I am learning the ELK Stack for log analysis and one thing I can't digest is whenever we have to populate index, the JSON format we have to use is not really a JSON.
For Eg:
Following is from tutorialspoint website, to populate a "Schools" index with
"School" type for bulk population :-
{ "index":{ "_index":"schools", "_type":"school", "_id":"1" }}
{"name":"Central School", "description":"CBSE Affiliation", "street":"Nagan","city":"paprola", "state":"HP", "zip":"176115", "location"[31.8955385, 76.8380405],"fees":2000, "tags":["Senior Secondary", "beautiful campus"], "rating":"3.5"}
{ "index":{ "_index":"schools", "_type":"school", "_id":"2" }}
{ "name":"Saint Paul School", "description":"ICSE Afiliation", "street":"Dawarka", "city":"Delhi", "state":"Delhi", "zip":"110075","location":[28.5733056, 77.0122136], "fees":5000,"tags":["Good Faculty", "Great Sports"], "rating":"4.5"}
Now I do understand space is the delimiter here and without giving one entity (index info or school info) in one line,it will return with parsing error. But according to my understanding and this (for reference), I think we do need a top level Key(Schools in below) with array ([ ]) of set of data(index info and School info) in above maybe like below :
{"Schools" : [
{
{"index":{ "_index":"schools", "_type":"school", "_id":"1" }},
{"name":"Central School", "description":"CBSE Affiliation", "street":"Nagan","city":"paprola", "state":"HP", "zip":"176115", "location"[31.8955385, 76.8380405],"fees":2000, "tags":["Senior Secondary", "beautiful campus"], "rating":"3.5"}
},
{
{ "index":{ "_index":"schools", "_type":"school", "_id":"2" }},
{ "name":"Saint Paul School", "description":"ICSE Afiliation", "street":"Dawarka", "city":"Delhi", "state":"Delhi", "zip":"110075","location":[28.5733056, 77.0122136], "fees":5000,"tags":["Good Faculty", "Great Sports"], "rating":"4.5"}
}
]}
I also tried changing the type of payload in POSTMAN from JSON(applicaton/json) to TEXT but it gives back an error stating that former payload is required.
I am clearly missing something here. It will be very helpful if someone can reason above behaviour.
Related
I just discovered glom and the tutorial makes sense, but I can't figure out the right spec to use for chrome BrowserHistory.json entries to create a data structure grouped by client_id or if this is even the right use of glom. I think I can accomplish this using other methods by looping over the json, but was hoping to learn more about glom and its capabilities.
The json has Browser_History with a list for each history entry as follows:
{
"Browser_History": [
{
"favicon_url": "https://www.google.com/favicon.ico",
"page_transition": "LINK",
"title": "Google Takeout",
"url": "https://takeout.google.com",
"client_id": "abcd1234",
"time_usec": 1424794867875291
},
...
I'd like a data structure where everything is grouped by the client_id, like with the client_id as the key to a list of dicts, something like:
{ 'client_ids' : {
'abcd1234' : [ {
"title" : "Google Takeout",
"url" : "https://takeout.google.com",
...
},
...
],
'wxyz9876' : [ {
"title" : "Google",
"url" : "https://www.google.com",
...
},
...
}
}
Is this something glom is suited for? I've been playing around with it and reading, but I can't seem to get the spec correct to accomplish what I need. Best I've got without error is:
with open(history_json) as f:
history_list = json.load(f)['Browser_History']
spec = {
'client_ids' : ['client_id']
}
pprint(glom(data, spec))
which gets me a list of all the client_ids, but I can't figure out how to group them together as keys rather than have them as a big list. any help would be appreciated, thanks!
This should do the trick although I'm not sure if this is the most "glom"-ic way to achieve this.
import glom
grouping_key = "client_ids"
def group_combine (existing,incoming):
# existing is a dictionary used for accumulating the data
# incoming is each item in the list (your input)
if incoming[grouping_key] not in existing:
existing[incoming[grouping_key]] = []
if grouping_key in incoming:
existing[incoming[grouping_key]].append(incoming)
return existing
data ={ 'Browser_History': [{}] } # your data structure
fold_spec = glom.Fold(glom.T,init = dict, op = group_combine )
results = glom.glom(data["Browser_History"] ,{ grouping_key:fold_spec })
I'm trying to get the titles of reddit posts from a subreddit in React, using Axios for fetching.
I can fetch the JSON from here, and would like to get the data of each object (post/comment) separately (so I can show titles, post text and the like in the render() part of the component).
Here's the first lines of that JSON, pretty-printed:
{
"kind": "Listing",
"data": {
"modhash": "",
"dist": 27,
"children": [
{
"kind": "t3",
"data": {
"approved_at_utc": null,
"subreddit": "reactjs",
"selftext": "Previous two threads - [June 2019](https:\/\/www.reddit.com\/r\/reactjs\/comments\/bvxng8\/beginners_thread_easy_questions_june_2019\/) and [May 2019](https:\/\/www.reddit.com\/r\/reactjs\/comments\/bjgval\/beginners_thread_easy_questions_may_2019\/). \n\nGot questions about React or anything else in its ecosystem? Stuck making progress on your app? Ask away! We\u2019re a friendly bunch. \n\nNo question is too simple. \ud83e\udd14\n\n---------------------------------------------\n\n\ud83c\udd98 **Want Help with your Code?** \ud83c\udd98\n\n* **Improve your chances** by putting a minimal example to either [JSFiddle](https:\/\/jsfiddle.net\/Luktwrdm\/) or [Code Sandbox](https:\/\/codesandbox.io\/s\/new). Describe what you want it to do, and things you've tried. Don't just post big blocks of code!\n\n* **Pay it forward!** Answer questions even if there is already an answer - multiple perspectives can be very helpful to beginners. Also there's no quicker way to learn than [being wrong on the Internet](https:\/\/xkcd.com\/386\/). \n\n**Have a question regarding code \/ repository organization?**\n\nIt's most likely answered within this [tweet](https:\/\/twitter.com\/dan_abramov\/status\/1027245759232651270?lang=en).\n\n---------------------------------------------------\n\n**New to React?**\n\n**Check out the sub's sidebar!**\n\n\ud83c\udd93 Here are great, **free** resources! \ud83c\udd93\n\n* [Create React App](https:\/\/facebook.github.io\/create-react-app\/)\n* [Read the **official** Getting Started page](https:\/\/reactjs.org\/docs\/getting-started.html) on the docs.\n* [\/u\/acemarke](https:\/\/www.reddit.com\/u\/acemarke)'s [suggested resources for learning React](http:\/\/blog.isquaredsoftware.com\/2017\/12\/blogged-answers-learn-react\/)\n* [Kent Dodd's Egghead.io course](http:\/\/kcd.im\/beginner-react)\n* [Tyler McGinnis' 2018 Guide](https:\/\/medium.freecodecamp.org\/a-comprehensive-guide-to-react-js-in-2018-ba8bb6975597)\n* [Codecademy's React courses](https:\/\/www.codecademy.com\/catalog\/language\/javascript)\n* [Scrimba's React Course](https:\/\/scrimba.com\/g\/glearnreact)\n* [Robin Wieruch's Road to React](https:\/\/roadtoreact.com)\n\n-----\n\nAny ideas\/suggestions to improve this thread - feel free to comment here!\n\n----\n_Finally, an ongoing thank you to all who post questions and those who answer them. We're a growing community and helping each other only strengthens it!_",
"author_fullname": "t2_2aun3ozb",
"saved": false,
"mod_reason_title": null,
"gilded": 0,
"clicked": false,
"title": "Beginner's Thread \/ Easy Questions (July 2019)",
"link_flair_richtext": [
],
Here's my component (without the render part, just what happens after that fetch is complete - i.e. the componentDidMount() function):
componentDidMount() {
axios.get(`https://www.reddit.com/r/reactjs.json`)
.then (
res => {
const posts = res.data.data.children.map(obj => obj.data);
this.setState({posts});
}
);
}
This works perfectly.
My question is - why?
My logic would go - get res -> look for data -> look for children -> look for data, then map() that data object into the data object for posts.
Instead, what seems to work is - get res -> look for data -> look for data -> look for children, then do the mapping.
That's how axios parses the response json object. They store it under their own personal data key. It just so happens that within the json object provided by Reddit, they also have a field called data that holds the children array you want.
It might be more helpful to name your variables like so:
componentDidMount() {
axios.get(`https://www.reddit.com/r/reactjs.json`)
.then((res) => {
const redditJson = res.data
const posts = redditJson.data.children.map(obj => obj.data)
this.setState({
posts: posts
})
})
}
I wish to be able to access the same data from multiple keys in a JSON file.
"TypeA","TypeB","TypeC": "These types are not supported",
"TypeR": "Vroom vroom!"
The actual problem is more complex, the data is a serialized object. But this is the current crux of the issue.
You could have something like:
{
"TypeA" : "COMMON_TYPE_VALUE",
"TypeB" : "COMMON_TYPE_VALUE",
"TypeC" : "COMMON_TYPE_VALUE",
"COMMON_TYPE_VALUE" : "here is the common value"
}
You'd have to know all the common keys up front of course...
Sounds like you might need to make it relational.
"Types": [
"These types are not supported",
"Vroom vroom!"
],
...
"TypeA": 0,
"TypeB": 0,
"TypeC": 0,
"TypeR": 1
Maybe not the best structure but it highlights the idea.
Let's say my JSON looks like this (example provided here) -
{
"year" : 2013,
"title" : "Turn It Down, Or Else!",
"info" : {
"directors" : [
"Alice Smith",
"Bob Jones"
],
"release_date" : "2013-01-18T00:00:00Z",
"rating" : 6.2,
"genres" : [
"Comedy",
"Drama"
],
"image_url" : "http://ia.media-imdb.com/images/N/O9ERWAU7FS797AJ7LU8HN09AMUP908RLlo5JF90EWR7LJKQ7##._V1_SX400_.jpg",
"plot" : "A rock band plays their music at high volumes, annoying the neighbors.",
"rank" : 11,
"running_time_secs" : 5215,
"actors" : [
"David Matthewman",
"Ann Thomas",
"Jonathan G. Neff"
]
}
}
I would like to query all movies where genres contains Drama.
I went through all of the examples but it seems that I can query only on hash key and sort key. I can't have JSON document as key itself as that is not supported.
You cannot. DynamoDB requires that all attributes you are filtering for have an index.
As you want to query independently of your main index, you are limited to Global Secondary Indexes.
The documentation lists on what kind of attributes indexes are supported:
The index key attributes can consist of any top-level String, Number, or Binary attributes from the base table; other scalar types, document types, and set types are not allowed.
Your type would be an array of Strings. So this query operation isn't supported by DynamoDB at this time.
You might want to consider other NoSQL document based databases which are more flexible like MongoDB Atlas, if you need this kind of querying functionality.
String filterExpression = "coloumnname.info.genres= :param";
Map valueMap = new HashMap();
valueMap.put(":param", "Drama");
ItemCollection scanResult = table
.scan(new ScanSpec().
withFilterExpression(filterExpression).
withValueMap(valueMap));
One example that I took from AWS Developer Forums is as follows.
We got some hints for you from our team. Filter/condition expressions for maps have to have key names at each level of the map specified separately in the expression attributeNames map.
Your expression should look like this:
{
"TableName": "genericPodcast",
"FilterExpression": "#keyone.#keytwo.#keythree = :keyone",
"ExpressionAttributeNames": {
"#keyone": "attributes",
"#keytwo": "playbackInfo",
"#keythree": "episodeGuid"
},
"ExpressionAttributeValues": {
":keyone": {
"S": "podlove-2018-05-02t19:06:11+00:00-964957ce3b62a02"
}
}
}
I have an index with a couple of fields of type Edm.String and Collection(Edm.String). I want to have another index with the same fields plus another field of type Edm.Double. When I create such an index and try to add the same values (plus the newly added Edm.Double value) as I did to the first index, I'm getting the following error:
{
"error": {
"code": "",
"message": "The request is invalid. Details: parameters : An unexpected 'StartArray' node was found when reading from the JSON reader. A 'PrimitiveValue' node was expected.\r\n"
}
}
Does anyone know what this error means? I tried looking for it on the Internet but I couldn't find anything related to my situation. A sample request I'm sending to the new index looks like this:
POST https://myservicename.search.windows.net/indexes/newindexname/docs/index?api-version=2016-09-01
{
"value": [{
"#search.action": "upload",
"keywords": ["red", "lovely", "glowing", "cute"],
"name": "sample document",
"weight": 0.5,
"id": "67"
}]
}
The old index is the same but it doesn't have the "weight" parameter.
Edit: I created the index using the portal, so I don't have the exact JSON to create the index, but the fields are roughly like this:
Field Type Attributes Analyzer
---------------------------------------------------------------------------------------
id Edm.String Key, Retrievable
name Edm.String Searchable, Filterable, Retrievable Eng-Microsoft
keywords Collection(Edm.String) Searchable, Filterable, Retrievable Eng-Microsoft
weight Edm.Double Filterable, Sortable
The reason I got the error was because I made a mistake and was trying to send a Collection(Edm.String) when the actual type on the index was Edm.String.