I am using nodejs.
I am having the following problem -
There is a very big json object which i want to map to a much smaller object with a specific format which i will be using later on.
I want to have specific schemas which i will be able to use on that big object, each schema will have its own purpose. I want to be able to customize the structure via those schemas and no where else.
I was searching for a library which can help me with that.
I found one via npm which is called deep-map.
https://www.npmjs.com/package/deep-map
I played with it for a bit and it seem to answer my basic needs.
But i also going to be needing to do some more complex mapping.
A simplified example -
"testObj": { "myArr": [ {"type": "x", name:"test1"}, {"type": "y", name:"test2"}] }
and i need to look in myArr only for the name for which type equals x.
So basically i need some sort of for each loop and if term in between.
Since deep-map uses lodash/template i thought maybe there is a way to use the capabilities of lodash to solve this problem but so far i didnt find how to combine those two ( lodash + deep-map ) to solve the more complex mapping.
I am also open to other libraries which might help me with this problem.
Lodash has an at method that will do the simple cases of this. It lets you specify things like field.subfieldThatsAnArray[1].subsubfield and get the value.
For less of a "built in your own" solution, there's a couple bits of tech that might be interesting here too:
JSONPath
SelectTransform
There's some others out there, but JSONPath is used some other places in the space, and SelectTransform just looks cool.
Related
I need to be able to access all the available JSON data, the problem is that a lot of it is nested.
I currently have this query.
SELECT * FROM `system_log` WHERE entry->"$[0]" LIKE "%search_term%";
I need instead of entry->"$[0]", something like entry->"$*"
I think the arrow syntax is short for JSON_EXTRACT which I think would mean that a solution for extract would work for the arrow syntax.
{
" Name": {
"after": "Shop",
"before": "Supermarket"
}
}
This is an example of my JSON data and as you can see there are multiple levels to it meaning that entry->"$[0]" won't catch it.
version 8.0.19 of SQL
What I've tried so far is entry->"$[0]" and then prepending [0] after, but this solution does not seem very dynamic as the JSON data could get deeper and deeper.
JSON_SEARCH() won't work for the search you describe, because JSON_SEARCH() only searches for full string matches, not wildcards.
If you truly cannot predict the structure of your JSON, and you just want to find if the pattern '%search_term%' appears anywhere, then just treat the whole JSON document as a string, and use LIKE:
SELECT * FROM `system_log` WHERE entry LIKE "%search_term%";
If you have more specific search requirements, then you'll have to come up with a way to predict the path to the value you're searching for in your JSON document. That's something I cannot help you with, because I don't know your usage of JSON.
Here is my JSON code.
{
"user_email": "{User.Email}",
"activity_date": "{Lead.LastAction.Date}",
"record_id": "{Lead.Id}-{Lead.LastAction.Date}",
"action_type": "{Lead.LastAction}",
"milestone": "{Lead.Milestone}",
"date_added": "{Lead.Date}"
}
Is it possible to add calculations in the code?
For example, can I add a line where the date_added is subtracted from activity_date?
No: JSON is a way to transport JS Objects.
You can do that while you format the JSON in your native language ( for example in PHP or JS serverside), basically creating the JSON object with the result of the calculation.
In JSON just by itself you cannot do that, it's just a data format, it's totally passive, like a text file. (If you happen to use JSONP, then the story would be a bit different, it might be possible, but using JSONP to do such things would step into area of 'hack/exploit' and it probably should not be used in that way:) )
However, I see you are using not only JSON - there is some extra markup like {User.Email}. This is totally outside JSON spec so clearly you are using some form text-templating engine. These can be quite intelligent at times. Check that path, see which one you are using, see what are its features, maybe you can write a custom function or expression to do that subtraction for you. Maybe, just maybe, it's as easy as
"inactivity_period": "{Lead.LastAction.Date - Lead.Date}"
or
"inactivity_period": "{myFunctionThatIWrote(Lead.LastAction.Date, Lead.Date)}"
but that all depends on the templating engine.
Using PIG (0.14), i'm interested in the following use-case: I wish to process my raw JSON into multiple output directories based upon their key and store the result (aggregated data) as JSON. The JSON has an evolving (dynamic) schema which is read in with elephant-bird, and (so-far) has not caused any problems.
I can either store the output in the correct directories (using MultiStorage) or as JSON (using JsonStorage) but not both. As far as i can tell, there is no publicly available UDF for this purpose.
Have I missed something, or is it just a case of writing my own UDF to perform this? This seems like a simple use-case and I would have thought would have been supported.
For those who are looking for an answer to this; a UDF is required.
It is possible (and relatively straight forward) to combine the piggybank UDFs of JsonStorage and MultiStorage to create a pseudo "JsonMultiStorage" class.
I've looked at the very similarly titled post (Is there a C equivalent to Perls' Dumper() method in Data::Dumper?), regarding a C equivalent to Data::Dumper::Dumper();. I have a similar question for the Go language.
I'm a Perl Zealot by trade, and am a progamming hobbyist, and make use of Data::Dumper and similar offspring literally hundreds of times a day. I've taken up learning Go, because it looks like a fun and interesting language, something that will get me out of the Perl rut I'm in, while opening my eyes to new ways of doing stuffz... One of the things I really want is something like:
fmt.Println(dump.Dumper(decoded_json))
to see the resulting data structure, like Data::Dumper would turn the JSON into an Array of Hashes. Seeing this in Go, will help me to understand how to construct and work with the data. Something like this would be considered a major lightbulb moment in my learning of Go.
Contrary to the statements made in the C counterpart post, I believe we can write this, and since I'll be passing Dumper to Println, after compilation what ever JSON string or XML page I pass in and decode. I should be able to see the result of the decoding, in a Dumper like state... So, does any more know of anything like this that exists? or maybe some pointers to getting something like this done?
Hi and welcome to go I'm former perl hacker myself.
As to your question the encoding/json package is probably the closest you will find to a go data pretty printer. I'm not sure you really need it though. One of the reasons Data::Dumper was awesome in perl is because many times you really didn't know the structure of the data you were consuming without visually inspecting it. With go though everything is a specific type and every specific type has a specific structure. If you want to know what the data will look like then you probably just need to look at it's definition.
Some other tools you should look at include:
fmt.Println("%#v", data) will print the data in go-syntax form.
fmt.Println("%T", data) will print the data's type in go-syntax
form.
More fmt format string options are documented here: http://golang.org/pkg/fmt/
I found a couple packages to help visualize data in Go.
My personal favourite - https://github.com/davecgh/go-spew
There's also - https://github.com/tonnerre/golang-pretty
I'm not familiar with Perl and Dumper, but from what I understand of your post and the related C post (and the very name of the function!), it outputs the content of the data structure.
You can do this using the %v verb of the fmt package. I assume your JSON data is decoded into a struct or a map. Using fmt.Printf("%v", json_obj) will output the values, while %+v will add field names (for a struct - no difference if its a map, %v will output both keys and values), and %#v will output type information too.
JSON appears to be a nice way to represent a complex data structure in plain text. If we think of this complex data structure as analogous to an OOP object - an instance of a class - then is there a commonly used JSON-like format that represents the class itself (just the data part - forget methods)? Can JSON itself be used for this?
To put it another way, if JSON encodes name-value pairs, what should I use if I want to encode only the names?
The reason I want this is that I am designing a protocol to use with jQuery (to which I am a complete novice by the way). The client will communicate to the server the structure of the JSON object it wants back, and the server will return a JSON object of that structure with the values added.
The key point is that it is the client that is in full control of what data fields (name-value pairs) the server returns. It's a bit different from all the examples of jQuery that I've found so far on the web where the client makes a request (which usually includes a very limited set of parameters, if any) and the server makes the decision as to what fields to return in the JSON reply.
(Obviously, what the client asks for must be congruent with the server's data model; if the server has an array of widgets each with its own price, the client can't ask for an array of prices each with its own widget.)
This must be a common problem, and I don't want to reinvent the wheel. I want to adopt a solution that is already in common use across the web.
Edit
I just found JSON Schema. This is not what I am looking for. It contains way more than I need.
Edit
I'm looking more for a 'this is how it is usually done' answer, rather than a 'you could try…' answer. (I can invent dozens of possible answers myself.)
To encode only names within JSON, you could use a key/value pair where the key is either the class name or just a key named 'values' - with the value being an array of strings that are the names to be returned by the server. For example:
{ 'class_name' : [ "name1", "name2", "name3" ] }
The server can then either detect the class name from the key used and return the supplied values for the names in the array if the class supports it or ignore if it does not.
I'm looking more for a 'this is how it is usually done' answer
There is no single "correct" way to do what you want. Many people have their implementation. It depends on various factors -- what you want to do, where you want to do, how efficiently you want it to do?
For simple structures I would prefer and suggest the answer given by #dbr9979.
For nested structures, you can have nested arrays. Something like:
{
"nestedfield1": {
"nestedfield11":["nestedfield111", "nestedfield112"],
"nestedfield12":["nestedfield121", "nestedfield122"],
"__SIMPLE_FIELDS__": ["simplefield13", "simplefield14"]
}
}
The point is, if the key is __SIMPLE_FIELDS__, the value is an array of simple fields (string, numbers etc..), else the key stands for the key in the object.
For something more complex, what I would suggest is you have predefined structures, that both the server and the client know of. This is particularly useful when you have to make multiple identical requests. Assign some unique number for each of them. Something like:
1 => <the structure above>
2 => ["simplefield1", "simplefield2" ..]
3 => etc .. etc
The server stores the above structure and the relevant number in the database or something. And now, as it may be obvious by now, client sends across the id of the required structure, and the server responds in the appropriate fashion.
I think what you meant by this:
the client that is in full control of what data fields (name-value pairs) the server returns.
is like the difference between SELECT * FROM Bags and SELECT color, price FROM Bag in SQL. Am I interpreting you correctly?
You could query with:
{
'resource': 'Bag',
'field_names': ['color', 'price']
}
which will return the response:
{
'status': 'success',
'result': [
{'color': 'red', 'price': 50},
{'color': 'blue', 'price': 45},
]
}
most likely though, you may not actually need your request to be a JSON object; I've seen implementations where the field names is taken from the query string, like http://foo.com/bag?fields=color,price
I was looking for Partial Response.
RESTful API Design: can your API give developers just the information they need? explains it all and gives examples from LinkedIn, Facebook, and Google. Google and Facebook both have similar approaches. Here's how Lie Ryan's example would look using Google's approach:
url?fields=status,result(color,price)
Since Google and Facebook are behind this, I would not be surprised to see this become a de facto standard.
In my case I am likely to run into a length limitation on the URL and so have to use POST instead, but this is an excellent starting point for me.