I need to be able to access all the available JSON data, the problem is that a lot of it is nested.
I currently have this query.
SELECT * FROM `system_log` WHERE entry->"$[0]" LIKE "%search_term%";
I need instead of entry->"$[0]", something like entry->"$*"
I think the arrow syntax is short for JSON_EXTRACT which I think would mean that a solution for extract would work for the arrow syntax.
{
" Name": {
"after": "Shop",
"before": "Supermarket"
}
}
This is an example of my JSON data and as you can see there are multiple levels to it meaning that entry->"$[0]" won't catch it.
version 8.0.19 of SQL
What I've tried so far is entry->"$[0]" and then prepending [0] after, but this solution does not seem very dynamic as the JSON data could get deeper and deeper.
JSON_SEARCH() won't work for the search you describe, because JSON_SEARCH() only searches for full string matches, not wildcards.
If you truly cannot predict the structure of your JSON, and you just want to find if the pattern '%search_term%' appears anywhere, then just treat the whole JSON document as a string, and use LIKE:
SELECT * FROM `system_log` WHERE entry LIKE "%search_term%";
If you have more specific search requirements, then you'll have to come up with a way to predict the path to the value you're searching for in your JSON document. That's something I cannot help you with, because I don't know your usage of JSON.
Related
I am using nodejs.
I am having the following problem -
There is a very big json object which i want to map to a much smaller object with a specific format which i will be using later on.
I want to have specific schemas which i will be able to use on that big object, each schema will have its own purpose. I want to be able to customize the structure via those schemas and no where else.
I was searching for a library which can help me with that.
I found one via npm which is called deep-map.
https://www.npmjs.com/package/deep-map
I played with it for a bit and it seem to answer my basic needs.
But i also going to be needing to do some more complex mapping.
A simplified example -
"testObj": { "myArr": [ {"type": "x", name:"test1"}, {"type": "y", name:"test2"}] }
and i need to look in myArr only for the name for which type equals x.
So basically i need some sort of for each loop and if term in between.
Since deep-map uses lodash/template i thought maybe there is a way to use the capabilities of lodash to solve this problem but so far i didnt find how to combine those two ( lodash + deep-map ) to solve the more complex mapping.
I am also open to other libraries which might help me with this problem.
Lodash has an at method that will do the simple cases of this. It lets you specify things like field.subfieldThatsAnArray[1].subsubfield and get the value.
For less of a "built in your own" solution, there's a couple bits of tech that might be interesting here too:
JSONPath
SelectTransform
There's some others out there, but JSONPath is used some other places in the space, and SelectTransform just looks cool.
Here is my JSON code.
{
"user_email": "{User.Email}",
"activity_date": "{Lead.LastAction.Date}",
"record_id": "{Lead.Id}-{Lead.LastAction.Date}",
"action_type": "{Lead.LastAction}",
"milestone": "{Lead.Milestone}",
"date_added": "{Lead.Date}"
}
Is it possible to add calculations in the code?
For example, can I add a line where the date_added is subtracted from activity_date?
No: JSON is a way to transport JS Objects.
You can do that while you format the JSON in your native language ( for example in PHP or JS serverside), basically creating the JSON object with the result of the calculation.
In JSON just by itself you cannot do that, it's just a data format, it's totally passive, like a text file. (If you happen to use JSONP, then the story would be a bit different, it might be possible, but using JSONP to do such things would step into area of 'hack/exploit' and it probably should not be used in that way:) )
However, I see you are using not only JSON - there is some extra markup like {User.Email}. This is totally outside JSON spec so clearly you are using some form text-templating engine. These can be quite intelligent at times. Check that path, see which one you are using, see what are its features, maybe you can write a custom function or expression to do that subtraction for you. Maybe, just maybe, it's as easy as
"inactivity_period": "{Lead.LastAction.Date - Lead.Date}"
or
"inactivity_period": "{myFunctionThatIWrote(Lead.LastAction.Date, Lead.Date)}"
but that all depends on the templating engine.
I run a request in Jmeter which response looks something like this:
{
"Items":[
{
"Available":3,
"Info":[
{
"Sample1":1,
"Sample2":33,
"Sample3":50,
"Sample4":"asd",
"Sample5":88,
"Sample6":null,
"Sample7":null,
"Sample8":null,
"Sample9":35,
"Sample0":35
}
]
}
]
}
And my goal is to go through the list of items (in my sample there is only one but there can be more) and if 'Available' is greater than 0 then save some values from 'Info' into a variable to use them for the next request.
Right now my solution is that I added JSON path postprocessor and there I separate the values like this:
$.Items[?(#.Available > 0)].Info[0].Sample1[0];
$.Items[?(#.Available > 0)].Info[0].Sample2[0];
$.Items[?(#.Available > 0)].Info[0].Sample3[0]...
but obviously this is not a very beautiful solution and I also think that this will take too much resource if I have to do it many times.
So my question is that is it somehow possible to separate the
$.Items[?(#.Available > 0)].Info[0]
element and then process it to get the fields I need?
I believe you can do it in a single JSON Path query using an asterisk - * - which stands for wildcard character like:
$.Items[?(#.Available > 0)].Info[0].*
References:
JSON Path Operators
Advanced Usage of the JSON Path Extractor in JMeter
After I solved the problem for myself, then I completely forgot about this question. But I'll post my solution now if someone should be struggling with the same problem.
I'm sure there are ways how to solve this problem using JMeter but it got way too complicated for me to manipulate JSON with regular expressions and also I felt that this is a little ridiculous since there are so nice tools for this. So I wrote a little Java script using GSON, packed it into a .jar and added it to JMeter.
The problem I'm facing here is that I have a blob of text which contains structured data (in the form of a JSON payload) and I'm interested in extracting the value of one of the keys for a specific JSON instance, picture the structured data inside as the following:
"Item 1": {"key1":"item1_key1_value", "key2":"item1_key2_value", "key3":"item1_key3_value"}, "Item 2": {"key1":"item2_key1_value", "key2":"item2_key2_value", "key3":"item2_key3_value"}
What I would like to use is use regex to grab item1_key2_value for instance. The keys all have the same name but the items are different. So I know which key for which Item I need but am not quite sure of the regex to retrieve that value. I've tried a few approaches to some basic matching but was wondering if any other more experienced regex users could direct me a bit here and explain what I'm doing wrong
1(.)(?=item1_key2_value.) will match a chunk of data from here but I'm not sure of the best way to reduce it to the value that I need.
The regex syntax for JSON is clearly specified at http://www.json.org. If you scroll down a little to where it says "A string is a sequence of", you will find the proper string structure.
Assuming the string follows the correct JSON structure, you could use
"key2"\s*:\s*"((\\.|[^\\"])*)"
where \s means whitespace and * means 0 or more times. \\ means a slosh (backslash) character and can be followed by . (any character). If it does not encounter a slosh, then it instead looks for [^\\"], which means not slosh nor quote.
If you want to be a little more strict to the exact JSON form, you could try
"key2"\s*:\s*"((\\["\\/bfnrtu]|[^\\"])*)"
which you can see follows the string form on the webpage more closely.
JSON appears to be a nice way to represent a complex data structure in plain text. If we think of this complex data structure as analogous to an OOP object - an instance of a class - then is there a commonly used JSON-like format that represents the class itself (just the data part - forget methods)? Can JSON itself be used for this?
To put it another way, if JSON encodes name-value pairs, what should I use if I want to encode only the names?
The reason I want this is that I am designing a protocol to use with jQuery (to which I am a complete novice by the way). The client will communicate to the server the structure of the JSON object it wants back, and the server will return a JSON object of that structure with the values added.
The key point is that it is the client that is in full control of what data fields (name-value pairs) the server returns. It's a bit different from all the examples of jQuery that I've found so far on the web where the client makes a request (which usually includes a very limited set of parameters, if any) and the server makes the decision as to what fields to return in the JSON reply.
(Obviously, what the client asks for must be congruent with the server's data model; if the server has an array of widgets each with its own price, the client can't ask for an array of prices each with its own widget.)
This must be a common problem, and I don't want to reinvent the wheel. I want to adopt a solution that is already in common use across the web.
Edit
I just found JSON Schema. This is not what I am looking for. It contains way more than I need.
Edit
I'm looking more for a 'this is how it is usually done' answer, rather than a 'you could try…' answer. (I can invent dozens of possible answers myself.)
To encode only names within JSON, you could use a key/value pair where the key is either the class name or just a key named 'values' - with the value being an array of strings that are the names to be returned by the server. For example:
{ 'class_name' : [ "name1", "name2", "name3" ] }
The server can then either detect the class name from the key used and return the supplied values for the names in the array if the class supports it or ignore if it does not.
I'm looking more for a 'this is how it is usually done' answer
There is no single "correct" way to do what you want. Many people have their implementation. It depends on various factors -- what you want to do, where you want to do, how efficiently you want it to do?
For simple structures I would prefer and suggest the answer given by #dbr9979.
For nested structures, you can have nested arrays. Something like:
{
"nestedfield1": {
"nestedfield11":["nestedfield111", "nestedfield112"],
"nestedfield12":["nestedfield121", "nestedfield122"],
"__SIMPLE_FIELDS__": ["simplefield13", "simplefield14"]
}
}
The point is, if the key is __SIMPLE_FIELDS__, the value is an array of simple fields (string, numbers etc..), else the key stands for the key in the object.
For something more complex, what I would suggest is you have predefined structures, that both the server and the client know of. This is particularly useful when you have to make multiple identical requests. Assign some unique number for each of them. Something like:
1 => <the structure above>
2 => ["simplefield1", "simplefield2" ..]
3 => etc .. etc
The server stores the above structure and the relevant number in the database or something. And now, as it may be obvious by now, client sends across the id of the required structure, and the server responds in the appropriate fashion.
I think what you meant by this:
the client that is in full control of what data fields (name-value pairs) the server returns.
is like the difference between SELECT * FROM Bags and SELECT color, price FROM Bag in SQL. Am I interpreting you correctly?
You could query with:
{
'resource': 'Bag',
'field_names': ['color', 'price']
}
which will return the response:
{
'status': 'success',
'result': [
{'color': 'red', 'price': 50},
{'color': 'blue', 'price': 45},
]
}
most likely though, you may not actually need your request to be a JSON object; I've seen implementations where the field names is taken from the query string, like http://foo.com/bag?fields=color,price
I was looking for Partial Response.
RESTful API Design: can your API give developers just the information they need? explains it all and gives examples from LinkedIn, Facebook, and Google. Google and Facebook both have similar approaches. Here's how Lie Ryan's example would look using Google's approach:
url?fields=status,result(color,price)
Since Google and Facebook are behind this, I would not be surprised to see this become a de facto standard.
In my case I am likely to run into a length limitation on the URL and so have to use POST instead, but this is an excellent starting point for me.