I am trying to parse below JSON using YAJL. YAJLGEN generated below data structure but the issue i am facing is the number of arrays ex: KEY, CUSTOMER are not fixed. These arrays are returned for each field in the response. I am trying to avoid defining an array for each field from the response.
Could you please, advise if there is a better way to read the below json and parse dyanic arrays. I tried using "yajl_array_loop", "yajl_array_elem" but i couldn't able to make it work in my program for some reason. Thank is in advance.
{
"errstatus": 400,
"errors": {
"Key": [
"The Key field is required."
],
"Customer": [
"The Customer field is required."
]
}
}
dcl-ds jsonDoc qualified;
errstatus packed(3) inz(0);
dcl-ds ERRORS;
num_KEY int(10) inz(0);
KEY varchar(37) inz('') dim(1);
num_CUSTOMER int(10) inz(0);
CUSTOMER varchar(43) inz('') dim(2);
end-ds;
end-ds;
If yajl is not working then it is probably not a good choice for your case. If your JSON is not hundreds of megabyte big then you may try a DOM like approach like using noxDB (https://github.com/sitemule/noxDB). It reads the whole JSON into memory and you can evaluate the in-memory JSON the way you want. Seems like a much better approach for your situation.
Related
I am working on a Power Automate flow to get a JSON file from SharePoint and Parse it. On one of my previous questions I received a solution that worked with a testing JSON file. However when I ran a couple of tests with some JSON files that I need to use, the Parse JSON step gives out errors regarding "missing" required properties.
Basically, the JSON file's arrays do not always have exactly the same elements (properties). For example (below) the element "minimun_version" does not always appear under the element "link", like the image below
and because of this syntax I get the errors below
How can I Parse such a JSON file successfully?
Any idea or suggestion will help me get unstuck.
Thank you
You can allow null values in your Parse Json schema by simply adding that to the schema. April Dunnam has a nice article about this approach:
https://www.sharepointsiren.com/2018/10/flow-parse-json-null-error-fix/
I assume you have something like below in your schema?
"minimum_version": {
"type": "number"
}
You can change that to this to allow null values
"minimum_version": {
"type": [
"number",
"null"
]
}
I was tasked to filter through data using the elasticsearch-kibana stack. My data comes in JSON format like so,
{
"instagram_account": "christywood",
"full_name": "Christy Wood",
"bio": "Dog mom from Houston",
"followers_count": 1000,
"post_count": 1000,
"email": christy#gmail.com,
"updated_at": "2022-07-18 02:06:29.998639"
}
However, when I try to import the data into Kibana, I get an error that states my data does not match the default GROK pattern.
I tried writing my own GROK, using the list of acceptable syntaxes in this repo, but the debugger always parses the key rather than the actual desired value. For instance, the GROK pattern
%{USERNAME:instagram_account}
returns this undesired data structure
{
"instagram_account": "instagram_account"
}
I've tried a couple other syntax options, but it seems that my debugger always grabs the key and not the actual value. No wonder elastic search cannot make sense of my data!
I've searched for examples, but I am unable to find any that use JSON data. To be fair, I'm very unfamiliar with GROK and would like to understand what % , /n , and other delimiters mean within this context.
Please tell me what I'm doing wrong, and point me in the right direction. Thank you!
I´m getting a json from an application with a couple of nested subdocuments. Some of those documents are optional and not present all the time. I´m wondering if there is a best practice how to handel this.
e.g. (The document is just an example, the real one looks differnt but I can´t post it, the Example is copied from: How to represent sub-documents in JSON array as Java Collection using Jackson?): The Adreess subdocument is not present in every document I receive.
{
"attributes": {
"type": "Lead",
"url": "/services/data/v30.0/sobjects/Lead/00Qi000000Jr44XEAR"
},
"Id": "00Qi000000Jr44XEAR",
"Name": "Kristen Akin",
"Address": {
"city": null,
"country": "USA",
"state": "CA",
"stateCode": null,
"street": null
},
"Phone": "(434) 369-3100"
}
Currently I´m receiving the data in the worst possible way I can imagine with a differnt type, which is like:
{
"attributes": {
"type": "Lead",
"url": "/services/data/v30.0/sobjects/Lead/00Qi000000Jr44XEAR"
},
"Id": "00Qi000000Jr44XEAR",
"Name": "Kristen Akin",
"Address": "",
"Phone": "(434) 369-3100"
}
I want to suggest better ways and I´m wondering whats the best one?
Leaving the adress subdocument out completely
receiving "Adress: null"
receiving Adress: {}
receiving Adress: {"city": null, "country": null, ...}
anything else
Personally I would go with Nr. 3 because I still get a (sub)document and can treat it the usual way. Does anythin speak against it or are there any best practices for this situation?
Thanks in advance.
Best regards.
Go with 3.
Leaving the adress subdocument out completely
Would work for many deserialization tools, but it is hard to identify the structure and identify if something is missing on debugging easily
receiving "Adress: null"
Would work for many deserialization tools, but it is not a good practice to deliver null for more complex attributes like arrays or objects. You cannot identify, that this is a complex object easily.
receiving Adress: {}
It is a good practice to deliver empty arrays if they are empty and empty objects, if they are empty. You can identify that there could be a complex object but it is not available here. Please go with this solution
receiving Adress: {"city": null, "country": null, ...}
Don't do this. It gives you more details for the complex object, but you cannot identify easily if the address was not added on purpose or if the API partner sends incomplete address data by accident or if incomplete data is valid on their side.
I always differentiate between values which are:
set but empty: We usually interpret these values as valid values which are intended to be empty, like an empty address book, which may contain no entries at all.
undefined: usually this is an optional value. The application has to handle if it needs the data from somewhere else.
null: setting a value intentionally to null means to invalidate the value. We often use this to reset the data. In case of the address book means: there is no address book at all, even no empty one.
I would prefer these options:
1.: if it is left out, it is undefined and means that it is up to the application to handle undefined values. Especially for optional values, you should be aware of handling undefined values.
3.: if it is empty, you still have a valid address book, but an empty one, which makes the handling in code easier.
What I would avoid:
4.: You get an valid address with invalid data, so you have to deep-check if the address is usable, which increases the efforts on validation, so I would not use this option.
5.: changing the data type to "" is also bad because for typed languages it will make it hard to parse because it expects an object but receives a string.
I am facing an issue with parsing a json using Jmespath based on a string condition. I would want to get values of a key based on condition on string value of another key. Both of them are in same hierarchy of a given dictionary. However , the parent key of the dictionary is a variable.I am familiar with jq but jmespath is new to me. I am using it , as the current project depends on it. Hence no much choice on changing the parser.
Sample Json below :-
{
"people": {
"a": {"First":"James", "last": "1"},
"b": {"First":"Jacob", "last": "2"},
"c": {"First":"Jayden", "last": "3"},
"d": {"First":"different", "last" : "4"}
}
}
I would want to get the last's value where First's value starts with "J".
I have tried referring to articles provided in official site at http://jmespath.org/tutorial.html. However , most of them concentrate on a standard key structure and not much on a variable key dictionary structure. Hence am unable to write a jmespath query for a given json.
The jq equivalent for achieving the intended result is given below :-
.people | .[] | select (.First | startswith("J")) | .last
The closest jmespath query , that I could logically arrive at based on my understanding is :-
people.*[?starts_with(First,`J`)].last
However , the above query returns blank result.
Expected output is
"d","e","f"
I am unable to understand where I am going wrong.
It would be nice if someone can point me to a good article or help me find the solution to the above issue.
Thanks alot
UPDATE :
The solution is to use values(#).
Reference link
https://github.com/jmespath/jmespath.site/issues/24
So one of the possible solution for the above ask is
people.values(#)[?starts_with(First,`J`)].last
So for any variable key , we can use values(#) to filter projections further down the structure.
In Postgres I have a table like this:
CREATE TABLE storehouse
(
user_id bigint NOT NULL,
capacity integer NOT NULL,
storehouse json NOT NULL,
last_modified timestamp without time zone NOT NULL,
CONSTRAINT storehouse_pkey PRIMARY KEY (user_id)
)
And storehouse.storehouse is storing data like this:
{
"slots":[
{
"slot" : 1,
"id" : 938
},
{
"slot" : 2,
"id" : 127
},
]
}
The thing is, I want to update storehouse.storehouse.slots[2], but I do not have an idea on how to do it.
I know how to alter the entire storehouse.storehouse field, but I am wondering since Postgres supports json type, it should support partial modify, otherwise that would be no difference between json type and text type. (I know json type also has type validation which is differ to text)
JSON indexing and partial updates are not currently supported. The JSON support in PostgreSQL 9.2 is rudimentary, limited to validating JSON and to converting rows and arrays to JSON. Internally, json is indeed pretty much just text.
There's ongoing work for enhancements like partial updates,indexing, etc. No matter what, though, PostgreSQL won't be able to avoid rewriting the whole row when part of a JSON value changes, because that's inherent to the MVCC model of concurrency. The only way to make that possible would be to split JSON values out into multiple tuples in a side relation, like TOAST tables - something that's possible, but likely to perform poorly and that's very far from being considered at this point.
As Chris Travers points out, you can use PL/V8 functions or functions in other languages with json support like Perl or Python to extract values, then create expression indexes on those functions.
Since PostgreSQL 9.5, there a function called jsonb_set which takes as input parameters:
a JSON object
an array indicating the path (keys and subkeys)
the new value to be stored (also a JSON object)
Example:
# SELECT jsonb_set('{"name": "James", "contact": {"phone": "01234 567890", "fax": "01987 543210"}}'::jsonb,
'{contact,phone}',
'"07900 112233"'::jsonb);
jsonb_replace
--------------------------------------------------------------------------------
{"name": "James", "contact": {"fax": "01987 543210", "phone": "07900 112233"}}
(1 row)