Parsing JSON in SAS - json

Does anyone know how to convert the following JSON to table format in SAS? Appreciate in advance any help!
JSON
{
"totalCount": 2,
"facets": {},
"content": [
[
{
"name": "customer_ID",
"value": "1"
},
{
"name": "customer_name",
"value": "John"
}
],
[
{
"name": "customer_ID",
"value": "2"
},
{
"name": "customer_name",
"value": "Jennifer"
}
]
]
}
Desired Output
customer_ID
customer_name
1
John
2
Jennifer
Steps I've Taken
1- Call API
filename request "C:\path.request.txt";
filename response "C:\path.response.json";
filename status "C:\path.status.json";
proc http
url="http://httpbin.org/get"
method="POST"
in=request
out=response
headerout=status;
run;
2- I have the following JSON MAP file save:
{
"DATASETS": [
{
"DSNAME": "customers",
"TABLEPATH": "/root/content",
"VARIABLES": [
{
"NAME": "name",
"TYPE": "CHARACTER",
"PATH": "/root/content/name"
},
{
"NAME": "value",
"TYPE": "CHARACTER",
"PATH": "/root/content/value"
}
]
}
]
}
3- I use the above JSON Map file as follow:
filename jmap "C:\path.jmap.map";
libname cust json map=jmap access=readonly;
proc copy inlib=cust outlib=work;
run;
4- This generates a table like this, which is not what I need:
name
value
customer_id
1
customer_value
John
customer_id
2
customer_value
Jennifer

From where you are, you have a very trivial step to convert to what you want - PROC TRANSPOSE.
filename test "h:\temp\test.json";
libname test json;
data pre_trans;
set test.content;
if name='customer_ID' then row+1;
run;
proc transpose data=pre_trans out=want;
by row;
id name;
var value;
run;
You could also do this directly in the data step; there are advantages to going either way.
data want;
set test.content;
retain customer_ID customer_name;
if name='customer_ID' then customer_ID=input(value,best.);
else if name='customer_name' then do;
customer_name = value;
output;
end;
run;
This data step works okay for the example above - the proc transpose works better for more complex examples, as you only have to hardcode the one value.
I suspect you could do this more directly with a proper JSON map, but I don't usually do this sort of thing that way - it's easier for me to just get it into a dataset and then work with it from there.
In this case, SAS is getting tripped up by the double arrays with no content before the second array - if there was some (any) content there, it would parse more naturally. Since there's nothing for SAS to really judge what you want to do with that Content array, it just lets you do whatever you want with it - which is easy enough.

Related

Parse JSON object dynamically in Bigquery + dbt

I have a json message like below. I am using dbt and with Big query plug in. I need to create table dynamically in Big query
{
"data": {
"schema":"dev",
"payload": {
"lastmodifieddate": "2022-11-122 00:01:28",
"changeeventheader": {
"changetype": "UPDATE",
"changefields": [
"lastmodifieddate",
"product_value"
],
"committimestamp": 18478596845860,
"recordIds":[
"568069"
]
},
"product_value" : 20000
}
}
}
I need to create table dynamically with recordIds and changed fields. This field list changes dynamically whenever source sends update..
Expected output:
recordIds | product_value | lastmodifieddate |changetype
568069 | 20000 | 2022-11-122 00:01:28 |UPDATE
Thanks for your suggestions and help!.
JSON objects can be saved in a BigQuery table. There is no need to use dbt here.
with tbl as (select 5 row, JSON '''{
"data": {
"schema":"dev",
"payload": {
"lastmodifieddate": "2022-11-122 00:01:28",
"changeeventheader": {
"changetype": "UPDATE",
"changefields": [
"lastmodifieddate",
"product_value"
],
"committimestamp": 18478596845860,
"recordIds":[
"568069"
]
},
"product_value" : 20000
}
}
}''' as JS)
select *,
JSON_EXTRACT_STRING_ARRAY(JS.data.payload.changeeventheader.recordIds) as recordIds,
JSON_EXTRACT_SCALAR(JS.data.payload.product_value) as product_value,
Json_value(JS.data.payload.lastmodifieddate) as lastmodifieddate,
Json_value(JS.data.payload.changeeventheader.changetype) as changetype
from tbl
If the JSON is saved as string in a BigQuery table, please use PARSE_JSON(column_name) to convert the string to JSON first.

Extracting multiple values having same path in json using json map in sas

Dose anyone can help me get multiple values in a json having same path using a json map. Any help is appreciated. Thank you.
JSON
{
"totalCount": 2,
"facets": {},
"content": [
[
{
"name": "customer_ID",
"value": "1"
},
{
"name": "customer_name",
"value": "John"
}
]
]
}
JSON MAP
{
"DATASETS": [
{
"DSNAME": "customers",
"TABLEPATH": "/root/content",
"VARIABLES": [
{
"NAME": "name",
"TYPE": "CHARACTER",
"PATH": "/root/content/name"/*output as customer_ID*/
},
{
"NAME": "name",
"TYPE": "CHARACTER",
"PATH": "/root/content/name"/*output as customer_name*/
},
{
"NAME": "value",
"TYPE": "CHARACTER",
"PATH": "/root/content/value"/*output as 1*/
},
{
"NAME": "value",
"TYPE": "CHARACTER",
"PATH": "/root/content/value"/*output as John*/
}
]
}
]
}
When i use the above json map I get the output for name as only "customer_name", but i need both "customer_ID" and "customer_name" in the output.
Similarly i need both values of "value"
JSON is a hierarchy of name-value pairs. The JSON engine in SAS will take the "name" and assign as a variable name, and then populate with the value. In your JSON, there are two sets of name-values, one being the name of an intended variable, and another being its value. This is a common output scheme we find in GraphQL responses -- and these require a little manipulation to turn into 2-D data sets.
For your example, you could use PROC TRANSPOSE:
libname j json fileref=test;
proc transpose
data=j.content
out=want;
id name;
var value;
run;
Output:
customer_ customer_
Obs _NAME_ ID name
1 value 1 John
You can also do more manual seek/assignment by using DATA step to process what you see in the ALLDATA member in the JSON libname. In your example, SAS sees that as:
Obs P P1 P2 V Value
1 1 totalCount 1 2
2 1 facets 0
3 1 content 0
4 1 content 0
5 2 content name 1 customer_ID
6 2 content value 1 1
7 1 content 0
8 2 content name 1 customer_name
9 2 content value 1 John
Processing the ALLDATA member is not as friendly as using the relational data that the JSON engine can create, but I find with GraphQL responses that's what you need to do to get more control over the name, length, and type/format for output variables.

Updating Nested JSON Array with new key and value from another key

I have have a JSON file where I have IDs with tasks. Some tasks can be empty. I want to put the ID into the tasks where tasks are not empty.
[
{
"id": 1961126,
"tasks": [
{
"id": 70340700,
"title": "Test1",
},
{
"id": 69801130,
"title": "Test15A",
}
]
},
{
"id": 1961126,
"tasks": []
}
]
I would like to get the tasks array updated to look like
[
{
"id": 1961126,
"tasks": [
{
**"sId":1961126,**
"id": 70340700,
"title": "Test1",
},
{
**"sId":1961126,**
"id": 69801130,
"title": "Test15A",
}
]
},
{
"id": 1961126,
"tasks": []
}
]
I can't figure out how to get the id from the object into the nested array. Here is what I have come up with
jq 'map(.tasks[0]|select( . != null )|.sId = .id)' file.json
This is only pulling in the same id. I have tired to put in [].id but I get a error Cannot index number with string "id". I am still learning how to deal with nested arrays and objects.
Save the ID in a variable and add it as a new field to each array member.
jq 'map(.id as $sId | .tasks[] += {$sId})' file.json
Demo
Note #1: Get rid of the final , within each object (see the Demo), as it's not proper JSON.
Note #2: Object fields generally have no order, but if you want to have the propagated ID shown first, as seen in your expected output, you could try to replace += {$sId} (which by itself is shorthand for |= . + {$sId}) with |= {$sId} + . to flip the order of generation (Demo). Although there is no guarantee that it stays that way with further processing.

Pentaho Kettle: How to dynamically fetch JSON file columns

Background: I work for a company that basically sells passes. Every order that is placed by the customer will contain N number of passes.
Issue: I have these JSON event-transaction files coming into a S3 bucket on a daily basis from DocumentDB (MongoDB). This JSON file is associated to the relevant type of event (insert, modify or delete) for every document key (which is an order in my case). The example below illustrates a "Insert" type of event that came through to the S3 bucket:
{
"_id": {
"_data": "11111111111111"
},
"operationType": "insert",
"clusterTime": {
"$timestamp": {
"t": 11111111,
"i": 1
}
},
"ns": {
"db": "abc",
"coll": "abc"
},
"documentKey": {
"_id": {
"$uuid": "abcabcabcabcabcabc"
}
},
"fullDocument": {
"_id": {
"$uuid": "abcabcabcabcabcabc"
},
"orderNumber": "1234567",
"externalOrderId": "12345678",
"orderDateTime": "2020-09-11T08:06:26Z[UTC]",
"attraction": "abc",
"entryDate": {
"$date": 2020-09-13
},
"entryTime": {
"$date": 04000000
},
"requestId": "abc",
"ticketUrl": "abc",
"tickets": [
{
"passId": "1111111",
"externalTicketId": "1234567"
},
{
"passId": "222222222",
"externalTicketId": "122442492"
}
],
"_class": "abc"
}
}
As we see above, every JSON file might contain N number of passes and every pass is - in turn - is associated to an external ticket id, which is a different column (as seen above). I want to use Pentaho Kettle to read these JSON files and load the data into the DW. I am aware of the Json input step and Row Normalizer that could then transpose "PassID 1", "PassID 2", "PassID 3"..."PassID N" columns into 1 unique column "Pass" and I would have to have to apply a similar logic to the other column "External ticket id". The problem with that approach is that it is quite static, as in, I need to "tell" Pentaho how many Passes are coming in advance in the Json input step. However what if tomorrow I have an order with 10 different passes? How can I do this dynamically to ensure the job will not break?
If you want a tabular output like
TicketUrl Pass ExternalTicketID
---------- ------ ----------------
abc PassID1Value1 ExTicketIDvalue1
abc PassID1Value2 ExTicketIDvalue2
abc PassID1Value3 ExTicketIDvalue3
And make incoming value dynamic based on JSON input file values, then you can download this transformation Updated Link
I found everything work dynamic in JSON input.

Obtain a different JSON object structure in AngularJS

I'm Working on AngularJS.
In this part of the project my goal is to obtain a JSON structure after filling a form with some particulars values.
Here's the fiddle of my simple form: Fiddle
With the form I will do a query to KairosDB, that is my NoSql Database, I will query data from it by a JSON object. The form is structured in this way:
a Name
a certain Number of Tags, with Tag Id ("ch" for example) and tag value ("932" for example)
a certain Number of Aggregators to manipulate data coming from DB
Start Timestamp and End Timestamp (now they are static and only included in the final JSON Object)
After filling this form, with my code I'll obtain for example this JSON object:
{
"metrics": [
{
"tags": [
{
"id": "ch",
"value": "932"
},
{
"id": "ch",
"value": "931"
}
],
"aggregators": {
"name": "sum",
"sampling": [
{
"value": "1",
"unit": "milliseconds",
"type": "SUM"
}
]
}
}
],
"cache_time": 0,
"start_absolute": 123,
"end_absolute": 1234
}
Unfortunately, KairosDB accepts a different structure, and as you could see, Tag id "ch" doesn't hase an "id" string before, or for example, Tag values coming from the same tag id are grouped together
{
"metrics": [
{
"tags": {
"ch": [
"932",
"931"
]
},
"name": "AIENR",
"aggregators": [
{
"name": "sum",
"sampling": {
"value": "1",
"unit": "milliseconds"
}
}
]
}
],
"cache_time": 0,
"start_absolute": 1367359200000,
"end_absolute": 1386025200000
}
My question is: Is there a way to obtain the JSON structure like the one accepted by Kairos DB with an Angular JS form?. Thanks to everyone.
I've seen this topic as the one more similar to mine but it isn't in AngularJS.
Personally, I'd do the refactoring work in the backend - Have what ever server interfaces sends and receives data do the manipulation - Otherwise you'll end up needing to refactor your data inside Angular anywhere you want to use that dataset.
Where as doing it in the backend would put it in a single access point.
Of course, you could do it in Angular, just replace userString in the submitData method with a copy of the array and replace the tags section with data in the new format, and likewise refactor the returned result to the correct format when you get a reply.