I'm trying to parse json output from an API call. The output has an array of orders, and each order has an array of items. I want to parse the output such that I have a single CSV output of each individual item with its parent order ID.
So if a single order contains multiple items, I need the orderID repeated for each item in its order. I've read the jq documentation and dozens of samples, and I've tried some trial and error for hours. I'm SO confused as to how to do this.
I'm struggling very much with the jq parsing syntax. None of the examples are really helping, and I'm just confused. Here's the basics:
curl -s https://api.site.com/orders?page=1&pageSize=10 | jq '.'
A sample of the json is below.
{
"orders": [
{
"orderId": 217356098,
"items": [
{
"orderItemId": 327010821,
"lineItemKey": "1",
"sku": "AJC-C10S",
"name": "TestDescription",
"imageUrl": null,
"weight": null,
"quantity": 2,
"unitPrice": 106.85,
"taxAmount": null,
"shippingAmount": null,
"warehouseLocation": null,
"options": [],
"productId": null,
"fulfillmentSku": null,
"adjustment": false,
"upc": null,
"createDate": "2016-11-09T02:11:28.307",
"modifyDate": "2016-11-09T02:11:28.307"
},
{
"orderItemId": 327010822,
"lineItemKey": "1",
"sku": "AJC-C106",
"name": "AnotherTestDescription",
"imageUrl": null,
"weight": null,
"quantity": 2,
"unitPrice": 106.85,
"taxAmount": null,
"shippingAmount": null,
"warehouseLocation": null,
"options": [],
"productId": null,
"fulfillmentSku": null,
"adjustment": false,
"upc": null,
"createDate": "2016-11-09T02:11:28.307",
"modifyDate": "2016-11-09T02:11:28.307"
}
]
},
],
"total": 359934,
"page": 1,
"pages": 179968
}
Expected output (without column headers of course):
orderId,orderItemId,sku,name
217356098,327010821,"JC-C10S","TestDescription"
217356098,327010822,"JC-C106","AnotherTestDescription"
As you can see, each item has its own line, but if they came from the same order, the orderId should be repeated on each line.
How can I do this?
With the -r command-line option, the following jq filter:
.orders[]
| .orderId as $oid
| .items[]
| [$oid, .orderItemId, .sku, .name]
| #csv
produces the desired output.
If there's any chance that any of the selected values might be [], then consider adding a line like the following immediately before the last line above:
| map_values(if . == [] then "NONE" else . end)
Thanks! That worked with a slight alteration:
.orders[]
| .orderId as $oid
| .items[]
| [$oid, .items.orderItemId, .items.sku, .items.name | tostring]
| #csv
Related
SELECT JSON_query([json], '$') from mytable
Returns fine the contents of [json] field
SELECT JSON_query([json], '$.Guid') from mytable
Returns null
SELECT JSON_query([json], '$.Guid[1]') from mytable
Returns null
I've also now tried:
SELECT JSON_query([json], '$[1].Guid')
SELECT JSON_query([json], '$[2].Guid')
SELECT JSON_query([json], '$[3].Guid')
SELECT JSON_query([json], '$[4].Guid')
and they all return null
So I'm stuck as to figuring out how create the path to get to the info. Maybe SQL Server json_query can't handle the null as the first array?
Below is the string that is stored inside of the [json] field in the database.
[
null,
{
"Round": 1,
"Guid": "15f4fe9d-403c-4820-8e35-8a8c8d78c33b",
"Team": "2",
"PlayerNumber": "78"
},
{
"Round": 1,
"Guid": "8e91596b-cc33-4ce7-bfc0-ac3d1dc5eb67",
"Team": "2",
"PlayerNumber": "54"
},
{
"Round": 1,
"Guid": "f53cd74b-ed5f-47b3-aab5-2f3790f3cd34",
"Team": "1",
"PlayerNumber": "23"
},
{
"Round": 1,
"Guid": "30297678-f2cf-4b95-a789-a25947a4d4e6",
"Team": "1",
"PlayerNumber": "11"
}
]
You need to follow the comments below your question. I'll just summarize them:
Probably the most appropriate approach in your case is to use OPENJSON() with explicit schema (the WITH clause).
JSON_QUERY() extracts a JSON object or a JSON array from a JSON string and returns NULL. If the path points to a scalar JSON value, the function returns NULL in lax mode and an error in strictmode. The stored JSON doesn't have a $.Guid key, so NULL is the actual result from the SELECT JSON_query([json], '$.Guid') FROM mytable statement.
The following statements provide a working solution to your problem:
Table:
SELECT *
INTO Data
FROM (VALUES
(N'[
null,
{
"Round": 1,
"Guid": "15f4fe9d-403c-4820-8e35-8a8c8d78c33b",
"Team": "2",
"PlayerNumber": "78",
"TheProblem": "doesn''t"
},
{
"Round": 1,
"Guid": "8e91596b-cc33-4ce7-bfc0-ac3d1dc5eb67",
"Team": "2",
"PlayerNumber": "54"
},
{
"Round": 1,
"Guid": "f53cd74b-ed5f-47b3-aab5-2f3790f3cd34",
"Team": "1",
"PlayerNumber": "23"
},
{
"Round": 1,
"Guid": "30297678-f2cf-4b95-a789-a25947a4d4e6",
"Team": "1",
"PlayerNumber": "11"
}
]')
) v (Json)
Statements:
SELECT j.Guid
FROM Data d
OUTER APPLY OPENJSON(d.Json) WITH (
Guid uniqueidentifier '$.Guid',
Round int '$.Round',
Team nvarchar(1) '$.Team',
PlayerNumber nvarchar(2) '$.PlayerNumber'
) j
SELECT JSON_VALUE(j.[value], '$.Guid')
FROM Data d
OUTER APPLY OPENJSON(d.Json) j
Result:
Guid
------------------------------------
15f4fe9d-403c-4820-8e35-8a8c8d78c33b
8e91596b-cc33-4ce7-bfc0-ac3d1dc5eb67
f53cd74b-ed5f-47b3-aab5-2f3790f3cd34
30297678-f2cf-4b95-a789-a25947a4d4e6
I have a table with a column that holds valid JSON, heres an example of the JSON structure:
{
"Requirements": {
"$values": [
{
"$type": "List",
"ListId": "956cf9c5-24ab-47d9-8082-940118f2f1a3",
"DefaultValue": "",
"MultiSelect": true,
"Selected": null,
"MultiSelected": {
"$type": "ListItem",
"$values": [
"Value1",
"Value2",
"Value3"
]
}
},
{
"$type": "List",
"ListId": "D11149DD-A682-4BC7-A87D-567954779234",
"DefaultValue": "",
"MultiSelect": true,
"Selected": null,
"MultiSelected": {
"$type": "ListItem",
"$values": [
"Value4",
"Value5",
"Value6",
"Value7"
]
}
}
]
}
}
I need to return the values from MultiSelected collection depending on the value of ListID.
I'm using the following JSON Path to retun value
$.Requirements."$values"[?(#.ListId=='956cf9c5-24ab-47d9-8082-940118f2f1a3')].MultiSelected."$values"
This worked fine in a JSON Expression tester.
But when I try to use it to query the table I get the following error:
JSON path is not properly formatted. Unexpected character '?' is found at position 25.
The query I'm using is as follows:
SELECT ID AS PayloadID,
Items.Item AS ItemsValues
FROM dbo.Payload
CROSS APPLY ( SELECT *
FROM OPENJSON( JSON_QUERY( Payload, '$.Requirements."$values"[?(#.ListId==''956cf9c5-24ab-47d9-8082-940118f2f1a3'')].MultiSelected."$values"' ) )
WITH ( Item nvarchar(200) '$' ) ) AS Items
WHERE ID = 3
I've tried replacing
?(#.ListId==''956cf9c5-24ab-47d9-8082-940118f2f1a3'')
with 0 and it works fine on SQL Server.
My question is, is filter syntax ?(...) supported in JSON_QUERY or is there something else I should be doing?
The database is running on Azure, were the database compatability level is set to SQL Server 2017 (140).
Thanks for your help in advance.
Andy
I would use openjson twice in stead
drop table if exists #payload
create table #payload(ID int,Payload nvarchar(max))
insert into #payload VALUES
(3,N'
{
"Requirements": {
"$values": [
{
"$type": "List",
"ListId": "956cf9c5-24ab-47d9-8082-940118f2f1a3",
"DefaultValue": "",
"MultiSelect": true,
"Selected": null,
"MultiSelected": {
"$type": "ListItem",
"$values": [
"Value1",
"Value2",
"Value3"
]
}
},
{
"$type": "List",
"ListId": "D11149DD-A682-4BC7-A87D-567954779234",
"DefaultValue": "",
"MultiSelect": true,
"Selected": null,
"MultiSelected": {
"$type": "ListItem",
"$values": [
"Value4",
"Value5",
"Value6",
"Value7"
]
}
}
]
}
}'
)
SELECT ID AS PayloadID,
Items.[value]
FROM #Payload a
CROSS APPLY OPENJSON( Payload, '$.Requirements."$values"' ) with ( ListId varchar(50),MultiSelected nvarchar(max) as json) b
CROSS APPLY OPENJSON( MultiSelected,'$."$values"' ) Items
where
a.id=3
AND b.listid='956cf9c5-24ab-47d9-8082-940118f2f1a3'
Reply:
+-----------+--------+
| PayloadID | value |
+-----------+--------+
| 3 | Value1 |
| 3 | Value2 |
| 3 | Value3 |
+-----------+--------+
I am parsing a curl output from gitlab api, and I need to add a sort_by to my query, then select only certain values.
sample input:
[
{
"id": 10,
"name": "another-test",
"path": "another-test",
"description": "",
"visibility": "private",
"lfs_enabled": true,
"avatar_url": null,
"web_url": "https://mygitlab/groups/another-test",
"request_access_enabled": false,
"full_name": "another-test",
"full_path": "another-test",
"parent_id": 9
},
{
"id": 11,
"name": "asdfg",
"path": "asdfg",
"description": "",
"visibility": "private",
"lfs_enabled": true,
"avatar_url": null,
"web_url": "https://mygitlab/groups/asdfg",
"request_access_enabled": false,
"full_name": "asdfg",
"full_path": "asdfg",
"parent_id": 7
}
I parse the JSON with jq as follows:
curl http://..... | jq -r '.[] | select(.parent_id!=null) | .name, .parent_id'
This works exactly as expected, but when I try to sort the results by parent_id, I get an error:
curl http://..... | jq -r '.[] | select(.parent_id!=null) | .name, .parent_id | sort_by(.parent_id)'
jq: error (at <stdin>:0): Cannot index number with string "parent_id"
I can use sort_by(), by putting a single dot instead than .[]:
curl http://..... | jq '. | sort_by(.parent_id) '
But I cannot combine the 2 functions.
Clarification: I need to extract name and parent_id, sorted by parent_id, when it is not null.
Thanks in advance
jq's sort_by() function accepts an array as input.
curl 'http://...' |
jq -r '
map(select(.parent_id != null))
| sort_by(.parent_id)[]
| [.name, .parent_id]
| #tsv
'
Sample output:
asdfg 7
another-test 9
Good day,
is there a way to do a coalesce once and replace all posible null column with empty string or the one you set.
suppose to be the way its done is like
select coalesce(col1,'') as col1, coalesce(col2,'') as col2, coalesce(col3,'') as col3 from table1
is there a way to do this more easily as I have to convert most of my queries into this to replace null fields with " " empty string..
something like
select coalesce(*,'') from tablename where col1=1
it really looks wrong though. but you'll get the idea
currently Im using laravel query
eg.
$data = DB::table('table_name)->where('col1',1)->get();
this converts to "select * from table_name where col1=1";
and result is an array of object:
[{
"id": 319,
"owner": 830,
"name": "new items22",
"date_added": "2017-10-05 22:12:59",
"last_modified": null,
"schedule": 54,
"day_index": 0,
"day": "Sunday",
"type": null,
"open": null,
"close": null,
"special": "closed"
},
{
"id": 320,
"owner": 830,
"name": "another ITEM",
"date_added": "2017-10-05 22:12:59",
"last_modified": null,
"schedule": 54,
"day_index": 1,
"day": "Monday",
"type": null,
"open": "09:00:00",
"close": "17:00:00",
"special": "open"
}]
but what I want to achieve is instead of having a null value, replace it with "" or emptry string.
I suggest that you set your default value on your database or make your custom default value.
I hope this will help.
Consider a table in MySQL 5.7.x having a JSON column.
-- CREATE TABLE "plans" -----------------------------------
CREATE TABLE `plans` (
`id` VarChar( 36 ) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NULL,
`name` VarChar( 50 ) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL,
`structure` JSON NOT NULL,
PRIMARY KEY ( `id` ),
CONSTRAINT `index_exam_plans_on_id` UNIQUE( `id` ),
CHARACTER SET = utf8mb4
COLLATE = utf8mb4_general_ci
ENGINE = InnoDB;
The structure column has a JSON. Please find an example structure of the JSON below:
{
"33aa1e1c-0c95-4860-9b71-ccd13f393dd0": {
"name": "Term 1",
"tags": [
"Term"
],
"type": "default",
"uuid": "33aa1e1c-0c95-4860-9b71-ccd13f393dd0",
"is_locked": false
},
"cb896a12-f07c-4bcc-9c22-7bdfa585f5f7": {
"name": "English",
"tags": [
"Paper",
"Course Paper"
],
"type": "course_paper",
"uuid": "cb896a12-f07c-4bcc-9c22-7bdfa585f5f7",
"course_id": 1,
"is_locked": false
},
"e6d2f9fb-0429-42b2-b704-c438e1695044": {
"name": "Written Work",
"tags": [
"Paper",
"Regular Paper"
],
"type": "regular_paper",
"uuid": "e6d2f9fb-0429-42b2-b704-c438e1695044",
"course_id": 2,
"is_locked": false
},
"d0d3eeff-9ffb-4f35-b0fb-b94373d1fe5b": {
"name": "Summative Assessment",
"tags": [
"Exam"
],
"type": "default",
"uuid": "d0d3eeff-9ffb-4f35-b0fb-b94373d1fe5b",
"is_locked": false
},
"0952e100-cd4a-473e-bd24-2370e0dfcc1c": {
"name": "Speaking skills",
"tags": [
"Paper",
"Regular Paper"
],
"type": "regular_paper",
"uuid": "0952e100-cd4a-473e-bd24-2370e0dfcc1c",
"course_id": 5,
"is_locked": false
}
}
Points to note:
The JSON is an object, containing any number of key-value pairs. The example structure has 5 key-value pairs.
The "value" in each key-value pair is another JS object of key-value pairs. Let's call this the "inner JSON".
Every inner JSON has a key called type.
In the example structure above, the first and fourth inner JSONs have default as the value for the type key.
The requirement
I want to find all the inner JSONs which have type set to default. This should be done in just SQL. For the example above, the output should be just the first and fourth inner JSONs.
How do I do this?
I have tried the following, but it does not return the inner JSONs based on the condition. It returns every inner JSON in the structure column.
select JSON_EXTRACT(structure, '$.*') from plans
where name = 'Academic'
and JSON_CONTAINS( structure->'$.*.type', '"default"' );
Could somebody give the right way to go about this?