how to select based on a substring in jq - json

With jq, how do I select only elements that contain the word "Star" in the following file movies:
$ cat movies
[
{
"title": "Dark Knight, The",
"year": 2008
},
{
"title": "Star Trek Beyond",
"year": 2016
},
{
"title": "Star Wars: Episode VII - The Force Awakens",
"year": 2015
}
]
Expected output:
{
"title": "Star Trek Beyond",
"year": 2016
}
{
"title": "Star Wars: Episode VII - The Force Awakens",
"year": 2015
}


jq '.[] | select(.title | contains("Star"))' movies

Related

How to handle nested json duplicate entries and save it in new row in csv?

I am trying to convert API response into a csv file.
What issue I am facing??
Look at the files and descriptions fields in the below attached json.
In the API response these array can have any no of entries as you can see in example json.
What I want to achieve?
I want a csv file which has columns as follow.
id, img1, img2, ...(all images)..., Market Description, Associated Comments, ...(all descriptions)...
here img1, img2,.. etc is in files.links.url
and
for descriptions... The value of discriptions.type should be the column name. and discriptions.discription should be the value in the respective column.
I tried to use json_normalize but, as these files and descriptions arrays are of random lengths I am completely clueless that how I can handle that.
Please help me out with this. Thanks for giving your valuable time. Any kind of help or guidance is valuable for me.
Also, let me know If you guys have any other approaches or anything which can help me out.
API Response::
"data": [
{
"id": "35776",
"type": "ProductData",
"files": [
{
"id": "2894907",
"type": "Image",
"file_extension": "JPG",
"media_content": "Photo - Primary",
"generic": true,
"links": [
{
"url": "https://d32vzsop7y1h3k.cloudfront.net/cc3a35729a75a8510c4a0b14a7fc6fd4.JPG",
"height": "1070.00",
"width": "800.00",
"size": "L"
}
]
},
{
"id": "6196485",
"type": "Image",
"file_extension": "JPG",
"media_content": "Photo - Primary",
"generic": true,
"links": [
{
"url": "https://d32vzsop7y1h3k.cloudfront.net/b8c7fa80b5e6fc9d35e2a8c8ee59e9ef.JPG",
"height": "570.00",
"width": "800.00",
"size": "L"
}
]
},
{
"id": "7079630",
"type": "Image",
"file_extension": "JPG",
"media_content": "Photo - Primary",
"generic": true,
"links": [
{
"url": "https://d32vzsop7y1h3k.cloudfront.net/7c39c695853f4a51ff1fe957cc4b2103.JPG",
"height": "533.00",
"width": "800.00",
"size": "L"
}
]
},
{
"id": "13400593",
"type": "Image",
"file_extension": "JPG",
"media_content": "Photo - Primary",
"generic": true,
"links": [
{
"url": "https://d32vzsop7y1h3k.cloudfront.net/95f42ba54e4106298efb5e84955e91a9.JPG",
"height": "800.00",
"width": "800.00",
"size": "L"
}
]
},
{
"id": "23536454",
"type": "Image",
"file_extension": "JPG",
"media_content": "Photo - Primary",
"generic": true,
"links": [
{
"url": "https://d32vzsop7y1h3k.cloudfront.net/9ef4515f90311a25e69f87111bee03f9.JPG",
"height": "800.00",
"width": "800.00",
"size": "L"
}
]
},
{
"id": "7079629",
"type": "Image",
"file_extension": "PNG",
"media_content": "Logo Image",
"generic": true,
"links": [
{
"url": "https://d32vzsop7y1h3k.cloudfront.net/b2e60a9f192a7032e8b87b966c826187.PNG",
"height": "390.00",
"width": "800.00",
"size": "L"
}
]
},
{
"id": "13400592",
"type": "Image",
"file_extension": "JPG",
"media_content": "Logo Image",
"generic": true,
"links": [
{
"url": "https://d32vzsop7y1h3k.cloudfront.net/99b17aa8393a06019fea7b4bb1b58727.JPG",
"height": "417.00",
"width": "800.00",
"size": "L"
}
]
}
],
"descriptions": [
{
"type": "Market Description",
"description": "All EBC rotors are made from OE spec grey cast iron, exactly as used on new car production and as many as we have tools for are BRITISH MADE wholly in our own UK foundry. Currently EVERY SINGLE ROTOR sold in Europe and the UK is BRITISH MADE and a majority of the designs specific to the USA are also BRITISH made.The British foundry is constantly expanding its in house tooled range but EBC like every brake manufacturer ....does need occasionally to buy some rotors in from approved sources to complete ranges in overseas markets. None of our competitors will ever tell you where their rotors come from and we at EBC are as disappointed as the rest of the world that cost pressure means there is not ONE foundry left in the USA to supply such parts, other brake brands when asked about origin will just sidestep the question. When we do buy rotors in, they are specified, sampled, tested for element analysis, grain structure, tensile strength, hardness and optimum ductility and tested 100 piece by piece to be perfect for runout on this in house machine."
},
{
"type": "Associated Comments",
"description": "DO NOT FIT UPGRADE PADS TO REAR OF VEHICLE UNLESS SAME UPGRADE FITTED TO FRONT, The ?Thermic Black coating? does NOT need to be removed before install. Do NOT use brake cleaner or any other method to remove Thermic Black coating from discs rotors. The Thermic black coating will be removed in the pad swept area by the brake pads with in the first 10 to 20 stops made after installation on the vehicle.The ?Thermic Black coating? is an anti corrosion coating for long resistance against rust. Attempting to remove Thermic Black Coating with brake cleaner or any other method is NOT covered under EBC warranty"
},
{
"type": "Associated Comments",
"description": "DO NOT FIT UPGRADE PADS TO REAR OF VEHICLE UNLESS SAME UPGRADE FITTED TO FRONT, The Thermic Black coating? does NOT need to be removed before install. Do NOT use brake cleaner or any other method to remove Thermic Black coating from discs rotors. The Thermic black coating will be removed in the pad swept area by the brake pads with in the first 10 to 20 stops made after installation on the vehicle.The Thermic Black coating? is an anti corrosion coating for long resistance against rust. Attempting to remove Thermic Black Coating with brake cleaner or any other method is NOT covered under EBC warranty"
},
{
"type": "Product Description - Extended",
"description": "EBC's GD sport rotors features wide aperture slots for cooling to reduce temperatures preventing brake fade. Dimple drilling avoids stress cracks with Thermic black finish with a sleek look whilst preventing corrosion."
},
{
"type": "Market Description",
"description": "Feature wider slots and blind drilled dimples to avoid cracks, these degas pads and remove dirt dust and debris from braking area"
},
{
"type": "Product Description - Extended",
"description": "EBC 3GD Series Sport Slotted Rotors; Rear; 11.1 in. Dia.; Set of Two;"
},
{
"type": "Product Description - Short",
"description": "EBC 09-14 Acura TSX 2.4 GD Sport Rear Rotors"
},
{
"type": "Application Summary",
"description": "2004 thru 2005 HONDA Accord Sedan 3.0 Hybrid"
},
{
"type": "Application Summary",
"description": "2006 thru 2007 HONDA Accord Sedan 3.0 Hybrid"
},
{
"type": "Application Summary",
"description": "2013 thru 2016 HONDA Accord Sedan 2.0 hybrid"
},
{
"type": "Application Summary",
"description": "2016 and up HONDA Accord Sedan 3.5 Touring"
},
{
"type": "Application Summary",
"description": "2016 and up HONDA Accord Coupe 3.5 Touring"
},
{
"type": "Application Summary",
"description": "2016 and up HONDA Accord Sedan 2.0 hybrid"
},
{
"type": "Product Description - Long",
"description": "GD Rotors REAR Disc Brake Rotor FMSI D537"
},
{
"type": "Application Summary",
"description": "2007 thru 2012 HONDA Accord Sedan 2.4 EX"
},
{
"type": "Application Summary",
"description": "2008 thru 2012 HONDA Accord Sedan 2.4 LX"
},
{
"type": "Application Summary",
"description": "2013 thru 2015 HONDA Accord Sedan 2.4 LX"
},
{
"type": "Application Summary",
"description": "2013 thru 2015 HONDA Accord Sedan 2.4 EX"
},
{
"type": "Application Summary",
"description": "2016 and up HONDA Accord Sedan 2.4 Sport"
},
{
"type": "Application Summary",
"description": "2008 thru 2012 HONDA Accord Coupe 2.4 EX"
},
{
"type": "Application Summary",
"description": "2008 thru 2012 HONDA Accord Coupe 2.4 LX"
},
{
"type": "Application Summary",
"description": "2013 thru 2015 HONDA Accord Coupe 2.4 LX"
},
{
"type": "Application Summary",
"description": "2013 thru 2015 HONDA Accord Coupe 2.4 EX"
},
{
"type": "Application Summary",
"description": "2016 and up HONDA Accord Coupe 2.4 LX-S"
},
{
"type": "Application Summary",
"description": "2016 and up HONDA Accord Coupe 3.5 EX-L"
},
{
"type": "Application Summary",
"description": "2016 and up HONDA Accord Sedan 2.4 EX-L"
},
{
"type": "Application Summary",
"description": "2016 and up HONDA Accord Coupe 2.4 EX-L"
},
{
"type": "Application Summary",
"description": "2016 and up HONDA Accord Sedan 3.5 EX-L"
},
{
"type": "Application Summary",
"description": "2016 and up HONDA Accord Coupe 2.4 EX"
},
{
"type": "Application Summary",
"description": "2016 and up HONDA Accord Sedan 2.4 EX"
},
{
"type": "Application Summary",
"description": "2016 and up HONDA Accord Sedan 2.4 LX"
},
{
"type": "Application Summary",
"description": "2008 thru 2012 HONDA Accord Sedan 3.5"
},
{
"type": "Application Summary",
"description": "2013 thru 2015 HONDA Accord Sedan 3.5"
},
{
"type": "Application Summary",
"description": "2008 thru 2012 HONDA Accord Coupe 3.5"
},
{
"type": "Application Summary",
"description": "2013 thru 2015 HONDA Accord Coupe 3.5"
},
{
"type": "Product Description - Invoice",
"description": "GD7440 REAR GD Pattern Rotors"
},
{
"type": "Application Summary",
"description": "2009 thru 2014 ACURA TSX 2.4"
},
{
"type": "Application Summary",
"description": "2010 thru 2014 ACURA TSX 3.5"
},
{
"type": "Application Summary",
"description": "2015 and up HONDA HR-V 1.8"
},
{
"type": "Application Summary",
"description": "2017 and up ACURA ILX 2.4"
},
{
"type": "Product Description - Long",
"description": "EBC GD SPORT ROTOR KIT"
},
{
"type": "Product Description - Short",
"description": "EBC Brakes GD Sport"
},
{
"type": "Application Summary",
"description": "2016 ACURA ILX 2.4"
},
{
"type": "Product Description - Short",
"description": "GD Pattern Rotors"
},
{
"type": "AAIA Part Type Description",
"description": "Disc Brake Rotor"
},
{
"type": "Product Description - Long",
"description": "EBC GD ROTOR KIT"
}
],
"relationships": {
"vehicle_fitments": {
"links": {
"self": "/v1/items/fitment/35776"
}
}
}
}
]
} ```

With the json file you provided, here is one way to do it:
df = pd.read_json("file.json")
# Get urls
images = pd.DataFrame.from_dict(df["files"][0])["links"].apply(lambda x: x[0]["url"])
# Reshape urls as 'horizontal' dataframe
images = pd.DataFrame(images).T.reset_index(drop=True)
# Renames columns
images.columns = [f"img{i}" for i in range(images.shape[1])]
# Get descriptions as 'horizontal' dataframe
descriptions = pd.DataFrame.from_dict(df["descriptions"][0]).T.reset_index(drop=True)
# Grab the first row for the header
new_header = descriptions.iloc[0]
# Take the data less the header row
descriptions = descriptions[1:]
# Set the header row as the df header and clean index
descriptions.columns = new_header
descriptions = descriptions.reset_index(drop=True)
# Final dataframe
df = pd.concat([images, descriptions], axis=1)
print(df)
# Output
img0 img1 img2 img3 \
0 https://d32... https://d32... https://d32... https://d32...
img4 ... Product Description - Short Application Summary \
0 https://d32... ... EBC Brakes ... 2016 ACURA ...
Product Description - Short AAIA Part Type Description \
0 GD Pattern ... Disc Brake ...
Product Description - Long
0 EBC GD ROTO...
[1 rows x 53 columns]

Setting up postgres database for recipes

I am new to PostgreSQL and trying to plan a database that will allow me to query recipes based on id, ingredients etc from a provided dataset, example below.
I am getting a bit thrownoff by the nested ingredients and how to lay out my tables.
I was initially thinking two tables one for instructions and image and another for the ingredients.
i was now wondering if i am on the right lines and if so how to loop through the nested ingredients to produce a useful second table
enter code here
[ {
"id": "recipe-88",
"imageUrl": "http://www.images.com/12",
"instructions": "blend with oat milk and ice, sprinkle with
salt",
"ingredients": [
{ "name": "blueberries", "grams": 114 },
{ "name": "coffee", "grams": 20 },
{ "name": "kale", "grams": 48 }
]
},
{
"id": "recipe-74",
"imageUrl": "http://www.images.com/2",
"instructions": "crush ingredients with mortar and pestle, mix
with whole milk, serve in bowl",
"ingredients": [
{ "name": "coffee", "grams": 25 },
{ "name": "lime", "grams": 140 },
{ "name": "strawberries", "grams": 3 },
{ "name": "apricots", "grams": 24 },
{ "name": "kale", "grams": 50 }
]
},
{
"id": "recipe-77",
"imageUrl": "http://www.images.com/25",
"instructions": "blend with oat milk and ice, sprinkle with
salt",
"ingredients": [
{ "name": "coconut", "grams": 14 },
{ "name": "coconut", "grams": 57 },
{ "name": "lime", "grams": 153 },
{ "name": "oat milk", "grams": 31 }
]
},]

I have parsed the json you provided and loaded it in a table.
Check the dbfiddle link.
Here is the parsed content loaded into a table.
postgres=# select * from parsedjj;
id | recipe_id | imageurl | instructions | ingredients
----+-----------+-------------------------+-------------------------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
1 | recipe-88 | http//www.images.com/12 | blend with oat milk and ice, sprinkle with salt | [{"name": "blueberries", "grams": 114}, {"name": "coffee", "grams": 20}, {"name": "kale", "grams": 48}]
2 | recipe-74 | http//www.images.com/2 | crush ingredients with mortar and pestle, mix with whole milk, serve in bowl | [{"name": "coffee", "grams": 25}, {"name": "lime", "grams": 140}, {"name": "strawberries", "grams": 3}, {"name": "apricots", "grams": 24}, {"name": "kale", "grams": 50}]
3 | recipe-77 | http//www.images.com/25 | blend with oat milk and ice, sprinkle with salt | [{"name": "coconut", "grams": 14}, {"name": "coconut", "grams": 57}, {"name": "lime", "grams": 153}, {"name": "oat milk", "grams": 31}]
(3 rows)
Note: I had changed the JSON key value from "imageUrl" to "image" because possibly "url" which is a keyword for postgres might have been impeding the parsing.

Reshape a array to object with jq

I am learning advance concepts of jq. And I made a tiny json with array with some films of Charles Chaplin...well this array in json:
[
{
"title": "The Great Dictator",
"year": 1940,
"country": "USA",
"genre": "political satire"
},
{
"title": "Modern Times ",
"year": 1936,
"country": "USA",
"genre": "comedy"
},
{
"title": "The Gold Rush",
"year": 1925,
"country": "USA",
"genre": "comedy"
},
{
"title": "The Kid",
"year": 1921,
"country": "USA",
"genre": "drama"
}
]
And I want to convert or reshape into a object with the genres as the keys and the list of the films as array (comedy is only has two element in the array):
{
"comedy": [
{
"title": "Modern Times ",
"year": 1936,
"country": "USA"
},
{
"title": "The Gold Rush",
"year": 1925,
"country": "USA"
}
],
"political satire": [
{
"title": "The Great Dictator",
"year": 1940,
"country": "USA"
}
],
"drama": [
{
"title": "The Kid",
"year": 1921,
"country": "USA"
}
]
}
But I can't do it. I trying the first step to create a object with genre and foo string as var, but it fails: cat c.json | jq '{.[] | (.genre): "foo" ]}'

It can be done in three lines:
[group_by(.genre)[]
| {(.[0].genre): map_values(del(.genre))}]
| add
aggregate_by/3
The relevant generic abstraction here is:
def aggregate_by(s; f; g):
reduce s as $x (null; .[$x|f] += [$x|g]);
This allows the solution to be written directly as:
aggregate_by(.[]; .genre; del(.genre))

I found:
$ cat c.json | jq '
group_by(.genre)
| map({"genre": .[0].genre,
"film": map(. | del(.genre))})
| [ .[] | {(.genre): .film}]
| add'
{
"comedy": [
{
"title": "Modern Times ",
"year": 1936,
"country": "USA"
},
{
"title": "The Gold Rush",
"year": 1925,
"country": "USA"
}
],
"drama": [
{
"title": "The Kid",
"year": 1921,
"country": "USA"
}
],
"political satire": [
{
"title": "The Great Dictator",
"year": 1940,
"country": "USA"
}
]
}
Maybe it is not the best, because I think there are a lot of steps...but it runs.

You can use a modified version of Jeff Mercado's answer on the page you linked to.
jq 'reduce .[] as $i ({}; .[$i.genre] += [$i])'
That groups the objects as you want but leaves the genre key-value pair. You can delete them like so.
jq 'reduce .[] as $i ({}; .[$i.genre] += [$i|del(.genre)])'
Really, this is just a concrete version of peak's "generic abstraction".

How to select a object in jq that matches on a greater or less than for multiple values in a attribute

Using jq, I am searching through a large json file that has objects that contain multiple year values in a attribute. I want to be able to narrow my search and only retrieve objects that are greater or less(older or younger) than YEAR(example 2015). This is tricky because the attributes can have multiple year values, some greater and less than the year I am searching for.
Here is a example of the code I am searching through. It's all basically this.
cat stackover.json
{
"tokens": [
{
"Name": "stack overflow",
"year": [
"1997",
"1998",
"1997"
]
},
{
"Name": "Return pizza",
"year": [
"1998",
"2015",
"2014",
"1998"
]
}
]
}
I know how to search for a contains one of the values... but can figure out how to search all the values and determine if all of them are less than 2000.
If I search for less than '<', then I get no out put back. If I search for greater than '>', then I get back both object multiple times
cat stackover.json | jq '.tokens[] | select(.year[] > 2000)'
{
"Name": "stack overflow",
"year": [
"1997",
"1998",
"1997"
]
}
{
"Name": "stack overflow",
"year": [
"1997",
"1998",
"1997"
]
}
{
"Name": "stack overflow",
"year": [
"1997",
"1998",
"1997"
]
}
{
"Name": "Return pizza",
"year": [
"1998",
"2015",
"2014",
"1998"
]
}
{
"Name": "Return pizza",
"year": [
"1998",
"2015",
"2014",
"1998"
]
}
{
"Name": "Return pizza",
"year": [
"1998",
"2015",
"2014",
"1998"
]
}
{
"Name": "Return pizza",
"year": [
"1998",
"2015",
"2014",
"1998"
]
}
What I am trying to accomplish.
If I search for anything less than 2000, I want a query that will only return "stack overflow" since it does not have a 20XX value in the available values.
cat stackover.json | code searching for only objects(stack overflow) where all values are less than 2000.
{
"Name": "stack overflow",
"year": [
"1997",
"1998",
"1997"
]
}

In brief, all/2 is your friend.
Assuming we have defined $mx (a string) as the cutoff year, a suitable filter would be:
.tokens[] | select( all(.year[]; . < $mx) )
The command-line option '--arg mx N' interprets N as a string, so a suitable invocation would be like so:
jq --arg mx 2000 -f filter.jq stackover.json
(Notice that we can avoid converting the dates to numbers by letting $mx be a string.)

How to extract a JSON field that has a random string with the jq command?

Please bear with me I'm still learning about JSON and programming in general. So I have this JSON file:
{
"root_200888546292191": {
"fields": {
"buyerId": "31392191"
},
"id": "200718546292191",
"tag": "root",
"type": "biz"
},
"shippingInfo_#package#OF04472002179150#WAREHOUSE_ACCEPTED": {
"fields": {
"delivery": {
"createdAt": "Sen 09 Apr - Rab 11 Apr",
"desc": "Standar",
"email": null,
"method": "Standard",
"status": "info"
},
"statusMap": {
"active": "Dalam proses",
"all": ["Dalam proses", "Dalam pengiriman", "Telah diterima"]
},
"trackingList": [{
"info": "Status One",
"updatedAt": "05 Apr 2018 - 11:00"
}, {
"info": "Status Two",
"updatedAt": "05 Apr 2018 - 11:00"
}]
},
"id": "#package#OF04472002179150#WAREHOUSE_ACCEPTED",
"tag": "shippingInfo",
"type": "biz"
},
"shippingInfo_#package#AAAAAAAAAAAAA#NOT_WAREHOUSE_ACCEPTED": {
"fields": {
"delivery": {
"createdAt": "Sen 09 Apr - Rab 11 Apr",
"desc": "Standar",
"email": null,
"method": "Standard",
"status": "info"
},
"statusMap": {
"active": "Dalam proses",
"all": ["Dalam proses", "Dalam pengiriman", "Telah diterima"]
},
"trackingList": [{
"info": "Status Three",
"updatedAt": "05 Apr 2018 - 11:00"
}, {
"info": "Status Four",
"updatedAt": "05 Apr 2018 - 11:00"
}]
},
"id": "#package#AAAAAAAAAAAAA#NOT_WAREHOUSE_ACCEPTED",
"tag": "shippingInfo",
"type": "biz"
},
"login_200718577292191": {
"fields": {
"buyerEmail": "myemail#gmail.com",
"buyerName": "myname"
},
"id": "200718522292191",
"tag": "login",
"type": "biz"
}
}
And I want to extract Info in shippingInfo_ > fields > trackingList So the output that I want is like this:
Status One
Status Two
Status Three
Status Four
The string after shippingInfo_ is always random, how do I extract it with jq?
This is as far as I've got jq '.shippingInfo_*.fields.trackingList.info'

Direct approaches
There are many direct approaches, e.g.:
Using paths
paths as $p
| select( $p|length == 5 and
($p[0] | startswith("shippingInfo_")) and
$p[1:3] == ["fields", "trackingList"] and
$p[4] == "info")
| getpath($p)
Using to_entries
to_entries[]
| select(.key | startswith("shippingInfo_"))
| .value
| .fields.trackingList[]
| .info
Indirect approaches
There are also some indirect approaches that are worth mentioning, e.g.
Using a helper function
def dot(s):
to_entries[] | select(.key|test(s)) | .value ;
dot("^shippingInfo_")
| .fields.trackingList[]
| .info
The last-mentioned filter can be abbreviated to:
dot("^shippingInfo_").fields.trackingList[].info
Relaxed requirements
If it is acceptable to ignore the "^shippingInfo_" requirement, the following may be worth considering as well:
.[].fields.trackingList[]?.info
or even:
.. | objects.fields.trackingList[]?.info