I would like to add some values from json file separated by pipe. It's working well so far until a value is a number and not a string.
Here what I've done so far: jq -r '.content[] | {seasonTitle, number, name} | join("|")' file.json
I've tried to convert number to string without any success jq -r '.content[] | {seasonTitle, "episodeNumber|tostring", name} | join("|")' file.json
Actual Result:
Top Master||Last Chance / Season 12
Top Master||Épisode 8 / Season 12
Top Master||Épisode 7 / Season 12
Expected Result:
Top Master|236|Last Chance / Season 12
Top Master|235|Épisode 8 / Season 12
Top Master|234|Épisode 7 / Season 12
Here the file.json
{
"page": 0,
"size": 3,
"count": 3,
"content": [
{
"name": "Last Chance / Season 12",
"releaseDate": "2008",
"duration": 2100,
"episodeNumber": 236,
"title": "Last Chance / Season 12",
"seasonTitle": "Top Master"
},
{
"name": "Épisode 8 / Season 12",
"releaseDate": "2008",
"duration": 7320,
"episodeNumber": 235,
"title": "Épisode 8 / Season 12",
"seasonTitle": "Top Master"
},
{
"name": "Épisode 7 / Season 12",
"releaseDate": "2008",
"duration": 7200,
"episodeNumber": 234,
"title": "Épisode 7 / Season 12",
"seasonTitle": "Top Master"
}
]
}
You are using join to concatenate values of different types, which works fine under jq v1.6:
.content[] | {seasonTitle, episodeNumber, name} | join("|")
Top Master|236|Last Chance / Season 12
Top Master|235|Épisode 8 / Season 12
Top Master|234|Épisode 7 / Season 12
Demo
However, with jq v1.5 it doesn't, and you need to convert non-strings to strings using tostring. As you are using a shortcut to create an object for join, introducing this conversion sacrifices the conciseness of your solution. So either stick with it:
.content[] | {seasonTitle, episodeNumber: (.episodeNumber | tostring), name} | join("|")
Or use an array instead, as you are going for the values only anyway:
.content[] | [.seasonTitle, (.episodeNumber | tostring), .name] | join("|")
Related
I am basically wanting to update multiple scholars for an NFT game (axie infinity). It requires a JSON file that looks like this:
{
"name": "Scholar 1",
"ronin": "ronin:<account_s1_address>",
"splits": [
{
"persona": "Manager",
"percentage": 44,
"ronin": "ronin:<manager_address>"
},
{
"persona": "Scholar",
"percentage": 40,
"ronin": "ronin:<scholar_1_address>"
},
{
"persona": "Other Person",
"percentage": 6,
"ronin": "ronin:<other_person_address>"
},
{
"persona": "Trainer",
"percentage": 10,
"ronin": "ronin:<trainer_address>"
}
]
}
But since there are multiple scholars/players, I wanted to know if there was anyway to format something on a CSV file that if I convert or import it using a JSON tool it will look like like the JSON above?
Your help is much appreciated.. Thank you!
PS:
The first lines:
"name": "Scholar 1",
"ronin": "ronin:<account_s1_address>",
"splits":
Would need to be repeated since again there are multiple scholars, i.e. Scholar 1, Scholar 2, Scholar 3...
CSV file structure is column-base, if Axie infinity require JSON file, you can create a CSV file by Excel or Google sheet and convert to JSON.
there is a similar answer to convert CSV to JSON
starting from this CSV that has this structure
name
ronin
id_persona
persona
percentage
split_ronin
Scholar 1
ronin:<account_s1_address>
1
Manager
44
ronin:<manager_address>
Scholar 1
ronin:<account_s1_address>
2
Scholar
40
ronin:<scholar_1_address>
Scholar 1
ronin:<account_s1_address>
3
Other Person
6
ronin:<other_person_address>
Scholar 1
ronin:<account_s1_address>
4
Trainer
10
ronin:<trainer_address>
you can run this Miller command
mlr --c2j reshape -r "^(p|s)" -o k,v then \
put '$k="splits".".".${id_persona}.".".$k' then \
cut -x -f id_persona then \
reshape -s k,v out.csv
to have
[
{
"name": "Scholar 1",
"ronin": "ronin:<account_s1_address>",
"splits": [
{
"persona": "Manager",
"percentage": 44,
"split_ronin": "ronin:<manager_address>"
},
{
"persona": "Scholar",
"percentage": 40,
"split_ronin": "ronin:<scholar_1_address>"
},
{
"persona": "Other Person",
"percentage": 6,
"split_ronin": "ronin:<other_person_address>"
},
{
"persona": "Trainer",
"percentage": 10,
"split_ronin": "ronin:<trainer_address>"
}
]
}
]
Some notes:
reshape -r "^(p|s)" -o k,v, to transform the input from wide to long;
put '$k="splits".".".${id_persona}.".".$k', to create values that I will use as field names (splits.1.persona,splits.1.percentage,splits.1.split_ronin,splits.2.persona,splits.2.percentage, ....
cut -x -f id_persona, to remove the field id_persona;
reshape -s k,v, to transform all from long to wide.
The real goal is to build, starting from that input, this kind of CSV
+-----------+----------------------------+------------------+---------------------+-------------------------+------------------+---------------------+---------------------------+------------------+---------------------+------------------------------+------------------+---------------------+-------------------------+
| name | ronin | splits.1.persona | splits.1.percentage | splits.1.split_ronin | splits.2.persona | splits.2.percentage | splits.2.split_ronin | splits.3.persona | splits.3.percentage | splits.3.split_ronin | splits.4.persona | splits.4.percentage | splits.4.split_ronin |
+-----------+----------------------------+------------------+---------------------+-------------------------+------------------+---------------------+---------------------------+------------------+---------------------+------------------------------+------------------+---------------------+-------------------------+
| Scholar 1 | ronin:<account_s1_address> | Manager | 44 | ronin:<manager_address> | Scholar | 40 | ronin:<scholar_1_address> | Other Person | 6 | ronin:<other_person_address> | Trainer | 10 | ronin:<trainer_address> |
+-----------+----------------------------+------------------+---------------------+-------------------------+------------------+---------------------+---------------------------+------------------+---------------------+------------------------------+------------------+---------------------+-------------------------+
and than use it to create the final JSON output
I am trying to convert a csv where the headers are keys and the values in the column are a list.
For example I have the following csv
mpg cyl disp hp drat wt qsec vs am gear carb
Mazda RX4 21 6 160 110 3.9 2.62 16.46 0 1 4 4
Mazda RX4 Wag 21 6 160 110 3.9 2.875 17.02 0 1 4 4
Datsun 710 22.8 4 108 93 3.85 2.32 18.61 1 1 4 1
I would like the following format.
{
"field1" : [Mazda RX4 ,Mazda RX4 Wag,Datsun 710],
"mpg" : [21,21,22.8],
"cyl" : [6,6,6],
"disp" : [160,160,108],
...
}
Note that the numerical values are not quoted. I am assuming that the columns all have the same type.
I am using the following jq command.
curl https://raw.githubusercontent.com/vincentarelbundock/Rdatasets/master/csv/datasets/mtcars.csv cars.csv | head -n4 | csvtojson | jq '.'
[
{
"field1": "Mazda RX4",
"mpg": "21",
"cyl": "6",
"disp": "160",
"hp": "110",
"drat": "3.9",
"wt": "2.62",
"qsec": "16.46",
"vs": "0",
"am": "1",
"gear": "4",
"carb": "4"
},
{
"field1": "Mazda RX4 Wag",
"mpg": "21",
"cyl": "6",
"disp": "160",
"hp": "110",
"drat": "3.9",
"wt": "2.875",
"qsec": "17.02",
"vs": "0",
"am": "1",
"gear": "4",
"carb": "4"
},
{
"field1": "Datsun 710",
"mpg": "22.8",
"cyl": "4",
"disp": "108",
"hp": "93",
"drat": "3.85",
"wt": "2.32",
"qsec": "18.61",
"vs": "1",
"am": "1",
"gear": "4",
"carb": "1"
}
]
Complete working solution
cat <csv_data> | csvtojson | jq '. as $in | reduce (.[0] | keys_unsorted[]) as $k ( {}; .[$k] = ($in|map(.[$k])))'
jq play - Converting all numbers to strings
https://jqplay.org/s/HKjHLVp9KZ
Here's a concise, efficient, and conceptually simple solution based on just map and reduce:
. as $in
| reduce (.[0] | keys_unsorted[]) as $k ( {}; .[$k] = ($in|map(.[$k])))
Converting all number-valued strings to numbers
. as $in
| reduce (.[0] | keys_unsorted[]) as $k ( {};
.[$k] = ($in|map(.[$k] | (tonumber? // .))))
1) I am trying to generate a CSV file using jq from a json.
2) I need parent keys along with one key-value pair from the child array
3) Which ever value has latest date in it , will be the resulting key-value pair
4) Need to generate a csv out of that result
This is my json
{
"students": [
{
"name": "Name1",
"class": "parentClass1",
"teacher": "teacher1",
"attendance": [
{
"key": "class1",
"value": "01-DEC-2018"
},
{
"key": "class1",
"value": "28-Nov-2018"
},
{
"key": "class1",
"value": "26-Oct-2018"
}
]
},
{
"name": "Name2",
"class": "parentClass2",
"teacher": "teacher2",
"attendance": [
{
"key": "class2",
"value": "05-DEC-2018"
},
{
"key": "class2",
"value": "25-Nov-2018"
},
{
"key": "class2",
"value": "20-Oct-2018"
}
]
}
]
}
I did not made much progress I am trying to create csv like this
jq '.students[] | [.name, .class, attendance[].key,.properties[].value] | #csv ' main.json
Below is expected CSV from that json
Name ParentClass key dateValue Summary
Name1 parentClass1 class1 150 days ago(difference with today date with latest date i.e 01-DEC-2018 ) Teacher1.parentClass1
Name2 parentClass2 class2 150 days ago(difference with today date with latest date i.e 05-DEC-2018 ) Teacher2.parentClass2
Parse dates using strptime and assign the result to values, thus you can get the latest attendance using max_by. Convert the value to seconds since Epoch using mktime, substract it from now, divide by 24 * 60 * 60 to get number of days since.
$ jq -r '
def days_since:
(now - .) / 86400 | floor;
.students[]
| [ .name, .class ] +
( .attendance
| map(.value |= strptime("%d-%b-%Y"))
| max_by(.value)
| [ .key, "\(.value | mktime | days_since) days ago" ]
) +
[ .teacher + "." + .class ]
| #tsv' file
Name1 parentClass1 class1 148 days ago teacher1.parentClass1
Name2 parentClass2 class2 144 days ago teacher2.parentClass2
Note that this solution doesn't deal with daylight saving time changes.
For production purposes jq can't be used here because it doesn't allow to perform daylight saving time safe date calculations.
I would use Python because it allows to perform daylight saving time safe date calculations, comes with json support by default and is installed on most to all UNIX derivates.
#!/usr/bin/env python
import argparse
from datetime import datetime
import json
def parse_args():
parser = argparse.ArgumentParser()
parser.add_argument('filename')
return parser.parse_args()
def main():
args = parse_args()
with open(args.filename) as file_desc:
data = json.load(file_desc)
print('Name\tParentClass\tkey\tdateValue')
today = datetime.today()
for record in data['students']:
for a in record['attendance']:
date = datetime.strptime(a['value'], '%d-%b-%Y')
a['since'] = (today - date).days
last = sorted(record['attendance'], key=lambda x: x['since'])[0]
print('\t'.join([
record['name'],
record['class'],
last['key'],
'{} days ago'.format(last['since']),
'{}.{}'.format(record['teacher'], record['class']),
]))
if __name__ == '__main__':
main()
Output (on the day when this answer was written):
Name ParentClass Key DateValue Summary
Name1 parentClass1 class1 148 days ago teacher1.parentClass1
Name2 parentClass2 class2 144 days ago teacher2.parentClass2
I'm trying to parse JSON and store certain values as metrics in Graphite.
In order to make my Graphite more user-friendly I have to form a metric name, that contains some values from its object.
I got working solution on bash loops + jq, but it's really slow. So I'm asking for help :)
Here is my input:
{
...
},
"Johnny Cage": {
"firstname": "Johnny",
"lastname": "Cage",
"height": 183,
"weight": 82,
"hands": 2,
"legs": 2,
...
},
...
}
Desired output:
mk.fighter.Johnny.Cage.firstname Johnny
mk.fighter.Johnny.Cage.lastname Cage
mk.fighter.Johnny.Cage.height 183
mk.fighter.Johnny.Cage.weight 82
mk.fighter.Johnny.Cage.hands 2
mk.fighter.Johnny.Cage.legs 2
...
With single jq command:
Sample input.json:
{
"Johnny Cage": {
"firstname": "Johnny",
"lastname": "Cage",
"height": 183,
"weight": 82,
"hands": 2,
"legs": 2
}
}
jq -r 'to_entries[] | (.key | sub(" "; ".")) as $name
| .value | to_entries[]
| "mk.fighter.\($name).\(.key) \(.value)"' input.json
To get $name as a combination of inner firstname and lastname keys replace (.key | sub(" "; ".")) as $name with "\(.value.firstname).\(.value.lastname)" as $name
The output:
mk.fighter.Johnny.Cage.firstname Johnny
mk.fighter.Johnny.Cage.lastname Cage
mk.fighter.Johnny.Cage.height 183
mk.fighter.Johnny.Cage.weight 82
mk.fighter.Johnny.Cage.hands 2
mk.fighter.Johnny.Cage.legs 2
Imagine I have a categories tree like this JSON file:
[
{
"id": "1",
"text": "engine",
"children": [
{
"id": "2",
"text": "exhaust",
"children": []
},
{
"id": "3",
"text": "cooling",
"children": [
{
"id": "4",
"text": "cooling fan",
"children": []
},
{
"id": "5",
"text": "water pump",
"children": []
}
]
}
]
},
{
"id": "6",
"text": "frame",
"children": [
{
"id": "7",
"text": "wheels",
"children": []
},
{
"id": "8",
"text": "brakes",
"children": [
{
"id": "9",
"text": "brake calipers",
"children": []
}
]
},
{
"id": "10",
"text": "cables",
"children": []
}
]
}
]
How can I convert it to this flat table?
id parent_id text
1 NULL engine
2 1 exhaust
3 1 cooling
4 3 cooling fan
5 3 water pump
6 NULL frame
7 6 wheels
8 6 brakes
9 8 brake calipers
10 6 cables
I found similar questions and inverted questions (from table to JSON) but I can't figure it out with jq and its #tsv filter. Also I noticed the "flatten" filter is not often referenced in the answers (while it looks to be the exact tool I need) but it might be because it was introduced recently in the latests versions of jq.
Here is another solution which uses jq's recurse builtin:
["id","parent_id","text"]
, (
.[]
| recurse(.id as $p| .children[] | .parent=$p )
| [.id, .parent, .text]
)
| #tsv
Sample Run (assumes filter in filter.jq and sample data in data.json)
$ jq -Mr -f filter.jq data.json
id parent_id text
1 engine
2 1 exhaust
3 1 cooling
4 3 cooling fan
5 3 water pump
6 frame
7 6 wheels
8 6 brakes
9 8 brake calipers
10 6 cables
Try it online!
The key here is to define a recursive function, like so:
def children($parent_id):
.id as $id
| [$id, $parent_id, .text],
(.children[] | children($id)) ;
With your data, the filter:
.[]
| children("NULL")
| #tsv
produces the tab-separated values shown below. It is now easy to add headers, convert to fixed-width format if desired, etc.
1 NULL engine
2 1 exhaust
3 1 cooling
4 3 cooling fan
5 3 water pump
6 NULL frame
7 6 wheels
8 6 brakes
9 8 brake calipers
10 6 cables
Here is a solution which uses a recursive function:
def details($parent):
[.id, $parent, .text], # details for current node
(.id as $p | .children[] | details($p)) # details for children
;
["id","parent_id","text"] # header
, (.[] | details(null)) # details
| #tsv # convert to tsv
Sample Run (assumes filter in filter.jq and sample data in data.json)
$ jq -Mr -f filter.jq data.json
id parent_id text
1 engine
2 1 exhaust
3 1 cooling
4 3 cooling fan
5 3 water pump
6 frame
7 6 wheels
8 6 brakes
9 8 brake calipers
10 6 cables
Try it online!