I need to create a JSON file with the following contents:
{
"destinationVersion":"4",
"sourceVersion":"0",
"props":{
"METADATA_SIZE":"91669",
"METADATA_HASH":"O7CLdR2j7qoD0RI2k1AGc8b+xoWYn20Ic24eZ1ZWUWE=",
"FILE_SIZE":"980374602",
"FILE_HASH":"+XW4QKN5Y4ynTx43m4NYbMuk1x3P91f1biAVZBpj4fI="
}
}
The main snag with the props block.
These values for props must be read from the text file.
In it they are in the following format:
FILE_HASH=+XW4QKN5Y4ynTx43m4NYbMuk1x3P91f1biAVZBpj4fI=
FILE_SIZE=980374602
METADATA_HASH=O7CLdR2j7qoD0RI2k1AGc8b+xoWYn20Ic24eZ1ZWUWE=
METADATA_SIZE=91669
My task is to read the file and somehow create the final JSON (append formatted text to props block).
I try to do this through jq, but I don’t know how to convert the file to be added to the final JSON. Preferred way - bash + jq
Since the Q only indicates the source for the key=value strings, the following focuses on the conversion of these strings to a JSON object.
Invocation: jq -n -R -f program.jq props.txt
program.jq:
[inputs | capture("^(?<key>[^=]*)=(?<value>.*)")]
| from_entries
Using inputs here has several small advantages but is inessential.
Related
I know we should use JQ for parsing json data, but I want to parse it using regex. I want to fetch the value of a json key into a variable in my shell script. As of now, I am using JQ for parsing.
So my abc.json is
{"value1":5.0,"value2":2.5,"value3":"2019-10-24T15:26:00.000Z","modifier":[],"value4":{"value41":{"value411":5}}}
Currently, my XYZ.sh has these lines to fetch the data
data1 =$(cat abc.json | jq -r '.value4.value41.value411')
I want data1 to have value of value411. How can I achieve this?
ps- The JSON is mutable. The above JSON is just a part of the JSON file that I want to fetch.
Is your json structure immutable? If you have to use it, consider the following
┌──[root#vms83.liruilongs.github.io]-[~]
└─$cat abc.json | awk -F: '{print $NF}' | grep -o '[[:digit:]]'
5
I think your problem was you had a space between data and =. There can't be a space there.
This works as you want it to (I removed the unnecessary cat)
data1=$(jq -r '.value4.value41.value411' abc.json)
echo $data1
I have a json file that is getting created at runtime using the sh script within groovy code. The json file has below contents.
cat.json
{
"user1":"pass1",
"user2":"pass2",
"user3":"pass3"
}
Now I want to create a file at runtime which stores key value pairs in below format
test
user1:pass1
user2:pass2
user3:pass3
can some one help me out shell codes for writing this.
You have literally dozen ways to convert that JSON document to a tabular data file (pretty much like CSV/colon-SV) since you mentioned Java, Groovy, including Java-driven scripting engines (BeanShell, JavaScript, Groovy itself), but if you can use jq then you can extract k/v pairs at least for simple values that do not require any escaping:
#!/bin/sh
jq -r 'to_entries[] | "\(.key):\(.value)"' \
< cat.json
This answer is inspired by searching for extracting entries using jq (or converting a JSON file to a CSV file) and especially by the answer https://stackoverflow.com/a/50496145/12232870 by #peak.
I have a lot of rather large JSON logs which need to be imported into several DB tables.
I can easily parse them and create 1 CSV for import.
But how can I parse the JSON and get 2 different CSV files as output?
Simple (nonsense) example:
testJQ.log
{"id":1234,"type":"A","group":"games"}
{"id":5678,"type":"B","group":"cars"}
using
cat testJQ.log|jq --raw-output '[.id,.type,.group]|#csv'>testJQ.csv
I get one file testJQ.csv
1234,"A","games
5678,"B","cars"
But I would like to get this
types.csv
1234,"A"
5678,"B"
groups.csv
1234,"games"
5678,"cars"
Can this be done without having to parse the JSON twice, first time creating the types.csv and second time the groups.csv like this?
cat testJQ.log|jq --raw-output '[.id,.type]|#csv'>types.csv
cat testJQ.log|jq --raw-output '[.id,.group]|#csv'>groups.csv
I suppose one way you could hack this up is to output the contents of one file to stdout and the others to stderr and redirect to separate files. Of course you're limited to two files though.
$ <testJQ.log jq -r '([.id,.type]|#csv),([.id,.group]|#csv|stderr|empty)' \
1>types.csv 2>groups.csv
stderr outputs to stderr but the value propagates to the output, so you'll want to follow that up with empty to swallow that up.
Personally I wouldn't recommend doing this, I would just write a python script (or other language) to parse this if you needed to output to multiple files.
You will either need to run jq twice, or to run jq in conjunction with another program to "split" the output of the call to jq. For example, you could use a pipeline of the form: jq -c ... | awk ...
The potential disadvantage of the pipeline approach is that if JSON is the final output, it will be JSONL; but obviously that doesn't apply here.
There are many ways to craft such a pipeline. For example, assuming there are no raw newlines in the CSV:
< testJQ.log jq -r '
"types", ([.id,.type] |#csv),
"groups", ([.id,.group]|#csv)' |
awk 'NR % 2 == 1 {out=$1; next} {print >> out".csv"}'
Or:
< testJQ.log jq -r '([.id,.type],[.id,.group])|#csv' |
awk '{ out = ((NR % 2) == 1) ? "types" : "groups"; print >> out".csv"}'
For other examples, see e.g.
Using jq how can I split a very large JSON file into multiple files, each a specific quantity of objects?
Splitting / chunking JSON files with JQ in Bash or Fish shell?
Split JSON into multiple files
Handling raw new-lines
Whether or not you split the CSV into multiple files, there is a potential issue with embedded raw newlines. One approach is to change "\n" in JSON strings to "\\n", e.g.
jq -r '([.id,.type],[.id,.group])
| map(if type == "string" then gsub("\n";"\\n") else . end)
| #csv'
Is there a way to convert JSONL to JSON in Linux with the full JSONL file depth? I found some methods based on jq but they don't work with full depth of JSONL file
Would something like this work?
#!/bin/sh
echo "[" >$1.json
perl -pe 's/$/,/' <$1 >>$1.json
echo "]" >>$1.json
I am quite confused as to what you want to do. But when it comes to jq, normally I process things line by line, with each line being an atomic JSON object. Something like
cat file | jq some-options 'some commands' > output.txt
Sometimes I get the output in tsv format and pipe it into awk. jq is very friendly with line-by-line objects.
To convert a large JSON list into line by line format, just parse the large object in any programming language, and serialize the inner objects back to json line by line.
But if you have already parsed the large object, I suggest you do the required processing you want to do in jq directly, without serializing the inner objects back...
I want to get value from JSON file:
Example:
{"name":"ghprbActualCommitAuthorEmail","value":"test#gmail.com"},{"name":"ghprbPullId","value":"226"},{"name":"ghprbTargetBranch","value":"master"},
My expect is :
I want to get test#gmail.com, 226 and master.
sed is the wrong tool for processing JSON.
Assuming you have a file tmp.json with valid JSON like
[{"name":"ghprbActualCommitAuthorEmail","value":"test#gmail.com"},
{"name":"ghprbPullId","value":"226"},
{"name":"ghprbTargetBranch","value":"master"}]
you can use jq '.[].value' tmp.son.
If the file instead contains
{"name":"ghprbActualCommitAuthorEmail","value":"test#gmail.com"}
{"name":"ghprbPullId","value":"226"}
{"name":"ghprbTargetBranch","value":"master"}
(i.e., just a stream of 3 separate JSON objects, you could use jq '.value' tmp.json, as jq will apply the filter to each object in succession. You can also use jq -s '.[].value' tmp.son, where the -s flag tells jq to read the entire input into an array first. This lets you use the same filter in both cases.