Say I have input like
{"DESCRIPTION": "Need to run script to do stuff", "PRIORITY": "Medium"}
but also get input like
{"STACK_NAME": "applecakes", "BACKEND_OR_INTEGRATIONS": "integrations", "PRIORITY": "Medium"}
ie, the parameters can be completely different.
I need to get the output in a format more friendly to send to Jira to make tickets. Specifically, I would like to strip the json formatting away, and insert a \n between each keypair. Here's what the above samples should look like:
DESCRIPTION: Need to run script to do stuff\nPRIORITY: Medium
STACK_NAME: applecakes\nBACKEND_OR_INTEGRATIONS: integrations\nPRIORITY: Medium
There can be a little flexibility in that if, for example, more spaces were needed or whatever.
So far I've got this worked out (assuming my input is stored in a variable called description
echo $description | jq -r "to_entries|map(\"\(.key)=\(.value|tostring)\")|.[]"
This works to strip away the JSON formatting, but doesn't handle newlines. I'm stumped on how to make sure I split only on each keypair, not on say every space or anything equally messy. What do I need to add to include newlines? Is a map even my best choice?
Just join what the array of strings with \\n (the sequence of the \ character which we need to escape and the n character) and use raw-output :
jq --raw-output 'to_entries | map("\(.key) : \(.value)") | join("\\n")'
Try it here.
Or more efficiently and more simply:
jq -r 'to_entries[] | "\(.key) : \(.value)"'
This produces one line per key-value pair.
The two-character sequence \n as a join-string
With your sample JSON, the invocation:
jq -j -r 'to_entries[] | "\(.key) : \(.value)", "\\n" '
would produce:
STACK_NAME : applecakes\nBACKEND_OR_INTEGRATIONS : integrations\nPRIORITY : Medium\n
Notice the trailing "\n".
Related
I have a json file with thousands of lines and 90 json objects.
Each object come with the following structure:
{
"country_codes": [
"GB"
],
"institution_id": "ins_118309",
"name": "Barclaycard (UK) - Online Banking: Personal", // I want to extract this line only
"oauth": true,
"products": [
"assets",
"auth",
"balance",
"transactions",
"identity",
"standing_orders"
],
"routing_numbers": []
},
For the ninety objects, I would like to delete all the lines and keep only the one with the name of the institution.
I guess that I will have to use a regex here?
I'm happy to use with vim, sublime, vscode or any other code editor that will alow me to do so
How can I extract these lines so I will stay with the following 90 lines?
"name": "Barclaycard (UK) - Online Banking: Personal",
"name": "Metro Bank - Commercial and Business Online Plus",
...
...
"name": "HSBC (UK) - Business",
If you must use a code editor, then in Vim you can delete all lines not
matching a pattern with: :v/^\s*"name":/d
The above pattern says:
^ line begins with
\s* zero or more white spaces
"name:" (pretty explanatory)
Although it's better to use a dedicated tool for parsing json files
rather than regex as json is not a 'regular
language'.
Bonus
If you do end up doing it in Vim, you can finish up by left align all the lines, do :%left or even just :%le.
That doesn't sound like the job for a text editor or even for regular expressions. How about using the right tool for the job™?
# print only the desired fields to stdout
$ jq '.[] | .name' < in.json
# write only the desired fields to file
$ jq '.[] | .name' < in.json > out.json
See https://stedolan.github.io/jq/.
If you really want to do it from a text editor, the simplest is still to filter the current buffer through a specialized external tool. In Vim, it would look like this:
:%!jq '.[] | .name'
See :help filter.
FWIW, here it is with another right tool for the job™:
:%!jj \\#.name -l
See https://github.com/tidwall/jj.
you can use grep eventually :
grep '^\s*"name":' your_file.json
In VSC
select institution_id
execute Selection > Select All Occurenses
Arrow Left
Ctrl+X
Esc
Ctrl+V
In vscode (although I would think it is the same for any regex-handling editor), use this Find:
^(?!\s*"name":.*).*\n?|^\s*
and replace with nothing. See regex101 demo.
^(?!\s*"name":.*).*\n? : get all lines that are not followed by "name":... including the newline so that line is completely discarded.
^\s* gets the whitespace before "name":...... - also discarded since we are replacing all matches with nothing.
Parsing JSON in Vim natively:
call getline(1, '$')
\ ->join("\n")
\ ->json_decode()
\ ->map({_, v -> printf('"name": "%s",', v.name)})
\ ->append('$')
NB. Line continuation is only available when sourcing script from file. If run interactively then type command on a single line.
I am trying to parse a very large file which consists of JSON objects like this:
{"id": "100000002", "title": "some_title", "year": 1988}
Now I also have a very big list of ID's that I want to extract from the file, if they are there.
Now I know that I can do this:
jq '[ .[map(.id)|indices("1", "2")[]] ]' 0.txt > p0.json
Which produces the result I want, namely fills p0.json with only the objects that have "id" 1 and "2". Now comes the problem: my list of id's is very long too (100k or so). So I have a Python programm that outputs the relevant id's. My line of thought was, to first assign that to a variable:
REL_IDS=`echo python3 rel_ids.py`
And then do:
jq --arg ids "$REL_IDS" '[ .[map(.id)|indices($ids)[]] ]' 0.txt > p0.json
I tried both with brackets [$ids] and without brackets, but no luck so far.
My question is, given a big amount of arguments for the filter, how would I proceed with putting them into my jq command?
Thanks a lot in advance!
Since the list of ids is long, the trick is NOT to use --arg. However, the details will depend on the details regarding the "long list of ids".
In general, though, you'd want to present the list of ids to jq as a file so that you could use --rawfile or --slurpfile or some such.
If for some reason you don't want to bother with an actual file, then provided your shell allows it, you could use these file-oriented options with process substitution: <( ... )
Example
Assuming ids.json contains a lising of the ids as JSON strings:
"1"
"2"
"3"
then one could write:
< objects.json jq -c -n --slurpfile ids ids.json '
inputs | . as $in | select( $ids | index($in.id))'
Notice the use of the -n command-line option.
Consider a JSON object like
{
"foo": 42,
"baz": -12,
"bar{label1=\"value1\"}": 12.34
}
constructed by jq using some data source. The actual key names and their amount may vary, but the result will always be an object with numbers (int or float) as values. The keys may contain quotation marks, but no whitespaces.
Can I use jq to format the object into a Prometheus-compatible format so I can just use the output to push the data to a Prometheus Pushgateway?
The required result would look like
foo 42
bar{label1="value1"} 12.34
baz -12
i.e. space-separated with newlines (no \r) and without quotes except for the label value.
I can't use bash for post-processing and would therefore prefer a pure jq solution if possible.
Use keys_unsorted to get object keys (keys does the same as well but the former is faster), generate desired output by means of string interpolation.
$ jq -r 'keys_unsorted[] as $k | "\($k) \(.[$k])"' file
foo 42
baz -12
bar{label1="value1"} 12.34
And, by adding -j option and printing line feed manually as #peak suggested you can make this portable.
On a Windows platform, jq will normally use CR-LF for newlines; to prevent this, use the -j command-line option and manually insert the desired 'newline' characters like so:
jq -rj 'to_entries[] | "\(.key) \(.value)\n"' file
I am running the following command:
sudo clustat | grep primary | awk 'NF{print $1",""server:"$2 ",""status:"$3}'
Results are:
service:servicename,server:servername,status:started
service:servicename,server:servername,status:started
service:servicename,server:servername,status:started
service:servicename,server:servername,status:started
service:servicename,server:servername,status:started
My desired result is:
{"service":"servicename","server":"servername","status":"started"}
{"service":"servicename","server":"servername","status":"started"}
{"service":"servicename","server":"servername","status":"started"}
{"service":"servicename","server":"servername","status":"started"}
{"service":"servicename","server":"servername","status":"started"}
I can't seem to put the qoutation marks withour srewing up my output.
Use jq:
sudo clustat | grep primary |
jq -R 'split(" ")|{service:.[0], server:.[1], status:.[2]}'
The input is read as raw text, not JSON. Each line is split on a space (the argument to split may need to be adjusted depending on the actual input). jq ensures that values are properly quoted when constructing the output objects.
Don't do this: Instead, use #chepner's answer, which is guaranteed to generate valid JSON as output with all possible inputs (or fail with a nonzero exit status if no JSON representation is possible).
The below is only tested to generate valid JSON with the specific inputs shown in the question, and will quite certainly generate output that is not valid JSON with numerous possible inputs (strings with literal quotes, strings ending in literal backslashes, etc).
sudo clustat |
awk '/primary/ {
print "{\"service\":\"" $1 "\",\"server\":\"" $2 "\",\"status\":\""$3"\"}"
}'
For JSON conversion of common shell commands, a good option is jc (JSON Convert)
There is no parser for clustat yet though.
clustat output does look table-like, so you may be able to use the --asciitable parser with jc.
I have to curl to a site (statuscake.com) that sends multiple items back in a JSON, each line of which contains multiple items. I want to extract from each line two of them, WebsiteName and TestID, so I can check if WebsiteName matches the one I'm interested in, get the TestID out and pass this to a second curl statement to delete the test.
Although it's more complex, the JSON that comes back is essentially of the form
[{"TestID": 123, "WebsiteName": "SomeSite1"}, {"TestID": 1234, "WebsiteName": "SomeSite2"}]
I can't seem to find a magic jq command to do it all in one - if there is one, I'd be really happy to see it.
I've got
cat $data | jq '[.[] | .WebsiteName]'
to get an array of the website names (and a very similar one for the TestIDs, but I think I've done something daft. data is the information coming back from the curl to get the JSON and that's populated OK.
I want to be able to assign these to two arrays, names and ids, then search names for the index of the relevant name, grab the id from ids and pass that to the curl. Unless there's a better way.
Any advice please?
My Xidel can do it all at once by selecting the JSON with a XPath-like query:
E.g. return all ids where the WebsiteName contains "site2" from an array of objects:
xidel /tmp/x.json -e '$json()[contains((.).WebsiteName, "site2")]/TestID'
Or e.g. to download the original JSON and then make the HTTP request with the ids:
xidel http://statuscake.com/your-url... -f '$json()[contains((.).WebsiteName, "site2")]/TestID!x"/your-delete-url{.}..."'
If I'm getting your question right, it sounds like what you want is to, for each element, select those where .WebsiteName == "needle", and then get .TestID from it. You can do just that:
.[] | select(.WebsiteName == "needle") | .TestID
If you want an array as the result, you can wrap the above script in square brackets.
The jq filters startswith and endswith may be of interest to you. If you're going to pass the result back to cURL, you may also be interested in the #sh formatting filter and the -r command-line flag.
Assuming you have a bash 4+ and assuming the json is valid (does not contain newlines in strings, etc.) this works:
$ echo "$data"
[{"TestID": 123, "WebsiteName": "SomeSite1"}, {"TestID": 1234, "WebsiteName":
"SomeSite2"}, {"TestID": 555, "WebsiteName": "foo*ba#r blah[54]quux{4,5,6}"}]
$ declare -A arr
$ while IFS= read -r line; do
eval "$line"
done < <(jq -M -r '.[] | #sh "arr[\(.WebsiteName)]+=\(.TestID)"' <<<"$data")
$ declare -p arr
declare -A arr='(["foo*ba#r blah[54]quux{4,5,6}"]="555" [SomeSite2]="1234" [SomeSite1]="123" )'
Here is a solution using only jq primitives.
.[]
| if .WebsiteName == "SomeSite1" then .TestID else empty end
This is essentially the same as Santiago's answer but if you are new to jq it may be informative because select/1 is defined as
def select(f): if f then . else empty end;