I have json like this:
[ {"one": 1}, {"two": 2}]
and wish to convert it to this format:
{"one": 1}
{"two": 2}
to facilitate indexing it into ElasticSearch. (latter is called 'jsonl' format). JQ is my tool of preference but I can't figure out how to do this.
Thanks
The key is the -c command-line option, which produces JSONL:
jq -c '.[]' test_array.json
figured this out:
cat test_array.json |jq '.[]'
Related
I know that jq can take serialized JSON strings to format/filter/modify the content. However, as far as I know the output is always human readable form of JSON. Is there a way in Bash or jq itself, to stringify the output?
For example, given
echo '{"foo" : "bar"}' | jq -r .
I'm looking for an output that is "{\"foo\" : \"bar\"}"
You want -R, aka --raw-input, to make jq treat your input as a string instead of as an object.
echo '{"foo" : "bar"}' | jq -R .
By "stringify", you seem to mean "encode a JSON value as a JSON string".
$ echo '{"foo": "bar"}' | jq '#json'
"{\"foo\":\"bar\"}"
I want to create a JSON file from a list of files like:
$ ls -r *
folder1/
file1.pdf
file2.pdf
file3.pdf
folder2/
file4.pdf
file5.pdf
file6.pdf
I want a json file that looks like this:
{
'folder1' : [ 'file1.pdf', 'file2.pdf', 'file3.pdf' ],
'folder2' : [ 'file4.pdf', 'file5.pdf', 'file6.pdf' ]
}
For now I am able to create the list of files with jq but not sure how to name them. This is what I am doing now:
$ ls folder | jq -R -s 'split("\n") -[""]'
[
"file1.pdf",
"file2.pdf",
"file3.pdf"
]
Thanks a lot for the help!
PS. Additionally, I need to include a prefix in the name of files. I can try to do it with sed but if maybe there is an easier way to do it here, that would be even better.
You shouldn't parse the output of ls. Pass folder1/file1.pdf, folder1/file2.pdf, etc. to JQ as positional arguments instead, and parse them in there.
jq -n 'reduce ($ARGS.positional[] / "/") as [$k, $v] (.; .[$k] += ["prefix-\($v)"])' --args */*
I have the following JSON:
{
"contexts": {
"context1": {
"mydata": "value1"
},
"context2": {
"mydata": "value2"
}
},
"current_context": "context2"
}
I'd like to use jq to output the value of mydata for the context indicated by current_context. The above would output value2. If I change the JSON to have "current_context": "context1", I'd get value1.
Given two invocations of jq, and the above JSON content in a file called jq.json, this works:
jq -r --arg context "$(jq -r '.current_context' jq.json)" '.contexts | to_entries[] | select(.key == $context) | .value.mydata' jq.json
Is there a way to do this with a single invocation of jq?
Thanks!
Like this:
jq -r '.contexts[.current_context].mydata' file.json
You could also use a variable:
jq -r '.current_context as $cc|.contexts[$cc].mydata' file.json
and here's an alternative to jq solution, based on a walk-path unix utility jtc:
bash $ jtc -w'[current_context]<cc>v[^0]<cc>t[mydata]' file.json
"value2"
bash $
walk path (-w) breakdown:
[current_context]<cc>v - get to the current_context value and memorize it in the namespace cc (directive <cc>v does it)
[^0]<cc>t[mydata] - reset walk path back to root ([^0]) and recursively search for the label (tag) stored in the namespace (<cc>t), then address found JSON object by label mydata.
PS> Disclosure: I'm the creator of the jtc - shell cli tool for JSON operations
I have a JSON which looks like this:
{
"lorem": "ipsum",
"dolor": "sid",
"data": {
"key1": "value1",
"key2": "value2"
}
}
and I want an output which is ini like where I only need the content of 'data' (which is always flat, no branches). The output should look like this:
key1=value1
key2=value2
I can use jq (just don't get it running) but have to use a bash script for it. Can anyone help?
jq solution:
jq -r '.data | to_entries[] | "\(.key)=\(.value)"' input.json
The output:
key1=value1
key2=value2
This will work in BASH.
#!/bin/bash
all_keys=$( cat input.txt );
while read key
do
grep "$key" ./text.txt | awk -F':' '{ print $1"="$2}' | tr -d '[", ]'
done <<< "$all_keys"
Assuming you values are in text.txt and that you have your keys in input.txt.
Regards!
In python
This reads stdin and outputs the desired "key=value" one per line
#!/usr/bin/python
import json
import sys
data = sys.stdin.read()
json_structure=json.loads(data)
start_point=json_structure["data"]
for k in start_point.keys():
print("%s=%s" % (k, start_point[k]))
If I was using python to unmangle the json input however, I would probably rewrite the bash script in python
I have a JSON and I need to extract a base64-encoded value by particular key and decode it.
JSON has the following structure:
[
{
"LockIndex": 0,
"Key": "Arul/key1",
"Flags": 0,
"Value": "MzAKCg==",
"CreateIndex": 369,
"ModifyIndex": 554
}
]
In the above JSON, I need to extract only "Value":"MzAKCg==" and decode the base64-encoded "MzAKCg==" value. I would like to perform this using shell scripting.
Please assist.
jq has recently added support for base64 encoding and decoding
https://stedolan.github.io/jq/manual/#Formatstringsandescaping
#base64:
The input is converted to base64 as specified by RFC 4648.
#base64d:
The inverse of #base64, input is decoded as specified by RFC 4648.
Note: If the decoded string is not UTF-8, the results are undefined.
For your data, the command would be
jq -r 'map(.Value | #base64d)' < file.json
https://github.com/stedolan/jq/issues/47
It's not released yet, but you can install the latest development version to use it.
brew reinstall --HEAD jq
Once the next version of jq is released then you can switch back to the latest stable version.
Using jq and base64:
jq -r '.[].Value' < file.json | base64 --decode
if this JSON will have always the same structure, you can use cut -d and later decode the value, for example:
$echo "Value": "MzAKCg==" | cut -d ":" -f 2 | base64 -D
30
This worked for me: curl --silent localhost:8500/v1/kv/key1 | jq -r '.[0].Value | #base64d'
use built-in in jq base64 decoder:
echo '{"val" : "MzAKCg=="}' | jq -r '.val |=#base64d'
yields:
{ "val": "30\n\n" }
side note: jq is simply a file symbolic for jq-win64.exe
if you want to keep the origin struct, you could use like jq -r '.|=map(.key|=#base64d)
With xidel:
xidel -s input.json -e '$json()/binary-to-string(base64Binary(Value))'
or
xidel -s input.json -e 'binary-to-string(base64Binary($json//Value))'