I have a JSON file that looks like this:
{
"key1": "value1",
"key2": "value2",
...
"keyn": "valuen"
}
No arrays or nested objects, just a simple key->value map. I want to use jq to convert this file to a plain text configuration file of a following format:
KEY1=value1
KEY2=value2
...
KEYn=valuen
(the keys should be uppercased in the result file). I have searched various jq tutorials available on the Net expecting that such an obvious (in my opinion) example would be covered there (at least without uppercasing), but it is not. All tutorials I have found use more complicated examples, like extracting specific values from JSON file, handling arrays or nested structures etc. However, no simple JSON-to-text conversion. man jq doesn't help either, it's not too clear. Could you help me in obtaining the result I want?
Note: it must be done with jq, not any other tool, because it will be used in a script that has access to jq, but no other tool for handling JSON is guaranteed to be present on the system.
You can iterate over the entries in the object with to_entries, then format each to a string and use the -r (--raw-output) flag, like so:
$ cat example.json
{
"key1": "value1",
"key2": "value2",
"keyn": "valuen"
}
$ jq -r 'to_entries[] | (.key | ascii_upcase) + "=" + .value' < example.json
KEY1=value1
KEY2=value2
KEYN=valuen
Related
I'm new to jq (around 24 hours). I'm getting the filtering/selection already, but I'm wondering about advanced I/O features. Let's say I have an existing jq query that works fine, producing a stream (not a list) of objects. That is, if I pipe them to a file, it produces:
{
"id": "foo"
"value": "123"
}
{
"id": "bar"
"value": "456"
}
Is there some fancy expression I can add to my jq query to output each object individually in a subdirectory, keyed by the id, in the form id/id.json? For example current-directory/foo/foo.json and current-directory/bar/bar.json?
As #pmf has pointed out, an "only-jq" solution is not possible. A solution using jq and awk is as follows, though it is far from robust:
<input.json jq -rc '.id, .' | awk '
id=="" {id=$0; next;}
{ path=id; gsub(/[/]/, "_", path);
system("mkdir -p " path);
print >> path "/" id ".json";
id="";
}
'
As you will need help from outside jq anyway (see #peak's answer using awk), you also might want to consider using another JSON processor instead which offers more I/O features. One that comes to my mind is mikefarah/yq, a jq-inspired processor for YAML, JSON, and other formats. It can split documents into multiple files, and since its v4.27.2 release it also supports reading multiple JSON documents from a single input source.
$ yq -p=json -o=json input.json -s '.id'
$ cat foo.json
{
"id": "foo",
"value": "123"
}
$ cat bar.json
{
"id": "bar",
"value": "456"
}
The argument following -s defines the evaluation filter for each output file's name, .id in this case (the .json suffix is added automatically), and can be manipulated to further needs, e.g. -s '"file_with_id_" + .id'. However, adding slashes will not result in subdirectories being created, so this (from here on comparatively easy) part will be left over for post-processing in the shell.
I am having trouble accessing bash variable inside 'jq'.
The snippet below shows my bash loop to check for missing keys in a Json file.
#!/bin/sh
for key in "key1" "key2.key3"; do
echo "$key"
if ! cat ${JSON_FILE} | jq --arg KEY "$key" -e '.[$KEY]'; then
missingKeys+=${key}
fi
done
JSON_FILE:
{
"key1": "val1",
"key2": {
"key3": "val3"
}
}
The script works correctly for top level keys such as "key1". But it does not work correctly (returns null) for "key2.key3".
'jq' on the command line does return the correct value
cat input.json | jq '.key2.key3'
"val3"
I followed answers from other posts to come to this solution. However can't seem to figure out why it does not work for nested json keys.
Using --arg prevents your data from being incorrectly parsed as syntax. Usually, a shell variable you're passing into jq contains literal data, so this is the correct thing.
In this case, your variable contains syntax, not literal data: The . isn't part of the string you want to do a lookup by, but is instead an instruction to jq to do two separate lookups one after the other.
So, in this case, you should do the more obvious thing, instead of using --arg:
jq -e ".$KEY"
I need to transform an array by adding additional objects -
I have:
"user_id":"testuser"
"auth_token":"abcd"
I need:
"key":"user_id"
"value":"testuser"
"key":"auth_token"
"value":"abcd"
I have been using jq but cant figure out how to do it. Do i need to transform this into a multi-dimensional array first?
I have tried multiple jq queries but cant find the most suitable
When i try using jq i get
jq: error: syntax error, unexpected $end, expecting QQSTRING_TEXT or QQSTRING_INTERP_START or QQSTRING_END (Unix shell quoting issues?) at , line 1
Your input is not json, it's just a bunch of what could be thought of as key/value pairs. Assuming your json input actually looked like this:
{
"user_id": "testuser",
"auth_token": "abcd"
}
You could get an array of key/value pair objects using to_entries.
$ jq 'to_entries' input.json
[
{
"key": "user_id",
"value": "testuser"
},
{
"key": "auth_token",
"value": "abcd"
}
]
If on the other hand your input was actually that, you would need to convert it to a format that can be processed. Fortunately you could read it in as a raw string and probably parse using regular expressions or basic string manipulation.
$ jq -Rn '[inputs|capture("\"(?<key>[^\"]+)\":\"(?<value>[^\"]*)\"")]' input.txt
$ jq -Rn '[inputs|split(":")|map(fromjson)|{key:.[0],value:.[1]}]' input.txt
You can use to_entries filter for that.
Here is jqplay example
Robust conversion of key:value lines to JSON.
If the key:value specifications would be valid JSON except for the
missing punctuation (opening and closing braces etc), then a simple and quite robust approach to converting these key:value pairs to a single valid JSON object is illustrated by the following:
cat <<EOF | jq -nc -R '["{" + inputs + "}" | fromjson] | add'
"user_id": "testuser"
"auth_token" : "abcd"
EOF
Output
{
"user_id": "testuser",
"auth_token": "abcd"
}
I have a json file named output.json. It has a simple key:value format, e.g.:
{
"key":"value",
"key":"value",
"key":"value",
"key":"value",
}
I want to extract "value part".
If anyone can write me a command that will be really helpful.
With jq (which is much better suited for parsing and filtering JSON than grep/sed/awk/etc) you can extract all values with values function:
$ echo '{"a":1, "b":2, "c":3}' | jq '.[]|values'
1
2
3
Alternatively (since you mention you already use Python in your pipeline), you can do it like:
#!/usr/bin/env python
import json
my_values = json.load('output.json').values()
I'm tailing some logs, and to be able to read them easier, I use jq (http://stedolan.github.io/jq/), but either it's missing something, or I don't know how to do what I need to do.
So all lines are json, and currently I'm doing:
tail -f /path/to/log | jq .
Issue is, sometimes, I have stuff like this (when logging http responses):
{
"foo": "bar",
"fi": "bo",
"stream": "{\n \"json\": \"asAString\"\n}"
}
And obviously, would like to end up with something like this:
{
"foo": "bar",
"fi": "bo",
"stream": {
"json": "asAString"
}
}
Although we can assume in a first time I know the specific name of this fields that needs to be processed in a particular way, if you have an automated way to do that :-)
Thanks!
That's not a nested object... it's a json string. It's already pretty printed as it should. You need to parse the string if that's what you want.
.stream |= fromjson
For me what worked is to take your json string and pipe it to jq using the -r (raw) flag. for example if you have a json object where the value is a long stringified log message like
{"message":"some long string with \"escaped characters \n"}
You take that input and pipe it to jq like | jq -r .message