I have a list of IPv4 addresses being output in a list each separated by \n. The program I would like to import these into is expecting it in this format:
{
"data":[
{ "IP":"127.0.0.1" },
{ "IP":"192.168.0.1" }
]
}
Input data for the above would have been this:
127.0.0.1
192.168.0.1
I've looked in the jq cookbook for ideas but the closest I've been able to string together is using [] not {}, not inside data, and only has the value without key.
jq -sR '[sub("\n$";"") | splits("\n") | sub("^ +";"") | [splits(" +")]]'
Outputs:
[
[
"127.0.0.1"
],
[
"192.168.0.1"
]
]
Here is a solution:
jq -Rn '{data: [ {IP: inputs} ] }' input.txt
If this seems a bit magical, you might like to use the more mundane variant:
jq -Rn '{data: [ inputs | {IP: .} ] }' input.txt
Of course, in practice, you might also want to remove extraneous whitespace in the input, filter out comments, perform validity checking or filter out invalid input ...
Related
I am using jq to parse a json file. This is the current output from jq:
[
"key1.childk2",
"key2.childk3"
]
I would like to turn this into a readable json format itself as list of lists like below:
[
["key1","childk2"],
["key2","childk3"]
]
I would ideally prefer to do this with jq, however any other shell tool that can work on shell variables is fair game.
You can use jq split filter:
jq '[.[] | split(".")]'
[
[
"key1",
"childk2"
],
[
"key2",
"childk3"
]
]
You should use it somewhere in the original jq command
With jq you can structurally turn the input into your desired JSON document using simple filters like map(./"."), but regarding the requested readability, the output wouldn't have exactly your desired formatting.
Without any further flags, jq would pretty-print the output as:
[
[
"key1",
"childk2"
],
[
"key2",
"childk3"
]
]
Demo
Using the --compact-output or -c flag would compress the whole JSON document into one line, not just the elements of the outer array:
[["key1","childk2"],["key2","childk3"]]
Demo
So, if you really wanted to, you could also glue together the parts yourself as you like them to be formatted from within jq, but honestly, I would discourage you from doing so as by circumventing jq's internal JSON composer you might end up outputting invalid JSON.
jq -r '"[", " " + (map(./"." | tojson) | .[:-1][] += ",")[], "]"'
[
["key1","childk2"],
["key2","childk3"]
]
Demo
Here is a ruby to do that:
ruby -r json -e 'puts JSON.parse($<.read).map{|e| e.split(".")}.to_json' file
[["key1","childk2"],["key2","childk3"]]
Or if you want it pretty:
ruby -r json -e 'puts JSON.pretty_generate(
JSON.parse($<.read).map{|e| e.split(".")})
' file
[
[
"key1",
"childk2"
],
[
"key2",
"childk3"
]
]
Or you can produce your precise format:
ruby -r json -e '
l=[]
JSON.parse($<.read).map{|e| l << e.split(".").to_s}
puts "[\n\t#{l.join(",\n\t")}\n]"
' file
[
["key1", "childk2"],
["key2", "childk3"]
]
I've a folder with more than 1000 request logs (generated per hour/day) which are in the following format:
[
{
"input": {
"random_param_name_1": "random_value_1",
"random_param_name_2": "random_value_2",
"random_param_name_3": "random_value_3",
"random_param_name_4": "random_value_4"
},
"output": {
"some_key_we_dont_care_about": "some_value_we_dont_care_about"
},
"status_code": 200
},
{
"input": {
"random_param_name_1": "random_value_1",
"random_param_name_4": "random_value_4",
"random_param_name_3": "random_value_3",
"random_param_name_5": "random_value_5"
},
"output": {
"some_key_we_dont_care_about": "some_value_we_dont_care_about"
},
"status_code": 200
}
]
And I need to find all the input requests that are unique. For this, I need to do two things:
sort the keys in input as different inputs might have same keys but in different order
print all the key and value in a single line, so that I can pipe the output to sort | uniq to get all the unique input combinations.
Please note that the input keys are random, most existing questions in stackoverflow of the similar kind, know the keys in advance, but that's not the case here.
I can print the key and values like this:
jq -r 'keys[] as $k | "\($k):(.[$k])"'
but they end up being on new lines.
to summarise, for the above json, I need a magic_expression
$ jq 'magic_expression' log.json
that will return
"random_param_name_1":"random_value_1","random_param_name_2":"random_value_2","random_param_name_3":"random_value_3","random_param_name_4":"random_value_4"
"random_param_name_1":"random_value_1","random_param_name_3":"random_value_3","random_param_name_4":"random_value_4","random_param_name_5":"random_value_5"
Here is a "magic expression" to get you started.
It uses to_entries to make the objects appearing in .input more managable.
def format: "\"\(.key)\":\"\(.value)\"" ;
map(.input) | unique | map(to_entries)[] | map(format) | join(",")
When run with -r / --raw-output it produces
"random_param_name_1":"random_value_1","random_param_name_2":"random_value_2","random_param_name_3":"random_value_3","random_param_name_4":"random_value_4"
"random_param_name_1":"random_value_1","random_param_name_4":"random_value_4","random_param_name_3":"random_value_3","random_param_name_5":"random_value_5"
Try it online!
EDIT: if as customcommander points out you want the keys to be sorted you can move the format before the unique. e.g.
def format: "\"\(.key)\":\"\(.value)\"" ;
map(.input | to_entries | map(format) | sort ) | unique[] | join(",")
which produces
"random_param_name_1":"random_value_1","random_param_name_2":"random_value_2","random_param_name_3":"random_value_3","random_param_name_4":"random_value_4"
"random_param_name_1":"random_value_1","random_param_name_3":"random_value_3","random_param_name_4":"random_value_4","random_param_name_5":"random_value_5"
when run with -r / --raw-output
Try it online!
Consider this:
/workspaces # jq 'map(.input)' data.json
[
{
"random_param_name_1": "random_value_1",
"random_param_name_2": "random_value_2",
"random_param_name_3": "random_value_3",
"random_param_name_4": "random_value_4"
},
{
"random_param_name_1": "random_value_1",
"random_param_name_4": "random_value_4",
"random_param_name_3": "random_value_3",
"random_param_name_5": "random_value_5"
}
]
You can sort the keys of each object with --sort-keys:
/workspaces # jq --sort-keys 'map(.input)' data.json
[
{
"random_param_name_1": "random_value_1",
"random_param_name_2": "random_value_2",
"random_param_name_3": "random_value_3",
"random_param_name_4": "random_value_4"
},
{
"random_param_name_1": "random_value_1",
"random_param_name_3": "random_value_3",
"random_param_name_4": "random_value_4",
"random_param_name_5": "random_value_5"
}
]
Then pipe this into another jq filter:
/workspaces # jq --sort-keys 'map(.input)' data.json | jq -r 'map(to_entries)[] | map("\"\(.key)\":\"\(.value)\"") | join(",")'
"random_param_name_1":"random_value_1","random_param_name_2":"random_value_2","random_param_name_3":"random_value_3","random_param_name_4":"random_value_4"
"random_param_name_1":"random_value_1","random_param_name_3":"random_value_3","random_param_name_4":"random_value_4","random_param_name_5":"random_value_5"
I need to find all the input requests that are unique.
This can be done within jq, without any sorting of keys, since jq's == operator ignores the key order. For example, the following will produce the unique input requests in their original form (i.e. without the keys being sorted):
map(.input)
| group_by(.)
| map(.[0])
Since group_by uses ==, uniqueness is guaranteed.
If you really want the keys to be sorted, then you could use the -S command-line option:
jq -S -f program.jq input.json
And if for some reason you really want the non-standard output format, you could use the following modification of the above program:
map(.input)
| group_by(.)
| map(.[0])
| .[]
| . as $in
| [ keys[] as $k | "\"\($k)\":\"\($in[$k])\"" ] | join(",")
With your sample input, this last produces:
"random_param_name_1":"random_value_1","random_param_name_2":"random_value_2","random_param_name_3":"random_value_3","random_param_name_4":"random_value_4"
"random_param_name_1":"random_value_1","random_param_name_3":"random_value_3","random_param_name_4":"random_value_4","random_param_name_5":"random_value_5"
I have a JSON file containing application clients and their associated application features:
{
"client-A": [
"feature-x"
],
"client-B": [
"feature-x",
"feature-y"
],
"client-C": [
"feature-z"
],
"client-D": [
"feature-x",
"feature-z"
],
...
}
I'm trying to turn this into the following CSV:
client,feature
client-A,feature-x
client-B,feature-x
client-B,feature-y
client-C,feature-z
client-D,feature-x
client-D,feature-z
What's an easy way using jq to get this done?
Not sure whether this is the most efficient way of doing it, but you can convert use the following pipeline:
<yourfile.json jq -r 'to_entries | .[] | { key: .key, value: .value[] } | [ .key, .value ] | #csv'
to_entries converts the structure into "key value" pairs, which can then be operated on. The { key: .key, value: .value[] } bit will convert the array into multiple rows...
Using jq how can I convert an array into object indexed by filename, or read multiple files into one object indexed by their filename?
e.g.
jq -s 'map(select(.roles[]? | contains ("mysql")))' -C dir/file1.json dir/file2.json
This gives me the data I want, but I need to know which file they came from.
So instead of
[
{ "roles": ["mysql"] },
{ "roles": ["mysql", "php"] }
]
for output, I want:
{
"file1": { "roles": ["mysql"] },
"file2": { "roles": ["mysql", "php"] }
}
I do want the ".json" file extension stripped too if possible, and just the basename (dir excluded).
Example
file1.json
{ "roles": ["mysql"] }
file2.json
{ "roles": ["mysql", "php"] }
file3.json
{ }
My real files obviously have other stuff in them too, but that should be enough for this example. file3 is simply to demonstrate "roles" is sometimes missing.
In other words: I'm trying to find files that contain "mysql" in their list of "roles". I need the filename and contents combined into one JSON object.
To simplify the problem further:
jq 'input_filename' f1 f2
Gives me all the filenames like I want, but I don't know how to combine them into one object or array.
Whereas,
jq -s 'map(input_filename)' f1 f2
Gives me the same filename repeated once for each file. e.g. [ "f1", "f1" ] instead of [ "f1", "f2" ]
If your jq has inputs (as does jq 1.5) then the task can be accomplished with just one invocation of jq.
Also, it might be more efficient to use any than iterating over all the elements of .roles.
The trick is to invoke jq with the -n option, e.g.
jq -n '
[inputs
| select(.roles and any(.roles[]; contains("mysql")))
| {(input_filename | gsub(".*/|\\.json$";"")): .}]
| add' file*.json
jq approach:
jq 'if (.roles[] | contains("mysql")) then {(input_filename | gsub(".*/|\\.json$";"")): .}
else empty end' ./file1.json ./file2.json | jq -s 'add'
The expected output:
{
"file1": {
"roles": [
"mysql"
]
},
"file2": {
"roles": [
"mysql",
"php"
]
}
}
jq is suppose to
process/filter JSON inputs and producing the filter's results as JSON
However, I found that after the jq process/filter, output result is no longer in JSON format any more.
E.g., https://stedolan.github.io/jq/tutorial/#result5, i.e.,
$ curl -s 'https://api.github.com/repos/stedolan/jq/commits?per_page=5' | jq '.[] | {message: .commit.message, name: .commit.committer.name}'
{
"message": "Merge pull request #162 from stedolan/utf8-fixes\n\nUtf8 fixes. Closes #161",
"name": "Stephen Dolan"
}
{
"message": "Reject all overlong UTF8 sequences.",
"name": "Stephen Dolan"
}
. . .
Is there any workaround?
UPDATE:
How to wrap the whole return into a json structure of:
{ "Commits": [ {...}, {...}, {...} ] }
I've tried:
jq '.[] | Commits: [{message: .commit.message, name: .commit.committer.name}]'
jq 'Commits: [.[] | {message: .commit.message, name: .commit.committer.name}]'
but neither works.
Found it, on the same page,
https://stedolan.github.io/jq/tutorial/#result6
If you want to get the output as a single array, you can tell jq to “collect” all of the answers by wrapping the filter in square brackets:
jq '[.[] | {message: .commit.message, name: .commit.committer.name}]'
Technically speaking, unless otherwise instructed (notably with the -r command-line option), jq produces a stream of JSON entities.
One way to convert an input stream of JSON entities into a JSON array containing them is to use the -s command-line option.
Response to UPDATE
To produce a JSON object of the form:
{ "Commits": [ {...}, {...}, {...} ] }
you could write something like:
jq '{Commits: [.[] | {message: .commit.message, name: .commit.committer.name}]}'
(jq understands the '{Commits: _}' shorthand.)