Append new element to JSON object in specific format using bash and jq - json

I would like to append an element to an existing JSON file where I have the working directory as the key and the working directory + the contents as the value in a string array format.
Lets say I have the following structure:
Docs (Directory)
|
+-- RandomFile.json
|
+-- Readme (Working Directory)
| |
| +-- Readme.md
| +-- Readyou.md
What I would like to achieve is the structure below with the working directory as the prefix for every element in the array.
"Readme": ["Readme/Readme.md", "Readme/Readyou.md"]
From the output above, I would like to append that to the contents of the RandomFile.json which currently looks like this:
{
"docs": {
"Doc": ["doc1"]
}
}
to this:
{
"docs": {
"Doc": ["doc1"],
"Readme": ["Readme/Readme.md", "Readme/Readyou.md"]
}
}
Is it something that can be managed straightforward using bash and jq?

This requires jq 1.6 in order to use the --args option.
$ jq --arg wd "$(basename "$PWD")" '.docs+={($wd): $ARGS.positional | map("\($wd)/\(.)")}' ../RandomFile.json --args *
{
"docs": {
"Doc": [
"doc1"
],
"Readme": [
"Readme/Readme.md",
"Readme/Readyou.md"
]
}
}
The shell is used to pass the base name of the current working directory as the variable $wd.
The shell is also used to pass the names of all the files in the current working directory as separate arguments.
The file to edit is assumed to be ../RandomFile.json; if you only know that there is a JSON file in the parent, you can use ../*.json instead.
Use += to update the .docs object of the original with a new key (the working directory) and list of file names. map prefixes each element of $ARGS.positional with $wd.

Related

jq bash Adding a json field to a json file

i'm stuck on jq input problem. I have a json file that looks like this:
{
"main_object": {
"child1": ["banana", "apple", "orange"]
}
}
I need to add another child object and rewrite this file, the problem is that this child object needs to be generated dynamically. so i'm doing this:
added_string=$(printf '.main_object += {%s: %s}' "$child_name" "$fruits")
Then I wrote this line, which worked well on my mac shell:
edited_json=$(cat $json_variable_file | jq $added_string)
When i tried to run all of this from a bash script i got this error:
jq: error: Could not open file +=: No such file or directory
jq: error: Could not open file {"child2":: No such file or directory
jq: error: Could not open file ["orange","potato","watermelon"]}: No such file or directory
So I tried many things so far, most of them still give me the same error, also tried doing this:
edited_json=$(cat $json_variable_file | jq <<< $added_string)
The error i got is this:
parse error: Invalid numeric literal at line 1, column 23
Really appreciate your time, the weird thing here is that it works completely fine, generating the needed json on my zsh but it does not work on bash.
With bash and zsh:
child_name="child2"
fruits='["orange","potato","watermelon"]'
added_string=$(printf '.main_object += {%s: %s}' "$child_name" "$fruits")
cat file | jq "$added_string" # quotes are important
Output:
{
"main_object": {
"child1": [
"banana",
"apple",
"orange"
],
"child2": [
"orange",
"potato",
"watermelon"
]
}
}

How to find something in a json file using Bash

I would like to search a JSON file for some key or value, and have it print where it was found.
For example, when using jq to print out my Firefox' extensions.json, I get something like this (using "..." here to skip long parts) :
{
"schemaVersion": 31,
"addons": [
{
"id": "wetransfer#extensions.thunderbird.net",
"syncGUID": "{e6369308-1efc-40fd-aa5f-38da7b20df9b}",
"version": "2.0.0",
...
},
{
...
}
]
}
Say I would like to search for "wetransfer#extensions.thunderbird.net", and would like an output which shows me where it was found with something like this:
{ "addons": [ {"id": "wetransfer#extensions.thunderbird.net"} ] }
Is there a way to get that with jq or with some other json tool?
I also tried to simply list the various ids in that file, and hoped that I would get it with jq '.id', but that just returned null, because it apparently needs the full path.
In other words, I'm looking for a command-line json parser which I could use in a way similar to Xpath tools
The path() function comes in handy:
$ jq -c 'path(.. | select(. == "wetransfer#extensions.thunderbird.net"))' input.json
["addons",0,"id"]
The resulting path is interpreted as "In the addons field of the initial object, the first array element's id field matches". You can use it with getpath(), setpath(), delpaths(), etc. to get or manipulate the value it describes.
Using your example with modifications to make it valid JSON:
< input.json jq -c --arg s wetransfer#extensions.thunderbird.net '
paths as $p | select(getpath($p) == $s) | null | setpath($p;$s)'
produces:
{"addons":[{"id":"wetransfer#extensions.thunderbird.net"}]}
Note
If there are N paths to the given value, the above will produce N lines. If you want only the first, you could wrap everything in first(...).
Listing all the "id" values
I also tried to simply list the various ids in that file
Assuming that "id" values of false and null are of no interest, you can print all the "id" values of interest using the jq filter:
.. | .id? // empty

Converting a list of files to json map

I've been sturggling to figure out how to use jq to map a list of files to a given JSON structure. I have the following list of files:
/tmp/jboss/ejb-jar_3_2.xsd
/tmp/jboss/example.crt
/tmp/jboss/example.key
/tmp/jboss/incomingrunasidentityloginmodule.zip
/tmp/jboss/jboss-eap-7.0.0.zip
/tmp/jboss/jboss-eap-7.0.8-patch.zip
/tmp/jboss/keystore.jks
/tmp/jboss/logger-api.sh
/tmp/jboss/mgmt-api-example.txt
/tmp/jboss/wildfly-dist-10.1.0.Final.zip
/tmp/jboss/wildfly-dist-7.1.0.GA-redhat-8.jar
that I would like to convert to the following JSON structure, where the dirname is the final path in the directory structure.
[
{ dirname: "jboss", filename: "/tmp/jboss/ejb-jar_3_2.xsd" },
{ dirname: "jboss", filename: "/tmp/jboss/example.crt" },
...
...
]
I figure the regex I can use to extract the path is fairly simple: .*/(.*)/ or something close to that. However, I can't figure out how to structure the jq filter. I figure it needs to be something like:
cat /tmp/files.txt | jq --raw-input '. | { dirname: .match(regex).capture[0], filename: .}'
is as close as I've gotten, but that has several issues. 1) I can''t figure out how to use a regex for the dirname, and 2) this doesn't actually build the list of objects - it just generates multiple objects.
Is there an easy jq filter that I can use for this?
jq -nR '
[inputs | capture(".*/(?<dirname>[^/]*)/[^/]+$") + { filname: .}]'

validating file name matches directory name

Have to write a ruby script which can validate that the key in s3 matches directory name.
I have a json file, path name- Apple/Employee/Background/test.tf
How can I validate that my directory path Apple/Employee/Background/ matches with the "key": "Apple/Employee/Background/" in json file.
I used jq and parsed json file to retrive the value of "key".
cat conf.json | jq '.terraform[] | .backend[] | .s3[] | .key'
"Apple/Employee/Background/terraform.tfstate"
I want to compare only Apple/Employee/Background with directory path and see if they are same or not.
json file looks like:
"terraform": [
{
"backend": [
{
"s3": [
{
"bucket": "terraform-dev",
"dynamodb": "terraform_files",
"encrypt": "true",
"key": "Apple/Employee/Background/terraform.tfstate"
}
]
}
],
"required_version": "~> 0.11.8"
}
]
the directory path looks like- Apple/Employee/Background/conf.tf
You can use File.dirname to get the directory path minus the base entry (in this case, 'terraform.tfstate')
# Assuming `key` contains the value from the JSON file
dir = File.dirname(key) # => "Apple/Employee/Background"
Note that the path is returned without a trailing slash.

Parsing this JSON to obtain a human readable list

I have a JSON file whit this structure. For every URL field, we have a RESULT field which contains hundreds of LINKS. Would it be somehow possible to parse it and obtain a (ie. csv) list which contains all the LINKS for every URL?
[{
"url": "https://example.org/yyy",
"result": "{\"links\":[{\"link\":\"https://example.org/xxx/xxx\",\"text\":\"\"
},
{
\"link\":\"https://example.org/xxx/xxx\",\"text\":\"\"
},
{
\"link\":\"https://example.org/xxx/xxx\",\"text\":\"yyy\"}[.......]
Thanks in advance
Here is a solution using jq. If data.json contains the sample data
[{"url": "https://example.org/yyy", "result": "{\"links\":[{\"link\":\"https://example.org/xx1/xx1\",\"text\":\"\"},{\"link\":\"https://example.org/xx2/xx2\",\"text\":\"\"}]}"}]
then the command
$ jq -Mr '.[].result | fromjson | .links[].link' data.json
produces
https://example.org/xx1/xx1
https://example.org/xx2/xx2
If you would like both the url and the links, the command
$ jq -Mr '.[] | .url as $url | .result | fromjson | "\($url),\(.links[].link)"' data.json
produces
https://example.org/yyy,https://example.org/xx1/xx1
https://example.org/yyy,https://example.org/xx2/xx2