How to delete all nested keys with JQ - json

I would like to delete all the resloved from a npm shrinwrap json file. this is causing a problem when running npm install on other machine.
"cssstyle": {
"version": "0.2.37",
"from": "cssstyle#>=0.2.29 <0.3.0",
"resolved": "https://registry.npmjs.org/cssstyle/-/cssstyle-0.2.37.tgz"
},
"dashdash": {
"version": "1.14.0",
"from": "dashdash#>=1.12.0 <2.0.0",
"resolved": "https://registry.npmjs.org/dashdash/-/dashdash-1.14.0.tgz",
"dependencies": {
"assert-plus": {
"version": "1.0.0",
"from": "assert-plus#>=1.0.0 <2.0.0",
"resolved": "https://registry.npmjs.org/assert-plus/-/assert-plus-1.0.0.tgz"
}
}
},
"debug": {
"version": "2.2.0",
"from": "debug#>=2.2.0 <3.0.0",
"resolved": "https://registry.npmjs.org/debug/-/debug-2.2.0.tgz"
}
How can I delete the resolved key from all the file
I'm using the pattern :
jq 'del(.resolved)' file.json

In my opinion, the simplest approach to this kind of problem is to use walk/1:
walk(if type == "object" and has("resolved") then del(.resolved) else . end)
If your jq does not have walk/1 (which was only included as a builtin after the release of jq 1.5), then simply add its definition (easily available on the web) before the above line, or perhaps include it in your ~/.jq file.

Related

Integrate json values into another file

I'm trying to update an existing json file from values in another json file using jq in a bash shell.
I've got a settings json file
{
"Logging": {
"MinimumLevel": {
"Default": "Information",
"Override": "Warning"
},
"WriteTo": [
{
"Name": "File",
"Args": {
"path": "./logs/log-.txt",
"rollingInterval": "Day"
}
}
]
},
"Settings": {
"DataServerUrl": "https://address.to.server.com",
"ServerKey": "1f969476798adfe95114dd28ed3a3ff"
"ServerTimeZone": "Mountain Standard Time",
"MaxOccupantCount": 6
}
}
In an integration step, I'm attempting to incorporate values for specific environments (think dev/staging/prod) from an external json file with limited setting values. An example of such a file is
{
"DataServerUrl": "https://dev.server.addr.com",
"ServerKey": "2a4d99233efea456b95114aa23ed342ae"
}
I can get to the data using jq. I can update the data using jq if I hard-code the updates. I'm looking for something general to take in any environment settings values and update them in the base settings file. My searches suggest I can do this in a single step without knowing the specific values. A command similar to
jq -r 'to_entries[]' settings.dev.json |
while IFS= read -r key value; do
jq -r '.[$key] |= [$value]' settings.json
done
What happens is I get error messages stating jq: error: $key is not defined at <top-level> (as well as the same message for $value). The messages appear several times in pairs. settings.json is not changed. Now, this makes partial sense because the output from just jq -r 'to_entries[]' settings.dev.json looks like (empty space in this output is included as produced by the command).
"key": "DataServerUrl",
"value": "https://dev.server.addr.com"
"key": "ServerKey",
"value": "2a4d99233efea456b95114aa23ed342ae"
How do I go about iterating over the values in the environment settings file such that I can use those values to update the base settings file for further processing (i.e., publishing to the target environment)?
The simplest way is to provide both files and address the second one using input. That way, all you need is the assignment:
jq '.Settings = input' settings.json insert.json
{
"Logging": {
"MinimumLevel": {
"Default": "Information",
"Override": "Warning"
},
"WriteTo": [
{
"Name": "File",
"Args": {
"path": "./logs/log-.txt",
"rollingInterval": "Day"
}
}
]
},
"Settings": {
"DataServerUrl": "https://dev.server.addr.com",
"ServerKey": "2a4d99233efea456b95114aa23ed342ae"
}
}
Demo
You could do something like
jq -s '.[1] as $insert | .[0].Settings |= $insert | .[0]' settings.json insert.json
Where we :
slurp both files
Save insert.json to a variable called $insert
Append (|=) $insert to .[0].Settings
Show only the first file .[0]
So the output will become:
{
"Logging": {
"MinimumLevel": {
"Default": "Information",
"Override": "Warning"
},
"WriteTo": [
{
"Name": "File",
"Args": {
"path": "./logs/log-.txt",
"rollingInterval": "Day"
}
}
]
},
"Settings": {
"DataServerUrl": "https://dev.server.addr.com",
"ServerKey": "2a4d99233efea456b95114aa23ed342ae"
}
}

How to define a variable within a JSON file and use it within JSON file

I'm trying to find if JSON file supports defining variables and using them within that JSON file?
{
"artifactory_repo": "toplevel_virtual_NonSnapshot",
"definedVariable1": "INSTANCE1",
"passedVariable2": "${passedFromOutside}",
"products": [
{ "name": "product_${definedVariable1}_common",
"version": "1.1.0"
},
{ "name": "product_{{passedVariable2}}_common",
"version": 1.5.1
}
]
}
I know YAML files allow this but now sure if JSON file allows this behavior or not. My plan is that a user will pass "definedVariable" value from Jenkins and I'll create a target JSON file (after substi
This might help you:
{
"artifactory_repo": "toplevel_virtual_NonSnapshot",
"definedVariable1": "INSTANCE1",
"passedVariable2": `${passedFromOutside}`,
"products": [
{ "name": `product_${definedVariable1}_common`,
"version": "1.1.0"
},
{ "name": `product_${passedVariable2}_common`,
"version": 1.5.1
}
]
}
*Note the use of `` instead of ''

Recursive extraction of object values and parent key name using jq

I have a requirement to parse the output of the npm ls --global --json command, such that I get a list of all the installed npm packages in the following format:
$package;$version;js;$resolved
Where:
$package is the key containing the package name, from each dependencies object.
$version is the version value taken from each package
js is just a literal string
$resolved is the resolved value taken from each package
I have gotten as far as this command syntax and output:
$ jq --raw-output 'select( has("dependencies") ) .dependencies[] | . as $d | "parentkey" + ";" + $d.version + ";js;" + $d.resolved'`
parentkey;5.5.1;js;
parentkey;1.1.3;js;https://registry.npmjs.org/yaml-table/-/yaml-table-1.1.3.tgz
The parts that I am specificly having difficulty with are as follows:
How can I get the key name value that I am iterating over in .dependencies that contains that package name. It seems that by that point I am looking at the contents of that object itself.
How can I recurse through ALL dependency objects? At the moment I'm only looking at the top level records in the root .dependencies object. I've discovered .. recursion, but I'm not quite sure how to apply it here.
Based on the example data below, I am trying to reach the following output results:
npm;5.5.1;js;
JSONStream;1.3.1;js;https://registry.npmjs.org/JSONStream/-/JSONStream-1.3.1.tgz
jsonparse;1.3.1;js;https://registry.npmjs.org/jsonparse/-/jsonparse-1.3.1.tgz
through;2.3.8;js;https://registry.npmjs.org/through/-/through-2.3.8.tgz
yaml-table;1.1.3;js;https://registry.npmjs.org/yaml-table/-/yaml-table-1.1.3.tgz
js-yaml;3.4.6;js;https://registry.npmjs.org/js-yaml/-/js-yaml-3.4.6.tgz
argparse;1.0.9;js;https://registry.npmjs.org/argparse/-/argparse-1.0.9.tgz
Some (much reduced) sample output npm ls --global --json that I have used for the above example, is as follows:
{
"dependencies": {
"npm": {
"version": "5.5.1",
"dependencies": {
"JSONStream": {
"version": "1.3.1",
"from": "JSONStream#~1.3.1",
"resolved": "https://registry.npmjs.org/JSONStream/-/JSONStream-1.3.1.tgz",
"dependencies": {
"jsonparse": {
"version": "1.3.1",
"from": "jsonparse#^1.2.0",
"resolved": "https://registry.npmjs.org/jsonparse/-/jsonparse-1.3.1.tgz"
},
"through": {
"version": "2.3.8",
"from": "through#>=2.2.7 <3",
"resolved": "https://registry.npmjs.org/through/-/through-2.3.8.tgz"
}
}
}
}
},
"yaml-table": {
"version": "1.1.3",
"from": "yaml-table#latest",
"resolved": "https://registry.npmjs.org/yaml-table/-/yaml-table-1.1.3.tgz",
"dependencies": {
"js-yaml": {
"version": "3.4.6",
"from": "js-yaml#3.4.6",
"resolved": "https://registry.npmjs.org/js-yaml/-/js-yaml-3.4.6.tgz",
"dependencies": {
"argparse": {
"version": "1.0.9",
"from": "argparse#>=1.0.2 <2.0.0",
"resolved": "https://registry.npmjs.org/argparse/-/argparse-1.0.9.tgz"
}
}
}
}
}
}
}
By using .., it will recurse through all values in the json tree. So you'll want to filter those out by objects that have the structure you're expecting. In this case, things that have a valid dependencies object. Once you've located the objects, you could extract the values you want.
jq -r '.. | .dependencies? | objects
| to_entries[] | [.key, .value.version, "js", .value.resolved] | join(";")' input.json
produces the results:
npm;5.5.1;js;
yaml-table;1.1.3;js;https://registry.npmjs.org/yaml-table/-/yaml-table-1.1.3.tgz
JSONStream;1.3.1;js;https://registry.npmjs.org/JSONStream/-/JSONStream-1.3.1.tgz
jsonparse;1.3.1;js;https://registry.npmjs.org/jsonparse/-/jsonparse-1.3.1.tgz
through;2.3.8;js;https://registry.npmjs.org/through/-/through-2.3.8.tgz
js-yaml;3.4.6;js;https://registry.npmjs.org/js-yaml/-/js-yaml-3.4.6.tgz
argparse;1.0.9;js;https://registry.npmjs.org/argparse/-/argparse-1.0.9.tgz

Get field from another json using jq

I have two .json-files.
The first is 1.json
{
"id": "107709375",
"type": "page",
"title": "SomeTitle",
"space": {
"key": "BUSINT"
},
"version": {
"number": 62
}
}
And the second one logg.json:
{
"id": "228204270",
"type": "page",
"status": "current",
"title": "test-test",
"version": {
"when": "2016-11-23T16:54:18.313+07:00",
"number": 17,
"minorEdit": false
},
"extensions": {
"position": "none"
}
}
Can I paste version.number from logg.json into version.number 1.json using jq? I need something like that (it's absolutely wrong):
jq-win64 ".version.number 1.json" = ".version.number +1" logg.json
Read logg.json as an argument file. You could then access its values to make changes to the other.
$ jq --argfile logg logg.json '.version.number = $logg.version.number + 1' 1.json
Of course you'll need to use double quotes to work in the Windows Command prompt.
> jq --argfile logg logg.json ".version.number = $logg.version.number + 1" 1.json
Although the documentation says to use --slurpfile instead, we only have a single object in the file so it would be totally appropriate to use --argfile instead.

Update inner attribute of JSON with jq

Could somebody help me to deal with jq command line utility to update JSON object's inner value?
I want to alter object interpreterSettings.2B263G4Z1.properties by adding several key-values, like "spark.executor.instances": "16".
So far I only managed to fully replace this object, not add new properties with command:
cat test.json | jq ".interpreterSettings.\"2B188AQ5T\".properties |= { \"spark.executor.instances\": \"16\" }"
This is input JSON:
{
"interpreterSettings": {
"2B263G4Z1": {
"id": "2B263G4Z1",
"name": "sh",
"group": "sh",
"properties": {}
},
"2B188AQ5T": {
"id": "2B188AQ5T",
"name": "spark",
"group": "spark",
"properties": {
"spark.cores.max": "",
"spark.yarn.jar": "",
"master": "yarn-client",
"zeppelin.spark.maxResult": "1000",
"zeppelin.dep.localrepo": "local-repo",
"spark.app.name": "Zeppelin",
"spark.executor.memory": "2560M",
"zeppelin.spark.useHiveContext": "true",
"spark.home": "/usr/lib/spark",
"zeppelin.spark.concurrentSQL": "false",
"args": "",
"zeppelin.pyspark.python": "python"
}
}
},
"interpreterBindings": {
"2AXUMXYK4": [
"2B188AQ5T",
"2AY8SDMRU"
]
}
}
I also tried the following but this only prints contents of interpreterSettings.2B263G4Z1.properties, not full object.
cat test.json | jq ".interpreterSettings.\"2B188AQ5T\".properties + { \"spark.executor.instances\": \"16\" }"
The following works using jq 1.4 or jq 1.5 with a Mac/Linux shell:
jq '.interpreterSettings."2B188AQ5T".properties."spark.executor.instances" = "16" ' test.json
If you have trouble adapting the above for Windows, I'd suggest putting the jq program in a file, say my.jq, and invoking it like so:
jq -f my.jq test.json
Notice that there is no need to use "cat" in this case.
p.s. You were on the right track - try replacing |= with +=