I have a bash script that outputs a json like this:
{
"name": "some",
"desc": "this is a desc",
"env": "this is an env type",
"dd": {
"one": "rr",
"two": "aa"
},
"url": "http://someurl",
//etc......
}
I would like to pipe a new command in my script, to return the final json output as:
{
"name": "some",
"env": "this is an env type",
"dd": {
"one": "rr",
"two": "aa"
}
}
How can i achieve this without installing new tools/libs like jq etc..
Any clue?
I know you've stipulated that you do this without external tools but hopefully this will change your mind:
jq '{ name, env, dd }' file.json
That was easy!
jq is very easy to obtain.
A quick and dirty python script would also work:
import sys
import json
with open(sys.argv[1]) as file:
obj = json.load(file)
print json.dumps({ key: obj[key] for key in ("name", "env", "dd") })
It can be run like python script.py file.json. To improve the formatting, you can pass extra arguments to json.dumps (see the docs).
How can i achieve this without installing new tools/libs like jq etc..
Any clue?
Well, to make an analogy it's like you were asking "how can I become a world champion in 100m sprint run, but I have no legs and I want no fake legs". The simple answer is that you cannot, or if you do, it won't be flexible and generic enough to be really useful.
Doing shell scripting is like having a toolbox, with many tools, each being designed to do one task and do it well. So if you refuse the tool that might be the right thing for what you need, then you cannot do it.
So the right way to do it is to use a tool such as jq, or a small python/ruby/... script that will take keys out of your json data.
Related
I'm trying to write a script that can add the package.json dependencies from one project to another one programmatically.
This is probably trivial jq, although I did search around and couldn't find anything that does this exactly.
What I ultimately want to do is this:
yarn add $(get-my-deps.sh)
And so I'm writing get-my-deps right now.
I know the path to my package.json file
I know that yarn add package-name#^1.23.45 package-name-2#^2.3.4 does what I want
I figure I want to do something like:
jq ".dependencies | keys" package.json
// ^ this is where I'm stuck
I don't want just a list of package names, I want to convert the hash of package-name/version-range data into a string with their names and ranges, like this:
babel#^7.3.12 react#^16.0.0 react-dom#^16.0.0 webpack#^4.0.0
I think I just don't understand how the... caret works in jq. I imagine it's the conceptual equivalent of regex backreferencing.
Here's the JS code:
module.exports = () => {
const package = require('./package.json')
// or, if you prefer
const FS = require('fs')
const package = FS.readFileSync('./package.json', 'utf8')
return Object.keys(package.dependencies)
// convert to array of strings: "{name}#{range}"
.reduce((modules, modName) => modules.concat(`${modName}#${package.dependencies[modName]}`), [])
// convert to single string for bash one-liner
.join(' ')
}
I'm sure this is a pretty trivial task for anyone who is fluent with jq, so I'm going to wait a couple days so that folks can take time to put together a good explanation.
Thanks for helping me get my head wrapped around jq.
Here's an example input:
{
"name": "chalk",
"version": "3.0.0",
"description": "Terminal string styling done right",
"license": "MIT",
"repository": "chalk/chalk",
"main": "source",
"engines": {
"node": ">=8"
},
"scripts": {
"test": "xo && nyc ava && tsd",
"bench": "matcha benchmark.js"
},
"dependencies": {
"ansi-styles": "^4.1.0",
"supports-color": "^7.1.0"
},
"devDependencies": {
"ava": "^2.4.0",
"coveralls": "^3.0.7",
"execa": "^3.2.0",
"import-fresh": "^3.1.0",
"matcha": "^0.7.0",
"nyc": "^14.1.1",
"resolve-from": "^5.0.0",
"tsd": "^0.7.4",
"xo": "^0.25.3"
}
}
What must be produced:
ansi-styles#^4.1.0 supports-color#^7.1.0
You were close; all you need was to realize how to store keys in a variable, so that you can combine each key with its corresponding value to form a single string.
Here's how you do it:
.dependencies | [ keys[] as $k | "\($k)#\(.[$k])" ] | join(" ")
Online demo at jqplay.org
keys[] expands to a stream of keys in dependencies; with as $k, we're telling jq to store each member of this stream in a variable called $k and make it available on the right hand side of the pipe. "\($k)#\(.[$k])" simply forms a string by combining $k, # and .[$k] (i.e: the value corresponding to $k). By wrapping this expression between brackets, we're storing each value it yields in an array, and by piping this array to join(" "), we're joining the array elements by a space to form a single string. That's all.
Well, I came up with this. It works. But if this isn't the idiomatic way to do it, I'd welcome an improved version.
jq ".dependencies | to_entries | map_values( .key + \"#\" + .value ) | join(\" \")" package.json
//> "ansi-styles#^4.1.0 supports-color#^7.1.0"
POC repl
I have a JSON file very similar to the following:
[
{
"uuid": "832390ed-58ed-4338-bf97-eb42f123d9f3",
"name": "Nacho"
},
{
"uuid": "5b55ea5e-96f4-48d3-a258-75e152d8236a",
"name": "Taco"
},
{
"uuid": "a68f5249-828c-4265-9317-fc902b0d65b9",
"name": "Burrito"
}
]
I am trying to figure out how to use the JQ command line processor to first find the UUID that I input and based on that output the name of the associated item. So for example, if I input UUID a68f5249-828c-4265-9317-fc902b0d65b9 it should search the JSON file, find the matching UUID and then return the name Burrito. I am doing this in Bash. I realize it may require some outside logic in addition to JQ. I will keep thinking about it and put an update here in a bit. I know I could do it in an overly complicated way, but I know there is probably a really simple JQ method of doing this in one or two lines. Please help me.
https://shapeshed.com/jq-json/#how-to-find-a-key-and-value
You can use select:
jq -r --arg query Burrito '.[] | select( .name == $query ) | .uuid ' tst.json
I need to alter some values in JSON data, and would like to include it in an already existing shell script. I'm trying to do so using jq, and will need the "sub()" function to cut off a piece of a string value.
Using this command line:
jq '._meta[][].ansible_ssh_pass | sub(" .*" ; "")'
with the data below will correctly replace the value (cutting off anything including the first space in the data), but only prints out the value, not the complete JSON structure.
Here's sample JSON data:
{_meta": {
"hostvars": {
"10.1.1.3": {
"hostname": "core-gw1",
"ansible_user": "",
"ansible_ssh_pass": "test123 / ena: test2",
"configsicherung": "true",
"os": "ios",
"managementpaket": ""
}
}
}}
Output should be something like this:
{"_meta": {
"hostvars": {
"10.1.1.3": {
"hostname": "core-gw1",
"ansible_user": "",
"ansible_ssh_pass": "test123",
"configsicherung": "true",
"os": "ios",
"managementpaket": ""
}
}
}}
I assume I have to add some sort of "if... then" based arguments, but haven't been able to get jq to understand me ;) Manual is a bit sketchy and I haven't been able to find any example I could get to match up with what I need to do ...
OK, as usual ... once you post a public question, you then manage to find a solution yourself ... ;)
This jq-call does what I need:
jq '. ._meta.hostvars[].ansible_ssh_pass |= sub(" .*";"" )'
I have a onelined json file that looks similar to this
{"fieldA":1, "fieldB":"foo"}
{"fieldA":2, "fieldB":"bar"}
{"fieldA":4, "fieldB":"foobar"}
...
How can I properly read this file using jq?
I tried doing:
cat myFile.json | jq [.]
but this returns something like:
[{
"fieldA":1,
"fieldB":"foo"
}]
[{
"fieldA":2,
"fieldB":"bar"
}]
[{
"fieldA":4,
"fieldB":"foobar"
}]
...
but I would like to receive this instead:
[{
"fieldA":1,
"fieldB":"foo"
},
{
"fieldA":2,
"fieldB":"bar"
},
{
"fieldA":4,
"fieldB":"foobar"
},
...]
Thanks in advance!
Are you sure you want that? What's your end goal? You can merge all of the inputs into a single array using jq -n '[inputs]' (the -n avoids the usual reading of stdin into ., allowing it all to appear on inputs), but that means that it can't produce any output, or do any further processing, until the entire input has been read, which may or may not be what you want.
For some extreme reason, I can't use jq or other cli tool. I need to extract the value of "name" from any json matching this puppet metadata.json. format.
the json might not be properly formatted and indented but will be valid. Meaning, white spaces, and line breaks, carriage backs might be inserted in eligible places.
Note that there could be "name" elements in dependencies array.
So, how to extract the value only using standard unix commands and/or shell script without installing any application like jq or other tools?
Thank you!!
{
"name": "examplecorp-mymodule",
"version": "0.0.1",
"author": "Pat",
"license": "Apache-2.0",
"summary": "A module for a thing",
"source": "https://github.com/examplecorp/examplecorp-mymodule",
"project_page": "https://forge.puppetlabs.com/examplecorp/mymodule",
"issues_url": "https://github.com/examplecorp/examplecorp-mymodule/issues",
"tags": ["things", "stuff"],
"operatingsystem_support": [
{
"operatingsystem":"RedHat",
"operatingsystemrelease":[ "5.0", "6.0" ]
},
{
"operatingsystem": "Ubuntu",
"operatingsystemrelease": [ "12.04", "10.04" ]
}
],
"dependencies": [
{ "name": "puppetlabs/stdlib", "version_requirement": ">=3.2.0 <5.0.0" },
{ "name": "puppetlabs/firewall", "version_requirement": ">= 0.0.4" }
]
}
It's ugly, awful, horrible, not structure-aware and will give you incorrect results if you have extra contents in your input file that look similar to what you're trying to find -- but...
#!/bin/bash
# ^- NOT /bin/sh; shell-native regexes are a bash extension
contents=$(<in.json)
if [[ $contents =~ '"name":'[[:space:]]*'"'([^\"]*)'"' ]]; then
echo "Found name: ${BASH_REMATCH[1]}"
fi
Now, let's talk about some of the ways this answer is broken (and using jq would be better):
It finds the first name, even if it's not one at an outer nesting layer. That is to say, if "dependencies": [ { "name": "puppetlabs/stdlib", "version_requirement": ">=3.2.0 <5.0.0" } ] comes before "name": "examplecorp-mymodule", guess which result is being found? (The easy workarounds to this would involve making assumptions about whitespace/formatting, and are thus not proof against all possible JSON expressions of the same data).
It won't unescape contents inside your name that require, well, unescaping (think about names containing symbols encoded as &foo;).
It isn't multibyte-character aware, and thus isn't guaranteed to emit output that aligns on codepoint boundaries.
If you have a name with an escaped \" subsequence... well, guess what happens there?
Etc. It's not quite as awful as trying to parse XML with regular expressions (JSON is easier!), but it's still quite a mess.
This should work for you:
jq '.name' metadata.json