Haven't used jq before but I'm wanting to build a shell script that will get a JSON response and extract just the values. To learn I thought I would try on my blog's WP API but for some reason I'm getting an error of:
jq: error (at :322): Cannot index array with string "slug"
When researching for and testing previous questions:
jq: Cannot index array with string
jq is sed for JSON
JSON array to bash variables using jq
How to use jq in a shell pipeline?
How to extract data from a JSON file
The above reading I've tried to code:
URL="http://foobar.com"
RESPONSE=$(curl -so /dev/null -w "%{http_code}" $URL)
WPAPI="/wp-json/wp/v2"
IDENTIFIER="categories"
if (("$RESPONSE" == 200)); then
curl -s {$URL$WPAPI"/"$IDENTIFIER"/"} | jq '.' >> $IDENTIFIER.json
result=$(jq .slug $IDENTIFIER.json)
echo $result
else
echo "Not returned status 200";
fi
An additional attempt changing the jq after the curl:
curl -s {$URL$WPAPI"/"$IDENTIFIER"/"} | jq '.' | $IDENTIFIER.json
result=(jq -r '.slug' $IDENTIFIER.json)
echo $result
I can modify the uncompress with the python JSON tool:
result=(curl -s {$URL$WPAPI"/"$IDENTIFIER"/"} | python -m json.tool > $IDENTIFIER.json)
I can save the JSON to a file but when I use jq I cannot get just the slug and here are my other trys:
catCalled=$(curl -s {$URL$WPAPI"/"$IDENTIFIER"/"} | python -m json.tool | ./jq -r '.slug')
echo $catCalled
My end goal is to try to use jq in a shell script and build a slug array with jq. What am I doing wrong in my jq and can I use jq on a string without creating a file?
Return from curl after uncompress per comment request:
[
{
"id": 4,
"count": 18,
"description": "",
"link": "http://foobar.com/category/foo/",
"name": "Foo",
"slug": "foo",
"taxonomy": "category",
},
{
"id": 8,
"count": 9,
"description": "",
"link": "http://foobar.com/category/bar/",
"name": "Bar",
"slug": "bar",
"taxonomy": "category",
},
{
"id": 5,
"count": 1,
"description": "",
"link": "http://foobar.com/category/mon/",
"name": "Mon",
"slug": "mon",
"taxonomy": "category",
},
{
"id": 11,
"count": 8,
"description": "",
"link": "http://foobar.com/category/fort/",
"name": "Fort",
"slug": "fort",
"taxonomy": "category",
}
]
eventually my goal is trying to get the name of the slug's into an array like:
catArray=('foo','bar','mon', 'fort')
There are 2 issues here:
slug is not a root level element in your example json. The root level element is an array. If you want to access the slug property of each element of the array, you can do so like this:
jq '.[].slug' $IDENTIFIER.json
Your example json has trailing commas after the last property of each array element. Remove the commas after "taxonomy": "category".
If I take your sample json, remove the errant commas, save it to a plain text file called test.json and run the following command:
jq '.[].slug' test.json
I get the following output:
"foo"
"bar"
"mon"
"fort"
Preprocessing
Unfortunately, the JSON-like data shown as having been produced by curl is not strictly JSON. jq does not have a "relaxed JSON" mode, so in order to use jq, you will have to preprocess the JSON-like data, e.g. using hjson (see http://hjson.org/):
$ hjson -j input.qjson > input.json
jq
With the JSON in input.json:
$ jq -c 'map(.slug)' input.json
["foo","bar","mon","fort"]
your string is not json, notice how the last member of your objects ends with a comma,
{foo:"bar",baz:9,}
this is legal in javascript, but it's illegal in json. if you are supposed to be receiving json from that endpoint, then contact the people behind it and tell them to fix the bug (it's breaking the json specs by ending objects's last member with a comma, which is illegal in json.) - until it's fixed, i guess you can patch it with a little regex, but it's a dirty quickfix, and probably not very reliable, but running it through
perl -p -0777 -e 's/\"\,\s*}/\"}/g;' makes it legal json..
Related
I'm working a bash script to extraxt specific field from json output using jq.
USERNAME=$(echo "$OUTPUT" | jq -r '.[] | .name')
Due to jq it always fails with parse error: Invalid numeric literal at line 1, column 2 error.
My restapi result has the below output.
[
{
"url": "#/systemadm/groups/uuid-d6e4e05",
"options": {},
"group_id": 313,
"owner": "abc-123-mec",
"owner_id": "ad1337884",
"id": "c258d7b330",
"name": "abc-group"
},
{
"options": {},
"id": "global%3Regmebers",
"name": "Udata-123"
},
{
"url": "#/systemadm/groups/uuid-38943000",
"options": {},
"group_id": 910,
"owner": "framework-abcc",
"owner_id": "78d4472b738bc",
"id": "38943000057a",
"name": "def-group"
},
........................
............................
......................................
So what's wrong with this jq response of code to get "name" ?
jq can only process valid JSON.
If the value of OUTPUT is literally "id": "c258d7b330","name": "abc-group", then you could enclose it in curly braces to make it valid JSON. No guarantees though; this depends on the exact format of your input.
OUTPUT='"id": "c258d7b330",
"name": "abc-group"'
USERNAME="$(printf '%s\n' "{$OUTPUT}" | jq -r '.name')"
printf '%s\n' "$USERNAME"; // abc-group
If it cannot be converted to valid JSON, maybe a simple solution using grep+cut or awk would suffice?
OUTPUT='"id": "c258d7b330",
"name": "abc-group"'
USERNAME="$(printf '%s\n' "$OUTPUT" | grep '^"name":' | cut -d'"' -f4)"
awk:
printf '%s\n' "$OUTPUT" | awk -F'"' '/^"name":/{print $4}'
Or even use jq to parse the input as array of strings and then filter for the line in which you are interested:
jq -Rr '(select(startswith("\"name\":")) / "\"")[3]'
All options are really fragile and I recommend to fix your input to be actual, valid JSON
Here is a json file named test.json for testing
{
"name": "Google",
"location": {
"street": "1600 Amphitheatre Parkway",
"city": "Mountain View",
"state": "California",
"country": "US"
},
"employees": [
{
"name": "Michael",
"division": "Engineering"
},
{
"name": "Laura",
"division": "HR"
},
{
"name": "Elise",
"division": "Marketing * test"
}
]
}
if I use the jq code to parser it like below:
cat test.json | jq -r '.employees[2].division'
it will work well and give a correct result:
Marketing * test
but I use $(), the bad thing will happen!
echo $(cat test.json | jq -r '.employees[2].division')
the result will list all file names under current folder! like:
my1.json my2.json test.json test ...
I guess it $() run asterisk * as a shell script, but a string.
so how to make asterisk (*) in json file just as a string when I am using jq ?. I am using Google cloud platform and Ubuntu 17.10
Always use double-quotes around command-substitution to avoid * to be treated literally. The * is a special character in shell that is a wildcard entry that expands to all the files available in the current working directory. You need to quote it to deprive of its special meaning (Refer GNU bash man page under Parameters section).
Also jq can process the file directly, you can avoid useless cat usage.
result="$(jq -r '.employees[2].division' < test.json)"
echo "$result"
should produce the result as expected.
Why egrep is not giving me all the matching entries?
This is my simple JSON blob:
[nukaNUKA#dev-machine csv]$ cat jsonfile.json
{"number": 303,"projectName": "giga","queueId":8881,"result":"SUCCESS"}
This is my pattern file (so that I don't scare the editor):
[nukaNUKA#dev-machine csv]$ cat egrep-pattern.txt
\"number\":.*\"projectName
\"projectName\":.*,\"queueId
\"queueId\":.*,\"result
\"result\":\".*$
This is egrep/grep command for individual searches, which works!:
[nukaNUKA#dev-machine csv]$ egrep -o "\"number\":.*\"projectName" jsonfile.json
"number": 303,"projectName
[nukaNUKA#dev-machine csv]$ egrep -o "\"projectName\":.*,\"queueId" jsonfile.json
"projectName": "giga","queueId
[nukaNUKA#dev-machine csv]$ egrep -o "\"queueId\":.*,\"result" jsonfile.json
"queueId":8881,"result
[nukaNUKA#dev-machine csv]$ egrep -o "\"result\":\".*$" jsonfile.json
"result":"SUCCESS"}
So, wth this didn't work? I don't wear glasses, yes.
[nukaNUKA#dev-machine csv]$ egrep -o "\"number\":.*\"projectName|\"projectName\":.*,\"queueId|\"queueId\":.*,\"result|\"result\":\".*$" jsonfile.json
"number": 303,"projectName
"queueId":8881,"result
[nukaNUKA#dev-machine csv]$ egrep -o -f egrep-pattern.txt jsonfile.json
"number": 303,"projectName
"queueId":8881,"result
[nukaNUKA#dev-machine csv]$
I have a complex nested JSON blob and because everything is unstructured, it seems like, I can't use JQ or JSONV or anything other Python script (as the data that I'm looking for is stored in arrays containing 1 dictionary entries (key=value) with same key names for what I'm looking for (ex: { "parameters": [ { "name": "jobname", "value": "shenzi" }, { "name": "pipelineVersion", "value": "1.2.3.4" }, ...so on..., ... ]) and the index for jobname and pipelineVersion or similar parameter names is not at the same index[X] location in every JSON entry I have.
Worst case, I can add conditional checks to see if the key at every index matches, jobname etc and then I get those fields what I looking for, but then, there are hundreds of such fields that I want to grab. I don't want to hard code those if possible.
I thought as my JSON entry is per line, I can simply write a cool patterns (ugly I know) but at least then I don't need to worry about the conditional code or just use BASH/sed/tr/cut power to get what I need but it seems like egrep -f or -o ... didn't work as shown above.
Sample JSON blob object (from one Jenkins job). There are different Jenkins build job's JSON blob entries (each having different JSON structures, parameters etc) in a single JenkinsJobsBuild collection in MongoDB.
See attached for sample JSON blob object.
{
"_id": {
"$oid": "5120349es967yhsdfs907c4f"
},
"actions": [
{
"causes": [
{
"shortDescription": "Started by an SCM change"
}
]
},
{
},
{
"oneClickDeployPossible": false,
"oneClickDeployReady": false,
"oneClickDeployValid": false
},
{
},
{
},
{
},
{
"cspec": "element * ...\/MyProject_latest_int\/LATESTnelement * ...\/MyProject_integration\/LATESTnelement \/vobs\/some_vob\/gigi \/main\/myproject_integration\/MyProject_Slot_0_maint_int\/LATESTnelement * ...\/myproject_integration\/LATESTnelement \/vobs\/some_vob \/main\/LATEST",
"latestBlsOnConfiguredStream": null,
"stream": null
},
{
},
{
"parameters": [
{
"name": "CLEARCASE_VIEWTAG",
"value": "jenkins_MyProject_latest"
},
{
"name": "BUILD_DEBUG",
"value": false
},
{
"name": "CLEAN_BUILD",
"value": true
},
{
"name": "BASEVERSION",
"value": "7.4.1"
},
{
"name": "ARTIFACTID",
"value": "lowercaseprojectname"
},
{
"name": "SYSTEM",
"value": "myprojectSystem"
},
{
"name": "LOT",
"value": "02"
},
{
"name": "PIPENUMBER",
"value": "7.4.1.303"
}
]
},
{
},
{
},
{
"parameters": [
{
"name": "DESCRIPTION_SETTER_DESCRIPTION",
"value": "lowercaseprojectname_V7.4.1.303"
}
]
},
{
},
{
},
{
},
{
}
],
"artifacts": [
],
"building": false,
"builtOn": "servername",
"changeSet": {
"items": [
{
"affectedPaths": [
"vobs\/some_vob\/myproject\/apps\/app1\/Java\/test\/src\/com\/giga\/highlevelproject\/myproject\/schedule\/validation\/SomeActivityTest.java"
],
"author": {
"absoluteUrl": "http:\/\/11.22.33.44:8080\/user\/hitj1620",
"fullName": "name1, name2 A"
},
"commitId": null,
"date": {
"$numberLong": "1489439532000"
},
"dateStr": "13\/03\/2017 21:12:12",
"elements": [
{
"action": "create version",
"editType": "edit",
"file": "vobs\/some_vob\/myproject\/apps\/app1\/Java\/test\/src\/com\/giga\/highlevelproject\/myproject\/schedule\/validation\/SomeActivityTest.java",
"operation": "checkin",
"version": "\/main\/MyProject_latest_int\/2"
}
],
"msg": "",
"timestamp": -1,
"user": "user111"
}
],
"kind": null
},
"culprits": [
{
"absoluteUrl": "http:\/\/11.22.33.44:8080\/user\/nuka1620",
"fullName": "nuka, Chuck"
}
],
"description": "lowercaseprojectname_V7.4.1.303",
"displayName": "#303",
"duration": 525758,
"estimatedDuration": 306374,
"executor": null,
"fullDisplayName": "MyProject \u00bb MyProject-build #303",
"highlevelproject_metrics_source_url": "http:\/\/11.22.33.44:8080\/job\/MyProject\/job\/MyProject-build\/303\/\/api\/json",
"id": "303",
"keepLog": false,
"number": 303,
"projectName": "MyProject-build",
"queueId": 8201,
"result": "SUCCESS",
"timeToRepair": null,
"timestamp": {
"$numberLong": "1489439650307"
},
"url": "http:\/\/11.22.33.44:8080\/job\/MyProject\/job\/MyProject-build\/303\/"
}
When the regexes are in a file, you don't have to escape double quotes; you don't have to fight to get your double quotes past the shell.
"number":.*"projectName
"projectName":.*,"queueId
"queueId":.*,"result
"result":".*$
When that's fixed, I get:
$ egrep -o -f egrep-pattern.txt jsonfile.json
"number": 303,"projectName
"queueId":8881,"result
$
The trouble now is, I think, that you've consumed the projectName with the first pattern, so the others don't get a chance to match it. Change the patterns to read up to a comma and you can get better results:
"number":[^,]*
"projectName":[^,]*
"queueId":[^,]*
"result":".*$
yields:
"number": 303
"projectName": "giga"
"queueId":8881
"result":"SUCESS"}
You could try to be more delicate, but you rapidly reach a point where a JSON-aware tool becomes more sensible. Commas in a string value would mess up the modified regexes, for example. (So, if the project name was "Giga, if not Tera", you'd have problems.)
Matching more general JSON name:value notation
As long as you're looking for simple "key":"quoted value" objects, you can use the following grep -E (aka egrep) command:
grep -Eoe '"[^"]+":"((\\(["\\/bfnrt]|u[0-9a-fA-F]{4}))|[^"])*"' data
Given the JSON-like data (in the file called data):
{"key1":"value","key2":"value2 with \"quoted\" text","key3":"value3 with \\ and \/ and \f and \uA32D embedded"}
that script produces:
"key1":"value"
"key2":"value2 with \"quoted\" text"
"key3":"value3 with \\ and \/ and \f and \uA32D embedded"
You can upgrade it to handle almost any valid JSON "key":value by using:
grep -Eoe '"[^"]+":(("((\\(["\\/bfnrt]|u[0-9a-fA-F]{4}))|[^"])*")|true|false|null|(-?(0|[1-9][0-9]*)(\.[0-9]+)?([eE][-+]?[0-9]+)?))' data
With a new data file containing:
{"key1":"value","key2":"value2 with \"quoted\" text"}
{"key3":"value3 with \\ and \/ and \f and \uA32D embedded"}
{"key4":false,"key5":true,"key6":null,"key7":7,"key8":0,"key9":0.123E-23}
{"key10":10,"key11":3.14159,"key12":0.876,"key13":-543.123}
the script produces:
"key1":"value"
"key2":"value2 with \"quoted\" text"
"key3":"value3 with \\ and \/ and \f and \uA32D embedded"
"key4":false
"key5":true
"key6":null
"key7":7
"key8":0
"key9":0.123E-23
"key10":10
"key11":3.14159
"key12":0.876
"key13":-543.123
You can follow the railroad diagrams in the outline JSON specification at http://json.org to see how I created the regex.
It could be enhanced by the judicious addition of [[:space:]]* in places where spaces are permitted but not required — before the key string, before the colon, after the colon (you could add it after the value too, but you probably don't want that).
Another simplification that I've taken is that the key doesn't allow for the various escape characters that the value string does. You could repeat that.
And, of course, this only works for 'leaf' name:value pairs; if a value is itself an object {…} or an array […], this doesn't handle the value as a whole.
However, this just goes to emphasize that it gets very messy very quickly and you would be better off using a special-purpose JSON query tool. One such tool is jq, as mentioned in a comment to the main query.
The complex JSON blob I had, was from Jenkins (i.e. Jenkins job's RestAPI data) that I had in MongoDB database.
To grab it from MongoDB, I used mongoexport command for generating (non-JsonArray or non-Pretty format) JSON blob successfully.
#/bin/bash
server=localhost
collectionFile=collections.txt
## Generate collection file contains all collections in the Jenkins database in MongoDB.
( set -x
mongo "mongoDbServer.company.com/database_Jenkins" --eval "rs.slaveOk();db.getCollectionNames()" --quiet > ${collectionFile}
)
## create collection based JSON files
for collection in $(cat ${collectionFile} | sed -e 's:,: :g')
do
mongoexport --host ${server} --db ${db} --collection "${collection}" --out ${exportDir}/${collection}.json
##mongoexport --host ${server} --db ${db} --collection "${collection}" --type=csv --fieldFile ~/mongoDB_fetch/get_these_csv_fields.txt --out ${exportDir}/${collection}.csv; ## This didn't work if you have nested fields. fieldFile file was just containing field name per line in a particular xyz.IndexNumber.yyy format.
done
Tried inbuilt mongoexport command's --type=csv with -f fields to catch topfield.0.subField, field2, field3.7.parameters.7.. nothing worked.
PS: The number after the . mark is how you define indexes if you are going to create a CSV file and using fields (mandatory) using mongoexport command.
As my JSON structure was all unstructured (Jenkins version bumps/upgrades happened in past and the data about a job was not the same structure), I tried this final sed trick (as JSON data per entry was in each individual line).
This sed command (as shown below) will give you all the keys and it's values (in the key=value format) per line at the LEAF field key=value level of almost any JSON blob / at least from the Jenkins JSON blob . Once you have this info, you can feed the output of this command to temporary file, then read all the value part (after the = mark) and create your CSV file acc. YES, you have to sort it so that your CSV file's fields are maintained in order for the header names and thus values are inserted to the right column/field. I calculated the fields names from all different collection JSON file's temporary key=value generated key names. Then, read all temporary collection files and added the values acc. into the final CSV file under respective header/field/column.
OK this is a weird solution but at least it's a solution - in one liner.
cat myJenkinsJob.json | sed "s/{}//g;s/,,*/,/g;s/},\"/\n/g;s/},{/\n/g;s/\([^\"a-zA-Z]\),\"/\1\n/g;s/:\[{/\n/g;s/\"name\":\"//g;s/\",\"value//g;s/,\"/\n/g;s/\":\"*/=/g;s/\"//g;s/[\[}\]]//g;s/[{}]//g;s/\$[a-zA-Z][a-zA-Z]*=//g"|grep "=" | sed "s/,$//"|egrep -v "=-|=$|=\[|^_class="
Tweak this acc. to your own solution for the sed part a little bit, if your JSON blob shows you funny characters that you don't want. The order of sed operations below is important. I'm also excluding any redundant variables (that I don't need at this time, for ex: JSON blob contained _class="..." values) so I'm excluding those via egrep -v after the last | pipe.
I'd like to extract the "id" key from this single line of JSON.
I believe this can be accomplished with grep, but I am not sure on the correct way.
If there is a better way that does not have dependencies, I would be interested.
Here is my example output:
{
"data": {
"name": "test",
"id": "4dCYd4W9i6gHQHvd",
"domains": ["www.test.domain.com", "test.domain.com"],
"serverid": "bbBdbbHF8PajW221",
"ssl": null,
"runtime": "php5.6",
"sysuserid": "4gm4K3lUerbSPfxz",
"datecreated": 1474597357
},
"actionid": "WXVAAHQDCSILMYTV"
}
If you have a grep that can do Perl compatible regular expressions (PCRE):
$ grep -Po '"id": *\K"[^"]*"' infile.json
"4dCYd4W9i6gHQHvd"
-P enables PCRE
-o retains nothing but the match
"id": * matches "id" and an arbitrary amount of spaces
\K throws away everything to its left ("variable size positive look-behind")
"[^"]*" matches two quotes and all the non-quotes between them
If your grep can't do that, you an use
$ grep -o '"id": *"[^"]*"' infile.json | grep -o '"[^"]*"$'
"4dCYd4W9i6gHQHvd"
This uses grep twice. The result of the first command is "id": "4dCYd4W9i6gHQHvd"; the second command removes everything but a pair of quotes and the non-quotes between them, anchored at the end of the string ($).
But, as pointed out, you shouldn't use grep for this, but a tool that can parse JSON – for example jq:
$ jq '.data.id' infile.json
"4dCYd4W9i6gHQHvd"
This is just a simple filter for the id key in the data object. To get rid of the double quotes, you can use the -r ("raw output") option:
$ jq -r '.data.id' infile.json
4dCYd4W9i6gHQHvd
jq can also neatly pretty print your JSON:
$ jq . infile.json
{
"data": {
"name": "test",
"id": "4dCYd4W9i6gHQHvd",
"domains": [
"www.test.domain.com",
"test.domain.com"
],
"serverid": "bbBdbbHF8PajW221",
"ssl": null,
"runtime": "php5.6",
"sysuserid": "4gm4K3lUerbSPfxz",
"datecreated": 1474597357
},
"actionid": "WXVAAHQDCSILMYTV"
}
Just pipe your data to jq and select by keys
"data": {
"name": "test",
"id": "4dCYd4W9i6gHQHvd",
"domains": [
"www.test.domain.com",
"test.domain.com"
],
"serverid": "bbBdbbHF8PajW221",
"ssl": null,
"runtime": "php5.6",
"sysuserid": "4gm4K3lUerbSPfxz",
"datecreated": 1474597357
},
"actionid": "WXVAAHQDCSILMYTV"
} | jq '.data.id'
# 4dCYd4W9i6gHQHvd
Tutorial Here
I found myself that the best way is to use python, as it handles JSON natively and is preinstalled on most systems these days, unlike jq:
$ python -c 'import sys, json; print(json.load(sys.stdin)["data"]["id"])' < infile.json
4dCYd4W9i6gHQHvd
No python ,jq, awk, sed just GNU grep:
#!/bin/bash
json='{"data": {"name": "test", "id": "4dCYd4W9i6gHQHvd", "domains": ["www.test.domain.com", "test.domain.com"], "serverid": "bbBdbbHF8PajW221", "ssl": null, "runtime": "php5.6", "sysuserid": "4gm4K3lUerbSPfxz", "datecreated": 1474597357}, "actionid": "WXVAAHQDCSILMYTV"}'
echo $json | grep -o '"id": "[^"]*' | grep -o '[^"]*$'
Tested & working here: https://ideone.com/EG7fv7
source: https://brianchildress.co/parse-json-using-grep
$ grep -oP '"id": *"\K[^"]*' infile.json
4dCYd4W9i6gHQHvd
Hopefully it will work for all. As this will work for me to print without quotes.
In my Unix shell script, when I execute a curl command, the result will be displayed as below which I am redirecting to file:
{"type":"Show","id":"123","title":"name","description":"Funny","channelTitle":"ifood.tv","lastUpdateTimestamp":"2014-04-20T20:34:59","numOfVideos":"15"}
But, I want this output to put in the readable JSON format like below in the file:
{"type":"Show",
"id":"123",
"title":"name",
"description":"Funny",
"channelTitle":"ifood.tv",
"lastUpdateTimestamp":"2014-04-20T20:34:59",
"numOfVideos":"15"}
How do I format the output this way?
A few solutions to choose from:
json json is a fast CLI tool for working with JSON. It is a single-file node.js script with no external deps (other than node.js itself).
$ echo '{"type":"Bar","id":"1","title":"Foo"}' | json
{
"type": "Bar",
"id": "1",
"title": "Foo"
}
Require:
# npm install -g json
json_pp: command utility available in Linux systems for JSON decoding/encoding
echo '{"type":"Bar","id":"1","title":"Foo"}' | json_pp -json_opt pretty,canonical
{
"id" : "1",
"title" : "Foo",
"type" : "Bar"
}
You may want to keep the -json_opt pretty,canonical argument for predictable ordering.
jq: lightweight and flexible command-line JSON processor. It is written in portable C, and it has zero runtime dependencies.
echo '{"type":"Bar","id":"1","title":"Foo"}' | jq '.'
{
"type": "Bar",
"id": "1",
"title": "Foo"
}
The simplest jq program is the expression ., which takes the input and produces it unchanged as output.
For additional jq options check the manual
python yq yq: Command-line YAML/XML/TOML processor - jq wrapper for YAML, XML, TOML documents
$ echo '{"type":"Bar","id":"1","title":"Foo"}' | yq
{
"type": "Bar",
"id": "1",
"title": "Foo"
}
The go version go yq doesn't work here
With xidel Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
$ echo '{"type":"Bar","id":"1","title":"Foo"}' | xidel -e '$json'
{
"type": "Bar",
"id": "1",
"title": "Foo"
}
with python:
echo '{"type":"Bar","id":"1","title":"Foo"}' | python -m json.tool
{
"id": "1",
"title": "Foo",
"type": "Bar"
}
with nodejs and bash:
echo '{"type":"Bar","id":"1","title":"Foo"}' | node -e "console.log( JSON.stringify( JSON.parse(require('fs').readFileSync(0) ), 0, 1 ))"
{
"type": "Bar",
"id": "1",
"title": "Foo"
}
I am guessing that you want to prettify the JSON output.
That could be achieved using python:
curl http://localhost:8880/test.json | python -mjson.tool > out.json
This is to add to of Gilles' Answer. There are many ways to get this done but personally I prefer something lightweight, easy to remember and universally available (e.g. come with standard LTS installations of your preferred Linux flavor or easy to install) on common *nix systems.
Here are the options in their preferred order:
Python Json.tool module
echo '{"foo": "lorem", "bar": "ipsum"}' | python -mjson.tool
pros: almost available everywhere; cons: no color coding
jq (may require one time installation)
echo '{"foo": "lorem", "bar": "ipsum"}' | jq
cons: needs to install jq; pros: color coding and versatile
json_pp (available in Ubuntu 16.04 LTS)
echo '{"foo": "lorem", "bar": "ipsum"}' | json_pp
For Ruby users
gem install jsonpretty
echo '{"foo": "lorem", "bar": "ipsum"}' | jsonpretty
You can use the json node module:
npm i -g json
then simply append | json after curl.
curl http://localhost:8880/test.json | json
python -m json.tool
Curl http://127.0.0.1:5000/people/api.json | python -m json.tool
can also help.
You can install jq and make the query like below:
curl http://localhost:8080/observations/station/221 | jq
I found json_reformat to be very handy. So I just did the following:
curl http://127.0.0.1:5000/people/api.json | json_reformat
that's it!
Motivation: You want to print prettify JSON response after curl command request.
Solution: json_pp - commandline tool that converts between some input and output formats (one of them is JSON). This program was copied from json_xs and modified. The default input format is json and the default output format is json with pretty option.
Synposis:
json_pp [-v] [-f from_format] [-t to_format] [-json_opt options_to_json1[,options_to_json2[,...]]]
Formula: <someCommand> | json_pp
Example:
Request
curl -X https://jsonplaceholder.typicode.com/todos/1 | json_pp
Response
{
"completed" : false,
"id" : 1,
"title" : "delectus aut autem",
"userId" : 1
}
Check out curljson
$ pip install curljson
$ curljson -i <the-json-api-url>
With xidel:
curl <...> | xidel - -se '$json'
xidel can probably retrieve the JSON for you as well.
A lot more features (slice, filter and map and transform structured ) apart from formatting.
https://stedolan.github.io/jq/