Related
I write this script
#!/bin/bash
# cm.sh
curl -i \
-H "Content-Type: application/json" \
-d '
{ "auth": {
"identity": {
"methods": ["password"],
"password": {
"user": {
"name": "admin",
"domain": { "id": "default" },
"password": "secret"
}
}
}
}
}' \
"http://localhost/identity/v3/auth/tokens" ; echo
echo $tokenizer1
echo $tokenizer2
But all of them(awk or sed) it's the same
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 540 100 312 100 228 312 228 0:00:01 --:--:-- 0:00:01 5142
My goal is to put the token in a variable for later.
Thanks guys in advance.
Instead of using the direct result of cURL, you could save the result in a file, and use your grep command on it.
Something like this maybe :
curl -o boulou.txt http://localhost/identity/v3/auth/tokens && cat boulou.txt | grep "X-Subject-Token" | awk '{printf $2}'
Edit, if you just want you desired output, add the --silent to the cURL command :
curl -o boulou.txt http://localhost/identity/v3/auth/tokens --silent && cat boulou.txt | grep "X-Subject-Token" | awk '{printf $2}'
Edit 2: If you want to export it, and delete your file, you could use something like this :
export OS_TOKEN=$(curl -o billy.txt hhttp://localhost/identity/v3/auth/tokens --silent && cat billy.txt | grep "X-Subject-Token" | awk '{printf $2}') && rm billy.txt
"How do I use grep/awk to extract a header in a field from curl when I pass it a JSON document when the contents is stored in a variable?" is a very tricky and unique problem indeed.
However, if you gradually mock out every part of your code to narrow it down, you'll discover that this is the much easier question you could have researched or asked instead:
How do I use grep/awk on contents from a variable?
I have a variable containing HTTP headers, and I want to extract the value of one of them. Here's an example:
variable='Foo: 1
Bar: 2
Baz: 3'
This is what I've tried to get 2 from Bar:
# Just hangs
tokenizer1=$variable `grep "Bar" | awk '{printf $2}'`
# Is empty
tokenizer2=$variable | `grep "Bar" | awk '{printf $2}'`
The answer here is to use echo to pipe the contents so that grep can read it on stdin:
tokenizer3=$(echo "$variable" | grep "Bar" | awk '{printf $2}')
This is easily applied to your example:
tokenizer3=$(echo "$token" | grep "X-Subject-Token" | awk '{printf $2}')
echo "The value is $tokenizer3"
Is there a (Unix) shell script to format JSON in human-readable form?
Basically, I want it to transform the following:
{ "foo": "lorem", "bar": "ipsum" }
... into something like this:
{
"foo": "lorem",
"bar": "ipsum"
}
With Python 2.6+ you can do:
echo '{"foo": "lorem", "bar": "ipsum"}' | python -m json.tool
or, if the JSON is in a file, you can do:
python -m json.tool my_json.json
if the JSON is from an internet source such as an API, you can use
curl http://my_url/ | python -m json.tool
For convenience in all of these cases you can make an alias:
alias prettyjson='python -m json.tool'
For even more convenience with a bit more typing to get it ready:
prettyjson_s() {
echo "$1" | python -m json.tool
}
prettyjson_f() {
python -m json.tool "$1"
}
prettyjson_w() {
curl "$1" | python -m json.tool
}
for all the above cases. You can put this in .bashrc and it will be available every time in shell. Invoke it like prettyjson_s '{"foo": "lorem", "bar": "ipsum"}'.
Note that as #pnd pointed out in the comments below, in Python 3.5+ the JSON object is no longer sorted by default. To sort, add the --sort-keys flag to the end. I.e. ... | python -m json.tool --sort-keys.
You can use: jq
It's very simple to use and it works great! It can handle very large JSON structures, including streams. You can find
their tutorials here.
Usage examples:
$ jq --color-output . file1.json file1.json | less -R
$ command_with_json_output | jq .
$ jq # stdin/"interactive" mode, just enter some JSON
$ jq <<< '{ "foo": "lorem", "bar": "ipsum" }'
{
"bar": "ipsum",
"foo": "lorem"
}
Or use jq with identity filter:
$ jq '.foo' <<< '{ "foo": "lorem", "bar": "ipsum" }'
"lorem"
I use the "space" argument of JSON.stringify to pretty-print JSON in JavaScript.
Examples:
// Indent with 4 spaces
JSON.stringify({"foo":"lorem","bar":"ipsum"}, null, 4);
// Indent with tabs
JSON.stringify({"foo":"lorem","bar":"ipsum"}, null, '\t');
From the Unix command-line with Node.js, specifying JSON on the command line:
$ node -e "console.log(JSON.stringify(JSON.parse(process.argv[1]), null, '\t'));" \
'{"foo":"lorem","bar":"ipsum"}'
Returns:
{
"foo": "lorem",
"bar": "ipsum"
}
From the Unix command-line with Node.js, specifying a filename that contains JSON, and using an indent of four spaces:
$ node -e "console.log(JSON.stringify(JSON.parse(require('fs') \
.readFileSync(process.argv[1])), null, 4));" filename.json
Using a pipe:
echo '{"foo": "lorem", "bar": "ipsum"}' | node -e \
"\
s=process.openStdin();\
d=[];\
s.on('data',function(c){\
d.push(c);\
});\
s.on('end',function(){\
console.log(JSON.stringify(JSON.parse(d.join('')),null,2));\
});\
"
I wrote a tool that has one of the best "smart whitespace" formatters available. It produces more readable and less verbose output than most of the other options here.
underscore-cli
This is what "smart whitespace" looks like:
I may be a bit biased, but it's an awesome tool for printing and manipulating JSON data from the command-line. It's super-friendly to use and has extensive command-line help/documentation. It's a Swiss Army knife that I use for 1001 different small tasks that would be surprisingly annoying to do any other way.
Latest use-case: Chrome, Dev console, Network tab, export all as HAR file, "cat site.har | underscore select '.url' --outfmt text | grep mydomain"; now I have a chronologically ordered list of all URL fetches made during the loading of my company's site.
Pretty printing is easy:
underscore -i data.json print
Same thing:
cat data.json | underscore print
Same thing, more explicit:
cat data.json | underscore print --outfmt pretty
This tool is my current passion project, so if you have any feature requests, there is a good chance I'll address them.
I usually just do:
echo '{"test":1,"test2":2}' | python -mjson.tool
And to retrieve select data (in this case, "test"'s value):
echo '{"test":1,"test2":2}' | python -c 'import sys,json;data=json.loads(sys.stdin.read()); print data["test"]'
If the JSON data is in a file:
python -mjson.tool filename.json
If you want to do it all in one go with curl on the command line using an authentication token:
curl -X GET -H "Authorization: Token wef4fwef54te4t5teerdfgghrtgdg53" http://testsite/api/ | python -mjson.tool
If you use npm and Node.js, you can do npm install -g json and then pipe the command through json. Do json -h to get all the options. It can also pull out specific fields and colorize the output with -i.
curl -s http://search.twitter.com/search.json?q=node.js | json
Thanks to J.F. Sebastian's very helpful pointers, here's a slightly enhanced script I've come up with:
#!/usr/bin/python
"""
Convert JSON data to human-readable form.
Usage:
prettyJSON.py inputFile [outputFile]
"""
import sys
import simplejson as json
def main(args):
try:
if args[1] == '-':
inputFile = sys.stdin
else:
inputFile = open(args[1])
input = json.load(inputFile)
inputFile.close()
except IndexError:
usage()
return False
if len(args) < 3:
print json.dumps(input, sort_keys = False, indent = 4)
else:
outputFile = open(args[2], "w")
json.dump(input, outputFile, sort_keys = False, indent = 4)
outputFile.close()
return True
def usage():
print __doc__
if __name__ == "__main__":
sys.exit(not main(sys.argv))
It is not too simple with a native way with the jq tools.
For example:
cat xxx | jq .
a simple bash script for pretty json printing
json_pretty.sh
#/bin/bash
grep -Eo '"[^"]*" *(: *([0-9]*|"[^"]*")[^{}\["]*|,)?|[^"\]\[\}\{]*|\{|\},?|\[|\],?|[0-9 ]*,?' | awk '{if ($0 ~ /^[}\]]/ ) offset-=4; printf "%*c%s\n", offset, " ", $0; if ($0 ~ /^[{\[]/) offset+=4}'
Example:
cat file.json | json_pretty.sh
With Perl, use the CPAN module JSON::XS. It installs a command line tool json_xs.
Validate:
json_xs -t null < myfile.json
Prettify the JSON file src.json to pretty.json:
< src.json json_xs > pretty.json
If you don't have json_xs, try json_pp . "pp" is for "pure perl" – the tool is implemented in Perl only, without a binding to an external C library (which is what XS stands for, Perl's "Extension System").
On *nix, reading from stdin and writing to stdout works better:
#!/usr/bin/env python
"""
Convert JSON data to human-readable form.
(Reads from stdin and writes to stdout)
"""
import sys
try:
import simplejson as json
except:
import json
print json.dumps(json.loads(sys.stdin.read()), indent=4)
sys.exit(0)
Put this in a file (I named mine "prettyJSON" after AnC's answer) in your PATH and chmod +x it, and you're good to go.
That's how I do it:
curl yourUri | json_pp
It shortens the code and gets the job done.
The JSON Ruby Gem is bundled with a shell script to prettify JSON:
sudo gem install json
echo '{ "foo": "bar" }' | prettify_json.rb
Script download: gist.github.com/3738968
$ echo '{ "foo": "lorem", "bar": "ipsum" }' \
> | python -c'import fileinput, json;
> print(json.dumps(json.loads("".join(fileinput.input())),
> sort_keys=True, indent=4))'
{
"bar": "ipsum",
"foo": "lorem"
}
NOTE: It is not the way to do it.
The same in Perl:
$ cat json.txt \
> | perl -0007 -MJSON -nE'say to_json(from_json($_, {allow_nonref=>1}),
> {pretty=>1})'
{
"bar" : "ipsum",
"foo" : "lorem"
}
Note 2:
If you run
echo '{ "Düsseldorf": "lorem", "bar": "ipsum" }' \
| python -c'import fileinput, json;
print(json.dumps(json.loads("".join(fileinput.input())),
sort_keys=True, indent=4))'
the nicely readable word becomes \u encoded
{
"D\u00fcsseldorf": "lorem",
"bar": "ipsum"
}
If the remainder of your pipeline will gracefully handle unicode and you'd like your JSON to also be human-friendly, simply use ensure_ascii=False
echo '{ "Düsseldorf": "lorem", "bar": "ipsum" }' \
| python -c'import fileinput, json;
print json.dumps(json.loads("".join(fileinput.input())),
sort_keys=True, indent=4, ensure_ascii=False)'
and you'll get:
{
"Düsseldorf": "lorem",
"bar": "ipsum"
}
UPDATE I'm using jq now as suggested in another answer. It's extremely powerful at filtering JSON, but, at its most basic, also an awesome way to pretty print JSON for viewing.
jsonpp is a very nice command line JSON pretty printer.
From the README:
Pretty print web service responses like so:
curl -s -L http://<!---->t.co/tYTq5Pu | jsonpp
and make beautiful the files running around on your disk:
jsonpp data/long_malformed.json
If you're on Mac OS X, you can brew install jsonpp. If not, you can simply copy the binary to somewhere in your $PATH.
Try pjson. It has colors!
Install it with pip:
⚡ pip install pjson
And then pipe any JSON content to pjson.
Or, with Ruby:
echo '{ "foo": "lorem", "bar": "ipsum" }' | ruby -r json -e 'jj JSON.parse gets'
You can use this simple command to achieve the result:
echo "{ \"foo\": \"lorem\", \"bar\": \"ipsum\" }"|python -m json.tool
I use jshon to do exactly what you're describing. Just run:
echo $COMPACTED_JSON_TEXT | jshon
You can also pass arguments to transform the JSON data.
Check out Jazor. It's a simple command line JSON parser written in Ruby.
gem install jazor
jazor --help
JSONLint has an open-source implementation on GitHub that can be used on the command line or included in a Node.js project.
npm install jsonlint -g
and then
jsonlint -p myfile.json
or
curl -s "http://api.twitter.com/1/users/show/user.json" | jsonlint | less
Simply pipe the output to jq ..
Example:
twurl -H ads-api.twitter.com '.......' | jq .
You can simply use standard tools like jq or json_pp.
echo '{ "foo": "lorem", "bar": "ipsum" }' | json_pp
or
echo '{ "foo": "lorem", "bar": "ipsum" }' | jq
will both prettify output like the following (jq even more colorful):
{
"foo": "lorem",
"bar": "ipsum"
}
The huge advantage of jq is that it can do A LOT more if you'd like to parse and process the json.
With Perl, if you install JSON::PP from CPAN you'll get the json_pp command. Stealing the example from B Bycroft you get:
[pdurbin#beamish ~]$ echo '{"foo": "lorem", "bar": "ipsum"}' | json_pp
{
"bar" : "ipsum",
"foo" : "lorem"
}
It's worth mentioning that json_pp comes pre-installed with Ubuntu 12.04 (at least) and Debian in /usr/bin/json_pp
Pygmentize
I combine Python's json.tool with pygmentize:
echo '{"foo": "bar"}' | python -m json.tool | pygmentize -g
There are some alternatives to pygmentize which are listed in my this answer.
Here is a live demo:
You only need to use jq
If jq is not installed then you need to install jq first:
sudo apt-get update
sudo apt-get install jq
After installing jq then only need to use jq:
echo '{ "foo": "lorem", "bar": "ipsum" }' | jq
Output looks like
{
"foo": "lorem",
"bar": "ipsum"
}
I recommend using the json_xs command line utility which is included in the JSON::XS perl module. JSON::XS is a Perl module for serializing/deserializing JSON, on a Debian or Ubuntu machine you can install it like this:
sudo apt-get install libjson-xs-perl
It is obviously also available on CPAN.
To use it to format JSON obtained from a URL you can use curl or wget like this:
$ curl -s http://page.that.serves.json.com/json/ | json_xs
or this:
$ wget -q -O - http://page.that.serves.json.com/json/ | json_xs
and to format JSON contained in a file you can do this:
$ json_xs < file-full-of.json
To reformat as YAML, which some people consider to be more humanly-readable than JSON:
$ json_xs -t yaml < file-full-of.json
jj is super-fast, can handle ginormous JSON documents economically, does not mess with valid JSON numbers, and is easy to use, e.g.
jj -p # for reading from STDIN
or
jj -p -i input.json
It is (2018) still quite new so maybe it won’t handle invalid JSON the way you expect, but it is easy to install on major platforms.
bat is a cat clone with syntax highlighting:
Example:
echo '{"bignum":1e1000}' | bat -p -l json
-p will output without headers, and -l will explicitly specify the language.
It has colouring and formatting for JSON and does not have the problems noted in this comment: How can I pretty-print JSON in a shell script?
Install yajl-tools with the command below:
sudo apt-get install yajl-tools
then,
echo '{"foo": "lorem", "bar": "ipsum"}' | json_reformat
Can someone suggest how I can get the value 45 after parsing an example json text as shown below :
....
"test": 12
"job": 45
"task": 11
.....
Please note that I am aware of tools like jq and others but this requires it to be installed.
I am hoping to get this executed using grep, awk or sed command.
awk -F'[[:space:]]*:[[:space:]]*' '/^[[:space:]]*"job"/{ print $2 }'
sed -n 's/^[[:space:]]*"job"[[:space:]]*:[[:space:]]*//p'
You can use grep -oP (PCRE):
grep -oP '"job"\s*:\s*\K\d+' file
45
\K is used for reseting the previously matched data.
Using awk, if you just want to print it:
awk -F ':[ \t]*' '/^.*"job"/ {print $2}' filename
Above command matches any line that has "job" at the beginning of a line, and then prints the second column of that line. awk option -F is used to set the column separator as : followed by any number of spaces or tabs.
If you want to store this value in bash variable job_val:
job_val=$(awk -F ':[ \t]*' '/^.*"job"/ {print $2}' filename)
Use specialized tools like jq for the task :
Had your file looked like
[
{
"test": 12,
"job": 45,
"task": 11
}
]
below stuff would get you home
jq ".[].job" file
Had your file looked like
{
"stuff" :{
.
.
"test": 12,
"job": 45,
"task": 11
.
.
}
}
below
jq ".stuff.job" file
would get you home.
I'm curling an endpoint:
#!/bin/bash
instance_info=$(curl -sk https://internal.admin.com/app/instance)
which gives a json response:
{
"basePath": "/install",
"metadata": {
"deployed_artifact": "app01",
"docker": "True",
"http_port": "7471",
"url": "www.google.com"
},
"name": "app-01",
"server": "webserver1"
}
I'm trying to avoid curling more than once to get the variables I need from the json using JQ.
Using bash I'd really appreciate if someone can show me how to store the response as another var and then use this to variablize name: server: url: http_port:
The following seems to run the curl twice:
#!/bin/bash
instance_info=$(curl -sk https://internal.admin.com/app/instance)
server_name=$(echo instance_info | /usr/bin/jq --raw-output '.server')
url=$(echo instance_info | /usr/bin/jq --raw-output '.url')
You are calling curl once and this suffices. Then you have the content in a variable, so you can access it without calling curl again.
Regarding your code, your approach is fine but you are missing $ when you are echoing the variable:
server_name=$(echo $instance_info | /usr/bin/jq --raw-output '.server')
# ^
See a sample. Here I hardcode the JSON:
your_json='
{
"basePath": "/install",
"metadata": {
"deployed_artifact": "app01",
"docker": "True",
"http_port": "7471",
"url": "www.google.com"
},
"name": "app-01",
"server": "webserver1"
}'
for the server:
$ echo "$your_json" | jq --raw-output '.server'
webserver1
For the url you need to indicate the block where it lies on. That is, metadata:
$ echo "$your_json" | jq --raw-output '.metadata.url'
www.google.com
To store into a variable, say:
your_field=$(echo "$your_json" | jq --raw-output 'XXXX')
# ^^^^
Here is a script which demonstrates how to use jq's #sh formatting directive along with bash eval to set bash variables using the output of a filter. In this case we hardcode the json which presumably would have come from curl.
#!/bin/bash
instance_info='
{
"basePath": "/install",
"metadata": {
"deployed_artifact": "app01",
"docker": "True",
"http_port": "7471",
"url": "www.google.com"
},
"name": "app-01",
"server": "webserver1"
}'
eval "$(jq -M -r '#sh "server_name=\(.server) url=\(.metadata.url)"' <<< "$instance_info")"
echo $server_name
echo $url
When run this produces the output
webserver1
www.google.com
The Extract data and set shell variables section of the JQ Cookbook has more examples of #sh.
sed -rn 's/[ ]*"([^"]*)"[^"]*("[^"]*").*/\1=\2/gp'' file.json
O/P:
basePath="/install"
deployed_artifact="app01"
docker="True"
http_port="7471"
url="www.google.com"
name="app-01"
server="webserver1"
eval this output you can get the variable
eval $(sed -rn 's/[ ]*"([^"]*)"[^"]*("[^"]*").*/\1=\2/gp' file.json )
echo $name $server
O/p:
app-01 webserver1
Read the values with one jq pass then there's no need to store the json:
read name server url http_port < <(
curl -sk https://internal.admin.com/app/instance | \
jq --raw-output '[.name, .server, .metadata.url, .metadata.http_port] | join(" ")'
)
echo -e "\$name: $name\n\$server: $server\n\$url: $url\n\$http_port $http_port"
$name: app-01
$server: webserver1
$url: www.google.com
$http_port 7471
Is there a (Unix) shell script to format JSON in human-readable form?
Basically, I want it to transform the following:
{ "foo": "lorem", "bar": "ipsum" }
... into something like this:
{
"foo": "lorem",
"bar": "ipsum"
}
With Python 2.6+ you can do:
echo '{"foo": "lorem", "bar": "ipsum"}' | python -m json.tool
or, if the JSON is in a file, you can do:
python -m json.tool my_json.json
if the JSON is from an internet source such as an API, you can use
curl http://my_url/ | python -m json.tool
For convenience in all of these cases you can make an alias:
alias prettyjson='python -m json.tool'
For even more convenience with a bit more typing to get it ready:
prettyjson_s() {
echo "$1" | python -m json.tool
}
prettyjson_f() {
python -m json.tool "$1"
}
prettyjson_w() {
curl "$1" | python -m json.tool
}
for all the above cases. You can put this in .bashrc and it will be available every time in shell. Invoke it like prettyjson_s '{"foo": "lorem", "bar": "ipsum"}'.
Note that as #pnd pointed out in the comments below, in Python 3.5+ the JSON object is no longer sorted by default. To sort, add the --sort-keys flag to the end. I.e. ... | python -m json.tool --sort-keys.
You can use: jq
It's very simple to use and it works great! It can handle very large JSON structures, including streams. You can find
their tutorials here.
Usage examples:
$ jq --color-output . file1.json file1.json | less -R
$ command_with_json_output | jq .
$ jq # stdin/"interactive" mode, just enter some JSON
$ jq <<< '{ "foo": "lorem", "bar": "ipsum" }'
{
"bar": "ipsum",
"foo": "lorem"
}
Or use jq with identity filter:
$ jq '.foo' <<< '{ "foo": "lorem", "bar": "ipsum" }'
"lorem"
I use the "space" argument of JSON.stringify to pretty-print JSON in JavaScript.
Examples:
// Indent with 4 spaces
JSON.stringify({"foo":"lorem","bar":"ipsum"}, null, 4);
// Indent with tabs
JSON.stringify({"foo":"lorem","bar":"ipsum"}, null, '\t');
From the Unix command-line with Node.js, specifying JSON on the command line:
$ node -e "console.log(JSON.stringify(JSON.parse(process.argv[1]), null, '\t'));" \
'{"foo":"lorem","bar":"ipsum"}'
Returns:
{
"foo": "lorem",
"bar": "ipsum"
}
From the Unix command-line with Node.js, specifying a filename that contains JSON, and using an indent of four spaces:
$ node -e "console.log(JSON.stringify(JSON.parse(require('fs') \
.readFileSync(process.argv[1])), null, 4));" filename.json
Using a pipe:
echo '{"foo": "lorem", "bar": "ipsum"}' | node -e \
"\
s=process.openStdin();\
d=[];\
s.on('data',function(c){\
d.push(c);\
});\
s.on('end',function(){\
console.log(JSON.stringify(JSON.parse(d.join('')),null,2));\
});\
"
I wrote a tool that has one of the best "smart whitespace" formatters available. It produces more readable and less verbose output than most of the other options here.
underscore-cli
This is what "smart whitespace" looks like:
I may be a bit biased, but it's an awesome tool for printing and manipulating JSON data from the command-line. It's super-friendly to use and has extensive command-line help/documentation. It's a Swiss Army knife that I use for 1001 different small tasks that would be surprisingly annoying to do any other way.
Latest use-case: Chrome, Dev console, Network tab, export all as HAR file, "cat site.har | underscore select '.url' --outfmt text | grep mydomain"; now I have a chronologically ordered list of all URL fetches made during the loading of my company's site.
Pretty printing is easy:
underscore -i data.json print
Same thing:
cat data.json | underscore print
Same thing, more explicit:
cat data.json | underscore print --outfmt pretty
This tool is my current passion project, so if you have any feature requests, there is a good chance I'll address them.
I usually just do:
echo '{"test":1,"test2":2}' | python -mjson.tool
And to retrieve select data (in this case, "test"'s value):
echo '{"test":1,"test2":2}' | python -c 'import sys,json;data=json.loads(sys.stdin.read()); print data["test"]'
If the JSON data is in a file:
python -mjson.tool filename.json
If you want to do it all in one go with curl on the command line using an authentication token:
curl -X GET -H "Authorization: Token wef4fwef54te4t5teerdfgghrtgdg53" http://testsite/api/ | python -mjson.tool
If you use npm and Node.js, you can do npm install -g json and then pipe the command through json. Do json -h to get all the options. It can also pull out specific fields and colorize the output with -i.
curl -s http://search.twitter.com/search.json?q=node.js | json
Thanks to J.F. Sebastian's very helpful pointers, here's a slightly enhanced script I've come up with:
#!/usr/bin/python
"""
Convert JSON data to human-readable form.
Usage:
prettyJSON.py inputFile [outputFile]
"""
import sys
import simplejson as json
def main(args):
try:
if args[1] == '-':
inputFile = sys.stdin
else:
inputFile = open(args[1])
input = json.load(inputFile)
inputFile.close()
except IndexError:
usage()
return False
if len(args) < 3:
print json.dumps(input, sort_keys = False, indent = 4)
else:
outputFile = open(args[2], "w")
json.dump(input, outputFile, sort_keys = False, indent = 4)
outputFile.close()
return True
def usage():
print __doc__
if __name__ == "__main__":
sys.exit(not main(sys.argv))
It is not too simple with a native way with the jq tools.
For example:
cat xxx | jq .
a simple bash script for pretty json printing
json_pretty.sh
#/bin/bash
grep -Eo '"[^"]*" *(: *([0-9]*|"[^"]*")[^{}\["]*|,)?|[^"\]\[\}\{]*|\{|\},?|\[|\],?|[0-9 ]*,?' | awk '{if ($0 ~ /^[}\]]/ ) offset-=4; printf "%*c%s\n", offset, " ", $0; if ($0 ~ /^[{\[]/) offset+=4}'
Example:
cat file.json | json_pretty.sh
With Perl, use the CPAN module JSON::XS. It installs a command line tool json_xs.
Validate:
json_xs -t null < myfile.json
Prettify the JSON file src.json to pretty.json:
< src.json json_xs > pretty.json
If you don't have json_xs, try json_pp . "pp" is for "pure perl" – the tool is implemented in Perl only, without a binding to an external C library (which is what XS stands for, Perl's "Extension System").
On *nix, reading from stdin and writing to stdout works better:
#!/usr/bin/env python
"""
Convert JSON data to human-readable form.
(Reads from stdin and writes to stdout)
"""
import sys
try:
import simplejson as json
except:
import json
print json.dumps(json.loads(sys.stdin.read()), indent=4)
sys.exit(0)
Put this in a file (I named mine "prettyJSON" after AnC's answer) in your PATH and chmod +x it, and you're good to go.
That's how I do it:
curl yourUri | json_pp
It shortens the code and gets the job done.
The JSON Ruby Gem is bundled with a shell script to prettify JSON:
sudo gem install json
echo '{ "foo": "bar" }' | prettify_json.rb
Script download: gist.github.com/3738968
$ echo '{ "foo": "lorem", "bar": "ipsum" }' \
> | python -c'import fileinput, json;
> print(json.dumps(json.loads("".join(fileinput.input())),
> sort_keys=True, indent=4))'
{
"bar": "ipsum",
"foo": "lorem"
}
NOTE: It is not the way to do it.
The same in Perl:
$ cat json.txt \
> | perl -0007 -MJSON -nE'say to_json(from_json($_, {allow_nonref=>1}),
> {pretty=>1})'
{
"bar" : "ipsum",
"foo" : "lorem"
}
Note 2:
If you run
echo '{ "Düsseldorf": "lorem", "bar": "ipsum" }' \
| python -c'import fileinput, json;
print(json.dumps(json.loads("".join(fileinput.input())),
sort_keys=True, indent=4))'
the nicely readable word becomes \u encoded
{
"D\u00fcsseldorf": "lorem",
"bar": "ipsum"
}
If the remainder of your pipeline will gracefully handle unicode and you'd like your JSON to also be human-friendly, simply use ensure_ascii=False
echo '{ "Düsseldorf": "lorem", "bar": "ipsum" }' \
| python -c'import fileinput, json;
print json.dumps(json.loads("".join(fileinput.input())),
sort_keys=True, indent=4, ensure_ascii=False)'
and you'll get:
{
"Düsseldorf": "lorem",
"bar": "ipsum"
}
UPDATE I'm using jq now as suggested in another answer. It's extremely powerful at filtering JSON, but, at its most basic, also an awesome way to pretty print JSON for viewing.
jsonpp is a very nice command line JSON pretty printer.
From the README:
Pretty print web service responses like so:
curl -s -L http://<!---->t.co/tYTq5Pu | jsonpp
and make beautiful the files running around on your disk:
jsonpp data/long_malformed.json
If you're on Mac OS X, you can brew install jsonpp. If not, you can simply copy the binary to somewhere in your $PATH.
Try pjson. It has colors!
Install it with pip:
⚡ pip install pjson
And then pipe any JSON content to pjson.
Or, with Ruby:
echo '{ "foo": "lorem", "bar": "ipsum" }' | ruby -r json -e 'jj JSON.parse gets'
You can use this simple command to achieve the result:
echo "{ \"foo\": \"lorem\", \"bar\": \"ipsum\" }"|python -m json.tool
I use jshon to do exactly what you're describing. Just run:
echo $COMPACTED_JSON_TEXT | jshon
You can also pass arguments to transform the JSON data.
Check out Jazor. It's a simple command line JSON parser written in Ruby.
gem install jazor
jazor --help
JSONLint has an open-source implementation on GitHub that can be used on the command line or included in a Node.js project.
npm install jsonlint -g
and then
jsonlint -p myfile.json
or
curl -s "http://api.twitter.com/1/users/show/user.json" | jsonlint | less
Simply pipe the output to jq ..
Example:
twurl -H ads-api.twitter.com '.......' | jq .
You can simply use standard tools like jq or json_pp.
echo '{ "foo": "lorem", "bar": "ipsum" }' | json_pp
or
echo '{ "foo": "lorem", "bar": "ipsum" }' | jq
will both prettify output like the following (jq even more colorful):
{
"foo": "lorem",
"bar": "ipsum"
}
The huge advantage of jq is that it can do A LOT more if you'd like to parse and process the json.
With Perl, if you install JSON::PP from CPAN you'll get the json_pp command. Stealing the example from B Bycroft you get:
[pdurbin#beamish ~]$ echo '{"foo": "lorem", "bar": "ipsum"}' | json_pp
{
"bar" : "ipsum",
"foo" : "lorem"
}
It's worth mentioning that json_pp comes pre-installed with Ubuntu 12.04 (at least) and Debian in /usr/bin/json_pp
Pygmentize
I combine Python's json.tool with pygmentize:
echo '{"foo": "bar"}' | python -m json.tool | pygmentize -g
There are some alternatives to pygmentize which are listed in my this answer.
Here is a live demo:
You only need to use jq
If jq is not installed then you need to install jq first:
sudo apt-get update
sudo apt-get install jq
After installing jq then only need to use jq:
echo '{ "foo": "lorem", "bar": "ipsum" }' | jq
Output looks like
{
"foo": "lorem",
"bar": "ipsum"
}
I recommend using the json_xs command line utility which is included in the JSON::XS perl module. JSON::XS is a Perl module for serializing/deserializing JSON, on a Debian or Ubuntu machine you can install it like this:
sudo apt-get install libjson-xs-perl
It is obviously also available on CPAN.
To use it to format JSON obtained from a URL you can use curl or wget like this:
$ curl -s http://page.that.serves.json.com/json/ | json_xs
or this:
$ wget -q -O - http://page.that.serves.json.com/json/ | json_xs
and to format JSON contained in a file you can do this:
$ json_xs < file-full-of.json
To reformat as YAML, which some people consider to be more humanly-readable than JSON:
$ json_xs -t yaml < file-full-of.json
jj is super-fast, can handle ginormous JSON documents economically, does not mess with valid JSON numbers, and is easy to use, e.g.
jj -p # for reading from STDIN
or
jj -p -i input.json
It is (2018) still quite new so maybe it won’t handle invalid JSON the way you expect, but it is easy to install on major platforms.
bat is a cat clone with syntax highlighting:
Example:
echo '{"bignum":1e1000}' | bat -p -l json
-p will output without headers, and -l will explicitly specify the language.
It has colouring and formatting for JSON and does not have the problems noted in this comment: How can I pretty-print JSON in a shell script?
Install yajl-tools with the command below:
sudo apt-get install yajl-tools
then,
echo '{"foo": "lorem", "bar": "ipsum"}' | json_reformat