Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 3 years ago.
Improve this question
I'm trying to convert a large JSON file to a CSV format. (I've just learned how to use jq so I'm still a beginner).
I've successfully managed to convert most of the data, however, I'm stuck at an array. Every JSON object in the file is supposed to be converted to a single CSV row, and I can't get that to work.
I've been trying to help myself with an existing answer:
Convert/Export JSON to CSV
But the problem is that this method writes a row for each item in the array, needlessly repeating information.
I am using the same type of command as the above answer, the only difference being the name of the columns, but the array blocks...
For instance, I could have a JSON file resembling:
{resources:[
{"id":"001","name"="Robert","items":[
{"label":"00A","name":"Pen"},
{"label":"00B","name":"Paper"}],
{"id":"002","name"="Bruce","items":[
{"label":"00A","name":"Pen"},
{"label":"00B","name":"Computer"},
{"label":"00C","name":"Headphones"}]
]
}
That I would like to become:
001,Robert,Pen,Paper,
002,Bruce,Pen,Computer,Headphones
I only need the name columns of the array
For the moment, the result is:
001,Robert,Pen
001,Robert,Paper
002,Bruce,Pen,
002,Bruce,Computer
002,Bruce,Headphones
The problem is the actual array is about 30 items long for each JSON object, making it impossible to use this way.
$ jq -r '.resources[] | [.id,.name,.items[].name] | #csv' < /tmp/b.json
"001","Robert","Pen","Paper"
"002","Bruce","Pen","Computer","Headphones"
You can use awk to fix up your current output file.
awk -F, '{a[$1","$2]=a[$1","$2]","$3}; END{for (v in a) print v a[v]}' in.txt | sort >out.txt
Input:
001,Robert,Pen
001,Robert,Paper
002,Bruce,Pen,
002,Bruce,Computer
002,Bruce,Headphones
Output:
001,Robert,Pen,Paper
002,Bruce,Pen,Computer,Headphones
Related
This question already has answers here:
change json file by bash script
(6 answers)
Closed 6 months ago.
I have a file called usr.json, and this file contains a JSON object. The object looks as following:
{
"users": {
"accessRights":"FFFFF",
"name": "admin"
}
}
Now I want to replace the string for accessRights with a new value which is stored in a variable called newUsrCfg.
Since I'm working with a remote device I need to ssh to it, and for that I have a script.
However, I call all this in Python code:
def test_tc():
newUsrCfg = "F7FFF"
cmd = f'sed -ie \'s/\"accessRights\":.*/\"accessRights\": \"{newUsrCfg}\"/g\' "/etc/hgp/runtime/user/user1.json"'
subprocess.call(["bash", pathToSshCommand, cmd])
The commands need to be either called with single quotes or double ones. And when opening the file the output of this is:
{
"users":{
"accessRights": "f7fff"
I tried many other sed commands with different formats, but nothing worked.
If jq is available to you, you can
jq '.users.accessRights = "f7fff"' usr.json
As also commented below, manipulating json with text replacement is likely to break if your input does not exactly meet your expectations (for instance though differences in whitespace). If you have a tool like jq which understands json, that is always the preferable option.
However, in case you still want to do it with sed you could adjust your regex to be less inclusive. Since you want to replace 5 hex values, something along the following lines should work (I hope I escaped everything correctly, can't test since I'm on the way).
cmd = 'sed -ie \'s/\"accessRights\":\s*\"[0-9,a-f,A-F]\{5\}\"/\"accessRights\": \"{newUsrCfg}\"/g\' "/etc/hgp/runtime/user/user1.json"'
See regex101.
A third option would to both load the file via ssh and manipulate the json in Python, or do it directly in Python on the target system, if Python is available there as well.
Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 1 year ago.
Improve this question
To change the md5 checksum of a json file, I am using the following filter in my httpd.conf:
ExtFilterDefine jsonfilter mode=output intype=application/json cmd="/usr/bin/perl -pe 'END { unless (-f q{/tmp/md5_filter.tmp}) { print qq(\\n\,\"STRING\"\: \") . time() . qq(\x0D\"\\n) }'"
But after the filter run, I receive an error, probally because the new string / timestamp get added after the last bracket } and leads to an unvalid json format:
SyntaxError: JSON.parse: unexpected non-whitespace character after JSON data at line 224 column 2 of the JSON data
Does someone know how to get this run correctly? Just to be sure: I just want to change the output of the file, before it gets response to the client browser. I don't want to change the original file itself. This should be untouched.
probally because the new string / timestamp get added after the last bracket } and leads to an unvalid json format
I don't know any details of the environment that you're working in, but I suspect that your suspicion is correct here.
You could try to fix your string-manipulating solution (perhaps by working out when you're processing the final line of the input and then printing your addition before printing the closing brace. But the best solution is to gather the input into a single string and then process it using a JSON parser.
Something like this (untested):
perl -MJSON -ne '$json .= $_; END { my $data = decode_json($json); $data->{STRING} = time; print encode_json($data);'
Update: To explain the various parts of my code.
-MJSON - loads the JSON module
-ne - n iterates over the input (putting each line, in turn, into $_); e executes code from the command line
$json .= $_ - this is run for every line of the input. It just takes each line and gathers them all together in a variable called $json
END { ... } - this block of code is only run once - after the input has all been read
my $data = decode_json($json) - takes the JSON text string (now stored in $json) and decodes it into a Perl data structure. That data structure is stored in $data
$data->{STRING} = time - adds another key/value pair to the data structure
print encode_json($data) - encodes the altered data structure back into a JSON string and prints that string to STDOUT
I have the following JSON structure:
{
"host1": "$PROJECT1",
"host2": "$PROJECT2",
"host3" : "xyz",
"host4" : "$PROJECT4"
}
And the following environment variables in the shell:
PROJECT1="randomtext1"
PROJECT2="randomtext2"
PROJECT4="randomtext3"
I want to check the values for each key, if they have a "$" character in them, replace them with their respective environment variable(which is already present in the shell) so that my JSON template is rendered with the correct environment variables.
I can use the --args option of jq but there are quite a lot of variables in my actual JSON template that I want to render.
I have been trying the following:
jq 'with_entries(.values as v | env.$v)
Basically making each value as a variable, then updating its value with the variable from the env object but seems like I am missing out on some understanding. Is there a straightforward way of doing this?
EDIT
Thanks to the answers on this question, I was able to achieve my larger goal for a part of which this question was asked
iterating over each value in an object,
checking its value,
if it's a string and starts with the character "$"
use the value to update it with an environment variable of the same name .
if it's an array
use the value to retrieve an environment variable of the same name
split the string with "," as delimiter, which returns an array of strings
Update the value with the array of strings
jq 'with_entries(.value |= (if (type=="array") then (env[.[0][1:]] | split(",")) elif (type=="string" and startswith("$")) then (env[.[1:]]) else . end))'
You need to export the Bash variables to be seen by jq:
export PROJECT1="randomtext1"
export PROJECT2="randomtext2"
export PROJECT4="randomtext3"
Then you can go with:
jq -n 'with_entries((.value | select(startswith("$"))) |= env[.[1:]])'
and get:
{
"host1": "randomtext1",
"host2": "randomtext2",
"host3": "xyz",
"host4": "randomtext3"
}
Exporting a large number of shell variables might not be such a good idea and does not address the problem of array-valued variables. It might therefore be a good idea to think along the lines of printing the variable=value details to a file, and then combining that file with the template. It’s easy to do and examples on the internet abound and probably here on SO as well. You could, for example, use printf like so:
printf "%s\t" ${BASH_VERSINFO[#]}
3 2 57 1
You might also find declare -p helpful.
See also https://github.com/stedolan/jq/wiki/Cookbook#arbitrary-strings-as-template-variables
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 2 years ago.
Improve this question
I downloaded a 203775480 bytes (~200 MiB, exact size is important for a later error) JSON file which has all entries all on one line. Needless to say, my text editor (ViM) cannot efficiently navigate in it and I'm not able to understand anything from it. I'd like to prettify it. I tried to use cat file.json | jq '.', jq '.' file.json, cat file.json | python -m json.tool but none worked. The former two commands print nothing on stdout while the latter says Expecting object: line 1 column 203775480 (char 203775479).
I guess it's broken somewhere near the end, but of course I cannot understand where as I cannot even navigate it.
Have you got some other idea for prettifying it? (I've also tried gg=G in ViM: it did not work).
I found that the file was indeed broken: I accidentally noticed a ']' at the beginning of the file so I struggled to go to the end of the file and added a ']' at the end (it took me maybe 5 minutes).
Then I've rerun cat file.json | python -m json.tool and it worked like a charm.
This question already has an answer here:
jq special characters in nested keys
(1 answer)
Closed 3 years ago.
I am trying to use the jq command line JSON processor https://shapeshed.com/jq-json/ (which works great) to process a JSON file that seems to have been made using some poor choices.
Normally your id and value in the JSON file would not contain any periods such as:
{"id":"d9s7g9df7sd9","name":"Tacos"}
To get Tacos from the file you would do the following in bash:
echo $json | jq -r '.name'
This will give you Tacos (There may be some extra code missing from that example but you get the point.)
I have a JSON file that looks like this:
{"stat.blah":123,"stat.taco":495,"stat.yum... etc.
Notice how they decided to use a period in the identifying field associated with the value? This makes using jq very difficult because it associates the period as a separator to dig down into child values in the JSON. Sure, I could first load my file, replace all "." with "_" and that would fix the problem, but this seems like a really dumb and hackish solution. I have no way to change how the initial JSON file is generated. I just have to deal with it. Is there a way in bash I can do some special escape to make it ignore the period?
Thanks
Use generic object index syntax, e.g:
.["stat.taco"]
If you use the generic object syntax, e.g. .["stat.taco"], then chaining is done either using pipes as usual, or without the dot, e.g.
.["stat.taco"]["inner.key"]
If your jq is sufficiently recent, then you can use the chained-dot notation by quoting the keys with special characters, e.g.
."stat.taco"."inner.key"
You can also mix-and-match except that expressions such as: .["stat.taco"].["inner.key"] are not (as of jq 1.6) supported.