If I have a JSON template in which some variable should be replaced with their actual value, is there a good way to handle the proper escape?
For example, $value may be replaced with a string that contains characters like " that should be treated specially in JSON.
{ "x": $value }
The template could be arbitrary complex. So it is not a good solution to code the template in some programming language like python, then perform the replacement in that language, then dump the JSON output.
Could anybody show me a generic but succinct way to perform the replacement?
Note that I tagged this question with jq. I am not sure it is strictly relevant to this question. If not, please remove the tag. I tagged jq because people who know jq may also know solutions to my question, although jq is just for transforming a JSON file. An elegant solution may be similar to jq in the sense that a domain-specific language is defined.
jq works quite nicely as a template engine, but there are choices to be made, e.g. depending on whether the "template" itself is valid JSON.
In the example you gave, the template is not valid JSON, but it is potentially valid jq, so the strategy using jq "$-variables" would make sense, e.g. along the lines of:
jq -n --arg value "someValue" -f template.jq
where template.jq is your template.
Three different strategies for using jq as a template engine are explained at some length in the jq Cookbook:
https://github.com/stedolan/jq/wiki/Cookbook#using-jq-as-a-template-engine
Related
I have a json file that is getting created at runtime using the sh script within groovy code. The json file has below contents.
cat.json
{
"user1":"pass1",
"user2":"pass2",
"user3":"pass3"
}
Now I want to create a file at runtime which stores key value pairs in below format
test
user1:pass1
user2:pass2
user3:pass3
can some one help me out shell codes for writing this.
You have literally dozen ways to convert that JSON document to a tabular data file (pretty much like CSV/colon-SV) since you mentioned Java, Groovy, including Java-driven scripting engines (BeanShell, JavaScript, Groovy itself), but if you can use jq then you can extract k/v pairs at least for simple values that do not require any escaping:
#!/bin/sh
jq -r 'to_entries[] | "\(.key):\(.value)"' \
< cat.json
This answer is inspired by searching for extracting entries using jq (or converting a JSON file to a CSV file) and especially by the answer https://stackoverflow.com/a/50496145/12232870 by #peak.
I'm working with multiple JSON files that are located in the same folder.
Files contain objects with the same properties and they are such as:
{
"identifier": "cameraA",
"alias": "a",
"rtsp": "192.168.1.1"
}
I want to replace a property for all the objects in the JSON files at the same time for a certain condition.
For example, let's say that I want to replace all the rtsp values of the objects with identifier equal to "cameraA".
I've been trying with something like:
jq 'if .identifier == \"cameraA" then .rtsp=\"cameraX" else . end' -c *.json
But it isn't working.
Is there a simple way to replace the property of an object among multiple JSON files?
jq can only write to STDIN and STDOUT, so the simplest approach would be to process one file at a time, e.g. putting your jq program inside a shell loop. sponge is often used when employing this approach.
However, there is an alternative that has the advantage of efficiency. It requires only one invocation of jq, the output of which would include the filename information (obtained from input_filename). This output would then be the input of an auxiliary process, e.g. awk.
Given an invalid JSON string such as: { foo: bar } is it possible to get jq to process and format correctly as { "foo": "bar" }
No, or at least not without complex programming, though jq can handle objects with unquoted key names, e.g. {foo: "bar"}. (Hint: read the quasi-JSON as a jq program.)
The jq FAQ, however, does have a section giving details about a number of command-line tools that can be recommended for this kind of task, e.g. any-json and hjson. That page provides links as well.
This question already has an answer here:
jq special characters in nested keys
(1 answer)
Closed 3 years ago.
I am trying to use the jq command line JSON processor https://shapeshed.com/jq-json/ (which works great) to process a JSON file that seems to have been made using some poor choices.
Normally your id and value in the JSON file would not contain any periods such as:
{"id":"d9s7g9df7sd9","name":"Tacos"}
To get Tacos from the file you would do the following in bash:
echo $json | jq -r '.name'
This will give you Tacos (There may be some extra code missing from that example but you get the point.)
I have a JSON file that looks like this:
{"stat.blah":123,"stat.taco":495,"stat.yum... etc.
Notice how they decided to use a period in the identifying field associated with the value? This makes using jq very difficult because it associates the period as a separator to dig down into child values in the JSON. Sure, I could first load my file, replace all "." with "_" and that would fix the problem, but this seems like a really dumb and hackish solution. I have no way to change how the initial JSON file is generated. I just have to deal with it. Is there a way in bash I can do some special escape to make it ignore the period?
Thanks
Use generic object index syntax, e.g:
.["stat.taco"]
If you use the generic object syntax, e.g. .["stat.taco"], then chaining is done either using pipes as usual, or without the dot, e.g.
.["stat.taco"]["inner.key"]
If your jq is sufficiently recent, then you can use the chained-dot notation by quoting the keys with special characters, e.g.
."stat.taco"."inner.key"
You can also mix-and-match except that expressions such as: .["stat.taco"].["inner.key"] are not (as of jq 1.6) supported.
I have this which works to declare a JSON string in a bash script:
local my_var="foobar"
local json=`cat <<EOF
{"quicklock":"${my_var}"}
EOF`
The above heredoc works, but I can't seem to format it any other way, it literally has to look exactly like that lol.
Is there any way to get the command to be on one line, something like this:
local json=`cat <<EOF{"quicklock":"${my_var}"}EOF`
that would be so much nicer, but doesn't seem to take, obviously simply because that's not how EOF works I guess lol.
I am looking for a shorthand way to declare JSON in a file that:
Does not require a ton of escape chars.
That allows for dynamic interpolation of variables.
Note: The actual JSON I want to use has multiple dynamic variables with many key/value pairs. Please extrapolate.
I'm not a JSON guy, don't really understand the "well-formed" arguments in the discussion above, but, you can use a 'here-string' rather than a 'here-document', like this:
my_var="foobar"
json=`cat <<<{\"quicklock\":\"${my_var}\"}`
why not use jq? It's pretty good at managing string interpolation and it lints your structure.
$ echo '{}' >> foo.json
$ declare myvar="assigned-var"
$ jq --arg ql "$myvar" '.quicklock=$ql' foo.json
the text that comes out on the other end of that call to jq can then be cat into a file or whatever you wanna do. text would look something like this:
{"quicklock": "assigned-var"}
You can do this with printf:
local json="$(printf '{"quicklock":"%s"}' "$my_var")"
(Never mind that SO's syntax highlighting looks odd here. Posix shell command substitution allows nesting one level of quotes.)
A note (thanks to Charles Duffy's comment on the question): I'm assuming $my_var is not controlled by user input. If it is, you'll need to be careful to ensure it is legal for a JSON string. I highly recommend barring non-ASCII characters, double quotes, and backslashes. If you have jq available, you can use it as Charles noted in the comments to ensure you have well-formed output.
You can define your own helper function to address the situation with missing bash syntax:
function begin() { eval echo $(sed "${BASH_LINENO[0]}"'!d;s/.*begin \(.*\) end.*/\1/;s/"/\\\"/g' "${BASH_SOURCE[0]}"); }
Then you can use it as follows.
my_var="foobar"
json=$(begin { "quicklock" : "${my_var}" } end)
echo "$json"
This fragment displays the desired output:
{ "quicklock" : "foobar" }
This is just a proof of concept. You can define your syntax in any way you want (such as end of the input by the custom EOF string, correctly escape invalid characters). For example, since Bash allows function identifiers using characters other than alphanumeric characters, it is possible to define such a syntax:
json=$(/ { "quicklock" : "${my_var}" } /)
Moreover, if you relax the first criterion (escape characters), ordinary assignment will nicely solve this problem:
json="{ \"quicklock\" : \"${my_var}\" }"
How about just using the shell's natural concatenation of strings? If you concatenate ${mybar} rather than interpolate it, you can avoid escapes and get everything on one line:
my_var1="foobar"
my_var2="quux"
json='{"quicklock":"'${my_var1}'","slowlock":"'$my_var2'"}'
That said, this is a pretty crude scheme, and as others have pointed out you'll have problems if the variables, say, contain quote characters.
Since no escape chars is strong requirement here is a here-doc based solution:
#!/bin/bash
my_var='foobar'
read -r -d '' json << EOF
{
"quicklock": "$my_var"
}
EOF
echo "$json"
It will give you the same output as the first solution I mentioned.
Just be careful, if you would put first EOF inside double quotes:
read -r -d '' json << "EOF"
$my_var would not be considered as a variable but as a plain text, so you would get this output:
{
"quicklock": "$my_var"
}