Stuctured logs in shell script using JQ - json

I've been wanting to logs some shells scripts to a file to be picked up by an agent like Fluentbit and shipped off to Cloudwatch and Datadog.
I found this example on the web which works like a charm using jq.
__timestamp(){
date "+%Y%m%dT%H%M%S"
}
__log(){
log_level="$1"
message="$2"
echo '{}' | jq \
--arg timestamp "$(__timestamp)"
--arg log_level "$log_level" \
--arg message "$message" \
'.timestamp=$timestamp|.log_level=$log_level|.message=$message' >> logs.log
}
__log "INFO" "Hello, World!"
The one issue is that the output is rendering a full json with newlines and tabs. Easy on the eyes, but not great for cloud logging services.
{
"timestamp": "20220203T162320",
"log_level": "INFO",
"message": "Hello, World!"
}
How would I modify to render the output on one line like so?
{"timestamp": "20220203T171908","log_level": "INFO","message": "Hello, World!"}

Use the --compact-output or -c option: jq -c --arg … '…' >> logs.log
From the manual:
--compact-output / -c:
By default, jq pretty-prints JSON output. Using this option will
result in more compact output by instead putting each JSON object
on a single line.
After having read your linked article about "Structured Logging", this is how I would have implemented said __log function:
__log(){
jq -Mcn --arg log_level "$1" --arg message "$2" \
'{timestamp: now|todate, $log_level, $message}'
}
__log "INFO" "Hello, World!"

🤦 Somehow I completely missed Step 3.0 The Finishing Touches of this terrific guide which answers my question perfectly.
In case anyone else was curious, the solution is here:
__timestamp(){
date "+%Y%m%dT%H%M%S"
}
__log(){
log_level="$1"
message="$2"
echo '{}' | \
jq --monochrome-output \
--compact-output \
--raw-output \
--arg timestamp "$(__timestamp)" \
--arg log_level "$log_level" \
--arg message "$message" \
'.timestamp=$timestamp|.log_level=$log_level|.message=$message'
}
__log "INFO" "Hello, World!"
Which outputs:
{"timestamp":"20210812T191730","log_level":"INFO","message":"Hello, World!"}

Related

Can you separate distinct JSON attributes into two files using jq?

I am following this tutorial from Vault about creating your own certificate authority. I'd like to separate the response (change the output to API call using cURL to see the response) into two distinct files, one file possessing the certificate and issuing_ca attributes, the other file containing the private_key. The tutorial is using jq to parse JSON objects, but my unfamiliarity with jq isn't helpful here, and most searches are returning info on how to merge JSON using jq.
I've tried running something like
vault write -format=json pki_int/issue/example-dot-com \
common_name="test.example.com" \
ttl="24h" \
format=pem \
jq -r '.data.certificate, .data.issuing_ca > test.cert.pem \
jq -r '.data.private_key' > test.key.pem
or
vault write -format=json pki_int/issue/example-dot-com \
common_name="test.example.com" \
ttl="24h" \
format=pem \
| jq -r '.data.certificate, .data.issuing_ca > test.cert.pem \
| jq -r '.data.private_key' > test.key.pem
but no dice.
It is not an issue with jq invocation, but the way the output files get written. Per your usage indicated, after writing the file test.cert.pem, the contents over the read end of the pipe (JSON output) is no longer available to extract the private_key contents.
To duplicate the contents over at the write end of pipe, use tee along with process substitution. The following should work on bash/zsh or ksh93 and not on POSIX bourne shell sh
vault write -format=json pki_int/issue/example-dot-com \
common_name="test.example.com" \
ttl="24h" \
format=pem \
| tee >( jq -r '.data.certificate, .data.issuing_ca' > test.cert.pem) \
>(jq -r '.data.private_key' > test.key.pem) \
>/dev/null
See this in action
jq -n '{data:{certificate: "foo", issuing_ca: "bar", private_key: "zoo"}}' \
| tee >( jq -r '.data.certificate, .data.issuing_ca' > test.cert.pem) \
>(jq -r '.data.private_key' > test.key.pem) \
>/dev/null
and now observe the contents of both the files.
You could abuse jq's ability to write to standard error (version 1.6 or later) separately from standard output.
vault write -format=json pki_int/issue/example-dot-com \
common_name="test.example.com" \
ttl="24h" \
format=pem \
| jq -r '.data as $f | ($f.private_key | stderr) | ($f.certificate, $f.issuing_ca)' > test.cert.pem 2> test.key.pem
There's a general technique for this type of problem that is worth mentioning
because it has minimal prerequisites (just jq and awk), and because
it scales well with the number of files. Furthermore it is quite efficient in that only one invocation each of jq and awk is needed. The idea is to setup a pipeline of the form: jq ... | awk ...
There are many variants
of the technique but in the present case, the following would suffice:
jq -rc '
.data
| "test.cert.pem",
"\t\(.certificate)",
"\t\(.issuing_ca)",
"test.key.pem",
"\t\(.private_key)"
' | awk -F\\t 'NF == 1 {fn=$1; next} {print $2 > fn}'
Notice that this works even if the items of interest are strings with embedded tabs.

How to use positional argument with embed quotes in bash?

I'm trying to create a bash script that automates configuration of some letsencrypt related stuff.
The file that I have to edit is json so I would just use jq to edit it and pass the site name to it from the positional arguments of the script, but I can't get the positional argument passed into the json text.
I'm trying to do domething like the following:
JSON=`jq '. + { "ssl_certificate": "/etc/letsencrypt/live/$2/fullchain.pem" }' <<< echo site_config.json`
JSON=`jq '. + { "ssl_certificate_key": "/etc/letsencrypt/live/$2/fullchain.pem" }' <<< ${JSON}`
echo -e "$JSON" > site_config.json
Where the second positional argument ($2) contain the domain name required to be set in the json file.
How this can be done?
Original json:
{
"key1":"value1",
"key2":"value2"
}
Wanted json:
{
"key1":"value1",
"key2":"value2",
"ssl_certificate": "/etc/letsencrypt/live/somesite.com/fullchain.pem",
"ssl_certificate_key": "/etc/letsencrypt/live/somesite.com/fullchain.pem"
}
1. String construction under bash
I use printf and octal representation for nesting quotes and double quotes:
printf -v JSON 'Some "double quoted: \047%s\047"' "Any string"
echo "$JSON"
Some "double quoted: 'Any string'"
2. Using jq, strictly answer to edited question:
myFunc() {
local file="$1" site="$2" JSON
printf -v JSON '. + {
"ssl_certificate": "/etc/letsencrypt/live/%s/fullchain.pem",
"ssl_certificate_key": "/etc/letsencrypt/live/%s/fullchain.pem"
}' "$site" "$site"
jq "$JSON" <"$file"
}
Then run:
myFunc site_config.json test.com
{
"key1": "value1",
"key2": "value2",
"ssl_certificate": "/etc/letsencrypt/live/test.com/fullchain.pem",
"ssl_certificate_key": "/etc/letsencrypt/live/test.com/fullchain.pem"
}
myFunc site_config.json test.com >site_config.temp && mv site_config.{temp,conf}
Or even:
myFunc <(
printf '{ "key1":"value1","key2":"value2","comment":"Let\047s doit\041" }'
) test.com
Will render:
{
"key1": "value1",
"key2": "value2",
"comment": "Let's doit!",
"ssl_certificate": "/etc/letsencrypt/live/test.com/fullchain.pem",
"ssl_certificate_key": "/etc/letsencrypt/live/test.com/fullchain.pem"
}
2b. Better written with jq's --arg option:
Thanks to peak's detailed answer!
I use bash arrays to store strings with quotes, spaces and other special characters. This is more readable as there is no need to escape end-of-line (backslashes) and permit comments:
myFunc() {
local file="$1" site="$2"
local JSON=(
--arg ssl_certificate "/etc/letsencrypt/live/$site/fullchain.pem"
--arg ssl_certificate_key "/etc/letsencrypt/live/$site/fullchain.pem"
'. + {$ssl_certificate, $ssl_certificate_key}' # this syntax
# do offer two advantages: 1: no backslashes and 2: permit comments.
)
jq "${JSON[#]}" <"$file"
}
3. Inline edit function
For editing a small script. I prefer to use cp -a in order to preserve
attributes and ensure a valid operation before replacement.
If you plan to use this, mostly for replacing, you could add replacement in your function:
myFunc() {
local REPLACE=false
[ "$1" = "-r" ] && REPLACE=true && shift
local file="$1" site="$2"
local JSON=( --arg ssl_certificate "/etc/letsencrypt/live/$site/fullchain.pem"
--arg ssl_certificate_key "/etc/letsencrypt/live/$site/fullchain.pem"
'. + {$ssl_certificate, $ssl_certificate_key}' )
if $REPLACE;then
cp -a "$file" "${file}.temp"
exec {out}>"${file}.temp"
else
exec {out}>&1
fi
jq "${JSON[#]}" <"$file" >&$out &&
$REPLACE && mv "${file}.temp" "$file"
exec {out}>&-
}
Then to modify file instead of dumping result to terminal, you have to add -r option:
myFunc -r site_config.json test.org
I cannot get the positional argument passed into the json text.
In general, by far the best way to do that is to use jq's --arg and/or --argjson command-line options. This is safe, and in the present case means that you would only have to call jq once. E.g.:
< site_config.json \
jq --arg sslc "/etc/letsencrypt/live/$2/fullchain.pem" \
--arg sslck "/etc/letsencrypt/live/$2/fullchain.pem" '
. + {ssl_certificate: $sslc, ssl_certificate_key: $sslck }'
Once you're sure things are working properly, feel free to use moreutils's sponge :-)
A DRY-er solution
Thanks to a neat convenience feature of jq, one can write more DRY-ly:
< site_config.json \
jq --arg ssl_certificate "/etc/letsencrypt/live/$2/fullchain.pem" \
--arg ssl_certificate_key "/etc/letsencrypt/live/$2/fullchain.pem" '
. + {$ssl_certificate, $ssl_certificate_key }'
``

use curl/bash command in jq

I am trying to get a list of URL after redirection using bash scripting. Say, google.com gets redirected to http://www.google.com with 301 status.
What I have tried is:
json='[{"url":"google.com"},{"url":"microsoft.com"}]'
echo "$json" | jq -r '.[].url' | while read line; do
curl -LSs -o /dev/null -w %{url_effective} $line 2>/dev/null
done
So, is it possible for us to use commands like curl inside jq for processing JSON objects.
I want to add the resulting URL to existing JSON structure like:
[
{
"url": "google.com",
"redirection": "http://www.google.com"
},
{
"url": "microsoft.com",
"redirection": "https://www.microsoft.com"
}
]
Thank you in advance..!
curl is capable of making multiple transfers in a single process, and it can also read command line arguments from a file or stdin, so, you don't need a loop at all, just put that JSON into a file and run this:
jq -r '"-o /dev/null\nurl = \(.[].url)"' file |
curl -sSLK- -w'%{url_effective}\n' |
jq -R 'fromjson | map(. + {redirection: input})' file -
This way only 3 processes will be spawned for the whole task, instead of n + 2 where n is the number of URLs.
I would generate a dictionary with jq per url and slurp those dictionaries into the final list with jq -s:
json='[{"url":"google.com"},{"url":"microsoft.com"}]'
echo "$json" | jq -r '.[].url' | while read url; do
redirect=$(curl -LSs \
-o /dev/null \
-w '%{url_effective}' \
"${url}" 2>/dev/null)
jq --null-input --arg url "${url}" --arg redirect "${redirect}" \
'{url:$url, redirect: $redirect}'
done | jq -s
Alternative (first) solution:
You can output the url and the effective_url as tab separated data and create the output json with jq:
json='[{"url":"google.com"},{"url":"microsoft.com"}]'
echo "$json" | jq -r '.[].url' | while read line; do
prefix="${line}\t"
curl -LSs -o /dev/null -w "${prefix}"'%{url_effective}'"\n" "$line" 2>/dev/null
done | jq -r --raw-input 'split("\t")|{"url":.[0],"redirection":.[1]}'
Both solutions will generate valid json, independently of whatever characters the url/effective_url might contain.
Trying to keep this in JSON all the way is pretty cumbersome. I would simply try to make Bash construct a new valid JSON fragment inside the loop.
So in other words, if $url is the URL and $redirect is where it redirects to, you can do something like
printf '{"url": "%s", "redirection": "%s"}\n' "$url" "$redirect"
to produce JSON output from these strings. So tying it all together
jq -r '.[].url' <<<"$json" |
while read -r url; do
printf '{"url:" "%s", "redirection": "%s"}\n' \
"$url" "$(curl -LSs -o /dev/null -w '%{url_effective}' "$url")"
done |
jq -s
This is still pretty brittle; in particular, if either of the printf input strings could contain a literal double quote, that should properly be escaped.

File content to single JSON string value with bash

I would like to read a file and put the whole content into a single string that is escaped to be used in a JSON object.
And I want to do it on the commandline/terminal (Linux).
Version 1
WARNING: With this solution the content of the file can be too big to fit in an argument!
jq -n \
--arg content "$(cat theFile.txt)" \
'{ theContent : $content }' \
| \
jq '.theContent'
Version 2
Jeff Mercado provided a more compact solution for the first part - so I adapted that in my code as follows:
jq -Rs \
'{ theContent: . }' \
theFile.txt \
| \
jq '.theContent'
Version 3
Now Jeff Mercado provided a more compact solution for what I was looking for:
jq -Rs '.' theFile.txt
A more direct way to do that is to use the raw input (-R) combined with slurp (-s) parameters to read the entire input as a single string. Then take that input and store in the appropriate property. You don't need to pass it in as a separate parameter.
$ jq -Rs '{ theContent: . }' theFile.txt

Modify Nested JSON with jq

I'm trying to modify nested JSON objects using the jq <map> function in a bash/shell script; something similar to this blog entry but attempting to adapt the examples here to nested objects.
The returned JSON to be modified as follows:
{
"name": "vendor-module",
"dependencies": {
"abc": {
"from": "abc#2.4.0",
"resolved": "https://some.special.url",
"version": "2.4.0"
},
"acme": {
"from": "acme#1.2.3",
"resolved": "<CHANGE_THIS>",
"version": "1.2.3"
}
}
}
This would be my attempt:
modules="`node -pe 'JSON.parse(process.argv[1]).dependencies.$dependency' \
"$(cat $wrapped)"`"
version="1.2.3"
resolved="some_url"
cat OLD.json |
jq 'to_entries |
map(if .dependencies[0].$module[0].from == "$module#$version"
then . + {"resolved"}={"$resolved"}
else .
end
) |
from_entries' > NEW.json
Obviously this doesn't work. When I run the script the NEW.json is created but without modifications or returned errors. If I don't target a nested object (e.g., "name": "vendor-module"), The script works as expected. I am sure there is a way to do it using native bash and jq..?? Any help (with the proper escaping) will be greatly appreciated.
UPDATE:
Thnx from the help of Charles Duffy's answer, and his suggestion of using sponge, The solution that works well for me is:
jq --arg mod "acme" --arg resolved "Some URL" \
'.dependencies[$mod].resolved |= $resolved' \
OLD.json | sponge OLD.json
If you know the name of the dependency you want to update, you could just index into it.
$ jq --arg dep "$dep" --arg resolved "$resolved" \
'.dependencies[$dep].resolved = $resolved' \
OLD.json > NEW.json
Otherwise, to modify a dependency based on the name (or other property), search for the dependency and update.
$ jq --arg version "$version" --arg resolved "$resolved" \
'(.dependencies[] | select(.version == $version)).resolved = $resolved' \
OLD.json > NEW.json
For your existing sample data, the following suffices:
jq --arg mod "acme" \
--arg resolved "some_url" \
'.dependencies[$mod].resolved=$resolved' \
<in.json >out.json
...to filter on the from, by contrast:
jq --arg new_url "http://new.url/" \
--arg target "acme#1.2.3" \
'.dependencies=(.dependencies
| to_entries
| map(if(.value.from == $target)
then .value.resolved=$new_url
else . end)
| from_entries)' \
<in.json >out.json