I am running a puppet bolt command query certain information from a set of servers in json format. I am piping it to jq.. Below is what I get
$ bolt command run "cat /blah/blah" -n #hname.txt -u uid --no-host-key-check --format json |jq -jr '.items[]|[.node],[.result.stdout]'
[
"node-name"
][
"stdout data\n"
]
What do I need to do to make it appear like below
["nodename":"stdout data"]
If you really want output that is not valid JSON, you will have to construct the output string, which can easily be done using string interpolation, e.g.:
jq -r '.items[] | "[\"\(.node)\",\"\(.result.stdout)\"]"'
#peak thank you.. that helped. Below is how it looks like
$ bolt command run "cat /blah/blah" -n #hname.txt -u UID --no-host-key-check --format json |jq -r '.items[] | "[\"\(.node)\",\"\(.result.stdout)\"]"'
["node name","stdout data
"]
I used a work around to get the data I needed by using the #csv flag to the command itself. Sharing with you below what worked.
$ bolt command run "cat /blah/blah" -n #hname.txt -u uid --no-host-key-check --format json |jq -jr '.items[]|[.node],[.result.stdout]|#csv'
""node-name""stdout.data
"
Related
Hoping someone can help me. I'm trying to understand the qBittorrent Web API. At the moment I'm listing all the paused torrents with:
curl -i http://localhost:8080/api/v2/torrents/info?category=test
The problem is that lists the whole JSON array - my question is can I just display the "name" or "hash" fields? This is all using curl through cmd, but I've tried this in Git Bash & Powershell:
[{"eta":8640000,"f_l_piece_prio":false,"force_start":false,"hash":"8419d48d86a14335c83fdf4930843438a2f75a6b","last_activity":1664863523,"magnet_uri":"","max_seeding_time":0,"**name**":"TestTorrentName","num_complete":12,"num_incomplete":1,"num_leechs":0,"num_seeds":0,"priority":0,"progress":1,"ratio":0,"ratio_limit":-2,"save_path":"F:\\Completed\\test\\","seeding_time":0,"seeding_time_limit":-2,"seen_complete":1664863523,"seq_dl":false,"size":217388295,"state":"pausedUP","super_seeding":false,"tags":"","time_active":569,"total_size":217388295,"tracker":"udp://open.stealth.si:80/announce","trackers_count":10,"up_limit":-1,"uploaded":0,"uploaded_session":0,"upspeed":0}]
I've tried the following that should work according to https://jqplay.org/ - see screenshot
curl -i http://localhost:8080/api/v2/torrents/info?category=test | jq --raw-output '.[] | .name'
But unfortunately I'm getting the following error:
curl -i http://localhost:8080/api/v2/torrents/info?category=test | jq --raw-output '.[] | .name'
% Total % Received % Xferd Average Speed Time '.name'' is not recognized as an internal or external command,
operable program or batch file.
Ti
curl -i http://localhost:8080/api/v2/torrents/info?category=test | jq --raw-output '.[] | .name'
The -i let curl give some header info, that is parsed to jq, but jq can only parse JSON end therefore fails.
Remove the -i and optionally replace it with -s to remove the stats:
curl -s http://localhost:8080/api/v2/torrents/info?category=test | jq --raw-output '.[] | .name'
I have a bunch of secrets (key/value) pairs stored in AWS Secrets Manager. I tried to parse the secrets using jq as:
aws secretsmanager get-secret-value --secret-id <secret_bucket_name> | jq --raw-output '.SecretString' | jq -r .PASSWORD
It retrieves the value stored in .PASSWORD, but the problem is I not only want to retrieve the value stored in key but also want to retrieve the key/value in the following manner:
KEY_1="1234"
KEY_2="0000"
.
.
.
so on...
By running the above command I am not able to parse in this format and also for every key/value I have to run this command many times which is tedious. Am I doing something wrong or is there a better way of doing this?
This isn't related to python, but more related to behaviour of aws cli and jq. I come up with something like this.
aws secretsmanager get-secret-value --secret-id <secret_name> --output text --query SecretString | jq ".[]"
There are literally hundred different ways to format something like this.
aws cli itself has lot of options to filter output using --query option https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-output.html
Exact conversion you are looking for would require somwthing like this:
aws secretsmanager get-secret-value --secret-id <secret_name> --output text --query SecretString \
| jq -r 'to_entries[] | [.key, "=", "\"", .value, "\"" ] | #tsv' \
| tr -d "\t"
There has to be some better way of doing this!!
Try the snippet below. I tend to put these little helper filters into their own shell function <3
tokv() {
jq -r 'to_entries|map("\(.key|ascii_upcase)=\"\(.value|tostring)\"")|.[]'
}
$ echo '{"foo":"bar","baz":"fee"}' | tokv
FOO="bar"
BAZ="fee"
I'm trying to perform a bulk upload to Elasticsearch (around 1mln documents). In order to do that, I'm using jq to reformat the JSON file extracted from MySQL database and curl to post the data to Elasticsearch:
cat dataset.json | jq -r -c '.[] | { "index" : { } }, .' | curl -u login:password -H "Content-Type: application/json" -XPOST "https://.../skills/default/_bulk?pretty" --data-binary #-
I get an error:
parse error: Invalid string: control characters from U+0000 through U+001F must be escaped at line 276249, column 317
I found that the character that jq can't parse is \u2022. I tried adding "-r" jq command but the error stil occurs. How can I handle this for all occurrences of \u2022?
Here's verification that \u2022 is properly handled by various versions of jq in a Mac environment:
$ echo '"\u2022"' | jq-1.4 .
"•"
$ echo '"•"' | jq-1.6 .
"•"
$ echo '"•"' | jq-1.5 .
"•"
$ echo '"•"' | jq-1.4 .
"•"
$
Perhaps the problem is related to a bug that was fixed since the release of jq 1.5 (see e.g. https://github.com/stedolan/jq/issues/1311).
If you are having difficulties with jq version 1.6 (the current version), please provide a minimal complete verifiable example
with further details about the computing environment.
I have a file containing:
{"id":1,"jsonrpc":"2.0","result":{"speed":0}}
How would I be able to grep "0" after "speed":"?
I have tried 'grep -o -P "speed":{1}', not what I am looking for.
You should use jq (sudo apt-get install jq on raspbian) for this task.
echo '{"id":1,"jsonrpc":"2.0","result":{"speed":0}}' | jq .result.speed
Result: 0
Since you said in your question that you have a file "containing" this line, you might want to use grep first to get only the line you're interested in, otherwise jq might throw an error.
Example file:
abc
{"id":1,"jsonrpc":"2.0","result":{"speed":0}}
123
Running grep "speed" yourfile.txt | jq .result.speed would output 0.
I am trying to use a grep to search a JSON output, I used a curl command to return the data from a particular codeship build and I want to use GREP to store said ID value in a variable. However after I run the command and try to echo out the value of the variable its blank.
Below are the commands:
export API_KEY=abc123
export PROJECT_ID=123456
export LAST_BUILD_ID=$(curl -s https://codeship.com/api/v1/projects/$PROJECT_ID.json?api_key=$API_KEY | grep -Eo '"builds":\[{"id":\d+' | grep -Eo --color=never '\d+' | tail -1)
export LAST_BUILD_URL=$(echo "https://codeship.com/api/v1/builds/$LAST_BUILD_ID/restart.json?api_key=$API_KEY")
My response : never use grep nor regex to parse json.
Instead, use a proper json parser.
In shell, take a look to jq.
Example, adapt it a bit :
#!/bin/bash
API_KEY=abc123
PROJECT_ID=123456
html=$(curl -s https://codeship.com/api/v1/projects/$PROJECT_ID.json?api_key=$API_KEY)
LAST_BUILD_ID=$(jq '.builds | .[] | .never' <<< "$html") # just guessing
LAST_BUILD_URL=$(echo "https://codeship.com/api/v1/builds/$LAST_BUILD_ID/restart.json?api_key=$API_KEY")
Note
If you provide the JSON, I will be able to be more specific with the jq command