How do I convert these two text strings into separate json objects
Text strings:
start process: Mon May 15 03:14:09 UTC 2017
logfilename: log_download_2017
Json output:
{
"start process": "Mon May 15 03:14:09 UTC 2017",
}
{
"logfilename": "log_download_2017",
}
Shell script:
logfilename="log_download_2017"
echo "start process: $(date -u)" | tee -a $logfilename.txt | jq -R split(:) >> $logfilename.json
echo "logfilename:" $logfilename | tee -a $logfilename.txt | jq -R split(:) >> $logfilename.json
One approach would be to use index/1, e.g. along these lines:
jq -R 'index(":") as $ix | {(.[:$ix]) : .[$ix+1:]}'
Or, if your jq supports regex, you might like to consider:
jq -R 'match( "([^:]*):(.*)" ) | .captures | {(.[0].string): .[1].string}'
or:
jq -R '[capture( "(?<key>[^:]*):(?<value>.*)" )] | from_entries'
Related
This is a variation on a question that's been asked before.
I'm using an external data source in Terraform to ask it for a list of volume snapshots in AWS Dublin, and JQ in a templatefile to extract the snapshot ids.
data "external" "volsnapshot_ids" {
program = [
"bash",
"-c",
templatefile("cli.tftpl", {input_string = "aws ec2 describe-snapshots --region=eu-west-1", top = "Snapshots", next = "| .SnapshotId"})]
}
And it uses this templatefile:
#!/bin/bash
set -e
OUTPUT=$(${input_string} | jq -r -c '.${top}[] ${next}' | jq -R -s -c 'split("\n")' | jq '.[:-1]')
jq -n -c --arg output "$OUTPUT" '{"output":$output}'
The basic CLI command with JQ works and looks like this:
aws ec2 describe-snapshots --region=eu-west-1 | jq -r -c '.Snapshots[] | .SnapshotId' | jq -R -s -c 'split("\n")' | jq '.[:-1]' | wc -l
It returns a lot of snapshot ids.
When I run it through Terraform though, it errors:
Error: External Program Execution Failed
│
│ with data.external.volsnapshot_ids,
│ on data.tf line 304, in data "external" "volsnapshot_ids":
│ 304: program = [
│ 305: "bash",
│ 306: "-c",
│ 307: templatefile("cli.tftpl", {input_string = "aws ec2 describe-snapshots --region=eu-west-1", top = "Snapshots", next = "| .SnapshotId"})]
│
│ The data source received an unexpected error while attempting to execute
│ the program.
│
│ Program: /bin/bash
│ Error Message: bash: line 6: /usr/local/bin/jq: Argument list too long
│
│ State: exit status 1
I think it's the size of the dataset being returned because it works in regions with less snapshot ids - London works.
Sizewise, here's London:
aws ec2 describe-snapshots --region=eu-west-2 | jq -r -c '.Snapshots[] | .SnapshotId' | jq -R -s -c 'split("\n")' | jq '.[:-1]' | wc -l
20000
And here's Dublin:
aws ec2 describe-snapshots --region=eu-west-1 | jq -r -c '.Snapshots[] | .SnapshotId' | jq -R -s -c 'split("\n")' | jq '.[:-1]' | wc -l
42500
Is there a way to fix up the JQ in my templatefile so it can handle big JSON files?
I wouldn't recommend using command inside TF datasource. Might be hard to debug. There is a data_source for EBS snapshots.
As for your command inside template, in order to debug it you need to simulate the same environment. E.g. instead of running as is, try to repeat what you have in template, like bash -c and so on. Also you can add output to see the template rendered to see if there are any issues.
Scroll to bottom of answer.
Don't provide the value as argument, but via directly standard input:
aws ... \
| jq -rc '.${top}[] ${next}' \
| jq -Rsc './"\n"' \
| jq -c '.[:-1]'
| jq -Rc '{output:.}'
Note that you are can probably combine most of the separate jq invocations into a single jq program.
This pipeline of jq invocations is a massively, massively overcomplicated non-solution. Why convert back and forth between strings and JSON objects, parsing those strings again, when jq can already process the data directly?
aws ... | jq -c '{ output: .Snapshots | map(.SnapshotId) | tostring }'
Example output:
{"output":"[\"snap-cafebabe\",\"snap-deadbeef\",\"snap-0123abcd\"]"}
If you have to use variables:
top=Snapshots
next=SnapshotId
aws ... | jq --arg top "$top" --arg next "$next" -c '{ output: .[$top] | map(.[$next]) | tostring }'
or .[$top] | map(.[$next]) | tostring | { output: . } or .[$top] | map(.[$next]) | { output: tostring }.
Even if you want or need to string together multiple jq invocations, there's little sense in consuming raw input (-R) and try to parse it, if you already have perfectly structured JSON items in stream form.
Here is what it would look like if you wanted to do it with multiple steps, but always stay in JSON land (and not play ping pong between structured JSON and unstructured text):
top=Snapshots
next=SnapshotId
aws ... \
| jq --arg top "$top" --arg next "$next" '.[$top][][$next]' \
| jq -sc '{ output: tostring }'
or the equivalent:
top=Snapshots
next=SnapshotId
aws ... \
| jq --arg top "$top" --arg next "$next" '.[$top] | map(.[$next])' \
| jq -c '{ output: tostring }'
I have a response trace file containing below response:
#RESPONSE BODY
#--------------------
{"totalItems":1,"member":[{"name":"name","title":"PatchedT","description":"My des_","id":"70EA96FB313349279EB089BA9DE2EC3B","type":"Product","modified":"2019 Jul 23 10:22:15","created":"2019 Jul 23 10:21:54",}]}
I need to fetch the value of the "id" key in a variable which I can put in my further code.
Expected result is
echo $id - should give me 70EA96FB313349279EB089BA9DE2EC3B value
With valid JSON (remove first to second row with sed and parse with jq):
id=$(sed '1,2d' file | jq -r '.member[]|.id')
Output to variable id:
70EA96FB313349279EB089BA9DE2EC3B
I would strongly suggest using jq to parse json.
But given that json is mostly compatible with python dictionaries and arrays, this HACK would work too:
$ cat resp
#RESPONSE BODY
#--------------------
{"totalItems":1,"member":[{"name":"name","title":"PatchedT","description":"My des_","id":"70EA96FB313349279EB089BA9DE2EC3B","type":"Product","modified":"2019 Jul 23 10:22:15","created":"2019 Jul 23 10:21:54",}]}
$ awk 'NR==3{print "a="$0;print "print a[\"member\"][0][\"id\"]"}' resp | python
70EA96FB313349279EB089BA9DE2EC3B
$ sed -n '3s|.*|a=\0\nprint a["member"][0]["id"]|p' resp | python
70EA96FB313349279EB089BA9DE2EC3B
Note that this code is
1. dirty hack, because your system does not have the right tool - jq
2. susceptible to shell injection attacks. Hence use it ONLY IF you trust the response received from your service.
Quick and dirty (don't use eval):
eval $(cat response_file | tail -1 | awk -F , '{ print $5 }' | sed -e 's/"//g' -e 's/:/=/')
It is based on the exact structure you gave, and hoping there is no , in any value before "id".
Or assign it yourself:
id=$(cat response_file | tail -1 | awk -F , '{ print $5 }' | cut -d: -f2 | sed -e 's/"//g')
Note that you can't access the name field with that trick, as it is the first item of the member array and will be "swallowed" by the { print $2 }. You can use an even-uglier hack to retrieve it though:
id=$(cat response_file | tail -1 | sed -e 's/:\[/,/g' -e 's/}\]//g' | awk -F , '{ print $5 }' | cut -d: -f2 | sed -e 's/"//g')
But, if you can, jq is the right tool for that work instead of ugly hacks like that (but if it works...).
When you can't use jq, you can consider
id=$(grep -Eo "[0-9A-F]{32}" file)
This is only working when the file looks like what I expect, so you might need to add extra checks like
id=$(grep "My des_" file | grep -Eo "[0-9A-F]{32}" | head -1)
input.json:-
{
"menu": {
"id": "file",
"value": "File",
"user": {
"address": "USA",
"email": "user#gmail.com"
}
}
}
Command:-
result=$(cat input.json | jq -r '.menu | keys[]')
Result:-
id
value
user
Loop through result:-
for type in "${result[#]}"
do
echo "--$type--"
done
Output:-
--id
value
user--
I want to do process the keys values in a loop. When I do the above, It result as a single string.
How can I do a loop with json keys result in bash script?
The canonical way :
file='input.json'
cat "$file" | jq -r '.menu | keys[]' |
while IFS= read -r value; do
echo "$value"
done
bash faq #1
But you seems to want an array, so the syntax is (missing parentheses) :
file='input.json'
result=( $(cat "$file" | jq -r '.menu | keys[]') )
for type in "${result[#]}"; do
echo "--$type--"
done
Output:
--id--
--value--
--user--
Using bash to just print an object keys from JSON data is redundant.
Jq is able to handle it by itself. Use the following simple jq solution:
jq -r '.menu | keys_unsorted[] | "--"+ . +"--"' input.json
The output:
--id--
--value--
--user--
How do I convert these two text strings into a single json object
Text strings:
start process: Mon May 15 03:14:09 UTC 2017
logfilename: log_download_2017
Json output:
{
"start process": "Mon May 15 03:14:09 UTC 2017",
"logfilename": "log_download_2017",
}
Shell script:
logfilename="log_download_2017"
echo "start process: $(date -u)" | tee -a $logfilename.txt | jq -R . >> $logfilename.json
echo "logfilename:" $logfilename | tee -a $logfilename.txt | jq -R . >> $logfilename.json
As mentioned e.g. at Use jq to turn x=y pairs into key/value pairs, the basic task of converting a key:value string can be accomplished in a number of ways. For example, you could start with:
index(":") as $ix | {(.[:$ix]) : .[$ix+1:]}
You evidently want to trim some spaces, which can be done using sub/2.
To combine the objects, you could use add. To do this in a single pass, you would use jq -R -s
Putting it all together, you could do worse than:
def trim: sub("^ +";"") | sub(" +$";"");
def s2o:
(index(":") // empty) as $ix
| {(.[:$ix]): (.[$ix+1:]|trim)};
split("\n") | map(s2o) | add
I try to convert the output of ps aux into Json format without using Perl or Python! For these I have read about jq. But I have success to convert the commandline output into json.
How to convert a simpe ps aux to Json?
ps aux | awk '
BEGIN { ORS = ""; print " [ "}
{ printf "%s{\"user\": \"%s\", \"pid\": \"%s\", \"cpu\": \"%s\"}",
separator, $1, $2, $3
separator = ", "
}
END { print " ] " }';
Just adjust columns which you need from ps aux output.
jq can read non-JSON input. You'll want to pre-process the input with awk first:
ps aux |
awk -v OFS=, '{print $1, $2}' |
jq -R 'split(",") | {user: .[0], pid: .[1]}'
If you want an array instead of a sequence of objects, pipe the output through jq --slurp 'add'. (I swear there's a way to do that without an extra call to jq, but it escapes me at the moment.)
Here's an only-jq solution based on tokenization.
Tokenization can be done using:
def tokens:
def trim: sub("^ +";"") | sub(" +$";"");
trim | splits(" +");
For illustration and brevity, let's consider only the first 10 tokens:
[tokens] | .[0:9]
Invocation:
$ ps aux | jq -c -R -f tokens.jq
Or as a one-liner, you could get away with:
$ ps aux | jq -cR '[splits(" +")] | .[0:9]'
First few lines of output:
["USER","PID","%CPU","%MEM","VSZ","RSS","TT","STAT","STARTED"]
["p","1595","55.9","0.4","2593756","32832","??","R","24Jan17"]
["p","12472","26.6","12.6","4951848","1058864","??","R","Sat01AM"]
["p","13239","10.9","1.5","4073756","128324","??","R","Sun12AM"]
["p","12482","7.8","1.2","3876628","101736","??","R","Sat01AM"]
["p","32039","7.7","1.4","4786968","118424","??","R","12Feb17"]
["_windowserver","425","7.6","0.8","3445536","65052","??","Ss","24Jan17"]
Using the headers as object keys
See e.g.
https://github.com/stedolan/jq/wiki/Cookbook#convert-a-csv-file-with-headers-to-json
I have a gist that to convert ps output to json. It uses jq under the covers so you need to install that. But you do not need to know jq
Dumps the fields specified by the -o flag as an array of PID objects:
ps ax -o "stat,euid,ruid,tty,tpgid,sess,pgrp,ppid,pid,wchan,sz,pcpu,command" \
| jq -sRr ' sub("\n$";"") | split("\n") | ([.[0]|splits(" +")]) as $header | .[1:] | [.[] | [. as $x | range($header|length) | {"key": $header[.], "value": (if .==($header|length-1) then ([$x|splits(" +")][.:]|join(" ")|tojson|.[1:length-1]) else ([$x|splits(" +")][.]) end) } ] | from_entries]'
This builds an array of the header fields, maps an array of {key, value} objects per output object, and then uses the built-in from_entries filter to object-aggregate these into the outputs.