I have a json file that I want to manipulate; but I can't seem to get the data to change as I want.
If I execute 1; the return is "true" because the folder is paused.
1: jq '.folders[] | select(.label=='\"$folder\"') | .paused' "$f"
If I execute 2; a single record is selected and the "true" is replaced with "false".
2: jq '.folders[] | select(.label=='\"$folder\"') .paused = false' "$f"
If I execute 3; the entire file is returned but no change is made.
3: jq 'if (.folders[] | .label == '\"$folder\"') then .paused = false else . end' "$f"
I want the entire file returned with the change made so I can post it back to update the config I'm trying to change.
What am I doing wrong here?
You need to use select on the label selection, otherwise the context changes and you can't return to the whole object.
(.folders[] | select (.label == '\"$folder\"')).paused = false
Also, it's cleaner to propagate the variable value as a variable into jq instead of handling or corner cases of quoting yourself:
jq --arg folder "$folder" '(.folders[] | select (.label == $folder)).paused = false' "$f"
All working now!
See: Syncthing forum for more info.
I am trying to pause / resume specific folders on a schedule, using cron. This is not supported by the API yet so I needed to do it manually by editing the config.
Thanks for all the help.
Related
I have a JSON which goes like this:
{
"results":[
{
"uri":"www.xxx.com"
}
]
}
EDIT
When uri is not present, JSON looks like this:
{
"results":[
]
}
In some cases, uri is present and in some cases, it is not.
Now, I want to use jq to return boolean value if uri is present or not.
This is what I wrote so far but despite uri being present, it gives null.
${search_query_response} contains the JSON
file_status=$(jq -r '.uri' <<< ${search_query_response})
Can anyone guide me?
Since you use jq, it means you are working within a shell script context.
If the boolean result is to be handled by the shell script, you can make jq set its EXIT_CODE depending on the JSON request success or failure status, with jq -e
Example shell script using the EXIT_CODE from jq:
if uri=$(jq -je '.results[].uri') <<<"$search_query_response"
then
printf 'Search results contains an URI: %s.\n' "$uri"
else
echo 'No URI in search results.'
fi
See man jq:
-e / --exit-status:
Sets the exit status of jq to 0 if the last output values was neither false nor null, 1 if the last output value was either false or null, or 4 if no valid result was ever produced. Normally jq exits with 2 if there was any usage problem or system error, 3 if there was a jq program compile error, or 0 if the jq program ran.
Another way to set the exit status is with the halt_error builtin function.
The has function does the job:
jq '.results|map(has("uri"))|.[]'
map the has function on .results.
I'm using jq to format the log entries from journalctl -o json in a readable way. While that works nicely, I'm missing the journalctl indicator for a reboot. This information isn't directly included in the json format output. But it can be inferred by comparing the _BOOT_ID value of the current object with the one on the previous line. If they are different, the message belongs to another boot session and I need to insert that "-- Reboot --" line.
How can I compare that value in jq?
The following is a simple version of what I'm doing:
journalctl -o json --since today |jq -r '
"\(._SOURCE_REALTIME_TIMESTAMP // .__REALTIME_TIMESTAMP |tonumber |. / 1000000 |strflocaltime("%m-%d %H:%M:%S")) \(.SYSLOG_IDENTIFIER // .UNIT): \(.MESSAGE)"
' |less
It shows a timestamp, unit name, and the message. What I'm looking for is something like the following (where $$previous$$ does not exist):
journalctl -o json --since today |jq -r '
"\(if ._BOOT_ID != $$previous$$._BOOT_ID then "-- Reboot --\n" else "" end)\(._SOURCE_REALTIME_TIMESTAMP // .__REALTIME_TIMESTAMP |tonumber |. / 1000000 |strflocaltime("%m-%d %H:%M:%S")) \(.SYSLOG_IDENTIFIER // .UNIT): \(.MESSAGE)"
' |less
I'd also accept setting variables at the end of one line and accessing them at the beginning of the next line; but variables don't seem to exist, I could only set a property of the current object which won't help.
A solution can be obtained using foreach along the following lines:
jq -nr '
foreach inputs as $entry (null;
$entry
+ if (. != null) and (._BOOT_ID != $entry._BOOT_ID)
then {emit: true} else null end;
if .emit then "-- Reboot --" else empty end,
.foo)
'
You would replace .foo with the filter defining the projection of interest.
I'm working on parsing JSON data using JSON.sh. And I wanted to read data from json file (test.json) whose content will be something like,
{
"/home/ukrishnan/projects/test.yml": {
"LOG_DRIVER": "syslog",
"IMAGE": "mysql:5.6"
},
"/home/ukrishnan/projects/mysql/app.xml": {
"ENV_ACCOUNT_BRIDGE_ENDPOINT": "/u01/src/test/sample.txt"
}
}
And I try to parse this JSON using JSON.sh by using,
test_parser=`sh ./lib/JSON.sh < test/test.json`
echo $test_parser
It prints,
["/home/ukrishnan/projects/test.yml","LOG_DRIVER"] "syslog" ["/home/ukrishnan/projects/test.yml","IMAGE"] "mysql:5.6" ["/home/ukrishnan/projects/test.yml"] {"LOG_DRIVER":"syslog","IMAGE":"mysql:5.6"} ["/home/ukrishnan/projects/mysql/app.xml","ENV_ACCOUNT_BRIDGE_ENDPOINT"] "/u01/src/test/sample.txt" ["/home/ukrishnan/projects/mysql/app.xml"] {"ENV_ACCOUNT_BRIDGE_ENDPOINT":"/u01/src/test/sample.txt"} [] {"/home/ukrishnan/projects/test.yml":{"LOG_DRIVER":"syslog","IMAGE":"mysql:5.6"},"/home/ukrishnan/projects/mysql/app.xml":{"ENV_ACCOUNT_BRIDGE_ENDPOINT":"/u01/src/test/sample.txt"}}
Whereas, the same command (sh ./lib/JSON.sh < test/test.json), if I run through terminal, it is printing with line breaks,
["/home/ukrishnan/projects/test.yml","LOG_DRIVER"] "syslog"
["/home/ukrishnan/projects/test.yml","IMAGE"] "mysql:5.6"
["/home/ukrishnan/projects/test.yml"] {"LOG_DRIVER":"syslog","IMAGE":"mysql:5.6"}
["/home/ukrishnan/projects/mysql/app.xml","ENV_ACCOUNT_BRIDGE_ENDPOINT"] "/u01/src/test/sample.txt"
["/home/ukrishnan/projects/mysql/app.xml"] {"ENV_ACCOUNT_BRIDGE_ENDPOINT":"/u01/src/test/sample.txt"}
[] {"/home/ukrishnan/projects/test.yml":{"LOG_DRIVER":"syslog","IMAGE":"mysql:5.6"},"/home/ukrishnan/projects/mysql/app.xml":{"ENV_ACCOUNT_BRIDGE_ENDPOINT":"/u01/src/test/sample.txt"}}
I wanted to read this and assign to bash variables like,
file_name='/home/ukrishnan/projects/test.yml'
key='LOG_DRIVER'
value='syslog'
As I'm almost completely new to shell script and grep or awk, I don't have much idea of how to achieve this. Any help on this would be greatly appreciated.
I wrote a JSON serializer / deserializer for gawk, if you're interested. Save that script and modify it, replacing everything above # === FUNCTIONS === with the following:
#!/usr/bin/gawk -f
# capture JSON string from beginning to end into a scalar variable
{ json = json ORS $0 }
END {
# objectify JSON string to the multilevel array "obj"
deserialize(json, obj)
for (filename in obj) {
print "file_name=" quote(filename)
for (key in obj[filename]) {
# print key="value"
print key "=" quote(obj[filename][key])
}
}
}
Do chmod 755 json.awk and execute it. Output will resemble this:
$ ./json.awk test5.json
file_name="/home/ukrishnan/projects/mysql/app.xml"
ENV_ACCOUNT_BRIDGE_ENDPOINT="/u01/src/test/sample.txt"
file_name="/home/ukrishnan/projects/test.yml"
LOG_DRIVER="syslog"
IMAGE="mysql:5.6"
Hopefully the logic is reasonably easy to follow. If you prefer to output filename=, key=, and value= on every loop iteration, modify the nested for loops accordingly:
for (filename in obj) {
for (key in obj[filename]) {
print "file_name=" quote(filename)
print "key=" quote(key)
print "value=" quote(obj[filename][key])
}
}
That change will result in the following output:
$ ./json.awk test5.json
file_name="/home/ukrishnan/projects/mysql/app.xml"
key="ENV_ACCOUNT_BRIDGE_ENDPOINT"
value="/u01/src/test/sample.txt"
file_name="/home/ukrishnan/projects/test.yml"
key="LOG_DRIVER"
value="syslog"
file_name="/home/ukrishnan/projects/test.yml"
key="IMAGE"
value="mysql:5.6"
Anyway, with that output, you can do something silly in BASH like this to populate and act upon the variables:
#!/bin/bash
./test.awk test5.json | while read -r line; do {
eval $line
[ "${line/=*/}" = "value" ] && {
echo "bash: file_name=$file_name"
echo "bash: key=$key"
echo "bash: value=$value"
echo "------"
}
}; done
It'd probably be more graceful just to do all processing within gawk from start to finish and not mess with the polyglot handoff, though.
Getting back to json.awk, if you prefer to keep json.awk modular for easy reuse in future projects, you could remove everything above # === FUNCTIONS ===, create a separate main.awk containing the code block at the top of this answer, and #include "json.awk" as a helper library pretty much anywhere outside of END {...} (just below the shbang, for example).
JSON.sh (from http://json.org) offers a nice bash friendly means of flattening out a JSON file. Which you've already provided how it looks in your question. So, the flatten form is the format:
[node] tab value
You have to think in UNIX script in extracting the information you want, you'll note the lines you're interested in actually follow this pattern:
["filename","key"] tab ["value"]
In regex notation, we replace:
filename with (.*)
key with (.*)
tab with \t
value with (.*)
We can retrieve the first, second and third matching groups with \1, \2, \3 respectively.
When used in sed we also note that these symbols []() need to be escaped with a backslash \, resulting in the following script:
./lib/JSON.sh < test/test.json | sed 's/\["\(.*\)","\(.*\)\"]\t"\(.*\)"/\1,\2,\3/;t;d'
/home/ukrishnan/projects/test.yml,LOG_DRIVER,syslog
/home/ukrishnan/projects/test.yml,IMAGE,mysql:5.6
/home/ukrishnan/projects/mysql/app.xml,ENV_ACCOUNT_BRIDGE_ENDPOINT,/u01/src/test/sample.txt
Now we put the lines in a loop and for each line, we can extract out filename,key,value:
for line in $(./lib/JSON.sh < test/test.json | sed 's/\["\(.*\)","\(.*\)\"]\t"\(.*\)"/\1,\2,\3/;t;d')
do
IFS="," read -ra arr <<< $line
filename=${arr[0]}
key=${arr[1]}
value=${arr[2]}
cat <<EOF
filename : $filename
key : $key
value : $value
EOF
done
Which outputs:
filename : /home/ukrishnan/projects/test.yml
key : LOG_DRIVER
value : syslog
filename : /home/ukrishnan/projects/test.yml
key : IMAGE
value : mysql:5.6
filename : /home/ukrishnan/projects/mysql/app.xml
key : ENV_ACCOUNT_BRIDGE_ENDPOINT
value : /u01/src/test/sample.txt
Below is the Json I receive as Response from url.
{"flows":[{"version":"OF_13","cookie":"0","tableId":"0x0","packetCount":"24","byteCount":"4563","durationSeconds":"5747","priority":"0","idleTimeoutSec":"0","hardTimeoutSec":"0","flags":"0","match":{},"instructions":{"instruction_apply_actions":{"actions":"output=controller"}}},
{"version":"OF_13","cookie":"45036000240104713","tableId":"0x0","packetCount":"0","byteCount":"0","durationSeconds":"29","priority":"6","idleTimeoutSec":"0","hardTimeoutSec":"0","flags":"1","match":{"eth_type":"0x0x800","ipv4_src":"10.0.0.10","ipv4_dst":"10.0.0.12"},"instructions":{"none":"drop"}},
{"version":"OF_13","cookie":"45036000240104714","tableId":"0x0","packetCount":"0","byteCount":"0","durationSeconds":"3","priority":"7","idleTimeoutSec":"0","hardTimeoutSec":"0","flags":"1","match":{"eth_type":"0x0x800","ipv4_src":"10.0.0.10","ipv4_dst":"127.0.0.1"},"instructions":{"none":"drop"}},
{"version":"OF_13","cookie":"0","tableId":"0x1","packetCount":"0","byteCount":"0","durationSeconds":"5747","priority":"0","idleTimeoutSec":"0","hardTimeoutSec":"0","flags":"0","match":{},"instructions":{"instruction_apply_actions":{"actions":"output=controller"}}}]}
So, I have for example four flows and I want to extract only the field "byteCount" for a specific flow identify by the ipv4_src and ipv4_dst that i have to give it as input
How can I do this?
json_array := JSON.parse(json_string)
foreach (element in json_array.flows):
if(element.match.hasProperty('ipv4_src') && element.match.hasProperty('ipv4_dst')):
if(element.match.ipv4_src == myValue && element.match.ipv4_dst == otherValue):
print element.byteCount ;
The above is a pseudo-code to find byteCount based on ipv4_src and ipv4_dst. Note that these two properties are within match property, which may or may not contain them. Hence, first check for their existence and then process.
Note: When formatted property, each element in the array is like
{
"version":"OF_13",
"cookie":"45036000240104713",
"tableId":"0x0",
"packetCount":"0",
"byteCount":"0",
"durationSeconds":"29",
"priority":"6",
"idleTimeoutSec":"0",
"hardTimeoutSec":"0",
"flags":"1",
"match":{
"eth_type":"0x0x800",
"ipv4_src":"10.0.0.10",
"ipv4_dst":"10.0.0.12"
},
"instructions":{
"none":"drop"
}
}
Here's how to perform the selection and extraction task using the command-line tool jq:
First create a file, say "extract.jq", with these three lines:
.flows[]
| select(.match.ipv4_src == $src and .match.ipv4_dst == $dst)
| [$src, $dst, .byteCount]
Next, assuming the desired src and dst are 10.0.0.10 and 10.0.0.12 respectively, and that the input is in a file named input.json, run this command:
jq -c --arg src 10.0.0.10 --arg dst 10.0.0.12 -f extract.jq input.json
This would produce one line per match; in the case of your example, it would produce:
["10.0.0.10","10.0.0.12","0"]
If the JSON is coming from some command (such as curl), you can use a pipeline along the following lines:
curl ... | jq -c --arg src 10.0.0.10 --arg dst 10.0.0.12 -f extract.jq
I have to extract data from JSON file depending on a specific key. The data then has to be filtered (based on the key value) and separated into different fixed width flat files. I have to develop a solution using shell scripting.
Since the data is just key:value pair I can extract them by processing each line in the JSON file, checking the type and writing the values to the corresponding fixed-width file.
My problem is that the input JSON file is approximately 5GB in size. My method is very basic and would like to know if there is a better way to achieve this using shell scripting ?
Sample JSON file would look like as below:
{"Type":"Mail","id":"101","Subject":"How are you ?","Attachment":"true"}
{"Type":"Chat","id":"12ABD","Mode:Online"}
The above is a sample of the kind of data I need to process.
Give this a try:
#!/usr/bin/awk
{
line = ""
gsub("[{}\x22]", "", $0)
f=split($0, a, "[:,]")
for (i=1;i<=f;i++)
if (a[i] == "Type")
file = a[++i]
else
line = line sprintf("%-15s",a[i])
print line > file ".fixed.out"
}
I made assumptions based on the sample data provided. There is a lot based on those assumptions that may need to be changed if the data varies much from what you've shown. In particular, this script will not work properly if the data values or field names contain colons, commas, quotes or braces. If this is a problem, it's one of the primary reasons that a proper JSON parser should be used. If it were my assignment, I'd push back hard on this point to get permission to use the proper tools.
This outputs lines that have type "Mail" to a file named "Mail.fixed.out" and type "Chat" to "Chat.fixed.out", etc.
The "Type" field name and field value ("Mail", etc.) are not output as part of the contents. This can be changed.
Otherwise, both the field names and values are output. This can be changed.
The field widths are all fixed at 15 characters, padded with spaces, with no delimiters. The field width can be changed, etc.
Let me know how close this comes to what you're looking for and I can make some adjustments.
perl script
#!/usr/bin/perl -w
use strict;
use warnings;
no strict 'refs'; # for FileCache
use FileCache; # avoid exceeding system's maximum number of file descriptors
use JSON;
my $type;
my $json = JSON->new->utf8(1); #NOTE: expect utf-8 strings
while(my $line = <>) { # for each input line
# extract type
eval { $type = $json->decode($line)->{Type} };
$type = 'json_decode_error' if $#;
$type ||= 'missing_type';
# print to the appropriate file
my $fh = cacheout '>>', "$type.out";
print $fh $line; #NOTE: use cache if there are too many hdd seeks
}
corresponding shell script
#!/bin/bash
#NOTE: bash is used to create non-ascii filenames correctly
__extract_type()
{
perl -MJSON -e 'print from_json(shift)->{Type}' "$1"
}
__process_input()
{
local IFS=$'\n'
while read line; do # for each input line
# extract type
local type="$(__extract_type "$line" 2>/dev/null ||
echo json_decode_error)"
[ -z "$type" ] && local type=missing_type
# print to the appropriate file
echo "$line" >> "$type.out"
done
}
__process_input
Example:
$ ./script-name < input_file
$ ls -1 *.out
json_decode_error.out
Mail.out