Related
I have file containing Multiple JSON Objects and need to covert them to JSON. I have bash and Excel installed, but cannot install any other tool.
{"name": "a","age":"17"}
{"name":"b","age":"18"}
To:
[{"name": "a","age":"17"},
{"name":"b","age":"18"}]
Assuming one object per line as shown by OP question
echo -n "["; while read line; do echo "${line},"; done < <(cat test.txt) | sed -re '$ s/(.*),/\1]/'
Result:
[{"name": "a","age":"17"},
{"name":"b","age":"18"}]
Inspired by https://askubuntu.com/a/475804
awk '(NR==FNR){count++} (NR!=FNR){ if (FNR==1) printf("["); printf("%s", $0); print (FNR==count)?"]":"," } END {if (count==0) print "[]"}' file file
A less compact but more readable version:
awk '
(NR==FNR) {
count++;
}
(NR!=FNR) {
if (FNR==1)
printf("[");
printf("%s", $0);
if (FNR==count)
print "]";
else
print ",";
}
END {
if (count==0) print "[]";
}' file file
The trick is to give the same file twice to awk. Because NR==FNR is always true for the first file, there is a first parse dedicated to counting the number of lines into variable count.
The second parse with NR!=FNR will apply the following algorithm for each line:
Write [ for the first line only
Then write the record, using printf instead of print in order to avoid the newline ending
Then write either ] or , depending on whether we are on the last line or not, using print in order to end with a newline
The END command is just a failsafe to output an empty array in case the file is empty.
Assumptions:
no requirement to (re)format the input data
Sample input:
$ cat raw.dat
{"name": "a","age":"17"}
{"name":"b","age":"18"}
{"name":"C","age":"23"}
One awk idea:
awk 'BEGIN {pfx="["} {printf "%s%s",pfx,$0; pfx=",\n"} END {printf "]\n"}' raw.dat
Where:
for each input line we printf the line without a terminating linefeed
for the first line we use a prefix (pfx) of [
for subsequent lines the prefix (pfx) is set to ,\n (ie, terminate the previous line with ,\n)
once the file has been processed we terminate the last input line with a printf "]\n"
requires a single pass through the input file
This generates:
[{"name": "a","age":"17"},
{"name":"b","age":"18"},
{"name":"C","age":"23"}]
Making sure #chepner's comment (re: a sed solution) isn't lost in the mix:
sed '1s/^/[/;2,$s/^/,/;$s/$/]/' raw.dat
This generates:
[{"name": "a","age":"17"}
,{"name":"b","age":"18"}
,{"name":"C","age":"23"}]
NOTE: I can remove this if #chepner wants to post this as an answer.
I'm working on parsing JSON data using JSON.sh. And I wanted to read data from json file (test.json) whose content will be something like,
{
"/home/ukrishnan/projects/test.yml": {
"LOG_DRIVER": "syslog",
"IMAGE": "mysql:5.6"
},
"/home/ukrishnan/projects/mysql/app.xml": {
"ENV_ACCOUNT_BRIDGE_ENDPOINT": "/u01/src/test/sample.txt"
}
}
And I try to parse this JSON using JSON.sh by using,
test_parser=`sh ./lib/JSON.sh < test/test.json`
echo $test_parser
It prints,
["/home/ukrishnan/projects/test.yml","LOG_DRIVER"] "syslog" ["/home/ukrishnan/projects/test.yml","IMAGE"] "mysql:5.6" ["/home/ukrishnan/projects/test.yml"] {"LOG_DRIVER":"syslog","IMAGE":"mysql:5.6"} ["/home/ukrishnan/projects/mysql/app.xml","ENV_ACCOUNT_BRIDGE_ENDPOINT"] "/u01/src/test/sample.txt" ["/home/ukrishnan/projects/mysql/app.xml"] {"ENV_ACCOUNT_BRIDGE_ENDPOINT":"/u01/src/test/sample.txt"} [] {"/home/ukrishnan/projects/test.yml":{"LOG_DRIVER":"syslog","IMAGE":"mysql:5.6"},"/home/ukrishnan/projects/mysql/app.xml":{"ENV_ACCOUNT_BRIDGE_ENDPOINT":"/u01/src/test/sample.txt"}}
Whereas, the same command (sh ./lib/JSON.sh < test/test.json), if I run through terminal, it is printing with line breaks,
["/home/ukrishnan/projects/test.yml","LOG_DRIVER"] "syslog"
["/home/ukrishnan/projects/test.yml","IMAGE"] "mysql:5.6"
["/home/ukrishnan/projects/test.yml"] {"LOG_DRIVER":"syslog","IMAGE":"mysql:5.6"}
["/home/ukrishnan/projects/mysql/app.xml","ENV_ACCOUNT_BRIDGE_ENDPOINT"] "/u01/src/test/sample.txt"
["/home/ukrishnan/projects/mysql/app.xml"] {"ENV_ACCOUNT_BRIDGE_ENDPOINT":"/u01/src/test/sample.txt"}
[] {"/home/ukrishnan/projects/test.yml":{"LOG_DRIVER":"syslog","IMAGE":"mysql:5.6"},"/home/ukrishnan/projects/mysql/app.xml":{"ENV_ACCOUNT_BRIDGE_ENDPOINT":"/u01/src/test/sample.txt"}}
I wanted to read this and assign to bash variables like,
file_name='/home/ukrishnan/projects/test.yml'
key='LOG_DRIVER'
value='syslog'
As I'm almost completely new to shell script and grep or awk, I don't have much idea of how to achieve this. Any help on this would be greatly appreciated.
I wrote a JSON serializer / deserializer for gawk, if you're interested. Save that script and modify it, replacing everything above # === FUNCTIONS === with the following:
#!/usr/bin/gawk -f
# capture JSON string from beginning to end into a scalar variable
{ json = json ORS $0 }
END {
# objectify JSON string to the multilevel array "obj"
deserialize(json, obj)
for (filename in obj) {
print "file_name=" quote(filename)
for (key in obj[filename]) {
# print key="value"
print key "=" quote(obj[filename][key])
}
}
}
Do chmod 755 json.awk and execute it. Output will resemble this:
$ ./json.awk test5.json
file_name="/home/ukrishnan/projects/mysql/app.xml"
ENV_ACCOUNT_BRIDGE_ENDPOINT="/u01/src/test/sample.txt"
file_name="/home/ukrishnan/projects/test.yml"
LOG_DRIVER="syslog"
IMAGE="mysql:5.6"
Hopefully the logic is reasonably easy to follow. If you prefer to output filename=, key=, and value= on every loop iteration, modify the nested for loops accordingly:
for (filename in obj) {
for (key in obj[filename]) {
print "file_name=" quote(filename)
print "key=" quote(key)
print "value=" quote(obj[filename][key])
}
}
That change will result in the following output:
$ ./json.awk test5.json
file_name="/home/ukrishnan/projects/mysql/app.xml"
key="ENV_ACCOUNT_BRIDGE_ENDPOINT"
value="/u01/src/test/sample.txt"
file_name="/home/ukrishnan/projects/test.yml"
key="LOG_DRIVER"
value="syslog"
file_name="/home/ukrishnan/projects/test.yml"
key="IMAGE"
value="mysql:5.6"
Anyway, with that output, you can do something silly in BASH like this to populate and act upon the variables:
#!/bin/bash
./test.awk test5.json | while read -r line; do {
eval $line
[ "${line/=*/}" = "value" ] && {
echo "bash: file_name=$file_name"
echo "bash: key=$key"
echo "bash: value=$value"
echo "------"
}
}; done
It'd probably be more graceful just to do all processing within gawk from start to finish and not mess with the polyglot handoff, though.
Getting back to json.awk, if you prefer to keep json.awk modular for easy reuse in future projects, you could remove everything above # === FUNCTIONS ===, create a separate main.awk containing the code block at the top of this answer, and #include "json.awk" as a helper library pretty much anywhere outside of END {...} (just below the shbang, for example).
JSON.sh (from http://json.org) offers a nice bash friendly means of flattening out a JSON file. Which you've already provided how it looks in your question. So, the flatten form is the format:
[node] tab value
You have to think in UNIX script in extracting the information you want, you'll note the lines you're interested in actually follow this pattern:
["filename","key"] tab ["value"]
In regex notation, we replace:
filename with (.*)
key with (.*)
tab with \t
value with (.*)
We can retrieve the first, second and third matching groups with \1, \2, \3 respectively.
When used in sed we also note that these symbols []() need to be escaped with a backslash \, resulting in the following script:
./lib/JSON.sh < test/test.json | sed 's/\["\(.*\)","\(.*\)\"]\t"\(.*\)"/\1,\2,\3/;t;d'
/home/ukrishnan/projects/test.yml,LOG_DRIVER,syslog
/home/ukrishnan/projects/test.yml,IMAGE,mysql:5.6
/home/ukrishnan/projects/mysql/app.xml,ENV_ACCOUNT_BRIDGE_ENDPOINT,/u01/src/test/sample.txt
Now we put the lines in a loop and for each line, we can extract out filename,key,value:
for line in $(./lib/JSON.sh < test/test.json | sed 's/\["\(.*\)","\(.*\)\"]\t"\(.*\)"/\1,\2,\3/;t;d')
do
IFS="," read -ra arr <<< $line
filename=${arr[0]}
key=${arr[1]}
value=${arr[2]}
cat <<EOF
filename : $filename
key : $key
value : $value
EOF
done
Which outputs:
filename : /home/ukrishnan/projects/test.yml
key : LOG_DRIVER
value : syslog
filename : /home/ukrishnan/projects/test.yml
key : IMAGE
value : mysql:5.6
filename : /home/ukrishnan/projects/mysql/app.xml
key : ENV_ACCOUNT_BRIDGE_ENDPOINT
value : /u01/src/test/sample.txt
I wrote some script that takes all user data of aws ec2 instance, and echo to local.json. All this happens when I install my node.js modules.
I don't know how to delete last comma in the json file. Here is the bash script:
#!/bin/bash
export DATA_DIR=/data
export PATH=$PATH:/usr/local/bin
#install package from git repository
sudo -- sh -c "export PATH=$PATH:/usr/local/bin; export DATA_DIR=/data; npm install git+https://reader:secret#bitbucket.org/somebranch/$1.git#$2"
#update config files from instance user-data
InstanceConfig=`cat /instance-config`
echo '{' >> node_modules/$1/config/local.json
while read line
do
if [ ! -z "$line" -a "$line" != " " ]; then
Key=`echo $line | cut -f1 -d=`
Value=`echo $line | cut -f2 -d=`
if [ "$Key" = "Env" ]; then
Env="$Value"
fi
printf '"%s" : "%s",\n' "$Key" "$Value" >> node_modules/*/config/local.json
fi
done <<< "$InstanceConfig"
sed -i '$ s/.$//' node_modules/$1/config/local.json
echo '}' >> node_modules/$1/config/local.json
To run him im doing that way: ./script
I get json(OUTPUT), but with comma in all lines. Here is local.json that I get:
{
"Env" : "dev",
"EngineUrl" : "engine.url.net",
}
All I trying to do, is delete in last line of the json file - comma(",").
I try many ways, that I found in internet. I know that it should be after last "fi"(end of the loop). And I know that it should be something like this line:
sed -i "s?${Key} : ${Value},?${Key} : ${Value}?g" node_modules/$1/config/local.json
Or this:
sed '$ s/,$//' node_modules/$1/config/local.json
But they not work for me.
Can someone help me with that? Who knows Bash scripting well?
Thanks!
If you know that it is the last comma that needs to be replaced, a reasonably robust way is to use GNU sed in "slurp" mode like this:
sed -zr 's/,([^,]*$)/\1/' local.json
Output:
{
"Env" : "dev",
"EngineUrl" : "engine.url.net"
}
If you'd just post some sample input/output it'd remove the guess-work but IF this is your input file:
$ cat file
Env=dev
EngineUrl=engine.url.net
Then IF you're trying to do what I think you are then all you need is:
$ cat tst.awk
BEGIN { FS="="; sep="{\n" }
{
printf "%s \"%s\" : \"%s\"", sep, $1, $2
sep = ",\n"
}
END { print "\n}" }
which you'd execute as:
$ awk -f tst.awk file
{
"Env" : "dev",
"EngineUrl" : "engine.url.net"
}
Or you can execute the awk script inline within a shell script if you prefer:
awk '
BEGIN { FS="="; sep="{\n" }
{
printf "%s \"%s\" : \"%s\"", sep, $1, $2
sep = ",\n"
}
END { print "\n}" }
' file
{
"Env" : "dev",
"EngineUrl" : "engine.url.net"
}
The above is far more robust, portable, efficient and better in every other way than the shell script you posted because it's using the right tool for the job. A UNIX shell is an environment from which to call tools with a language to sequence those calls. It is NOT a language to process text which is why it's so difficult to get it right. The UNIX tool for general text processing is awk so when you need to process text in UNIX, you just have shell call awk, that's all.
Here a jq version if it's available:
jq --raw-input 'split("=") | {(.[0]):.[1]}' /instance-config | jq --slurp 'add'
There might be a way to do it with one jqpass, but I couldn't see it.
You an remove all trailing commas from invalid json with:
sed -i.bak ':begin;$!N;s/,\n}/\n}/g;tbegin;P;D' FILE
sed -i.bak = creates a backup of the original file, then applies changes to the file
':begin;$!N;s/,\n}/\n}/g;tbegin;P;D' = anything ending with , followed by
"new line and }". Remove the , on the previous line
FILE = the file you want to make the change to
If you're willing to use it, xidel is rather forgiving for trailing commas:
xidel -s local.json -e '$json'
{
"Env": "dev",
"EngineUrl": "engine.url.net"
}
xidel - -se '$json' <<< '{"Env":"dev","EngineUrl":"engine.url.net",}'
#or
xidel - -se 'parse-json($raw,{"liberal":true()})' <<< '{"Env":"dev","EngineUrl":"engine.url.net",}'
{
"Env": "dev",
"EngineUrl": "engine.url.net"
}
I want to convert my sqlite data from my database to JSON format.
I would like to use this syntax:
sqlite3 -line members.db "SELECT * FROM members LIMIT 3" > members.txt
OUTPUT:
id = 1
fname = Leif
gname = Håkansson
genderid = 1
id = 2
fname = Yvonne
gname = Bergman
genderid = 2
id = 3
fname = Roger
gname = Sjöberg
genderid = 1
How to do this with nice and structur code in a for loop?
(Only in Bash)
I have tried some awk and grep but not with a great succes yet.
Would be nice with some tips.
I want a result similar to this:
[
{
"id":1,
"fname":"Leif",
"gname":"Hakansson",
"genderid":1
},
{
"id":2,
"fname":"Yvonne",
"gname":"Bergman",
"genderid":2
},
{
"id":3,
"fname":"Roger",
"gname":"Sjberg",
"genderid":1
}
}
If your sqlite3 is compiled with the json1 extension (or if you can obtain a version of sqlite3 with the json1 extension), then you can use it to generate JSON objects (one JSON object per row). For example:
select json_object('id', id, 'fname', fname, 'gname', gname, 'genderid', genderid) ...
You can then use a tool such as jq to convert the stream of objects into an array of objects, e.g. pipe the output of the sqlite3 to jq -s ..
(A less tiresome alternative might be to use the sqlite3 function json_array(), which produces an array, which you can reassemble into an object using jq.)
If the json1 extension is unavailable, then you could use the following as a starting point:
awk 'BEGIN { print "["; }
function out() {if (n++) {print ","}; if (line) {print "{" line "}"}; line="";}
function trim(x) { sub(/^ */, "", x); sub(/ *$/, "", x); return x; }
NF==0 { out(); next};
{if (line) {line = line ", " }
i=index($0,"=");
line = line "\"" trim(substr($0,1,i-1)) ": \"" substr($0, i+2) "\""}
END {out(); print "]"} '
Alternatively, you could use the following jq script, which converts numeric strings that occur on the RHS of "=" to numbers:
def trim: sub("^ *"; "") | sub(" *$"; "");
def keyvalue: index("=") as $i
| {(.[0:$i] | trim): (.[$i+2:] | (tonumber? // .))};
[foreach (inputs, "") as $line ({object: false, seed: {} };
if ($line|trim) == "" then { object: .seed, seed : {} }
else {object: false,
seed: (.seed + ($line | keyvalue)) }
end;
.object | if . and (. != {}) then . else empty end ) ]
Just type -json argument with SQLite 3.33.0 or higher and get json output:
$ sqlite3 -json database.db "select * from TABLE_NAME"
from SQLite Release 3.33.0 note:
...
CLI enhancements:
Added four new output modes: "box", "json", "markdown", and "table".
The "column" output mode automatically expands columns to contain the longest output row and automatically turns ".header" on if it has
not been previously set.
The "quote" output mode honors ".separator"
The decimal extension and the ieee754 extension are built-in to the CLI
...
I think I would prefer to parse sqlite output with a single line per record rather than the very wordy output format you suggested with sqlite3 -line. So, I would go with this:
sqlite3 members.db "SELECT * FROM members LIMIT 3"
which gives me this to parse:
1|Leif|Hakansson|1
2|Yvonne|Bergman|2
3|Roger|Sjoberg|1
I can now parse that with awk if I set the input separator to | with
awk -F '|'
and pick up the 4 fields on each line with the following and save them in an array like this:
{ id[++i]=$1; fname[i]=$2; gname[i]=$3; genderid[i]=$4 }
Then all I need to do is print the output format you need at the end. However, you have double quotes in your output and they are a pain to quote in awk, so I temporarily use another pipe symbol (|) as a double quote and then, at the very end, I get tr to replace all the pipe symbols with double quotes - just to make the code easier on the eye. So the total solution looks like this:
sqlite3 members.db "SELECT * FROM members LIMIT 3" | awk -F'|' '
# sqlite output line - pick up fields and store in arrays
{ id[++i]=$1; fname[i]=$2; gname[i]=$3; genderid[i]=$4 }
END {
printf "[\n";
for(j=1;j<=i;j++){
printf " {\n"
printf " |id|:%d,\n",id[j]
printf " |fname|:|%s|,\n",fname[j]
printf " |gname|:|%s|,\n",gname[j]
printf " |genderid|:%d\n",genderid[j]
closing=" },\n"
if(j==i){closing=" }\n"}
printf closing;
}
printf "]\n";
}' | tr '|' '"'
Sqlite-utils does exactly what you're looking for. By default, the output will be JSON.
Better late than never to plug jo.
Save sqlite3 to a text file.
Get jo (jo's also available in distro repos)
and use this bash script.
while read line
do
id=`echo $line | cut -d"|" -f1`
fname=`echo $line | cut -d"|" -f2`
gname=`echo $line | cut -d"|" -f3`
genderid=`echo $line | cut -d"|" -f4`
jsonline=`jo id="$id" fname="$fname" gname="$gname" genderid="$genderid"`
json="$json $jsonline"
done < "$1"
jo -a $json
Please don't create (or parse) json with awk. There are dedicated tools for this. Tools like xidel.
While first and foremost a html, xml and json parser, xidel can also parse plain text.
I'd like to offer a very elegant solution using this tool (with much less code than jq).
I'll assume your 'members.txt'.
First to create a sequence of each json object to-be:
xidel -s members.txt --xquery 'tokenize($raw,"\n\n")'
Or...
xidel -s members.txt --xquery 'tokenize($raw,"\n\n") ! (position(),.)'
1
id = 1
fname = Leif
gname = Håkansson
genderid = 1
2
id = 2
fname = Yvonne
gname = Bergman
genderid = 2
3
id = 3
fname = Roger
gname = Sjöberg
genderid = 1
...to better show you the individual items in the sequence.
Now you have 3 multi-line strings. To turn each item/string into another sequence where each item is a new line:
xidel -s members.txt --xquery 'tokenize($raw,"\n\n") ! x:lines(.)'
(x:lines(.) is a shorthand for tokenize(.,'\r\n?|\n'))
Now for each line tokenize on the " = " (which creates yet another sequence) and save it to a variable. For the first line for example this sequence is ("id","1"), for the second line ("fname","Leif"), etc.:
xidel -s members.txt --xquery 'tokenize($raw,"\n\n") ! (for $x in x:lines(.) let $a:=tokenize($x," = ") return ($a[1],$a[2]))'
Finally remove leading whitespace (normalize-space()), create a json object ({| {key-value-pair} |}) and put all json objects in an array ([ ... ]):
xidel -s members.txt --xquery '[tokenize($raw,"\n\n") ! {|for $x in x:lines(.) let $a:=tokenize($x," = ") return {normalize-space($a[1]):$a[2]}|}]'
Prettified + output:
xidel -s members.txt --xquery '
[
tokenize($raw,"\n\n") ! {|
for $x in x:lines(.)
let $a:=tokenize($x," = ")
return {
normalize-space($a[1]):$a[2]
}
|}
]
'
[
{
"id": "1",
"fname": "Leif",
"gname": "Håkansson",
"genderid": "1"
},
{
"id": "2",
"fname": "Yvonne",
"gname": "Bergman",
"genderid": "2"
},
{
"id": "3",
"fname": "Roger",
"gname": "Sjöberg",
"genderid": "1"
}
]
Note: For xidel-0.9.9.7173 and newer --json-mode=deprecated is needed to create a json array with [ ]. The new (XQuery 3.1) way to create a json array is to use array{ }.
I have created a JSON file which in this case contains:
{"ipaddr":"10.1.1.2","hostname":"host2","role":"http","status":"active"},
{"ipaddr":"10.1.1.3","hostname":"host3","role":"sql","status":"active"},
{"ipaddr":"10.1.1.4","hostname":"host4","role":"quad","status":"active"},
On other side I have a variable with values for example:
arr="10.1.1.2 10.1.1.3"
which comes from a subsequent check of the server status for example. For those values I want to change the status field to "inactive". In other words to grep the host and change its "status" value.
Expected output:
{"ipaddr":"10.1.1.2","hostname":"host2","role":"http","status":"inactive"},
{"ipaddr":"10.1.1.3","hostname":"host3","role":"sql","status":"inactive"},
{"ipaddr":"10.1.1.4","hostname":"host4","role":"quad","status":"active"},
$ arr="10.1.1.2 10.1.1.3"
$ awk -v arr="$arr" -F, 'BEGIN { gsub(/\./,"\\.",arr); gsub(/ /,"|",arr) }
$1 ~ "\"(" arr ")\"" { sub(/active/,"in&") } 1' file
{"ipaddr":"10.1.1.2","hostname":"host2","role":"http","status":"inactive"},
{"ipaddr":"10.1.1.3","hostname":"host3","role":"sql","status":"inactive"},
{"ipaddr":"10.1.1.4","hostname":"host4","role":"quad","status":"active"},
Here is a quick perl "wrap-around one-liner": that uses the JSON module and slurps with the -0 switch:
perl -MJSON -n0E '$j = decode_json($_);
for (#{$j->{hosts}}){$_->{status}=inactive if $_->{ipaddr}=~/2|3/} ;
say to_json( $j->{hosts}, {pretty=>1} )' status_data.json
might be nicer or might violate PBP recommendations for map:
perl -MJSON -n0E '$j = decode_json($_);
map { $_->{status}=inactive if $_->{ipaddr}=~/2|3/ } #{ $j->{hosts} } ;
say to_json( $j->{hosts} )' status_data.json
A shell script that resets status using jq would also be possible. Here's a quick way to parse and output changes to JSON using jq:
cat status_data.json| jq -r '.hosts |.[] |
select(.ipaddr == "10.1.1.2"//.ipaddr == "10.1.1.3" )' |jq '.status = "inactive"'
EDIT In an earlier comment I was uncertain whether the OP was more interested in an application than a quick search and replace (something about the phrases "On other side..." and "check on the server status"). Here is a (still simple) perl approach in script form:
use v5.16; #strict, warnings, say
use JSON ;
use IO::All;
my $status_data < io 'status_data.json';
my $network = JSON->new->utf8->decode($status_data) ;
my #changed_hosts= qw/10.1.1.2 10.1.1.3/;
sub status_report {
foreach my $host ( #{ $network->{hosts} }) {
say "$host->{hostname} is $host->{status}";
}
}
sub change_status {
foreach my $host ( #{ $network->{hosts} }){
foreach (#changed_hosts) {
$host->{status} = "inactive" if $host->{ipaddr} eq $_ ;
}
}
status_report;
}
defined $ENV{CHANGE_HAPPENED} ? change_status : status_report ;
The script reads the JSON file status_data.json (using IO::All which is great fun) then decodes it with JSON into a hash. It is hard to tell if this us a complete a solution because if you are "monitoring" host status then we should check the JSON data file periodically and compare it to our hash and then run the main body of the script one when changes have occurred.
To simulate changes occurring you can define/undefine CHANGE_HAPPENED in your environment with export CHANGE_HAPPENED=1 (or setenv if in in tcsh) and unset CHANGE_HAPPENED and the script will then either update the messages and the hash or "report". For this to be complete the data in our hash should be updated to match the the data file either periodically or when an event occurs. The status_report() subroutine could be changed so that it builds arrays of #inactive_hosts and #active_hosts when update_status() told it to do so: if ( something_happened() ) { update_status() }, etc.
Hope that helps.
status_data.json
{
"hosts":[
{"ipaddr":"10.1.1.2","hostname":"host2","role":"http","status":"active"},
{"ipaddr":"10.1.1.3","hostname":"host3","role":"sql","status":"active"},
{"ipaddr":"10.1.1.4","hostname":"host4","role":"quad","status":"active"}
]
}
output:
~/ % perl network_status_json.pl
host2 is active
host3 is active
host4 is active
~/ % export CHANGE_HAPPENED=1
~/ % perl network_status_json.pl
host2 is inactive
host3 is inactive
host4 is active
Version 1:
Using a simple regex based transformation. This can be done in several ways. From the initial question, the list of ipaddr is in variable in arr. Example using a Bash env variable:
$ export var="... ..."
It would be a possible solution to provide this information by command line parameters.
#!/usr/bin/perl
my %inact; # ipaddr to inactivate
my $arr=$ENV{arr} ; # from external var (export arr=...)
## $arr=shift; # from command line arg
for( split(/\s+/, $arr)){ $inact{$_}=1 }
while(<>){ # one "json" line at the time
if(/"ipaddr":"(.*?)"/ and $inact{$1}){
s/"active"/"inactive"/}
print $_;
}
Version 2:
Using Json parser we can do more complex transformations; as the input is not real JSON we will process one line of "almost json" at the time:
use JSON;
use strict;
my ($line, %inact);
my $arr=$ENV{arr} ;
for( split(/\s+/, $arr)){ $inact{$_}=1 }
while(<>){ # one "json" line at the time
if(/^\{.*\},/){
s/,\n//;
$line = from_json( $_);
if($inact{$line->{ipaddr}}){
$line->{status} = "inactive" ;}
print to_json($line), ",\n"; }
else { print $_;}
}
#!/bin/ksh
# your "array" of IP
arr="10.1.1.2 10.1.1.3"
# create and prepare temporary file for sed action
SedAction=/tmp/Action.sed
# --- for/do generating SedAction --------
echo "#sed action" > ${SedAction}
#take each IP from the arr variable one by one
for IP in ${arr}
do
# prepare for a psearch pattern use
IP_RE="$( echo "${IP}" | sed 's/\./\\./g' )"
# generate sed action in temporary file.
# final action will be like:
# s/\("ipaddr":"10\.1\.1\.2".*\)"active"}/\1"inactive"}/;t
# escape(double) \ for in_file espace, escape(simple) " for this line interpretation
echo "s/\\\(\"ipaddr\":\"${IP_RE}\".*\\\)\"active\"}/\\\1\"inactive\"}/;t" >> ${SedAction}
done
# --- sed generating sed action ---------------
echo "${arr}" \
| tr " " "\n" \
| sed 's/\./\\./g
s#.*#s/\\("ipaddr":"&".*\\)"active"}/\\1"inactive"}/;t#
' \
> ${SedAction}
# core of the process (use -i for inline editing or "double" redirection for non GNU sed)
sed -f ${SedAction} YourFile
# clean temporary file
rm ${SedAction}
Self commented, tested in ksh/AIX.
2 way to generate the SedAction depending of action you want to do also (if any). You only need one to work, i prefer the second
This is very simple indeed in Perl, using the JSON module.
use strict;
use warnings;
use JSON qw/ from_json to_json /;
my $json = JSON->new;
my $data = from_json(do { local $/; <DATA> });
my $arr = "10.1.1.2 10.1.1.3";
my %arr = map { $_ => 1 } split ' ', $arr;
for my $item (#$data) {
$item->{status} = 'inactive' if $arr{$item->{ipaddr}};
}
print to_json($data, { pretty => 1 }), "\n";
__DATA__
[
{"ipaddr":"10.1.1.2","hostname":"host2","role":"http","status":"active"},
{"ipaddr":"10.1.1.3","hostname":"host3","role":"sql","status":"active"},
{"ipaddr":"10.1.1.4","hostname":"host4","role":"quad","status":"active"}
]
output
[
{
"role" : "http",
"hostname" : "host2",
"status" : "inactive",
"ipaddr" : "10.1.1.2"
},
{
"hostname" : "host3",
"role" : "sql",
"ipaddr" : "10.1.1.3",
"status" : "inactive"
},
{
"ipaddr" : "10.1.1.4",
"status" : "active",
"hostname" : "host4",
"role" : "quad"
}
]