How to access JSON data from perl? - json

I have a JSON file as below.
{
"card":{
"cardName":"10AN10G",
"portSignalRates":[
"10AN10G-1-OTU2",
"10AN10G-1-OTU2E",
"10AN10G-1-TENGIGE",
"10AN10G-1-STM64"
],
"listOfPort":{
"10AN10G-1-OTU2":{
"portAid":"10AN10G-1-OTU2",
"signalType":"OTU2",
"tabNames":[
"PortDetails"
],
"requestType":{
"PortDetails":"PTP"
},
"paramDetailsMap":{
"PortDetails":[
{
"type":"dijit.form.TextBox",
"name":"signalType",
"title":"Signal Rate",
"id":"",
"options":[
],
"label":"",
"value":"OTU2",
"checked":"",
"enabled":"false",
"selected":""
},
{
"type":"dijit.form.TextBox",
"name":"userLabel",
"title":"Description",
"id":"",
"options":[
],
"label":"",
"value":"",
"checked":"",
"enabled":"true",
"selected":""
},
{
"type":"dijit.form.Select",
"name":"Frequency",
"title":"Transmit Frequency"
}
]
}
}
}
}
}
I require the output to be:
signalType:"Signal Rate",
userLabel:"Description",
Frequency:"Transmit Frequency",.. ,.....
I tried with:
grep -oP '(?<=\"title\":\")[^"]*' file > outfile
but this just splits the value of title and returns.
Can I use perl to access elements of the JSON data that I want?

I suggest to use decode_json instead of a regex. First, install the JSON module from CPAN:
sudo perl -MCPAN -e 'install JSON'
Alternatively you can use apt-get on Ubuntu:
sudo apt-get install libjson-pp-perl
Once it is installed, you can use this code:
my $json = '{... your json string ...}';
my $decoded = decode_json($json);
$decoded->{'card'}{'listOfPort'}{'10AN10G-1-OTU2'}{'signalType'}
You can find more details in this article.

Yes you can use the JSON perl module. Of course it meeds to be installed via cpan, cpanm or your system's packaging system. Parse the JSON into a hash and then use that in the normal way from Perl. Here's a quick example:
use JSON;
use IO::All;
use strict;
use warnings;
my $data < io '/tmp/data.json';
my $j = decode_json($data);
use DDP; # for quick debug printing
p $j->{card}{listOfPort}{"10AN10G-1-OTU2"}{paramDetailsMap}{PortDetails}[0]{title}
"Signal Rate"
You might want to use some deep diving techniques to get at the inner values more easily/programmatically (take a look at the Data::Diver module for that) - my example is meant to show only that it is possible and a bit of the mechanics of mapping JSON into a perl hash with the excellent JSON module. The documentation has lots of useful examples.

Perl on the command line:
perl -pe 's/"name":"([^"]+)","title":/"$1":/g' file > outfile
Output (pertinent part):
{"card": ... "signalType":"Signal Rate", ... "userLabel":"Description", ... "Frequency":"Transmit Frequency" ... }

Related

How to add commas in between JSON objects using Linux Shell and SnowSQL?

While there are several posts about this topic on Stack Overflow, none match my exact use case. I am using a Linux shell script to run SnowSQL to generate a json file.
========================
My json file needs to have a comma between json objects.
This:
{
"CAMPAIGN": "Welcome_New",
"UUID": "fe881781-bdc2-41b2-95f2-e0e8c19dc597"
}
{
"CAMPAIGN": "Welcome_Existing",
"UUID": "77a41c02-beb9-48bf-ada4-b2074c1a78cb"
}
...needs to look this:
{
"CAMPAIGN": "Welcome_New",
"UUID": "fe881781-bdc2-41b2-95f2-e0e8c19dc597"
},
{
"CAMPAIGN": "Welcome_Existing",
"UUID": "77a41c02-beb9-48bf-ada4-b2074c1a78cb"
}
Here is my complete ksh script:
#!/usr/bin/ksh
. /appl/.snf_logon
export SNOW_PKEY_FILE=$(mktemp ./pkey-XXXXXX)
trap "rm -f ${SNOW_PKEY_FILE}" EXIT
LibGetSnowCred
{
outFile=JSON_FILE_TYPE_TEST.json
inDir=/testing
outFileNm=#my_db.my_schema.my_file_stage/${outFile}
snowsql \
--private-key-path $SNOW_PKEY_FILE \
-o exit_on_error=true \
-o friendly=false \
-o timing=false \
-o log_level=ERROR \
-o echo=true <<!
COPY INTO ${outFileNm}
FROM (SELECT object_construct(
'UUID',UUID
,'CAMPAIGN',CAMPAIGN)
FROM my_db.my_schema.JSON_Test_Table
LIMIT 2)
FILE_FORMAT=(
TYPE=JSON
COMPRESSION=NONE
)
OVERWRITE=True
HEADER=False
SINGLE=True
MAX_FILE_SIZE=4900000000
;
get ${outFileNm} file://${inDir}/;
rm ${outFileNm};
!
if [ $? -eq 0 ]; then
echo "Export successful"
else
echo "ERROR in export"
fi
}
Is the best practice to add the comma during the SELECT or after the file is generated and how?
With or without that comma, the text is still not JSON but just a random text that looks like JSON. You export several rows, each row as an independent object. You need to gather all these objects into an array to produce a valid JSON.
A JSON that encodes an array of rows looks like this:
[
{
"CAMPAIGN": "Welcome_New",
"UUID": "fe881781-bdc2-41b2-95f2-e0e8c19dc597"
},
{
"CAMPAIGN": "Welcome_Existing",
"UUID": "77a41c02-beb9-48bf-ada4-b2074c1a78cb"
}
]
The easiest way to produce this output would be to ask the database, if it supports this option (to wrap all the records into a list before generating the JSON, to not export each record in a separate JSON).
If this is not possible then you have a file that contains multiple JSONs. You can use jq to convert these individual JSONs into a JSON similar to the one described above (encoding an array of objects).
It is as simple as that:
jq --slurp '.' input_file > output_file
The option --slurp tells jq to read all the JSONs from the file input_file in memory, to parse them and to put them into an array. That is the program input.
'.' is the jq program. It says "dump the current object". It does not do any processing to the input data. The current object is the array.
After it executes the program (which, in this case doesn't do anything), jq dumps the modified value (as JSON, of course) to the standard output (by default, on screen).
The > output_file part redirects this output to a file (named output_file) instead of showing it on screen.
You can see how it works on the jq playground.

Delete unused key:value properties in JSON file

I have key:value JSON object that is used in my JavaScript project. Value is a string and this object looks like this
{
key1:{
someKey: "Some text",
someKey2: "Some text2"
},
key2:{
someKey3:{
someKey4: "Some text3",
someKey5: "Some text4"
}
}
}
I use it in the project like this: key1.someKey and key2.someKey3.someKey4. Do you have idea how to delete unused properties? Let's say we don't use key2.someKey3.someKey5 in any file in a project, so i want it to be deleted from a JSON file. To people in the comments. I did't say i want to use JavaScript for this. I don't want to use it in browser or server. I just want the script that can do that on my local computer.
If you live within javascript and node, you can use something like this to get all the paths:
Using some modified code from here: https://stackoverflow.com/a/70763473/999943
var lodash=require('lodash') // use this if calling from the node REPL
// import lodash from 'lodash'; // use this if calling from a script
const allPaths = (o, prefix = '', out = []) => {
if (lodash.isObject(o) || lodash.isArray(o)) Object.entries(o).forEach(([k, v]) => allPaths(v, prefix === '' ? k : `${prefix}.${k}`, out));
else out.push(prefix);
return out;
};
let j = {
key1: { someKey: 'Some text', someKey2: 'Some text2' },
key2: { someKey3: { someKey4: 'Some text3', someKey5: 'Some text4' } }
}
allPaths(j)
[
'key1.someKey',
'key1.someKey2',
'key2.someKey3.someKey4',
'key2.someKey3.someKey5'
]
That's all well and good, but now you want to take that list and look through your codebase for usage.
The main choices available are text searching with grep or awk or ag, or parse the language and look through the symbolic representation of the language after it's loaded into your project. Tree-shaking can do this for libraries... I haven't looked into how to do tree-shaking for dictionary keys, or some other undefined reference check like a linter may do for a language.
Then once you have all the instances found, then you either manually modify your list or use a json library to modify it.
My weapons of choice in this instance are:
jq and bash and grep
It's not infallible. But it's a start. (use with caution).
setup_test.sh
#!/usr/bin/env bash
mkdir src
echo "key2.someKey3.someKey4" > src/a.js
echo "key1.someKey2" > src/b.js
echo "key3.otherKey" > src/c.js
test.json
{
"key1":{
"someKey": "Some text",
"someKey2": "Some text2"
},
"key2":{
"someKey3":{
"someKey4": "Some text3",
"someKey5": "Some text4"
}
}
}
check_for_dict_references.sh
#!/usr/bin/env bash
json_input=$1
code_path=$2
cat << HEREDOC
json_input=$json_input
code_path=$code_path
HEREDOC
echo "Paths found in json"
paths="$(cat "$json_input" | jq -r 'paths | join(".")')"
no_refs=
for path in $paths; do
escaped_path=$(echo "$path" | sed -e "s|\.|\\\\.|g")
if ! grep -r "$escaped_path" "$code_path" ; then
no_refs="$no_refs $path"
fi
done
echo "Missing paths..."
echo "$no_refs"
echo "Creating a new json file without the unused paths"
del_paths_list=
for path in $no_refs; do
del_paths_list+=".$path, "
done
del_paths_list=${del_paths_list:0:-2} # remove trailing comma space
cat "$json_input" | jq -r 'del('$del_paths_list')' > ${json_input}.new.json
Running the setup_test.sh, then we can test the jq + grep solution
$ ./check_for_dict_references.sh test.json src
json_input=test.json
code_path=src
Paths found in json
src/b.js:key1.someKey2
src/b.js:key1.someKey2
src/b.js:key1.someKey2
src/a.js:key2.someKey3.someKey4
src/a.js:key2.someKey3.someKey4
src/a.js:key2.someKey3.someKey4
Missing paths...
key2.someKey3.someKey5
Creating a new json file without the unused paths
If you look closely you would want it to also print key1.someKey, but this got "found" in the middle of the name key1.someKey2. There are some more fancy regex things you can do, but for the purpose of this script it may be enough.
Now look in your directory for the new json file:
$ cat test.json.new.json
{
"key1": {
"someKey": "Some text",
"someKey2": "Some text2"
},
"key2": {
"someKey3": {
"someKey4": "Some text3"
}
}
}
Hope that helps.

How do have multiline strings for script call in packer

I'm doing this:
{
"type": "powershell",
"script": "myscript.ps1 -param1 {{user `aaaa`}} -param2 {{user `bbbb`}} -param3 {{user `cccc`}}"
}
I'm adding more params all the time. How do I break this across multiple lines?
I don't want to use a list of env vars because I want the script to explicitly define its params.
I don't want to use hcl because that's still in beta.
Ran into this myself. Packer does support multi-line here documents. And they are serious about the "The lines of the string and the end marker must not be indented."
So a solution (in HCL) would look like:
provisioner "shell-local" {
inline = [ <<EOF
myscript.ps1 \
-param1 {{user `aaaa`}} \
-param2 {{user `bbbb`}} \
-param3 {{user `cccc`}}
EOF
]
}
I'd imagine something similar should work in powershell.

How can I prettyprint JSON on the command line, but allow invalid JSON objects to pass though?

I'm currently tailing some logs in bash that are half JSON, half text like below:
{"response":{"message":"asdfasdf"}}
{"log":{"example":"asdfasdf"}}
here is some text
{"another":{"example":"asdfasdf"}}
more text
Each line is either a full valid JSON object or some text that would fail a JSON parser.
I've looked at jq and underscore-cli to see if they have options to return the invalid object in the case of failure, but I'm not seeing any.
I've also tried to use a || operator to cat the piped input, but I'm losing the value somehow. Maybe I should read up on pipes more? Example: getLogs -t | (underscore print || cat)
I think I could write a script that stores the input. Format it, and return the output if successful. If it fails returned the stored value. I feel like there should be a simpler way though. Any thoughts?
You can use this node library
install with
$ npm install -g js-beautify
Here is what I did:
$ js-beautify -r test.js
beautified test.js
I tested it with an incomplete json file and it worked
jq can check for invalid json
#!/bin/bash
while read p; do
if jq -e . >/dev/null 2>&1 <<<"$p"; then
echo $p | jq
else
echo 'Skipping invalid json'
fi
done < /tmp/tst.txt
{
"response": {
"message": "asdfasdf"
}
}
{
"log": {
"example": "asdfasdf"
}
}
Skipping invalid json
{
"another": {
"example": "asdfasdf"
}
}
Skipping invalid json

unix command to filter the json

[
{
"name":"sandboxserver.tar.gz.part-aa",
"hash":"010d126f8ccf199f3cd5f468a90d5ae1",
"bytes":4294967296,
"last_modified":"2018-10-10T01:32:00.069000",
"content_type":"binary/octet-stream"
},
{
"name":"sandboxserver.tar.gz.part-ab",
"hash":"49a6f22068228f51488559c096aa06ce",
"bytes":397973601,
"last_modified":"2018-10-10T01:32:22.395000",
"content_type":"binary/octet-stream"
},
{
"name":"sandboxserver.tar.gz.part-ac",
"hash":"2c5e845f46357e203214592332774f4c",
"bytes":5179281858,
"last_modified":"2018-10-11T08:20:11.566000",
"content_type":"binary/octet-stream"
}
]
I am getting above JSON as response while listing the objects in cloud object storage using curl -l -X GET. How can I get the object "name" assigned to an array while looping through all the objects.
for example
array[1]="sandboxserver.tar.gz.part- aa"
array[2]="sandboxserver.tar.gz.part- ab"
array[3]="sandboxserver.tar.gz.part- ac"
You can use jq.
jq is a powerful tool that lets you read, filter, and write JSON in bash.
You might need to install it first.
Try this:
I've pasted your json into a file:
~$ cat n1.json
[
{
"name":"sandboxserver.tar.gz.part-aa",
"hash":"010d126f8ccf199f3cd5f468a90d5ae1",
"bytes":4294967296,
"last_modified":"2018-10-10T01:32:00.069000",
"content_type":"binary/octet-stream"
},
{
"name":"sandboxserver.tar.gz.part-ab",
"hash":"49a6f22068228f51488559c096aa06ce",
"bytes":397973601,
"last_modified":"2018-10-10T01:32:22.395000",
"content_type":"binary/octet-stream"
},
{
"name":"sandboxserver.tar.gz.part-ac",
"hash":"2c5e845f46357e203214592332774f4c",
"bytes":5179281858,
"last_modified":"2018-10-11T08:20:11.566000",
"content_type":"binary/octet-stream"
}
]
And then used jq to find the names:
~$ jq -r '.[].name' n1.json
sandboxserver.tar.gz.part-aa
sandboxserver.tar.gz.part-ab
sandboxserver.tar.gz.part-ac
If you don't want to depend on external utility like jq, use can use python + bash combo do the trick.
response="$(cat data.json)"
declare -a array
array=($(python -c "import json,sys; data=[arr['name'] for arr in json.loads(sys.argv[1])]; print('\n'.join(data));" "$response"))
echo "${array[#]}"
Advice: Writing embedded python code may soon become unreadable so you may want to put the python code in a separate script and run the script.