Getting individual key values from Perl JSON one at a time - json

Question

I have JSON code that I'm pulling with key names that are the same and I'm trying to pull the values from the keys one at a time and pass them to variables (in a loop) in a perl script but it pulls all of the values at one time instead of iterating through them. I'd like to pull a value from a key and pass it to a variable then iterate through the loop again for the next value. The amount of data changes in JSON so the amount of identical keys will grow.
Perl Script Snippet
#!/usr/bin/perl
use warnings;
use strict;
use JSON::XS;
my $res = "test.json";
my $txt = do {
local $/;
open my $fh, "<", $res or die $!;
<$fh>;
};
my $json = decode_json($txt);
for my $mdata (#{ $json->{results} }) {
my $sitedomain = "$mdata->{custom_fields}->{Domain}";
my $routerip = "$mdata->{custom_fields}->{RouterIP}";
#vars
my $domain = $sitedomain;
my $host = $routerip;
print $domain;
print $host;
}
Print $host variable
print $host;
192.168.201.1192.168.202.1192.168.203.1
Print $domain variable
print $domain;
site1.global.localsite2.global.localsite3.global.local
JSON (test.json)
{
"results": [
{
"id": 37,
"url": "http://global.local/api/dcim/sites/37/",
"display": "Site 1",
"name": "Site 1",
"slug": "site1",
"custom_fields": {
"Domain": "site1.global.local",
"RouterIP": "192.168.201.1"
}
},
{
"id": 38,
"url": "http://global.local/api/dcim/sites/38/",
"display": "Site 2",
"name": "Site 2",
"slug": "site2",
"custom_fields": {
"Domain": "site2.global.local",
"RouterIP": "192.168.202.1"
}
},
{
"id": 39,
"url": "http://global.local/api/dcim/sites/39/",
"display": "Site 3",
"name": "Site 3",
"slug": "site3",
"custom_fields": {
"Domain": "site3.global.local",
"RouterIP": "192.168.203.1"
}
}
]
}

Answer 1

Your code produces expected result if you add \n to print statement. You can utilize say instead of print if there is no format required.
use warnings;
use strict;
use feature 'say';
use JSON::XS;
my $res = "test.json";
my $txt = do {
local $/;
open my $fh, "<", $res or die $!;
<$fh>;
};
my $json = decode_json($txt);
for my $mdata (#{ $json->{results} }) {
my $sitedomain = "$mdata->{custom_fields}->{Domain}";
my $routerip = "$mdata->{custom_fields}->{RouterIP}";
#vars
my $domain = $sitedomain;
my $host = $routerip;
say "$domain $host";
}
The code can be re-written in shorter form as following
use strict;
use warnings;
use feature 'say';
use JSON;
my $fname = 'router_test.json';
my $txt = do {
local $/;
open my $fh, "<", $fname or die $!;
<$fh>;
};
my $json = from_json($txt);
say "$_->{custom_fields}{Domain} $_->{custom_fields}{RouterIP}" for #{$json->{results}};

Answer 2

It sounds like you want to "slice" the data. You could buffer in code, or collect unique values later. Let's modify what you started with, and make some tweaks:
n.b. No need to quote my $sitedomain = "$mdata->{custom_fields}->{Domain}";. The content of the JSON is already a string, and forcing Perl to make another string by interpolating it is unnecessary.
n.b.2 JSON::XS works automatically if it's installed.
my %domains;
my %ips;
for my $mdata (#{ $json->{results} }) {
my $sitedomain = $mdata->{custom_fields}->{Domain};
my $routerip = $mdata->{custom_fields}->{RouterIP};
# Collect and count all the unique domains and IPs by storing them as hash keys
$domains{$sitedomain} += 1;
$ips{$routerip} += 1;
}
for my $key (keys %domains) {
printf "%s %s\n", $key, $domains{$key};
# and so on
}
If we don't know the custom fields, we can play with nested hashes to collect it all:
my %fields;
for my $mdata (#{ $json->{results} }) {
for my $custom_field (keys %{ $mdata->{custom_fields} }) {
$fields{$custom_field}{$mdata->{custom_fields}{$custom_field}} += 1;
}
}
for my $custom_field (keys %fields) {
print "$custom_field:\n";
for my $unique_value (keys %{ $fields{$custom_field} }){
printf "%s - %s\n", $unique_value, $fields{$custom_field}{$unique_value};
}
}
Example output:
RouterIP:
192.168.201.1 - 1
192.168.203.1 - 1
192.168.202.1 - 1
Domain:
site2.global.local - 1
site1.global.local - 1
site3.global.local - 1
... or something like that. Nested structures lead very quickly to messy code. You can mitigate it by dereferencing the substructures. It could also be more predictable if we work with a known list of keys e.g.
my #known_keys = qw/RouterIP Domain/;
for my $mdata (#{ $json->{results} }) {
for my $custom_field (#known_keys) {
if (exists $fields{$custom_field}) {
$fields{$custom_field}{$mdata->{custom_fields}{$custom_field}} += 1;
}
}
}
If the JSON file is massive you may run out of memory. For this you would need to look into a package like JSON::SL or JSON::Streaming::Reader. They're more involved to use but prevent you from needing to load the whole file into memory. There are also unix tools like jq that provide the same powers.