I am using the LWP::UserAgent module to issue a GET request to one of our APIs.
#!/usr/bin/perl
use strict;
use warning;
use LWP::UserAgent;
use Data::Dumper;
my $ua = LWP::UserAgent->new;
my $request = $ua->get("http://example.com/foo", Authorization => "Bearer abc123", Accept => "application/json" );
print Dumper $request->content;
The request is successful. Dumper returns the following JSON.
$VAR1 = '{
"apiVersion": "v1",
"data": {
"ca-bundle.crt": "-----BEGIN CERTIFICATE-----abc123-----END CERTIFICATE-----\\n"
},
"kind": "ConfigMap",
"metadata": {
"creationTimestamp": "2021-07-16T17:13:01Z",
"labels": {
"auth.openshift.io/managed-certificate-type": "ca-bundle"
},
"managedFields": [
{
"apiVersion": "v1",
"fieldsType": "FieldsV1",
"fieldsV1": {
"f:data": {
".": {},
"f:ca-bundle.crt": {}
},
"f:metadata": {
"f:labels": {
".": {},
"f:auth.openshift.io/managed-certificate-type": {}
}
}
},
"manager": "cluster-kube-apiserver-operator",
"operation": "Update",
"time": "2021-09-14T17:07:39Z"
}
],
"name": "kube-control-plane-signer-ca",
"namespace": "openshift-kube-apiserver-operator",
"resourceVersion": "65461225",
"selfLink": "/api/v1/namespaces/openshift-kube-apiserver-operator/configmaps/kube-control-plane-signer-ca",
"uid": "f9aea067-1234-5678-9101-9d4073f5ae53"
}
}';
Let's say I want to print the value of the apiVersion key, which should print v1.
print "API Version = $request->content->{'apiVersion'} \n";
The following is being printed. I am not sure how to print the value v1. Since HTTP::Response is included in the output, I suspect I might have to use the HTTP::Response module?
API Version = HTTP::Response=HASH(0x2dffe80)->content->{'apiVersion'}
Perl doesn't expand subroutine calls in a double-quoted string.
print "API Version = $request->content->{'apiVersion'} \n";
In this line of code, content() is a subroutine call. So Perl sees this as:
print "API Version = $request" . "->content->{'apiVersion'} \n";
And if you try to print most Perl objects, you'll get the hash reference along with the name of the class - hence HTTP::Response=HASH(0x2dffe80).
You might think that you just need to break up your print() statement like this:
print 'API Version = ', $request->content->{'apiVersion'}, "\n";
But that's not going to work either. $request->content doesn't return a Perl data structure, it returns a JSON-encoded string. You need to decode it into a data structure before you can access the individual elements.
use JSON;
print 'API Version = ', decode_json($request->content)->{'apiVersion'}, "\n";
But it might be cleaner to do the decoding outside of the print() statement.
use JSON;
my $data = decode_json($request->content);
In which case you can go back to something more like your original code:
print "API Version = $data->{'apiVersion'} \n";
The JSON content must be decoded first. There are several modules for that, like JSON:
use JSON;
# ...
my $href = decode_json $request->content;
And then use it like a normal hash reference: $href->{apiVersion}
Related
I have JSON code that I'm pulling with key names that are the same and I'm trying to pull the values from the keys one at a time and pass them to variables (in a loop) in a perl script but it pulls all of the values at one time instead of iterating through them. I'd like to pull a value from a key and pass it to a variable then iterate through the loop again for the next value. The amount of data changes in JSON so the amount of identical keys will grow.
Perl Script Snippet
#!/usr/bin/perl
use warnings;
use strict;
use JSON::XS;
my $res = "test.json";
my $txt = do {
local $/;
open my $fh, "<", $res or die $!;
<$fh>;
};
my $json = decode_json($txt);
for my $mdata (#{ $json->{results} }) {
my $sitedomain = "$mdata->{custom_fields}->{Domain}";
my $routerip = "$mdata->{custom_fields}->{RouterIP}";
#vars
my $domain = $sitedomain;
my $host = $routerip;
print $domain;
print $host;
}
Print $host variable
print $host;
192.168.201.1192.168.202.1192.168.203.1
Print $domain variable
print $domain;
site1.global.localsite2.global.localsite3.global.local
JSON (test.json)
{
"results": [
{
"id": 37,
"url": "http://global.local/api/dcim/sites/37/",
"display": "Site 1",
"name": "Site 1",
"slug": "site1",
"custom_fields": {
"Domain": "site1.global.local",
"RouterIP": "192.168.201.1"
}
},
{
"id": 38,
"url": "http://global.local/api/dcim/sites/38/",
"display": "Site 2",
"name": "Site 2",
"slug": "site2",
"custom_fields": {
"Domain": "site2.global.local",
"RouterIP": "192.168.202.1"
}
},
{
"id": 39,
"url": "http://global.local/api/dcim/sites/39/",
"display": "Site 3",
"name": "Site 3",
"slug": "site3",
"custom_fields": {
"Domain": "site3.global.local",
"RouterIP": "192.168.203.1"
}
}
]
}
Your code produces expected result if you add \n to print statement. You can utilize say instead of print if there is no format required.
use warnings;
use strict;
use feature 'say';
use JSON::XS;
my $res = "test.json";
my $txt = do {
local $/;
open my $fh, "<", $res or die $!;
<$fh>;
};
my $json = decode_json($txt);
for my $mdata (#{ $json->{results} }) {
my $sitedomain = "$mdata->{custom_fields}->{Domain}";
my $routerip = "$mdata->{custom_fields}->{RouterIP}";
#vars
my $domain = $sitedomain;
my $host = $routerip;
say "$domain $host";
}
The code can be re-written in shorter form as following
use strict;
use warnings;
use feature 'say';
use JSON;
my $fname = 'router_test.json';
my $txt = do {
local $/;
open my $fh, "<", $fname or die $!;
<$fh>;
};
my $json = from_json($txt);
say "$_->{custom_fields}{Domain} $_->{custom_fields}{RouterIP}" for #{$json->{results}};
It sounds like you want to "slice" the data. You could buffer in code, or collect unique values later. Let's modify what you started with, and make some tweaks:
n.b. No need to quote my $sitedomain = "$mdata->{custom_fields}->{Domain}";. The content of the JSON is already a string, and forcing Perl to make another string by interpolating it is unnecessary.
n.b.2 JSON::XS works automatically if it's installed.
my %domains;
my %ips;
for my $mdata (#{ $json->{results} }) {
my $sitedomain = $mdata->{custom_fields}->{Domain};
my $routerip = $mdata->{custom_fields}->{RouterIP};
# Collect and count all the unique domains and IPs by storing them as hash keys
$domains{$sitedomain} += 1;
$ips{$routerip} += 1;
}
for my $key (keys %domains) {
printf "%s %s\n", $key, $domains{$key};
# and so on
}
If we don't know the custom fields, we can play with nested hashes to collect it all:
my %fields;
for my $mdata (#{ $json->{results} }) {
for my $custom_field (keys %{ $mdata->{custom_fields} }) {
$fields{$custom_field}{$mdata->{custom_fields}{$custom_field}} += 1;
}
}
for my $custom_field (keys %fields) {
print "$custom_field:\n";
for my $unique_value (keys %{ $fields{$custom_field} }){
printf "%s - %s\n", $unique_value, $fields{$custom_field}{$unique_value};
}
}
Example output:
RouterIP:
192.168.201.1 - 1
192.168.203.1 - 1
192.168.202.1 - 1
Domain:
site2.global.local - 1
site1.global.local - 1
site3.global.local - 1
... or something like that. Nested structures lead very quickly to messy code. You can mitigate it by dereferencing the substructures. It could also be more predictable if we work with a known list of keys e.g.
my #known_keys = qw/RouterIP Domain/;
for my $mdata (#{ $json->{results} }) {
for my $custom_field (#known_keys) {
if (exists $fields{$custom_field}) {
$fields{$custom_field}{$mdata->{custom_fields}{$custom_field}} += 1;
}
}
}
If the JSON file is massive you may run out of memory. For this you would need to look into a package like JSON::SL or JSON::Streaming::Reader. They're more involved to use but prevent you from needing to load the whole file into memory. There are also unix tools like jq that provide the same powers.
I have a JSON file, and I want to check if keys exists or not, if keys are empty or not.
I've already done this kind of check in the script below.
But, here I have "children" which is an empty array.
How can I see if this array exists or not and if this array is empty or not?
Here the JSON sample:
{
"id": "Store::STUDIO",
"name": "Studio",
"categories": [
{
"id": "Category::556",
"name": "Cinéma",
"children": []
},
{
"id": "Category::557",
"name": "Séries",
"children": []
}
],
"images": [
{
"format": "iso",
"url": "http://archive.ubuntu.com/ubuntu/dists/bionic-updates/main/installer-amd64/current/images/netboot/mini.iso",
"withTitle": false
}
],
"type": "PLAY"
}
Here is the script:
#!/usr/bin/perl
use strict;
use warnings;
use utf8;
use JSON qw( decode_json );
use JSON qw( from_json );
# JSON file
my $json_f = '/home/test';
# Variable
my $curl_cmd = "curl -o /dev/null --silent -Iw '%{http_code}'";
# JSON text
my $json_text = do {
open (TOP, "<", $json_f);
local $/;
<TOP>
};
my $decoded = from_json($json_text);
# Display value provider if exist
my $provider = $decoded->{"categories"}[0]{"provider"};
print $provider, "\n" if scalar $provider;
# Display value children if exist
my #child = $decoded->{"categories"}[0]{"children"};
print $child[0], "\n" if scalar #child;
# Checking an url is exist
my $url_src = $decoded->{"images"}[0]{"url"};
my $http_res = qx{$curl_cmd $url_src}; # Checking if URL is correct
# Display categories with others values
my #categories = #{ $decoded->{'categories'} };
foreach my $f ( #categories ) {
print $decoded->{"id"} . "|" . $f->{"id"} . "|" . $f->{"name"} . "|" . $http_res . "\n";
}
In your code, #child is an array of arrays. Dereference the array. Change:
my #child = $decoded->{"categories"}[0]{"children"};
to:
my #child = #{ $decoded->{"categories"}[0]{"children"} };
I'm trying to grab some information out of a json export from Ping. My rusty Perl skills are failing me as I'm getting lost in the weeds with the dereferencing. Rather than bang my head against the wall some more I thought I'd post a question since all the google searches are leading here.
My understanding is that decode_json converts items into an array of hashes and each hash has strings and some other arrays of hashes as contents. This seems to bear out when attempting to get to an individual string value but only if I manually specify a specific array element. I can't figure out how to loop through the items.
The JSON comes back like this:
{
"items":[
{
#lots of values here are some examples
"type": "SP",
"contactInfo": {
"company": "Acme",
"email": "john.doe#acme.com"
}
]
}
I had no problems getting to actual values
#!/usr/bin/perl
use JSON;
use Data::Dumper;
use strict;
use warnings;
use LWP::Simple;
my $json;
{
local $/; #Enable 'slurp' mode
open my $fh, "<", "idp.json";
$json = <$fh>;
close $fh;
}
my $data = decode_json($json);
#array print $data->{'items'};
#hash print $data->{'items'}->[0];
#print $data->{'items'}->[0]->{'type'};
But, I can't figure out how to iterate through the array of items. I've tried for and foreach and various combinations of dereferencing, and it keeps telling me that the value I'm looping thru is still an array. If $data->{'items'} is an array, then presumably I should be able to do some variation of
foreach my $item ($data->{'items'})
or
my #items = $data->{'items'};
for (#items)
{
# stuff
}
But, I keep getting arrays back and I have to add in the ->[0] to get to a specific value.
$data->{'items'} is a reference to an array (of hash references). You need to dereference it, with #{ }:
use JSON;
use strict;
use warnings;
my $json;
{
local $/; #Enable 'slurp' mode
$json = <DATA>;
}
my $data = decode_json($json);
for my $item (#{ $data->{items} }) {
print "$item->{type}\n";
}
__DATA__
{
"items":[
{
"type": "SP",
"contactInfo": {
"company": "Acme",
"email": "john.doe#acme.com"
}
}
]
}
Output:
SP
This question already has answers here:
Which Perl module would you recommend for JSON manipulation?
(6 answers)
Closed 2 years ago.
how can i parse perl json object which has spaces in its keys
{
"abc" : [
"lmn" : {
"Ab Cd" : "Xy Zw",
"Ef Gh" : "Pq Rs",
}
]
}
By definition, one parses JSON using a JSON parser. There exists multiple JSON parsers on CPAN, including Cpanel::JSON::XS. It handles keys with spaces in them without issue, as should every other JSON parser.
Note that what you have isn't JSON. I'm assuming the errors are typos since you asked about JSON.
Spaces in a key will present no problems at all to any JSON parser.
There are, however, two problems in your JSON that will cause problems for any parser. Others have noted the extra comma after "Pq Rs", but you also have an array that contains a key/value pair (with the key "lnm") which needs to be inside an object.
Originally, I just removed the comma and ran this code:
#!/usr/bin/perl
use strict;
use warnings;
use feature 'say';
use Data::Dumper;
use JSON;
my $json = '{
"abc" : [
"lmn" : {
"Ab Cd" : "Xy Zw",
"Ef Gh" : "Pq Rs"
}
]
}';
my $data = decode_json($json);
say Dumper $data;
This gives an error:
, or ] expected while parsing array, at character offset 28 (before ": {\n "Ab C...")
I fixed it, by inserting { ... } around the lnm object.
#!/usr/bin/perl
use strict;
use warnings;
use feature 'say';
use Data::Dumper;
use JSON;
my $json = '{
"abc" : [ {
"lmn" : {
"Ab Cd" : "Xy Zw",
"Ef Gh" : "Pq Rs"
}
} ]
}';
my $data = decode_json($json);
say Dumper $data;
And then I got this output:
$VAR1 = {
'abc' => [
{
'lmn' => {
'Ab Cd' => 'Xy Zw',
'Ef Gh' => 'Pq Rs'
}
}
]
};
Which is, I think, what you are expecting.
I get the JSON from request:
use HTTP::Tiny;
my $response = HTTP::Tiny->new->get('https://jsonplaceholder.typicode.com/todos/1');
print "-------------------**------------------- \n";
my $content = $response->{content};
print $content->[0]->{name};
Response:
[
{
"id": 1,
"name": "Leanne Graham", "username": "Bret", "email": "Sincere#april.biz",
"address": {
"street": "Kulas Light", "suite": "Apt. 556",
"city": "Gwenborough", "zipcode": "92998-3874", "geo": { "lat": "-37.3159",
"lng": "81.1496" } }, "phone": "1-770-736-8031 x56442", "website": "hildegard.org",
"company": {
"name": "Romaguera-Crona",
"catchPhrase": "Multi-layered client-server neural-net",
"bs": "harness real-time e-markets"
}
},
{
"id": 2,
"name": "Ervin Howell",
"username": "Antonette",
"email": "Shanna#melissa.tv",
"address": {
"street": "Victor Plains",
"suite": "Suite 879",
"city": "Wisokyburgh",
"zipcode": "90566-7771",
"geo": {
"lat": "-43.9509",
"lng": "-34.4618"
}
}
]
How to read every content of the json return variable. I've tried this:
print $content->[0]->{name};
, but return nothing.
How to read through the JSON structure of perl?
Your variable contains a string that represents a data structure in the JSON format. You need to convert it to a Perl data structure in order to use traverse it in Perl. At this point, it's just a bunch of text, and HTTP::Tiny does not care what kind of data it returns.
Core Perl brings the JSON::PP module starting from version 5.13.9 (with the 5.14 release).
use strict;
use warnings;
use JSON::PP 'decode_json';
use Data::Printer;
my $json = qq({ "foo" : "bar" });
my $decoded = decode_json($json);
p $decoded;
print $decoded->{foo};
This will output:
\ {
foo "bar"
}
bar
If you've got a newer Perl and have some other stuff installed, you probably also have JSON::MaybeXS, which will pick the fastest JSON parser available automatically.
Now if you wanted your user agent to know how to do this in multiple places, you can easily create a sub class. I've done a rudimentary implementation here. Save it in a new file HTTP/Tiny/DecodeJSON.pm in the right folder. I would place it under lib in your script's directory.
.
├── lib
│ └── HTTP
│ └── Tiny
│ └── DecodeJSON.pm
└── script.pl
I would also suggest adding extensive error handling.
package HTTP::Tiny::DecodeJSON;
use strict;
use warnings;
use JSON::PP 'decode_json';
use parent 'HTTP::Tiny';
# we need this to not throw a warning in HTTP::Tiny::_agent()
use constant VERSION => '0.01';
sub get_json {
my $self = shift;
my $res = $self->get(#_);
# add error handling here ...
return decode_json $res->{content};
}
1;
You can then reuse it wherever you like. To use it in your script, you need to add the lib directory to the list of directories that Perl looks for it's modules.
use strict;
use warnings;
use Data::Printer;
use lib 'lib';
use HTTP::Tiny::DecodeJSON;
my $decoded = HTTP::Tiny::DecodeJSON->new->get_json(
'https://jsonplaceholder.typicode.com/todos/1'
);
p $decoded;
simbabque has explained a lot,
and it is useful to have an example of subclassing HTTP::Tiny. I would add the following
I believe that Cpanel::JSON::XS, despite its convoluted name, is the superior JSON module on CPAN
There is no $content->[0]->{name} element in the data returned from that URL, although I imagine that is because you are working on it. Thank you for posting a usable data source: it makes questions so much more pleasant to answer
It's pretty much essential to check whether the HTTP request has succeeded, and die with an explanatory message if there was a problem. It's just an extra statement
die $response->{reason} unless $response->{success};
Here's how I would write your code. Instead of selecting the field as you do I have used Data::Dump
to display the contents of the structure
use strict;
use warnings 'all';
use HTTP::Tiny;
use Cpanel::JSON::XS 'decode_json';
my $response = HTTP::Tiny->new->get('https://jsonplaceholder.typicode.com/todos/1');
die $response->{reason} unless $response->{success};
my $data = decode_json $response->{content};
use Data::Dump;
dd $data;
output
{
completed => bless(do{\(my $o = 0)}, "JSON::PP::Boolean"),
id => 1,
title => "delectus aut autem",
userId => 1,
}
As you can see, $content->[0]->{name} would never work because the data is a hash rather than an array, and there is no hash key name anywhere. But the Latin is a strong indicator that the server has been updated since your question so this is not a problem
The value $data->{completed} is boolean, and should probably be tested with
if ( $data->{completed} ) { ... }
to decide what to do with the response