When reading some JSON data structures, and then trying to Dump them using YAML::Tiny, I sometimes get the error
YAML::Tiny does not support JSON::XS::Boolean
I understand why this is the case (in particular YAML::Tiny does not support booleans, which JSON is keen to clearly distinguish from other scalars), but is there a quick hack to turn those JSON::XS::Boolean objects into plain 0's and 1's just for quick dump-to-the-screen purposes?
YAML::Tiny doesn't support objects. Unfortunately, it doesn't even have an option to just stringify all objects, which would handle JSON::XS::Boolean.
You can do that fairly easily with a recursive function, though:
use strict;
use warnings;
use 5.010; # for say
use JSON::XS qw(decode_json);
use Scalar::Util qw(blessed reftype);
use YAML::Tiny qw(Dump);
my $hash = decode_json('{ "foo": { "bar": true }, "baz": false }');
# Stringify all objects in $hash:
sub stringify_objects {
for my $val (#_) {
next unless my $ref = reftype $val;
if (blessed $val) { $val = "$val" }
elsif ($ref eq 'ARRAY') { stringify_objects(#$val) }
elsif ($ref eq 'HASH') { stringify_objects(values %$val) }
}
}
stringify_objects($hash);
say Dump $hash;
This function doesn't bother processing scalar references, because JSON won't produce them. It also doesn't check whether an object actually has overloaded stringification.
Data::Rmap doesn't work well for this because it will only visit a particular object once, no matter how many times it appears. Since the JSON::XS::Boolean objects are singletons, that means it will only find the first true and the first false. It's possible to work around that, but it requires delving into the source code to determine how keys are generated in its seen hash:
use Data::Rmap qw(rmap_ref);
use Scalar::Util qw(blessed refaddr);
# Stringify all objects in $hash:
rmap_ref { if (blessed $_) { delete $_[0]->seen->{refaddr $_};
$_ = "$_" } } $hash;
I think the recursive function is clearer, and it's not vulnerable to changes in Data::Rmap.
Related
For some Perl diagnostic tests, I'm recording assorted bits of information formatted as JSON using JSON::MaybeXS.
I get an error when I want to record the current Perl version, which I obtain from the special variable $^V.
As the minimal demonstration script shows, the error occurs unless I quote $^V as "$^V".
json_perl_version_test.pl
#!/usr/bin/env perl
use strict;
use warnings;
use v5.18;
use JSON::MaybeXS;
say "Running Perl version $^V";
my $item = 'Wut?';
my %hash1 = (
something => $item,
v_unquoted => $^V
);
eval { say say 'Hash1: ', encode_json \%hash1 };
say "Oops - JSON encode error: $#" if $#;
my %hash2 = (
something => $item,
v_quoted => "$^V"
);
say 'Hash2: ', encode_json \%hash2;
# Running Perl version v5.34.0
# Oops - JSON encode error: encountered object 'v5.34.0',
# but neither allow_blessed, convert_blessed nor allow_tags
# settings are enabled (or TO_JSON/FREEZE method missing) at
# /Users/bw/Documents/Dev/tests/json_perl_version_test.pl line 17.
# Hash2: {"something":"Wut?","v_quoted":"v5.34.0"}
Note that it wasn't necessary to quote $item.
The error message refers to some ways to handle other cases, but seemingly not including canonical Perl version dotted-decimal strings. I've looked through the main Perl JSON modules (recent versions of JSON::MaybeXS, JSON, and Cpanel::JSON::XS), but can't find anything referring to $^V or dotted-decimal strings. Also don't find a relevant question on SO :(.
Perhaps I'm missing something? Or am I stuck with needing to quote $^V?
Reasons?
Thanks,
The $^V variable is really an object
The revision, version, and subversion of the Perl interpreter, represented as a version object.
An object cannot be stored in JSON just so. Quoting it stringifies it.
It is possible to make JSON::XS (and its Cpanel::) take a blessed reference but it involves more work. See Object Serialization. The cleanest complete solution is with convert_blessed, when the encode method will look for a TO_JSON method (in the class of the object that is to be added to JSON), which would return a JSON-ready string.
Alas, there is no such a thing for the version (nor for a few other classes I tried, like DateTime). One can add that method to the class† but just thinking of it makes quotes look nice.
Another way is to get explicit in making the version object stringify
my $json = JSON::XS->new->encode( { ver => $^V->stringify } )
This is yet more elaborate but at least now it's clear what the matter is, without magic quotes.
Or just quote it and add a comment.
† By "monkey-patching" it, for example
add the sub, with a fully qualified name
perl -Mstrict -wE'use JSON::XS; say $^V;
sub version::TO_JSON { return $_[0]->stringify };
my $json = JSON::XS->new->convert_blessed->encode( { ver => $^V } );
say $json'
Can also add a sub via string eval at runtime but that doesn't seem needed.
write the code reference to the class's symbol table at runtime
perl -wE'use JSON::XS; say $^V;
*{"version"."::"."TO_JSON"} = sub { return $_[0]->stringify };
$json = JSON::XS->new->convert_blessed->encode( { ver => $^V } );
say $json'
well, or really with strict in effect we need to allow the symbolic refs
perl -Mstrict -wE'use JSON::XS; say $^V;
NO_STRICT_REFS: {
no strict "refs";
my ($class, $method) = qw(version TO_JSON);
*{$class."::".$method} = sub { return $_[0]->stringify }
};
my $json = JSON::XS->new->convert_blessed->encode( { ver => $^V } );
say $json'
where I also added variables for class name and method, not necessary but better.
This is packaged for far nicer use in Sub::Install
use Sub::Install;
Sub::Install::install_sub({
code => sub { ... }, into => $package, as => $subname
});
There are expected defaults and a bit more in the module.
Or, of course by writing a wrapper class or some such but that's something yet else.
Blessed Perl objects can't be stored in JSON without extra steps (mentioned by the error).
print ref $^V; # version
A possible workaround:
my $j = Cpanel::JSON::XS->new->convert_blessed; # Allow stringification.
say 'Hash1: ', $j->encode(\%hash1);
All of the documentation and examples I have seen for the Perl JSON::XS module use a OO interface, e.g.
print JSON::XS->new->ascii()->pretty()->canonical()->encode($in);
But I don't necessarily want all those options every time, I'd prefer to send them in a hash like you can with the basic JSON module, e.g.
print to_json($in, { canonical => 1, pretty => 1, ascii => 1 } );
sending to that encode_json yields
Too many arguments for JSON::XS::encode_json
Is there any way to do that?
JSON's to_json uses JSON::XS if it's installed, so if you want a version of to_json that uses JSON::XS, simply use the one from JSON.
Or, you could recreate to_json.
sub to_json
my $encoder = JSON::XS->new();
if (#_ > 1) {
my $opts = $_[1];
for my $method (keys(%$opts)) {
$encoder->$_($opts->{$_});
}
}
return $encoder->encode($_[0]);
}
But doesn't help stop passing in the options every time. If you're encoding multiple data structures, it's best to create a single object and reuse it.
my $encoder = JSON::XS->new->ascii->pretty->canonical;
print $encoder->encode($in);
Run cpanm --look DBIx::Class ; cd examples/Schema/ to use the example database.
use 5.024;
use strictures;
use JSON::MaybeXS qw(encode_json);
use MyApp::Schema qw();
use Sub::Install qw();
my $s = MyApp::Schema->connect('dbi:SQLite:db/example.db');
# Yes, I know Helper::Row::ToJSON exists.
Sub::Install::install_sub({
code => sub {
my ($self) = #_;
return { map {$_ => $self->$_} keys %{ $self->columns_info } };
},
into => $s->source('Track')->result_class,
as => 'TO_JSON',
});
my ($t) = $s->resultset('Cd')->first->tracks;
say ref $t->can('TO_JSON'); # 'CODE', ok
say ref $t->TO_JSON; # 'HASH', ok
say encode_json $t;
# encountered object 'MyApp::Schema::Result::Track=HASH(0x1a53b48)',
# but neither allow_blessed, convert_blessed nor allow_tags settings
# are enabled (or TO_JSON/FREEZE method missing) at …
I expect the serialiser to find the installed hook and use it, but instead I get the error above. What's going wrong?
In order to make JSON::XS consider TO_JSON, you have to explicitly enable convert_blessed option:
my $coder = JSON::XS->new;
$coder->convert_blessed(1);
say $coder->encode($t);
According to docs:
$json = $json->convert_blessed ([$enable])
$enabled = $json->get_convert_blessed
See "OBJECT SERIALISATION" for details.
If $enable is true (or missing), then encode, upon encountering a blessed object, will check for the availability of the TO_JSON method
on the object's class. If found, it will be called in scalar context
and the resulting scalar will be encoded instead of the object.
The TO_JSON method may safely call die if it wants. If TO_JSON returns other blessed objects, those will be handled in the same way.
TO_JSON must take care of not causing an endless recursion cycle (==
crash) in this case. The name of TO_JSON was chosen because other
methods called by the Perl core (== not by the user of the object) are
usually in upper case letters and to avoid collisions with any to_json
function or method.
If $enable is false (the default), then encode will not consider this type of conversion.
This setting has no effect on decode.
(emphasis mine)
In order to create an API that's consistent for strict typing languages, I need to modify all JSON to return quoted strings in place of integers without going through one-by-one and modifying underlying data.
This is how JSON is generated now:
my $json = JSON->new->allow_nonref->allow_unknown->allow_blessed->utf8;
$output = $json->encode($hash);
What would be a good way to say, "And quote every scalar within that $hash"?
Both of JSON's backends (JSON::PP and JSON::XS) base the output type on the internal storage of the value. The solution is to stringify the non-reference scalars in your data structure.
sub recursive_inplace_stringification {
my $reftype = ref($_[0]);
if (!length($reftype)) {
$_[0] = "$_[0]" if defined($_[0]);
}
elsif ($reftype eq 'ARRAY') {
recursive_inplace_stringification($_) for #{ $_[0] };
}
elsif ($reftype eq 'HASH') {
recursive_inplace_stringification($_) for values %{ $_[0] };
}
else {
die("Unsupported reference to $reftype\n");
}
}
# Convert numbers to strings.
recursive_inplace_stringification($hash);
# Convert to JSON.
my $json = JSON->new->allow_nonref->utf8->encode($hash);
If you actually need the functionality provided by allow_unknown and allow_blessed, you will need to reimplement it inside of recursive_inplace_stringification (perhaps by copying it from JSON::PP if licensing allows), or you could use the following before calling recursive_inplace_stringification:
# Convert objects to strings.
$hash = JSON->new->allow_nonref->decode(
JSON->new->allow_nonref->allow_unknown->allow_blessed->encode(
$hash));
I am using PowerShell v3 and the Windows PowerShell ISE. I have the following function that works fine:
function Get-XmlNode([xml]$XmlDocument, [string]$NodePath, [string]$NamespaceURI = "", [string]$NodeSeparatorCharacter = '.')
{
# If a Namespace URI was not given, use the Xml document's default namespace.
if ([string]::IsNullOrEmpty($NamespaceURI)) { $NamespaceURI = $XmlDocument.DocumentElement.NamespaceURI }
# In order for SelectSingleNode() to actually work, we need to use the fully qualified node path along with an Xml Namespace Manager, so set them up.
[System.Xml.XmlNamespaceManager]$xmlNsManager = New-Object System.Xml.XmlNamespaceManager($XmlDocument.NameTable)
$xmlNsManager.AddNamespace("ns", $NamespaceURI)
[string]$fullyQualifiedNodePath = Get-FullyQualifiedXmlNodePath -NodePath $NodePath -NodeSeparatorCharacter $NodeSeparatorCharacter
# Try and get the node, then return it. Returns $null if the node was not found.
$node = $XmlDocument.SelectSingleNode($fullyQualifiedNodePath, $xmlNsManager)
return $node
}
Now, I will be creating a few similar functions, so I want to break the first 3 lines out into a new function so that I don't have to copy-paste them everywhere, so I have done this:
function Get-XmlNamespaceManager([xml]$XmlDocument, [string]$NamespaceURI = "")
{
# If a Namespace URI was not given, use the Xml document's default namespace.
if ([string]::IsNullOrEmpty($NamespaceURI)) { $NamespaceURI = $XmlDocument.DocumentElement.NamespaceURI }
# In order for SelectSingleNode() to actually work, we need to use the fully qualified node path along with an Xml Namespace Manager, so set them up.
[System.Xml.XmlNamespaceManager]$xmlNsManager = New-Object System.Xml.XmlNamespaceManager($XmlDocument.NameTable)
$xmlNsManager.AddNamespace("ns", $NamespaceURI)
return $xmlNsManager
}
function Get-XmlNode([xml]$XmlDocument, [string]$NodePath, [string]$NamespaceURI = "", [string]$NodeSeparatorCharacter = '.')
{
[System.Xml.XmlNamespaceManager]$xmlNsManager = Get-XmlNamespaceManager -XmlDocument $XmlDocument -NamespaceURI $NamespaceURI
[string]$fullyQualifiedNodePath = Get-FullyQualifiedXmlNodePath -NodePath $NodePath -NodeSeparatorCharacter $NodeSeparatorCharacter
# Try and get the node, then return it. Returns $null if the node was not found.
$node = $XmlDocument.SelectSingleNode($fullyQualifiedNodePath, $xmlNsManager)
return $node
}
The problem is that when "return $xmlNsManager" executes the following error is thrown:
Cannot convert the "System.Object[]" value of type "System.Object[]" to type "System.Xml.XmlNamespaceManager".
So even though I have explicitly cast my $xmlNsManager variables to be of type System.Xml.XmlNamespaceManager, when it gets returned from the Get-XmlNamespaceManager function PowerShell is converting it to an Object array.
If I don't explicitly cast the value returned from the Get-XmlNamespaceManager function to System.Xml.XmlNamespaceManager, then the following error is thrown from the .SelectSingleNode() function because the wrong data type is being passed into the function's 2nd parameter.
Cannot find an overload for "SelectSingleNode" and the argument count: "2".
So for some reason PowerShell is not maintaining the data type of the return variable. I would really like to get this working from a function so that I don't have to copy-paste those 3 lines all over the place. Any suggestions are appreciated. Thanks.
What's happening is PowerShell is converting your namespace manager object to a string array.
I think it has to do with PowerShell's nature of "unrolling" collections when sending objects down the pipeline. I think PowerShell will do this for any type implementing IEnumerable (has a GetEnumerator method).
As a work around you can use the comma trick to prevent this behavior and send the object as a whole collection.
function Get-XmlNamespaceManager([xml]$XmlDocument, [string]$NamespaceURI = "")
{
...
$xmlNsManager.AddNamespace("ns", $NamespaceURI)
return ,$xmlNsManager
}
More specifically, what is happening here is that your coding habit of strongly typing $fullyQualifiedModePath is trying to turn the result of the Get (which is a list of objects) into a string.
[string]$foo
will constrain the variable $foo to only be a string, no matter what came back. In this case, your type constraint is what is subtly screwing up the return and making it Object[]
Also, looking at your code, I would personally recommend you use Select-Xml (built into V2 and later), rather than do a lot of hand-coded XML unrolling. You can do namespace queries in Select-Xml with -Namespace #{x="..."}.