I'm relatively new to Perl and trying to self teach. However i have read all of the related threads on this page and others and none of them seem to work for me.
Below is my code - trying to get a lot of data from a webpage in Perl format and export it to update values in an SQL table.
Currently i can't even data dumper the results of the url out.
Any help would be great.
#!/usr/bin/perl
#
use LWP::Simple;
use warnings;
use strict;
use JSON qw( decode_json from_json );
use LWP::Simple;
use Data::Dumper;
use utf8;
my $url = "http://.sensitivedata.txt";
my #json= from_json(get ( $url ));
die "Couldn't get $url" if not defined #json;
##my $decoded_json = decode_json( #json);
print Dumper #json;
exit 0;
This is the error message it is giving me:
defined(#array) is deprecated at alarms.pl line 14.
(Maybe you should just omit the defined()?)
malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "(end of string)") at /opt/csw/share/perl/csw/JSON.pm line 168
The error message is pretty clear about a) what the problem is and b) how to get rid of it.
defined(#array) is deprecated at alarms.pl line 14. (Maybe you should
just omit the defined()?)
Calling defined() on #json is pointless. You're really just checking to see if there is any data in the array so replace if not defined #json with if not #json.
That will get rid of the error message. But you'll still have a problem as your program will almost certainly now die on the same line with the error message "Couldn't get http://.sensitivedata.txt". And that's probably not an accurate error message.
The problem is that this error can be caused by two problems. Either you can't get the data or you can't parse the data. Your error message only mentions one of these possibilities. Better to split the error checking into two.
# Step 1: Get the data
my $raw_json = get($url);
die "Can't get data from $url" unless $raw_json;
# Step 2: Parse the data
my #json = from_json($raw_json);
if (!#json) {
warn $raw_json;
die "Can't parse data from $url";
}
With code more like this, you'll be able to see what the problem is.
There's another little problem here, so to pre-empt your next question...
from_json always returns a scalar. It will either be a hash reference or an array reference (depending on the JSON you get). Looks like you're expecting an array. You'll need to store the reference in a scalar and dereference it.
my $json_array_ref = from_json($raw_json);
if (!#$json_array_ref) {
warn $raw_json;
die "Can't parse data from $url";
}
my #json = #$json_array_ref;
Related
string s = "%7BparentAsin%3Aasin_1%2C+businessType%3A+%22AHS%22%2CrenderType%3ARenderAll%2Cconstraints%3A%5B%7Btype%3A+Delete%2CmutuallyInclusive%3Afalse%7D%5D%7D"
I want this to be converted into a JSON in Mason Language. (Mason is very similar to perl).
I am doing this and it is working partly:
URI::Escape::uri_unescape($ItemAssociationGroupData)
This is returning:
{parentAsin:asin_1,+businessType:+"AHS",renderType:RenderAll,constraints:[{type:+Delete,mutuallyInclusive:false}]}
Here I dont want the "+" signs and the final output should be a Json and not a String. Like this can be done online on this tool, but I want to do same in code.
https://www.url-encode-decode.com/
I have tried: JSON::XS::to_json && HTML::Entities.. n all but they are not working and returning undef values.
Any help here is appreciated
Just replace the + with spaces.
uri_unescape( $ItemAssociationGroupData =~ s/\+/ /rg )
That produces
{parentAsin:asin_1, businessType: "AHS",renderType:RenderAll,constraints:[{type: Delete,mutuallyInclusive:false}]}
But that string isn't JSON. The keys of objects must be string literals in JSON, and string literals must be quoted.
Cpanel::JSON::XS's allow_barekey option will make it accept unquoted keys, but no JSON parser is going to accept the other unquoted string literals (asin_1, RenderAll, Delete). Not even JavaScript would accept that.
I don't know where you're getting that string from, but it's not really very close to JSON.
!/usr/bin/perl
use strict;
use warnings;
use feature 'say';
use JSON;
use URI::Escape;
use Data::Dumper;
my $str = '%7BparentAsin%3Aasin_1%2C+businessType%3A+%22AHS%22%2CrenderType%3ARenderAll%2Cconstraints%3A%5B%7Btype%3A+Delete%2CmutuallyInclusive%3Afalse%7D%5D%7D';
my $json = uri_unescape($str);
say $json;
say Dumper decode_json($json);
We get this output:
{parentAsin:asin_1,+businessType:+"AHS",renderType:RenderAll,constraints:[{type:+Delete,mutuallyInclusive:false}]}
And then this error:
'"' expected, at character offset 1 (before "parentAsin:asin_1,+b...") at json_decode line 21.
That's caused by the keys in your objects not being in quoted strings. Ok, we can fix that. We'll also replace the '+' signs with spaces.
#!/usr/bin/perl
use strict;
use warnings;
use feature 'say';
use JSON;
use URI::Escape;
use Data::Dumper;
my $str = '%7BparentAsin%3Aasin_1%2C+businessType%3A+%22AHS%22%2CrenderType%3ARenderAll%2Cconstraints%3A%5B%7Btype%3A+Delete%2CmutuallyInclusive%3Afalse%7D%5D%7D';
# ADDED THIS LINE
$str =~ s/\+/ /g;
my $json = uri_unescape($str);
# ADDED THIS LINE
$json =~ s/(\w+?):/"$1":/g;
say $json;
say Dumper decode_json($json);
Now we get better output:
{"parentAsin":asin_1, "businessType": "AHS","renderType":RenderAll,"constraints":[{"type": Delete,"mutuallyInclusive":false}]}
But we still get an error:
malformed JSON string, neither tag, array, object, number, string or atom, at character offset 14 (before "asin_1,+"businessTyp...") at json_decode line 21.
This is because your values also need to be quoted strings. But fixing this is harder because some of your values are already quoted (e.g. "AHS") and some values don't need to be quoted (e.g. false).
So it's hard to know the best approach to take from here. My first instinct would be to go back to whatever is generating that original string and see if you can get the bugs fixed so you get a proper JSON string.
Below listed program fails with the following error:
JSON text must be an object or array (but found number, string, true, false or null, use allow_nonref to allow this) at json_test.pl line 10.
Works fine when I comment out thread startup/join, or when JSON is parsed before thread is run.
Message seems to be coming from JSON library, so I suppose something is wrong with it.
Any ideas what's going on and how to fix it?
# json_test.pl
use strict;
use warnings;
use threads;
use JSON;
use Data::Dumper;
my $t = threads->new(\&DoSomething);
my $str = '{"category":"dummy"}';
my $json = JSON->new();
my $data = $json->decode($str);
print Dumper($data);
$t->join();
sub DoSomething
{
sleep 10;
return 1;
}
JSON uses JSON::XS if installed which is not compatible with Perl threads (please don't take the author's words at face value - threads are discouraged and difficult to use effectively, but not deprecated and there are no plans to remove them). The community-preferred fork Cpanel::JSON::XS is thread safe and will be used by JSON::MaybeXS by default, which is a mostly drop-in replacement for JSON.
I have written Perl code that was working until recently, when I tried to run it again. The problem seems to originate from the JSON::XS "decode_json" method.
Code Snippet:
use warnings;
use strict;
use MooseX::Singleton;
use Array::Utils qw(:all);
use Data::Dumper;
use JSON::XS qw(encode_json decode_json);
use Storable;
use Tie::IxHash;
open (my $observations_fh, '<', 'observations.json') or die "Could not open observations.json\n";
my $observations_json = <$obserations_fh>;
my #decoded_observations = #{decode_json($observations_json)};
Usually, after this code I was able to go through each JSON component in a for loop and take specific information, but now I get the error:
, or ] expected while parsing array, at character offset 5144816
(before "(end of string)")
I saw a similar question here, but it didn't resolve my problem.
I also have similar json decoding going on that doesn't utilize #{decode_json($variable)}, but when I tried that with this observations.json file, the same error was output.
I also tried just using the JSON module, but same error occurred.
Any insight would be greatly appreciated!
-cookersjs
That probably indicates you have incomplete JSON in $observations_json. Your assumption that the entire file consists of just one line is probably incorrect. Use
my $observations;
{
open (my $observations_fh, '<', 'observations.json')
or die("Can't open observations.json: $!\n");
local $/;
my $observations_json = <$obserations_fh>;
$observations = decode_json($observations_json);
}
If that doesn't help, observations.json doesn't contain valid JSON.
I have written a perl script which accessed JIRA REST API to GET a list of issues that match a specific JQL query. Sometimes thee results are only one issue and other times I get many back.
$client->GET(
$apiPath.$jql.$fieldRes,
$headers);
#a perl hash of results
my $response = from_json($client->responseContent());
while $response is a perl hash, if I try to drill down into the hash I hit an issue.
There is an array of "issues" within the hash.
I am trying to pull data with "foreach" for each specific issue but I keep getting errors:
foreach my $issues ($response->{'issues'})
{
print STDERR Dumper($issues->{'key'});
}
Error...
Pseudo-hashes are deprecated at script.pl line #.
Argument "JIRA-10011" isn't numeric in hash element at script.pl line #.
Bad index while coercing array into hash at script.pl line #.
Any help is appreciated
There is an array of "issues" within the hash.
You cannot put arrays into hashes in Perl, this is only possible for array references. So you need to dereference it when iterating the hash(ref) with your foreach.
foreach my $issues ( #{ $response->{'issues'} } )
{
print STDERR Dumper($issues->{'key'});
}
Since you will get one issue per iteration, you should rename $issues to $issue so you won't get confused later.
I'm using Perl's DBI module. I prepare a statement using placeholders, then execute the query.
Is it possible to print out the final query that was executed without manually escaping the parameters and dropping them into the placeholders?
Thanks
See Tracing in DBI. The following works using DBD::SQLite but produces a lot of output:
$dbh->trace($dbh->parse_trace_flags('SQL|1|test'));
Output:
<- prepare('SELECT ... FROM ... WHERE ... = ?')= DBI::st=HASH(0x21ee924) at booklet-excel.pl line 213
<- execute('Inhaler')= '0E0' at booklet-excel.pl line 215
etc etc.
You could plug your own filter in to the trace stream to only keep prepares.
You can do a debug print of a prepared statement using the Statement attribute. This can be accessed either with a "statement handle" or a "database handle".
print $sth->{Statement} # with a statement handle
print $dbh->{Statement} # with a database handle
Not in general, because DBI doesn't necessarily produce such a query. If your database supports prepared statements and placeholders in its API, DBI will pass them through and let the database do the work, which is one of the reasons to use prepared statements.
This works for DBD::mysql with server-side prepare disabled (the default):
$ DBI_TRACE=2 perl your-script-here
It will print each statement twice, once before binding parameters and once after. The latter will be well-formed SQL that you can run yourself.
There is also a module, DBI::Log, which only prints SQL statements (no other debug noise), and optional timing information and caller stacktraces. It's really useful.
If you don't want to create your own tracer module (as suggested by Sinan), you are better off just trying to print the argument hash before it is passed to $sth->execute(). This is especially true, since the "Trace" functionality is DBMS dependent and $sth->{Statement} only returns the SQL placeholder statement. Here's what I did.
...
while (my $row = $csv->getline_hr($fh)) {
my $cval = "";
my $tquery = $query;
foreach my $j (#cols) {
$cval = $row->{$j};
$tquery =~ s/\?/\'$cval\'/;
}
print "$tquery\n\n";
$rc = $sth->execute(#{$row}{#cols});
}
Where I have used Text::CSV...
NOTE: This is not exact, due to DBMS implementation dependent handling of {'}s.
As masto says in general the placeholders in the SQL are not directly replaced with your parameters. The whole point of parameterized SQL is the SQL with placeholders is passed to the database engine to parse once and then it just receives the parameters.
As idssl notes you can obtain the SQL back from the statement or connection handle and you can also retrieve the parameters from ParamValues. If you don't want to do this yourself you can use something like DBIx::Log4perl to log just the SQL and parameters. See DBIX_L4P_LOG_DELAYBINDPARAM which outputs something like this:
DEBUG - prepare(0.1): 'insert into mje values(?,?)'
DEBUG - $execute(0.1) = [{':p1' => 1,':p2' => 'fred'},undef];
Of course as it uses Log::Log4perl you can omit the "DEBUG - " if you want. There is a small tutorial for using DBIx::Log4perl here.
You should be able to use DBIx::Log4perl with any DBD and if you cannot for some reason RT it and I will look at it.
If you don't want to go with DBIx::Log4perl and the DBI trace options don't suit your needs you can write callbacks for DBI's prepare/select*/execute methods and gather whatever you like in them.
For the majority of queries, the simplest debugging is to use the following...
If you prepare and execute a single statement using do method, use:
use feature 'say';
say $dbh->{Statement};
If you use prepare and execute methods separately, use:
use feature 'say';
use Data::Dumper;
$Data::Dumper::Sortkeys = 1;
say $sth->{Statement};
say Dumper($sth->{ParamValues});
For perl neophytes, my solution, copied from not2qubit and simplified/hopefully made a bit more generic/reuseable:
sub dump_query {
my $tquery = shift;
my #args = shift;
my $j;
foreach my $j (#args) { $tquery =~ s/\?/\'$j\'/; }
print STDERR "$tquery\n\n";
}