i'm new in perl (i don't know almost nothing :-))
I had a script which worked on my local computer but not in server :-(
It looks like:
my $json = JSON->new;
my $json_map = [];
for (my $i = 0; $i <= $#commitlist; $i++) {
my %co = %{$commitlist[$i]};
...
push $json_map, {esc_html($co{'id'})=> {author=>esc_html($co{'author'}),pubDate=>$cd{'rfc2822'},link=>$co_url, title=>esc_html($co{'title'})}};
}
my $output = $json->encode($json_map);
print $output . "\n";
It worked and result was like this:
[{"id":{"author":"johny","title":"some title","link":"http://127.0.0.1","pubDate":"Fri, 14 Mar 2014 12:31:17 +0000"}}]
But now I have on server following problem (there is perl 5.8.8 version, but i would like to fix it in the script):
Type of arg 1 to push must be array (not private variable) at XX line XX, near "};"
When you do a push you need to de-reference the first argument (i.e. the array you're pushing things into). Like this:
push #$json_map, ...
Hope this helps.
Related
Please tell me such a thing. There is a self-written site, the code works on it, when I transfer the same code to the yii2 console application, it will get an error converting an array to a string
I will describe in more detail what is happening.
There is a text file inside which the following.
5|02.10.2020 20:20|Name 1
5|02.10.2020 22:25|Name 2
Day of the week, date and time, program name. It is necessary to add this case to the database.
$data = iconv("windows-1251", "UTF-8", file_get_contents( './import/' . $filename . '.txt' ) );
$tv = explode("\n", $data);
foreach ($tv as $line)
{
$ex1 = explode("|",$line);
$ex2 = explode(" ",$ex1[1]);
... next insert to db $ex1[0] - day of week, $ex2[0] - date, $ex2[1] - time, $ex1[2] - name
}
In a regular .php file, all this works, and when the same code is in yii2, it swears at conversion. Why is that? How to do it right?
It turns out that it cannot use $ ex1 [0], $ ex2 [0] and so on as a string.
so I have to update a perl script that goes through a json file, fetches keys called “items”, and transforms these items into perl output.
I’m a noob at Perl/coding in general, so plz bear with me🥺. The offset variable is set as each url is iterated through. A curl command is passed to the terminal, the file is put through a "#lines" array, and in the end, whatever json data is stored in $data gets decoded and transformed. and in the blocks below (where # populate %manager_to_directs, # populate %user_to_management_chain, and # populate %manager_to_followers are commented) is where fetch_json gets called and where the hash variables get the data from the decoded json. (***Please feel free to correct me if I interpreted this code incorrectly)
There’s been a problem where the $cmd doesn’t account for the HTTP Responses every time this program is executed. I only want the results to be processed if and only if the program gets http 200 (OK) or http 204 (NO_CONTENT) because the program will run and sometimes partially refresh our json endpoint (url in curl command output from terminal below), or sometimes doesn’t even refresh at all.
All I’m assuming is that I’d probably have to import the HTTP::Response pragma and somehow pull that out of the commands being run in fetch_json, but I have no other clue where to go from there.
Would I have to update the $cmd to pull the http code? And if so, how would I interpret that in the fetch_json sub to exit the process if anything other than 200 or 204 is received?
Oh and also, how would I save the previous output from the last execution in a backup file?
Any help I can get here would be highly appreciated!
See code below:
Pulling this from a test run:
curl -o filename -w "HTTP CODE: %{http_code}\n" --insecure --key <YOUR KEY> --cert <YOUR CERT> https://xxxxxxxxxx-xxxxxx-xxxx.xxx.xxxxxxxxxx.com:443/api/v1/reports/active/week > http.out
#!/usr/bin/env perl
use warnings;
use strict;
use JSON qw(decode_json);
use autodie qw(open close chmod unlink);
use File::Basename;
use File::Path qw(make_path rmtree);
use Cwd qw(abs_path);
use Data::Dumper;
use feature qw(state);
sub get_fetched_dir {
return "$ENV{HOME}/tmp/mule_user_fetched";
}
# fetch from mulesoft server and save local copy
sub fetch_json {
state $now = time();
my ($url) = #_;
my $dir = get_fetched_dir();
if (!-e $dir) {
make_path($dir);
chmod 0700, $dir;
}
my ($offset) = $url =~ m{offset=(\d+)};
if (!defined $offset) {
$offset = 0;
}
$offset = sprintf ("%03d", $offset);
my $filename = "$dir/offset${offset}.json";
print "$filename\n";
my #fields = stat $filename;
my $size = $fields[7];
my $mtime = $fields[9];
if (!$size || !$mtime || $now-$mtime > 24*60*60) {
my $cmd = qq(curl \\
--insecure \\
--silent \\
--key $ENV{KEY} \\
--cert $ENV{CERT} \\
$url > $filename
);
#print $cmd;
system($cmd);
chmod 0700, $filename;
}
open my $fh, "<", $filename;
my #lines = <$fh>;
close $fh;
return undef if !#lines;
my $data;
eval {
$data = decode_json (join('',#lines));
};
if ($#) {
unlink $filename;
print "Bad JSON detected in $filename.\n";
print "I have deleted $filename.\n";
print "Please re-run script.\n";
exit(1);
}
return $data;
}
die "Usage:\n KEY=key_file CERT=cert_file mule_to_jira.pl\n"
if !defined $ENV{KEY} || !defined $ENV{CERT};
print "fetching data from mulesoft\n";
# populate %manager_to_directs
my %manager_to_directs;
my %user_to_manager;
my #users;
my $url = "https://enterprise-worker-data.eip.vzbuilders.com/api/v1/reports/active/week";
while ($url && $url ne "Null") {
my $data = fetch_json($url);
last if !defined $data;
$url = $data->{next};
#print $url;
my $items = $data->{items};
foreach my $item (#$items) {
my $shortId = $item->{shortId};
my $manager = $item->{organization}{manager};
push #users, $shortId;
next if !$manager;
$user_to_manager{$shortId} = $manager;
push #{$manager_to_directs{$manager}}, $shortId;
}
}
# populate %user_to_management_chain
# populate %manager_to_followers
my %user_to_management_chain;
my %manager_to_followers;
foreach my $user (keys %user_to_manager) {
my $manager = $user_to_manager{$user};
my $prev = $user;
while ($manager && $prev ne $manager) {
push #{$manager_to_followers{$manager}}, $user;
push #{$user_to_management_chain{$user}}, $manager;
$prev = $manager;
$manager = $user_to_manager{$manager}; # manager's manager
}
}
# write backyard.txt
open my $backyard_fh, ">", "backyard.txt";
foreach my $user (sort keys %user_to_management_chain) {
my $chain = join ',', #{$user_to_management_chain{$user}};
print $backyard_fh "$user:$chain\n";
}
close $backyard_fh;
# write teams.txt
open my $team_fh, ">", "teams.txt";
foreach my $user (sort #users) {
my $followers = $manager_to_followers{$user};
my $followers_joined = $followers ? join (',', sort #$followers) : "";
print $team_fh "$user:$followers_joined\n";
}
close $team_fh;
my $dir = get_fetched_dir();
rmtree $dir, {safe => 1};
So, if you want to keep the web fetch and the Perl processing decoupled, you can modify the curl command so that it includes the response header in the output by adding the -i option. That means that the Perl will have to be modified to read and process the headers before getting to the body. A successful http.out will look something like this:
HTTP/1.1 200 OK
Server: somedomain.com
Date: <date retrieved>
Content-Type: application/json; charset=utf-8
Content-Length: <size of JSON>
Status: 200 OK
Maybe: More Headers
Blank: Line signals start of body
{
JSON object here
}
An unsuccessful curl will have something other than 200 OK on the first line next to the HTTP/1.1, so you can tell that something went wrong.
Alternatively, you can let the Perl do the actual HTTP fetch instead of relying on curl; you can use LWP::UserAgent or any of a number of other HTTP client libraries in Perl, which will give you the entire response, not just the body.
I am using Perl to read in variables from a json file and handle them accordingly. The spot I need help with is when I read a time in from the file that could look like the following:
"StartTime":"2015-07-08T03:38:08Z",
"EndTime":"2015-07-10T03:38:08Z"
This is easy to handle, however here is the tricky part:
"StartTime":"now-10",
"EndTime":"now+10"
I have a function which gets these variables from the json file and checks if the string contains the word "now". But after that, I'm not sure what to do. I'm trying to convert "now" to localtime(time), but it's getting ugly fast. Here is my code:
my $_StartTime = getFromJson("StartTime");
my $_EndTime = getFromJson("EndTime");
if($_StartTime =~ /now/) {
(my $sec, my $min, my $hour, my $mday, my $mon, my $year, my $wday, my $yday, my $isdst) = localtime(time);
my $now = sprintf("%04d-%02d-%02dT%02d:%02d:%02dZ", $year+1900, $mon+1, $mday, $hour, $min, $sec);
}
# end time is handled the same way
Am I on the right track? And if so, how can I add the "+/-10" after the "now" in the file? (Note: assume the +/-10 always refers to hours)
There are lots of good modules on the CPAN that could help in this instance. You don't need to use them but it's worth knowing about them nonetheless.
Firstly, JSON might make your life easier when parsing the JSON files as it has easy methods for converting the JSON into native Perl structures.
Secondly, the DateTime family of modules might make it easier to parse and manipulate the dates. Specifically, instead of using sprintf, you could use DateTime::Format::ISO8601 to parse the date:
my $dt = DateTime::Format::ISO8601->parse_datetime( $_StartTime );
DateTime has methods for accessing the day, year, month and so on. These are documented on the main module page.
You could then keep your special case for the now input and do something like:
# work out if it's addition or subtraction and grab the amount
# then use the appropriate DateTime function:
my $dt = DateTime->now()->add( seconds => 10 );
# or
my $dt = DateTime->now()->subtract( seconds => 10 );
Using POSIX::strftime will make your life easier.
use POSIX 'strftime';
my #test_times = qw[now+10 now now-10];
foreach my $start_time (#test_times) {
if (my ($adjust) = $start_time =~ /^now([-+]\d+)?/) {
$adjust //= 0;
$adjust *= 60 * 60; # Convert hours to seconds
my $time = strftime '%Y-%m-%dT%H:%M:%SZ', gmtime(time + $adjust);
say $time;
}
}
Thinking about it further, I think I'd prefer to use Time::Piece. The principle is almost identical.
use Time::Piece;
my #test_times = qw[now+10 now now-10];
foreach my $start_time (#test_times) {
if (my ($adjust) = $start_time =~ /^now([-+]\d+)?/) {
$adjust //= 0;
$adjust *= 60 * 60; # Convert hours to seconds
my $time = gmtime(time + $adjust);
say $time->strftime('%Y-%m-%dT%H:%M:%SZ');
}
}
I would change this to:
my $_StartTime = getFromJson("StartTime");
my $_EndTime = getFromJson("EndTime");
if($_StartTime =~ s/now//) {
my $time = time;
if ($_StartTime =~ /^([-+]?)([0-9]+)/) {
my ($sign, $number) = ($1, $2);
$time += ($sign eq '-' ? -1 : 1) * $number * 3_600;
}
(my $sec, my $min, my $hour, my $mday, my $mon, my $year, my $wday, my $yday, my $isdst) = localtime($rime);
$_StartTime = sprintf("%04d-%02d-%02dT%02d:%02d:%02dZ", $year+1900, $mon+1, $mday, $hour, $min, $sec);
}
You give little information about the format of the original data, and what result you want from this. I assume the code you show is to convert the times formatted with now to one that you recognize so that you can go on from there. But it's best to handle both formats in one place to generate the same final result regardless of the input
This program uses an imaginary JSON data structure and processes all elements inside it. The core is the use of the Time::Piece module, which will parse and format times for you and do date/time arithmetic
I have encapsulated the code that processes both sorts of time values in a subroutine convert_time which returns a Time::Piece object. The code just uses the module's own stringify method to make the value readable, but you can generate any form of string you want using the object's methods
use strict;
use warnings 'all';
use feature 'say';
use JSON 'from_json';
use Time::Piece;
use Time::Seconds 'ONE_HOUR';
my $json = <<END;
[
{
"StartTime": "2015-07-08T03:38:08Z",
"EndTime": "2015-07-10T03:38:08Z"
},
{
"StartTime": "now-10",
"EndTime": "now+10"
}
]
END
my $data = from_json($json);
for my $item ( #$data ) {
for my $key ( keys %$item ) {
my $time = $item->{$key};
say "$key $time";
my $ans = convert_time($time);
print $ans, "\n\n";
}
}
sub convert_time {
my ($time) = #_;
if ( $time =~ /now([+-]\d+)/ ) {
return localtime() + $1 * ONE_HOUR;
}
else {
return Time::Piece->strptime($time, '%Y-%m-%dT%H:%M:%SZ');
}
}
output
StartTime 2015-07-08T03:38:08Z
Wed Jul 8 03:38:08 2015
EndTime 2015-07-10T03:38:08Z
Fri Jul 10 03:38:08 2015
StartTime now-10
Wed Jan 6 05:57:04 2016
EndTime now+10
Thu Jan 7 01:57:04 2016
I have a simple Perl script and I want to remove everything up to the word "city". Or remove everything up to the nth occurrence (the 2nd in my particular case) of the comma's " , ". Here's what is looks like below.
#!/usr/bin/perl
use warnings;
use strict;
my $CMD = `curl http://ip-api.com/json/8.8.8.8`;
chomp($CMD);
my $find = "^[^city]*city";
$CMD =~ s/$find//;
print $CMD;
The output is this:
{"as":"AS15169 Google Inc.","city":"Mountain View","country":"United States","countryCode":"US","isp":"Google","lat" :37.386,"lon":-122.0838,"org":"Google","query":"8.8.8.8","region":"CA","regionName":"California","status":"success","timezone":"America/Los_Angeles","zip":"94035"}
So i want do drop
" {"as":"AS15169 Google Inc.","
or drop up to
{"as":"AS15169 Google Inc.","city":"Mountain View",
EDIT:
I see I was doing far too much when matching the string. I simplified the fix for my problem with removing all before "city". My $find has been changed to
my $find = ".*city";
While I also changed the replace function like so,
$CMD =~ s/$find/city/;
Still haven't figured out how to remove all before the nth occurrence of a comma or any character / string for that matter.
The content you get back is JSON, so you can easily turn it into a Perl data structure, play with it, and even turn it back into JSON if you like. That's the point! And, it's so easy:
use Mojo::UserAgent;
use Mojo::JSON qw(decode_json encode_json);
my $ua = Mojo::UserAgent->new;
my $tx = $ua->get( 'http://ip-api.com/json/8.8.8.8' );
my $json = $tx->res->body;
my $perl = decode_json( $json );
delete $perl->{'as'};
my $new_json = encode_json( $perl );
print $new_json;
Mojolicious is wonderful for this. It's my preferred way for dealing with JSON even without the user-agent stuff. If you play with the JSON string directly, you're likely to have problems when the order of elements change or it contains wide characters.
You don't have to manually decode_json() with Mojolicious. Simply do this:
my $tx = $ua->get('http://ip-api.com/json/8.8.8.8');
my $json = $tx->res->json;
my $as = $json->{as}
You can even go fancy with JSON pointers:
my $as = $tx->res->json("/as");
Something like
#!/usr/bin/perl -w
my $results = `curl http://ip-api.com/json/8.8.8.8`;
chomp $results;
$results =~ s/^.*city":"\w+\s?\w+",//g;
print $results . "\n";
should do the trick.. unless there's a misunderstanding of what you want to keep v.s. remove.
FYI, http://regexr.com/ is totally my go to for regex happiness.
Script works well when run manually, but when I schdule it in cronjob it shows :
malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "<html>\r\n<head><tit...") at /usr/local/lib/perl5/site_perl/5.14.2/JSON.pm line 171.
script itself:
#rest config vaiables
$ENV{'PERL_LWP_SSL_VERIFY_NONE'} = 0;
print "test\n";
my $client = REST::Client->new();
$client->addHeader('Authorization', 'Basic YWRtaW46cmFyaXRhbg==');
$client->addHeader('content_type', 'application/json');
$client->addHeader('accept', 'application/json');
$client->setHost('http://10.10.10.10');
$client->setTimeout(1000);
$useragent = $client->getUseragent();
print "test\n";
#Getting racks by pod
$req = '/api/v2/racks?name_like=2t';
#print " rekvest {$req}\n";
$client->request('GET', qq($req));
$racks = from_json($client->responseContent());
$datadump = Dumper (from_json($client->responseContent()));
crontab -l
*/2 * * * * /usr/local/bin/perl /folder/api/2t.pl > /dmitry/api/damnitout 2>&1
Appreciate any suggestion
Thank you,
Dmitry
It is difficult to say what is really happening, but in my experience 99% issues of running stuff in crontab stems from differences in environment variables.
Typical way to debug this: in the beginning of your script add block like this:
foreach my $key (keys %ENV) {
print "$key = $ENV{$key}\n";
}
Run it in console, look at the output, save it in log file.
Now, repeat the same in crontab and save it into log file (you have already done that - this is good).
See if there is any difference in environment variables when trying to run it both ways and try to fix it. In Perl, probably easiest is to alter environment by changing %ENV. After all differences are sorted out, there is no reason for this to not work right.
Good luck!