Script works well when run manually, but when I schdule it in cronjob it shows :
malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "<html>\r\n<head><tit...") at /usr/local/lib/perl5/site_perl/5.14.2/JSON.pm line 171.
script itself:
#rest config vaiables
$ENV{'PERL_LWP_SSL_VERIFY_NONE'} = 0;
print "test\n";
my $client = REST::Client->new();
$client->addHeader('Authorization', 'Basic YWRtaW46cmFyaXRhbg==');
$client->addHeader('content_type', 'application/json');
$client->addHeader('accept', 'application/json');
$client->setHost('http://10.10.10.10');
$client->setTimeout(1000);
$useragent = $client->getUseragent();
print "test\n";
#Getting racks by pod
$req = '/api/v2/racks?name_like=2t';
#print " rekvest {$req}\n";
$client->request('GET', qq($req));
$racks = from_json($client->responseContent());
$datadump = Dumper (from_json($client->responseContent()));
crontab -l
*/2 * * * * /usr/local/bin/perl /folder/api/2t.pl > /dmitry/api/damnitout 2>&1
Appreciate any suggestion
Thank you,
Dmitry
It is difficult to say what is really happening, but in my experience 99% issues of running stuff in crontab stems from differences in environment variables.
Typical way to debug this: in the beginning of your script add block like this:
foreach my $key (keys %ENV) {
print "$key = $ENV{$key}\n";
}
Run it in console, look at the output, save it in log file.
Now, repeat the same in crontab and save it into log file (you have already done that - this is good).
See if there is any difference in environment variables when trying to run it both ways and try to fix it. In Perl, probably easiest is to alter environment by changing %ENV. After all differences are sorted out, there is no reason for this to not work right.
Good luck!
Related
Please tell me such a thing. There is a self-written site, the code works on it, when I transfer the same code to the yii2 console application, it will get an error converting an array to a string
I will describe in more detail what is happening.
There is a text file inside which the following.
5|02.10.2020 20:20|Name 1
5|02.10.2020 22:25|Name 2
Day of the week, date and time, program name. It is necessary to add this case to the database.
$data = iconv("windows-1251", "UTF-8", file_get_contents( './import/' . $filename . '.txt' ) );
$tv = explode("\n", $data);
foreach ($tv as $line)
{
$ex1 = explode("|",$line);
$ex2 = explode(" ",$ex1[1]);
... next insert to db $ex1[0] - day of week, $ex2[0] - date, $ex2[1] - time, $ex1[2] - name
}
In a regular .php file, all this works, and when the same code is in yii2, it swears at conversion. Why is that? How to do it right?
It turns out that it cannot use $ ex1 [0], $ ex2 [0] and so on as a string.
I'm using PTC Integrity at my firm. Here we have an Excel file which I need to transfer to our SQL Database with a Perl script.
In Integrity itself, there is a number for a Member Revision. I can see this Revision Number when I type:
echo %MKSSI_REVISION1%
in the command line. I tried to write this in Perl, but it's really hard for me. The Perl script should look for the Excel file Database.xlsx in the path C:\Integrity_Sandbox\Database\Database.xlsx, then read the Member Revision number, and write this number to my SQL Database.
Does anyone have any ideas on how I can manage to do this?
Edit Solution:
my #result = `si revisioninfo --project=/Database/project.pj ´Database.xlsm`;
my $integrity_version = #result[2];
chomp $integrity_version;
my #fields = split(/: /,$integrity_version);
$integrity_version = #fields[1];
chomp $integrity_version;
Bodaggnmo solved his problem with the following Perl code:
my #result = `si revisioninfo --project=/Database/project.pj ´Database.xlsm`;
my $integrity_version = #result[2];
chomp $integrity_version;
my #fields = split(/: /,$integrity_version);
$integrity_version = #fields[1];
chomp $integrity_version;
yesterday I got a very easy task, but unfortunatelly looks like i can't do with a nice code.
The task briefly: I have a lot of parameters, that I want to ask with whiptail "interactive" mode in the installer script.
The detail of code:
#!/bin/bash
address="192.168.0.1" # default address, what the user can modify
addressT="" # temporary variable, that I want to check and modify, thats why I don't modify in the function the original variable
port="1234"
portT=""
... #there is a lot of other variable that I need for the installer
function parameter_set {
$1=$(whiptail --title "Installer" --inputbox "$2" 12 60 "$3" 3>&1 1>&2 2>&3) # thats the line 38
}
parameter_set "addressT" "Please enter IP address!" "$address"
parameter_set "portT" "Please enter PORT!" "$port"
But i got the following error:
"./install.sh: line: 38: address=127.0.0.1: command not found"
If I modify the variable to another (not a parameter of function), works well.
function parameter_set {
foobar=$(whiptail --title "Installer" --inputbox "$2" 12 60 "$3" 3>&1 1>&2 2>&3)
echo $foobar
}
I try to use global retval variable, and assign to outside of the function to the original variable, it works, but I think it's not the nicest solution for this task.
Could anybody help me, what I do it wrong? :)
Thanks in advance (and sorry for my bad english..),
Attila
It seems that your whiptail command is not producing anyoutput because of the redirections. So the command substitution leaves the value empty. Try removing them. Also it's better to save the new value to a local variable first:
parameter_set() {
local NAME=$1
local NEWVALUE=$(whiptail --title "Installer" --inputbox "$2" 12 60 "$3")
export $NAME="$NEWVALUE"
}
I'm using Symfony's console component to write some command line tools, one of which uses WP-CLI to set up a WordPress site. Specifically using wp option update I'm running into issues with both JSON and quotes.
For example, running something like:
shell_exec("wp option update blogdescription ''");
Results in literal '' in the database. In fact, any time I use that command, if it's successful, whatever I've tried to set in the database is wrapped in the single quotes.
And then something like this:
$github_updater_args = [
'github_access_token'=>'',
'bitbucket_username'=>'username',
'bitbucket_password'=>'password',
'all-in-one-seo-populate-keywords'=>'1'
];
$github_updater_args = json_encode($github_updater_args);
var_dump($github_updater_args);
shell_exec("wp option update github_updater '$github_updater_args' --format=json");
The dump results in:
string(200) "{"github_access_token":"","bitbucket_username":"devs#webspecdesign.com","bitbucket_password":"xxxxxxxxx","webspec-design-wordpress-core":"1","all-in-one-seo-populate-keywords":"1","webspec-smtp":"1"}"
Which is valid JSON, but the wp command resulted in the following:
Error: Invalid JSON: '{github_access_token:,bitbucket_username:username,bitbucket_password:password,all-in-one-seo-populate-keywords:1}'
You'll notice the quotes have been stripped out, which I assume is what it's complaining about? Or is that just how it dumps out the error?
Either way, I then decided I would hardcode the JSON, so just:
shell_exec('wp option update github_updater \'{"github_access_token": "", "bitbucket_username": "username", "bitbucket_password": "password"}\' --format=json');
Which gave me the following error:
Error: Too many positional arguments: , bitbucket_username: username, bitbucket_password: password}'
I wager the two issues are related. I thought maybe it was a WP-CLI issue, so I opened an issue there, but it was closed without reaching an answer. The more I stare at it, however, the more I think it might by a Symfony thing. Perhaps I definitely need to use the Process Component rather than shell_exec? Is that what it's designed for?
Update: Tried the Symfony Process Component:
$process = new Process("wp option update blogdescription ''");
$process->run();
Still got my two quotes. So nothing doing there.
You are looking for escapeshellarg() to escape your json data. escapeshellcmd() is for sanitizing an entire command, not a single argument. Try something like this:
$github_updater_args = [
'github_access_token'=>'',
'bitbucket_username'=>'username',
'bitbucket_password'=>'password',
'all-in-one-seo-populate-keywords'=>'1'
];
$github_updater_args = json_encode($github_updater_args);
$cmd = sprintf("wp option update github_updater %s --format=json", escapeshellarg($github_updater_args));
shell_exec($cmd);
Edit
There is something missing from your question or there is a bug in the CLI script you are executing. I wrote two quick test php scripts to illustrate that the json escaping works correctly.
test.php
<?php
$github_updater_args = [
'github_access_token'=>'',
'bitbucket_username'=>'username',
'bitbucket_password'=>'password',
'all-in-one-seo-populate-keywords'=>'1'
];
$github_updater_args = json_encode($github_updater_args);
$cmd = sprintf("php j.php %s", escapeshellarg($github_updater_args));
echo shell_exec($cmd);
echo PHP_EOL;
j.php
<?php
var_dump(json_decode($argv[1]));
output
~$ php test.php
object(stdClass)#1 (4) {
["github_access_token"]=>
string(0) ""
["bitbucket_username"]=>
string(8) "username"
["bitbucket_password"]=>
string(8) "password"
["all-in-one-seo-populate-keywords"]=>
string(1) "1"
}
I have to extract data from JSON file depending on a specific key. The data then has to be filtered (based on the key value) and separated into different fixed width flat files. I have to develop a solution using shell scripting.
Since the data is just key:value pair I can extract them by processing each line in the JSON file, checking the type and writing the values to the corresponding fixed-width file.
My problem is that the input JSON file is approximately 5GB in size. My method is very basic and would like to know if there is a better way to achieve this using shell scripting ?
Sample JSON file would look like as below:
{"Type":"Mail","id":"101","Subject":"How are you ?","Attachment":"true"}
{"Type":"Chat","id":"12ABD","Mode:Online"}
The above is a sample of the kind of data I need to process.
Give this a try:
#!/usr/bin/awk
{
line = ""
gsub("[{}\x22]", "", $0)
f=split($0, a, "[:,]")
for (i=1;i<=f;i++)
if (a[i] == "Type")
file = a[++i]
else
line = line sprintf("%-15s",a[i])
print line > file ".fixed.out"
}
I made assumptions based on the sample data provided. There is a lot based on those assumptions that may need to be changed if the data varies much from what you've shown. In particular, this script will not work properly if the data values or field names contain colons, commas, quotes or braces. If this is a problem, it's one of the primary reasons that a proper JSON parser should be used. If it were my assignment, I'd push back hard on this point to get permission to use the proper tools.
This outputs lines that have type "Mail" to a file named "Mail.fixed.out" and type "Chat" to "Chat.fixed.out", etc.
The "Type" field name and field value ("Mail", etc.) are not output as part of the contents. This can be changed.
Otherwise, both the field names and values are output. This can be changed.
The field widths are all fixed at 15 characters, padded with spaces, with no delimiters. The field width can be changed, etc.
Let me know how close this comes to what you're looking for and I can make some adjustments.
perl script
#!/usr/bin/perl -w
use strict;
use warnings;
no strict 'refs'; # for FileCache
use FileCache; # avoid exceeding system's maximum number of file descriptors
use JSON;
my $type;
my $json = JSON->new->utf8(1); #NOTE: expect utf-8 strings
while(my $line = <>) { # for each input line
# extract type
eval { $type = $json->decode($line)->{Type} };
$type = 'json_decode_error' if $#;
$type ||= 'missing_type';
# print to the appropriate file
my $fh = cacheout '>>', "$type.out";
print $fh $line; #NOTE: use cache if there are too many hdd seeks
}
corresponding shell script
#!/bin/bash
#NOTE: bash is used to create non-ascii filenames correctly
__extract_type()
{
perl -MJSON -e 'print from_json(shift)->{Type}' "$1"
}
__process_input()
{
local IFS=$'\n'
while read line; do # for each input line
# extract type
local type="$(__extract_type "$line" 2>/dev/null ||
echo json_decode_error)"
[ -z "$type" ] && local type=missing_type
# print to the appropriate file
echo "$line" >> "$type.out"
done
}
__process_input
Example:
$ ./script-name < input_file
$ ls -1 *.out
json_decode_error.out
Mail.out