I am complete newbie in Perl. I have this little sub
sub processOpen{
my($filename, $mdbh)=#_;
my($qr, $query);
# parse filename to get the date extension only
# we will import the data into the table with this extension
# /home//logs/open.v7.20120710_2213.log
my(#fileparts) = split(/\./, $filename);
my(#filedateparts) = split(/_/, $fileparts[2]);
my($tableext) = $filedateparts[0];
$query = "LOAD DATA INFILE '" . $filename . "' INTO TABLE open_" . $tableext . " FIELDS TERMINATED BY '||' LINES TERMINATED BY '\n'
(open_datetime, open_date, period,tag_id)";
$qr = $$mdbh->prepare($query);
$qr->execute(); # causes error (see below)
$qr->finish();
}
And I'm getting the following error:
DBD::mysql::st execute failed: Can't get stat of '/home/logs/open..v7.20120710_2213.log' (Errcode: 2) at /home/thisfile.pm line 32.
Line 32 is the $qr->execute();
Error Code 2 is most likely "File not found".
Does your file exist? Note that if you are running the perl on a separate host from the MySQL database, the file must be on the database host, not the client host.
Related
I'm trying to let a windows perl script run on linux, but it's not work, I want to know what mistake I made.
Originally, it was running on windows and connect to the local mysql db, but now I want to transfer it to linux. I installed docker on Linux and wanted to connect to the mysql db in container, but the revised script kept reporting an error
On Windows mysql version is 5.1
on Linux mysql version is 8.0
original script on windows
foreach my $file (#files) {
$command = '-e "LOAD DATA LOCAL INFILE \''.$file.'\' REPLACE INTO TABLE test FIELDS TERMINATED BY \'|\' LINES TERMINATED BY \'\\r\\n\' IGNORE 1 LINES (#v001, #v002, #v003, #v004, #v005) SET `CA` = TRIM(#v001), `CB` = TRIM(#v002),`CD` = TRIM(#v003),`CE` = TRIM(#v004),`CF` = TRIM(#v005);"';
system('mysql', '-hlocalhost', $user, $password, $command) == 0 or err("ERROR:Failed to load file to test: $! \n");
$nfile = "D:\\DONE\\".getmodifydate($file); #file last modify time
move($file, $nfile);
}
On Linux
sub docker_mysql {
$docker_mysql = "mysql", "-h$localhost", $database, $user, $password;
}
system(docker_mysql(), '-e', "DELETE FROM test WHERE 1") == 0 or
err("ERROR:Failed to delete record from test: $! ");
foreach my $file (#files) {
$command = "LOAD DATA LOCAL INFILE \'$file'\' REPLACE INTO TABLE test FIELDS TERMINATED BY \'|\' LINES TERMINATED BY \'\\n\' IGNORE 1 LINES (\#v001, \#v002, \#v003, \#v004, \#v005) SET `CA` = TRIM(\#v001), `CB` = TRIM(\#v002),`CD` = TRIM(\#v003),`CE` = TRIM(\#v004),`CF` = TRIM(\#v005);";
system(docker_mysql(), '-e', $command,) == 0 or
err("ERROR:Failed to load file to test: $! \n");
$nfile = "./DONE/".getmodifydate($file); #file last modify time
move($file, $nfile);
}
This is the error code after execution
ERROR 1064 (42000) at line 1: You have an error in your SQL syntax; check the manual that
corresponds to your MySQL server version for the right syntax to use
near '|' LINES TERMINATED BY '
' IGNORE 1 LINES (#v001, #v002, #v003, #v004, #v005) SE' at line 1
You need to escape the backslash that is part of \n to send an actual line-feed, not an n.
$command = "[...] LINES TERMINATED BY \'\n\' [...]";
^^ This should be \\n
I have a .dat file that I've pulled from another application that has many rows of comma delimitated data. I opened the file with Excel, and saved it as a .csv. I am able to upload that file without issue to my database using LOAD DATA INFILE.
But here's where it gets weird. If I take the original file and try to copy, for example, 20 rows and make a new .csv file, when I upload it to the same database, same table schema, it will only load the first row. Did Excel do something with my formatting when I saved as a new .csv? It looks the same to me...I can't see a difference when opening with other text editors.
Here's my SQL for creating a new table for the data to be imported (I first check if the table already exists):
// Create table - Table name matches name of file without extension
$sql = "CREATE TABLE " . $file_name . " (
history_id INT,
stream_id INT,
record_time DATETIME,
ALEQ FLOAT,
APEAK FLOAT,
AFASTSPL FLOAT,
dose FLOAT,
color INT
)";
Here's my SQL for loading the .csv:
// Insert data from uploaded .csv file into table
$sql = "LOAD DATA LOCAL INFILE '" . $target_file . "'
INTO TABLE " . $file_name . "
FIELDS OPTIONALLY ENCLOSED BY '''' TERMINATED BY ','
LINES TERMINATED BY '\n'";
Here's what the .csv for the file that does work looks like:
1,1,'2014-08-28 07:00:31',35.2,56.0,35.1,0.0000000,0
2,1,'2014-08-28 07:00:31',35.1,56.0,35.4,0.0000000,0
3,1,'2014-08-28 07:00:31',35.1,56.0,35.2,0.0000000,0
4,1,'2014-08-28 07:00:31',35.3,56.0,35.0,0.0000000,0
5,1,'2014-08-28 07:00:31',35.2,56.0,35.3,0.0000000,0
6,1,'2014-08-28 07:00:31',35.2,56.0,35.2,0.0000000,0
And here's the .csv for the file that doesn't work:
60135,1,'2014-08-28 15:19:19',52,68.3,52.6,0.43186,0
60136,1,'2014-08-28 15:19:20',56.9,71.6,51.3,0.43186,0
60137,1,'2014-08-28 15:19:20',56.5,68.8,58.3,0.43186,0
60138,1,'2014-08-28 15:19:21',57,70.1,56.4,0.43186,0
60139,1,'2014-08-28 15:19:21',56.5,69.2,56.2,0.43186,0
60140,1,'2014-08-28 15:19:22',58.7,71.4,56.7,0.43186,0
If it only loads the first row then it is not detecting the line endings properly. I suggest using a hex editor to check what line endings you have in your source file.
Perhaps you saved as "CSV (Macintosh)" which uses \r as line endings instead. If this is the case, you should be able to use
LINES TERMINATED BY '\r'";
A more generic solution if you dont know the line endings...
MySQL has no automatic line ending detection and that is why you have to tell it how to parse the data. You can use this short function to do some basic frequency analysis.
function detectLineEndings($file)
{
$data = file_get_contents($file);
$le_linux = substr_count($data, "\n");
$le_windows = substr_count($data, "\r\n");
$le_mac = substr_count($data, "\r");
if ($le_linux > $le_windows && $le_linux > $le_mac) {
return '\n';
}
if ($le_mac > $le_linux && $le_mac > $le_windows) {
return '\r';
}
return '\r\n';
}
$endings = detectLineEndings($target_file);
$sql = "LOAD DATA LOCAL INFILE '" . $target_file . "'
INTO TABLE " . $file_name . "
FIELDS OPTIONALLY ENCLOSED BY '''' TERMINATED BY ','
LINES TERMINATED BY '" . $endings ."'";
Below is my perl code
I referred to the following link http://www.tol.it/doc/MySQL/chapter6.html. If there is a better please post.
use Mysql;
my $db = Mysql->connect($mysqlhost, $mysqlDB, $mysqlUser, $mysqlPassword);
$db->selectdb("$mysqlDB");
my $loadQuery="LOAD DATA INFILE '$filename' INTO TABLE $pageURLTable FIELDS TERMINATED BY '\\t' LINES TERMINATED BY '\\n'";
print "Executing $loadQuery";
my $loadresult=$db->query($loadQuery);
if(!$loadresult){
print "Error: MySQl Load failed.System error message:$!.";
return -1;
}
print "Info:".$loadresult->info"; ## this raises error MySQL::Statement info not loadable;
What is wrong ?
Can you suggest a better way of coding this so that Load data file errors are better captured?
Thanks,
Neetesh
Try to use ->do method instead of ->query.
Make sure that Load Data Inline is enabled in your database.
If above don't work, you can use system(mysql -e "LOAD DATA INFILE...")
I generally use DBI recipes. "executing sql statement" I like to do something like $rows = $sth->execute(); Gives affected rows by last statement, and if rows >0 after loading csv files I'know that I've successfully loaded data in database.
my $sth = $dbh->prepare($sql)
or die "Can't prepare SQL statement: ", $dbh->errstr(), "\n";
$rows = $sth->execute();
if ($rows>0)
{
print "Loaded $rows rows\n" if ($Debug);
}
else {
print "Can't execute SQL statement: ", $sth->errstr(), "\n";
}
I have tried the below code for laoding the inline file in drupal db
$sql = "LOAD DATA INFILE '".db_escape_string($this->file_name).
"' INTO TABLE `".$this->table_name.
"` FIELDS TERMINATED BY '".db_escape_string($this->field_separate_char).
"' OPTIONALLY ENCLOSED BY '".db_escape_string($this->field_enclose_char).
"' ESCAPED BY '".db_escape_string($this->field_escape_char).
"' ".
($this->use_csv_header ? " IGNORE 1 LINES " : "")
."(`".implode("`,`", $this->arr_csv_columns)."`)";
$res = db_query($sql);
$this->error = mysql_error();
I got the error as but the file was present in the folder
user warning: File 'http:\localhost\example\sites\default\files\example_1.csv' not found (Errcode: 22) query: LOAD DATA INFILE 'http://localhost/example/sites/default/files/example_1.csv' INTO TABLE temp_21_10_2010_06_38_00 FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '\"' ESCAPED BY '\' IGNORE 1 LINES (PROGRAMNAME,OFFLINE,ONLINE,INSTOCK) in C:\wamp\www\example\sites\all\modules\importmodule\Quick_CSV_import.php on line 65.
On line 65 the above php code $res = db_query($sql); is present.
you need to specify the full/absolute or a relative path to your csv file not a URL
load data infile '/myproj/imports/example_1.csv' into table...
load data infile 'c:\\myproj\\imports\\example_1.csv' into table...
suggest you check the manual again http://dev.mysql.com/doc/refman/5.1/en/load-data.html
Perhaps is it due to open_files_limit ?
Is there a way to dynamically specify a file name in the LOAD DATA INFILE? Can it be parameterized like for instance (syntax may be incorrect) LOAD DATA INFILE '$filename'?
A citation from MySQL documentation:
The LOAD DATA INFILE statement reads rows from a text file into a table at a very high speed. The file name must be given as a literal string.
That means that it can not be a parameter of a prepared statement. But no one forbids to make the string interpolation while the statement is just a string in your PHP code.
Unfortunately, this feature is not yet supported in MySQL and is currently listed as bug/feature request #39115 http://bugs.mysql.com/bug.php?id=39115
Or you can make a temporary copy of the file (BATCH example):
LOAD_DATA.bat
COPY %1 TempFileToLoad.csv
mysql --user=myuser --password=mypass MyDatabase < ScriptLoadMyDatabase.sql
DEL TempFileToLoad.csv
the SQL (for info) :
ScriptLoadMyDatabase.sql
load data infile 'TempFileToLoad.csv' IGNORE
into table tLoad
FIELDS TERMINATED BY ';' OPTIONALLY ENCLOSED BY '"'
lines terminated by '\r\n'
IGNORE 1 LINES
(#DateCrea, NomClient, PrenomClient, TypeMvt, #Montant, NumeroClient)
set DateCrea = str_to_date(#DateCrea, '%Y-%m-%d'), Montant = (round(#Montant / 1000)*2) ;
And finished to put a link to the BAT file in SendTo windows folder.
If you're asking if it can be used in a script; you can do some thing like this with php:
<?php
$mysqli = new mysqli("host", "user", "pwd", "db");
/* check connection */
if (mysqli_connect_errno()) {
printf("Connect failed: %s\n", mysqli_connect_error());
exit();
}
$sql = "CREATE TABLE number1 (id INT PRIMARY KEY auto_increment,data TEXT)";
if ($result = $mysqli->query($sql)) {
} else {
printf("<br>%s",$mysqli->error);
}
$host = $_SERVER['HTTP_HOST'];
$uri = rtrim(dirname($_SERVER['PHP_SELF']), '/\\');
$filename = "data.csv";
$sql = "LOAD DATA LOCAL INFILE '$host$uri$filename' INTO TABLE number1";
if ($result = $mysqli->query($sql)) {
} else {
printf("<br>%s",$mysqli->error);
}
// Close the DB connection
$mysqli->close();
exit;
%>
If the file is in the same folder as the script just use $filename a instead of $host$uri$filename. I put this together quick from a couple scripts I'm using, sorry if it doesn't work without debug, but it should be pretty close. It requires mysqli.