I'm trying to let a windows perl script run on linux, but it's not work, I want to know what mistake I made.
Originally, it was running on windows and connect to the local mysql db, but now I want to transfer it to linux. I installed docker on Linux and wanted to connect to the mysql db in container, but the revised script kept reporting an error
On Windows mysql version is 5.1
on Linux mysql version is 8.0
original script on windows
foreach my $file (#files) {
$command = '-e "LOAD DATA LOCAL INFILE \''.$file.'\' REPLACE INTO TABLE test FIELDS TERMINATED BY \'|\' LINES TERMINATED BY \'\\r\\n\' IGNORE 1 LINES (#v001, #v002, #v003, #v004, #v005) SET `CA` = TRIM(#v001), `CB` = TRIM(#v002),`CD` = TRIM(#v003),`CE` = TRIM(#v004),`CF` = TRIM(#v005);"';
system('mysql', '-hlocalhost', $user, $password, $command) == 0 or err("ERROR:Failed to load file to test: $! \n");
$nfile = "D:\\DONE\\".getmodifydate($file); #file last modify time
move($file, $nfile);
}
On Linux
sub docker_mysql {
$docker_mysql = "mysql", "-h$localhost", $database, $user, $password;
}
system(docker_mysql(), '-e', "DELETE FROM test WHERE 1") == 0 or
err("ERROR:Failed to delete record from test: $! ");
foreach my $file (#files) {
$command = "LOAD DATA LOCAL INFILE \'$file'\' REPLACE INTO TABLE test FIELDS TERMINATED BY \'|\' LINES TERMINATED BY \'\\n\' IGNORE 1 LINES (\#v001, \#v002, \#v003, \#v004, \#v005) SET `CA` = TRIM(\#v001), `CB` = TRIM(\#v002),`CD` = TRIM(\#v003),`CE` = TRIM(\#v004),`CF` = TRIM(\#v005);";
system(docker_mysql(), '-e', $command,) == 0 or
err("ERROR:Failed to load file to test: $! \n");
$nfile = "./DONE/".getmodifydate($file); #file last modify time
move($file, $nfile);
}
This is the error code after execution
ERROR 1064 (42000) at line 1: You have an error in your SQL syntax; check the manual that
corresponds to your MySQL server version for the right syntax to use
near '|' LINES TERMINATED BY '
' IGNORE 1 LINES (#v001, #v002, #v003, #v004, #v005) SE' at line 1
You need to escape the backslash that is part of \n to send an actual line-feed, not an n.
$command = "[...] LINES TERMINATED BY \'\n\' [...]";
^^ This should be \\n
Related
I want to write a TCL script in which i need to use MySQL. I have to (1) read a file; from that (2) copy data into a SQL table, and after that (3) query data from that table based on requirement. But I am unable to find how to copy data from file to SQL table and then how to query for that table in TCL.
LOAD DATA INFILE 'path/filename.csv' #loading data from a csv file
INTO TABLE tablename #tablename is user defined
FIELDS TERMINATED BY ',' #csv files use comma separated values
LINES TERMINATED BY '\n' #till the last column
IGNORE 1 LINES #ignores the first row if you need
proc mysql_connect {} {
variable mysql_dbh ; variable command
set mysql_host "hostwebsite"
set mysql_user "username"
set mysql_password "password"
set mysql_db "databasename"
## loading the driver
set libmysql_path "driver path"
if {[catch {load $libmysql_path}]} {
puts "$command Error: Unable to load mysql file" ; exit
}
## making connection with mysql db
if {[catch {set ::mysql_dbh [::mysql::connect -h $mysql_host -u $mysql_user -password $mysql_password -db $mysql_db]}]} {
puts "$command Error: Unable to connect mysql DB"; exit
}
puts "Db connected"
}
proc QueryDisplay {} {
mysql_connect
set rows [::mysql::sel $::mysql_dbh "select *from tablename" -list]
foreach data $rows {
puts $data
}
}
proc mysql_disconnect {} {
variable mysql_dbh
::mysql::close $::mysql_dbh
}
Below is my perl code
I referred to the following link http://www.tol.it/doc/MySQL/chapter6.html. If there is a better please post.
use Mysql;
my $db = Mysql->connect($mysqlhost, $mysqlDB, $mysqlUser, $mysqlPassword);
$db->selectdb("$mysqlDB");
my $loadQuery="LOAD DATA INFILE '$filename' INTO TABLE $pageURLTable FIELDS TERMINATED BY '\\t' LINES TERMINATED BY '\\n'";
print "Executing $loadQuery";
my $loadresult=$db->query($loadQuery);
if(!$loadresult){
print "Error: MySQl Load failed.System error message:$!.";
return -1;
}
print "Info:".$loadresult->info"; ## this raises error MySQL::Statement info not loadable;
What is wrong ?
Can you suggest a better way of coding this so that Load data file errors are better captured?
Thanks,
Neetesh
Try to use ->do method instead of ->query.
Make sure that Load Data Inline is enabled in your database.
If above don't work, you can use system(mysql -e "LOAD DATA INFILE...")
I generally use DBI recipes. "executing sql statement" I like to do something like $rows = $sth->execute(); Gives affected rows by last statement, and if rows >0 after loading csv files I'know that I've successfully loaded data in database.
my $sth = $dbh->prepare($sql)
or die "Can't prepare SQL statement: ", $dbh->errstr(), "\n";
$rows = $sth->execute();
if ($rows>0)
{
print "Loaded $rows rows\n" if ($Debug);
}
else {
print "Can't execute SQL statement: ", $sth->errstr(), "\n";
}
I have a .txt file that has a bunch of formatted data in it that looks like the following:
...
1 75175.18 95128.46
1 790890.89 795829.16
1 875975.98 880914.25
8 2137704.37 2162195.53
8 2167267.27 2375275.28
10 2375408.74 2763997.33
14 2764264.26 2804437.77
15 2804504.50 2881981.98
16 2882048.72 2887921.25
16 2993093.09 2998031.36
19 3004104.10 3008041.37
...
I am trying to load each row as an entry into a table in my database, where each column is a different field. I am having trouble getting mySQL to separate all of the data properly. I think the issue is coming from the fact that not all of the numbers are separated with an equidistant white-space amount.
Here are two queries I have tried so far (I have also tried several variations of these queries):
LOAD DATA LOCAL INFILE
'/some/Path/segmentation.txt'
INTO TABLE clip (slideNum, startTime, endTime)
SET presID = 1;
LOAD DATA LOCAL INFILE
'/some/Path/segmentation.txt'
INTO TABLE clip
FIELDS TERMINATED BY ' '
LINES TERMINATED BY '\n'
(slideNum, startTime, endTime)
SET presID = 1;
Any ideas how to get this to work?
These are what we call "fixed-width" records and LOAD DATA doesn't play well with them. Options:
Clean up data in Excel first, or
Load up the data to a temp table with only 1 column, shoving an entire text row into that column. Then you can use SUBSTR() and TRIM() to slice out the columns you need into the final table.
Or with user variables (#row) you can do it all within the LOAD DATA statement.
LOAD DATA LOCAL INFILE
'/some/Path/segmentation.txt'
INTO TABLE clip
(#row)
SET slideNum = TRIM(SUBSTR(#row,1,4)),
startTime = TRIM(SUBSTR(#row,5,13)),
endTime = TRIM(SUBSTR(#row,18,13))
;
LOAD DATA
CHARACTERSET AL32UTF8
INFILE 'DCF Master 14APR2013 VSPCFM_reduced size.txt'
INTO TABLE EMPLOYEE3
(
a = TRIM(SUBSTR(#row,1,11)),
b = TRIM(SUBSTR(#row,33,38)),
c = TRIM(SUBSTR(#row,70,86))
)
If you're on unix/linux then you can put it through sed to strip out spaces. The solution is here
You can programmatically replace spaces with a different delimiter. I decided to use PHP, you can also safely do it in Python
<?php
$mysqli = new mysqli(
"***",
"***",
"***",
"***",
3306
);
mysqli_options($mysqli, MYSQLI_OPT_LOCAL_INFILE, true);
if (mysqli_connect_errno()) {
printf("Connect failed: %s\n", mysqli_connect_error());
exit();
}
function createTempFileWithDelimiter($filename, $path){
$content = file_get_contents($filename);
$replaceContent = preg_replace('/\ +/', ',', $content); // NOT \s+
$onlyFileName = explode('\\',$filename);
$newFileName = $path.end($onlyFileName);
file_put_contents($newFileName, $replaceContent);
return $newFileName;
}
$pathTemp = 'C:\\TempDir\\';
$pathToFile = 'C:\\some\\Path\\segmentation.txt';
$file = createFileWithDelimiter($pathToFile, $pathTemp);
$file = str_replace(DIRECTORY_SEPARATOR, '/', $file);
$sql = "LOAD DATA LOCAL INFILE '".$file."' INTO TABLE `clip`
COLUMNS TERMINATED BY ','
LINES TERMINATED BY '\n' // or '\r\n'
(slideNum, startTime, endTime)
SET presID = 1;";
if (!($stmt = $mysqli->query($sql))) {
echo "\nQuery execute failed: ERRNO: (" . $mysqli->errno . ") " . $mysqli->error;
};
unlink($file);
?>
Don't use '/\s+/' in preg_replace because \s matches any whitespace character (equivalent to [\r\n\t\f\v ]) and the formatting will change, columns and line breaks will disappear.
I am complete newbie in Perl. I have this little sub
sub processOpen{
my($filename, $mdbh)=#_;
my($qr, $query);
# parse filename to get the date extension only
# we will import the data into the table with this extension
# /home//logs/open.v7.20120710_2213.log
my(#fileparts) = split(/\./, $filename);
my(#filedateparts) = split(/_/, $fileparts[2]);
my($tableext) = $filedateparts[0];
$query = "LOAD DATA INFILE '" . $filename . "' INTO TABLE open_" . $tableext . " FIELDS TERMINATED BY '||' LINES TERMINATED BY '\n'
(open_datetime, open_date, period,tag_id)";
$qr = $$mdbh->prepare($query);
$qr->execute(); # causes error (see below)
$qr->finish();
}
And I'm getting the following error:
DBD::mysql::st execute failed: Can't get stat of '/home/logs/open..v7.20120710_2213.log' (Errcode: 2) at /home/thisfile.pm line 32.
Line 32 is the $qr->execute();
Error Code 2 is most likely "File not found".
Does your file exist? Note that if you are running the perl on a separate host from the MySQL database, the file must be on the database host, not the client host.
Is there a way to dynamically specify a file name in the LOAD DATA INFILE? Can it be parameterized like for instance (syntax may be incorrect) LOAD DATA INFILE '$filename'?
A citation from MySQL documentation:
The LOAD DATA INFILE statement reads rows from a text file into a table at a very high speed. The file name must be given as a literal string.
That means that it can not be a parameter of a prepared statement. But no one forbids to make the string interpolation while the statement is just a string in your PHP code.
Unfortunately, this feature is not yet supported in MySQL and is currently listed as bug/feature request #39115 http://bugs.mysql.com/bug.php?id=39115
Or you can make a temporary copy of the file (BATCH example):
LOAD_DATA.bat
COPY %1 TempFileToLoad.csv
mysql --user=myuser --password=mypass MyDatabase < ScriptLoadMyDatabase.sql
DEL TempFileToLoad.csv
the SQL (for info) :
ScriptLoadMyDatabase.sql
load data infile 'TempFileToLoad.csv' IGNORE
into table tLoad
FIELDS TERMINATED BY ';' OPTIONALLY ENCLOSED BY '"'
lines terminated by '\r\n'
IGNORE 1 LINES
(#DateCrea, NomClient, PrenomClient, TypeMvt, #Montant, NumeroClient)
set DateCrea = str_to_date(#DateCrea, '%Y-%m-%d'), Montant = (round(#Montant / 1000)*2) ;
And finished to put a link to the BAT file in SendTo windows folder.
If you're asking if it can be used in a script; you can do some thing like this with php:
<?php
$mysqli = new mysqli("host", "user", "pwd", "db");
/* check connection */
if (mysqli_connect_errno()) {
printf("Connect failed: %s\n", mysqli_connect_error());
exit();
}
$sql = "CREATE TABLE number1 (id INT PRIMARY KEY auto_increment,data TEXT)";
if ($result = $mysqli->query($sql)) {
} else {
printf("<br>%s",$mysqli->error);
}
$host = $_SERVER['HTTP_HOST'];
$uri = rtrim(dirname($_SERVER['PHP_SELF']), '/\\');
$filename = "data.csv";
$sql = "LOAD DATA LOCAL INFILE '$host$uri$filename' INTO TABLE number1";
if ($result = $mysqli->query($sql)) {
} else {
printf("<br>%s",$mysqli->error);
}
// Close the DB connection
$mysqli->close();
exit;
%>
If the file is in the same folder as the script just use $filename a instead of $host$uri$filename. I put this together quick from a couple scripts I'm using, sorry if it doesn't work without debug, but it should be pretty close. It requires mysqli.