I'm trying to load data into a MySQL database using the LOAD DATA LOCAL INFILE statement. On normal files, this works fine.
If I create a temporary file with File::Temp, store CSV data in it, close the file and then directly LOAD it into the database using
$dbh->do("LOAD DATA LOCAL INFILE '$tempfile' INTO TABLE $temptable" FIELDS TERMINATED BY ',');
the last two records are reproducibly omitted. However, if I do anything with the tempfile between creation and LOADing, for example with
`touch $tempfile`;
everything works as expected.
Is this an issue with the MySQL driver having trouble with freshly created tempfiles? Is it a filesystem (ext4) issue, maybe a cache flush not happening in time? Am I missing something here?
EDIT: Actually, all records are omitted if the temporary CSV file is not created by a format-converter subroutine, but by hand as shown below. I also included the code for the database interaction. Note the commented touch $tmpfh, which, when uncommented, would make the example work.
Adding UNLINK => 0 to File::Temp->new() does not make a difference.
my $tmpfh = File::Temp->new();
print $tmpfh <<EOT;
record1,textfield1
record2,textfield2
record3,textfield3
record4,textfield4
record5,textfield5
EOT
# `touch $tmpfh`; # uncomment this line to make it work
# get db handle
my $dbh = DBI->connect("DBI:mysql:$dbname:$dbserver", $username, $pwd);
# drop and recreate temp table
$dbh->do("DROP TABLE IF EXISTS $temptable") or die;
$dbh->do("CREATE TABLE $temptable (
`id` INT(11) NOT NULL PRIMARY KEY AUTO_INCREMENT,
`header` VARCHAR(255) NOT NULL,
`sequence` MEDIUMBLOB)")
or die;
# load data into temp table
my $nrecords = $dbh->do("LOAD DATA LOCAL INFILE '$tmpfh'
INTO TABLE $temptable
FIELDS TERMINATED BY ','
(header, sequence)")
or die;
$dbh->disconnect();
printf "Loaded %d records from %s into %s on %s.\n", $nrecords, $tmpfh, $dbname, $dbserver;
Close the file handle to flush the buffer. Keep the "UNLINK => 0" if you want the file to remain when the object goes out of scope.
Related
I have a MySql database dump saved as a text file and am trying to find a way of extracting the indivdual pdf documents stored in one of the tables. All my research online so far has drawn a blank.
The data in the exported text file is in the following format:
DROP TABLE IF EXISTS `codocs`;
CREATE TABLE `codocs` (
`ID` int(10) unsigned NOT NULL AUTO_INCREMENT,
`COCODE` varchar(8) NOT NULL,
`FILENAME` varchar(100) NOT NULL,
`DATE` date NOT NULL,
`USER` varchar(10) NOT NULL,
`DOCUMENT` mediumblob NOT NULL,
PRIMARY KEY (`ID`),
KEY `oc` (`COCODE`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
LOCK TABLES `codocs` WRITE;
/*!40000 ALTER TABLE `codocs` DISABLE KEYS */;
INSERT INTO `codocs` (`ID`, `COCODE`, `FILENAME`, `DATE`, `USER`, `DOCUMENT`)
VALUES
(1,’123456’,’document-2016-01-18.pdf','2016-01-21’,’user1’,X’8CB7638C2840B32D3AB66DDBB66DDBB66DDBB6BDC7B66DDBB6B1C7336F9F736EDECD4DBE1FE7477752D555ABBB562A59D5A40A2262B48C74CC0450A48747734B04508C040C04F64656 …
D2495CC3D8C1FCB8845D1D6F6C717E5EFB493B431B1250782FFFC12FD518D0E4EBF951D3B98F3C7971C1235F54B793172A427FF0F'),
(2,’234567’,’document-2016-01-18.pdf','2016-01-22’,’user1’,X’8CF763702E4EF02D0AC7B6ED64C7B66DDB7E62DBB6EDECD8C98E6DDBB66D3B797FE79C5BEFAD5BF5FF70AA66BAAA7B7AD674AD999A5A4DAE282A4EC744CF4204437E7038BB4804C344C448646F6C4504C3CB4B04C3A0EAE900206210317231B2B137FFCF57343207381331FF9 …
971C1235F54B793172A427FF0F'),
(3,’…
Any assistance would be greatly appreciated.
Update: 20220112
I have since restored the database from the sql dump and have subsequently created the following php files to try to display the pdfs stored in the codocs table:
db.php - contains the mysql database connection - this is working
records_list.php - lists all the records in the codocs table including a button on each returned row to view the stored pdf - this is working
view_pdf.php - receives the ID for the record clicked on from the records_list.php file and passes the selected record ID to the SELECT statement and displays the correct (presumably, as different data is returned for each separate record clicked on in the records_list.php file) raw mediumblob code stored in the database -
this is not working as intended
The following code is for the view_pdf.php file:
<?php
$pdf_id = $_REQUEST['pdfID'];
require_once "db.php";
if(isset($pdf_id)) {
$myID = $pdf_id;
$sql = "select * from codocs where ID='" . $myID . "'";
if (!$result=mysqli_query($con, $sql)){
echo mysqli_error($con);
} else {
$row = mysqli_fetch_array($result);
echo $row["DOCUMENT"];
mysqli_close($con);
}
}
?>
As mentioned just the raw mediumblob data appears to be being returned.
If the following line is replaced:
echo $row["DOCUMENT"];
with
echo '<object data="data:application/pdf;base64,'.base64_decode($row['DOCUMENT']).'" type="application/pdf" style="height:1000px;width:100%"></object>';
or
echo base64_decode($row['DOCUMENT']);
it makes no difference. Raw code continues to be returned.
If the original line of code referred to above is replaced with
header('Content-type: application/pdf');
echo $row["DOCUMENT"];
a downloadable pdf is offered and can be saved but is unreadable with the following warning: "This PDF document might not be displayed correctly." and the following error: "Unable to open document file:///...document.pdf. File type unknown (application/octet-stream) is not supported."
Can anyone advise how the code above can be amended to allow the retrieval of the stored pdf files?
Is the X that precedes the single quotations marks shown surrounding the mediumblob data in the sql dump file of any significance?
Any assistance would be greatly appreciated.
Further Update 20220112:
The following are example unreadable pdf restores but generate 'pdf' files of differing sizes:
Record 554:
Using the following replacement code:
header('Content-type: application/pdf');
echo $row["DOCUMENT"];
generates an unreadable file 82.2Kb in size.
Using the following replacement code:
header('Content-type: application/pdf');
echo '<object data="data:application/pdf;base64,'.base64_decode($row['DOCUMENT']).'" type="application/pdf" style="height:1000px;width:100%"></object>';
generates an unreadable file 15.6Kb in size.
Using the following replacement code:
header('Content-type: application/pdf');
echo '<object data="data:application/pdf;base64,'.base64_encode($row['DOCUMENT']).'" type="application/pdf" style="height:1000px;width:100%"></object>';
generates an unreadable file 109.7Kb in size.
Any thoughts on helping to resolve the issue would be very welcome.
I use parametric queries for normal insert/updates for security.
How do I do that for queries like this:
LOAD DATA INFILE '/filepath' INTO TABLE mytable
In my case, the path to the file would be different everytime (for different requests). Is it fine to proceed like this (since I am not getting any data from outside, the file is from the server itself):
path = /filepath
"LOAD DATA INFILE" + path + "INTO TABLE mytable"
Since LOAD DATA is not listed in SQL Syntax Allowed in Prepared Statements you can't prepare something like
LOAD DATA INFILE ? INTO TABLE mytable
But SET is listed. So a workaround could be to prepare and execute
SET #filepath = ?
And then execute
LOAD DATA INFILE #filepath INTO TABLE mytable
Update
In Python with MySQLdb the following query should work
LOAD DATA INFILE %s INTO TABLE mytable
since no prepared statement is used.
To answer your "is it fine to proceed like this" question, your example code will fail because the resulting query will be missing quotes around the filename. If you changed it to the following it could run, but is still a bad idea IMO:
path = "/filepath"
sql = "LOAD DATA INFILE '" + path + "' INTO TABLE mytable" # note the single quotes
While you may not be accepting outside input today, code has a way of sticking around and getting reused/copied, so you should use the API in a way that will escape your parameters:
sql = "LOAD DATA INFILE %s INTO TABLE mytable"
cursor.execute(sql, (path,))
And don't forget to commit if autocommit is not enabled.
I am attempting to set up a large database on my desktop (~100GB) and one of the csv files is about 40 GB. My MySQL workbench executes the query for about 30-60 minutes then loses connection, with a report of error code 2013.
What is the typical upload time per GB ?
Do I need to modify my INNODB Option File or other parameters? I cannot seem to figure out the perfect settings...below I have listed my LOAD DATA code for reference.
LOAD DATA LOCAL INFILE '/Users/ED/desktop/mirror2/CHARTEVENTS.csv'
INTO TABLE CHARTEVENTS
FIELDS TERMINATED BY ',' ESCAPED BY '\\' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES
(#ROW_ID,#SUBJECT_ID,#HADM_ID,#ICUSTAY_ID,#ITEMID,#CHARTTIME,#STORETIME,#CGID,#VALUE,#VALUENUM,#VALUEUOM,#WARNING,#ERROR,#RESULTSTATUS,#STOPPED)
SET
ROW_ID = #ROW_ID,
SUBJECT_ID = #SUBJECT_ID,
HADM_ID = IF(#HADM_ID='', NULL, #HADM_ID),
ICUSTAY_ID = IF(#ICUSTAY_ID='', NULL, #ICUSTAY_ID),
ITEMID = #ITEMID,
CHARTTIME = #CHARTTIME,
STORETIME = IF(#STORETIME='', NULL, #STORETIME),
CGID = IF(#CGID='', NULL, #CGID),
VALUE = IF(#VALUE='', NULL, #VALUE),
VALUENUM = IF(#VALUENUM='', NULL, #VALUENUM),
VALUEUOM = IF(#VALUEUOM='', NULL, #VALUEUOM),
WARNING = IF(#WARNING='', NULL, #WARNING),
ERROR = IF(#ERROR='', NULL, #ERROR),
RESULTSTATUS = IF(#RESULTSTATUS='', NULL, #RESULTSTATUS),
STOPPED = IF(#STOPPED='', NULL, #STOPPED);
I don't know the details of the connection between your local machine and the MySQL server, but the connection could be getting cut off for any number of reasons. One simple workaround here would be to just upload the 40GB file directly to the same remote machine running MySQL, and then use LOAD DATA (without LOCAL). With this approach, the LOAD DATA statement should take orders of magnitude less time to parse the input file, there no longer being any network latency to slow things down.
I am relatively new to Perl.I have been trying to insert data to database from a text file using a CGI script.I have written code for it and it's working properly.but when i try to impose a limit on the data that is inserted using LIMIT keyword there is a problem.Please check where i am going wrong and what needs to be amended.Thanks for your advice.
here is the code
#!/usr/bin/perl
use warnings;
use strict;
use CGI ':standard';
use DBI;
if(param())
{
my #params=param();
my $limit=param('limit')||'';
my $dbh =DBI->connect("DBI:mysql:sample","root","");
my $var="LOAD DATA LOCAL INFILE 'C:/test.txt' INTO TABLE sample2 FIELDS TERMINATED BY '\n' WHERE LIMIT 0,$limit";
my $sth = $dbh->do($var) or die "prepare failed: " . $dbh->errstr();
print $sth ."Records inserted";
$sth->finish();
$dbh->disconnect();
print
header(),
start_html(
-title=>'Welcome',
-text=>'#520063'
),
#h1("Records have been displayed"),
end_html();
}
else
{
print
header(),
start_html('A Simple Form'),
h1('Please enter the limit '),
start_form(),
'Limit: ',
textfield(-name=>'limit'),
br(),
#'Phone Number: ',
#textfield(-name => 'number'),
#br(),
submit(),
end_form(),
end_html();
}
Without an error message I can't be totally sure, but the problem is likely in your SQL.
LOAD DATA LOCAL INFILE 'C:/test.txt'
INTO TABLE sample2
FIELDS TERMINATED BY '\n'
WHERE LIMIT 0,$limit
There's problems.
There WHERE clause is empty (LIMIT is not part of the WHERE clause)
LOAD DATA INFILE does not take a WHERE clause
LOAD DATA INFILE does not take a LIMIT clause
Basically that whole last line won't work. In general, if a SQL command doesn't work in a program try firing up the MySQL command line and debug the command there.
LOAD DATA INFILE does not have a way to limit how many lines it will pull in, this is an oft requested feature. To work around it I would suggest copying and truncating the file before feeding it to MySQL.
We use MySQL tables to which we add new fields from time to time as our product evolves.
I'm looking for a way to export the structure of the table from one copy of the db, to another, without erasing the contents of the table I'm importing to.
For example say I have copies A and B of a table, and I add fields X,Y,Z to table A. Is there a way to copy the changed structure (fields X,Y,Z) to table B while keeping its content intact?
I tried to use mysqldump, but it seems I can only copy the whole table with its content, overriding the old one, or I can use the "-d" flag to avoid copying data (dumping structure only), but this will create an empty table when imported, again overriding old data.
Is there any way to do what I need with mysqldump, or some other tool?
What I usually do is store each and every ALTER TABLE statement run on the development table(s), and apply them to the target table(s) whenever necessary.
There are more sophisticated ways to do this (like structure comparison tools and such), but I find this practice works well. Doing this on a manual step by step basis also helps prevent accidental alteration or destruction of data by structural changes that change a field's type or maximum length.
I just had the same problem and solved it this way:
Export the structure of the table to update.
Export the structure of the development table.
run this code for the first file "update.sql" needs to be changed according to your exported filename.
cat update.sql|awk -F / '{
if(match($0, "CREATE TABLE")) {
{ FS = "`" } ; table = $2
} else {
if(match($0," `")) {
gsub(",",";",$0)
print "ALTER TABLE `" table "` ADD" $0
}
}
}' > update_alter.sql
run the same command for the second file
cat development.sql|awk -F / '{
if(match($0, "CREATE TABLE")) {
{ FS = "`" } ; table = $2
} else {
if(match($0," `")) {
gsub(",",";",$0)
print "ALTER TABLE `" table "` ADD" $0
}
}
}' > development_alter.sql
run this command to find the differences in the output files
diff --changed-group-format='%<' --unchanged-group-format='' development_alter.sql update_alter.sql > update_db.sql
In the file update_db.sql there will now be the code you are looking for.
Lazy way: export your old data and struct, import your actual struct, import only your old data. Works to me in the test.
for your case, it might just need to perform an update
alter table B add column x varchar(255);
alter table B add column y varchar(255);
alter table B add column z varchar(255);
update A,B
set
B.x=A.x,
B.y=A.y,
B.z=A.z
where A.id=B.id; <-- a key that exist on both tables
There is a handy way of doing this but need a little bit editing in a text editor :
This takes about Max 10Min in Gedit Under Linux !!
Export you table & save it in : localTable.sql
Open it in a text edior (Gedit) You will see something like this :
CREATE TABLE IF NOT EXISTS `localTable` (
`id` int(8) NOT NULL AUTO_INCREMENT,
`date` int(10) NOT NULL,
# Lot more Fields .....
#Other Fields Here
After Just Remove :
Anything after the closing ) parenthese
CREATE TABLE IF NOT EXISTS localTable (
Change all , to ; in each line like thats you execute all this once (,\n to ;\n)
remove all ADDPRIMARY KEY (id);ADDKEY created_by (created_by) !
And just Keep Fields you are interested in
You will have this
`id` int(8) NOT NULL AUTO_INCREMENT,
`date` int(10) NOT NULL,
# Lot more Fields .....
#Other Fields Here
Add to the begining of each line ALTER TABLE localTable ADD
ALTER TABLE `localTable` ADD `id` int(8) NOT NULL AUTO_INCREMENT,
ALTER TABLE `localTable` ADD `date` int(10) NOT NULL,
ALTER TABLE `localTable` ADD #to each more Fields .....
#Other Fields Here
That's it we can make this ab Automated Script by adding a Shell Script to do this job .
After you know what you have to do Import it in the 'remoteTable' ;)
Thanks
No it isn't possible because MySql is using mariaDB version. In mariaDB version structure of a table are arranged in memory and that memory shared with your byte data.
So when we try to import a structure (or a table) it alter that whole memory block.