sample
user id User Name
U456 Mathew
U457 Leon
U458 Cris
U459 Yancy
U460 Jane
and so on up to 500k.
I need to read this text file and insert to MySQL in two columns say User ID and User Name. How do I do this in PHP?
LOAD DATA INFILE
Example:
NOTE: if you run this from Windows you need to escape the forward slashes in the file path.
EXAMPLE:
C:\\path\to\file.txt
Looks like:
C:\\\\path\\to\\file.txt
Here is the query:
LOAD DATA INFILE '/path/to/sample.txt'
INTO TABLE `database_name`.`table_name`
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\r\n'
IGNORE 1 LINES
(
user_id, user_name
)
Delete the first line and use this command
LOAD DATA INFILE 'C:\\sample.txt'
INTO TABLE Users
FIELDS TERMINATED BY ' '
LINES TERMINATED BY '\r\n';
For more information visit
http://tech-gupshup.blogspot.com/2010/04/loading-data-in-mysql-table-from-text.html
Using PHP, Possibly Something similar to this:
$file = "/path/to/text/file.txt";
$fp = fopen($file, "r");
$data = fread($fp, filesize($file));
fclose($fp);
The above reads the text file into a variable
$output = explode("\n", $output);
foreach($output as $var) {
$tmp = explode("|", $var);
$userId = $tmp[0];
$userName = $tmp[1];
Tell it to explode at each Endline and then store the data in temp variables
$sql = "INSERT INTO table SET userId='$userId', userName='$userName'";
mysql_query($sql);
Execute the query for each line
Depends. If the two fields are separated using a TAB character, then fgetcsv($f,1000,"\t") would work to read in each line. Otherwise use substr() if it's a fixed width column text file to split up the fields (apply trim() eventually).
Loop over the rows and fields, and use your database interface of choice:
db("INSERT INTO tbl (user_id, user_name) VALUES (?,?)", $row);
Use the fopen function of PHP to access and read the file.. From there on the rest is pretty much simple. Read the file line by line and insert the values into the database.
http://us3.php.net/manual/en/function.fopen.php
The above link gives a very good description of how the fopen function works.. Using a loop will be easy in this task.
Related
The Code i tried is
$source = fopen('filename.csv', 'r') or die("Problem open file");
while (($data = fgetcsv($source, 1000, ",")) !== FALSE)
{
$name = $data[0];
`$mobile = $data[1];
mysql_query("INSERT INTO `table` (`name`,`mobile`) VALUES ('".$name."','".$mobile ."') ");
}
fclose($source);
but it takes only last record...
Why not just add the whole file using MySQL rather than breaking it up by line? Something like
LOAD DATA INFILE 'filename.csv' INTO TABLE table FIELDS TERMINATED BY ',';
or if you've got specific columns that need to go into specific fields,
LOAD DATA INFILE 'filename.csv' INTO TABLE table FIELDS TERMINATED BY ',' (#c1, #c2)
set name = #c1, mobile = #c2;
Here is an easy way to do it, using PHP fgetcsv() and mysqli():
1. Connect to mysqli:
$mysqli = new mysqli("localhost", "my_user", "my_password", "my_db");
2. Open your CSV file:
$source = fopen('filename.csv', 'r') or die("Problem open file");
3. With a loop read the CSV and put your inserts queries in a variable:
$query="";
do {
if ($fields[0]) {
$query.="INSERT INTO `table_name` (`name`, `mobile`) VALUES
(
'".addslashes($fields[0])."',
'".addslashes($fields[1])."'
)
;";
}
} while ($fields = fgetcsv($source,1000,",","'"));
4. Excute your query and close mysqli:
if($mysqli->multi_query($query)) echo "Data successfuly added";
$mysqli->close();
fgetcsv documentation.
Myqli multiquery documentation
I have a .dat file that I've pulled from another application that has many rows of comma delimitated data. I opened the file with Excel, and saved it as a .csv. I am able to upload that file without issue to my database using LOAD DATA INFILE.
But here's where it gets weird. If I take the original file and try to copy, for example, 20 rows and make a new .csv file, when I upload it to the same database, same table schema, it will only load the first row. Did Excel do something with my formatting when I saved as a new .csv? It looks the same to me...I can't see a difference when opening with other text editors.
Here's my SQL for creating a new table for the data to be imported (I first check if the table already exists):
// Create table - Table name matches name of file without extension
$sql = "CREATE TABLE " . $file_name . " (
history_id INT,
stream_id INT,
record_time DATETIME,
ALEQ FLOAT,
APEAK FLOAT,
AFASTSPL FLOAT,
dose FLOAT,
color INT
)";
Here's my SQL for loading the .csv:
// Insert data from uploaded .csv file into table
$sql = "LOAD DATA LOCAL INFILE '" . $target_file . "'
INTO TABLE " . $file_name . "
FIELDS OPTIONALLY ENCLOSED BY '''' TERMINATED BY ','
LINES TERMINATED BY '\n'";
Here's what the .csv for the file that does work looks like:
1,1,'2014-08-28 07:00:31',35.2,56.0,35.1,0.0000000,0
2,1,'2014-08-28 07:00:31',35.1,56.0,35.4,0.0000000,0
3,1,'2014-08-28 07:00:31',35.1,56.0,35.2,0.0000000,0
4,1,'2014-08-28 07:00:31',35.3,56.0,35.0,0.0000000,0
5,1,'2014-08-28 07:00:31',35.2,56.0,35.3,0.0000000,0
6,1,'2014-08-28 07:00:31',35.2,56.0,35.2,0.0000000,0
And here's the .csv for the file that doesn't work:
60135,1,'2014-08-28 15:19:19',52,68.3,52.6,0.43186,0
60136,1,'2014-08-28 15:19:20',56.9,71.6,51.3,0.43186,0
60137,1,'2014-08-28 15:19:20',56.5,68.8,58.3,0.43186,0
60138,1,'2014-08-28 15:19:21',57,70.1,56.4,0.43186,0
60139,1,'2014-08-28 15:19:21',56.5,69.2,56.2,0.43186,0
60140,1,'2014-08-28 15:19:22',58.7,71.4,56.7,0.43186,0
If it only loads the first row then it is not detecting the line endings properly. I suggest using a hex editor to check what line endings you have in your source file.
Perhaps you saved as "CSV (Macintosh)" which uses \r as line endings instead. If this is the case, you should be able to use
LINES TERMINATED BY '\r'";
A more generic solution if you dont know the line endings...
MySQL has no automatic line ending detection and that is why you have to tell it how to parse the data. You can use this short function to do some basic frequency analysis.
function detectLineEndings($file)
{
$data = file_get_contents($file);
$le_linux = substr_count($data, "\n");
$le_windows = substr_count($data, "\r\n");
$le_mac = substr_count($data, "\r");
if ($le_linux > $le_windows && $le_linux > $le_mac) {
return '\n';
}
if ($le_mac > $le_linux && $le_mac > $le_windows) {
return '\r';
}
return '\r\n';
}
$endings = detectLineEndings($target_file);
$sql = "LOAD DATA LOCAL INFILE '" . $target_file . "'
INTO TABLE " . $file_name . "
FIELDS OPTIONALLY ENCLOSED BY '''' TERMINATED BY ','
LINES TERMINATED BY '" . $endings ."'";
I have one problem, I am totally beginner and I would like Insert or convert data, which I have on my PC like txt file to my tables created by Mysql language. Is it possible?
Table has same columns like txt file.
Thank you so much
You can write direct sql query Like this
LOAD DATA INFILE 'C://path/to/yourfilename.txt'
INTO TABLE 'database_name'.'table_name'
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\n\r'
(column1,column2)
Here i assumed that your fields are terminated by semi column.
for line termination Character and Escape sequence you can take reference from this thread
http://dev.mysql.com/doc/refman/5.1/en/load-data.html
You can write a simple program that loops through the content of the file and inserts the data into the database. Here is something to get you started.
This simply inserts the data you have put in the PHP code.
What you will want to do is to open a file and insert the data from it
mysql_connect('localhost', 'user', 'pass') or die(mysql_error());
mysql_select_db('db_name') or die(mysql_error());
$lines = file('company.txt');
$company_names = ""; $insert_string
= "INSERT INTO company(company_name) VALUES"; $counter = 0; $maxsize = count($lines);
foreach($lines as $line => $company) {
$insert_string .= "('".$company."')"; $counter++; if($counter <
$maxsize) { $insert_string .= ","; }//if }//foreach
mysql_query($insert_string) or die(mysql_error());
I have a .txt file that has a bunch of formatted data in it that looks like the following:
...
1 75175.18 95128.46
1 790890.89 795829.16
1 875975.98 880914.25
8 2137704.37 2162195.53
8 2167267.27 2375275.28
10 2375408.74 2763997.33
14 2764264.26 2804437.77
15 2804504.50 2881981.98
16 2882048.72 2887921.25
16 2993093.09 2998031.36
19 3004104.10 3008041.37
...
I am trying to load each row as an entry into a table in my database, where each column is a different field. I am having trouble getting mySQL to separate all of the data properly. I think the issue is coming from the fact that not all of the numbers are separated with an equidistant white-space amount.
Here are two queries I have tried so far (I have also tried several variations of these queries):
LOAD DATA LOCAL INFILE
'/some/Path/segmentation.txt'
INTO TABLE clip (slideNum, startTime, endTime)
SET presID = 1;
LOAD DATA LOCAL INFILE
'/some/Path/segmentation.txt'
INTO TABLE clip
FIELDS TERMINATED BY ' '
LINES TERMINATED BY '\n'
(slideNum, startTime, endTime)
SET presID = 1;
Any ideas how to get this to work?
These are what we call "fixed-width" records and LOAD DATA doesn't play well with them. Options:
Clean up data in Excel first, or
Load up the data to a temp table with only 1 column, shoving an entire text row into that column. Then you can use SUBSTR() and TRIM() to slice out the columns you need into the final table.
Or with user variables (#row) you can do it all within the LOAD DATA statement.
LOAD DATA LOCAL INFILE
'/some/Path/segmentation.txt'
INTO TABLE clip
(#row)
SET slideNum = TRIM(SUBSTR(#row,1,4)),
startTime = TRIM(SUBSTR(#row,5,13)),
endTime = TRIM(SUBSTR(#row,18,13))
;
LOAD DATA
CHARACTERSET AL32UTF8
INFILE 'DCF Master 14APR2013 VSPCFM_reduced size.txt'
INTO TABLE EMPLOYEE3
(
a = TRIM(SUBSTR(#row,1,11)),
b = TRIM(SUBSTR(#row,33,38)),
c = TRIM(SUBSTR(#row,70,86))
)
If you're on unix/linux then you can put it through sed to strip out spaces. The solution is here
You can programmatically replace spaces with a different delimiter. I decided to use PHP, you can also safely do it in Python
<?php
$mysqli = new mysqli(
"***",
"***",
"***",
"***",
3306
);
mysqli_options($mysqli, MYSQLI_OPT_LOCAL_INFILE, true);
if (mysqli_connect_errno()) {
printf("Connect failed: %s\n", mysqli_connect_error());
exit();
}
function createTempFileWithDelimiter($filename, $path){
$content = file_get_contents($filename);
$replaceContent = preg_replace('/\ +/', ',', $content); // NOT \s+
$onlyFileName = explode('\\',$filename);
$newFileName = $path.end($onlyFileName);
file_put_contents($newFileName, $replaceContent);
return $newFileName;
}
$pathTemp = 'C:\\TempDir\\';
$pathToFile = 'C:\\some\\Path\\segmentation.txt';
$file = createFileWithDelimiter($pathToFile, $pathTemp);
$file = str_replace(DIRECTORY_SEPARATOR, '/', $file);
$sql = "LOAD DATA LOCAL INFILE '".$file."' INTO TABLE `clip`
COLUMNS TERMINATED BY ','
LINES TERMINATED BY '\n' // or '\r\n'
(slideNum, startTime, endTime)
SET presID = 1;";
if (!($stmt = $mysqli->query($sql))) {
echo "\nQuery execute failed: ERRNO: (" . $mysqli->errno . ") " . $mysqli->error;
};
unlink($file);
?>
Don't use '/\s+/' in preg_replace because \s matches any whitespace character (equivalent to [\r\n\t\f\v ]) and the formatting will change, columns and line breaks will disappear.
Is there a way to dynamically specify a file name in the LOAD DATA INFILE? Can it be parameterized like for instance (syntax may be incorrect) LOAD DATA INFILE '$filename'?
A citation from MySQL documentation:
The LOAD DATA INFILE statement reads rows from a text file into a table at a very high speed. The file name must be given as a literal string.
That means that it can not be a parameter of a prepared statement. But no one forbids to make the string interpolation while the statement is just a string in your PHP code.
Unfortunately, this feature is not yet supported in MySQL and is currently listed as bug/feature request #39115 http://bugs.mysql.com/bug.php?id=39115
Or you can make a temporary copy of the file (BATCH example):
LOAD_DATA.bat
COPY %1 TempFileToLoad.csv
mysql --user=myuser --password=mypass MyDatabase < ScriptLoadMyDatabase.sql
DEL TempFileToLoad.csv
the SQL (for info) :
ScriptLoadMyDatabase.sql
load data infile 'TempFileToLoad.csv' IGNORE
into table tLoad
FIELDS TERMINATED BY ';' OPTIONALLY ENCLOSED BY '"'
lines terminated by '\r\n'
IGNORE 1 LINES
(#DateCrea, NomClient, PrenomClient, TypeMvt, #Montant, NumeroClient)
set DateCrea = str_to_date(#DateCrea, '%Y-%m-%d'), Montant = (round(#Montant / 1000)*2) ;
And finished to put a link to the BAT file in SendTo windows folder.
If you're asking if it can be used in a script; you can do some thing like this with php:
<?php
$mysqli = new mysqli("host", "user", "pwd", "db");
/* check connection */
if (mysqli_connect_errno()) {
printf("Connect failed: %s\n", mysqli_connect_error());
exit();
}
$sql = "CREATE TABLE number1 (id INT PRIMARY KEY auto_increment,data TEXT)";
if ($result = $mysqli->query($sql)) {
} else {
printf("<br>%s",$mysqli->error);
}
$host = $_SERVER['HTTP_HOST'];
$uri = rtrim(dirname($_SERVER['PHP_SELF']), '/\\');
$filename = "data.csv";
$sql = "LOAD DATA LOCAL INFILE '$host$uri$filename' INTO TABLE number1";
if ($result = $mysqli->query($sql)) {
} else {
printf("<br>%s",$mysqli->error);
}
// Close the DB connection
$mysqli->close();
exit;
%>
If the file is in the same folder as the script just use $filename a instead of $host$uri$filename. I put this together quick from a couple scripts I'm using, sorry if it doesn't work without debug, but it should be pretty close. It requires mysqli.