I have the following file dumped daily into one of our online directories:
dat-part2-489359-43535-toward.txt
The numbers change each day randomly.
I have the following code to try and LOAD the file:
mysql_query("LOAD DATA LOCAL INFILE 'dat-part2-%-toward.txt'
REPLACE INTO TABLE my_table
FIELDS TERMINATED BY ',' ENCLOSED BY ''
LINES TERMINATED BY '\r\n'
IGNORE 1 LINES") or die(mysql_error());
And of course no luck. What's the best way to do this?
assuming this is a scheduled job, why not check the directory for the most recent file that matches your filename template. Store the name of said file in a variable and then sub the variable into your query. Check out glob()
I would do it via a shell script or windows cmd file. This assumes you created the file with mysqldump or some program that creates a valid sql script.
You can run dir /B in windows commend to get a directory of files. So do a dir /B > input.txt then use Windows scripting to read the input file and for each line, pipe the file in using the mysql client.
# echo off set infile= %%1
for /f "eol= tokens=*
delims= usebackq" %%i in (%infile%)
do ( mysql -u userName --password=pwd < %%i )
It's been a long time since I wrote any windows scripts, but that should give you an idea of an approach.
Related
I am new to SQL Server and SSIS. I want to shedule the loading of .csv file into SQL Server.I want to run the loading for a specific time daily.Please help.Thanks in Advance.
You can do this by first creat the table(s) in SQL with the appropriate data types. Then by importing into SQL first and then write your script to update the table but a .bat file would require in order to run your scripts. The following is an example command for a .bat file: sqlcmd -S ServerName -U UserName P Password -i "C:\folder\update.sql"-o "C:\folder\output.txt"
You will need to write the .csv file to a text file and change the extension to .sql
You can use the "Bulk Insert Command"
Example:
BULK INSERT FILENAME
DATABASENAME.DBO.TABLENAME
FROM 'C:\sqlscript\fileName.csv'
WITH
(
FIELDTERMINATOR = ',',
ROWTERMINATOR = '\n'
)
GO
Hope this helps
I use the following .bat script
set varSearch="C:\Users\User1\Desktop\Test-folder\*.crypt8"
for /f %%i in ('dir %varSearch% /B ') do set varSearch= %%i
WhatsAppViewer.exe -decrypt8 %myName% key exp.db
sqlite3.exe exp.db<command.txt
cd C:\xampp\mysql\bin
mysql -u admin -p1234 < query.txt
The basic function is to find a file thats ending with .crypt8, decrypt it, save as csv and import to mysql. Its working correctly
But i need some extra features
Case1
The folder contains more than 1 file, and every file has to be processed, but only once
Case 2
Everyday at least one file gets added. It would be superb if the .bat could be scheduled as a task, and run every night and just process the new added files.
Does anybody has a solution for this?
Case 2
The forfiles command processes groups of files based on date. This does files made today only.
forfiles /d 0 /m *.crypt8 /c "cmd /c echo #fname in #path"
Case 1
Your code has errors, it may work but not under all conditions.
The easist way is to put the sequence of commands in a batchfile for a file (%1) which is passed on command line, and use forfiles to call it.
i trying load a CSV in a table.
I have my CSV in a folder of my server. (wwww.myweb.com/temp/file.csv)
I use this sentence:
LOAD DATA INFILE 'http://wwww.myweb.com/temp/file.csv' INTO TABLE ga_tmpActivosDocumentos FIELDS TERMINATED BY ';' LINES TERMINATED BY '\n' IGNORE 1 LINES (idTipoSuelo,C_Latitud,C_Longitud,Referencia,Zona,idProvincia,Poblacion,TituloActivo,Descripcion,Superficie,Gastos,Equipamiento,EquipamientoEN,GestionDocumental,PrecioVenta,CampoLibre1_Texto,CampoLibre1_Titulo,CampoLibre1_TextoEN,CampoLibre1_TituloEN,Activo, IMG1,IMG_Desc1,IMG_Desc1EN,IMG2,IMG_Desc2,IMG_Desc2EN,IMG3,IMG_Desc3,IMG_Desc3EN,IMG4,IMG_Desc4,IMG_Desc4EN,DOC1,DOC_Desc1,DOC_Desc1EN,DOC2,DOC_Desc2,DOC_Desc2EN,DOC3,DOC_Desc3,DOC_Desc3EN,DOC4,DOC_Desc4,DOC_Desc4EN,URL1,URL_Desc1,URL_Desc1EN,URL2,URL_Desc2,URL_Desc2EN) SET idCliente = 23
The sentence not work for me. I try to change the path to .../temp/file.csv, and other combinations but not work.
Also use "LOAD DATA LOCAL INFILE" but does not work.
I have read other topics, but only look examples with a relative URL, never absolute.
Thanks, and sorry for my english
#vadym-tyemirov answer works but if you don't want to create a temporary file, one solution is to load it from '/dev/stdin' and pipe it to the mysql cli:
wget -O - 'http://wwww.myweb.com/temp/file.csv' |
mysql \
--user=root \
--password=password \
--execute="LOAD DATA LOCAL INFILE '/dev/stdin' INTO TABLE table_name"
Save CSV file on your LOCAL computer.
Connect to the DB from your LOCAL computer
Issue the following command: load data LOCAL infile '/tmp/file.csv' INTO TABLE table_name;
You can also load data files by using the mysqlimport utility; it operates by sending a LOAD DATA INFILE statement to the server. The --local option causes mysqlimport to read data files from the client host.
MySQL cannot access the file in that location. Try moving it somewhere simple like /tmp (or copy it) on the local filesystem, and not via a URL parameter.
The MySQL process likely cannot load the folders BEFORE "temp/file.csv"
I have looked all over and found no solution, any help on this would be great.
Query:
LOAD DATA INFILE '/Users/name/Desktop/loadIntoDb/loadIntoDB.csv'
INTO TABLE `tba`.`tbl_name`
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\r\n'
IGNORE 1 LINES
(
field1, field2, field3
)
Error:
Can't get stat of '/Users/name/Desktop/loadIntoDb/loadIntoDB.csv' (Errcode:2)
NOTE:
I'm running MySQL Query browser on OSX 10.6.4 connecting to MySQL 5.x
Things I've tried:
Drag-n-drop
Chmod 777
Put in a folder with 777 permissions
as well as the file having 777
permissions
try to use LOAD DATA LOCAL INFILE instead of LOAD DATA INFILE
otherwise check if apparmor is active for your directory
I had a similar problem. The resolution was a mildly ugly hack, but much easier to remember than apparmor workarounds provided that you can 'sudo'. First, I had to put the input file in the mysql sub-directory for the database I was using:
sudo cp myfile.txt /var/lib/mysql/mydatabasename
This does a copy and leaves 'root' as the file owner. After getting into mysql and doing a USE mydatabasename, I was able to populate appropriate table using
LOAD DATA INFILE 'mytabdelimitedtextfile.txt' INTO TABLE mytablename;
Using --local parameter will help with this.
Example: mysqlimport --local databasename file.txt -p
source:
http://dev.mysql.com/doc/refman/5.1/en/load-data.html
"The --local option causes mysqlimport to read data files from the client host"
For me, copying the contents to /tmp and using that as the source folder did the trick.
I use MariaDB, and my version does not allow using the "LOCAL" modifier.
Interestingly, giving read-write access to the CSV folder did not work either.
I had the same problem while populating a table in mysql on a AWS instance.
In my case i had the csv file in the instance itself.
Putting the Absolute path solved my problem.
Here's the line from MySQL documentation
If LOCAL is specified, the file is read by the client program on the client host and sent to the server. The file can be given as a full path name to specify its exact location. If given as a relative path name, the name is interpreted relative to the directory in which the client program was started.
http://dev.mysql.com/doc/refman/5.7/en/load-data.html
I've been playing with mysqlimport and I've run into the restriction where the filename has to be the same as the table name. Is there any way to work round this?
I can't rename the file as it is used by other processes and I don't want to copy the file as there will be many of them, some being very large.
I want to use mysqlimport not LOAD INFILE.
EDIT: Unfortunately this needs to run on windows so no tricks with symbolic links I'm afraid.
You didn't say what platform you are on. On unix you can create a symbolic link to the file:
ln -s filename.txt tablename.txt
Then use that in the mysqlimport command.
But mysqlimport is just a command line interface to LOAD INFILE so you could also do this on the command line:
mysql -e "load data infile 'filename' into table TBL_NAME" dbname
mysqlimport uses the filename to determine the name of the table into which the data should be loaded. The program does this by stripping off any filename extension (the last period and anything following it); the result is then used as the table name. For example, mysqlimport treats a file named City.txt or City.dat as input to be loaded into a table named City.
Have you tried using the alias command, assuming you are on a Linux system?
Just create a symbolic link:
ln -s /tmp/real_file.txt /tmp/your_table_name.txt