I am writing a program which deals with data transformations via MySQL and it deals with big files.
I made a question earlier about another issue I was having, while I was trying out someone's answer I got the following error
[MySQL][ODBC 5.3(a) Driver][mysqld-5.5.5-10.1.9-MariaDB]File 'C:\xampp\mysql\data\ingram\' not found (Errcode: 2 "No such file or directory")
I am certain that directory exists and when I change the code to its original state it works perfectly.
What is going on there?
This is the piece of code that gives me the problem
Cmd.CommandText = String.Format("LOAD DATA INFILE ""{0}"" IGNORE INTO TABLE libros_nueva FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '""' ESCAPED BY '""' LINES TERMINATED BY '\r\n';", filepath)
Cmd.Execute()
Any help will be appreciated!
Given the salient portion of the error message:
File 'C:\xampp\mysql\data\ingram\' not found (Errcode: 2 "No such file or directory")
I am pretty sure you are passing just a path when a full path and file name are required. There is certainly no file name in the path it echoed back.
Can you please explain it [MySqlBulkLoader] to me?
Another way to import is to use MySqlBulkLoader from the MySql.Data.MySqlClient namespace:
' columns in the order they appear in the CSV file:
Dim cols As String() = {"Name", "Descr", "`Group`", "ValueA",
"Bird", "Fish", "zDate", "Color", "Active"}
Dim csvFile As String = "C:\Temp\mysqlImport.csv"
Dim rows As Int32
Using dbcon As New MySqlConnection(MySQLConnStr)
Dim bulk = New MySqlBulkLoader(dbcon)
bulk.TableName = "importer"
bulk.FieldTerminator = "," ' this is a CSV
bulk.LineTerminator = "\r\n" ' == CR/LF
bulk.FileName = csvFile ' full file path name to CSV
bulk.NumberOfLinesToSkip = 0 ' has a header?
bulk.Columns.Clear()
For Each s In cols
bulk.Columns.Add(s) ' tell MySQL the order
Next
rows = bulk.Load() ' Make it so.
End Using
Times to import 100k rows: 3619, 2719 and 2987 ms. There is also a LoadAsync method which may be of interest given your last question.
If there are data transforms to do before the insert, CSVHelper can provide an easy way to load records so you can do whatever needs to be done, then use normal SQL Inserts to update the DB.
Part of this answer shows using CSVHelper to import into Access in batches of 50k and which was pretty fast.
Related
This issue is rather old for me, but so far it was bearable. But now I need to get rid of it. I am delevoping Winforms app, which among others has the function to import CSV files to MySQL database on another server.
As far as the source CSV file is not opened in Excel there are absolutely no issues with importing. Also if file is opened in Notepad or Notepad++ all is fine. But if file is opened in Excel it cannot be imported in database. When showing processlist in MySQL client you can see the state as "Receiving from client" for a minute or two and then VisualStudio ends execution with exception 'ContextSwitchDeadlock' (the complete exception message is shown below). Just for comparison if I import this CSV file using HeidiSQL there are no issues when file is opened in Excel.
The code for loading is below:
importStr = "Load data local infile '" + selectedPath.Replace("\", "\\") + fileName +
"' into table " + targetTable + columns terminated by '" + TextBox1.Text + "'
optionally enclosed by '""' lines terminated by '" + TextBox2.Text + "' ignore 1 lines"
This results in following command:
Load data local infile 'E:\\New\\def_charts_cell_3ser.csv'
into table def_charts_cell_3ser_temp
columns terminated by ','
optionally enclosed by '"'
lines terminated by '\r\n'
ignore 1 lines
Importing string is used as follows:
Queries.ExecQueryAdminNoSess(importStr, MysqlConn)
The definition of sub is below:
Public Shared Sub ExecQueryAdminNoSess(queryString As String, MysqlConn As MySqlConnection)
Dim SQL As New MySqlCommand
Try
MysqlConn.Open()
SQL.Connection = MysqlConn
SQL.CommandText = queryString
SQL.CommandTimeout = 1200
SQL.ExecuteNonQuery()
Catch Err As MySqlException
MsgBox("Error: " & Err.Number & " - " & Err.Message)
Finally
MysqlConn.Close()
MysqlConn.Dispose()
End Try
End Sub
Full text of exception:
Managed Debugging Assistant 'ContextSwitchDeadlock'
Message=Managed Debugging Assistant 'ContextSwitchDeadlock' : 'The CLR has been unable to transition from COM context 0xf35008 to COM context 0xf34f50 for 60 seconds. The thread that owns the destination context/apartment is most likely either doing a non pumping wait or processing a very long running operation without pumping Windows messages. This situation generally has a negative performance impact and may even lead to the application becoming non responsive or memory usage accumulating continually over time. To avoid this problem, all single threaded apartment (STA) threads should use pumping wait primitives (such as CoWaitForMultipleHandles) and routinely pump messages during long running operations.'
The work-around that I implemented and solved my question is to copy the CSV file in the same folder, but with different name. After that this new file is loaded in MySQL database and finally this new file is deleted. Below is the code.
If File.Exists(selectedPath + fileNameOrig) Then
My.Computer.FileSystem.CopyFile(selectedPath + fileNameOrig, selectedPath + fileNameCopy),
FileIO.UIOption.AllDialogs, FileIO.UICancelOption.DoNothing)
*do file contents injection in database*
My.Computer.FileSystem.DeleteFile(selectedPath + fileNameCopy)
End If
I am trying to use luasql.mysql and execute multiple INSERT statements from a file to a mysql database. I can not seem to figure out how to use a string variable instead of using a string. I want to execute from a for loop, looping though a file of strings. Any help would be appreciated.
mysql = require "luasql.mysql"
local env = mysql.mysql()
local conn = env:connect('mydb','myusername','mypassword','myip')
print(env,conn)
file = io.open("sqldumps.sql")
lines = file:lines()
print("Contents of file:");
for line in lines do
status,errorString = conn:execute( '"line"' )
print(status,errorString )
end
It's actually really simple:
for line in file:lines() do
status, errorString = conn:execute(line)
print(status, errorString)
end
Lua knows that line is a string, so you don't need any quotes around it. In fact, if you add them, Lua will treat it as the actual text "line", not the content of the line variable.
A hint on for loops: file:lines() just so happens to return a single function that you can use in your for loop, but this isn't always the case.
I've got 200k csv files and I need to import them all to a single postgresql table. It's a list of parameters from various devices and each csv's file name contains device's serial number and I need it to be in one of the colums for each row.
So to simplify, I've got few columns of data (no headers), let's say that columns in each csv file are: Date, Variable, Value and file name contains SERIALNUMBER_and_someOtherStuffIDontNeed.csv
I'm trying to use cygwin to write a bash script to iterate over files and do it for me, however for some reason it won't work, showing 'syntax error at or near "as" '
Here's my code:
#!/bin/bash
FILELIST=/cygdrive/c/devices/files/*
for INPUT_FILE in $FILELIST
do
psql -U postgres -d devices -c "copy devicelist
(
Date,
Variable,
Value,
SN as CURRENT_LOAD_SOURCE(),
)
from '$INPUT_FILE
delimiter ',' ;"
done
I'm learning SQL so it might be an obvious mistake, but I can't see it.
Also I know that in that form I will get full file name, not just the serial number bit I want but I can probably handle that somehow later.
Please advise.
Thanks.
I dont think there is a CURRENT_LOAD_SOURCE() function in postgres. A work-around is to leave the name-column NULL on copy, and patch is to the desired value just after the copy. I prefer a shell here-document because that make quoting inside the SQL body easier. (BTW: for 10K of files, the globbing needed to obtain FILELIST might exceed argmax for the shell ...)
#!/bin/bash
FILELIST="`ls /tmp/*.c`"
for INPUT_FILE in $FILELIST
do
echo "File:" $INPUT_FILE
psql -U postgres -d devices <<OMG
-- I have a schema "tmp" for testing purposes
CREATE TABLE IF NOT EXISTS tmp.filelist(name text, content text);
COPY tmp.filelist ( content)
from '/$INPUT_FILE' delimiter ',' ;
UPDATE tmp.filelist SET name = '$FILELIST'
WHERE name IS NULL;
OMG
done
For anyone interested in an answer, I've used a python script to change file names and then another script using psycopg2 to connect to the database and then done everyting in one connection. Took 10 minutes instead of 10 hours.
Here's the code:
Renaming files (also apparently to import from CSV you need all the rows to be filled and the information I needed was in first 4 columns anyway, therefore I've put together a solution to generate whole new CSVs instead of just renaming them):
import os
import csv
path='C:/devices/files'
os.chdir(path)
i=0
for file in os.listdir(path):
try:
i+=1
if i%10000 == 0:
#just to see the progress
print(i)
serial_number = (file[:8])
creader = csv.reader(open(file))
cwriter = csv.writer(open('processed_'+file, 'w'))
for cline in creader:
new_line = [val for col, val in enumerate(cline) if col not in (4, 5, 6, 7)]
new_line.insert(0, serial_number)
#print(new_line)
cwriter.writerow(new_line)
except:
print('problem with file: ' + file)
pass
Updating database:
import os
import psycopg2
path="C:\\devices\\files"
directory_listing = os.listdir(path)
conn = psycopg2.connect("dbname='devices' user='postgres' host='localhost'")
cursor = conn.cursor()
print(len(directory_listing))
i=100001
while i < 218792:
current_file=(directory_listing[i])
i+=1
full_path = "C:/devices/files/" + current_file
with open(full_path) as f:
cursor.copy_from(file=f, table='devicelistlive', sep=",")
conn.commit()
conn.close()
Don't mind while and weird numbers, it's just because I was doing it in portions for testing purposes. Can easily be replaced with for
I am trying to convert and evtx log file to csv from log parser 2.2. I just want to copy all of the data into a csv.
LogParser "Select * INTO C:\Users\IBM_ADMI
N\Desktop\sample.csv FROM C:\Users\IBM_ADMIN\Desktop\Event
Logs\sample.evtx" -i:EVTX -o:csv
But I am getting the error below.
Error: Syntax Error: extra token(s) after query: 'Logs\sample.evtx'
Please assist in solving this error.
I know this has been a year but if you (or other people) still need it and for sake of reference, this is what I do:
LogParser "Select * INTO C:\Users\IBM_ADMIN\Desktop\sample.csv FROM 'C:\Users\IBM_ADMIN\Desktop\Event Logs\sample.evtx'" -i:evt -o:csv
Correct input type is evt, not evtx.
If there is space in the Event Logs folder, enclose with single quote.
The Problem was due to the extra space in between the folder name Event Logs. Changed the folder name to a single workd and it worked.
you have to convert .evtx file to .csv than you can read from this .csv file.
like this .enter image description here
//String command = "powershell.exe your command";
//called the PowerShell from java code
String command = "powershell.exe Get-WinEvent -Path C:\windows\System32\winevt\Logs\System.evtx | Export-Csv system.csv";
File seys = new File("system.csv");
Process powerShellProcess = Runtime.getRuntime().exec(command);
I try to insert the data from a csv file into a mysql database using a powershell script. When using a (dummy) variable in the LOAD DATA query I run into troubles.
Reproducible example:
Create a Mysql database and table with
CREATE DATABASE loadfiletest;
USE loadfiletest;
CREATE TABLE testtable (field1 INT, field2 INT DEFAULT 0);
Create a csv file named loadfiletestdata.csv containing
1,3
2,4
Create the powershell script (don't forget to change the db password and possibly the username)
[system.reflection.assembly]::LoadWithPartialName("MySql.Data")
$mysqlConn = New-Object -TypeName MySql.Data.MySqlClient.MySqlConnection
$mysqlConn.ConnectionString = "SERVER=localhost;DATABASE=loadfiletest;UID=root;PWD=pwd"
$mysqlConn.Open()
$MysqlQuery = New-Object -TypeName MySql.Data.MySqlClient.MySqlCommand
$MysqlQuery.Connection = $mysqlConn
$MysqlQuery.CommandText = "LOAD DATA LOCAL INFILE 'C:/path/to/files/loadfiletestdata.csv' INTO TABLE loadfiletest.testtable FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '""' LINES TERMINATED BY '\r\n' (field1, field2)"
$MysqlQuery.ExecuteNonQuery()
Put everything in the folder C:/path/to/files/ (should also be your path in the powershell script) and run the script. This populates the table testtable with
field1 field2
1 3
2 4
as one would expect. This implies that quotes and such are like they should be. Each time the script is executed, those values are inserted in the table. Now, when I replace in the one but last line of the powershell script (field1, field2) by (field1, #dummy), I would expect that the values
field1 field2
1 0
2 0
are inserted into the table. However, I receive the error
Exception calling "ExecuteNonQuery" with "0" argument(s): "Fatal error encountered during command execution."
At C:\path\to\files\loadfiletest.ps1:8 char:1
+ $queryOutput = $MysqlQuery.ExecuteNonQuery()
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : MySqlException
When running the query with #dummy from a mysql client it works. Also the syntax looks the same to me as what can be found in the mysql manual (somewhere in the middle of the page, look for #dummy).
A few further experiment that I did, suggest that any LOAD DATA query containing a variable #whatever gives the error.
So the questions:
Why doesn't it work?
Is there a way to execute a LOAD DATA query with (dummy) variables from powershell?
If not, is there an elegant workaround?
Obvious workarounds are creating an intermediate csv file according to the layout of the table or creating an intermediate table matching the layout of the csv file. However that seems ugly and cumbersome for something that imho should "just work".
Note: The present question is a follow up and generalization of this question. I chose to start a new one since replacing the old content would make the answers already given obsolete and adding the content of this question would make the old question veeeeery long and full of useless sidetracks.
I know this is old, but I had the same problem and I found the solution here:
http://blog.tjitjing.com/index.php/2009/05/mysqldatamysqlclientmysqlexception-parameter-id-must-be-defined.html
Quoting from the above blog:
"Starting from version 5.2.2 of the Connector you should add the Allow User Variables=True Connection String Setting in order to use User Defined Variables in your SQL statements.
Example of Connection String:
Database=testdb;Data Source=localhost;User Id=root;Password=hello;Allow User Variables=True"
Thank you for down-voting my answer.