Further to an earlier question here
Fast import of csv file into access database via VB.net 2010
I tried using the following code in my .NET application (VB.NET 2010)
cmd.CommandText =
"SELECT F1 AS id, F2 AS firstname " &
"INTO MyNewTable " &
"FROM [Text;FMT=Delimited;HDR=No;CharacterSet=850;DATABASE=C:\__tmp].table1.csv;"
and it seemed to work, but when I opened the database in Access the table showed garbled characters.
I think maybe CharacterSet=850 is not the correct setting for my CSV file. I tried searching for a character set list, but I couldn't find it.
My .csv file uses UTF-8. What should I use for the CharacterSet number?
The CharacterSet number for UTF-8 is CharacterSet=65001, so your CommandText should be
cmd.CommandText =
"SELECT F1 AS id, F2 AS firstname " &
"INTO MyNewTable " &
"FROM [Text;FMT=Delimited;HDR=No;CharacterSet=65001;DATABASE=C:\__tmp].table1.csv;"
Note also that this approach requires that the UTF-8 file be saved without a BOM (byte order mark), which is unusual for the Windows platform. (If the file does include a BOM then the first record will be imported as blank fields.)
Related
How to convert a DBF to CSV?
I need, use this library but it gave error: http://pythonhosted.org/dbf
import dbf
dbf.export('crop1-fx')
print 'Done'
"C:\Users\User\Anaconda2\python.exe"
"C:/Users/User/Desktop/Python/23/dbf/insertValuesDBF.py" Traceback
(most recent call last): File
"C:/Users/User/Desktop/Python/23/dbf/insertValuesDBF.py", line 3, in
dbf.export('crop1-fx') File "C:\Users\User\Anaconda2\lib\site-packages\dbf\ver_2.py", line 7824,
in export
table = source_table(table_or_records[0]) File "C:\Users\User\Anaconda2\lib\site-packages\dbf\ver_2.py", line 7956,
in source_table
table = thingie._meta.table() AttributeError: 'str' object has no attribute '_meta'
Process finished with exit code 1
You almost had it:
import dbf
db = dbf.Table('crop1-fx')
dbf.export(db)
The above will create a crop1-fx.csv file; however, I'm not sure this will work with a 24-digit numeric field in the table.
To convert a .DBF file to .CSV, download dBASE III PLUS or any other dBASE
software available on NET. Please note I am referring to 16 bit platform on
DOS.
Once dBASE is downloaded, go to the DOT prompt and give the following commands:
Type
use <the dbf file in question without the extension .dbf>
You will see the name of the dbf file on the display bar
Then, type "copy to" <file name you want, limited to 8 characters>
"delimited"
Now the data in the dbf file is sent to a TEXT file with data in each field surrounded by " " (double inverted commas) and separated by , (comma)
Now this file can be used to export the data to any other DATABASE SYSTEM which has got provision to convert this .CSV or DELIMITED FILE to the new database.
If only comma-separated file without the " " marks are required a procedure
can be written in dBASE to achieve that also.
I am using MS Access and have to create a file from a query, some of the fields are quite large and the program I am importing into only allows a max length of 65 characters. How do I insert line feed into those fields. The information is already in access so I cant do it by data validation on the fields.
Try this:
SELECT Left(LongLine, 65) & Chr(13) + Chr(10) & Right(LongLine, Len(LongLine) - 65) As _Line
FROM YourTable
I'm trying to save some data gathered from fields in MySQL db. Text contains some Polish characters, but Livecode sends all Polish chars as '?'. Here's part of my code:
Declare variable
put the unicodeText of field "Title" into tTitle
put uniEncode(tTitle, "UTF8") into tTitle
Send this to db:
put "UPDATE magazyn SET NAZWA='" & tTitle & "'" into tSQLStatement
revExecuteSQL gConnectionID,tSQLStatement, "SET NAMES 'utf8'"
For example, word "łąka" is saved as "??ka". I've tried uniEncode, uniDecode, everything is going wrong.
Don't use any encoders/decoders. They will only add to the confusion.
When trying to use utf8/utf8mb4, if you see Question Marks (regular ones, not black diamonds),
The bytes to be stored are not encoded as utf8. Fix this. (Getting rid of the encoders may fix it.)
The column in the database is CHARACTER SET utf8 (or utf8mb4). Fix this.
Also, check that the connection during reading is utf8. I don't know the details of "Livecode"; look in its documentation. If you can't find anything, execute this SQL after connecting: SET NAMES utf8.
Problem solved! Here's code:
get the unicodeText of field "Title"
put unidecode(it,"polish") into tTitle
it will save polish characters in a strange version, but for downloading i'm using this code:
set the unicodetext of fld "List" to uniencode(tList,"polish")
tList variable contains all data gathered from MySQL
Ensure the column in the database is set to utf8 encoding.
Starting with LiveCode 7 all text in fields is unicode, specifically UTF-16. Before you send the text out to any external file or datastore, you need to encode it as UTF-8 (or whatever format you want to store it in. Use the LiveCode textEncode() function for this:
put textEncode(field "Title","utf-8") into tTitle
put "UPDATE magazine SET nazwa = :1" into tSQLStatement
revExecuteSQL gConnectionID, tSQLStatement, "tTitle"
Note: It's also a good idea to use the :N variable substitution method to reduce the risk of SQL code injection attacks.
When you read the data from the database use textDecode to convert back to UTF-16:
put textDecode(tRawDataFromDB,"UTF-16") into old tTitle
I'm looking for a clever way to extract 500 plus lines of data from an excel spreadsheet and enter is into my database.
The spreadsheet is like this
My table 'tbl_foot_teams' is set out as
id | name | rating
Quite simply, I need to enter get the two columns from the spreadsheet into the database fields name and rating.
Is there any efficient way to achieve this?
Individually, it will take me a ridiculous amount of time!
Thanks
Save Excel file as CSV and use LOAD DATA INFILE command to import data.
Your excel file has no id field. Make id field in the table as AUTO_INCREMENT, and use command like this -
LOAD DATA INFILE 'file_name.csv' INTO TABLE tbl_foot_teams
FIELDS TERMINATED BY ','
LINES TERMINATED BY '\r\n'
-- IGNORE 1 LINES -- if csv file has column headers
(name, rating)
SET id = NULL; -- this will set unique value for each row
Also, have a look at GUI Data Import tool (Excel or CSV format) in dbForge Studio for MySQL.
In phpmyadmin you have an Import From Excel option.
If you don't have one, you may have Import From CSV, so just convert the spreadsheet to CSV.
If you have none of above, you can write a php function that opens a text file, makes explode by rows and then explode by values
If we are talking about 50 rows, you can create easily a new column on spreadsheet with a formula to concatenate your values to a insert statement. Something like:
=concat( "insert into tbl_foot_teams ( name , rating) values ( " , $b8 , " ...
then, copy paste calculate formula text result on your database.
You don't specify what database you're using, but an easy way to do this with MySQL would be to export the spreadsheet as a csv file and then import to MySQL with mysqlimport.
This is described in a comment on this MySQL page, from user Philippe Jausions:
If you are one of the many people trying to import a CSV file into
MySQL using mysqlimport under MS-Windows command/DOS prompt, try the
following:
mysqlimport --fields-optionally-enclosed-by=""" --fields-terminated-by=, --lines-terminated-by="\r\n" --user=YOUR_USERNAME --password YOUR_DATABASE YOUR_TABLE.csv
Between quotes " and backslashes \ it can really give you a hard time
finding the proper combination under Windows...
I usually run this command from the folder containing the
YOUR_TABLE.csv file.
If you have a header in your .csv file with the name of columns or
other "junk" in it, just add a --ignore-lines=X to skip the first X
lines (i.e. --ignore-lines=1 to skip 1 line)
If your fields are (optionally) enclosed by double-quotes " and which
themselves are doubled inside a value (i.e. a double double-quote "" =
1 double-quote ") then also use --fields-escaped-by=\ (default) and
NOT --fields-escaped-by="""
Working from the Excel end, you can use ADO, for example Excel VBA: writing to mysql database
Dim cn As ADODB.Connection
''Not the best way to get the name, just convenient for notes
strFile = Workbooks(1).FullName
strCon = "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" & strFile _
& ";Extended Properties=""Excel 8.0;HDR=Yes;IMEX=1"";"
Set cn = CreateObject("ADODB.Connection")
''For this to work, you must create a DSN and use the name in place of
''DSNName, however, you can also use the full connection string
strSQL = "INSERT INTO [ODBC;DSN=DSNName;].NameOfMySQLTable " _
& "Select AnyField As NameOfMySQLField FROM [Sheet1$];"
cn.Execute strSQL
After converting to a CSV file, you can import into any database that supports importing from CSV files (MySQL, PostgreSQL, etc.) but you would have to do this from the command-line:
Importing CSV files in PostgreSQL
Import CSV to Oracle table
MySQL 5.1 LOAD DATA INFILE syntax
Import CSV File into MSSQL
You can use a programming language such as Python and use a library for it such as the Excel spreadsheet reading library and then another library for interfacing with your SQL database.
You connect to the data, load the Excel file, and loop through each row and extract whichever column data you want. Then you take that data and execute the INSERT statement.
If you have an installation of phpMyAdmin on a server, you can use the Import from CSV option though you would first have to re-save your Excel spreadsheet as a CSV (Comma-Separated Values) file.
I'm working with a client who has an existing system, built on what is apparently a Paradox database. I've got the database, in the form of a zip file containing .DB, .MB and .PX files, one for each table.
I need to take (some) of this data and import it in to a Web application that's using MySQL. Does anybody have a way for me to extract this data, that doesn't involve installing Paradox?
If not, does Paradox export in some readable format? Either as SQL or something that can be parsed reasonably easily? The person in charge of this system for my client is a volunteer (they're a non-profit), so I'd like to go to him with a solution - because last time I asked for the data, I got this, which is clearly no good.
The wikipedia article about Paradox lists two other things, that might be interessant, both under GPL license:
pxlib: Library to read and write Paradox databases
pxtools: convert a Paradox-database into a SQL-database
And if you have Delphi and want to write a converter yourself (which would need the BDE to work) you can take a look at this article or at the source code of ConvertCodeLib on this web site. Both make use of TClientDataset, which can write a CDS (binary format) or an XML file.
Both the Paradox for DOS and Paradox for Windows platforms will export data tables in Delimited Text, Fixed-Length Text, and Lotus 1-2-3 formats. The older Paradox for DOS also writes Lotus Symphony, while the slightly less antique Paradox for Windows does a passable Excel 5.
However, someone will have to sit down and export the tables one by one, or write a script do to it. Of course you'd need to have Paradox installed to write the script.
-Al.
MS has instructions for using the MS Jet driver to read data from files produced by Paradox 3-5. That can act as (at least) an ODBC driver, so you can use it to read a Paradox file from just about anything that knows how to use ODBC.
You have a few options:
Get your hands on the original Paradox software, and use it to export the database into CSV format. Unfortunately, Borland no longer sells it and the most recent version doesn't run well on Windows XP or above.
Access the database using either a Paradox or dBase/xBase ODBC driver. Paradox and xBase are very similar, so you may be able to extract the data using drivers meant for either of them. You may be able to get a Paradox ODBC driver somewhere on firebirdsql.org.
Use Borland Delphi to write a program which will export the data you need. As someone else mentioned, you can get a free version called Turbo Explorer. You will also have to install the BDE seperately, as it doesn't come with Turbo Explorer.
I've been working on a gigantic data migration from Paradox to MySQL. My general approach has been to export CSV files from Paradox, and then import the CSV files from the MySQL command line. However this system breaks down when there are M (memo) fields in Paradox, because that data doesn't get pulled into the CSV file as expected.
Here's my long-winded process for getting Paradox data into MySQL, hopefully it helps somebody!
Open Paradox file in Paradox, export to dbase (.dbf) file. What this does is it exports the memo data into dbase's blob format.
Open the .dbf file in Paradox. It might be necessary to convert double format to long integer or number before opening in dbfviewer. Double format appears to not be working. Save file.
Use this program to open up the dbase file and then export to Excel: http://dbfviewer.org/
Export -> XLS-File … this opens it in Excel
Now we need to create a macro because Excel doesn't have any native way to enclose CSV fields with quotes or anything else. I've pasted the macro below, but here are the reference sites that I found. One site had better instructions but corrupted text:
http://www.mrexcel.com/forum/showthread.php?320531-export-as-csv-file-enclosed-quotes
http://www.markinns.com/articles/full/export_excel_csvs_with_double_quotes/
In Excel replace all " with ' by CTRL-F, replace... any " in records will mess stuff up
In Excel press ALT - F11 to open up macros
Insert -> Module
Create this macro to save CSV files enclosed with double quotes:
Sub CSVFile()
Dim SrcRg As Range
Dim CurrRow As Range
Dim CurrCell As Range
Dim CurrTextStr As String
Dim ListSep As String
Dim FName As Variant
FName = Application.GetSaveAsFilename("", "CSV File (*.csv), *.csv")
If FName <> False Then
ListSep = Application.International(xlListSeparator)
If Selection.Cells.Count > 1 Then
Set SrcRg = Selection
Else
Set SrcRg = ActiveSheet.UsedRange
End If
Open FName For Output As #1
For Each CurrRow In SrcRg.Rows
CurrTextStr = ""
For Each CurrCell In CurrRow.Cells
CurrTextStr = CurrTextStr & """" & CurrCell.Value & """" & ListSep
Next
While Right(CurrTextStr, 1) = ListSep
CurrTextStr = Left(CurrTextStr, Len(CurrTextStr) - 1)
Wend
Print #1, CurrTextStr
Next
Close #1
End If
End Sub
Then Run -> Run Macro
Set up target MySQL db schema with text fields where we want the blobs to go
In MySQL command line here's an example of how to do the import:
LOAD DATA LOCAL INFILE 'C:/data.csv'
INTO TABLE table_name
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
(column1, column2)
Paradox is a native format for the Borland Database Engine, which is included with various Delphi programming products. Ownership has changed hands at least once recently, but at one point there were free "Express" versions of Delphi available that would let you write a simple program to export this stuff. If a free version is no longer available, the lowest available SKU should include BDE functionality.
Using MS Access 2007 you can import Paradox 7 and below using the BDE distribution included with the free Paradox Database Editor program (google it). Use a connection such as:
DoCmd.TransferDatabase acImport, "ODBC Database", _
"Paradox 3.X;HDR=NO;IMEX=2;ACCDB=YES;DATABASE=C:\apache\Archive;TABLE=Messages#db", _
acReport, DailyArchiveName, "MyDatabase"