I am trying to read a CSV file with VBScript and it is causing huge problems because it isn't recognizing a line break. The way I have right now is:
Set objFSO = CreateObject("Scripting.FileSystemObject")
Set objFile = objFSO.OpenTextFile(server.mappath("my_csv_file.csv"), ForReading)
Do Until objFile.AtEndOfStream
strLine = objFile.ReadLine
arrFields = Split(strLine, ",")
LOOP_STUFF_HERE
Loop
The CSV file has several lines but the problem is that it is reading the CSV file all as one long line and the last item of each line is being combined with the first item of the next line because there is no comma after the last line (it's being created by a client of mine in Excel and then sent to me). My solution is that I have to open it up in a text editor, manually add a comma to the end of the lines and then remove the line breaks. This won't work in the long run because we are setting up an automated system.
Basically, I need to be able to to Split the lines on a line break (I've tried Split(strLine, "\n" but that doesn't seem to work) and then once they are split by line break, then split them by comma. It'd be a multidimensional array in other words.
I can't find out how to get VBScript to recognize the line breaks though. Any ideas? Thanks for your help.
Reading a CSV with FileSystem Object?. don't reinvent the wheel, there are different and proven ways to do exactly what you want in ASP.
The most easy way it's using OLEDB Jet or OLEDB ACE driver to read the file
Basiclyyou need to create a OLEDB.Connection Object with a specific Connection String to the get all the data in the CSV as rows; later you can pass all the data as array using GetRows method or you can use the Recordset object directly.
Post related of using this functionality
ASP.NET (Ace) When reading a CSV file using a DataReader and the OLEDB Jet data provider, how can I control column data types?
ASP.NET (Ace) Microsoft.ACE.OLEDB.12.0 CSV ConnectionString
ASP-Classic (Jet) Reading csv file in classic asp. Problem: column values are truncated up to 300 characters
the Connection is almost the same for Jet or Ace driver (it's the important part).
Related
I'm trying to import the following file into a MySQL Db:
https://drive.google.com/drive/folders/1WbRdNgqVre3wN4DpJZ-08jtGkJtCDJNQ?usp=sharing
Using the "data import wizard" on MySql Workbench, for some reason I'm getting "218\223 lines imported successfully", whereas the file contains close to 100K.
I tried looking for special chars around lines 210-230, also removing all of them, but still the same happens.
The file is a CSV of Microsoft Bing's geo locations, used in Microsoft Advertising campaigns, downloaded from Microsoft's website (using an ad account there).
I've been googling, reading, StackOverflowing, playing with the file and different import options...
I tried cutting the file into small bits, and the newly created file was completely corrupt somehow...
Encoding seems to be UTF-8, line breaks all "\n". I tried changing them all into "\r\n" using notepad++, but still the same happens.
File opens normally in Excel, looks normal, passes CSVlint.io...
The only weird thing is that the file contains quotes on some of the values but not on the rest (e.g. line 219. Yeah I know it sounds like this would be the problem, but I removed it, and all the rest of the lines with quotes, and it still happens... Also tried loading with ENCLOSED BY ", see below).
I also tried using SQL statements to import:
LOAD DATA LOCAL INFILE 'c:\\Users\\Gilad\\Downloads\\GeoLocations.csv'
INTO TABLE aw_geo_map_bmsl
FIELDS TERMINATED BY ','
(tried also with: ENCLOSED BY '"')
LINES TERMINATED BY '/n'
IGNORE 1 ROWS;
(had to add OPT_LOCAL_INFILE=1 to the connection on Advanced for MySQL Workbench to be allowed access to local files on my computer)
This gives 0 rows affected.
Help?
Epilogue: In the end I just gave up on all these import wizards and did it the old "make your SQL statements from Excel" way.
I imported the CSV data into Excel. Watch out: in this case I found I needed to use a data import wizard from Excel (but that one worked perfectly) to be able to change the encoding to UTF, which Excel 2010 chose as "windows" which was wrong.
After processing the data a bit to my liking, I used the following Excel code:
=CONCATENATE("INSERT INTO aw_geo_map_bmsl (`Location Id`,Name,`Canonical Name`,`Location Type`,Status,`Adwords Location Id`)
VALUES (",
A2,
",""",B2,"""",
",""",C2,"""",
",""",D2,"""",
",""",E2,"""",
",",F2,");")
to generate INSERT statements for every line, then copy-pasted and pasted only values, then pasted into an editor, removed additional quotes that Excel adds, and ran it in MySQL Workbench, which runs it line by line (takes some time), and you can see the progress.
Saved me hours of unsuccessfully playing around with "automatic tools" which fail for unknown reasons and don't give proper logs ootb.
Warning: do NOT do this for unsanitized code as it's vulnerable to SQL injection. In this case it was data from Microsoft so I know it's fine.
My Team's using IBM's DataStage ETL tool to read a CSV file into a Salesforce instance. If the last column is blank the file doesn't have a second comma to close out the Record. That is, it's just ',' instead of ',,' at the end of a line. That's causing ETL to reject the file. Excel Anyone know if ETL can be configured to handle the missing record deliminator? Thanks!
You can count the commas in every line and for the line if the final comma is missing add accordingly. All this you need to implement in sequncer command stage.
I have a CSV file that I can open in Excel 2012 and it comes in perfectly. When I try to setup the metadata for this CSV file in Talend the fields (columns) are not splitting the same was as Excel splits them. I suspect I am not properly setting the metadata.
The specific issue is that I have a column with string data in it which may contain commas within the string. For example suppose I have a CSV file with three columns: ID, Name and Age which looks like this:
ID,Name,Age
1,Ralph,34
2,Sue,14
3,"Smith, John", 42
When Excel reads this CSV file it looks at the second element of the third row ("Smith, John") as a single token and places it into a cell by itself.
In Talend it trys to break this same token into two since there is a comma within the token. Apparently Excel ignores all delimeters within a quoted string while Talend by default does not.
My question is how to I get Talend to behave the same as Excel?
if you use tfileinputdelimited component to read this csv file, you can use delimeter as "," and under csv options properties of this component you should enable Text Enclosure """ option or even if you use metadata there would be an option to define string/text enclosure - here you should mention """ to resolve your problem
I'd like to import a pipe delimited file in a table but this file is in UTF-8 with dynamic structure.
I had tried with TranfertText and FSO but only ADODB.Stream seems to deal well with such an encoding however it only read the full text...
How can I read such a file line per line to add rows in an existing table ?
Thanks in advance.
You can read a line from an ADO Stream with its ReadText method.
strLine = objStream.ReadText -2 ' adReadLine
You may need to set your stream's LineSeparator property first.
After you read the line, you can split on the pipe character.
Split(strLine, "|")
I'm working with a client who has an existing system, built on what is apparently a Paradox database. I've got the database, in the form of a zip file containing .DB, .MB and .PX files, one for each table.
I need to take (some) of this data and import it in to a Web application that's using MySQL. Does anybody have a way for me to extract this data, that doesn't involve installing Paradox?
If not, does Paradox export in some readable format? Either as SQL or something that can be parsed reasonably easily? The person in charge of this system for my client is a volunteer (they're a non-profit), so I'd like to go to him with a solution - because last time I asked for the data, I got this, which is clearly no good.
The wikipedia article about Paradox lists two other things, that might be interessant, both under GPL license:
pxlib: Library to read and write Paradox databases
pxtools: convert a Paradox-database into a SQL-database
And if you have Delphi and want to write a converter yourself (which would need the BDE to work) you can take a look at this article or at the source code of ConvertCodeLib on this web site. Both make use of TClientDataset, which can write a CDS (binary format) or an XML file.
Both the Paradox for DOS and Paradox for Windows platforms will export data tables in Delimited Text, Fixed-Length Text, and Lotus 1-2-3 formats. The older Paradox for DOS also writes Lotus Symphony, while the slightly less antique Paradox for Windows does a passable Excel 5.
However, someone will have to sit down and export the tables one by one, or write a script do to it. Of course you'd need to have Paradox installed to write the script.
-Al.
MS has instructions for using the MS Jet driver to read data from files produced by Paradox 3-5. That can act as (at least) an ODBC driver, so you can use it to read a Paradox file from just about anything that knows how to use ODBC.
You have a few options:
Get your hands on the original Paradox software, and use it to export the database into CSV format. Unfortunately, Borland no longer sells it and the most recent version doesn't run well on Windows XP or above.
Access the database using either a Paradox or dBase/xBase ODBC driver. Paradox and xBase are very similar, so you may be able to extract the data using drivers meant for either of them. You may be able to get a Paradox ODBC driver somewhere on firebirdsql.org.
Use Borland Delphi to write a program which will export the data you need. As someone else mentioned, you can get a free version called Turbo Explorer. You will also have to install the BDE seperately, as it doesn't come with Turbo Explorer.
I've been working on a gigantic data migration from Paradox to MySQL. My general approach has been to export CSV files from Paradox, and then import the CSV files from the MySQL command line. However this system breaks down when there are M (memo) fields in Paradox, because that data doesn't get pulled into the CSV file as expected.
Here's my long-winded process for getting Paradox data into MySQL, hopefully it helps somebody!
Open Paradox file in Paradox, export to dbase (.dbf) file. What this does is it exports the memo data into dbase's blob format.
Open the .dbf file in Paradox. It might be necessary to convert double format to long integer or number before opening in dbfviewer. Double format appears to not be working. Save file.
Use this program to open up the dbase file and then export to Excel: http://dbfviewer.org/
Export -> XLS-File … this opens it in Excel
Now we need to create a macro because Excel doesn't have any native way to enclose CSV fields with quotes or anything else. I've pasted the macro below, but here are the reference sites that I found. One site had better instructions but corrupted text:
http://www.mrexcel.com/forum/showthread.php?320531-export-as-csv-file-enclosed-quotes
http://www.markinns.com/articles/full/export_excel_csvs_with_double_quotes/
In Excel replace all " with ' by CTRL-F, replace... any " in records will mess stuff up
In Excel press ALT - F11 to open up macros
Insert -> Module
Create this macro to save CSV files enclosed with double quotes:
Sub CSVFile()
Dim SrcRg As Range
Dim CurrRow As Range
Dim CurrCell As Range
Dim CurrTextStr As String
Dim ListSep As String
Dim FName As Variant
FName = Application.GetSaveAsFilename("", "CSV File (*.csv), *.csv")
If FName <> False Then
ListSep = Application.International(xlListSeparator)
If Selection.Cells.Count > 1 Then
Set SrcRg = Selection
Else
Set SrcRg = ActiveSheet.UsedRange
End If
Open FName For Output As #1
For Each CurrRow In SrcRg.Rows
CurrTextStr = ""
For Each CurrCell In CurrRow.Cells
CurrTextStr = CurrTextStr & """" & CurrCell.Value & """" & ListSep
Next
While Right(CurrTextStr, 1) = ListSep
CurrTextStr = Left(CurrTextStr, Len(CurrTextStr) - 1)
Wend
Print #1, CurrTextStr
Next
Close #1
End If
End Sub
Then Run -> Run Macro
Set up target MySQL db schema with text fields where we want the blobs to go
In MySQL command line here's an example of how to do the import:
LOAD DATA LOCAL INFILE 'C:/data.csv'
INTO TABLE table_name
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\r\n'
(column1, column2)
Paradox is a native format for the Borland Database Engine, which is included with various Delphi programming products. Ownership has changed hands at least once recently, but at one point there were free "Express" versions of Delphi available that would let you write a simple program to export this stuff. If a free version is no longer available, the lowest available SKU should include BDE functionality.
Using MS Access 2007 you can import Paradox 7 and below using the BDE distribution included with the free Paradox Database Editor program (google it). Use a connection such as:
DoCmd.TransferDatabase acImport, "ODBC Database", _
"Paradox 3.X;HDR=NO;IMEX=2;ACCDB=YES;DATABASE=C:\apache\Archive;TABLE=Messages#db", _
acReport, DailyArchiveName, "MyDatabase"