I have a text file that has the field headers on the first line, and the following lines are space separated fields. The spacing between each field is different. There are 45 fields. Specifically, this is an eBird dataset. I want to load this text file into a newly created table within a database I created, so that each header is the field header in the database and each line below the headers are records. Here is a small example of the files format:
Header1 Header2 Header3 Header4
082739 United States US-CA-01 1
I haven't tried anything yet, because I want to know what to do before I load this data into a table. I have this command prepared:
LOAD DATA INFILE <file_path> INTO TABLE <table_name> FIELDS TERMINATED BY ' ';
Will this command populate the table so that each field corresponds to each header, or do I need to tell the command how many spaces there are between each field? Will the command know that each line is terminated by a newline if not told?
Looks like there's an R package for filtering and processing the EBD dataset. Using R you should be able to process the data into a more manageable format for insertion into MariaDB.
For example, you can separate the fields with commas to generate CSV-formatted data that can be readily ingested by MariaDB.
Related
I have a 13gb .txt file which I am importing into MySQL, however I don't want to import all of the data. For example there are many columns that are either completely empty or contain irrelevant information - I only want to import ~100/360 I've been provided. If I only create headers for the columns I want, can I select the specific corresponding data from the .txt file to be uploaded?
Normally I would use a text editor to remove the superfluous data, but I do not possess a text editor that can handle a file of this size.
You can ignore specific columns in the input file by assigning them to a user-defined variable instead of a database column.
For example if you had a CSV file with 4 columns and just wanted to import columns 1 and 4 into your table you could do something like this:
load data infile '/tmp/so42140337.csv'
into table so42140337
fields terminated by ','
lines terminated by '\n'
(c1,#dummy,#dummy,c2);
Given the size of your input file it may be more efficient to import it in chunks rather than importing the entire file in one command. You can use the pt-fifo-split tool for this, following the pattern in this blog post.
I'm doing a lad data infile in MySQL through MySQL Workbench. I'm pretty new to SQL in general, so this may be a simple fix, but I can't get it to work. It is throwing a a 1261 Error (doesn't contain data for all columns) on the last row, but the last row (like the rest of the CSV) doesn't have any blank or null values.
I've looked around for help and read the manual, but everything I've seen has been about dealing with null values.
I exported the CSV from Excel, to the extent that maters.
The code I'm using to import is (I've changed the field, file, and table names to be more generic):
load data infile '/temp/filename.csv'
into table table1
fields terminated by ","
lines terminated by '\r'
ignore 1 lines
(Col1,Col2,Col3,Col4,Col5,col6,col7,Col8,Col9);
The first two columns are varchar and char, respectively with the remaining columns all formatted as double.
Here's the last few lines of the csv file:
364,6001.009JR,43.96,0,0,0,0,0,0
364,6001.900FM,0,0,0,0,0,0,0
364,6001.900JR,0,0,0,0,0,0,0
The only thing I can think of is that I'm supposed to have some signal after the last line to indicate that the file is finished, but I haven't found anything to indicate what that would be.
Any help would be appreciated
When I've had similar errors, it's because there were unexpected newlines inside my data (a newline in one row would look like two too-short rows, upon import).
How can I load 10,000 rows of test.xls file into mysql db table?
When I use below query it shows this error.
LOAD DATA INFILE 'd:/test.xls' INTO TABLE karmaasolutions.tbl_candidatedetail (candidate_firstname,candidate_lastname);
My primary key is candidateid and has below properties.
The test.xls contains data like below.
I have added rows starting from candidateid 61 because upto 60 there are already candidates in table.
please suggest the solutions.
Export your Excel spreadsheet to CSV format.
Import the CSV file into mysql using a similar command to the one you are currently trying:
LOAD DATA INFILE 'd:/test.csv'
INTO TABLE karmaasolutions.tbl_candidatedetail
(candidate_firstname,candidate_lastname);
To import data from Excel (or any other program that can produce a text file) is very simple using the LOAD DATA command from the MySQL Command prompt.
Save your Excel data as a csv file (In Excel 2007 using Save As) Check
the saved file using a text editor such as Notepad to see what it
actually looks like, i.e. what delimiter was used etc. Start the MySQL
Command Prompt (I’m lazy so I usually do this from the MySQL Query
Browser – Tools – MySQL Command Line Client to avoid having to enter
username and password etc.) Enter this command: LOAD DATA LOCAL INFILE
‘C:\temp\yourfile.csv’ INTO TABLE database.table FIELDS TERMINATED
BY ‘;’ ENCLOSED BY ‘”‘ LINES TERMINATED BY ‘\r\n’ (field1, field2);
[Edit: Make sure to check your single quotes (') and double quotes (")
if you copy and paste this code - it seems WordPress is changing them
into some similar but different characters] Done! Very quick and
simple once you know it :)
Some notes from my own import – may not apply to you if you run a different language version, MySQL version, Excel version etc…
TERMINATED BY – this is why I included step 2. I thought a csv would default to comma separated but at least in my case semicolon was the deafult
ENCLOSED BY – my data was not enclosed by anything so I left this as empty string ”
LINES TERMINATED BY – at first I tried with only ‘\n’ but had to add the ‘\r’ to get rid of a carriage return character being imported into the database
Also make sure that if you do not import into the primary key field/column that it has auto increment on, otherwhise only the first row will be imported
Original Author reference
I've read through some other posts and nothing quite answers my question specifically.
I have an existing database in phpMyAdmin - a set of pin codes we use to collect contest entries.
The DB has about 10,000 pin codes in it.
I need to add 250 "New" codes to it. I have an excel file that is stripped down to a single column .csv, no header - just codes.
What I need to do is import this into the table named "pin2" and add these to the row called "pin"
The other rows are where entrants would add names and phone numbers, so are all "null"
I've uploaded a screen grab of the structure.
DB Structure http://www.redpointdesign.ca/sql.png
any help would be appreciated!
You need to use a LOAD DATA query similar to this:
LOAD DATA INFILE 'pincodes.csv'
INTO TABLE pin2 (pin)
If the pin codes in the csv file are enclosed in quotes you may also need to include an ENCLOSED BY clause.
LOAD DATA INFILE 'pincodes.csv'
INTO TABLE pin2
FIELDS ENCLOSED BY '"'
(pin)
If you wants to do using csv
Then you need to need to follow these steps
Manually define autoincremented value in first comlumn.
In other column you have to externally define it as a NULL,
otherwise you will get Invalid column count in CSV input on line 1.
because column with no value is not consider by phpmyadmin
Them click on import in phpmyadmin and you are done ..
I am trying to load data from a csv file that is still in excel. So far this is the statement that I have as my sql query:
LOAD DATA LOCAL INFILE 'C:\\Documents and Settings\\J03299\\Desktop\\TMETER.csv'
INTO TABLE edata
COLUMNS TERMINATED BY ',' ENCLOSED BY "" LINES TERMINATED BY '\n'
(Year,Month,Day,MJD,xpiles,xstacks,Utilites);
The file is called TMETER and it has 7 columns. It has 366 rows. I am able to only read the first row and only the first four columns(till MJD) but everything else is null after that. Secondly in the second row it puts all the columns from my file (TMETER.csv) in row 124 into the first column of the second row in my edata table. I am confused as to
Why doesn't it read the data from column piles,stacks ,utilites? (mind you the column names in the csv file are weird and not the same as my edata table e.g in database table it is piles while in actual csv file it is x(piles), stacks in table but y(stacks) in csv file. Mysql doesn't not allow me to create etable names with this format so I had to improvise. Could this be why it is not reading and mapping from the csv file to the table in mysql?
Why is my statement putting the first row in my csv file in the first row in mysql table but then skipping all the down to row 124 and then inserting all columns from csv file into first column of mysql database table?
Sorry my English is not good.
Any assistance will be greatly appreciated.
When you run the command, it should give a message like Records: 1 Deleted: 0 Skipped: 0 Warnings: 0. If there are any warnings, type SHOW WARNINGS to see what they were.
This sounds like behavior I sometimes get when the line ending is wrong. Try opening your document in a code editor and checking to make sure your lines actually end with \n. If that's inconvenient, you could also just try dropping the table and reimporting with LINES TERMINATED BY '\r' or LINES TERMINATED BY '\r\n' and see if that fixes it.
Regarding field names: This command ignores field names in your text file. All it does is match the first column to the first field indicated in parentheses (in your case, Year) the second column to the second field in parentheses (Month), and so on. If you have field names at the top of your file, you should skip over them by adding IGNORE 1 LINES just before the parentheses with the list of fields.