Every week I import a table of our clients into MySQL and every week a random client is missing from that list. That same client might have been imported without problems the week before and then next week again imported perfectly fine, however MySQL will always omit one client from the import which makes my job very frustrating in trying to find who it was and manually fix it.
Has anyone run into similar issues in the past or have any idea why they might be happening?
I am using the MySQL table import wizard. Normally I don't have issues, but this specific client table export I get from another platform is having this issue every single week.
I would do the import from the Excel into a TEMPORARY (ie: not the final table), but one of the same structure. Then, once that is done and the import of all names and columns are parsed, you can left-join from LIVE table to the TEMP table where the entry exists on LIVE but not in TEMP. This would help find the missing import. THEN, since the data is already in a valid table structure, you can pull in all new values from the temp table to the production, including adding any new that did not yet exist.
If you are exporting to excel using the Export wizard, it creates a sheet with the name of the table, but also creates a named range with the same name. The named range then contains only the region with data. Then when you add new data, this named region is not extended. And when you are later importing the data, you probably chose the name without the $ character on end. The name without the $ in the name represents the named region, which probably doesn't contain your newly inserted data. To load all the date chose the name with $ character or extent the named region to cover all the data in the sheet.
Related
I was given an excel (csv) sheet containing a database metadata.
I'm asking if there's a simple way to import the csv and create the tables from there?
Data is not part of this question. the csv looks like this:
logical_table_name, physical_table_name, logical_column_name, physcial_column_name, data_type, data_length
There's about 2000 rows of metadata. I'm hoping I don't have to manually create the tables. Thanks.
I don't know of any direct import or creation. However, if I had to do this and I couldn't find one, I would import the excel file into a staging table (just a direct data import). I'd make add a unique auto ID column to staging table to keep the rows in order.
Then I would use some queries to build table and column creation commands from the raw data. Unless this was something I was setting up to do a lot, I would keep it dead simple, not try and get fancy. Build individual add column commands for each column. Build a create Table command for the first row for each table. Sort them all by the order id, tables before columns. Then you should be able to just copy the script column, check the commands, and go.
I am trying to use an Index to import monthly data from Excel into a Table (by Appending new records to the Table).
I have created a unique index in the Table being posted to to prevent the import of duplicate records.
However when I click import (and assuming the records already exist) it still gives the user the option to import the same records again by clicking yes to the message below:
How would I get the macro to not run if the entries already exist and also not give the user the above message option?
I have tried suppressing messages temporarily using a bit of code as referred to below, but that just processes the import anyway and re-adds the entries to the table again.
Thanks
Excel imports often have issues (even if you haven't found any in this particular file), expecially if the Excel file is created or edited by humans.
My suggestion would be to import the file into a temporary table consisting of all text fields (F1, F2, etc) and an identity field.
Using all text fields, all data should import without any conversion problems.
Don't use the first row as column heads. Import it as the first row of data. That way you can easily check for correct column headings.
If correct, just delete that row.
Then validate and scrub the data using queries and/or code against the temporary table to identify any data that does not meet the requirements of your permanent tables. E.G. Select * From WHERE Not IsNumeric(F3).
Then remove any leading or trailing spaces, double spaces, etc. and whatever other cleanup you might like to do.
Then use a query with an outer join (on your PK fields) to append rows from the temp table, that are not already in the permanent table, to the permanent table.
I have to import CSV File for different clients in my system, some with [,] some [|] Etc… separated. Always very big files.
While importing I need to filter duplicate records, duplicate records should not insert in dB.
Problem is columns can be different for different clients. I have database for every client for CSV DATA which I import every day, Week or month depending on client need. I keep data for every import so we can generate reports & what data we receive in CSV file our system do processing after import.
Data structure example:
Client 1 database:
First_Name | Last_Name | Email | Phone | Ect…
95% data always same every new CSV file. Some records new comes & some records they delete from csv. so our system only process those records those newly imported .
Currently what happening we import data every time in new table. We keep table name with time stamp so we can keep track for import. it Is expensive process, it duplicate records and tables.
Im thinking solution and I need your suggestion on it.
Keeping one table every time import when I import CSV file data in table, I’ll alter existing table add new column, column name will be “current date” (byte or Boolean) add true false on import??
My other question is first time when I import CSV file …I need to write script:
While importing CSV data, if table already exists then my date logic will work else if table does not exist it should create table given or provided “client name” as table name. Challenge is columns I don’t know it should create columns from CSV file.
Table already exist some new records came in it should insert else update.
Is it do able in mysql??
although I have to do something for mssql also but right now I need solution for my sql.
Please help me... im not good in MySQL :(
You can certainly do an insert Or update statement when importing each record.
see here :
https://dev.mysql.com/doc/refman/5.0/en/insert-on-duplicate.html
I propose you create a script to dynamically create you table if it doesn't
What is the language that you would use to insert your csv?
I removed, what at the time I thought, was an unnecessary field/column from a table in mysql and now would like to import it back into the table. Just before I dropped the column, I did a backup of the whole table. Since then, close to 1000 new rows have been added.
How can I add the column and information back to the table?
I have a sandbox that I can "play" with. I have tried but have yet to get the proper end result. Thank you in advance.
Not sure if it matters but, the system is fedora 16.
What I would do is to load your dump into another table (e.g. table_bak, you might need to do this in your sandbox and redump). Then add your column back in to your live table with a sensible default value. Then you should be able to bring the old data back like this
UPDATE `table`,`table_bak` SET `table`.restored_column=`table_bak`.restored_column WHERE `table`.pk_id=`table_bak`.pk_id
Friends, I am using toad for MySQl, and have a huge database ready and validated.
Now i have an excel file which contains data-entries for a particular table. And i am also successfully able to import data into the db using import wizard, mapping the first row header with the column names etc.
But now i have appended a few data entries into it which i wish to insert into the database. However the old values also get selected and hence cause a primary_key_violation exception as the entry already exists! Otherwise a truncate table option is there which i dont wish to use as there may be many files from which i have inserted the data.
I tried my level best but didnt get any solution, atleast in toad for mysql. Please tell me what to do! the solution maybe simple but i need it SOS
An option may be to not append records to that excel file, but create a new excel file with only the new records