I'm using Firebird database and I need to load Excel file into a database table. I need a tool that does this well. I tried some I found on Google, but all of them have some bugs.
Since Excel data is not created by me, it would be good if it could scan the file and discover what kind of data is inside and suggest a table to be created in the database.
Also, it would be nice if I could compare the file against the data that is already in the database table, and I can pick which data to load and which not.
Tools that load CSV files are also fine, I can "Save as" CSV from Excel before loading.
Well, if you can use CSV, the I guess XMLWizard is the right tool for you. It can load a CSV file and compare with database data. And you can select the changes you wish to make to the table.
Don't let the name fool you, it does work with XML, but it also works very well with CSV files. And it can also estimate the column datatypes and offer CREATE TABLE statement for your file.
Have you tried FSQL?
It's a freeware very similar to Firebird's standard ISQL, but with some extra features, like import data from CSV files.
I've used it with DBF files and it worked fine.
There is also EMS Data import tool for Firebird and Interbase
http://www.sqlmanager.net/en/products/ibfb/dataimport
Not free, though, but it accepts a big variety of formats, including CSV and Excel.
EDIT
Another similar payware tool is Firebird Data Wizard http://www.sqlmaestro.com/products/firebird/datawizard/
There are some online tools which can help you to generate DDL/DML scripts from csv header/sample dump file, check out: http://www.convertcsv.com/csv-to-sql.htm
You can then use sql-workbench's Data Pumper or WbImport Tool from command line.
Orbada has GUI which support for importing csv file also.
DBeaver Free edition also support importing csv out of the box.
BULK INSERT
Other way is on Excell you build formula in new cells with data you want to export. The formula consists to format in strings and lenght to your field according lenght your field in firebird. So you can copy all this cells from excell and past on txt editor, so is possible to use the strategy of BULK INSERT in Firebird.
See more details in http://www.firebirdfaq.org/faq209/
The problem is if you have blob or null data to import, so see if you have this kind of values and if this way is to you.
If you have formated data in txt file, BULK INSERT will be quick way.
Hint: You can too to disable trigger and index associated with your table to accelerate BULK INSERT, and after enable them.
Roberto Novakosky
I load the excel file to lazarus spreadsheet and then export to firebird db. Everythong is fine and the only problem is fpspreadsheet will consider string field with numbers only as a number field. I can check the titles in the first row to see whether the excel file is valid or not.
As far as I can see all replies so far focus on tools that essentially read the Excel (or CSV) file and uses SQL inserts to insert the records into the Firebird database. While this works, I have always found this approach painstakingly slow.
That's why I created a tool that reads an Excel file and writes one file that has a (text) format suitable for Firebird external table (including support for UTF8 char columns) and one DDL file to create the external table in Firebird.
I then use regular SQL to select from the external table, cast as needed, and insert into whatever normal Firebird table I want. The performance with this approach is orders of magnitude faster than SQL inserts from a client app in my experience.
I would be willing to publish the tool. It's written in C#. Let me know if there's any interest.
Related
My office has a Filemaker database which they asked me to replace with a mySQL one. The mySQL one is now set up and running, but doesn't have exact same structure as the Filemaker one (they asked for more things to be added, redundant things were left out, etc.).
I've seen that the filemaker data can be exported as .xml files, could I use those to populate the mySQL database?
If so, I've only ever used
Cooktop,but I'm currently using mac10.6/lubuntu, is there maybe an equivalent (free) piece of software that could do that?
All suggestions are welcome.
Thanks
I can add some information about the various export formats that FileMaker provides. I've done extensive research and testing on this topic.
Below, you'll see a chart with all of the formats that FileMaker offers along the top. Along the side, you'll see various features of these file formats that are unique to FileMaker when exporting. Some are limitations of the FileMaker export process and others are general pros and cons of the format itself.
I'll explain them briefly:
Headers: column labels are exported
Delimiter: the type of separator symbol used
UTF 8/16: yes if either of these is available, could be of concern for special characters or some languages
Only 1 format: means that only one type of encoding is available
Other encoding: a list of all encoding options
Can be imported: FileMaker allows import (not important for this question)
Future proof: According to Wikipedia, format is still widely used and actively maintained
Open standard: open source format
Size: when exporting one of our tables, this was the size of the file
I would recommend also considering some of these factors when deciding which format will work for you. It will depend on the contents and type of your data.
MySQL is just the backend database, so you need a UI to perform the import. You could use FileMaker for this as well, if you set up the MySQL database as an ESS source. If you do this, then you can use familiar import steps in FileMaker to populate your new database.
This may be what the previous answer mentions, but just to designate between the ODBC insert via Execute SQL, which is limited, and External SQL Sources (ESS) that give a native UI in FM.
If the FileMaker database is hosted on a server, you could setup an ODBC link to the MySQL. You could then create a script, in FileMaker, to loop through the data, creating rows in MySQL with only the Columns you are looking to populate.
Other than that, you can export the data, from FileMaker, into many other formats including; TAB, CSV, Excel, xml and pushing it into MySQL.
I want to store 7000 records consisting of txt files as datatype BLOB in my DB using workbench.
But:
1. I don't know how to do it automatically? Should I put all the files in one catalogue and then write a script to take them one by one and insert in the adequate rows?
2. I am not sure if BLOB is fine for this type of file storage? Later I want to connect my DB with GUI so after clicking, it should be possible to open each txt file in new window.
Could you advice me how to solve my problem?
You should write a script, yes. If it's hard for you to put them all in one folder I think there are scripts and tools to do this.
You can use C#, PHP or any other lang to scan those files and then insert them into the database.
Bunch of tutorials:
http://www.csharp-examples.net/get-files-from-directory/
Inserting record in to MySQL Database using C#
http://www.codeproject.com/Articles/43438/Connect-C-to-MySQL
http://net.tutsplus.com/articles/news/scanning-folders-with-php/
http://www.homeandlearn.co.uk/php/php13p3.html
Blob should do, takes around 20 megabytes of text.
I have to make some queries on several Excel sheets and I think it would be easier if I can put them on a DB and make the queries with SQL.
Is there a tool that creates SQL tables from a CSV file with headers?
MySQL has functions built in to load data from CSV files. This is probably your best option and one that gives you the most control.
Have a look at Data Import tool (Excel, CSV or TEXT formats) in dbForge Studio for MySQL. Import allows to create new table and customize fields in a wizard.
How can I import initial table data to a .mwb file? I know that there is an inserts tab for each table, but I would like to import around 200 records and I don't want to do this by hand.
It is not possible with the modern version of MySQL Workbench. There is no way, essentially, to model data - you can only upload it to the server (not the model). The only way currently is to edit one by one which isn't practical. Even if you reverse engineer a table filed with data, the inserts table of the EER model will be blank. You'll note that right-clicking on the row of the inserts tab gives a number of greyed out options including "load from file". I suspect the team didn't have time to implement them or something. Anyway, there is a simple work around if you know phpMyAdmin, which seems to handle CSV files well, or MySQL Workbench, which I have not gotten to work at all with CSV files.
Solution:
Draw your DB model in MySQL Workbench, structure only. Place all your data in associated CSV files - I use Excel and save as CSV - very easy. Once you have your data modeled in Excel and the structure modeled in Workbench, forward engineer the DB, then use some other tool or technique to upload your Excel modeled data.
Not the greatest solution, but bug them to provide data modeling and maybe we'll be lucky in the next version.
Currently this seems not to be possible. I too was hoping to be able to Reverse engineer from the insert statements in a script file, but 1. it didn't work :P and 2. actually the documentation explicitly states that these will be ignored:
http://download.oracle.com/docs/cd/E19078-01/mysql/mysql-workbench/wb-data-modeling.html#wb-reverse-engineering
7.7.9.1. Reverse Engineering Using a Create Script
Reverse engineering using a create script is done by using the File, Import, Reverse Engineer MySQL Create Script ... menu options. Doing this opens a file open dialog box with the default file type set to an SQL script file, a file with the extension sql.
You can create a data definition (DDL) script by executing the mysqldump db_name --no-data > script_file.sql command. Using the --no-data option ensures that the script contains DDL statements only. However, if you are working with a script that also contains DML statements you need not remove them; they will be ignored.
It seems that the lesson is that we ought to handle such resources (that are too large to be manually inserted) through some other medium, such as a versioned sql file. :(
I have a shapefile that I would like to upload to a spatial SQL Server 2008. I have tried using this tool: SQL Server 2008 Spatial Tools. But without luck.
Does anyone know any other (free) tools for doing this?
You can use ogr2ogr to convert from shapefile to GML (or many other formats) and then use SQL Server's GeomFromGML to import. You will need to call GeomFromGML for each feature in your shapefile, but that's a relatively easy program to write.
I have written code in arcbjects for do this task. If you have license arcview or engine you can create a console application in c# and use this code: see https://gis.stackexchange.com/questions/33917/how-to-import-shapefiles-into-ms-sql-2008-and-then-view-that-data-using-qgis?lq=1
UPDATE: I decided I would just point people to the official Github repo instead: https://github.com/zer0infinity/OGR2GUI
This [ogr2ogr fork] tool will attempt to parse the content of the input file (in my case, it was a shapefile) and output it as a bunch of different formats (in my case, I needed it as a CSV, but you can even export your file as a SQLite file). Unfortunately it doens't do straight up SQL, but you can do a dump from a SQLite viewer such as SQLite Browser and import it onto MS SQL. I did notice some inefficiencies when converting to SQLite (I lost some attributes/tables). I also tried MobileMapper Office (MMO) (with better luck) to export to CSV and it did preserve a lot of the data, but you'd then need to write a script to import the data to SQL. If you are going that route, let me know, I'm currently writing a VBA script to deal with the exported data from MMO.
If you're hoping to build this into your application (a script of some sort), you may have some luck trying with ogr2ogr, but you never know what the data is going to look like. documentation is found here http://www.gdal.org/ogr2ogr.html
Original answer: Save yourself sometime and just use this amazing tool: http://ogr2gui.ca/
It's based off ogr2ogr, but with a nice GUI.
I've made an app for importing shapefiles into SQL Server. It's made primarily to suit my needs but i had some spare time so i made an installer for you.
Some nice things you can do with it:
choose encoding of input shapefile
rename/remap destination table column names
choose the destination table name
set the primary key for the destination table
It has a user interface and you can download it for free.
More details can be found on my blog, here's the link: Import shapefiles into SQL Server