Am basically from Microsoft background working much on SSIS for ETL sought of project.
Now I got another project on hand to deal with loading of .csv files into MySql database. In process of loading these tables data has to go through some transformations and then into destination table. It is much of ETL project.
Client doesn't have SSIS (BIDS) and am compelled to use open source tools.
I did bit of research and found Talend Data Integration tool best fits for my situation.
As am new to this environment and am sure there are experts in this area, I need some advice on best tools to do ETL of this type and best practices.
If need any futher information please let me know.
If I remember correctly, PhpMyAdmin can import CSV into MySQL, and this question is about a similar topic too, but these don't come close to what SSIS can offer...
Yes you are right Talend Open Studio is pretty good tool with hundreds of connector,
in your case just create job which take CSV as your source and MySQl is destination apply any transformation if required and load it.
you can get more information on CSV to MySQL load with examples Talend forum
if you have any base plan then, share with me, I can guide you how to transfer CSV to MySQL table.
Related
I was given a task to have a better understanding of several ETL packages that were created in a Database project using Business Intelligence Development Studio(SQL 2005).
Currently I have to open each master package, package and then data flow and so on to discover the relationships that exists with either the source tables and the destination tables.
I realized that probably a good way to more easily get that information would be having a tool similar to what SchemaSpy does with a normal Database. That would provide my a high level detail of the relationships that exist.
Anyone knows an application/script that could help me achieving this result?
I tried to search, but I must admit that I was getting the feeling that I wasn't really searching in the right direction as most of my searches ended up pointing for database comparisons.
Turned out, the only way I found to do this was to parse the xml inside the packages and extract the relationships. And then using Graphviz (the same visual component used by schema spy) create the diagrams.
Unfortunately this was an expensive thing to do and I never finished the project. Mainly due to lack of knowledge around the xml structure but it is definitely possible to be achieved
I'm currently developing a webapp to a costumer and I need to import data from their old application.
As far as I can tell, the database is dBase written with FoxPro.
I have the application exe and lots of dbf/dbb/dbi files and I can open the dbf table files thanks to the DBF Commander.
Although I can open each table and see the data, I really could use an EER diagram to understand how the tables connect to each other.
Is there any tool that could reverse engineer the database and draw the EER diagram?
Thank you for time.
Regards,
Hugo
A lot depends on whether it's just a bunch of DBF/CDX/FPT files, or whether they belong to a database container (DBC/DCT/DCX extensions). If there's no database container, i.e. 'free tables', they haven't necessarily been created by FoxPro, it could have been any of the xBase languages, like Clipper and so on.
If they are part of a database container then there may be metadata in it defining rules, triggers, relations.
Even so, you probably really need a copy of Visual FoxPro (do you have an MSDN sub maybe?). It has a built-in tool called GenDBC which will document the structure of a database if it has a database container.
There is also the Stonefield Database Toolkit which would give better tools in this regard.
Or you could use the Upsizing Wizard in Visual FoxPro to get the data into SQL Server, and from there you would have a much better choice of tools.
See whether XCase gives you what you need: http://www.xcase.com/
I have a shapefile that I would like to upload to a spatial SQL Server 2008. I have tried using this tool: SQL Server 2008 Spatial Tools. But without luck.
Does anyone know any other (free) tools for doing this?
You can use ogr2ogr to convert from shapefile to GML (or many other formats) and then use SQL Server's GeomFromGML to import. You will need to call GeomFromGML for each feature in your shapefile, but that's a relatively easy program to write.
I have written code in arcbjects for do this task. If you have license arcview or engine you can create a console application in c# and use this code: see https://gis.stackexchange.com/questions/33917/how-to-import-shapefiles-into-ms-sql-2008-and-then-view-that-data-using-qgis?lq=1
UPDATE: I decided I would just point people to the official Github repo instead: https://github.com/zer0infinity/OGR2GUI
This [ogr2ogr fork] tool will attempt to parse the content of the input file (in my case, it was a shapefile) and output it as a bunch of different formats (in my case, I needed it as a CSV, but you can even export your file as a SQLite file). Unfortunately it doens't do straight up SQL, but you can do a dump from a SQLite viewer such as SQLite Browser and import it onto MS SQL. I did notice some inefficiencies when converting to SQLite (I lost some attributes/tables). I also tried MobileMapper Office (MMO) (with better luck) to export to CSV and it did preserve a lot of the data, but you'd then need to write a script to import the data to SQL. If you are going that route, let me know, I'm currently writing a VBA script to deal with the exported data from MMO.
If you're hoping to build this into your application (a script of some sort), you may have some luck trying with ogr2ogr, but you never know what the data is going to look like. documentation is found here http://www.gdal.org/ogr2ogr.html
Original answer: Save yourself sometime and just use this amazing tool: http://ogr2gui.ca/
It's based off ogr2ogr, but with a nice GUI.
I've made an app for importing shapefiles into SQL Server. It's made primarily to suit my needs but i had some spare time so i made an installer for you.
Some nice things you can do with it:
choose encoding of input shapefile
rename/remap destination table column names
choose the destination table name
set the primary key for the destination table
It has a user interface and you can download it for free.
More details can be found on my blog, here's the link: Import shapefiles into SQL Server
After looking at a lot of questions..i found no real answer for this.
I redisigned an Database for our customer.
With Microsoft Access i found a good Tool to get old table Data in my new well formed Database Structure. It is really easy but takes a lot of time (cause handling old Data with a lot of care).
Are there any Open Source Tools that bring that facilities like Microsoft Access?
To clear it up: I "just" want to reorder old Firebird Database Data in a new "best-practise" Way.
Edit:
I would be really nice if i can get a Log File or something similar to have some documentation on the changes.
Update:
After checking some of the Tools of that Wikipedia Site. I found no real Logging Mechanism.
How do you documentate the changes on a Database? Simply by writing it down?
Result:
So i dont got an real answer...i ma still searching for an nice tool. thnak you guys for the hints and your thoughts regarding this question. I want to reward Kenneth Cochran with the Bounty cause he pointed me to ETL. Thank you!
Talend's Open Source ETL supports FireBird. Very cool tool.
http://www.talend.com/download.php?src=DataGovernanceBlog
It sounds like what you're asking for is an ETL(extract, transform, load) tool.
Wikipedia has a list of open source tools that may help with this. I've not used any of them personally.
Well, I used the Pentaho suite for doing ETL using their Kettle tool.
It's quite easy to use and should be more than enough to reach your intent.
And it's open source.
Give a look at it.
I advice you to use a tool like IBExpert or Database Workbench which are the best tools for Firebird.
For migrating Firebird 1.5 to Firebird 2.1 : you just have to make a backup of your database with Firebird 1.5 server and restore your database with Firebird 2.1 server
I've used Excel in the past to document data model changes - each worksheet used the application version in order to sync with our tags in CVS. Every thing was logged in it - columns that were removed as well as minor alterations to datatypes like varchar(10) to varchar(20) etc along with a note describing why the change was made.
Personally, I've only ever scripted things like these as DDL/DML scripts broken into a script that dealt with table creation, constraint dropping, index drops, DML script(s), constraint application, index application, and removing orphaned tables.
If you want a basic ETL tool, that is client based (and cheap at $300), look at Advanced Query Tool. It mainly queries any type of ODBC connection(including Excel files set up that way), but also has some extended features, including moving data. And has a command line interface. http://www.querytool.com/
I've used it instead of Informatica for one-off jobs, but I've also used to extract from Excel to another file for business users, for a few months, scheduled from my desktop.
I often have data in Excel or text that I need to get into SqlServer. I can use ODBC to query the Excel file and I can parse the text file. What I want though is some tool that will just grab the data and put it into tables with little / no effort. Does anyone know of such a tool?
Have you tried the SQL Server Import/Export Wizard ?
In SQL Server Management Studio, right-click your Database Name, and select Tasks menu, Import Data. For Data Source, select Microsoft Excel, browse to the .XLS...
If you are using Sql Server look at Integration Services (SSIS).
You can also take a look at parse-o-matic
Use DTS or SSIS depending on which version of SQL Server you have. There is an import wizard which can get you started, but data imports are rarely simple and usually involve some sort of data cleanup so that your incoming data is acceptable to the table where you intend to store it. Excel data, in my experience, is usually particularly bad inthis respect becasue it often isn't stored properly in Excel to begin with.
I haven't seen commercial tools that do this. I create this kind of tools at work all the time, and the data validation is not trivial. This just makes sure that you don't have bad data making it into your database.
I found that for simple data conversion needs something like FileHelpers is pretty good. It still needs programming though. This framework is fairly easy to use, and somebody with a little bit of experience could bang something out for you.
On further thought, you can use the SQL Server bcp utility to upload the contents of a text file. This is a command-line utility and has a lot of switches. I would suggest you experiment on a test table before you use this in a production table.
It's been a while since I used it, so I can't remember if you can directly use an Excel spreadsheet. Text files are always the easiest to deal with in any case.
Seems like it'd be pretty easy to write a script that reads the text file, and converts it to "INSERT * into TABLE" Sql statements. I suspect this has already been done, but a simple implementation would be less than 100 lines of code in your favorite scripting language.
Hey, Google says SQLServer comes with such a tool, BULK INSERT: