I need to convert the contents of Excel sheet to the corresponding tables in MySQL database. I referred the net and found that Pentaho data integration (ETL Kettle tool) can be used for this purpose. Can anyone suggest me which version can I download the software and tutorials on those ETL tools?
You can try looking at Pentaho site itself. You can download the latest version from there.
Here is a tutorial that covers the sort of transformations you are looking to do. While this tutorial uses .csv files as the data sources, you could just as easily substitute Excel files as the source. http://holowczak.com/building-etl-transformations-in-pentaho-data-integration-kettle/
Related
I am in need to convert an excel taken as a input into mysql database table. While searching for tools regarding it, I found that ETL tools (eg.pentaho kettle etl tool) . Can any one say whether the tool is correct or any other approach is there to do the task.
Sounds like a good option but you have to dig into the subject more yourself.
Take a look at this video tutorial for example (the tool mentioned in the tutorial - Spoon is part of Kettle).
Yes, Pentaho Data Integration, aka Kettle, will allow you to do that. As would any ETL tool for that matter. You may also find specific tools that are simpler to use for your purpose, installing a multi 100 Mb tool just for that may be a bit overkill.
Has anyone had much experience with data migration into and out of NetSuite? I have to export DB2 tables into MySQL, manipulate data, and then export ina CSV file. Then take a CSV file of accounts and manipulate the data again for accounts to match up from our old system to new. Anyone tried to do this in MySQL?
A couple of options:
Invest in a data transformation tool that connects to NetSuite and DB2 or MySQL. Look at Dell Boomi, IBM Cast Iron, etc. These tools allow you to connect to both systems, define the data to be extracted, perform data transformation functions and mappings and do all the inserts/updates or whatever you need to do.
For MySQL to NetSuite, php scripts can be written to access MySQL and NetSuite. On the NetSuite side, you can either do SOAP web services, or you can write custom REST APIs within NetSuite. SOAP is probably a bit slower than REST, but with REST, you have to write the API yourself (server side JavaScript - it's not hard, but there's a learning curve).
Hope this helps.
I'm an IBM i programmer; try CPYTOIMPF to create a pretty generic CSV file. I'll go to a stream file - if you have NetServer running you can map a network drive to the IFS directory or you can use FTP to get the CSV file from the IFS to another machine in your network.
Try Adeptia's Netsuite integration tool to perform ETL. You can also try Pentaho ETL for this (As far as I know Celigo's Netsuite connector is built upon Pentaho). Also Jitterbit does have an extension for Netsuite.
We primarily have 2 options to pump data into NS:
i)SuiteTalk ---> Using which we can have SOAP based transformations.There are 2 versions of SuiteTalk synchronous and asynchronous.
Typical tools like Boomi/Mule/Jitterbit use synchronous SuiteTalk to pump data into NS.They also have decent editors to help you do mapping.
ii)RESTlets ---> which are typical REST based architures by NS can also be used but you may have to write external brokers to communicate with them.
Depending on your need you can have whatever you need.IN most of the cases you will be using SuiteTalk to bring in data to Netsuite.
Hope this helps ...
We just got done doing this. We used an iPAAS platform called Jitterbit (similar to Dell Boomi). It can connect to mySql and to NetSuite and you can do transformations in the tool. I have been really impressed with the platform overall so far
There are different approaches, I like the following to process a batch job:
To import data to Netsuite:
Export CSV from old system and place it in Netsuite's a File Cabinet folder (Use a RESTlet or Webservices for this).
Run a scheduled script to load the files in the folder and update the records.
Don't forget to handle errors. Ways to handle errors: send email, create custom record, log to file or write to record
Once the file has been processed move the file to another folder or delete it.
To export data out of Netsuite:
Gather data and export to a CSV (You can use a saved search or similar)
Place CSV in File Cabinet folder.
From external server call webservices or RESTlet to grab new CSV files in the folder.
Process file.
Handle errors.
Call webservices or RESTlet to move CSV File or Delete.
You can also use Pentaho Data Integration, its free and the learning curve is not that difficult. I took this course and I was able to play around with the tool within a couple of hours.
I am going to customise Saiku analysis tool for my local MySQL data base. For that I have to change the their existing data source as following manner,
type=OLAP
name=steelwheels
driver=mondrian.olap4j.MondrianOlap4jDriver
location=jdbc:mondrian:Jdbc=jdbc:mysql://localhost/sampledata; \>Catalog=../webapps/saiku/steelwheels/steelwheels.mondrian.xml;JdbcDrivers=com.mysql.jdbc.Driver;
username=dbuser
password=password
According to above code, can any one please tell me how I generate Mondrian schema file called steelwheels.mondrian.xml with MySQL data base sampledata
Thanks
Download and open PSW or Pentaho Schema Workbench which can be found as part of the mondrian project on sourceforge.
This is going away with Mondrian4, but thats ages away at the moment.
If you are using Pentaho BI server then its really easy to create *.mondrian.xml file for your datasource using Pentaho User Console it will be easy to create and edit datasource and just use *.mondrian.xml file created under /biserver-ce/pentaho-solutions/admin/resources/metadata/.
I use that in my regular practice. It will be effective even if you dont use pentaho for your whole application but just use pentaho user console for this purpose.
At this moment my best solution is to use the data source wizard to create a new analysis, export it manually and edit it by hand in your favorite xml editor. (Example Notepad++ with XMLTools plugin).
Sadly it works better than using the old Pentaho Schema Workbench. You may try to create a DTD to validate the structure of XML against the schema. See the official documentation.
Maybe the Enterprise version has better tools to edit it.
Am basically from Microsoft background working much on SSIS for ETL sought of project.
Now I got another project on hand to deal with loading of .csv files into MySql database. In process of loading these tables data has to go through some transformations and then into destination table. It is much of ETL project.
Client doesn't have SSIS (BIDS) and am compelled to use open source tools.
I did bit of research and found Talend Data Integration tool best fits for my situation.
As am new to this environment and am sure there are experts in this area, I need some advice on best tools to do ETL of this type and best practices.
If need any futher information please let me know.
If I remember correctly, PhpMyAdmin can import CSV into MySQL, and this question is about a similar topic too, but these don't come close to what SSIS can offer...
Yes you are right Talend Open Studio is pretty good tool with hundreds of connector,
in your case just create job which take CSV as your source and MySQl is destination apply any transformation if required and load it.
you can get more information on CSV to MySQL load with examples Talend forum
if you have any base plan then, share with me, I can guide you how to transfer CSV to MySQL table.
I have a shapefile that I would like to upload to a spatial SQL Server 2008. I have tried using this tool: SQL Server 2008 Spatial Tools. But without luck.
Does anyone know any other (free) tools for doing this?
You can use ogr2ogr to convert from shapefile to GML (or many other formats) and then use SQL Server's GeomFromGML to import. You will need to call GeomFromGML for each feature in your shapefile, but that's a relatively easy program to write.
I have written code in arcbjects for do this task. If you have license arcview or engine you can create a console application in c# and use this code: see https://gis.stackexchange.com/questions/33917/how-to-import-shapefiles-into-ms-sql-2008-and-then-view-that-data-using-qgis?lq=1
UPDATE: I decided I would just point people to the official Github repo instead: https://github.com/zer0infinity/OGR2GUI
This [ogr2ogr fork] tool will attempt to parse the content of the input file (in my case, it was a shapefile) and output it as a bunch of different formats (in my case, I needed it as a CSV, but you can even export your file as a SQLite file). Unfortunately it doens't do straight up SQL, but you can do a dump from a SQLite viewer such as SQLite Browser and import it onto MS SQL. I did notice some inefficiencies when converting to SQLite (I lost some attributes/tables). I also tried MobileMapper Office (MMO) (with better luck) to export to CSV and it did preserve a lot of the data, but you'd then need to write a script to import the data to SQL. If you are going that route, let me know, I'm currently writing a VBA script to deal with the exported data from MMO.
If you're hoping to build this into your application (a script of some sort), you may have some luck trying with ogr2ogr, but you never know what the data is going to look like. documentation is found here http://www.gdal.org/ogr2ogr.html
Original answer: Save yourself sometime and just use this amazing tool: http://ogr2gui.ca/
It's based off ogr2ogr, but with a nice GUI.
I've made an app for importing shapefiles into SQL Server. It's made primarily to suit my needs but i had some spare time so i made an installer for you.
Some nice things you can do with it:
choose encoding of input shapefile
rename/remap destination table column names
choose the destination table name
set the primary key for the destination table
It has a user interface and you can download it for free.
More details can be found on my blog, here's the link: Import shapefiles into SQL Server