My office has a Filemaker database which they asked me to replace with a mySQL one. The mySQL one is now set up and running, but doesn't have exact same structure as the Filemaker one (they asked for more things to be added, redundant things were left out, etc.).
I've seen that the filemaker data can be exported as .xml files, could I use those to populate the mySQL database?
If so, I've only ever used
Cooktop,but I'm currently using mac10.6/lubuntu, is there maybe an equivalent (free) piece of software that could do that?
All suggestions are welcome.
Thanks
I can add some information about the various export formats that FileMaker provides. I've done extensive research and testing on this topic.
Below, you'll see a chart with all of the formats that FileMaker offers along the top. Along the side, you'll see various features of these file formats that are unique to FileMaker when exporting. Some are limitations of the FileMaker export process and others are general pros and cons of the format itself.
I'll explain them briefly:
Headers: column labels are exported
Delimiter: the type of separator symbol used
UTF 8/16: yes if either of these is available, could be of concern for special characters or some languages
Only 1 format: means that only one type of encoding is available
Other encoding: a list of all encoding options
Can be imported: FileMaker allows import (not important for this question)
Future proof: According to Wikipedia, format is still widely used and actively maintained
Open standard: open source format
Size: when exporting one of our tables, this was the size of the file
I would recommend also considering some of these factors when deciding which format will work for you. It will depend on the contents and type of your data.
MySQL is just the backend database, so you need a UI to perform the import. You could use FileMaker for this as well, if you set up the MySQL database as an ESS source. If you do this, then you can use familiar import steps in FileMaker to populate your new database.
This may be what the previous answer mentions, but just to designate between the ODBC insert via Execute SQL, which is limited, and External SQL Sources (ESS) that give a native UI in FM.
If the FileMaker database is hosted on a server, you could setup an ODBC link to the MySQL. You could then create a script, in FileMaker, to loop through the data, creating rows in MySQL with only the Columns you are looking to populate.
Other than that, you can export the data, from FileMaker, into many other formats including; TAB, CSV, Excel, xml and pushing it into MySQL.
Related
We are finally moving from Excel and .csv files to databases. Currently, most of my Tableau files are connected to large .csv files (.twbx).
Is there any performance differences between PostgreSQL and MySQL in Tableau? Which would you choose if you were starting from scratch?
Right now, I am using pandas to join files together and creating a new .csv file based on the join.(Example, I take a 10mil row file and drop duplicates and create a primary key, then I join it with the same key on a 5mil row file, then I export the new 'Consolidated' file to .csv and connect Tableau to it. Sometimes the joins are complicated involving dates or times and several columns).
I assume I can create a view in a database and then connect to that view rather than creating a separate file, correct? Each of my files could instead be a separate table which should save space and allow me to query dates rather than reading the whole file into memory with pandas.
Some of the people using the RDMS would be completely new to databases in general (dashboards here are just Excel files, no normalization, formulas in the raw data sheet, etc.. it's a mess) so hopefully either choice has some good documentation to lesson the learning curve (inserting new data and selecting data mainly, not the actual database design).
Both will work fine with Tableau. In fact, Tableau's internal data engine is based on Postgres.
Between the two, I think Postgres is more suitable for a central data warehouse. MySQL doesn’t allow certain SQL methods such as Common Table Expressions and Window Functions.
Also, if you’re already using Pandas, Postgres has a built-in Python extension called PL/Python.
However, if you’re looking to store a small amount of data and get to it really fast without using advanced SQL, MySQL would be a fine choice but Postgres will give you a few more options moving forward.
As stated, either database will work and Tableau is basically agnostic to the type of database that you use. Check out https://www.tableau.com/products/techspecs for a full list of all native (inbuilt & optimized) connections that Tableau Server and Desktop offer. But, if your database isn't on that list you can always connect over ODBC.
Personally, I prefer postgres over mysql (I find it really easy to use psycopg2 to write to postgres from python), but mileage will vary.
I am pretty new to databases and need help. I have n (large) files, each file contains m (very large) text file (numeric data). What is the best way to import those files into a mysql database concerning the names of the fields?
usually one would write script with perl (or whatever scripting language is preferred, offering MySQL Support ) and process files one after other, applying necessary processing to files / lines inside files. If you like more specific answer, ask more specific question
If you only need to do it once, or the import process remains fairly similar each time, I would recommend using the ETL software Kettle by Pentaho (this bit of software is commonly referred to as kettle). While this software is far from perfect, I've found that I can often import data in a fraction of the time I would have to spend writing a script for one specific file. You can select a text file input and specify the delimiters, fixed width, etc.. and then simply export directly into your SQL server (they support MySql, SQLite, Oracle, and much more).
If you would like to research other types of software like this, its often referred to as ETL software, short for Extract Transform Load.
If your familiar with python, I would also recommend the last post on this page
I need an alternative to exporting to a csv file. Basically, the requirement is to be able to allow the user to edit any amount of data from the database. Currently, the option is to export the data to csv and then the user can update them in Microsoft Excel. Thus, one can edit it without actually needing to have our web application installed. However, we are trying to avoid this approach due to some issues we encountered with working with CSV on Excel. Can someone recommend an alternative solution?
Most programming languages have libraries to allow you to import/export excel directly which avoids many of these issues. This means you won't have to tell you users to do special Save As when saving their files.
Java: http://poi.apache.org
Ruby: http://spreadsheet.rubyforge.org or http://roo.rubyforge.org
Google can provide options for other languages.
I currently have a database in MySQL, which I'd like to import in MS Access.
Is it possible to do this while keeping all relationships intact (i.e. without exporting to .csv, or by using ODBC)?
I'm a noob in this area so any help is greatly appreciated.
Thanks.
You need to solve two different problems:
Creating an empty MS Access database with a structure that matches the MySQL database structure.
Extracting the data from MySQL and loading it into MS Access.
This is not easy because different SQL databases offer different structural features, different datatypes, and so on. The more complex your use of MySQL is the more likely you'll run into some show-stopper during the conversion (for instance, Access doesn't support triggers at all). Conversely if you're using MySQL as a simple data store you may find the conversion fairly easy.
To get an MS Access database with the same structure as your MySQL database, your best bet is to find a database definition / diagramming tool that offers reverse engineering and supports both MySQL and MS Access. Use it to reverse engineer your MySQL database into a database diagram, then change the underlying database to MS Access and use the tool to generate a database.
Check out Dezign For Databases which (on paper, anyway) offers the features you would need to do this.
To pump the data across, there are any number of tools. This kind of operation is generically referred to as ETL (Extract, Translate, Load).
Do you mean SQL Server? A good starting point might be to check out SQL Server Integration Services (SSIS), which can be used for transferring data around like that.
Google will also be helpful, check out the first result:
http://support.microsoft.com/kb/237980
By the way, you said ".sql" in your question: a .SQL file is a script file, which could do anything from create a database, insert data, drop table, delete data, or given the right permissions, call system procedures and reboot a machine, format a drive, send an email.. Just for ref, .SQL files aren't the storage format used by SQL Server.
While you can script your database's schema into script files via something like SQLyog, you will find that the syntax varies enough from database to database (MySQL to Access, in your case) that you can't directly apply the scripts.
With much effort a conversion script could be created by editing the script (perhaps automated with a program, depending on the resulting script size). I think you would be better served using ODBC to copy the tables (and data) and then extracting and re-applying the relationships from the generated script by hand. Time consuming, but also a one time operation I would hope.
When both systems are the same database, there are tools that can do the comparison and script generation (TOAD for MySQL and RedGate Compare for Microsoft SQL), but they don't do cross database work (at least not the ones I am aware of).
If you create a ODBC DSN, you can use TransferDatabase to import from your MySQL database. You can do it manually with the GET EXTERNAL DATA command (or whatever it is in A2007/A2010) and see how well it works. It won't get all data types exactly right, but you could do some massaging and likely get it closer to what will work best.
Is there some reason you can't just link to the MySQL tables and use them directly? That is, why do you need to import into Access at all?
Access: run query. Just make sure to adapt the SQL code since every RDMS has its own sintaxis (despite SQL being an ANSI standard).
I have a shapefile that I would like to upload to a spatial SQL Server 2008. I have tried using this tool: SQL Server 2008 Spatial Tools. But without luck.
Does anyone know any other (free) tools for doing this?
You can use ogr2ogr to convert from shapefile to GML (or many other formats) and then use SQL Server's GeomFromGML to import. You will need to call GeomFromGML for each feature in your shapefile, but that's a relatively easy program to write.
I have written code in arcbjects for do this task. If you have license arcview or engine you can create a console application in c# and use this code: see https://gis.stackexchange.com/questions/33917/how-to-import-shapefiles-into-ms-sql-2008-and-then-view-that-data-using-qgis?lq=1
UPDATE: I decided I would just point people to the official Github repo instead: https://github.com/zer0infinity/OGR2GUI
This [ogr2ogr fork] tool will attempt to parse the content of the input file (in my case, it was a shapefile) and output it as a bunch of different formats (in my case, I needed it as a CSV, but you can even export your file as a SQLite file). Unfortunately it doens't do straight up SQL, but you can do a dump from a SQLite viewer such as SQLite Browser and import it onto MS SQL. I did notice some inefficiencies when converting to SQLite (I lost some attributes/tables). I also tried MobileMapper Office (MMO) (with better luck) to export to CSV and it did preserve a lot of the data, but you'd then need to write a script to import the data to SQL. If you are going that route, let me know, I'm currently writing a VBA script to deal with the exported data from MMO.
If you're hoping to build this into your application (a script of some sort), you may have some luck trying with ogr2ogr, but you never know what the data is going to look like. documentation is found here http://www.gdal.org/ogr2ogr.html
Original answer: Save yourself sometime and just use this amazing tool: http://ogr2gui.ca/
It's based off ogr2ogr, but with a nice GUI.
I've made an app for importing shapefiles into SQL Server. It's made primarily to suit my needs but i had some spare time so i made an installer for you.
Some nice things you can do with it:
choose encoding of input shapefile
rename/remap destination table column names
choose the destination table name
set the primary key for the destination table
It has a user interface and you can download it for free.
More details can be found on my blog, here's the link: Import shapefiles into SQL Server