We are finally moving from Excel and .csv files to databases. Currently, most of my Tableau files are connected to large .csv files (.twbx).
Is there any performance differences between PostgreSQL and MySQL in Tableau? Which would you choose if you were starting from scratch?
Right now, I am using pandas to join files together and creating a new .csv file based on the join.(Example, I take a 10mil row file and drop duplicates and create a primary key, then I join it with the same key on a 5mil row file, then I export the new 'Consolidated' file to .csv and connect Tableau to it. Sometimes the joins are complicated involving dates or times and several columns).
I assume I can create a view in a database and then connect to that view rather than creating a separate file, correct? Each of my files could instead be a separate table which should save space and allow me to query dates rather than reading the whole file into memory with pandas.
Some of the people using the RDMS would be completely new to databases in general (dashboards here are just Excel files, no normalization, formulas in the raw data sheet, etc.. it's a mess) so hopefully either choice has some good documentation to lesson the learning curve (inserting new data and selecting data mainly, not the actual database design).
Both will work fine with Tableau. In fact, Tableau's internal data engine is based on Postgres.
Between the two, I think Postgres is more suitable for a central data warehouse. MySQL doesn’t allow certain SQL methods such as Common Table Expressions and Window Functions.
Also, if you’re already using Pandas, Postgres has a built-in Python extension called PL/Python.
However, if you’re looking to store a small amount of data and get to it really fast without using advanced SQL, MySQL would be a fine choice but Postgres will give you a few more options moving forward.
As stated, either database will work and Tableau is basically agnostic to the type of database that you use. Check out https://www.tableau.com/products/techspecs for a full list of all native (inbuilt & optimized) connections that Tableau Server and Desktop offer. But, if your database isn't on that list you can always connect over ODBC.
Personally, I prefer postgres over mysql (I find it really easy to use psycopg2 to write to postgres from python), but mileage will vary.
Related
My office has a Filemaker database which they asked me to replace with a mySQL one. The mySQL one is now set up and running, but doesn't have exact same structure as the Filemaker one (they asked for more things to be added, redundant things were left out, etc.).
I've seen that the filemaker data can be exported as .xml files, could I use those to populate the mySQL database?
If so, I've only ever used
Cooktop,but I'm currently using mac10.6/lubuntu, is there maybe an equivalent (free) piece of software that could do that?
All suggestions are welcome.
Thanks
I can add some information about the various export formats that FileMaker provides. I've done extensive research and testing on this topic.
Below, you'll see a chart with all of the formats that FileMaker offers along the top. Along the side, you'll see various features of these file formats that are unique to FileMaker when exporting. Some are limitations of the FileMaker export process and others are general pros and cons of the format itself.
I'll explain them briefly:
Headers: column labels are exported
Delimiter: the type of separator symbol used
UTF 8/16: yes if either of these is available, could be of concern for special characters or some languages
Only 1 format: means that only one type of encoding is available
Other encoding: a list of all encoding options
Can be imported: FileMaker allows import (not important for this question)
Future proof: According to Wikipedia, format is still widely used and actively maintained
Open standard: open source format
Size: when exporting one of our tables, this was the size of the file
I would recommend also considering some of these factors when deciding which format will work for you. It will depend on the contents and type of your data.
MySQL is just the backend database, so you need a UI to perform the import. You could use FileMaker for this as well, if you set up the MySQL database as an ESS source. If you do this, then you can use familiar import steps in FileMaker to populate your new database.
This may be what the previous answer mentions, but just to designate between the ODBC insert via Execute SQL, which is limited, and External SQL Sources (ESS) that give a native UI in FM.
If the FileMaker database is hosted on a server, you could setup an ODBC link to the MySQL. You could then create a script, in FileMaker, to loop through the data, creating rows in MySQL with only the Columns you are looking to populate.
Other than that, you can export the data, from FileMaker, into many other formats including; TAB, CSV, Excel, xml and pushing it into MySQL.
I have a Database which was created using Java DB, Would it be possible for me to manipulate this Database using access?Ive tried using the wizard but it asks me for a dsn and I cant seem to figure out where to get that from
Edit: I would like to modify the database from both MS-Access and javadb
The tables are (relatively) simple currently & it contains 2 relationships; one to zero/one and one to many
Should I convert it to csv and then after I'm finished convert it back to Java DB again?
There are several options. If you want to be able to modify the database from both Derby and MS-Access, then you will need to locate an ODBC driver for MS-Access that would enable it to read and modify a Derby database. Off hand, I do not know if such a driver exists.
If you wish to simply convert the database to an MS-Access format, then ...
... much will depend on the structure of the underlying tables and how much of the schema you need to preserve.
The simplest thing would be that you have a database consisting of a single table that resembles a flat-file database. In this case, you can write a simple JDBC tool to write your database out as a CSV file (for example) that MS-Access can read.
If you have multiple tables or data that can't be easily represented by a CSV file, then things become more complex. You haven't told us anything about your needs or the structure of your database, so it is unlikely that anyone can be more specific.
I have a few XML files containing data for a research project which I need to run some statistics on. The amount of data is close to 100GB.
The structure is not so complex (could be mapped to perhaps 10 tables in a relational model), and given the nature of the problem, this data will never be updated again, I only need it available in a place where it's easy to run queries on.
I've read about XML databases, and the possibility of running XPATH-style queries on it, but I never used them and I'm not so comfortable with it. Having the data in a relational database would be my preferred choice.
So, I'm looking for a way to covert the data stored in XML into a relational database (think of a big .sql file similar to the one generated by mysqldump, but anything else would do).
The ultimate goal is to be able to run SQL queries for crunching the data.
After some research I'm almost convinced I have to write it on my own.
But I feel this is a common problem, and therefore there should be a tool which already does that.
So, do you know of any tool that would transform XML data into a relational database?
PS1:
My idea would be something like (it can work differently, but just to make sure you get my point):
Analyse the data structure (based on the XML themselves, or on a XSD)
Build the relational database (tables, keys) based on that structure
Generate SQL statements to create the database
Generate SQL statements to create fill in the data
PS2:
I've seen some posts here in SO but still I couldn't find a solution.
Microsoft's "Xml Bulk Load" tool seems to do something in that direction, but I don't have a MS SQL Server.
Databases are not the only way to search data. I can highly recommend Apache Solr
Strategies to Implement search on XML file
Keep your raw data as XML and search it using the Solr index
Importing XML files of the right format into a MySql database is easy:
https://dev.mysql.com/doc/refman/5.6/en/load-xml.html
This means, you typically have to transform your XML data into that kind of format. How you do this depends on the complexity of the transformation, what programming languages you know, and if you want to use XSLT (which is most probably a good idea).
From your former answers it seems you know Python, so http://xmlsoft.org/XSLT/python.html may be the right thing for you to start with.
Take a look at StAX instead of XSD for analyzing/extraction of data. It's stream based and can deal with huge XML files.
If you feel comfortable with Perl, I've had pretty good luck with XML::Twig module for processing really big XML files.
Basically, all you need is to setup few twig handlers and import your data into MySQL using DBI/DBD::mysql.
There is pretty good example on xmltwig.org.
If you comfortable with commercial products, you might want to have a look at Data Wizard for MySQL by the SQL Maestro Group.
This application is targeted especially at exporting and, of course, importing data from/ to MySQL databases. This also includes XML import. You can download a 30-day trial to check if this is what you are looking for.
I have to admit that I did not use the MySQL product line from them yet, but I had a good user experience with their Firebird Maestro and SQLite Maestro products.
I currently have a database in MySQL, which I'd like to import in MS Access.
Is it possible to do this while keeping all relationships intact (i.e. without exporting to .csv, or by using ODBC)?
I'm a noob in this area so any help is greatly appreciated.
Thanks.
You need to solve two different problems:
Creating an empty MS Access database with a structure that matches the MySQL database structure.
Extracting the data from MySQL and loading it into MS Access.
This is not easy because different SQL databases offer different structural features, different datatypes, and so on. The more complex your use of MySQL is the more likely you'll run into some show-stopper during the conversion (for instance, Access doesn't support triggers at all). Conversely if you're using MySQL as a simple data store you may find the conversion fairly easy.
To get an MS Access database with the same structure as your MySQL database, your best bet is to find a database definition / diagramming tool that offers reverse engineering and supports both MySQL and MS Access. Use it to reverse engineer your MySQL database into a database diagram, then change the underlying database to MS Access and use the tool to generate a database.
Check out Dezign For Databases which (on paper, anyway) offers the features you would need to do this.
To pump the data across, there are any number of tools. This kind of operation is generically referred to as ETL (Extract, Translate, Load).
Do you mean SQL Server? A good starting point might be to check out SQL Server Integration Services (SSIS), which can be used for transferring data around like that.
Google will also be helpful, check out the first result:
http://support.microsoft.com/kb/237980
By the way, you said ".sql" in your question: a .SQL file is a script file, which could do anything from create a database, insert data, drop table, delete data, or given the right permissions, call system procedures and reboot a machine, format a drive, send an email.. Just for ref, .SQL files aren't the storage format used by SQL Server.
While you can script your database's schema into script files via something like SQLyog, you will find that the syntax varies enough from database to database (MySQL to Access, in your case) that you can't directly apply the scripts.
With much effort a conversion script could be created by editing the script (perhaps automated with a program, depending on the resulting script size). I think you would be better served using ODBC to copy the tables (and data) and then extracting and re-applying the relationships from the generated script by hand. Time consuming, but also a one time operation I would hope.
When both systems are the same database, there are tools that can do the comparison and script generation (TOAD for MySQL and RedGate Compare for Microsoft SQL), but they don't do cross database work (at least not the ones I am aware of).
If you create a ODBC DSN, you can use TransferDatabase to import from your MySQL database. You can do it manually with the GET EXTERNAL DATA command (or whatever it is in A2007/A2010) and see how well it works. It won't get all data types exactly right, but you could do some massaging and likely get it closer to what will work best.
Is there some reason you can't just link to the MySQL tables and use them directly? That is, why do you need to import into Access at all?
Access: run query. Just make sure to adapt the SQL code since every RDMS has its own sintaxis (despite SQL being an ANSI standard).
Are there any ways to import data in databases such as MS SQL, MySQL into in-memory databases like HSQLDB, H2 etc ?
H2 supports a special database URL that initialized the database from a SQL script file:
"jdbc:h2:mem;INIT=RUNSCRIPT FROM '~/create.sql'"
HSQLDB and Apache Derby don't support such a feature as far as I know.
I think you need to do
query the data out from MS SQL
import the data into in-memory DB with its API
Either SQL expressions or DB related APIs
In Hibernate: Adding import.sql to the class path works great, hbm2dll checks if the file exists and executes it. The only details is that every sql command most be on one row, otherwise it will fail to execute
You could dump the data as SQL INSERT statements then read it back.
You could read to a temporay object (like a struct) then write back to the internal db.
Look at the free "universal database converter" http://eva-3-universal-database-converter-udc.optadat-com.qarchive.org/ -- it does claim to support MySQL, MS-SQL, and HSQLDB, among others.
It really depends on what ways you think about.
Is there a tool that could do it automatically without programming? Maybe.
Do you want to develop it? Then find out whether your favorite language supports both database engines(standard and in memory) and if it does, just write a script that does it.
Process everything in chunks(fetch n rows at a time then insert them; repeat). How big the chunk size? It's up to you, try different sizes(say 100, 500, 1k etc.) see which one performs better on your hardware, fine tune to the sweet spot.
If your favorite language on the other hand doesn't support both of them, try using something that does.
You can use dbunit for dumping the database to xml files and importing it back to another rdbms.
Latest versions of HSQLDB allow you to open a CSV (comma separated values) or other delimiter separated data file as a TEXT TABLE in HSQLDB even with mem: databases, which can then be copied to other tables.
As others have pointed out, there are also capable and well maintained third party tools for this purpose.