I don't know completely what I want, but surely someone has had the same need, and has solved it in a far better manner than I could:
I'm looking for some mechanism to extract the data definition of a mySQL table from the database and allow it to be queried for the list of columns and their definitions, as part of a routine to dynamically construct DML? It would also be good to have the table parameters (e.g. ENGINE, INDEX, etc.) available, too.
Our databases aren't particularly advanced, and I certainly don't have an encyclopedic knowledge of SQL DDL, so what I came up with probably wouldn't be of much use to anyone else. Is there something already out there in Perl - preferably object-oriented - to do this, at least for mySQL?
Yes, there's a Perl package SQL::Translator, part of a toolset called SQLFairy. It parses SQL DDL from an SQL script or from a live database instance. It supports several RDBMS, including MySQL.
Then it offers tools to do schema conversions, schema diffs, and a bunch of other cool stuff.
http://metacpan.org/pod/SQL::Translator
http://sqlfairy.sourceforge.net/
I found the docs are better than most Perl projects, but still I had to read the code to understand how to use it in the way I wanted to.
The DBI interface has a set of "Catalog Methods": http://metacpan.org/pod/DBI#Catalog-Methods.
There is a similar StackOverflow question you can look at: How do I get schemas from Perl's DBI?
Related
I have a few XML files containing data for a research project which I need to run some statistics on. The amount of data is close to 100GB.
The structure is not so complex (could be mapped to perhaps 10 tables in a relational model), and given the nature of the problem, this data will never be updated again, I only need it available in a place where it's easy to run queries on.
I've read about XML databases, and the possibility of running XPATH-style queries on it, but I never used them and I'm not so comfortable with it. Having the data in a relational database would be my preferred choice.
So, I'm looking for a way to covert the data stored in XML into a relational database (think of a big .sql file similar to the one generated by mysqldump, but anything else would do).
The ultimate goal is to be able to run SQL queries for crunching the data.
After some research I'm almost convinced I have to write it on my own.
But I feel this is a common problem, and therefore there should be a tool which already does that.
So, do you know of any tool that would transform XML data into a relational database?
PS1:
My idea would be something like (it can work differently, but just to make sure you get my point):
Analyse the data structure (based on the XML themselves, or on a XSD)
Build the relational database (tables, keys) based on that structure
Generate SQL statements to create the database
Generate SQL statements to create fill in the data
PS2:
I've seen some posts here in SO but still I couldn't find a solution.
Microsoft's "Xml Bulk Load" tool seems to do something in that direction, but I don't have a MS SQL Server.
Databases are not the only way to search data. I can highly recommend Apache Solr
Strategies to Implement search on XML file
Keep your raw data as XML and search it using the Solr index
Importing XML files of the right format into a MySql database is easy:
https://dev.mysql.com/doc/refman/5.6/en/load-xml.html
This means, you typically have to transform your XML data into that kind of format. How you do this depends on the complexity of the transformation, what programming languages you know, and if you want to use XSLT (which is most probably a good idea).
From your former answers it seems you know Python, so http://xmlsoft.org/XSLT/python.html may be the right thing for you to start with.
Take a look at StAX instead of XSD for analyzing/extraction of data. It's stream based and can deal with huge XML files.
If you feel comfortable with Perl, I've had pretty good luck with XML::Twig module for processing really big XML files.
Basically, all you need is to setup few twig handlers and import your data into MySQL using DBI/DBD::mysql.
There is pretty good example on xmltwig.org.
If you comfortable with commercial products, you might want to have a look at Data Wizard for MySQL by the SQL Maestro Group.
This application is targeted especially at exporting and, of course, importing data from/ to MySQL databases. This also includes XML import. You can download a 30-day trial to check if this is what you are looking for.
I have to admit that I did not use the MySQL product line from them yet, but I had a good user experience with their Firebird Maestro and SQLite Maestro products.
I am looking for a tool or a library for Rails to validate/parse queries that could be SQL and/or DDL. Currently, I did not find anything that I could use quickly and easily.
I found Parslet that I can use to define my own SQL/DDL language to validate SQL/DDL statements.
The goal to reach is to have a tool that we can use to validate the SQL/DDL syntax before any run on the database. For example, DDL queries are not transactional with MySQL and therefore, if one statement fails at the middle of a bigger script, we need to restore the database or run the script from the failure point (that is not really userfriendly). If we can, at least, validate the syntax, we will improve our daily work by removing a lot of "stupid" errors.
This post lists a few Ruby SQL parsers you might be interested in taking a look at. This one in particular has a Treetop grammar file you could probably use as a base for your own validations.
Does anyone know of a quick and easy test to see if a query is properly formatted for both MySQL & MSSQL. Perhaps other database types as well, such as SQL Server? I only have access to MySQL at this time.
Info: I'm working on an Open Source project called JJWDesign Google Maps for SugarCRM. Some of the queries use the SugarCRM classes; others I have to write custom. For example, some are special distance calculations against the geocode information stored in the tables.
http://www.sugarforge.org/projects/jjwgooglemaps/
More importantly, while there is an accepted syntax, each flavour of database has it's own specific functions, features and things you can do.
The best you can do is to make do with the most basic of features. Oracle has different functions for datetime compared to mysql compared to db2. While I would love to assist in a 'free as in beer' project, you really will need to check each function to see if it is the same across all major vendors. General functions most often are, so abs() will be fairly consistent, but others simply won't.
You're talking about a SQL parser so by definition it either isn't going to be quick and easy or it will do only the simplest checking.
Each RDBMS has its own flavour of SQL too so you'd really be limited to testing whether it was ANSI SQL.
This is kind of implementation question maybe. I wonder if I where to make a tool to convert some relational database to some other kind of database. What would the approach be?
If I for example want to convert data and the structure from a mysql database to mssql. Would I need to use regular expression to parse the SQL-file? Or maybe I could convert it to XML or JSON first and from that structure parse into my targeted database?
Using existing tools for converting mysql to mssql or anything similar is not in this scope. Since I want to know how it is actually done.
Well it's kind of a broad question, but generally speaking, having your own abstract representation of the structure and data would be a good thing, because you could extend your system "easily" by writing importers and exporters, and actually decouple your code a little by abstracting the relational db concepts into your own format.
The importers would "reverse engineer" a given database, by converting it to your own representation (as you say, xml/json or even your own query language -that would be better I guess-). Then the exporters would just convert from your format to the requested SQL dialect. No regular expressions, no other stuff "hardcoded".
This will allow you to extend your system and support a bigger number of sources and targets, and also handle errors like some SQL features from a "source" not supported in the selected "target".
My 2 cents, hope it helps!
I have an existing Schema definition in MySql database. I created the schema using MySql Workbench.
I wish to access the schema from my Lift-Scala-Squeryl code. I know that a simple way would be to manually define the schema structure using Squeryl data objects.
Is there an automated way to generate Squeryl data objects out of existing MySql schema?
I've found the following general question, but I'm sure there can be a way to generate a naive structure, although not accurate, it will allow a better starting point for the manual work.
Thanks, David.
Max, Squeryl's creator, had suggested that this would be a good idea a while back. Here is the google group discussion.
You may not be too pleased with me for this, but I think I talked him out of it :) So, to my knowledge, there isn't a way to do it. Besides the issues I pointed out in that thread, the fact that Squeryl can work in multiple modes (primitive types, custom types, lift record types) it would be a difficult thing to do and get right for everyone.