I am looking for a tool or a library for Rails to validate/parse queries that could be SQL and/or DDL. Currently, I did not find anything that I could use quickly and easily.
I found Parslet that I can use to define my own SQL/DDL language to validate SQL/DDL statements.
The goal to reach is to have a tool that we can use to validate the SQL/DDL syntax before any run on the database. For example, DDL queries are not transactional with MySQL and therefore, if one statement fails at the middle of a bigger script, we need to restore the database or run the script from the failure point (that is not really userfriendly). If we can, at least, validate the syntax, we will improve our daily work by removing a lot of "stupid" errors.
This post lists a few Ruby SQL parsers you might be interested in taking a look at. This one in particular has a Treetop grammar file you could probably use as a base for your own validations.
Related
Is there a solution that helps you manage database schema patches, preferably that runs in Node. I am looking at node-db-migrate which has support for MySQL but I want to be able to run plain vanilla SQL on it instead of using wrapper functions offered by this package.
Is there a solution that offers the same functionality but also lets you have plain SQL in patches?
I'm personally a fan of Sequelize, which has a (poorly named) sister project called Umzug designed to handle migrations. You can absolutely run vanilla SQL in a package like this, but be very careful doing so. Running raw SQL can be an easy way to do something like populate a new column with data (perhaps a new summary column that needs to be filled with the results of a more complex query you've been running until now). However, it can also make it very easy to make mistakes if you do this for things like ALTER TABLE. Whenever you can, it's best to use provided constructs for things like that.
I don't know completely what I want, but surely someone has had the same need, and has solved it in a far better manner than I could:
I'm looking for some mechanism to extract the data definition of a mySQL table from the database and allow it to be queried for the list of columns and their definitions, as part of a routine to dynamically construct DML? It would also be good to have the table parameters (e.g. ENGINE, INDEX, etc.) available, too.
Our databases aren't particularly advanced, and I certainly don't have an encyclopedic knowledge of SQL DDL, so what I came up with probably wouldn't be of much use to anyone else. Is there something already out there in Perl - preferably object-oriented - to do this, at least for mySQL?
Yes, there's a Perl package SQL::Translator, part of a toolset called SQLFairy. It parses SQL DDL from an SQL script or from a live database instance. It supports several RDBMS, including MySQL.
Then it offers tools to do schema conversions, schema diffs, and a bunch of other cool stuff.
http://metacpan.org/pod/SQL::Translator
http://sqlfairy.sourceforge.net/
I found the docs are better than most Perl projects, but still I had to read the code to understand how to use it in the way I wanted to.
The DBI interface has a set of "Catalog Methods": http://metacpan.org/pod/DBI#Catalog-Methods.
There is a similar StackOverflow question you can look at: How do I get schemas from Perl's DBI?
A normal UNION based SQL Injection can be blocked using the WAF, which filter keywords like UNION, SELECT. But I've that it can be bypassed by the inline comment statements of SQL such as /*!UNION*/ and /*!SELECT*/
Comment statements are meant for only reading purpose, right? If so, how a SQL server reads the injection query inside the comments and executes it?
Filtering keywords with a WAF is pointless. There is no way it could possibly succeed. Take a look at this list of ways to bypass it: http://ferruh.mavituna.com/sql-injection-cheatsheet-oku/ (And I found that link in just a few seconds with google.)
If the code was written correctly, it would not be necessary.
As for your question the /*! syntax of MySQL is for MySQL specific commands. It's intended for you to be able to write portable SQL (that can run on any database) and yet still be able to send MySQL special commands.
SQL injection should not be an issue at all if you're using a database driver that supports placeholders for data. What you'd be trying to do with an after-the-fact detection is futile, like trying to eradicate a roach infestation with a ratty fly-swatter. You can't possibly get them all.
The best practice is to ensure it's impossible to inject hostile data into your queries in the first place. There are many examples available on Booby Tables that illustrate how to do this properly.
Commenting out partial query by not closing them can be used for bypassing blacklisting, removing spaces, obfuscating and determining database versions.
The one that you've mentioned is a special comment syntax for MySQL. If you put a code into this comments it's going to execute in MySQL only. Also you can use this to execute some code only if the server is higher than supplied version. For example:
Classical Inline Comment SQL Injection Attack Samples
ID: /*!32302 10*/
ID: 10
You will get the same response if MySQL version is higher than 3.23.02
If I were to want to create a PHP function that does the same thing as the Export tab in phpMyAdmin, how could I do it? I don't know if there is a MySQL function that does this or if phpMyAdmin just builds the export file (in SQL that is) manually. Without shell access. Just using PHP.
I tried the documentation for mysqldump, but that seemed to require using the shell. I'm not quite sure what that even is -- maybe my question is: how do you use shell?
My silly idea is to allow non-technical users to build a site on one server (say a localhost) using MySQL then export the site, database and all, to another server (eg. a remote server).
I think I'm pretty clear on the Import process.
You can check the phpMyAdmin source code (an advantage of open-source software). Check the export.php script and the supporting functions in the libraries/export/sql.php script file.
In summary, what phpMyAdmin does is:
get a list of the tables in the given database (SHOW TABLES FROM...),
get the create query for each table (SHOW CREATE TABLE...),
parse it and extract column definitions from it,
get all data (SELECT * FROM...)
build a query according to column data.
I've written similar code for my own apps (for backup purposes, when the GPL license of phpMyAdmin doesn't allow me to use it), however I use DESCRIBE to get column definitions. I think they rather parse the SHOW CREATE TABLE output because contains more information than DESCRIBE output.
This way to generate SQL sentences requires a bit of care to handle the escaping but it allows for some flexibility, as you can convert types, filter or sanitize data, etc. It is also a lot slower than using a tool like mysqldump and you should take care of not consuming all available memory (write soon, write often, don't keep everything in memory).
If you will implement a migration process (from server to server) maybe it would be easier to do it with some shell scripting and calling mysqldump directly, unless you will do everything with PHP.
I've just been tasked with automating a reporting task at work. Previously, someone would run large, arbitrary SELECTs on a MySQL database using a GUI tool, then use that same tool to export the results to CSV. Now I want to write a Ruby script to do this.
I know about FasterCSV in Ruby, but as far as SQL queries, I've only used ActiveRecord, where you're generally not writing the queries but using models and associations. The last time I wrote out complete SQL in code was when I coded PHP.
What's the most straightforward way to do this sort of thing in Ruby? Should I use ActiveRecord?
If you are writing a complex application, there is absolutely nothing wrong with using ActiveRecord. Especially since you are already comfortable with it.
If you are writing a quick script and don't want to bother with ActiveRecord, you should also check out the mysql and mysql2 gems.
I highly recommend Sequel. It has great documentation, active development, a thriving and helpful community, and it is (IMHO) simpler and better than ActiveRecord, especially for simple use cases like you appear to have.
You might want to start by reading the README, Cheat Sheet and Sequel for SQL Users.
I often find that in reporting engines you end up with complex queries processing large amounts of data and ORMs like ActiveRecord just don't cut it. Unless your reports are really simple, I think you'll want to use raw SQL (with a tool like Sequel) or call stored procedures.