Is there a solution that helps you manage database schema patches, preferably that runs in Node. I am looking at node-db-migrate which has support for MySQL but I want to be able to run plain vanilla SQL on it instead of using wrapper functions offered by this package.
Is there a solution that offers the same functionality but also lets you have plain SQL in patches?
I'm personally a fan of Sequelize, which has a (poorly named) sister project called Umzug designed to handle migrations. You can absolutely run vanilla SQL in a package like this, but be very careful doing so. Running raw SQL can be an easy way to do something like populate a new column with data (perhaps a new summary column that needs to be filled with the results of a more complex query you've been running until now). However, it can also make it very easy to make mistakes if you do this for things like ALTER TABLE. Whenever you can, it's best to use provided constructs for things like that.
Related
I'm using MODx CMS and would like to use git for co-working. However, as you know, MODx stores some of code places in database. So, I'd like to know, how to version MySQL for co-working except dumping? Thanks
You'll need to store the database in some format, and it can't be SQL as you can't just tell a live database to update the schema using a SQL file.
Basically, you'll have to store an incremental set of commands that updates the database. Git won't really help you in this case, you are going to have to use your own system.
The easiest way to achieve this is by just numbering your database versions and storing only the changes in your favorite programming language.
Step 1: store the current database version somewhere.
Step 2: store all updates to the database in files.
update/mysql_1_to_2.py
update/mysql_2_to_3.py
# etc
Step 3: create the initial database
update/mysql_0_to_1.py
If you use that system, all you need to do is keep track of the current in use version and the version that's most recent in your code. When you need to update, just run all mysql_x_to_y scripts and you're done with the versioning. mysql_0_to_1 can do the initial bootstrap and if you somehow manage to ever reach 1000 revisions, just use mysql_0_to_1000 to take a shortcut.
The system I outlined above should work really well for linear repositories, but might be a bit harder to use for trees (branches, multiple developers, etc). I'm sure you can come up with something though.
It sounds like perhaps you are talking about stored procedures and/or user-defined functions?
In any case, the best method is mysqldump - presumably, you don't need the data, just the stored procs, user-defined functions, and perhaps table structures. There are options for mysqldump to control what gets output.
If you want to add versioning of database to versioning of code, you have to see at Liquibase. To avoid repeating everything again, my older full answer on topic
I am looking for a tool or a library for Rails to validate/parse queries that could be SQL and/or DDL. Currently, I did not find anything that I could use quickly and easily.
I found Parslet that I can use to define my own SQL/DDL language to validate SQL/DDL statements.
The goal to reach is to have a tool that we can use to validate the SQL/DDL syntax before any run on the database. For example, DDL queries are not transactional with MySQL and therefore, if one statement fails at the middle of a bigger script, we need to restore the database or run the script from the failure point (that is not really userfriendly). If we can, at least, validate the syntax, we will improve our daily work by removing a lot of "stupid" errors.
This post lists a few Ruby SQL parsers you might be interested in taking a look at. This one in particular has a Treetop grammar file you could probably use as a base for your own validations.
I've just been tasked with automating a reporting task at work. Previously, someone would run large, arbitrary SELECTs on a MySQL database using a GUI tool, then use that same tool to export the results to CSV. Now I want to write a Ruby script to do this.
I know about FasterCSV in Ruby, but as far as SQL queries, I've only used ActiveRecord, where you're generally not writing the queries but using models and associations. The last time I wrote out complete SQL in code was when I coded PHP.
What's the most straightforward way to do this sort of thing in Ruby? Should I use ActiveRecord?
If you are writing a complex application, there is absolutely nothing wrong with using ActiveRecord. Especially since you are already comfortable with it.
If you are writing a quick script and don't want to bother with ActiveRecord, you should also check out the mysql and mysql2 gems.
I highly recommend Sequel. It has great documentation, active development, a thriving and helpful community, and it is (IMHO) simpler and better than ActiveRecord, especially for simple use cases like you appear to have.
You might want to start by reading the README, Cheat Sheet and Sequel for SQL Users.
I often find that in reporting engines you end up with complex queries processing large amounts of data and ORMs like ActiveRecord just don't cut it. Unless your reports are really simple, I think you'll want to use raw SQL (with a tool like Sequel) or call stored procedures.
In our rails app we sometimes have db entries created by users that we'd like to make part of our dev environment, without exporting the whole table. So, we'd like to be able to have a special 'dev and testing' dump.
Any recommended best practices? mysqldump seems pretty cumbersome, and we'd like to pull in rails associations as well, so maybe a rake task would make more sense.
Ideas?
You could use an ETL tool like Pentaho Kettle. Once you have initial transformation setup that you want you could easily run it with different parameters in the future. This way you could also keep all your associations. I wrote a little blurb about Pentaho for another question here.
If you provide a rough schema I could probably help you get started on what your transformation would look like.
I had a similar need and I ended up creating a plugin for that. It was developed for Rails 2.x and worked fine for me, but I didn't have much use for it lately.
The documentation is lacking, but it's pretty simple. You basically install the plugin and then have a method to_sql available on all your models. Options are explained in README.
You can try it out and let me know if you have any issues, I'll try to help.
I'd go after it using a Rails runner script. That will allow your code to access the same things your Rails app would, including the database initializations. ActiveRecord will be able to take advantage of the model relationships you've defined.
Create some "transfer" tables in your production database and copy the desired data into those using the "runner" script. From there you could serialize the data, or use a dump tool, since you'll be dealing with a reduced amount of records. Reverse the process in the development environment to move the data into the database.
I had a need to populate the database in one of my apps from remote web logs and wrote a runner script that fired off periodically via cron, ftps the data from my site and inserts the data.
I have a small amount of experience using SVN on my development projects, and I have just as little experience with relational databases. I know the basic concepts like tables, and SQL statements, but I'm far from being an expert.
What I'd like to know is if there are any generic version control type systems like SVN, but that work with a database rather than files. I would like the same kind of features you get with SVN like the ability to create branches, create tags, and merge branches together. Rather than a revision number being associated to a version of a file repository it would be associated with a version of the database.
Are their any generic solutions available that can add this kind of functionality independent of the actual database schema? I'd be interested in solutions that work with MySQL or MS SQL Server.
I should also clarify that I'm trying to version control the data not the schema. I would expect the schema to remain constant. So really it seems like I want a way to create a log of all the INSERT, UPDATE, and DELETE requests sent the the database between each version of the data. That way any version could be recreated by resending all the SQL statements that have been saved up to the desired version.
You can script all your DDL, stored procedures and such to regular text files.
Then you can simply use SVN for database versioning.
I've never found a solution that works as well as Subversion, but here's a few things I've done that have helped:
Make scripts that will create the schema and populate any initial data. Then make an update script for each change after that. It's a fairly manual process, but it works. There's extra things that help like storing the current version number in a table in the db and making sure that the scripts are idempotent.
Store the full development db in Subversion. This doesn't usually work out too well for me if there is a lot of data or it is frequently changed. But in some projects is could work.
I keep and maintain create scripts in my version control system.
There are two things I can think of:
http://www.liquibase.org/ - provides a way of generally managing database changes. Creates files that get committed into source control, and it helps manage changes across different development databases, etc.
http://www.viget.com/extend/backup-your-database-in-git/ - this describes a strategy for backing up a database into source control, but the same strategy can be used just on the schema. In this scheme, the database would be in a separate area from your main code. (This can be used with other source control systems too.)