In our rails app we sometimes have db entries created by users that we'd like to make part of our dev environment, without exporting the whole table. So, we'd like to be able to have a special 'dev and testing' dump.
Any recommended best practices? mysqldump seems pretty cumbersome, and we'd like to pull in rails associations as well, so maybe a rake task would make more sense.
Ideas?
You could use an ETL tool like Pentaho Kettle. Once you have initial transformation setup that you want you could easily run it with different parameters in the future. This way you could also keep all your associations. I wrote a little blurb about Pentaho for another question here.
If you provide a rough schema I could probably help you get started on what your transformation would look like.
I had a similar need and I ended up creating a plugin for that. It was developed for Rails 2.x and worked fine for me, but I didn't have much use for it lately.
The documentation is lacking, but it's pretty simple. You basically install the plugin and then have a method to_sql available on all your models. Options are explained in README.
You can try it out and let me know if you have any issues, I'll try to help.
I'd go after it using a Rails runner script. That will allow your code to access the same things your Rails app would, including the database initializations. ActiveRecord will be able to take advantage of the model relationships you've defined.
Create some "transfer" tables in your production database and copy the desired data into those using the "runner" script. From there you could serialize the data, or use a dump tool, since you'll be dealing with a reduced amount of records. Reverse the process in the development environment to move the data into the database.
I had a need to populate the database in one of my apps from remote web logs and wrote a runner script that fired off periodically via cron, ftps the data from my site and inserts the data.
Related
I'm creating a javafx app with using wamp server for mysql database, how to make automatic backup even if database become large for a specific location, Or what's best practice in this case?
What can I do by java for this issue?
I would suggest to create a cron job for this.
You can either invoke mysqldump tool directly or create a script in any language to export data you want to, which requires more work, but is more flexible.
Alternatively you can search the Internet for some ready tool (like this which I just found, but didn't check if it works). I'm pretty sure you would find some, since it's pretty common thing to do.
We have a rails app that has a MySQL backend, each client has one DB and the schema is identical. We use a custom gem to change the DB based on the URL of the request (This is some legacy code that we are trying to move away from)
We need to capture some changes from those MySQL databases (Changes in inventory, some order information, etc) transform and store in a single MongoDB database (multitenant data store), this data will be used for analytics at first, but our idea is to move everything there.
There was something in place to do this, using AR callbacks and Rabbit, but to be honest it wasn't working correctly and it looked like it was more trouble to fix it than to start over with a fresh approach.
We did some research and found some tools to do ETL but they are overkill for our needs.
Does anyone have some experience with a similar problem?
Recommendations on how to architect and implement this simple ETL
Pentaho provides change-data-capture option which can solve Data-synchronization problems.
If by Overkill you mean Setup, Configuration, then Yes that is the common problem with ETL tools and PENTAHO is the easiest among them.
If you can provide more details, I'll be glad to provide an elaborate answer.
I'm trying to make some reports using meteor and raphael js. I have to report data from an existing MySQL database. I do not wish to write to that database. I need only the "R" from CRUD.
I have thought of various manual ways of: exporting .csv files from the MySQL db via the application itself (Limesurvey) and using mongoimport to populate a MongoDB collection, and then do my CollectionName.find() etc in Meteor.
or perhaps some way of exposing REST full endpoints only to consume data, and use the http package for Meteor.
Is there a good clean solution for using existing SQL data in a Meteor JS application?
How can one use pre-existing SQL data?
(I've no problem with duplication in MongoDB, mind you. however it has to be...)
Thank You
You can do it without any duplication completely from inside Meteor, but you will have to jump through a couple of hoops.
Firstly, use the mysql npm package to query the SQL database. Though Meteor provides Npm to require node packages, I find that using meteor-npm is an easier. Then to do the "R"eading form MySQL, create a Meteor.method on your server which queries the MySQL directly.
Then the second problem is that the mysql package is completely asynchronous. Hence, the execution of the SQL query returns value in a call back and by that point, your Meteor.method call would return leaving the client with an undefined. To fix that issue, we can use Future.
There are a couple of ways of smoothing over this step:
Using `meteor-sync-methods
Spinning out your own version from advice from the issue to allow this natively
Use this easy to implement one-time pattern: "fence has already activated -- too late to add writes"
Hope that helps.
I have written couple of methods to retrieve data from LDap and put it into MySql database. I put those methods in a Listener, so that it executes while deploying the War.
Now this is a one time action. That means, I have to take all the data from Ldap and put those into the MySql DB, and then work on the database tables. I have nothing to do with the LDap data farther.
Is there any better way to do the data migration thing? Since it is a one time work, and once the database is created successfully, there is no need of these methods.
Please Suggest!
Thanks. :)
For migration exercises, look into the Open Source Pentaho Data Integration tool (PDI, or commonly known as Kettle).
There is a slight learning curve, but it's easy to use, and you'll have it forever.
I'm working with another dev and together we're building out a MySQL database. We've each got our own local instances of MySQL 5.1 on our dev machines. We've not yet been able to identify a way for us to be able to make a local schema change (eg: add a field and some values for that field) and then export some kind of script or diff file that the other can import in. I've looked into Toad and Navicat's synchronization features but they seem oriented towards synchronizing between two instances, not an instance and an intermediate file. We thought MySQL Workbench would be great but this but the synchronization feature just seems plain broken. Any other ideas? How do you collaborate with others on the schema?
First of all put your final SQL schema into version control. So you'll always have a version of it with all changes. It can be a plain SQL file. Every developer in the team can use it as starting point to created his copy database. All changes must be applied to it. This will help you to find conflicts faster.
Also I used such file to create a test database to run unit-tests after each submit. So we were always sure that production code is working.
Then you can use any migration tool to move changed between developers. Here is similar question about this:
Mechanisms for tracking DB schema changes
If you're using PHP then look at Doctrine migrations.