I'm working on a project right now that required me to use a CMS that makes multiple changes to a database, I'll need those changes later in order to create a post install configuration file to reuse those changes. I know that there are lots of Windows based programs that will show you MySQL Diffs, but what about Linux? I would like the ability to keep an appending log of my changes so I know what exactly is going on under the hood.
The ideal scenario would be that I can capture a post and current state, compare them, and aggregate the output. Does anyone know a way to do this?
If these are the only changes made to your database then one way to do this is to enable the binary log, and use that as your change log. You can convert it to a SQL script using the mysqlbinlog tool.
Related
I want to know, is there a log that saves all the changes made by users to a database just like in git where all the commits can be viewed by each user so that if any error occur we know who did it.If there is one how to get it? or how to create one that will do it?
Problem I'm facing is that a table's data has been altered by one of the developers but there is no way to find when and who did it and also I am struggling to find all the changes that has been done to the table.
Databases typically do not provide auditing as standard. I typically implement it within my application. However, for a faster result, there are some plugins for mySQL which you could try.
I'm using MODx CMS and would like to use git for co-working. However, as you know, MODx stores some of code places in database. So, I'd like to know, how to version MySQL for co-working except dumping? Thanks
You'll need to store the database in some format, and it can't be SQL as you can't just tell a live database to update the schema using a SQL file.
Basically, you'll have to store an incremental set of commands that updates the database. Git won't really help you in this case, you are going to have to use your own system.
The easiest way to achieve this is by just numbering your database versions and storing only the changes in your favorite programming language.
Step 1: store the current database version somewhere.
Step 2: store all updates to the database in files.
update/mysql_1_to_2.py
update/mysql_2_to_3.py
# etc
Step 3: create the initial database
update/mysql_0_to_1.py
If you use that system, all you need to do is keep track of the current in use version and the version that's most recent in your code. When you need to update, just run all mysql_x_to_y scripts and you're done with the versioning. mysql_0_to_1 can do the initial bootstrap and if you somehow manage to ever reach 1000 revisions, just use mysql_0_to_1000 to take a shortcut.
The system I outlined above should work really well for linear repositories, but might be a bit harder to use for trees (branches, multiple developers, etc). I'm sure you can come up with something though.
It sounds like perhaps you are talking about stored procedures and/or user-defined functions?
In any case, the best method is mysqldump - presumably, you don't need the data, just the stored procs, user-defined functions, and perhaps table structures. There are options for mysqldump to control what gets output.
If you want to add versioning of database to versioning of code, you have to see at Liquibase. To avoid repeating everything again, my older full answer on topic
I'm working with a project which is using mysql as the database. The application is hosted with many clients and we are doing upgrades for the current live systems often.
There are some instances where the client has change the database structure(adding new tables) and causes some unexpected db crashes.
I need to log all the structural changes which were done at that database, so we can find the correct root cause for that. We can't do it 100% correct with diff tool because it will not show the intermediate changes.
I found http://www.liquibase.org/ tool but seems little bit complex.
Is there any well known technique or a tool to track database structural changes only.
well from mysql studio you can generate all object's schema definition and compare them with your standard schema definition and this way you can compare two database schema...
generate scrips of both database (One is client's Database and One is master copy database) and then compare it using file compare tool would be the best practice according to me because this way you can track which collumn was added, which column was deleted, which index was added like wise without any tool download.
Possiable duplication of Compare two MySQL databases ?
Hope this helps.
If you have an application for your clients to manage these schema changes, you can use a mechanism at application level. If you have a Python and Django-based solution, you could probably use South which provides schema change tracking and rollbacks.
We're an opensource project and would like to collaboratively edit our website through github public repo.
Any ideas on the best solution to export the mysql data to github, as mysql can hold some sensitive info in it, and how we can version the changes that happen in it ?
Answer is you don't hold data in the repo.
You may want to hold your ddl, and maybe some configuration data. But that's it.
If you want to version control your data, there are other options. GIT isn't one of them
It seems dbdeploy is what you are looking for
Use a blog engine "backend-ed by git", forget about mysql, commit on github.com, push and pull, dominate !
Here it is a list of the best:
http://jekyllrb.com/
http://nestacms.com/
http://cloudhead.io/toto
https://github.com/colszowka/serious
and just in case, ... a simple, Git-powered wiki with a sweet API and local frontend. :
https://github.com/github/gollum
Assuming that you have a small quantity of data that you wish to treat this way, you can use mysqldump to dump the tables that you wish to keep in sync, check that dump into git, and push it back into your database on checkout.
Write a shell script that does the equivalent of:
mysqldump [options] database table1 table2 ... tableN > important_data.sql
to create or update the file. Check that file into git and when your data changes in a significant way you can do:
mysql [options] database < important_data.sql
Ideally that last would be in a a git post-receive hook, so you'd never forget to apply your changes.
So that's how you could do it. I'm not sure you'd want to do it. It seems pretty brittle, esp. if Team Member 1 makes some laborious changes to the tables of interest while Team Member 2 is doing the same. One of them is going to check-in their changes first, and best case you'll have some nasty merge issues. Worst case is that one of them lose all their changes.
You could mitigate those issues by always making your changes in the important_data.sql file, but the ease or difficulty of that depend on your application. If you do this, you'll want to play around with the mysqldump options so you get a nice readable, and git-mergable file.
You can export each table as a separate SQL file. Only when a table is changed it can be pushed again.
If you were talking about configuration then I'd recommend sql dumps or similar to seed the database as per Ray Baxters answer.
Since you've mentioned Drupal, I'm guessing the data concerns users/ content. As such you really ought to be looking at having a single database that each developer connects to remotely - i.e. one single version. This is because concurrent modifications to mysql tables will be extremely difficult to reconcile (e.g. two new users both with user.id = 10 each making a new post with post.id = 1, post.user_id = 10 etc).
It may make sense, of course, to back this up with an sql dump (potentially held in version control) in case one of your developers accidentally deletes something critical.
If you just want a partial dump, PHPMyAdmin will do that. Run your SELECT statement and when it's displayed there will be an export link at the bottom of the page(the one at the top does the whole table).
You can version mysqldump files which are simply sql scripts as stated in the prior answers. Based on your comments it seems that your primary interest is to allow the developers to have a basis for a local environment.
Here is an excellent ERD for Drupal 6. I don't know what version of Drupal you are using or if there have been changes to these core tables between v6 and v7, but you can check that using a dump, or phpMyAdmin or whatever other tool you have available to you that lets you inspect the database structure. Drupal ERD
Based on the ERD, the data that would be problematic for a Drupal installation is in the users, user_roles, and authmap tables. There is a quick way to omit those, although it's important to keep in mind that content that gets added will have relationships to the users that added it, and Drupal may have problems if there aren't rows in the user table that correspond to what has been added.
So to script the mysqldump, you would simply exclude the problem tables, or at very least the user table.
mysqldump -u drupaldbuser --password=drupaluserpw 0-ignore-table=drupaldb.user drupaldb > drupaldb.sql
You would need to create a mock user table with a bunch of test users with known name/password combinations that you would only need to dump and version once, but ideally you want enough of these to match or exceed the number of real drupal users you'll have that will be adding content. This is just to make the permissions relationships match up.
For some security issues I'm in an envorinment where third party apps can't access my DB. For this reason I should have some service/tool/script (dunno what yet... i'm open to the best option, still reading to see what I'm gonna do...)
which enables me to generate on a regular basis(daily, weekly, monthly) some csv file with all new/modified records for a certain application.
I should be able to automate this process and also export at any time a new file.
So it should keep track for each application which records he still needs.
Each application will need some data in some other format (csv/xls/sql), also some fields will be needed for some application and some aren't... It should be fairly flexible...
What is the best option for me? Creating some custom tables for each application? Based on that extracting modified data?
I think you best thing here, assuming you have access to the server to let you set this up is to make a small command line program that can do the relativley simple task you need. Languages like pearl are good for this sort of thing I do believe.
once you have that 'tool' made you can schedule it through the OS of the server to run ever set amount of time. Either schedule task for a windows server or a cronjob for a linux server.
You can also (with out having to set up the scheduled task if you don't / can't want to) enable this small command line application to be called via 'CGI' this is a special way of letting applications on the server be executed at will by a web user. If you do enable this though, I suggest you add some sort of locking system so that it can only be run every so often and to stop it being run five times at once.
EDIT
You might also want to just look into database replication or adding read only users. This saves a hole lot of arseing around. Try to find a solution that dose not split or duplicate data. You can set up users to only be able to access certain parts of the database system in certain ways, such as SELECT data