I am trying to restore a DB using an SQL script, but things foreign key constraints get in the way
I am taking a MySQL DB and bringing it over to PostgreSQL.
Since the MySQL create table syntax ended up being quite different, I took another PostgreSQL DB with the same schema, but different data and restored the schema only, from that.
In other words, I now have a database with tables, constraints, sequences and all that shnaz but no data inside.
So, it's is time to restore data.
I take a backup of the MySQL DB with phpMyAdmin (data only) as an SQL script (pgAdmin does not seem to accept zip or gzip files for some reason) and run the SQL script.
Now, this is where the problems start to happen, it's only natural, I am going from MySQL to PostgreSQL, so syntax errors are bound to happen.
But, there are other non syntax related problems to, like this one:
ERROR: insert or update on table "_account" violates foreign key constraint "fk_1_account"
DETAIL: Key (accountid)=(2) is not present in table "_entity".
So, yeah, basically, a foreign constraint exists, the query is trying to insert data into the _account table, but the corresponding data has not been inserted into the _entity table yet.
How do I get around that? Is there a way to make pgAdmin3/PostgreSQL disable ALL OF the constraints, insert the data, and then re-enable the constraints?
A syntax related error I encountered, was this one:
INSERT INTO _accounttype_seq (id) VALUES (11);
The PostgreSQL equivalent of that statement (if I am correct) is
ALTER SEQUENCE _accounttype_seq INCREMENT BY 11;
But, it's a bit of a pain to run through the whole script and change all 200+ Sequence insert statements. So, I am being lazy here, but is there an easier way to deal with the sequences as well?
Or, do you guys have any suggestions for a different set of tools to make this easier?
Thanks for your time, have a good day.
Do not try to get around the foreign key constraints. That is the way to make sure the data is bad.
First look at the constraints and make sure you are inserting to the tables in the correct order. If _entity is parent of "_account, then it should be populated first.
Next you need to have the script move any failing records to an exception table. Then you can look at them and see what the data integrity issues is and if you need to throw the records away permanently or try to figure out what the missing parent value should be. If it is critical data such as orders where the customer no longer exists (possible in any system that didn't have correct fks to begin with) and you must keep the record and cannot determine what the parent value should have been, you can create an 'Unknown" record in the customer table and assign all bad orders to that customer id.
And manually changing the alter sequences shouldn't take long even if it is boring. There wil be plently of other things you need to handle manually in a conversion of this type.
I would try to find a data import tool for PostgreSQL - I live in SQL server world where I would use SSIS but you need the equivalent of SSIS for the PostgreSQL world.
Aparently the foreign keys weren't actually enforced in MySQL (maybe because of using MyISAM) or the generated SQL just does it in the wrong order.
If it's "only" the wrong order, I see two possible solutions:
edit the generated script and either move all FK definitions to the end of the script
Edit the definition of each FK constraint and set them all to initially deferred. Then run the script as one single transaction with only on commit at the very end.
Edit (because this is too much to be put as a comment)
Using SET CONSTRAINTS ALL DEFERRED will only work if the constraints have been created with the option DEFERRABLE.
To run everything in one single transaction, you have to make sure you have turned autocommit off. Then simply run the INSERTs and at the very end issue a COMMIT. A ; will only commit if you have autocommit on.
If you want to be independent of the autocommit setting, then start your script with [BEGIN][1] and make sure there is only a single COMMIT at the very end.
BEGIN DEFERRABLE
INSERT INTO table_one ... ;
INSERT INTO table_two ... ;
.....
COMMIT;
Related
i'd like to ask a question regarding Unique columns in MySQL.
Would like to ask experts on which is a better way to approach this problem, advantages or disadvantages if there is any.
Set a varchar column as unique
Do a SQL INSERT IGNORE
If affected rows > 0 proceed with running the code
versus
Leave a varchar column as not-unique
Do a search query to look for identical value
If there is no rows returned in query, Do a SQL INSERT
proceed with running the code
Neither of the 2 approaches is good.
You don't do INSERT IGNORE nor do you search. The searching part is also unreliable, because it fails at concurrency and compromises the integrity. Imagine this scenario: you and I try to insert the same info into the database. We connect at the same time. Code in question determines that there's no such record in the database, for both of us. We both insert the same data. Now your column isn't unique, therefore we'll end up with 2 records that are the same - your integrity now fails.
What you do is set the column to unique, insert and catch the exception in the language of your choice.
MySQL will fail in case of duplicate record, and any proper db driver for MySQL will interpret this as an exception.
Since you haven't mentioned what the language is, it's difficult to move forward with examples.
Defining a column as an unique index has a few advantages, first of all when you define it as an "unique index" MySQL can optimize your index for unique values (same as a primary key) because mysql doesn't have to check if there are more rows with the same value so it can use an optimized algoritme for the lookups.
Also you are assured that there never will be a double entry in your database instead of handeling this in multiple places in your code.
When you don't define it as UNIQUE you first need to check if an records exists in your table, and then insert something wich requires 2 queries (and even a full table lock) instead of 1 wich decreases your performance and is more error prone
http://dev.mysql.com/doc/refman/5.0/en/constraint-primary-key.html
I'm leaving the fact that you would use the INSERT IGNORE wich IGNORES the exception when the entry allready exists in the database (Still you could use it for high performance operations maybe in some sort of special case). A normal INSERT will give you the feedback if an entry allready exists
Putting a constraint like UNIQUE is better when it comes to query performance and data reliability. But there is also a trade-off when it comes to writing. So It's up to you which do you prefer. But in your case, since you also do INSERT IF NOT EXIST query, so I guess, it's better to just use the Constraint.
I have recently installed a new computer with Percona Server 5.6 instead of MySQL 5.6, and using InnoDB/XtraDB mostly, FWIW. The database I'm working on is merely a testing ground, but I have 1 issue: after I add a column to a table (or even remove one), I usually forget to INSERT or otherwise change another table's data, which keeps track of what column names are in which table; each table has ASCII name along with a number, and this number is the only difference between table names for simplicity. So, is there a way to auto-update the "relation" table so that the column name and table's number are added or changed, instead of using a cronjob ?
Now that I think, I could DROP that table and use information_schema instead ...
EDIT 0: Don't let the above realization stop you; it's just good to know if this is possible before going for a possible other way.
Yes, relying on the 'INFORMATION_SCHEMA.COLUMNS' may be best.
Unfortunately mysql does not support DDL TRIGGER events, as this would be what you are looking for.
triggers allow you to perform many SQL and procedural operations before insertion, update or deletion of rows in a specific table. However to the best of my knowledge - and I would be stoked if I were wrong - you cant set TRIGGER events on DDL statements like ALTER and DROP TABLE...
However still take the time to learn about triggers - they save a lot of time by eliminating the need for cronjobs and external updates for things like aggregate values.
https://dev.mysql.com/doc/refman/5.6/en/trigger-syntax.html
Is it possible to use any sort of logic in MySQL without using any procedures? My web hosting does not let me create any procedures so I'm looking for a workaround.
The type of thing I want to do is only add an item to a table if it doesn't already exist. Or add a column to a table if it's not already there. There are some operations that can be done such as CREATE TABLE IF NOT EXISTS and so on, but some operations I require do not have such luxuries :(
I realised late on that my lovely procs won't work and so I tried writing IF/ELSE logic as top-level queries, but for MySQL, IF ELSE blocks only seem to work inside functions/procs and not at the global scope.
Any workarounds greatfully received - I've already asked the hosting to grant me privileges to create procedures but no reply as yet...
I suppose you don't have access to the INFORMATION_SCHEMA either. You can possibly find solutions but it would be better, in my oninion, to:
Change your hosting provider. Seriously. Pay more - if needed - for a MySQL instance that you can configure to your needs. You only have a crippled DBMS if you are not allowed to create procedures and functions.
Posible workarounds for the specific task: You want to add a column if it doesn't exist.
1) Just ALTER TABLE and add the column. If it already exists, you'll get an error. You can catch that error, in your application.
2) (If you have no access to the INFORMATION_SCHEMA) maintain a version of the schema, for your database.
The best solution that I can think of would be to use an additional language with SQL. For example, you can run a query for a specific record, and based on the response that you get, you can conditionally run an INSERT statement.
For inserting a table if it doesn't exist, try using the SHOW TABLES statement and testing whether or not a name exists in the result set.
MySQL supports INSERT IGNORE. and INSERT ... ON DUPLICATE KEY UPDATE.
The following will insert a new row, but only if there is no existing row with id=10. (This assumes that id is defined as a unique or primary key).
INSERT IGNORE INTO my_table (id, col1, col2) values (10, "abc", "def");
The following will insert a new row, but if there is an existing row with id=10 (again, assuming id is unique or primary), the existing row will be updated to hold the new values, instead of inserting a new row.
INSERT INTO my_table (id, col1, col2) values (10, "abc", "def")
ON DUPLICATE KEY UPDATE col1=VALUES(col1), col2=VALUES(col2)
Also, CREATE TABLE supports the IF NOT EXISTS modifier. So you can do something like:
CREATE TABLE IF NOT EXISTS my_table ...
There are many other similar options and modifiers available in MySQL. Check the docs for more.
Originally I created a big script to create or update the database schema, to make it easier to deploy database changes from my local machine to the server.
My script was doing a lot of "if table 'abc' exists and it doesn't have a FK constraint called 'blah'" then create an FK constraint called 'blah' on table 'abc'... and so on.
I now realise it's not actually necessary to check whether a table has a certain column or constraint etc, because I can just maintain a schema-versioning system, and query the DB schema-version when my app starts, or when I navigate to a certain page.
e.g. let's say I want to add a new column to a table. It works like this:
Add a new migration script to the app code, containing the SQL required to add the column to the existing table
Increment the app's schema-version by 1
On app startup, the app queries the DB for the DB's schema-version
If DB schema-version < app schema-version, execute the SQL migration scripts between the two schema-versions, and then update the DB schema-version to be the same as the app
e.g. if the DB's schema-version is 5 and the app version is 8, the app will apply migration scripts 5-6, 6-7 and 7-8 to the DB. These can just be run without having to check anything on the DB side.
The app is therefore solely responsible for updating the DB schema and there's no need for me to ever have to execute schema change scripts on the local or remote DB.
I think it's a better system than the one I was trying to implement for my question.
Once a record is entered in a particular table, I think I don't ever want it to be deleted. The records are there to keep track of things for historical purposes. I'm using MySQL, InnoDB. Is there any way I can protect that table from accidental deletes by the application layer? If so, is this a good approach?
If you can set the permission for your user(s), don't give the delete permission. There're situations where this practice is a "must be" like the tables for statistics purpose. Thus, if your table is used to achieve one of this goal, it's a good approach.
I use a trigger that detects the deletion and does some illegal stuff so the whole operation fails. For example like this:
CREATE TRIGGER protect_delete before delete ON protected_table
FOR EACH ROW UPDATE non_existing_table SET non_existing_column=1;
So when someone will attempt a delete operation - the whole statement will fail. You also might use better names for non_existing_column and non_existing_table.
E.g it is possible to obtain an error message like this:
ERROR 1146 (42S02): Table 'database.delete_restricted_on_tableX'
doesn't exist
EDIT: also it is possible to create even better fail messages, please check here http://www.brokenbuild.com/blog/2006/08/15/mysql-triggers-how-do-you-abort-an-insert-update-or-delete-with-a-trigger/
One other option is switch to the ARCHIVE engine for historical tables.
This will prevent any DELETE or UPDATE actions on the table, and compress the data. One (major) downside to this approach is that you cannot index the columns on the table.
I think you should implement this logic in your application layer and insert a column where you put a no-delete flag.
Another idea would be to exclude the delete access for the db user
You may want to write a trigger that detects the deletion, then reinserts the record, but there may be some issues with that, so you can also just add a foreign key constraint that will prevent the deletion.
For some discussions on this you can look at: http://rpbouman.blogspot.com/2011/10/mysql-hacks-preventing-deletion-of.html.
I'm writing some code to inspect a MySQL database structure, and need information about Foreign Key constraints (on InnoDB tables).
There are two ways I know of to do this:
Parse the results of SHOW CREATE TABLE X
Use INFORMATION_SCEMA.REFERENTIAL_CONSTRAINTS
Unfortunately option two requires MySQL 5.1.16 or later, so I can't use it unless/until I can convince our server guy to update, And while I can probably get away with option 1, it feels messy and without writing a full SQL parser I wouldn't feel sure my code would always work with any table.
Is there another way of getting at this information?
Thanks
From the MySQL 5.0 manual online:
You can also display the foreign key constraints for a table like
this:
SHOW TABLE STATUS FROM db_name LIKE 'tbl_name';
The foreign key constraints are listed in the Comment column of the
output.
Poster indicates that this doesn't provide ON UPDATE and ON DELETE information which is an important part of foreign key behavior.
Another option:
Since you control the code involved, is it possible to set up another MySQL instance in the same environment which is version 5.1+? If so, let's call that instance dummy. Run the SHOW CREATE TABLE on the live database. Then, on dummy run a DROP TABLE IF EXIST followed by the output from the SHOW CREATE TABLE query.
Now you can use INFORMATION_SCHEMA on the dummy database to get the information.