Ihave been developing an app for some time. This involves entering and deleteing alot of useless data in the tables. Now that I want to go to production I want to get rid of all the data but also restore all the 'IDs' ( primary keys ) to 0 so that the live system can start fresh with sensible ID's like 1,2,3 etc.
Using MySQL and PHP / Codeigniter
Many Many Thanks for yoru help !
I would normally use TRUNCATE - this both removes the data and resets the AUTO_INCREMENT.
Note that MySQL will perform a row by row deletion if there is a foreign key relationship, which is quite convenient (compared to SQL Server).
If your pk is autoincrement, you can do
ALTER TABLE tbl AUTO_INCREMENT =1
Make sure table is empty before executing the query.
Related
I have a very large table (dozens of millions of rows) and a UNIQUE index needs to be added to a column on that table. I know for a fact that the table does contain duplicated values on that key, which I need to clean up (by deleting rows/resetting the value of the column to something unique that I can automatically generate). A plus is that the rows which are already duplicated do not get modified anymore.
What would be the right approach to perform a change like this, given that I will be probably using the Percona pt-osc tool and there are continuous deletes/inserts on the table? My plan was:
Add code that ensures no dupe IDs get inserted anymore. Probably I need to add a separate table for this temporarily, since I want the database to enforce this for me and not the application - so insert into the "shadow table" with a unique index in a transaction together with my main table, rollback all inserts that try to insert duplicate values
Backfill the table by zapping all invalid column values which are within the primary key range below $current_pkey_value
Then add the index and use pt-osc to changeover the table
Is there anything I am missing?
Since we use pt-online-schema-change we are using triggers for performing the synchronisation from the existing table to a temp table. The tool actually has a special configuration key for this, --no-check-unique-key-change, which will do exactly what we need - agree to perform the ALTER TABLE and set up triggers in such a way that if a conflict occurs, INSERT .. IGNORE will be applied and the first row having used the now-unique value will win in the insert during synchronisation. For us this is a good tradeoff because all the duplicates we have seen resulted from data races, not from actual conflicts in the value generation process.
We have two very large tables in our Mysql(MariaDb) database. Table_1 holds a many to many map. It has a auto incremented primary key and a composite key of two columns.
Table_2 refers to the primary key of Table_1. We wan't to fix this obvious error in design by,
Use a composite primary key on Table_1
Add the two columns to Table_2
Populate the composite key in Table_2 by copying data from Table_1, and create index on it.
Preferably delete the auto incremented key column from both tables.
These tables have ~300M rows, and the tables are ~10GB range in size. We need to make these updates within a ~6 hour service window.
I'm investigating how to do this efficiently and doing trials on a replica db. So far I have not tried to run anything with actual data, because ordinary scripts would be insufficient.
I'm not an experienced DB admin. So I need some light shedding to get this done.
My question is what would be the best approach/tips to do this efficiently?
Things I have attempted so far
I read about the new instant add column feature, but our production DB is on MariaDb version 10.0, which is older.
I have followed suggestions in this answer and ran below script on a latest DB version with instant add column support(Alter table was instant). The table had ~50M rows (1/6th of original). It took about two hours , that also is excluding creating new indexes. Therefore this would not be sufficient.
SET join_buffer_size = 4 * 50 * 1024 * 1024; -- 50M keys of 4 bytes each
SET optimizer_switch='mrr=on,mrr_cost_based=off,mrr_sort_keys=on,optimize_join_buffer_size=on';
SET join_cache_level = 8;
UPDATE TABLE_2
JOIN TABLE_1 ON TABLE_1_Id = TABLE_2_FKT1_Id
SET
TABLE_2_KeyPart_1 = TABLE_1_KeyPart_1,
TABLE_2_KeyPart_2 = TABLE_1_KeyPart_2
Also considering evaluating this tool
https://www.percona.com/doc/percona-toolkit/2.2/pt-online-schema-change.html
Plan A: Use Percona's tool: pt-online-schema-change.
Plan B: Use a competing product: gh-ost.
Plan C: Don't use UPDATE, that is the killer. Instead, rebuild the table(s) in a straightforward way, then use RENAME TABLE to swap the new version into place.
Partitioning is unlikely to help in any way. Daniel's link helps with doing a lengthy UPDATE, but trades off time (it takes longer) versus invasiveness (which is not an issue because you have a maintenance window).
Some more details into Plan C (which I prefer for this case):
CREATE TABLE(s) ... -- with new names, and all the new features except secondary indexes
INSERT INTO new SELECT ... FROM old table(s)
RENAME TABLE real1 TO old1,
new1 TO real1,
real2 TO old2,
new2 TO real2;
test -- you still undo the RENAME if necessary
DROP TABLE old1, old2;
I'm developing an Android application in which the data is stored in a SQLite database.
I have made sync with a MySQL database, in the web, to where I'm sending the data stored in the SQLite in the device.
The problem is that I don't know how to maintain the relations between tables, because the primary keys are going to be updated with AUTO_INCREMENT, and the foreign keys remain the same, breaking the relations between tables.
If this is a full migration, don't use auto increment during migration - create tables with normal columns. Use ALTER TABLE to change the model after import.
For incremental sync, the easiest way I see is additional column in each MySQL table called sqlite_id and filled with original id. Then you can update references using UPDATE (with joins).
Alternatives involve temporary tables for storing data and an auxiliary table used for pairing. Tedious for bigger data model.
The approach I tend to use, if possible, is to avoid auto increment in such situations. I have usaully an auxiliary table with four columns like this: t_import(tablename, operationid, sqlite_id, mysqlid).
Process is the following:
Import the primary keys into t_import. Use operationid to separate parallel imports if needed.
Generate new keys for data tables and store them into t_import table. This can be combined with step one.
Import the actual data and use t_import for setting new primary keys and restore relations.
That should work for most scenarios I know about.
Thanks or the help, you have given me some ideas.
I will try to add a id2 field to the tables that will store the same value as the primary key (_id)
When I send the information from SQLite to MySQL and the primary key is incremented I will have the id2 field with the original value of the primary key so that I can compare it with the foreign key of the other tables and update it.
Let’s see if it works.
Thanks
I work with SQL Server and haven't had such an issue adding a FK to a table like I am having with MySQL. The datatypes match up, the foreign key table contains 300K rows, and is taking forever to execute and eventually times out. Am I doing something wrong? I did this in Workbench and Toad, same thing happens.
The relationship is a one to one, with the User table primary key ID being used a a FK for the District Admins table primary key ID.
I dont know how to find the MySQL version, and can't tell you the amount of memory.
Found out the issue was the table collations were different. My new tables were using latin1, but the existing were using UTF8. That is what I get for relying on Workbench to muchy.
I'm trying to reset the seed for an autoincrement field in SQLITE using AIR/ActionScript/Flex 4.5.
I'm doing what should normally work:
DELETE FROM SQLITE_SEQUENCE WHERE NAME = 'myTable'
I've checked that the database actually contains a SQLITE_SEQUENCE table. I'm able to execute the above statement without errors directly via the Firefox SQLITE plugin (SQLite Manager).
However, when I try to do the same using actionscript, I get an error:
No such table 'SQLITE_SEQUENCE'.
All I could find in searching around was this one guy who posted everywhere he could find - only to be left without an answer:
here
and
here
and
here
Any ideas?
sqlite_sequence table is not created, until you define at least one autoincrement and primary key column in your schema.
You should to insert "Auto increment" to primary key column
at least to one table,
Then SQLite is creating "SQLITE_SEQUENCE" table.
To get all tables have Auto increment:
SELECT * FROM SQLITE_SEQUENCE;