Upload mysql database in chunks - mysql

I am trying to upload a 32mb MYSQL database into a pre-existing database, but the php admin on my shared hosting has a 10mb limit... I have tried zipping it up - but when the server unzips the database, the uncompressed file is too large for the server to handle.
Is it possible to split the database up and upload it by pasting it in parts as an SQL query - I assume I would need each chunk to have something at the start of it which says
"Import this data into the pre-existing tables in the database"
What would this be?
At the moment there is a few hundred lines saying things like "CREATE" and "INSERT INTO"

You might try connecting to the database remotely with mysql workbench, or command line tool mysql. If you can do that, you can run:
source c:/path/to/your/file.sql
and you won't be constrained by phpmyadmin's upload size restrictions. Most shared hosting I've seen allows it. If not, you may just need to grant permissions for the user#host in phpmyadmin (or whatever the interface is).

The dump file created by mysqldump is just a set of SQL statements that will rebuild your tables.
To load it in in chunks I'd recommend either dumping it out in sets of tables and loading them one by one or if required the dump file should be roughly in the same (pseudo) format:
Set things up ready for loading
CREATE TABLE t1;
INSERT INTO TABLE t1...;
INSERT INTO TABLE t1...;
CREATE TABLE t2;
INSERT INTO TABLE t2...;
INSERT INTO TABLE t2...;
Finalise stuff after loading
You can manually split the file up by keeping the commands at the start and finish and just choosing blocks for individual tables by looking for their CREATE TABLE statements.

Related

Accesing data from one mysql database to another in MYSQL Workbench

I have two different databases. I have to access data from one database and insert them into another ( with some data processing included, it is not only to copy data ) Also, the schema is really complex and each table has many rows, so copying data into schema in the second database is not an option. I have to do that using MySQL Workbench, so I have to do it using SQL queries. Is there a way to create a connection from one database to another and access its data?
While MySQL Workbench can be used to transfer data between servers (e.g. as part of a migration process) it is not useful when you have to process the data first. Instead you have 2 other options:
Use a dedicated tool you write yourself to do that (as eddwinpaz mentioned).
Use the capabilities of your server. That is, copy the data to the target server, into a temporary table (using dump and restore). Then use queries to modify the data as you need it. Finally copy it to the target table.

MySQL "source" command overwrites table

I have a MySQL Server which has one database called "Backup".
It only has one table with the name "storage".
In the Backup db the storage table contains about 5 Millions datarows.
Now I wanted to append new rows to the table by using the "source" command in the SQL command line.
So what happend is, that source uploaded all the new files in the table, but it overwrote the existing entries (seems that he first deleted all data)
What I have to say is that the sql file that I want to update comes from another server where this table has the same name and structure as "storage".
What I want is to append the new entries that are in the sql file to the one in my datebase. I do not want to overwrite them.
The structure in the two tables is exactly the same. I use the Backup datebase as the name says for backup uses, so that from time to time I can backup my data.
Has anyone an idea how to solve this?
Look in the .sql file you're reading with the SOURCE command, and remove the DROP TABLE and CREATE TABLE statements that appear there. They are the cause of your table being overwritten; what's actually happening is that the table is being replaced.
You could also look into using SELECT ... INTO OUTFILE and LOAD DATA INFILE as a faster and less potentially destructive way to get data from one server to the other in a file.

Rename file within MySQL Stored Procedure

I'm trying to backup some of my data stored in a big table with
SELECT ... INTO OUTFILE
statement.
The output file is on a network hard drive, so if the network connection breaks just during the dump (it takes one minute more or less) I find a partial file on my network hard drive and I'd like to mark such file as "wrong".
Is there a SQL command that I can give inside my MySQL Stored Procedure that let me rename such file?
Thank you very much
Best
cghersi
You can't rename file in mysql, but you could use two files for dump and rotate it only when operation was successful.
Example:
You make dump to 'a.csv', if operation was successful will use 'b.csv' for next dump, otherwise use 'a.csv' again. And so on...

question about MySQL database migration

If I have a MySQL database with several tables on a live server, now I would like to migrate this database to another server. Of course, the migration I mean here involves some database tables, for example: add some new columns to several tables, add some new tables etc..
Now, the only method I can think of is to use some php/python(two scripts I know) script, connect two databases, dump the data from the old database, and then write into the new database. However, this method is not efficient at all. For example: in old database, table A has 28 columns; in new database, table A has 29 columns, but the extra column will have default value 0 for all the old rows. My script still needs to dump the data row by row and insert each row into the new database.
Using MySQLDump etc.. won't work. Here is the detail. For example: I have FOUR old databases, I can name them as 'DB_a', 'DB_b', 'DB_c', 'DB_d'. Now the old table A has 28 columns, I want to add each row in table A into the new database with a new column ID 'DB_x' (x to indicate which database it comes from). If I can't differentiate the database ID by the row's content, the only way I can identify them is going through some user input parameters.
Is there any tools or a better method than writing a script yourself? Here, I dont need to worry about multithread writing problems etc.., I mean the old database will be down (not open to public usage etc.., only for upgrade ) for a while.
Thanks!!
I don't entirely understand your situation with the columns (wouldn't it be more sensible to add any new columns after migration?), but one of the arguably fastest methods to copy a database across servers is mysqlhotcopy. It can copy myISAM only and has a number of other requirements, but it's awfully fast because it skips the create dump / import dump step completely.
Generally when you migrate a database to new servers, you don't apply a bunch of schema changes at the same time, for the reasons that you're running into right now.
MySQL has a dump tool called mysqldump that can be used to easily take a snapshot/backup of a database. The snapshot can then be copied to a new server and installed.
You should figure out all the changes that have been done to your "new" database, and write out a script of all the SQL commands needed to "upgrade" the old database to the new version that you're using (e.g. ALTER TABLE a ADD COLUMN x, etc). After you're sure it's working, take a dump of the old one, copy it over, install it, and then apply your change script.
Use mysqldump to dump the data, then echo output.txt > msyql. Now the old data is on the new server. Manipulate as necessary.
Sure there are tools that can help you achieving what you're trying to do. Mysqldump is a premier example of such tools. Just take a glance here:
http://dev.mysql.com/doc/refman/5.1/en/mysqldump.html
What you could do is:
1) You make a dump of the current db, using mysqldump (with the --no-data option) to fetch the schema only
2) You alter the schema you have dumped, adding new columns
3) You create your new schema (mysql < dump.sql - just google for mysql backup restore for more help on the syntax)
4) Dump your data using the mysqldump complete-insert option (see link above)
5) Import your data, using mysql < data.sql
This should do the job for you, good luck!
Adding extra rows can be done on a live database:
ALTER TABLE [table-name] ADD [row-name] MEDIUMINT(8) default 0;
MySql will default all existing rows to the default value.
So here is what I would do:
make a copy of you're old database with MySql dump command.
run the resulting SQL file against you're new database, now you have an exact copy.
write a migration.sql file that will modify you're database with modify table commands and for complex conversions some temporary MySql procedures.
test you're script (when fail, go to (2)).
If all OK, then goto (1) and go live with you're new database.
These are all valid approaches, but I believe you want to write a sql statement that writes other insert statements that support the new columns you have.

Remove (merge) SQL Servers' database secondary data file

I have a database backup for which SQL Server Management Studio says that it has three files in it: an .mdf file, an .ndf file and one .ldf file. This secondary data file (the .ndf one) was created for no obvious reason, so I want to remove it altogether (without losing data, of course), preferably during while the database is being restored from the backup.
Is this at all doable?
Ok, found a solution.
First back up the database.
Execute this:
USE database_name;
Then execute this, and replace logical_ndf_file_name with the logical name of your NDF file (which you can easily find out via Database->Properties_Files):
DBCC SHRINKFILE('logical_ndf_file_name', EMPTYFILE);
ALTER DATABASE database_name REMOVE FILE logical_ndf_file_name;
I ran the empty followed by ndf drop during produciton load successfully. I think it is important to run the drop ndf in the same transaction as the empty to ensure the database doesn't try to write to the file you are deleting, but then after an empty the database marks the files unusable, evidenced by attempting another empty shorty after.