I have a database which has the size over 100 MB. It has the .sql.gz which means it is compressed. When I try to import it using PHPMyAdmin I get time out errors. I even tried partial imports ( Allow the interruption of an import in case the script detects it is close to the PHP timeout limit. (This might be good way to import large files, however it can break transactions.) ) which does not work for me. Given that I'm using CPanel & PHPMyAdmin to get the job done.
How can import this database?
You can use BigDump for this
Adjust the database configuration and charset in this file
Remove the old tables on the target database if your dump doesn't contain "DROP TABLE"
Create the working directory (e.g. dump) on your web server
Upload bigdump.php and your dump files (.sql, .gz) via FTP to the working directory
Run the bigdump.php from your browser via URL like
http://www.yourdomain.com/dump/bigdump.php
BigDump can start the next import session automatically if you enable the JavaScript
Wait for the script to finish, do not close the browser window
IMPORTANT: Remove bigdump.php and your dump files from the web server
If Timeout errors still occure you may need to adjust the $linepersession setting in this file. Read more
Very Simple and Effective Technique to import large mysql databases.
Using Command Line console
you can access command line using putty for server and execute following command
mysql -u 'database_user' -p'database_password' database_name < sql_file_name_with_full_path
Using Cron Job [if console is not available]
add a cron job in cpanel with following command
mysql -u 'database_user' -p'database_password' database_name < sql_file_name_with_full_path
Do not forget to delete cron job after importing database. Also delete sql file from server.
for both techniques commands are same
Example
mysql -u 'test_user' -p'123456' test_db < /home1/test/public_html/db.sql
or if you do not have password
mysql -u 'test_user' test_db < /home1/test/public_html/db.sql
In my opinion and experience with hepsiawebhosting.com server, it's better and faster to set those parameters under WHM settings called "Tweak setting". Under Tweak settings you can modify maximum upload and maximum post, and you can could, for example, change upload to 120 MiB and post to 125 MiB.
Also, in PHP settings under WHM, you can change the default socket and SQL time out. You need to increase those settings.
You can also unzip the database and try to upload it as a simple .sql file.
I hope this will help you.
why this is not published, what is missing here?
Related
I am using phpmyadmin on my windows os. I have a database with one table which has 100M records with the size of 20GB. I want to export this table and have the table.sql file. Whenever I try to do this, the size of the exported file is 0 bytes. When I check the apache error log, the following would show up:
Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 1066139648 bytes)
Any idea how to solve this problem?!
Thanks :)
I would suggest to try using the command line and the mysqldump.exe utility, as suggested here
If you have shared hosting and you are using Cpanel then they are providing you the option to backup your database in the following section.
Files => Backup => Download a MySQL Database Backup.
If you are on a shared hosting or you don't have access to shell, then use mysqldumper script; copy it to your server and start it in your browser under "yourDomain.com/path_to_mysqlumper/"
MySQLDumper is a PHP and Perl based tool for backing up MySQL
databases. You can easily dump your data into a backup file and - if
needed - restore it. It is especially suited for shared hosting
webspaces, where you don't have shell access.
If you have shell access to your host servers (if provided by your host since not all shared server hosters give this access) then you may use SSH access like in this tutorial using Putty that you install and configure then import or export your databases like in this third tutorial.
I try mysqldump for many hours but didnt work, until I started a superuser console.
First, start a superuser console
sudo su
Then, try the complete command
/opt/lampp/bin/mysqldump -u root -p [DATABASE NAME] > [PATH_FOR_BACKUPFILE]/[FILE_NAME].sql
I my case, it was something like /opt/lampp/bin/mysqldump -u root -p database > /home/user/backup.sql
MySQLDumper worked like a charm for me at my hosted website. I had to copy one database and "paste" it into a new database. In MySQLDumper, it isn't apparent right away how to do this, but the key is to create a new configuration file in MySQLDumper and that will allow you to copy/restore to different databases.
On the home screen in MySQLDumper, click Configuration, then Configuration Files. There is a text box at the top allowing you to create a new Configuration file. In there, put in the information for the second database you need (you created a connection to the first database when you install MySQLDumper). Save it. Then you can click Restore where you can select the dump of the first database and restore it in the second one.
This was a lifesaver. Thanks!
increase
max_post_size
variable in php.ini file. Then you will be able to download it.
I had a different issue when I was downloading from phpmyadmin in the middle like 180MB download stops with message - network error
So I used ssh connection which you can find in your cpanel sometimes they provide browser based terminal or sometime you have to access it using putty .
In terminal I go inside my public_html folder where all my files are stored . Followed by this command:
mysqldump -u [username] -p [database-you-want-to-dump] > [path-to-place-data-dump.sql]
This did the job in few minutes and saved a sql file in my public_html folder. Then I opened the folder in File manager and downloaded it from there.
You can also use FTP or you can download it directory by accessing by url.
Make sure you delete it after your download finishes.
We have a Magento site.
We have a large database of 1000MB for that site.
When we try to import database through phpmyadmin, it's not uploading the full database. Lot of tables are missing after uploading.
Qe are getting this error :
Script timeout passed, if you want to finish import, please resubmit
same file and import will resume
We have a GoDaddy cpanel and phpmyadmin.
Is there any way to upload a large database completely?
I am planning to change php.ini settings as mentioned in this question:
How to solve time out in phpmyadmin?
But I could not able to find php.ini file.
Have I need to edit the default Magento phi.ini.sample?
use sqlsplitter, It will created small chunk of you large sql file and then upload all these small files
Try like this in your Godaddy SSH,
Log on to your server.
At the command line, type su -. This gives you root access.
At the command line, type mysql -uroot -p
Enter your MySQL root password.
Select yourdatabase name for insert data by writing this command, USE YOURDATABASENAME
Then load source sql file as SOURCE D:\profile20\database.sql
Press enter for insert data.
This video link may help you.
I'm trying to export a database with phpmyadmin but the result is a file that can't be read. In phpmyadmin I leave the options and only add compression. After downloading I'm unable to open the resulted .sql file (after extracting) in geany and get a complaint about the file being in an encoding that's not supported. Every time I do the export the file seems to be of a random size as if the process just stops somewhere.
It's a small database with joomla stuff that needs to be moved to another site and I would like to use mysldump but don't have ssh access. Other than the phpmyadmin in the cpanel I can't think of another way to access that data but phpmyadmin doesn't seem to be up for doing the job.
Is there any setting I should have to have a look at or some other way to get that data exported?
Try disabling compression or using another compression method. There have been a few bugs in phpMyAdmin. Maybe you are stumbling about one of them. Also try opening the sql file in an ordinary text editor and check the content maybe you can get some more information about the problem you are experiencing.
If you have access to the commandline you can also try the following command: mysqldump -u <username> -p <database_name> > dumpfilename.sql.
I am using phpmyadmin on my windows os. I have a database with one table which has 100M records with the size of 20GB. I want to export this table and have the table.sql file. Whenever I try to do this, the size of the exported file is 0 bytes. When I check the apache error log, the following would show up:
Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 1066139648 bytes)
Any idea how to solve this problem?!
Thanks :)
I would suggest to try using the command line and the mysqldump.exe utility, as suggested here
If you have shared hosting and you are using Cpanel then they are providing you the option to backup your database in the following section.
Files => Backup => Download a MySQL Database Backup.
If you are on a shared hosting or you don't have access to shell, then use mysqldumper script; copy it to your server and start it in your browser under "yourDomain.com/path_to_mysqlumper/"
MySQLDumper is a PHP and Perl based tool for backing up MySQL
databases. You can easily dump your data into a backup file and - if
needed - restore it. It is especially suited for shared hosting
webspaces, where you don't have shell access.
If you have shell access to your host servers (if provided by your host since not all shared server hosters give this access) then you may use SSH access like in this tutorial using Putty that you install and configure then import or export your databases like in this third tutorial.
I try mysqldump for many hours but didnt work, until I started a superuser console.
First, start a superuser console
sudo su
Then, try the complete command
/opt/lampp/bin/mysqldump -u root -p [DATABASE NAME] > [PATH_FOR_BACKUPFILE]/[FILE_NAME].sql
I my case, it was something like /opt/lampp/bin/mysqldump -u root -p database > /home/user/backup.sql
MySQLDumper worked like a charm for me at my hosted website. I had to copy one database and "paste" it into a new database. In MySQLDumper, it isn't apparent right away how to do this, but the key is to create a new configuration file in MySQLDumper and that will allow you to copy/restore to different databases.
On the home screen in MySQLDumper, click Configuration, then Configuration Files. There is a text box at the top allowing you to create a new Configuration file. In there, put in the information for the second database you need (you created a connection to the first database when you install MySQLDumper). Save it. Then you can click Restore where you can select the dump of the first database and restore it in the second one.
This was a lifesaver. Thanks!
increase
max_post_size
variable in php.ini file. Then you will be able to download it.
I had a different issue when I was downloading from phpmyadmin in the middle like 180MB download stops with message - network error
So I used ssh connection which you can find in your cpanel sometimes they provide browser based terminal or sometime you have to access it using putty .
In terminal I go inside my public_html folder where all my files are stored . Followed by this command:
mysqldump -u [username] -p [database-you-want-to-dump] > [path-to-place-data-dump.sql]
This did the job in few minutes and saved a sql file in my public_html folder. Then I opened the folder in File manager and downloaded it from there.
You can also use FTP or you can download it directory by accessing by url.
Make sure you delete it after your download finishes.
I have a 400 MB large SQL backup file. I'm trying to import that file into MySQL database using WAMP → import, but the import was unsuccessful due to many reasons such as upload file size is too large and so on. So, I've gone ahead and made a few changes to the configuration of PHP and MySQL settings under WAMP. Changes made are
Under WAMP->MySQL->my.ini
max_allowed_packet = 800M
read_buffer_size = 2014K
Under WAMP->PHP->PHP.ini
max_input_time = 20000
memory_limit = 128M
Is this the right way to import such a large SQL file using WAMP → import?
If so, did I make the right changes to the configuration files? What are the maximum values allowed for the above variable?
You can import large files this command line way:
mysql -h yourhostname -u username -p databasename < yoursqlfile.sql
Since you state (in a clarification comment to another person's answer) that you are using MySQL Workbench, you could try using the "sql script" option there. This will avoid the need to use the commandline (although I agree with the other answer that it's good to climb up on that horse and learn to ride).
In MySQL Workbench, go to File menu, then select "open script". This is probably going to take a long time and might even crash the app since it's not made to open things that are as big as what you describe.
The better way is to use the commandline. Make sure you have MySQL client installed on your machine. This means the actual MySQL (not Workbench GUI or PhpMyAdmin or anything like that). Here is a link describing the command-line tool. Once you have that downloaded and installed, open a terminal window on your machine, and you have two choices for slurping data from your file system (such as in a backup file) up into your target database. One is to use 'source' command and the other is to use the < redirection operator.
Option 1:
from the directory where your backup file is:
$mysql -u username -p -h hostname databasename < backupfile.sql
Option 2:
from the directory where your backup file is:
$mysql -u username -p -h hostname
[enter your password]
> use databasename;
> source backupfile.sql
Obviously, both of these options require that you have a backup file that is SQL.
Use this from mysql command window:
mysql> use db_name;
mysql> source backup-file.sql;
You can make the import command from any SQL query browser using the following command:
source C:\path\to\file\dump.sql
Run the above code in the query window ... and wait a while :)
This is more or less equal to the previously stated command line version. The main difference is that it is executed from within MySQL instead. For those using non-standard encodings, this is also safer than using the command line version because of less risk of encoding failures.
We have experienced the same issue when moving the sql server in-house.
A good solution that we ended up using is splitting the sql file into chunks. There are several ways to do that. Use
http://www.ozerov.de/bigdump/ seems good (but never used it)
http://www.rusiczki.net/2007/01/24/sql-dump-file-splitter/ used it and it was very useful to get structure out of the mess and you can take it from there.
Hope this helps :)
If you are using the source command on Windows remember to use f:/myfolder/mysubfolder/file.sql and not f:\myfolder\mysubfolder\file.sql
I believe the easiest way is to upload the file using MYSQL command line.
using the command from the terminal to access MySQL command line and run source
mysql --host=hostname -uuser -ppassword
source filename.sql
or directly from the terminal
mysql --host=hostname -uuser -ppassword < filename.sql
at the prompt.
Three things you have to do, if you are doing it locally:
In php.ini or php.cfg of your PHP installation
post_max_size=500M
upload_max_filesize=500M
memory_limit=900M
Or set other values. Restart Apache.
Or
Use the PHP big dump tool. It’s best ever I have seen. It’s free and open source.
Had a similar problem, but in Windows. I was trying to figure out how to open a large MySQL SQL file in Windows, and these are the steps I had to take:
Go to the download website (http://dev.mysql.com/downloads/).
Download the MySQL Community Server and install it (select the developer or full install, so it will install client and server tools).
Open the MySQL command-line client from the Start menu.
Enter your password used in install.
In the prompt, mysql>, enter:
CREATE DATABASE database_name;
USE database_name;
SOURCE myfile.sql
That should import your large file.
The question is a few months old but for other people looking --
A simpler way to import a large file is to make a sub directory 'upload' in your folder c:/wamp/apps/phpmyadmin3.5.2
and edit this line in the config.inc.php file in the same directory to include the folder name
$cfg['UploadDir'] = 'upload';
Then place the incoming .sql file in the folder /upload.
Working from inside the phpmyadmin console, go to the new database and import. You will now see an additional option to upload files from that folder. Chose the correct file and be a little patient. It works.
If you still get a time out error try adding $cfg['ExecTimeLimit'] = 0; to the same config.inc.php file.
I have had difficulty importing an .sql file where the user name was root and the password differed from my the root password on my new server. I simply took off the password before I exported the .sql file and the import worked smoothly.
On item 1.16 of phpMyAdmin they say:
1.16 I cannot upload big dump files (memory, HTTP or timeout problems).
Starting with version 2.7.0, the import engine has been re–written and these problems should not occur. If possible, upgrade your phpMyAdmin to the latest version to take advantage of the new import features.
The first things to check (or ask your host provider to check) are the values of upload_max_filesize, memory_limit and post_max_size in the php.ini configuration file. All of these three settings limit the maximum size of data that can be submitted and handled by PHP. One user also said that post_max_size and memory_limit need to be larger than upload_max_filesize. There exist several workarounds if your upload is too big or your hosting provider is unwilling to change the settings:
Look at the $cfg['UploadDir'] feature. This allows one to upload a file to the server via scp, ftp, or your favorite file transfer method. PhpMyAdmin is then able to import the files from the temporary directory. More information is available in the Configuration of this document.
Using a utility (such as BigDump) to split the files before uploading. We cannot support this or any third party applications, but are aware of users having success with it.
If you have shell (command line) access, use MySQL to import the files directly. You can do this by issuing the “source” command from within MySQL:
source filename.sql;
The simplest solution is MySQL Workbench. Just copy the .sql file text to the query window of MySQL Workbench and just execute it. All renaming things will done by it.
if you want to see the import progress. Install pv with brew and run this:
brew install pv
pv -p dump.sql | mysql -h localhost -u root -p db_name
my.ini
max_allowed_packet = 800M
read_buffer_size = 2014K
PHP.ini
max_input_time = 20000
memory_limit = 128M
post_max_size=128M