I have a large amount of text data I need to import into MySQL. I'm doing this on a MacBook and don't have enough space for it so I want to store it in an external hard drive (I'm not really concerned about speed at this point - this is just for testing).
What's the best way to do it?
Install MySQL on the external hard drive (is this possible on a Mac?)
Install MySQL on the laptop's hard drive and have the tables on the external (how?)
One simple hack is to create an symbolic link replacing your current mysql database file location pointing to the external disk. Google symbolic link.
sample usage would be after you shutdown mysql, change the old mysql db folder name to something else, and create the symbolic link using the ln command like below
ln -s [EXTERNAL DRIVE PATH] [MYSQL DB FOLDER PATH]
Then move all the previous content of the mysql db folder to the new location.
Open /etc/mysql/my.cnf and find the value of the datadir. Alternatively, you can find this out in the mysql monitor with
mysql> select ##datadir;
Stop mysql
sudo systemctl stop mysql
Copy the data from there to your external drive
sudo rsync -av /var/lib/mysql /mnt/myHDD/somedir/mysql
Modify the location of the datadir in my.cnf.
Start mysql again
sudo systemctl start mysql
Verify that everything is still fine and remove the original data dir.
This page contains a more extensive guide but all the additional issues it warns about were not relevant for me on my raspberry PI. I.e. I skipped them and it worked.
For the second option, a tablespace might do the trick:
http://dev.mysql.com/doc/refman/5.1/en/create-tablespace.html
User user658991 answer is halfway there.
After adding the soft link, you will need to add the following line to /etc/apparmor.d/usr.sbin.mysqld beneath the 2 lines to the old mysql folder.
/path/to/mysql/folder/on/the/external/ r
/path/to/mysql/folder/on/the/external/ ** rwk
Without these 2 lines, MySQL fails to start complaining of:
Can't create test file /path/to/mysql/folder/on/the/external/hostname.lower-test
Can't create test file /path/to/mysql/folder/on/the/external/hostname.lower-test
mysqld: Can't change dir to '/path/to/mysql/folder/on/the/external/' (Errcode: 13)
Restart apparmor for the changes to take effect.
sudo invoke-rc.d apparmor restart
With this, MySQL starts normally.
Related
I have setup a private wiki (1.35.1) running on Ubuntu Mate which is a guest OS on VMWare Workstation 16. I'd like to run this wiki at 2 locations (A & B) that are isolated (no VPN connection). I will be the only user accessing since it is my private wiki.
I've got the wiki setup and running at location A and will simply archive the guest and bring it up at location B as an identical copy.
Question: After I spend the day at location A (editing my wiki there), can i just simply copy the entire /var/www/html/Mediawiki folder and the entire /var/lib/mysql folder (MariaDB) onto a thumb drive and dump onto location B?
The intent is for these to be identical wiki's - synchronized by me (sneakernet) with thumb drive.
UPDATE - this is working well so far. Below is how i do it.
Stop the mysql server - sudo service mysql stop
Copy (using rsync) all new or changed files from /var/lib/mysql/ to my external share with:
sudo rsync -cavurt --delete --info=del,name,stats2 "/var/lib/mysql/" "/home/rp/shares/VM_share_ubuntu/wiki_sql_files"
Copy all new or changed files from /var/www/html/mediawiki-1.35.1 to my external share with:
sudo rsync -cavurt --delete --info=del,name,stats2 "/var/www/html/mediawiki-1.35.1/" "/home/rp/shares/VM_share_ubuntu/wiki_mediawiki_files"
Start the mysql server - sudo service mysql start
Now, copy the new/changed files to the 2nd machine:
Stop the mysql server - sudo service mysql stop
Copy in (using rsync) all new or changed mysql files with:
sudo rsync -cavurt --delete --info=del,name,stats2 "/home/rp/shares/VM_share_ubuntu/wiki_sql_files/" "/var/lib/mysql"
Copy in all new or changed mediawiki files with:
sudo rsync -cavurt --delete --info=del,name,stats2 "/home/rp/shares/VM_share_ubuntu/wiki_mediawiki_files/" "/var/www/html/mediawiki-1.35.1"
Start the mysql server - sudo service mysql start
In those rsync commands note that the end of the source folder needs to be / and the target folder does NOT have the ending /. The significance of that is explained in this thread.
UPDATE 2: If you modify the /etc/php/7.4/apache2/php.ini file on one machine you will need to make sure to make the same update on the other machine. e.g. If you change the file upload size from the default 2M or some other change that affects php.ini.
This would mostly work as long as you set $wgServer dynamically. Pages that use absolute URLs and are loaded from cache would link to the wrong URL, but that should be very rare - almost everything uses relative URLs.
Starting on PHP/MySQL Development and as the title states, downloaded MySQL 8.0.15 DMG. Installed. However, when I attempt to run the terminal command:
"sudo /usr/local/mysql/support-files/mysql.server.start"
I get:
"Starting MysQl .......... "
which finally ends in:
ERROR! The server quit without updating PID file (/usr/local/mysql/data/Macbook.hsd.comcast.net.pid)
So far I have tried:
Tried to remove the "Macbook.hsd.comcast.net.pid" file located in the data directory but all i keep getting a "No Such file or directory"
To look for default "my.cnf" file to mod or to add the default values, HOWEVER I read that: "By default the OS X installation does not use a my.cnf, and My SQL just uses the default values. To set up your own my.cnf, you could just create a file straight in /etc.
An "auto.cnf" exists in the data folder, i'm not sure if MySQL uses that or would respect a "my.cnf" created in the /usr/local/sql/data/ folder.
I'm not sure of the proper location to create a "my.cnf" (see, all I want to do is set the proper value for the "pid-file" such as pid-file = /var/run/mysqld/mysqld.pid and then create the directory and give it the proper permissions. SEE UPDATE AT BOTTOM:
I'm cannot access the "data" folder in MySQL folder (which is alias of): "/usr/local/mysql-8.0.15-macos10.14-x86_64/". It keeps telling me I don't have permission. There is a little red dot
I have set checked the permissions in "/user/local/mysql/" and did a
"sudo chown -RL root:mysql /user/local/mysql"
"sudo chown -RL _mysql:mysql /usr/local/mysql/data"
In the /mysql/ directory now everything reads:
Stumped and not sure where to move on from here.
Is the fact that I can't access the data folder a result because I haven't made one/specified one? Can't seem to find an answer for getting MySQL working with HighSierra/Mojave. I can't believe that it's just dead.
I would appreciate any/all help.
UPDATE/EDIT:
Was able to add my user permissions to the "/user/local/mysql/data" folder and was finally able to access the error file ("Macbook.hsd.comcast.net.err") and a "auto.cnf".
The "auto.cnf" file is empty besides a single line: [auto] server-uuid=e7986494-331d-11e9-9691-bec9b3249da6 (not sure if I need to modify the auto.cnf and add PID entries). See error BELOW which seems to indicate otherwise.
The Error file had repeated Errors (occurred several Times):
Location of MySQL configuration file (ie: my.cnf) not specified
How can I fixed this thing?
I been trying to reinstall mysqlserver 5.5 / 5.7 and workbench countless of time. But I still not able to fix this issue.
I solved this problem. Step one: click here
Step two: and then click here to choose your path
Step three: my path is C:\ProgramData\MySQL\MySQL Server 5.7\my.ini
If you already know where your configuration file is located, but just don't know how to specify it in mySQLWorkbench, you can skip steps 1-3 and just follow step 4 in these instructions.
Step 1: Make sure which mysql is running (just in case you have multiple installations on your system) and how:
$ ps aux | grep mysql
may result in something like:
_mysql 120 0,0 3,4 5122736 565108 ?? Ss Fr01pm 61:05.93 /usr/local/mysql/bin/mysqld --user=_mysql --basedir=/usr/local/mysql --datadir=/usr/local/mysql/data --plugin-dir=/usr/local/mysql/lib/plugin --log-error=/usr/local/mysql/data/mysqld.local.err --pid-file=/usr/local/mysql/data/mysqld.local.pid
If, in this output, you see that mysqld has been started with the --defaults-file option, that option specifies the configuration file used by msql. If so, you have found the configuration file and can skip to step 4.
If not, check if this returns the same binary location as above command:
$ which mysqld
If not, then make sure you use the full path for mysqld from the first output in the following step.
Step 2: Ask mySQL which configuration files it is using:
$ mysqld --verbose --help | grep -A 1 "Default options"
should output something like:
Default options are read from the following files in the given order:
/etc/my.cnf /etc/mysql/my.cnf /usr/local/mysql/etc/my.cnf ~/.my.cnf
Step 3: Check the given locations in that order. The first finding should be your mySQL configuration file.
Step 4: In mySQLWorkbench go to Database > Manage Connections, then select the connection, select tab System Profile, and enter the path and file name in Configuration File.
This should be it.
And if still nothing happens, you may have used the installer Package. If you have, then you must select the Installation Type from the drop-down to reflect that before you locate the configuration file.
Here's a screenshot that follows from the above screenshots
Also, there may not be a config file and you can create one here, then get out of the Options File, go back in and you will see all the options.
I changed my setting from Custom to MacOSx (MySQL Package) under: Database → Manage Connections → MySQL Connections → Select DB Connection → System Profile → Installation Type → (default setting) MySQL Package.
Change from custom to default setting:
Reconnect by testing connection:
For maria-db users
back up the original /etc/mysql/my.cnf
copy and rename /etc/mysql/mariadb.conf.d/50-server.cnf to /etc/mysql/my.cnf.
This would fix the problem.
Remember to revert changes after finish.
For those with Mac OS Catalina:
I did not have a my.cnf file. I went to /usr/local/mysql/
and created a directory named etc, which I did not have either.
Then, inside that directory, I created the file (my.cnf).
This is the link that helped me How to fix --secure-file-priv option error
for mysql 5.7 users and MAMP: just create /etc/my.cnf
macOS 13 and MySQL 8 current:
Open MySQL Workbench on your Mac. Connect into your localhost instance of MySQL.
Click the little wrench icon next to INSTANCE on the left pane above Startup / Shutdown.
Select macOS as System Type, macOS as Installation Type, and the rest will default into place. Copy the location it gives you for the my.cnf file.
Click Close
Now click Options File on the left. It will notice that you don't have a file and will show you default values.
Click the Apply... in the bottom right corner.
It will then ask you for a password of your current logged in user of your Mac to create and save the my.cnf file.
Now in the Finder use the keyboard combo of Command-Shift-G and copy in your location we copied early (i.e. /etc/my.cnf)
UPDATE: The sysadmin in my company deleted the /tmp directory and created a
symbolic link to a mounted drive (e.g. ln -s /mnt/somewhere /tmp).
MySQL apparently didn't like this symbolic link.
Once a real /tmp directory on the local filesystem was created the problem went away.
I'm running a process (rake db:migrate) which causes MySQL to try to create a temp file.
However, the following error is appearing:
Mysql::Error: Can't create/write to file '/tmp/#sql_196_0.MYI' (Errcode: 13):
SHOW FIELDS FROM `user_rules`
Errcode 13 apparently is a file permission issue.
The user that is running the process has write permissions to the /tmp directory.
I also searched the web, and someone mentioned that the issue could be the file be prefixed with a pound sign (e.g. '#").
I went to the /tmp directory and tried to create the file and noticed that in certain cases I can't create the file. So I'm wondering if indeed the pound sign is causing the problem. And if it is, how can use MySQL or Rails to fix the problem:
cd /tmp
echo "Hello" > #sql_123
touch #sql_123
echo "Hello" > "#sql_123"
touch "#sql_123"
The 2nd and 3rd commands (without quotes) are failing. The 4th and 5th lines (with quotes) are working. So maybe the failure is because of the pound sign not being in quotes or escaped with a backslash. But again, even if that is the cause, I'm not sure how to tell MySQL or Rails how to deal with it.
Any ideas?
In a shell you need quotes because # introduces a comment, unless escaped by a backslash or quotes. Those #sql_123 files with a prefix of a pound-sign are temporary files owned by mysql, they are normal. You cannot move them however and whenever they are present you cannot move your database using the binary files, you either need to shutdown and move; or do a mysqldump to SQL.
Now for your sysadmin, did he delete /tmp while mysql was running? Mysql opens files in /tmp and then deletes them from the directory leaving the file-handle open. You would see differences in disk space while the server is up or down. MySQL creates all temporary files as hidden files. This ensures that the temporary files are removed if mysqld is terminated. The disadvantage of using hidden files is that you do not see a big temporary file that fills up the file system in which the temporary file directory is located.
So if you would like to keep the symlink (i.e. to a larger disk) then you would better shutdown mysql, create the link, and then start mysql to find it has no problem with symlinks.
I'm trying make a copy of a MySQL database on another server. I stop the server, tar up the mysql directory, copy it to the other server and untar it. I set all the permissions to match to the working server, and copied the my.cnf file so everything is the same except for server specific changes.
However, when I try to startup the server, I get the following InnoDB error:
InnoDB: Operating system error number 13 in a file operation.
This error means mysql does not have the access rights to
the directory.
File name /var/lib/mysql/ibdata1
File operation call: 'open'.
The owner/group for all the files is mysql. I even tried changing permissions to a+rw. I can su to the mysql user and access the ibdata1 file using head.
SOLUTION:
The problem was selinux was enabled and preventing the new files from being accessed.
A silly question, but people forget: you said you checked that all files have the same permissions; still, even though it said so in the message, might you possibly have forgotten to check the permissions on the containing directory?
UPDATE: Two more suggestions:
You might try inserting --console and --log-warnings flags, for lots of debugging output, something like this (on my Mac):
/usr/libexec/mysqld --console --log-warnings --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-external-locking --socket=/var/lib/mysql/mysql.sock
If all else fails, you can probably try strace mysqld ... to see what exactly is it failing. The error will be somewhere at the bottom.
UPDATE2: Interesting indeed... I can't see what your OS is. I normally don't use /sbin/service, it's a bit mysterious for me; on a Mac, it's deprecated in favour of launchctl with config file in /System/Library/LaunchDaemons/mysqld.plist, and on most Linux boxes you have /etc/init.d/mysqld. So you could insert strace there.
Or (untested, but manpage suggests it's possible) you could try stracing the service call:
strace -ff -o straces /sbin/service start mysqld
This should produce files straces.pid, one of which should be mysqld's, and hopefully you'll find your error there.
This isn't a direct answer to your question, but I would recommend trying one of these programs for your backup / restore needs.
Percona Xtrabackup: https://launchpad.net/percona-xtrabackup
Mydumper: http://www.mydumper.org/
Both are great tools, are free and open source, and will help you avoid that problem entirely.
Hope that helps.