I am having I/O related performance problems that would be solved if a few relatively small tables were running on a SSD. I can't move the entire DB to SSD because it is much too large.
I thought this was possible (map specific tables to different drives) but a tech at my managed hosting company says that the entire DB needs to be in a single directory. Is this correct? If he's wrong, can someone point me somewhere with basic instructions on how this is done? Or even provide the instructions here?
When you create a MySQL table you can specify the data directory and index directory.
Have a look at http://dev.mysql.com/doc/refman/5.1/en/create-table.html
So, to answer your question, you could create a new table in the different directory and migrate your data there.
Related
I have a MYSQL database on my SDA. It's mostly all one schema with "popular" tables in it. I want to store the less "popular" tables of the schema (which take up another 1TB or so) on my SDB partition.
What is the right way to do this? Do I need another MYSQL server running on that drive? Or can I simply set like DATA_DIRECTORY= or something? This is Ubuntu and MYSQL 5.7.38. Thank you for any help, it's much appreciated.
As of MySQL 8.0.21, the ability to specify the data directory per table has finally improved.
CREATE TABLE t1 (c1 INT PRIMARY KEY) DATA DIRECTORY = '/external/directory';
Read https://dev.mysql.com/doc/refman/8.0/en/innodb-create-table-external.html#innodb-create-table-external-data-directory for details.
In earlier versions of MySQL, you could use symbolic links. That is, the link still has to reside under the default data directory, but the link can point to a file on another physical device.
It was unreliable to use symbolic links for individual tables in this way, because OPTIMIZE TABLE or many forms of ALTER TABLE would recreate the file without the symbolic link, effectively moving it back to the primary storage device. To solve this, it was recommended to use a symbolic link for the schema subdirectory instead of individual tables.
To be honest, I've never found a case where I needed to use either of these techniques. Just keep it simple: one data directory on one filesystem, and don't put the data directory on the same device as the root filesystem. Make sure the data storage volume is large enough for all your data. Use software RAID if you need to use multiple devices to make one larger filesystem.
The project is an open source IDE like system to edit code, alter images,, run code, etc. for students in school.
It contains an upload of Archive/Folders which characteristics are:
Upload Archive, contains up to 30 files (txt,js,xul,xml,php, gif/png/jpg), av.size in total 500kb
Users can edit and fork
all the files are editable through Codemirror (like used on jsfiddle)
all the Gif/Png/Jpg are replaceable (10 - 40 per archive)
we expect daily at least 1'000 new uploads/forks with an average of 20 files, total-size min. 500 mb
Our enironment:
PHP
most likely MySQL database
Linux
To consider:
We don't require Searching through the folders/files on global scope
the User saved Data is as it is, never any changes from our side necessary
State-of-Development:
Ready besides the storage question and all their dependencies
Would you advise on SQL or a simple Filesystem?
Before starting the project we were 100% sure using MySQL, but with the added feature of Image modification and a growing database (atm 80/k files,2GB) we struggle. Also reading here let us hesitate too.
Advantages of MYSQL are surely the easy maintainment and simpler future restructure of the system.
Though it will get a huge database fast to search within.
By using a global php entryfile to read the Filesystem based on URL parameters, the searching can be ommitted and straight go displaying the fetched directory with its content.
We are not experienced in managing large data and rely on experience of people who faced such a situtation already.
Rather than just vote for Database or Filesystem please consider your own tips to make this environment more efficient to run (eg indexing, structure, tables, etc..) or elaborate your decision.
very thankful for any tips of you
btw, the project will be hosted at git
I believe that the best option to go is the file system taking into consideration your requirements. The pros of this choice are
Better performance of your database since the table will only hold a link to your file system where the file is stored.
It is easier to backup your database.
You have the possibility of storing the files in different locations by defining rules. This gives you a flexibility in managing your storage.
When we faced a similar problem for storing attachments in a service desk application, we have chosen to go with the filesystem. Till now everything is working as expected.
I hope that my answer is helpful to you.
I just spend a few hours putting together a table full of data, and just decided to drop a WP installation in the same database. I expected WordPress to leave my table as it was and simply add the WP_* tables, but apparently the install wiped my database before installing the wordpress tables.
I have full access to the server root. Is there any way for me to recover this table?
I'm sorry, but that can't be done. Once a table is truncated or dropped everything is gone.
The only way to get it back would be recovering the database files from the disk, but that is not very reliable. To do that, you first have to find out where the MySQL's data files are (see this), then try using a file recovery tool to get them back. Good luck.
Another one who learns to back up the hard way.
If tables were MyISAM, then I agree with NullUserException - you can't recover the tables. Well, the chance is very slim.
If InnoDB - check out my presentation on slideshare - there is a tool to extract records from raw bytes stream - percona data recovery toolkit. There are examples in the presentation, your scenario is pretty much typical.
UPDATE: Data recovery toolkit moved to GitHub
Few months ago I have asked a question regarding how to change database location at runtime - I didn't get any solution for this problem yet.
I am needing to create more than 32,000 databases in MySQL. The default data location of MySQL data folder, after creating 32,000 database on that location, I want to change the data directory to other location. I am planning to do this through Java code.
But before that can anyone tell me if this is possible?
I am really needing to implement this as an requirement. Please help me out with this.
I am sorry If I am unclear anywhere in this description, but let me know if you all need any more information.
That sounds difficult. I don't understand why 32000 - is mysql refusing to create more databases than that? What error do you get? Seems arbitrary - maybe this is a config variable that can be changed?
A few possibilities. You could run more than one mysql server, each with a different data directory. I don't think it would help, but you might look into the NDB storage engine, it can handle tablespaces which just might let you store data in multiple locations.
You can create several partitions and join them using LVM (this is in Linux) and mount the partition to the DATADIR path. Also you can use soft links the the databases moved to other folders/partitions.
Unfortunately MySQL supports only one DATADIR but it seemed to me that InnoDB tables can be places to separate path. Could you check this?
The application that I am working on generates files dynamically with use. This makes backup and syncronization between staging,development and production a real big challenge. One way that we might get smooth solution (if feasable) is to have a script that at the moment of backing up the database can backup the dynamically generated files inside the database and in restore time can bring those file out of the database and in the filesystem again.
I am wondering if there are any available (pay or free) application that could be use as scripts to make this happen.
Basically if I have
/usr/share/appname/server/dynamicdir
/usr/share/appname/server/otherdir/etc/resource.file
Then taking the examples above and with the script put them on the mysql database.
Please let me know if you need more information.
Do you mean that the application is storing a files as blobs in the MySQL database, and/or creating lots of temporary tables? Or that you just want temporary files - themselves unrelated to a database - to be stored in MySQL as a backup?
I'm not sure that trying to use MySQL as an net-new intermediary for backups of files is a good idea. If the app already uses it, thats one thing, if not, MySQL isn't the right tool here.
Anyway. If you are interested in capturing a filesystem at point-in-time, the answer is to utilize LVM snapshots. You would likely have to rebuild your server to get your filesystems onto LVM, and have enough free storage there for as many snapshots as you think you'd need.
I would recommend having a new mount point just for this apps temporary files. If your MySQL tables are using InnoDB, a simple script to run mysqldump --single-transaction in the background, and then the lvm snapshot process, you could get these synced up to less then a second.
the should be trivial to accomplish using PHP, perl, python, etc. are you looking for someone to write this for you?