Recently, I write a python code to insert HTML text into table. After my code writing about 200,000 html page, I can use select to retrieval all these data. However, I find that the MySQL server does not write any data into files. I check the memory usage and find that mysqld.exe program consumes more than 1.5GB memory. I search the whole disk about the table name but I only found a 9KB file related to my table name. By the way, I also checked the mysql.ini file. The path configuration is correct. Then, I use mysqldump to backup that table. This command gives me more than 7GB sql file. I check it again and find there is 20GB file ibdata in my datadir folder. What is that file mean? Why there is no file related to my table ? Does MySQL just store the data in memory?
Run SHOW TABLE STATUS and check the storage engine value. It might be using MEMORY, though I would be surprised if it is and you didn't know that (because you would have to set it explicitly).
Maybe you're on InnoDB but haven't enabled innodb_file_per_table, so everything's being written to one file.
Related
I have a MYSQL database on my SDA. It's mostly all one schema with "popular" tables in it. I want to store the less "popular" tables of the schema (which take up another 1TB or so) on my SDB partition.
What is the right way to do this? Do I need another MYSQL server running on that drive? Or can I simply set like DATA_DIRECTORY= or something? This is Ubuntu and MYSQL 5.7.38. Thank you for any help, it's much appreciated.
As of MySQL 8.0.21, the ability to specify the data directory per table has finally improved.
CREATE TABLE t1 (c1 INT PRIMARY KEY) DATA DIRECTORY = '/external/directory';
Read https://dev.mysql.com/doc/refman/8.0/en/innodb-create-table-external.html#innodb-create-table-external-data-directory for details.
In earlier versions of MySQL, you could use symbolic links. That is, the link still has to reside under the default data directory, but the link can point to a file on another physical device.
It was unreliable to use symbolic links for individual tables in this way, because OPTIMIZE TABLE or many forms of ALTER TABLE would recreate the file without the symbolic link, effectively moving it back to the primary storage device. To solve this, it was recommended to use a symbolic link for the schema subdirectory instead of individual tables.
To be honest, I've never found a case where I needed to use either of these techniques. Just keep it simple: one data directory on one filesystem, and don't put the data directory on the same device as the root filesystem. Make sure the data storage volume is large enough for all your data. Use software RAID if you need to use multiple devices to make one larger filesystem.
How should you go about restoring (and backing) up a MySQL database "safely"? By "safely" I mean: the restore should create/overwrite a desired database, but not risk altering anything outside that database.
I have already read https://dev.mysql.com/doc/refman/5.7/en/backup-types.html.
I have external users. They & I may want to exchange backups for restore. We do not have a commercial MySQL Enterprise Backup, and are not looking for a third-party commercial offering.
In Microsoft SQL Server there are BACKUP and RESTORE commands. BACKUP creates a file containing just the database you want; both its rows and all its schema/structure are included. RESTORE accepts such a file, and creates or overwrites its structure. The user can restore to a same-named database, or specify a different database name. This kind of behaviour is just what I am looking for.
In MySQL I have come across 3 possibilities:
Most people seem to use mysqldump to create a "dump file", and mysql to read that back in. The dump file contains a list of arbitrary MySQL statements, which are simply executed by mysql. This is quite unacceptable: the file could contain any SQL statements. (Limiting access rights of restoring user to try to ensure it cannot do anything "naughty" is not acceptable.) There is also the issue that the user may have created the dump file with the "Include CREATE Schema" option (MySQL Workbench), which hard-codes the original database name for recreation. This "dump" approach is totally unsuitable to me, and I find it surprising that anyone would use it in a production environment.
I have come across MySQL's SELECT ... INTO OUTFILE and LOAD DATA INFILE statements. At least they do not contain SQL code to execute. However, they look like a lot of work, deal with a table at time not the whole database, and don't deal with the structure of the tables, you have to know that yourself for restoring. There is a mysqlimport helper command-line utility, but I don't see anything for the export side, and I don't see it for restoring a complete database.
The last is to use what MySQL refers to as "Physical (Raw)" rather than "Logical" Backups. This works on the database directories and files themselves. It is the equivalent of SQL Server's detach/attach method for backing up/restoring. But, as per https://dev.mysql.com/doc/refman/5.7/en/backup-types.html, it has all sorts of caveats, e.g. "Backups are portable only to other machines that have identical or similar hardware characteristics." (I have no idea, e.g. some users are Windows versus Windows, I have no idea about their architecture) and "Backups can be performed while the MySQL server is not running. If the server is running, it is necessary to perform appropriate locking so that the server does not change database contents during the backup." (let alone restores).
So can anything satisfy (what I regard as) my modest requirements, as outlined above, for MySQL backup/restore? Am I really the only person who finds the above 3 as the only, yet unacceptable, possible solutions?
1 - mysqldump - I use this quite a bit, usually in environments where I am handling all the details myself. I do have one configuration where I use that to send copies of a development database - to be dumped/restored in its entirety - to other developers. It is probably the fastest solution, has some reasonable configuration options (e.g., to include/exclude specific tables) and generates very functional SQL code (e.g., each INSERT batch is small enough to avoid locking/speed issues). For a "replace entire database" or "replace key tables in a specific database" solution, it works very well. I am not too concerned about the "arbitrary SQL commands" problem - if that is an issue then you likely have other issues with users trying to "do their own thing".
2 - SELECT ... INTO OUTFILE and LOAD DATA INFILE - The problem with these is that if you have any really big tables then the LOAD DATA INFILE statement can cause problems because it is trying to load everything all at once. You also have to add code to create (if needed) or empty the tables before LOAD DATA.
3 - Physical (raw) file transfer. This can work but under limited circumstances. I had one situation with a multi-gigabyte database and decided to compress the raw files, move them to the new machine, uncompress and just tell MySQL "everything is already there". It mostly worked well. But I would not recommend it for any unattended/end-user process due to the MANY possible problems.
What do I recommend?
1 - mysqldump - live with its limitations and risks, set up a script to call mysqldump and compress the file (I am pretty sure there are options in mysqldump to do the compression automatically), include the date in the file name so that there is less confusion as the files are sent around, and make a simple script for users to load the file.
2 - Write your own program. I have done this a few times. This is more work initially but allows you to control every aspect of the process and transfer a file that only contains data without any actual SQL code. You can control the specific database, tables, etc. One catch is that if you make any changes to the table structure, indexes, etc. you will need to make sure that information is somehow transmitted to the receiving problem so that it can change the structures as needed - that is not a problem with mysqldump as it normally replaces the tables, creating the new structures, indexes, etc. This can be written in any language that can connect to MySQL - it does not have to be the same language as your application.
If you're not going to use third party tools (like innobackupex for example) then you're limited to use ... mysqldump, which is in the mysql package.
I can't understand why it is not acceptable for you, why you don't like sql commands in those dumps. Best practice,when restoring a single db into the server, which already contains other databases, is to have a separated user, with rights only to write into the restored db. Then even if the user performing restore, would change the sql commands and tried to write to another db, they will not be able to.
When doing raw backup (physical copy of database files) you need to have all the instances down, mysql server not running. Similar hardware means you need to have the same directories as the source server (unless you would change my.cnf before starting the server, and putting all the files to right directories).
When coming into mysql, try to not compare it to sql server - it's totally different approach and philosophy.
But if you would convinced yourself anyhow to use third party tool - I recommend innobackex from Percona, which is free btw.
The export tool that complements mysqlimport is mysqldump --tab. This outputs CSV files like SELECT...INTO OUTFILE. It also outputs the table structure in much smaller .sql files. So there are two files for each table.
Once you recreate your tables from the .sql files, you can use mysqlimport to import all the data files. You can even use the mysqlimport --use-threads option to make it load multiple data files in parallel.
This way you have more control over which schema to load the data into, and it should run a lot faster than loading a large SQL dump.
I am working on a solaris system running an optical routing EMS application that has generated an huge DATA_MAINTENANCE_LOG.ibd file.
It's taking up alot of space on the filesystem, I do not know mysql commands.
How do I effectively tail the content of this file, then safely clear it out?
How do I effectively tail the content of this file, then safely clear it out?
You don't, unless you like corrupting your database.
It's a database file. Delete the data from the database table, and if you're lucky MySQL will release the file space. If not, and your MySQL instance is configured to store one table per file, drop the table and the file will be deleted.
Any consequences from deleting data from that table, or dropping that table, are your responsibility to determine.
I am trying to save or retrieve data from my co-workers computer. Her MySQL instance does not work anymore (cannot figure out the reasons). I would like to know if there is a way to save all the data from the databases present in her computer.
Any suggestions on how to save the data, and maybe re-install MySQL again?
You can copy the tables to another computer with a working MySQL instance, as long as the floating point format is the same on both computers (see this page for reference, covered in the first paragraph).
As long as you copy the .frm, .myd and .myi files somewhere safe, you can also try reinstalling MySQL and see if that works for you.
First of all, errors may be recoverable. If you do not have ability to recover from errors, you can try to make a plain copy of MySQL data. BUT! If tables used InnoDB engine, that will be not so easy and chance of recovery will be not so big.
When I delete solr's index files on disk, (found in /solr/data/index and solr/data/spellchecker), solr throws an exception whenever I try to make a request to it:
java.lang.RuntimeException: java.io.FileNotFoundException: no segments* file found in org.apache.lucene.store.NIOFSDirectory#/…/solr/data/index:
The only way I've found to recover from this is to “seed” the data directory with the index files from elsewhere. It doesn't really matter where it seems. Once I do this, I can run a query to reload the schema and regenerate the index. Is this how this is supposed to work? It seems like there should be a way to tell solr to regenerate those files from scratch. Maybe I'm just mistaken in my assumption that these files are not part of the application itself (kind of implied by the name “data”)?
Solr will throw that exception at startup if the index directory exists but is empty. However if you delete the directory, Solr will create it and the empty segments files at startup.
If you are using sunspot solr on rails, sunspot can reindex all the data from the database into solr. However, solr standalone would not know where to pull the data to reindex. You would need a backup of the data.
The problem may lie with the segments file if you delete the index . the files are physically deleted but are present in the ram or cache of the solr . Avoid deleting files directly from solr index files physically . use delete query to delete the index , doing this would alter the segments of the index and you will not have to restart the solr
regards
Rajat
Exception FileNotFoundException signals that an attempt to open the file denoted by a specified pathname has failed. So either your index is invalid or corrupted.
NIOFSDirectory class is used for reading and writing index files. The directory is created at the named location if it does not yet exist.
So you should probably:
Delete the index directory or restore data from backups.
Restart the server (or at least the reload the config).