MySQL database with thousands of tables - mysql

Question

So I'm building a database in MySQL which contains approximately 20,000 tables, one for each human gene, where each gene's table has a single column listing the alternative names (synonyms) for this gene found in the literature, and often times there's no logic to these synonyms and they exist purely for historical reasons.
First off, is there a better way to set up this database with fewer tables?
The problem is that each gene has a variable number of alternative names, so it's not like I can make one big table with each row corresponding to a gene and a set number of columns. And even if each gene had the same number of alternative names, any particular column would basically be meaningless, since, for example, there would be no relationship between the synonym in column 1 for gene 1 and the synonym in column 1 for gene 2.
What exactly is bad about having thousands of tables in MySQL?
I could potentially break the database up into 23 databases (one for each chromosome), or something like that, and then each database would only have ~900 tables, would something like that be better?
I almost feel like maybe MySQL (a relational database) is the wrong tool for the job. If that's the case, what would be a better database paradigm?

Answer 1

20,000 tables is a lot of tables. There's nothing necessarily "bad" about having 20,000 tables, if you actually have a need 20,000 tables. We run with innodb_file_per_table, so that's a whole slew of files, and we'd potentially running up against some limits in MySQL (innodb_open_files, open_files_limit, table_cache_open) which are in turn limited by the OS ulimit.
Add to that the potential difficulty managing a large number of identical tables. If I need to add a column, I'd be needing to add that column to 20,000 tables. That's 20,000 ALTER TABLE statements. And if I miss some tables, the tables won't be identical anymore. I just don't want to go there, if I can help it.
I'd propose and consider a different design.
As a first cut, something like:
CREATE TABLE gene_synonym
( gene VARCHAR(64)
, synonym VARCHAR(255)
, PRIMARY KEY (gene, synonym)
) ENGINE=InnoDB
;
To add a synonym for a gene, rather than inserting a value into a single column of a particular table:
INSERT INTO gene_synonym (gene, synonym) VALUES ('alzwhatever','iforgot');
And the querying, instead of figuring out which of the 20,000 tables to be queries, we would query just one table and include a condition on gene column:
SELECT gs.synonym
FROM gene_synonym gs
WHERE gs.gene = 'alzwhatever'
ORDER BY gs.synonym
The WHERE clause makes it so we can view a subset of the one big table, the set returned will emulate one of the currently individual tables.
And if I needed to search for a synonym, I could query just this one table:
SELECT gs.gene
FROM gene_synonym gs
WHERE gs.synonym = 'iforgot'
To do that same search with 20,000 tables, I would need 20,000 different SELECTs, one for each each of the 20,000 tables.
I just took a swag at the datatypes. Since MySQL has a limit of 64 characters for a table name, I limited the gene column to 64 characters.
We could populate the gene column with the names of the tables in the current design.
However, what this table can't emulate is an empty table, a gene that doesn't have any synonyms. (Or maybe our design would be for the name of the gene to be a synonym of itself, so we'd have a row ('alzwhatever','alzwhatever')
Either case, we'd likely also want to add a table like this:
CREATE TABLE gene
( gene VARCHAR(64)
, PRIMARY KEY (gene)
) ENGINE=InnoDB
;
This is the table that would have the 20,000 rows, one row for each of the tables in your current design.
Further, we can add a foreign key constraint
ALTER TABLE gene_synonym
ADD CONSTRAINT FK_gene_synonym_gene (gene) REFERENCES gene (gene)
ON UPDATE CASCADE ON DELETE CASCADE
;
This design is much more in keeping with normative pattern for relational databases.
This isn't to say that other designs are "bad". Just this design would be more typical.

Answer 2

You should have a synonym table. One such table:
create table geneSynonyms (
geneSynonymId int auto_increment primary key,
geneId int not null,
synonym varchar(255),
constraint fk_geneSynonyms_geneId foreign key (geneId) references genes(geneId),
constraint unq_geneSynonyms_synonym unique (synonym) -- I assume this is unique
);
Then, you have one row for each synonym for all the genes in a single table.
What is bad about having thousands of tables? Here are a few things:
The data storage is very inefficient. The minimum space occupied by a table is a data page. If you don't fill the page you are wasting space.
By wasting space, you end up filling the page cache with nearly empty pages. This means less data fits into memory, which adversely affects performance.
Your queries are hard-wired to the table being accessed. You cannot write generic code for multiple genes.
You cannot make changes to your data structure, easily.
You cannot validate data, by having rules that say "a synonym needs to be unique across all genes".
You cannot readily find the gene that a synonym refers to.
Improving performance by, say, adding indexes or partitioning the data is a nightmare.