Time series data base for scientific experiments - mysql

Question

I have to perform scientific experiments using time series.
I intend to use MySQL as the data storage platform.
I'm thinking of using the following set of tables to store the data:
Table1 --> ts_id (store the time series index, I will have to deal with several time series)
Table2 --> ts_id, obs_date, value (should be indexed by {ts_idx,obs_date})
Because there will be many time series (hundreds) each with possibly millions of observations, table 2 may grow very large.
The problem is that I have to replicate this experiment several times, so I'm not sure what would be the best approach:
add an experiment_id to the tables and allow them to grow even more.
create a separate data base for each experiment.
if option 2 is better (I personally think so), what would be the best logical way to do this? I have many different experiments to perform, each needing replication. If I create a different data base for every replication, I'd get hundreds of data bases pretty soon. Is there a way to logically organize them, such as each replication as a "sub-database" of its experiment master database?

Answer 1

You might want to start out by considering how you will need to analyze your data.
Assumably your analysis will need to know about experiment name, experiment replica number, internal replicates (e.g. at each timepoint there 3 "identical" subjects measured for each treatment). So your db schema might be something like this:
experiments
exp_id int unsigned not null auto_increment primary key,
exp_name varchar(45)
other fields that any kind of experiment can have
replicates
rep_id int unsigned not null auto_increment primary key,
exp_id int unsigned not null foreign key to experiments
other fields that any kind of experiment replica can have
subjects
subject_id int unsigned not null auto_increment primary key,
subject_name varchar(45),
other fields that any kind of subject can have
observations
ob_id int unsigned not null auto_increment primary key,
rep_id int unsigned not null foreign key to replicates,
subject_id int unsigned not null foreign key to subjects,
ob_time timestamp
other fields to hold the measurements you make at each timepoint
If you have internal replicates you'll need another table to hold the internal replicate/subject relationship.
Don't worry about your millions of rows. As long as you index sensibly, there won't likely be any problems. But if worse comes to worst you can always partition your observation table (likely to be the largest) by rep_id.

Answer 2

Should you have more than one data base, one for each experiment?
The answer to your question hinges on your answer to this question: Will you want to do a lot of analysis that compares one experiment to another?
If you will do a lot of experiment-to-experiment comparison, it will be a horrendous pain in the neck to have a separate data base for every experiment.
I think your suggestion of an experiment ID column in your observation table is a fine idea. That way you can build an experiment table with an overall description of your experiment. That table can also hold the units of observation in your value column (e.g. temp, voltage, etc).
If you have some kind of complex organization of your multiple experiments, you can store that organization in your experiment table.
Notice that MySQL is quite efficient at slinging around short rows of data. You can buy a nice server for the cost of a few dozen hours of your labor, or rent one on a cloud service for the cost of a few hours of labor.
Notice also that MySQL offers the MERGE storage engine. http://dev.mysql.com/doc/refman/5.5/en/merge-storage-engine.html This allows a bunch of different tables with the same column structure to be accessed as if it were one table. This would allow you to store results from individual experiments or groups of them in their own tables, and then access them together. If you have problems scaling up your data collection system, you may want to consider this. But the good news is you can get your database working and then convert to this.
Another question: why do you have a table with nothing but ts_id values in it? I don't get that.