I have Drupal site with MySQL database on my hosting provider. He tells me that my DB is almost on max limit about 1GB but when I export DB to output file, the file is only 80 MB for whole DB.
It is logical DB to be smaller than my output file or same but almost when DB on hosting is 10 times larger than export file, I think it is impossible.
Can you help me to find out is it possible, or my hosting provider manipulate my data and write me every day messages to get bigger DB storage for more money?
Functioning MySQL databases use plenty of data storage for such things as indexes. A 12x ratio between your raw (exported) data size and the disk space your database uses is a large ratio, but it's still very possible.
MySQL makes the tradeoff of using lots of disk space to improve query performance. That's a wise tradeoff these days because the cost of disk space is low, and decreasing very fast.
You may (or may not) recover some data space by going in to phpmyadmin and optimizing all your tables. This query will optimize a list of tables.
OPTIMIZE TABLE tablename, tablename, tablename
It's smart to back up your data base before doing this.
A 1GiB limit on a MySQL database size is absurdly low. (A GiB of storage at AWS or another cloud provider costs less than $US 0.20 per month these days.) It may be time to get a better hosting service.
Related
Recently I've found these two MySQL functions compress and uncompress and I'm planning to use it in my database to compress some xml files (around 40 million ~ 600GB+) to reduce my database size. Unfortunately, I didn't found anything about performance and ram/hardware usage from those functions.
How would it be in terms of performance and server ram usage performing hundred/thousands of queries like these per minute?
SELECT cast(uncompress(document) AS char(10000))
FROM documents;
I have a table which is expected to have 22 billion records yearly. How much will be the RAM requirement if each of the records cost around 4 KB of data.
It is expected to have around 8 TB of storage for the same table.
[update]
There is no join queries involved. I just need the select queries to be executed efficiently.
I have found that there is no general rule of thumb when it comes to how much RAM you need for x amount of records in MYSQL.
The first factor which you need to look at is the design of the database itself. This is one of the most impacting factors of them all. If your database is poorly designed, throwing RAM at it isn't going to fix your problem.
Another factor is how this data is going to be accessed, i.e if a specific row is being accessed by 100 people SELECT * FROM table where column = value then you could get away with a tiny amount of RAM as you would just use query caching.
It MAY (not always) be a good idea to keep your entire database in RAM to allow it to be read quicker (Dependent on the total size of the database). I.e. if your database is 100GB in size then 128GB of RAM should be proficient to deal with any overheads such as the OS and other factors.
As per my system i am supporting daily Oracle 224GB CDR record to a network operator.
Also for another system daily 20 lack data retrieve from SQL database .
you can use 128 GB if you are using one server or else
if you are using load balancer then you can use 62 GB on every PC.
I have nearly 2.54GB database datas, which have tens millions of listing.
Now I have optimized mysql query as good as I can. but still I got 10 to 12 secs to get data. SO can anyone help me what should I do now ?
There are several things you could do:
If it's feasible, optimize your database by choosing the data sizes and types which fit best;
Add indexes to the most searched columns in your queries;
Choose the right configuration parameters for your database. You should use MySQLTuner-perl and/or the database configuration wizard from Percona (free registration required). Remember, tuning MySQL is a trial-and-error process; there is no "right" configuration, only one that works better for you. For instance, you could find that you get better performances with a large query cache, or with a disabled query cache altogether;
You could move your database to a SSD drive to increase disk access times.
I'm developing a database that holds large scientific datasets. Typical usage scenario is that on the order of 5GB of new data will be written to the database every day; 5GB will also be deleted each day. The total database size will be around 50GB. The server I'm running on will not be able to store the entire dataset in memory.
I've structured the database such that the main data table is just a key/value store consisting of a unique ID and a Value.
Queries are typically for around 100 consecutive values,
eg. SELECT Value WHERE ID BETWEEN 7000000 AND 7000100;
I'm currently using MySQL / MyISAM, and these queries take on the order of 0.1 - 0.3 seconds, but recently I've come to realize that MySQL is probably not the optimal solution for what is basically a large key/value store.
Before I start doing lots of work installing the new software and rewriting the whole database I wanted to get a rough idea of whether I am likely to see a significant performance boost when using a NoSQL DB (e.g. Tokyo Tyrant, Cassandra, MongoDB) instead of MySQL for these types of retrievals.
Thanks
Please consider also OrientDB. It uses indexes with RB+Tree algorithm. In my tests with 100GB of database reads of 100 items took 0.001-0.015 seconds on my laptop, but it depends how the key/value are distributed inside the index.
To make your own test with it should take less than 1 hour.
One bad news is that OrientDB not supports a clustered configuration yet (planned for September 2010).
I use MongoDB in production for a write intensive operation where I do well over the rates you are referring to for both WRITE and READ operations, the size of the database is around 90GB and a single instance (amazon m1.xlarge) does 100QPS I can tell you that a typical key->value query takes about 1-15ms on a database with 150M entries, with query times reaching the 30-50ms time under heavy load.
at any rate 200ms is way too much for a key/value store.
If you only use a single commodity server I would suggest mongoDB as it quite efficient and easy to learn
if you are looking for a distributed solution you can try any Dynamo clone:
Cassandra (Facebook) or Project Volemort (LinkedIn) being the most popular.
keep in mind that looking for strong consistency slows down these systems quite a bit.
I would expect Cassandra to do better where the dataset does not fit in memory than a b-tree based system like TC, MySQL, or MongoDB. Of course, Cassandra is also designed so that if you need more performance, it's trivial to add more machines to support your workload.
Do access .MDB files have an upper size limit? Will, for example, applications that connect to an .MDB file over 1GB have problems?
What problems/risks are there with MDB files over 1GB and what can be done?
Yes. The maximum size for an MDB file is 2 GB and yes any file over 1 GB is really pushing Access.
Original (now broken) link Access Database Specifications
See Access Database Specifications for more. (Wayback machine)
You may find data retrieval painfully slow with a large Access database. Indexing can reduce the pain considerably. For example if you have a query which includes "WHERE somefield = 27", data retrieval can be much faster if you create an index on somefield. If you have little experience with indexing, try the Performance Analyzer tool to get you started. In Access 2003 the Performance Analyzer is available from Tools -> Analyze -> Performance. I'm not sure about other Access versions.
One caveat about indexes is that they add overhead for Insert, Update, and Delete operations because the database engine must revise indexes in addition to the table where the changes occur. So if you were to index everything, you would likely make performance worse.
Try to limit the amount of data your client application retrieves from the big database. For example with forms, don't use a table as the form's data source. Instead create a query which returns only one or a few rows, and use the query as the form's data source. Give the user a method to select which record she wants to work on and retrieve only that record.
Your didn't mention whether you have performed Compact and Repair. If not, try it; it could shrink the size of your database considerably. In addition to reclaiming unused space, compact also updates the index statistics which helps the database engine determine how to access data more efficiently.
Tony Toews has more information about Access performance you may find useful, though it's not specific to large databases. See Microsoft Access Performance FAQ
If you anticipate bumping up against the 2 GB limit for MDB files, consider moving your data into SQL Server. The free Express version also limits the amount of data you can store, but is more generous than Access. SQL Server Express R2 will allow you to store 10 GB. Actually I would probably move to SQL Server well before Access' 2 GB limit.
2 GB total for all objects in the database
Total number of objects in a database 32,768