I just converted an Access Db to mysql (using Access as frontend, mysql as backend)
It is a simple, 4 table database
I have a Master table, and 3 linked tables to it
So an Access form displays has data from:
Master table (mainform)
Details1 table (subform)
Details2 table (subform)
Details3 table (subform)
The master table will always be one row, however all the linked tables ("details" tables) can have any number of records, usually around 10-30 rows in each detail table per master record.
Everything runs well, however when checking Mysql Administrator Health>Connection Health>Number of SQL Queries, the number of queries jump to 10 (queries) everytime I move between Master record pages.
I am running this on my own laptop, and I am worried this will become a problem (performance) when I put 100+ users in the work server all working at once.
Could anyone advise if this high "number of queries" reported by Mysql Admin will be a problem?
What number is considered "dangerous" for performance purposes?
The idea is to have a fast running system, so I would like to avoid too many queries to the database.
I also dont understand why (example) it displays 7 queries when there are only 4 tables in total..with only one row per table being dislayed
ANy ideas/comments will be appreciated
Can there something be changed in Access front end to make the number of queries lower ?
thanks so much
Those 10 queries probably don't take a long time, and they are very likely sequential. I doubt there will be a problem with 100 users since they won't all be running the queries at once. Even then, mysql can handle quite a load.
I'm not sure what is going on inside Access. "Queries" can be just about anything (i.e. meta data), not just queries for records from the table. For example, getting the total number of records in a table to display something like "showing 23 of 1,000". If access is doing this for each table, that's an extra 4 queries right there, leaving only 3 to get actual data to display.
It's hard to be sure because it depends on a lot of things like server's memory, cpu and complexity of the queries but...
Supposing the queries for the subforms are directly linked to the master table (with an indexed id field) and do not need to join with other tables (as you have only 4 tables), I think you're ok to run without problems as the number of queries is not too high.
As an example, some years ago I had an old machine (Athlon XP1600 with only 512MB or 1GGB RAM) running mysql and serving files for 20 users. Most of the queries were small Stored Procedures using mainly 20 tables but returning a lot of rows (usualy around 2000 for the most used query). Everything was fast. This old system ran 14 millions queries in 2 months (average > 700 per minute) so I think you will be OK.
Anyways, if you have a way to do a partial test it would be the best option. You could use a small script querying the database in a loop on several machines for example.
Related
So just FYI, I have very limited SQL training, and my new position is requiring that I learn so please be gentle with me :)
First off, I saw this:
Multi-table query in Microsoft Access
And that is exactly what I need in principle; however, my data structure is a little different. I have roughly 24 databases each averaging around 1gb of information over 8 years across 420 institutions (4 databases for each year - about 16 million observations in total). Every field is identical across databases.
I need to run analyses on all of this information, but ms-access databases are limited to 2gb, so I'm trying to figure out a workaround. My idea is to link each table into a master database, and then run queries using the selection query from the above link. My question is whether or not this will actually work. My computer has 32gb of physical memory, so I feel like I should be able to load all of it into active memory while performing any queries, but I still don't know. Is there a more efficient way?
Basically, what I need to be able to query institutions over years. Right now that's impossible and it's causing issues. Because institutions aren't subjected to any scrutiny regarding the information they report, we need to understand how reporting trends evolved within and between them over time.
I was given a list of about 40 questions that all involve different queries that we need to run, so I'm hoping against hope that I can figure out a solution that doesn't involve me losing my mind.
Thank you all for your help!
-David
It is perfectly acceptable to link all the tables into one master database. In my previous experience I had one master database with over 80 backends, many with one large table. The issue with querying them is that a single Access query can also only be 2Gb. So if it any time in the processing your query exceeds that limit you'll get an error. It can be very annoying. For me the best way to work around this is temp tables and driver/parameter tables. Use driver/parameter table to store what data you'll be pulling from a source. This creates more efficient queries. Especially if you index on the join fields. In my master database I always had a form with VB code to load the driver/parameter tables.
Now all that said you'll be much happier in a different environment, even SQL Server Express. Then you'll be able to use the 32Gb of memory. Otherwise you'll always be dealing with not only the Access 2Gb file/query limit but also that Access is a 32-bit application and it can't address all the memory you have.
Im re working an existing PHP/MySql/JS/Ajax web app that processes a LARGE number of table rows for users. Here's how the page works currently.
A user uploads a LARGE csv file. The test one I'm working with has 400,000 rows, (each row has 5 columns).
Php creates a brand new table for this data and inserts the hundreds of thousands of rows.
The page then sorts / processes / displays this data back to the user in a useful way. Processing includes searching, sorting by date and other rows and re displaying them without a huge load time (thats where the JS/Ajax comes in).
My question is should this app be placing the data into a new table for each upload or into one large table with an id for each file? I think the origional developer was adding seperate tables for speed purposes. Speed is very important for this.
Is there a faster way? Is there a better mouse trap? Has anyone ever delt with this?
Remember every .csv can contain hundreds of thousands of rows and hundreds of .csv files can be uploaded daily. Though they can be deleted about 24 hrs after they were last used (Im thinking cron job any opinions?)
Thank you all!
A few notes based on comments:
All data is unique to each user and changes so the user wont be Re accessing this data after a couple of hours. Only if they accidentally close the window and then come right back would they really re visit for the same .csv.
No Foreign keys required all csv's are private to each user and dont need to be cross referenced.
I would shy away from putting all the data into a single table for the simple reason that you cannot change the data structure.
Since the data is being deleted anyway and you don't have a requirement to combine data from different loads, there isn't an obvious reason for putting the data into a single table. The other argument is that the application now works. Do you really want to discover some requirement down the road that implies separate tables after you've done the work?
If you do decide on a single table, then use table partitioning. Since each user is using their own data, you can use partitions to separate each user load into a separate partition. Although there are limits on partitions (such as no foreign keys), this will make access the data in a single table as fast as accessing the original data.
Given 105 rows and 102 CSVs per day, you're looking at 10 million rows per day (and you say you'll clear that data down regularly). That doesn't look like a scary figure for a decent db (especially given that you can index within tables, and not across multiple tables).
Obviously the most regularly used CSVs could be very easily held in memory for speed of access - perhaps even all of them (a very simple calculation based on next to no data gives me a figure of 1Gb if you flush every over 24 hours. 1Gb is not an unreasonable amount of memory these days)
I am working with an application which has a 3 tables each with more than 10mm records and larger than 2GB.
Every time data is inserted there's at least one record added to each of the three tables and possibly more.
After every INSERT a script is launched which queries all these tables in order to extract data relevent to the last INSERT (let's call this the aggregation script).
What is the best way to divide the DB in smaller units and across different servers so that the load for each server is manageable?
Notes:
1. There are in excess of 10 inserts per second and hence the aggregation script is run the same number of times.
2. The aggregation script is resource intensive
3. The aggregation script has to be run on all the data in order to find which one is relevant to the last insert
4. I have not found a way of somehow dividing the DB into smaller units
5. I know very little about distributed DBs, so please use very basic terminology and provide links for further reading if possible
There are two answers to this from a database point of view.
Find a way of breaking up the database into smaller units. This is very dependent on the use of your database. This is really your best bet because it's the only way to get the database to look at less stuff at once. This is called sharding:
http://en.wikipedia.org/wiki/Shard_(database_architecture)
Have multiple "slave" databases in read only mode. These are basically copies of your database (with a little lag). For any read only queries where that lag is acceptable, they access these databases across the code in your entire site. This will take some load off of the master database you are querying. But, it will still be resource intensive on any particular query.
From a programming perspective, you already have nearly all your information (aside from ids). You could try to find some way of using that information for all your needs rather than having to requery the database after insert. You could have some process that only creates ids that you query first. Imagine you have tables A, B, C. You would have other tables that only have primary keys that are A_ids, B_ids, C_ids. Step one, get new ids from the id tables. Step two, insert into A, B, C and do whatever else you want to do at the same time.
Also, general efficiency/performance of all queries should be reviewed. Make sure you have indexes on anything you are querying. Do explain on all queries you are running to make sure they are using indexes.
This is really a midlevel/senior dba type of thing to do. Ask around your company and have them lend you a hand and teach you.
We have a report that users can run that needs to select records from 5 different services. Right not, I am using UNION to combine all the tables in one query, but sometimes, it was just too much for the server and it crashed!
I optimized bits and pieces of the query (where's and table joins) and there haven't been any crashes since, but the report still takes a long time to load (ie the query is very slow).
The question is, will mysql perform faster and more optimally if I create 5 temp tables for the different service types, and then select from all of the temps? Or is there a different idea?
I could, of course, just use 5 separate selects and then combine them in the code (php). But I imagine this would cause the report to load even slower...
Any ideas?
Usually the limiting factor in speed is the database, not PHP. I'd suggest running seperate queries and let the PHP do the combining, see if that is faster. If you're not storing all data in arrays or doing other heavy processing, I suspect the PHP way is much faster.
(this was actually meant as a comment but don't have those rights yet..)
I'm developping a chat application. I want to keep everything logged into a table (i.e. "who said what and when").
I hope that in a near future I'll have thousands of rows.
I was wondering : what is the best way to optimize the table, knowing that I'll do often rows insertion and sometimes group reading (i.e. showing an entire conversation from a user (look when he/she logged in/started to chat then look when he/she quit then show the entire conversation)).
This table should be able to handle (I hope though !) many many rows. (15000 / day => 4,5 M each month => 54 M of rows at the end of the year).
The conversations older than 15 days could be historized (but I don't know how I should do to do it right).
Any idea ?
I have two advices for you:
If you are expecting lots of writes
with little low priority reads. Then you
are better off with as little
indexes as possible. Indexes will
make insert slower. Only add what you really need.
If the log table
is going to get bigger and bigger
overtime you should consider log
rotation. Otherwise you might end up
with one gigantic corrupted table.
54 million rows is not that many, especially over a year.
If you are going to be rotating out lots of data periodically, I would recommend using MyISAM and MERGE tables. Since you won't be deleting or editing records, you won't have any locking issues as long as concurrency is set to 1. Inserts will then always be added to the end of the table, so SELECTs and INSERTs can happen simultaneously. So you don't have to use InnoDB based tables (which can use MERGE tables).
You could have 1 table per month, named something like data200905, data200904, etc. Your merge table would them include all the underlying tables you need to search on. Inserts are done on the merge table, so you don't have to worry about changing names. When it's time to rotate out data and create a new table, just redeclare the MERGE table.
You could even create multiple MERGE tables, based on quarter, years, etc. One table can be used in multiple MERGE tables.
I've done this setup on databases that added 30 million records per month.
Mysql does surprisingly well handling very large data sets with little more than standard database tuning and indexes. I ran a site that had millions of rows in a database and was able to run it just fine on mysql.
Mysql does have an "archive" table engine option for handling many rows, but the lack of index support will make it not a great option for you, except perhaps for historical data.
Index creation will be required, but you do have to balance them and not just create them because you can. They will allow for faster queries (and will required for usable queries on a table that large), but the more indexes you have, the more cost there will be inserting.
If you are just querying on your "user" id column, an index on there will not be a problem, but if you are looking to do full text queries on the messages, you may want to consider only indexing the user column in mysql and using something like sphynx or lucene for the full text searches, as full text searches in mysql are not the fastest and significantly slow down insert time.
You could handle this with two tables - one for the current chat history and one archive table. At the end of a period ( week, month or day depending on your traffic) you can archive current chat messages, remove them from the small table and add them to the archive.
This way your application is going to handle well the most common case - query the current chat status and this is going to be really fast.
For queries like "what did x say last month" you will query the archive table and it is going to take a little longer, but this is OK since there won't be that much of this queries and if someone does search like this he would be willing to wait a couple of seconds more.
Depending on your use cases you could extend this principle - if there will be a lot of queries for chat messages during last 6 months - store them in separate table too.
Similar principle (for completely different area) is used by the .NET garbage collector which has different storage for short lived objects, long lived objects, large objects, etc.