I am currently developing vtigercrm 7.1.0 open source. Vtiger uses mysql for its database and the crm is spilt into multiple modules. The opportunity module is the heart of the crm and contains most of the system fields. As i have been working on the system and adding more fields to the Opportunity module it has been getting progressively slower. I now have over 500 fields in the opportunity module. Each time i add a new field it creates a new column in mysql table vtiger_potentialscf.
If i run select * from vtiger_potentialscf it takes around 10 seconds to finish the query which has a detrimental effect on the end user who has to wait around 13 seconds for the webpage to load. I have read up on mysql and it doesn't like tables with too many columns.
I have been working on this system for months now but feel if i cant find a way to improve the speed i will have to look for an alternative CRM system. Does anyone have any helpful suggestions to improve the speed?
There are multiple things you should check -
Do you really need 500 fields for a module? Can they be moved to other modules or merged in someway to reduce the field count?
Run an explain on the query and see why it is taking that long.
Add indexes on the cf table - usually the potentialid column should be the index but it is possible that is fragmented. You can try defragmenting the table
Try moving some of the fields to the main vtiger_potential table or even a third vtiger_potentialcf1 table to split the data further.
Try altering the mysql configuration for optimal performance. There are multiple guides available over the internet (even stackoverflow has some).
Let me know how it goes for you.
Related
So just FYI, I have very limited SQL training, and my new position is requiring that I learn so please be gentle with me :)
First off, I saw this:
Multi-table query in Microsoft Access
And that is exactly what I need in principle; however, my data structure is a little different. I have roughly 24 databases each averaging around 1gb of information over 8 years across 420 institutions (4 databases for each year - about 16 million observations in total). Every field is identical across databases.
I need to run analyses on all of this information, but ms-access databases are limited to 2gb, so I'm trying to figure out a workaround. My idea is to link each table into a master database, and then run queries using the selection query from the above link. My question is whether or not this will actually work. My computer has 32gb of physical memory, so I feel like I should be able to load all of it into active memory while performing any queries, but I still don't know. Is there a more efficient way?
Basically, what I need to be able to query institutions over years. Right now that's impossible and it's causing issues. Because institutions aren't subjected to any scrutiny regarding the information they report, we need to understand how reporting trends evolved within and between them over time.
I was given a list of about 40 questions that all involve different queries that we need to run, so I'm hoping against hope that I can figure out a solution that doesn't involve me losing my mind.
Thank you all for your help!
-David
It is perfectly acceptable to link all the tables into one master database. In my previous experience I had one master database with over 80 backends, many with one large table. The issue with querying them is that a single Access query can also only be 2Gb. So if it any time in the processing your query exceeds that limit you'll get an error. It can be very annoying. For me the best way to work around this is temp tables and driver/parameter tables. Use driver/parameter table to store what data you'll be pulling from a source. This creates more efficient queries. Especially if you index on the join fields. In my master database I always had a form with VB code to load the driver/parameter tables.
Now all that said you'll be much happier in a different environment, even SQL Server Express. Then you'll be able to use the 32Gb of memory. Otherwise you'll always be dealing with not only the Access 2Gb file/query limit but also that Access is a 32-bit application and it can't address all the memory you have.
Just interested, maybe someone might know that. If I use lazy load to get all attributes, relations and so on it makes ~350 queries to database it takes about 2 sec to fully render the page. If i make one big query with multiple joins to get all relations I need it makes ~20 queries one is really big, and problem is that this big query first time takes about 10 sec to execute, after that it gets cached and it goes much faster and whole page loads in ~1.5 sec, but problem is that every user has different parameters to that query so for every user first time it goes for 10 sec.. why it goes so long for first time?
May I ask, if you are using a stored procedure? I have added a link with some advantages of using a stored procedure https://docs.oracle.com/cd/F49540_01/DOC/java.815/a64686/01_intr3.htm . Can you give some examples of your parameters for different users?
Thanks
As you gave no information on the data base schema, the data size and other parameters it is very difficult to determine the root cause of the bad performance. However, there is another answer here on StackOverflow that might be a great starting point for further investigation.
In general consider the following questions to start investigating / optimizing:
Do you really need all the information you fetch from the DB (at once)?
Is the database optimized for the queries you execute?
How often do you need to execute the queries and if you cache them, how often does the cache outdate?
Context
I'm currently developing a tool for managing orders and communicating between technicians and services. The industrial context is broadcast and TV. Multiple clients expecting media files each made to their own specs imply widely varying workflows even within the restricted scope of a single client's orders.
One client can ask one day for a single SD file and the next for a full-blown HD package containing up to fourteen files... In a MySQL db I am trying to store accurate information about all the small tasks composing the workflow, in multiple forms:
DATETIME values every time a task is accomplished, for accurate tracking
paths to the newly created files in the company's file system in VARCHARs
archiving background info in TEXT values (info such as user comments, e.g. when an incident happens and prevents moving forward, they can comment about it in this feed)
Multiply that by 30 different file types and this is way too much for a single table. So I thought I'd break it up by client: one table per client so that any order only ever requires the use of that one table that doesn't manipulate more than 15 fields. Still, this a pretty rigid solution when a client has 9 different transcoding specs and that a particular order only requires one. I figure I'd need to add flags fields for each transcoding field to indicate which ones are required for that particular order.
Concept
I then had this crazy idea that maybe I could create a temporary table to last while the order is running (that can range from about 1 day to 1 month). We rarely have more than 25 orders running simultaneously so it wouldn't get too crowded.
The idea is to make a table tailored for each order, eliminating the need for flags and unnecessary forever empty fields. Once the order is complete the table would get flushed, JSON-encoded, into a TEXT or BLOB so it can be restored later if changes need made.
Do you have experience with DBMS's (MySQL in particular) struggling from such practices if it has ever existed? Does this sound like a viable option? I am happy to try (which I already started) and I am seeking advice so as to keep going or stop right here.
Thanks for your input!
Well, of course that is possible to do. However, you can not use the MySQL temporary tables for such long-term storage, you will have to use "normal" tables, and have some clean-up routine...
However, I do not see why that amount of data would be too much for a single table. If your queries start to run slow due to much data, then you should add some indexes to your database. I also think there is another con: It will be much harder to build reports later on, when you have 25 tables with the same kind of data, you will have to run 25 queries and merge the data.
I do not see the point, really. The same kinds of data should be in the same table.
I am working with an application which has a 3 tables each with more than 10mm records and larger than 2GB.
Every time data is inserted there's at least one record added to each of the three tables and possibly more.
After every INSERT a script is launched which queries all these tables in order to extract data relevent to the last INSERT (let's call this the aggregation script).
What is the best way to divide the DB in smaller units and across different servers so that the load for each server is manageable?
Notes:
1. There are in excess of 10 inserts per second and hence the aggregation script is run the same number of times.
2. The aggregation script is resource intensive
3. The aggregation script has to be run on all the data in order to find which one is relevant to the last insert
4. I have not found a way of somehow dividing the DB into smaller units
5. I know very little about distributed DBs, so please use very basic terminology and provide links for further reading if possible
There are two answers to this from a database point of view.
Find a way of breaking up the database into smaller units. This is very dependent on the use of your database. This is really your best bet because it's the only way to get the database to look at less stuff at once. This is called sharding:
http://en.wikipedia.org/wiki/Shard_(database_architecture)
Have multiple "slave" databases in read only mode. These are basically copies of your database (with a little lag). For any read only queries where that lag is acceptable, they access these databases across the code in your entire site. This will take some load off of the master database you are querying. But, it will still be resource intensive on any particular query.
From a programming perspective, you already have nearly all your information (aside from ids). You could try to find some way of using that information for all your needs rather than having to requery the database after insert. You could have some process that only creates ids that you query first. Imagine you have tables A, B, C. You would have other tables that only have primary keys that are A_ids, B_ids, C_ids. Step one, get new ids from the id tables. Step two, insert into A, B, C and do whatever else you want to do at the same time.
Also, general efficiency/performance of all queries should be reviewed. Make sure you have indexes on anything you are querying. Do explain on all queries you are running to make sure they are using indexes.
This is really a midlevel/senior dba type of thing to do. Ask around your company and have them lend you a hand and teach you.
I have a large mysql MyISAM table with 1.5mil rows and 4.5GB big, still increasing everyday.
I have done all the necessary indexing and the performance has been greatly optimized. Yet, the database occasionally break down (showing 500 Internal Server error) usually due to query overload. Whenever there is a break down, the table will start to work very slowly and I'll have to do a silly but effective task : copy the entire table over to a new table and replace the new one with the old one!!
You may ask why such a stupid action. Why not repair or optimize the table? I've tried that but the time to do repair or optimization may be more than the time to simply duplicate the table and more importantly the new table performs much faster.
Newly built table usually work very well. But over time, it will become sluggish (maybe after a month) and eventually lead to another break down (500 internal server). That's when everything slow down significantly and I need to repeat the silly process of replacing table.
For your info:
- The data in the table seldom get deleted. So there isn't a lot of overhead in the table.
- Under optimal condition, each query takes 1-3 secs. But when it becomes sluggish, the same query can take more than 30 seconds.
- The table has 24 fields, 7 are int, 3 are text, 5 are varchar and the rest are smallint. It's used to hold articles.
If you can explain what cause the sluggishness or you have suggestion on how to improve the situation, feel free to share it. I will be very thankful.
Consider moving to InnoDB. One of its advantages is that it's crash safe. If you need full text capabilities, you can achieve that by implementing external tools like Sphinx or Lucene.
Partitioning is a common strategy here. You might be able to partition the articles by what month they were committed to the database (for example) and then have your query account for returning results from the month of interest (how you partition the table would be up to you and your application's design/behavior). You can union results if you will need your results to come from more than one table.
Even better, depending on your MySQL version, partitioning may be supported by your server. See this for details.