I need to fetch huge data in my code. I am using zf2, currently I am using Doctrine2 to deal with my database. But I just found that ORM queries are taking much time than plain mysql queries. So could you please suggest me the best way to fetch huge data from database in zf2.
There are a few things to consider here:
By default, Doctrine will hydrate the database records into php objects (entities): it fills the entities with the data from your query. This is done by "smart guessing" so to speak, which is relatively slow. Hydrate to arrays, and you are getting a much faster response. Read more about hydration in the manual.
By default, Doctrine does not use any caching to parse your mappings or when transforming the DQL into SQL. You can use caching to speed things up. Also, Doctrine is faster dealing with read-only entities than making them read/write. Read more about Doctrine performance issues in the manual.
Rendering 50,000 rows in a html table (or whatever) is a pain for your render engine (Zend\View / php) and your DOM (in the browser). The query might be optimized and load fairly quickly, rendering all these results into a view and displaying them into a browser will be really slow as well.
Using pagination will decrease the size of your dataset, speeding up the query and the rendering at the same time.
Using an ORM to help you with mapping, database abstraction and so on is really nice, but comes with the trade-off: performance. Using a library like Doctrine inherently increases execution time: you will never reach the performance of the php mysql functions in Doctrine.
So:
Try to decrease the number of results from one query, to speed up the database query, hydration and rendering
Try tuning performance with Doctrine by using array hydration, caching and read-only entities.
If you need to fastest way possible, don't use Doctrine. Try out Zend\Db or write all things in sql yourself
Related
I'd like to test how selecting and loading data in java application using hibernate and mysql can be optimized.
I divided results I found into 2 groups
MySQL
indexes - for sure
stored procedures - is there a difference if select is done in stored procedure?
views - is there a difference if select definition is kept in view?
query cache - does it work only if we do the same select second time?
Hibernate
hibernate cache - is this similar to query cache? how it can be configured?
lazy loading - can it help?
Are there any other ways? I use simple queries with several joins and aggregation functions.
I need to demonstate time changes between "before" and "after" optimization.
For more information I tried to read this, but language is to complicated for me.
Batch fetching is important when you read e.g. a collection. In that case Hibernate can get many rows of the collection in one SQL request, so that it would be faster.
Hibernate caching is very good solution (read about EHCache for instance) it can store retrieved data on memory and if nothing changes on his side it can retrieve it without even asking SQL engine what's going on there.
Lazy loading is a must for One to Many associations (you can kill your solution without this). But fortunately it is set by default in Hibernate for such associations.
You can also read about optimistic lock, which is faster than pessimistic lock in many cases.
Last but not least you should use proper transaction strategy, so that you shouldn't create new transactions when it is not necessary.
It is obvious that executing database query in loops has performance issues. but if the query is used as prepared statement, does it make any difference?
What is preferable joining together the tables and get the results or using prepared statement in loop?
Using join would almost always be preferred instead of looping over a result set to get additional results.
Relational Database Management Systems are built for combining related results, and does so very efficiently... additionally, this will you save many round trips to the database, which can become costly if used excessively - regardless of if you're using prepared statements or not.
The overhead to the prepared statements is probably not going to be the escaping of the inputs, it's going to be the connection to the database, or reconnection, or act of sending of the finalized sql statement. That interface between your code and the relational database is likely to be the slow point of the process more than anything else.
However, for my part, I would generally go for whatever is simplest and most able to be maintained from the start, and only worry about performance if the performance actually shows itself to be slow. Write the data-grabbing functionality in a separate function or method so that the implementation can change if the performance proves to need optimization, though.
At that point you can then start optimizing your sql, and use joins or unions as alternatives to multiple prepared statements.
I am developing with Codeigniter and when it gets to complicated database queries
I am using
$this->db->query('my complicated query');
then cast to array of object using $query->result();
and so far it's very good and useful
Now my question is
what if I want to create mysql view and select from it? Will
$this->db->from('mysql_view')
take the mysql view as it's a table or not?
And if I do that will be any difference in performance are views faster than normal database query?
What would be best practice with Codeigniter and MYSQL database dealing with complicated queries as I understand that ActiveRecord is just query builder and as some tests it's even a little slower
Thanks in advance for your advise
MySQL views are queried the same way as tables, on a side note, you can't have a table and a view share the same name.
Depends on the query you use in the view, views can be internally cached so in the long run - yes, they are faster.
Best practice in this case is to use whatever you find easy to use for yourself and your team, I personally stick to using $this->db->query(); as I find it's easier to change a simple query of this kind to have some advanced functionality like sub-queries or other things that are hard and/or impossible to do with CI query builder. My advice would be to stick to one way of queries - if you use ->query(), then use them everywhere, if you use a query builder, then use it wherever it is possible to achieve the result using it.
Is it possible to fetch data from Redis from within MySQL (using a native function or some other mechanism)? I would like to be able to use this information in ORDER BY statements for paging with LIMIT. Otherwise I will have to fetch all the data from MySQL, fetch additional data for each row from Redis, sort in my application and keep the page I need.
It would be much more efficient if MySQL could say call a function for every row to get data from Redis, do the sorting and only send me the page I need.
Even if this is possible (with open source everything is technically possible), it's unlikely to improve performance much over the cleaner approach of sorting within your app. If your data set is small, returning everything is not a problem. If your data set is large, you probably need to be sorting by an indexed column to get decent performance out of sql, and you can't index a function.
Also, if the result set isn't huge the dominant performance issue is usually latency rather than processing or data transfer. One query from sql and one mget from redis should be reasonably quick.
If you really want just one query when a page is viewed you will need to have both record and sorting data in one place - either add the data from redis as a column in sql or cache your queries in redis lists.
I want to create a query result page for a simple search, and i don't know , should i use views in my db, would it be better if i would write a query into my code with the same syntax like i would create my view.
What is the better solution for merging 7 tables, when i want to build a search module for my site witch has lots of users and pageloads?
(I'm searching in more tables at the same time)
you would be better off using a plain query with joins, instead of a view. VIEWS in MySQL are not optimized. be sure to have your tables properly indexed on the fields being used in the joins
If you always use all 7 tables, i think you should use views. Be aware that mysql changes your original query when creating the view so its always good practice to save your query elsewhere.
Also, remember you can tweak mysql's query cache env var so that it stores more data, therefore making your queries respond faster. However, I would suggest that you used some other method for caching like memcached. The paying version of mysql supports memcached natively, but Im sure you can implement it in the application layer no problem.
Good luck!