How to improve a Doctriny query to retrieve huge quantity of data? - mysql

Question

I'm working on a Symfony2 project using Doctrine.
I created a function to retrieve stocks items.
It works well if my database contains less than 50 000 rows.
However, my project needs to deal with more than 2 000 000 rows and If I use the following request, It needs more than 1 minutes to display my page.
Repository (StockRepository.php) :
public function findAllQuery() {
$query = $this->createQueryBuilder('s')
->select(array('s','st','p'))
->join('s.store','st')
->join('s.product','p')
->orderBy('st.name','ASC');
return $query;
}
Controller (StockController.php) :
public function indexAction() {
$em = $this->getDoctrine()->getManager();
$entities = $em->getRepository('LiveDataShopBundle:Stock')->findAllQuery();
$paginator = $this->get('knp_paginator');
$pagination = $paginator->paginate(
$entities, $this->get('request')->query->get('page', 1), 50
);
return array(
'pagination' => $pagination,
);
}
Entity (Stock.php) :
<?php
namespace LiveData\Bundle\ShopBundle\Entity;
use Doctrine\ORM\Mapping as ORM;
/** * Stock * * #ORM\Table() *
#ORM\Entity(repositoryClass="LiveData\Bundle\ShopBundle\Entity\StockRepository")
*/ class Stock {
/**
* #var integer
*
* #ORM\Column(name="id", type="integer")
* #ORM\Id
* #ORM\GeneratedValue(strategy="AUTO")
*/
private $id;
/**
* #var \stdClass
*
* #ORM\ManyToOne(targetEntity="Store", inversedBy="stocks")
* #ORM\JoinColumn(name="store_id", referencedColumnName="id", onDelete="set null")
*/
private $store;
/**
* #var integer
*
* #ORM\Column(name="quantity", type="integer")
*/
private $quantity;
/**
* #var \stdClass
*
* #ORM\ManyToOne(targetEntity="Product", inversedBy="stocks")
* #ORM\JoinColumn(name="product_id", referencedColumnName="id", onDelete="set null")
*/
private $product;
/**
* Get id
*
* #return integer
*/
public function getId()
{
return $this->id;
}
/**
* Set store
*
* #param \stdClass $store
* #return Stock
*/
public function setStore($store)
{
$this->store = $store;
return $this;
}
/**
* Get store
*
* #return \stdClass
*/
public function getStore()
{
return $this->store;
}
/**
* Set quantity
*
* #param integer $quantity
* #return Stock
*/
public function setQuantity($quantity)
{
$this->quantity = $quantity;
return $this;
}
/**
* Get quantity
*
* #return integer
*/
public function getQuantity()
{
return $this->quantity;
}
/**
* Set product
*
* #param \stdClass $product
* #return Stock
*/
public function setProduct($product)
{
$this->product = $product;
return $this;
}
/**
* Get product
*
* #return \stdClass
*/
public function getProduct()
{
return $this->product;
} }
Any idea to reduce the process time ?
Thanks in advance !

Answer 1

I do not know much about doctrine 2 only doctrine 1. Like i see in your code you only displaying 50 at one time. If this is the case than try to do a count(*) on stocks put this into the paginator and load only 50 (offset and limit) per page. That should help a lot.
Or use sphinx/solr. But i think the problem is the big amout of data that has to be moved from db to doctrine2/symphony2 (2.000.000 rows), try to debug and get futher informations.

Answer 2

If your problem is in mapping from db to objects, try to enable apc (for example) cache:
#app/config/config.yml
doctrine:
orm:
metadata_cache_driver: apc
query_cache_driver: apc