Doctrine Native SQL many-to-many query - mysql

I have a many-to-many relationship between Students and Programs with tables student, program, and student_program in my database.
I'm trying to join the two entities and perform some custom queries that require subqueries. This means that the Doctrine QueryBuilder cannot work because it does not support subqueries.
Instead, I'm trying the NativeSQL function and am making decent progress. However, when I try to SELECT something from the Program entity, I get the error Notice: Undefined index: Bundle\Entity\Program in vendor/doctrine/orm/lib/Doctrine/ORM/Internal/Hydration/ObjectHydrator.php line 180.
$mapping = new \Doctrine\ORM\Query\ResultSetMappingBuilder($em);
$mapping->addRootEntityFromClassMetadata('Student', 's');
$mapping->addJoinedEntityFromClassMetadata('Program', 'p', 's', 'programs', array('id' => 'program_id'));
// Query based on form
$sql = 'SELECT s.id, s.last_name, p.name <---- problem when this is added
FROM student s
JOIN program p
';
$query = $em->createNativeQuery($sql, $mapping);
$students = $query->getResult();

Not a direct answer but doctrine 2 does indeed support sub queries. Just create a query then feed the dql into a where class. This example is somewhat verbose but it works just fine:
public function queryGames($search)
{
// Pull params
$ages = $this->getValues($search,'ages');
$genders = $this->getValues($search,'genders');
$regions = $this->getValues($search,'regions');
$sortBy = $this->getValues($search,'sortBy',1);
$date1 = $this->getValues($search,'date1');
$date2 = $this->getValues($search,'date2');
$time1 = $this->getValues($search,'time1');
$time2 = $this->getValues($search,'time2');
$projectId = $this->getValues($search,'projectId');
// Build query
$em = $this->getEntityManager();
$qbGameId = $em->createQueryBuilder(); // ### SUB QUERY ###
$qbGameId->addSelect('distinct gameGameId.id');
$qbGameId->from('ZaysoCoreBundle:Event','gameGameId');
$qbGameId->leftJoin('gameGameId.teams', 'gameTeamGameId');
$qbGameId->leftJoin('gameTeamGameId.team','teamGameId');
if ($projectId) $qbGameId->andWhere($qbGameId->expr()->in('gameGameId.projectId',$projectId));
if ($date1) $qbGameId->andWhere($qbGameId->expr()->gte('gameGameId.date',$date1));
if ($date2) $qbGameId->andWhere($qbGameId->expr()->lte('gameGameId.date',$date2));
if ($time1) $qbGameId->andWhere($qbGameId->expr()->gte('gameGameId.time',$time1));
if ($time2) $qbGameId->andWhere($qbGameId->expr()->lte('gameGameId.time',$time2));
if ($ages) $qbGameId->andWhere($qbGameId->expr()->in('teamGameId.age', $ages));
if ($genders) $qbGameId->andWhere($qbGameId->expr()->in('teamGameId.gender',$genders));
if ($regions)
{
// $regions[] = NULL;
// $qbGameId->andWhere($qbGameId->expr()->in('teamGameId.org', $regions));
$qbGameId->andWhere($qbGameId->expr()->orX(
$qbGameId->expr()->in('teamGameId.org',$regions),
$qbGameId->expr()->isNull('teamGameId.org')
));
}
//$gameIds = $qbGameId->getQuery()->getArrayResult();
//Debug::dump($gameIds);die();
//return $gameIds;
// Games
$qbGames = $em->createQueryBuilder();
$qbGames->addSelect('game');
$qbGames->addSelect('gameTeam');
$qbGames->addSelect('team');
$qbGames->addSelect('field');
$qbGames->addSelect('gamePerson');
$qbGames->addSelect('person');
$qbGames->from('ZaysoCoreBundle:Event','game');
$qbGames->leftJoin('game.teams', 'gameTeam');
$qbGames->leftJoin('game.persons', 'gamePerson');
$qbGames->leftJoin('game.field', 'field');
$qbGames->leftJoin('gameTeam.team', 'team');
$qbGames->leftJoin('gamePerson.person', 'person');
$qbGames->andWhere($qbGames->expr()->in('game.id',$qbGameId->getDQL())); // ### THE TRICK ###
switch($sortBy)
{
case 1:
$qbGames->addOrderBy('game.date');
$qbGames->addOrderBy('game.time');
$qbGames->addOrderBy('field.key1');
break;
case 2:
$qbGames->addOrderBy('game.date');
$qbGames->addOrderBy('field.key1');
$qbGames->addOrderBy('game.time');
break;
case 3:
$qbGames->addOrderBy('game.date');
$qbGames->addOrderBy('team.age');
$qbGames->addOrderBy('game.time');
$qbGames->addOrderBy('field.key1');
break;
}
// Always get an array even if no records found
$query = $qbGames->getQuery();
$items = $query->getResult();
return $items;
}

Related

Mysql search on high number of rows

I try to import a relatively high number of data in a mysql database (around 6 millions entries coming from text files).
I have to check for each entry if there is not already a similar record in the database by comparing it with two text fields :
`ref` varchar(30) COLLATE utf8_unicode_ci NOT NULL
`labelCanonical` varchar(15) COLLATE utf8_unicode_ci DEFAULT NULL
Files are processed by batches of N entries (for this example 10), and I do a single query to check for all duplicates in the batch, like so :
SELECT p.`ref`, p.`labelCanonical`
FROM `rtd_piece` p
WHERE (p.`ref` = "6569GX" AND p.`labelCanonical` = "fsc-principal")
OR (p.`ref` = "6569GY" AND p.`labelCanonical` = "fsc-principal")
OR (p.`ref` = "6569GZ" AND p.`labelCanonical` = "fsc-principal")
OR (p.`ref` = "6569H0" AND p.`labelCanonical` = "fsc-habitacle")
OR (p.`ref` = "6569H1" AND p.`labelCanonical` = "support-fsc")
OR (p.`ref` = "6569H2" AND p.`labelCanonical` = "fsc-injection")
OR (p.`ref` = "6569H4" AND p.`labelCanonical` = "fsc-injection")
OR (p.`ref` = "6569H8" AND p.`labelCanonical` = "faisceau-mot")
OR (p.`ref` = "6569H9" AND p.`labelCanonical` = "faisceau-mot")
OR (p.`ref` = "6569HA" AND p.`labelCanonical` = "fsc-principal")
I use Doctrine 2 (without Symfony), and I do this query using "NativeQuery".
This problem is, even with a 600k entries in the database, this query takes 730ms (or 6.7 seconds for a batch of 100 records) to execute and it increases dramatically as records are added to the database.
I have no index on "ref" or "labelCanonical" fields for now, and I'm not sure if adding one will do any good with the kind of request I do.
Where I am wrong with this method so its so slow ?
Edit to add more information about the process.
I do an ajax query for each batch, also to give a feedback to the user.
When in the server side (PHP), I do the following procedure :
1) I seek on the current file on processing and extract next N records
2) I parse each line and add references and slugified labels to two different arrays
3) I try to get these records from the database to avoid duplicates :
$existing = array();
$results = getRepository('Piece')->findExistingPieces($refs, $labels);
for ($i = 0, $c = count($results); $i < $c; ++$i) {
$existing[] = $results[$i]['ref'].'|'.$results[$i]['labelCanonical'];
}
public function findExistingPieces(array $refs, array $labels)
{
$sql = '';
$where = array();
$params = array();
for ($i = 0, $c = count($refs); $i < $c; ++$i) {
$params[] = $refs[$i];
$params[] = $labels[$i];
$where[] = '(p.`ref` = ? AND p.`labelCanonical` = ?)';
}
$sql = 'SELECT p.`ref`, p.`labelCanonical` '.
'FROM `rtd_piece` p '.
'WHERE '.implode(' OR ', $where);
$rsm = new ResultSetMapping;
$rsm->addScalarResult('ref', 'ref');
$rsm->addScalarResult('labelCanonical', 'labelCanonical');
$query = $this->getEntityManager()
->createNativeQuery($sql, $rsm)
->setParameters($params);
return $query->getScalarResult();
}
4) I iterate through previously parsed data and check for duplicates :
for ($i = 0; $i < $nbParsed; ++$i) {
$data = $parsed[$i];
if (in_array($data['ref'].'|'.$data['labelCanonical'], $existing)) {
// ...
continue ;
}
// Add record
$piece = new PieceEntity;
$piece->setRef($data['ref']);
//...
$em->persist($piece);
}
5) I flush at the end of the batch
I've added some "profiling" code to track the time being spent for each step, here the result :
0.00024509429931641 (0.245 ms) : Initialized
0.00028896331787109 (0.289 ms) : Start doProcess
0.00033092498779297 (0.331 ms) : Read and parse lines
0.0054769515991211 (5.477 ms) : Check existence in database
6.9432899951935 (6,943.290 ms) : Process parsed data
6.9459540843964 (6,945.954 ms) : Finilize
6.9461529254913 (6,946.153 ms) : End of process
6.9464020729065 (6,946.402 ms) : End doProcess
6.9464418888092 (6,946.442 ms) : Return result
The first number show microseconds elapsed since the beginning of the request, then the same time in milliseconds and then what is being done.
So after some refactoring, here it's what I came with :
I check for duplicates using a new field named "hash" like so :
$existing = array();
$results = getRepository('Piece')->findExistingPiecesByHashes($hashes);
for ($i = 0, $c = count($results); $i < $c; ++$i) {
$existing[] = $results[$i]['hash'];
}
public function findExistingPiecesByHashes(array $hashes)
{
$sql = 'SELECT p.`ref`, p.`labelCanonical`, p.`hash` '.
'FROM `rtd_piece` p '.
'WHERE (p.`hash`) IN (?)';
$rsm = new ResultSetMapping;
$rsm->addScalarResult('ref', 'ref');
$rsm->addScalarResult('hash', 'hash');
$rsm->addScalarResult('labelCanonical', 'labelCanonical');
$query = $this->getEntityManager()
->createNativeQuery($sql, $rsm)
->setParameters(array($hashes));
return $query->getScalarResult();
}
The hash is automatically updated in the model like so :
// Entities/Piece.class.php
private function _updateHash()
{
$this->hash = md5($this->ref.'|'.$this->labelCanonical);
}
My hash field has no FULLTEXT index because I use the InnoDB engine and MySQL version 5.5, and from what I've read InnoDB only supports FULLTEXT indexes since MySQL 5.6.
I don't have the feel to update MySQL right now, too many databases and websites runs on it, it would be disastrous if the update goes wrong.
BUT, even without indexing the field, the performance gain is incredible :
0.00024199485778809 (0.242) : Initialized
0.00028181076049805 (0.282) : Start doProcess
0.0003199577331543 (0.320) : Read and parse lines
0.088779926300049 (88.780) : Check existence in database
0.8656108379364 (865.611) : Process parsed data
0.94273900985718 (942.739) : Finilize
1.3771109580994 (1,377.111) : End of process
1.3795168399811 (1,379.517) : End doProcess
1.3795938491821 (1,379.594) : Return result
And this is for a batch of 1000 with 650k records on the table.
Before this optimization, it took 6.7s for a check of 100 records, so it's around 9 times faster !
At this speed I should be able to import all the data in 1h30-2h.
Thanks you very much for your help.
First, let me suggest that you write this using row constructors:
SELECT p.`ref`, p.`labelCanonical`
FROM `rtd_piece` p
WHERE (p.`ref`, p.`labelCanonical`) IN ( ('6569GX', 'fsc-principal'),
('6569GY', 'fsc-principal'),
. . .
);
This will not affect performance, but it is easier to read. Then, you need an index, either rtd_piece(ref, labelCanonical) or rtd_piece(labelCanonical, ref).

Zend Framework - join query

I build a function
public function getBannedByLogin($commentId)
{
$sql = $this->getDbAdapter()->select()
->from(array('comments' => 'comments'), array())
->join(array('users' => 'qengine_users'),
'comments.bannedBy = users.userId',
array())
->where('commentId = ?', $commentId)
;
$row = $this->fetchRow($sql);
return $row['login'];
}
And there are problems, that does'nt work! :D
Let's I explain you. Column 'bannedBy' from comments returns id of user, who give a ban. I need to join this with table users to load a login field. Where i have mistakes?
I assume the code works in the sense of not throwing an exception. If so, your code is OK, you just specifically tell Zend_Db not to select any columns.
public function getBannedByLogin($commentId)
{
$sql = $this->getDbAdapter()->select()
->from(array('comments' => 'comments'))
->join(array('users' => 'qengine_users'),
'comments.bannedBy = users.userId')
->where('commentId = ?', $commentId)
;
$row = $this->fetchRow($sql);
return $row['login'];
}
The last argument to from() and join() functions is an array of columns you wish to select. If you pass in an empty array, no columns are selected. No argument = select everything. You can, of course, specify only the columns you need too.

Statements works in SQL, but not in PDO

I'm converting from SQL to PDO, and everything has gone well until this statement.
My SQL does what it should and does NOT output the message "This user has no private images". But for some reason, when changing to PDO, the same message is shown when it should not be.
Any ideas?
Original SQL:
$result = mysql_query("SELECT * FROM tbl_private_photos WHERE profile = $usernum AND photo_deleted != 'Yes' LIMIT 1");
if (mysql_num_rows($result)!==1) { die("This user has no private images");}
My PDO:
$sql = "SELECT * FROM tbl_private_photos WHERE profile = :usernum AND photo_deleted != 'Yes' LIMIT 1";
$q = $conn->prepare($sql); // the default way of PDO to manage errors is quite the same as `or die()` so no need for that
$q->bindValue(':usernum',$usernum,PDO::PARAM_INT);
$q->execute();
if($r = $q->fetch(PDO::FETCH_ASSOC)!==1)
{
die("This user has no private images");
}
PDO::fetch() returns in this case an array or false. You don't want to compare the fetch result explicitly to the integer 1 then assign it to a variable--it will always be true because a 1 !== array() is always true, and 1 !== false is always true.
Instead you should see if your result set is empty or false.
Try this instead:
$r = $q->fetch(PDO::FETCH_ASSOC);
if(empty($r))
{
die("This user has no private images");
}

Symfony2 - how can I get Entity Object based on custom query?

this is my custome query:
$query = $em->createQuery("SELECT max(d.id) FROM MyBundle:DBTableEntity d ");
$max_incoming_id = $query->execute();
I want it to return the Entity Object, just like this one below:
$EntityObj = $resource->getRepository("MyBundle:DBTableEntity")->findAll();
Any idea how to do that?
Something like this should work
$EntityObjects = $resource->getRepository('MyBundle:DBTableEntity')
->createQuery()
->orderBy('id', 'DESC')
->getResult();
$EntityObject = array_pop($EntityObjects);
Try This
$query = sprintf("SELECT s FROM BundleName:EntityClass s where s.field1 = %d and s.field2=%d", $field1, $field2);
$em = $this->getDoctrine()->getEntityManager();
$queryObj = $em->createQuery($query);
$entities = $queryObj->execute();
With excute you receive a array of results (entities).
so you can do $entities = $query.excute();
return $entities[0]; //this is if you have one result;
array_pop($entities) will not work this gives a error in symfony 2.6
I think you will like this style:
$EntityObj = $resource->getRepository("MyBundle:DBTableEntity")
->findOneBy(array(), array('id' => 'DESC'));
:)

codeigniter active records join with using?

I am wanting to write this query:
SELECT u.userId, uil.interestId, url.regionId FROM users u
JOIN userInterestLink uil USING (userId)
JOIN userRegionLink url USING (userId)
JOIN dealInterestLink dil USING (interestId)
JOIN dealRegionLink drl USING (regionId, dealId)
WHERE dealId = 1
using the code igniter active records class, however I have no ideda how to do the JOIN ... USING
There is no built-in support for JOIN ... USING in the active record class. Your best bet would probably change the join() function to be like this (the file is system/database/DB_active_rec.php in case you don't know)
public function join($table, $cond, $type = '')
{
if ($type != '')
{
$type = strtoupper(trim($type));
if ( ! in_array($type, array('LEFT', 'RIGHT', 'OUTER', 'INNER', 'LEFT OUTER', 'RIGHT OUTER')))
{
$type = '';
}
else
{
$type .= ' ';
}
}
// Extract any aliases that might exist. We use this information
// in the _protect_identifiers to know whether to add a table prefix
$this->_track_aliases($table);
// Strip apart the condition and protect the identifiers
if (preg_match('/([\w\.]+)([\W\s]+)(.+)/', $cond, $match))
{
$match[1] = $this->_protect_identifiers($match[1]);
$match[3] = $this->_protect_identifiers($match[3]);
$cond = $match[1].$match[2].$match[3];
}
// Assemble the JOIN statement
$type.'JOIN '.$this->_protect_identifiers($table, TRUE, NULL, FALSE);
$using_match = preg_match('/using[ (]/i', $cond);
if ($using_match)
{
$join .= $cond;
}
else
{
$join .= ' ON '.$cond;
}
$this->ar_join[] = $join;
if ($this->ar_caching === TRUE)
{
$this->ar_cache_join[] = $join;
$this->ar_cache_exists[] = 'join';
}
return $this;
}
So then, you can simply use this in your code join('table', 'USING ("something")')
Although, you might want to extend the class instead of modifying it so that you won't need to do same thing over and over again when you upgrade your CI.
Have a look at this article or this one (or search google) if you want to do so instead.
Or if you don't want to go all those troubles, you can write a simple helper function that can do the same thing.
<?php if ( ! defined('BASEPATH')) exit('No direct script access allowed');
function join_using($table, $key)
{
$CI = get_instance();
$join = 'JOIN '. $table .' USING (`'. $key .'`)';
return $CI->db->ar_join[] = $join;
}
Later on, just load the helper and call the function like this join_using('table', 'key'). It will then produce the same result as you would with the original join() except this one will give you USING instead of ON condition.
For example:
// $something1 and $something2 will produce the same result.
$something1 = $this->db->join('join_table', 'join_table.id = table.id')->get('table')->result();
print_r($something1);
join_using('join_table', 'id');
$something2 = $this->db->get('table')->result();
print_r($something2);