Having a spot of bother trying to grab some data out of my database.
I have the following model:
function GetRestaurants($options = array())
{
// Qualification
if(isset($options['restaurantID']))
$this->db->where('restaurantID', $options['restaurantID']);
if(isset($options['restaurantRegionID']))
$this->db->where('restaurantRegionID', $options['restaurantRegionID']);
if(isset($options['restaurantName']))
$this->db->where('restaurantName', $options['restaurantName']);
if(isset($options['restaurantAddress1']))
$this->db->where('restaurantAddress1', $options['restaurantAddress1']);
if(isset($options['restaurantAddress2']))
$this->db->where('restaurantAddress2', $options['restaurantAddress2']);
if(isset($options['restaurantSuburb']))
$this->db->where('restaurantSuburb', $options['restaurantSuburb']);
if(isset($options['restaurantCity']))
$this->db->where('restaurantCity', $options['restaurantCity']);
if(isset($options['restaurantInformation']))
$this->db->where('restaurantInformation', $options['restaurantInformation']);
// limit / offset
if(isset($options['limit']) && isset($options['offset']))
$this->db->limit($options['limit'], $options['offset']);
else if(isset($options['limit']))
$this->db->limit($options['limit']);
// sort
if(isset($options['sortBy']) && isset($options['sortDirection']))
$this->db->order_by($options['sortBy'], $options['sortDirection']);
$query = $this->db->get("tblRestaurants");
if(isset($options['count'])) return $query->num_rows();
if(isset($options['restaurantID']))
return $query->row(0);
if(isset($options['limit']) && $options['limit'] == '1')
return $query->row(0);
return $query->result();
}
Now the following code works fine:
$this->load->model('Restaurants_model');
$data['restaurant'] = $this->Restaurants_model->GetRestaurants(array(
'restaurantName' => 'shed 5',
'limit' => '1'
));
However the following does not work:
$this->load->model('Restaurants_model');
$data['restaurant'] = $this->Restaurants_model->GetRestaurants(array(
'restaurantName' => str_replace('-', ' ', $this->uri->segment(2)),
'limit' => '1'
));
Even though the result of
str_replace('-', ' ', $this->uri->segment(2))
is in this instance: ‘shed 5’.
I have compared var_dumps of the output of the str_replace and the string itself and determined them to be identical. So why does the straight string return a result yet the string generated from the uri segment doesn’t? Some kind of encoding issue? My database holds data in ‘utf8_general_ci’.
Thanks for any suggestions!
$restaurant=str_replace('-', ' ', $this->uri->segment(2));
get value outside array and try array_push
$data['restaurant'] = $this->Restaurants_model->GetRestaurants(array(
'restaurantName' => $restaurant,
'limit' => '1'
));
Related
I have a .csv file with more than 690 000 rows.
I found a solution to import data that works very well but it's a little bit slow... (around 100 records every 3 seconds = 63 hours !!).
How can I improve my code to make it faster ?
I do the import via a console command.
Also, I would like to import only prescribers that aren't already in database (to save time). To complicate things, no field is really unique (except for id).
Two prescribers can have the same lastname, firstname, live in the same city and have the same RPPS and professional codes. But, it's the combination of these 6 fields which makes them unique !
That's why I check on every field before create a new one.
<?php
namespace AppBundle\Command;
use Symfony\Bundle\FrameworkBundle\Command\ContainerAwareCommand;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;
use Symfony\Component\Console\Helper\ProgressBar;
use AppBundle\Entity\Prescriber;
class PrescribersImportCommand extends ContainerAwareCommand
{
protected function configure()
{
$this
// the name of the command (the part after "bin/console")
->setName('import:prescribers')
->setDescription('Import prescribers from .csv file')
;
}
protected function execute(InputInterface $input, OutputInterface $output)
{
// Show when the script is launched
$now = new \DateTime();
$output->writeln('<comment>Start : ' . $now->format('d-m-Y G:i:s') . ' ---</comment>');
// Import CSV on DB via Doctrine ORM
$this->import($input, $output);
// Show when the script is over
$now = new \DateTime();
$output->writeln('<comment>End : ' . $now->format('d-m-Y G:i:s') . ' ---</comment>');
}
protected function import(InputInterface $input, OutputInterface $output)
{
$em = $this->getContainer()->get('doctrine')->getManager();
// Turning off doctrine default logs queries for saving memory
$em->getConnection()->getConfiguration()->setSQLLogger(null);
// Get php array of data from CSV
$data = $this->getData();
// Start progress
$size = count($data);
$progress = new ProgressBar($output, $size);
$progress->start();
// Processing on each row of data
$batchSize = 100; # frequency for persisting the data
$i = 1; # current index of records
foreach($data as $row) {
$p = $em->getRepository('AppBundle:Prescriber')->findOneBy(array(
'rpps' => $row['rpps'],
'lastname' => $row['nom'],
'firstname' => $row['prenom'],
'profCode' => $row['code_prof'],
'postalCode' => $row['code_postal'],
'city' => $row['ville'],
));
# If the prescriber doest not exist we create one
if(!is_object($p)){
$p = new Prescriber();
$p->setRpps($row['rpps']);
$p->setLastname($row['nom']);
$p->setFirstname($row['prenom']);
$p->setProfCode($row['code_prof']);
$p->setPostalCode($row['code_postal']);
$p->setCity($row['ville']);
$em->persist($p);
}
# flush each 100 prescribers persisted
if (($i % $batchSize) === 0) {
$em->flush();
$em->clear(); // Detaches all objects from Doctrine!
// Advancing for progress display on console
$progress->advance($batchSize);
$progress->display();
}
$i++;
}
// Flushing and clear data on queue
$em->flush();
$em->clear();
// Ending the progress bar process
$progress->finish();
}
protected function getData()
{
// Getting the CSV from filesystem
$fileName = 'web/docs/prescripteurs.csv';
// Using service for converting CSV to PHP Array
$converter = $this->getContainer()->get('app.csvtoarray_converter');
$data = $converter->convert($fileName);
return $data;
}
}
EDIT
According to #Jake N answer, here is the final code.
It's very very faster ! 10 minutes to import 653 727 / 693 230 rows (39 503 duplicate items!)
1) Add two columns in my table : created_at and updated_at
2) Add a single index of type UNIQUE on every column of my table (except id and dates) to prevent duplicate items with phpMyAdmin.
3) Add ON DUPLICATE KEY UPDATE in my query, to update just the updated_at column.
foreach($data as $row) {
$sql = "INSERT INTO prescripteurs (rpps, nom, prenom, code_prof, code_postal, ville)
VALUES(:rpps, :nom, :prenom, :codeprof, :cp, :ville)
ON DUPLICATE KEY UPDATE updated_at = NOW()";
$stmt = $em->getConnection()->prepare($sql);
$r = $stmt->execute(array(
'rpps' => $row['rpps'],
'nom' => $row['nom'],
'prenom' => $row['prenom'],
'codeprof' => $row['code_prof'],
'cp' => $row['code_postal'],
'ville' => $row['ville'],
));
if (!$r) {
$progress->clear();
$output->writeln('<comment>An error occured.</comment>');
$progress->display();
} elseif (($i % $batchSize) === 0) {
$progress->advance($batchSize);
$progress->display();
}
$i++;
}
// Ending the progress bar process
$progress->finish();
1. Don't use Doctrine
Try to not use Doctrine if you can, it eats memory and as you have found is slow. Try and use just raw SQL for the import with simple INSERT statements:
$sql = <<<SQL
INSERT INTO `category` (`label`, `code`, `is_hidden`) VALUES ('Hello', 'World', '1');
SQL;
$stmt = $this->getDoctrine()->getManager()->getConnection()->prepare($sql);
$stmt->execute();
Or you can prepare the statement with values:
$sql = <<<SQL
INSERT INTO `category` (`label`, `code`, `is_hidden`) VALUES (:label, :code, :hidden);
SQL;
$stmt = $this->getDoctrine()->getManager()->getConnection()->prepare($sql);
$stmt->execute(['label' => 'Hello', 'code' => 'World', 'hidden' => 1);
Untested code, but it should get you started as this is how I have done it before.
2. Index
Also, for your checks, have you got an index on all those fields? So that the lookup is as quick as possible.
I'd like to translate a perl web site in several languages. I search for and tried many ideas, but I think the best one is to save all translations inside the mySQL database. But I get a problem...
When a sentence extracted from the database contains a variable (scalar), it prints on the web page as a scalar:
You have $number new messages.
Is there a proper way to reassign $number its actual value ?
Thank you for your help !
You can use printf format strings in your database and pass in values to that.
printf allows you to specify the position of the argument so only have know what position with the list of parameters "$number" is.
For example
#!/usr/bin/perl
use strict;
my $Details = {
'Name' => 'Mr Satch',
'Age' => '31',
'LocationEn' => 'England',
'LocationFr' => 'Angleterre',
'NewMessages' => 20,
'OldMessages' => 120,
};
my $English = q(
Hello %1$s, I see you are %2$s years old and from %3$s
How are you today?
You have %5$i new messages and %6$i old messages
Have a nice day
);
my $French = q{
Bonjour %1$s, je vous vois d'%4$s et âgés de %2$s ans.
Comment allez-vous aujourd'hui?
Vous avez %5$i nouveaux messages et %6$i anciens messages.
Bonne journée.
};
printf($English, #$Details{qw/Name Age LocationEn LocationFr NewMessages OldMessages/});
printf($French, #$Details{qw/Name Age LocationEn LocationFr NewMessages OldMessages/});
This would be a nightmare to maintain so an alternative might be to include an argument list in the database:
#!/usr/bin/perl
use strict;
sub fetch_row {
return {
'Format' => 'You have %i new messages and %i old messages' . "\n",
'Arguments' => 'NewMessages OldMessages',
}
}
sub PrintMessage {
my ($info,$row) = #_;
printf($row->{'Format'},#$info{split(/ +/,$row->{'Arguments'})});
}
my $Details = {
'Name' => 'Mr Satch',
'Age' => '31',
'LocationEn' => 'England',
'LocationFr' => 'Angleterre',
'NewMessages' => 20,
'OldMessages' => 120,
};
my $row = fetch_row();
PrintMessage($Details,$row)
I'm trying to query a page based on either a category ID or sub category name.
The variable $cat will either have an integer or varchar grabbed from my database.
I've been using cakephp 1.3 with a sql find all articles with a category of $cat OR sub-category LIKE $cat
It works great but a problem arises when $cat has a space between words, "google forms".
I've looked through this site and tried a number of methods with no luck. Appreciate any advice.
Here's my controller routines:
$cat = Sanitize::escape($cat);
$cat = trim($cat);
$title_a = str_replace($cat, "%".$cat."%", $cat);
$a_t = str_replace('"', $title_a, $title_a);
//var_dump($cat);
if(!empty($cat))
{
$sqlConditions = array('OR'=>array('Article.categories LIKE' => $a_t, 'Article.event_category_id' => $cat));
$sqlParams = array('conditions'=>$sqlConditions);
$catdata=$this->Article->find('all',$sqlParams);
return $catdata;
}
I've tried many different alternatives:
RLIKE instead of LIKE
Different query using MATCH
$sqlConditions = array(
'OR' => array(
'MATCH(Article.categories AGAINST(? IN BOOLEAN MODE)' => $cat,
'MATCH(Article.event_category_id) AGAINST(? IN BOOLEAN MODE)' => $cat
)
);
$sqlConditions = array('OR'=>array('Article.categories LIKE' => "%".$cat."%", 'Article.event_category_id' => $cat));
I think a decent solution would be to remove all of the spaces and make the characters of $cat lower case.
$likeCat = strtolower(str_replace(' ', '', trim($cat)));
$sqlConditions = array(
'OR'=> array(
'LOWER(REPLACE(Article.categories, ' ', ''))' => $likeCat,
'Article.event_category_id' => $cat
)
);
EDIT : I found the solution by replacing the SSP class by a customized SSP class I've found here : https://github.com/emran/ssp
I don't know if it's an understandable title, but here is my problem :
I have a DB-table (called projects) that needs to be inserted in a datatable. I've no problem to make a call using ServerSide and get results in the datatable.
But, for each project (each row), there is a project creator (column creator_id in the projects DB-table). What I need to do is to make a request to the creators DB-table in order to get the firstname/lastname of the creator, each time I get a row from the projects DB-table. Is that make sense?
Here is the code I use :
$table = 'projects';
$primaryKey = 'project_id';
$columns = array(
array(
'db' => 'project_id',
'dt' => 'DT_RowId',
'formatter' => function( $d, $row ) {
return $d;
}
),
array( 'db' => 'creator_id',
'dt' => 'creator',
'formatter' => function( $d, $row ) {
// Here I need to make a call to the creators DB-table and return the $creator_name value
return $creator_name;
}
)
);
// SQL server connection information
$sql_details = array(
'user' => '',
'pass' => '',
'db' => '',
'host' => ''
);
require(BASE_DIR.'/lib/dataTables/ssp.class.php');
$result = SSP::simple($_GET, $sql_details, $table, $primaryKey, $columns);
You can use formatter property to make a separate SQL query but it will affect performance greatly by increasing script response time.
Instead you can use the following trick when using ssp.class.php with sub-queries instead of table name to use GROUP BY, JOIN, etc.
<?php
$table = <<<EOT
(
SELECT projects.project_id, creators.creator_id, creators.creator_name
FROM projects
LEFT JOIN creators ON projects.creator_id=creators.creator_id
) t
EOT;
$primaryKey = 'project_id';
$columns = array(
array(
'db' => 'project_id',
'dt' => 'DT_RowId'
),
array(
'db' => 'creator_id',
'dt' => 'creator'
),
array(
'db' => 'creator_name',
'dt' => 'creator_name'
)
);
// SQL server connection information
$sql_details = array(
'user' => '',
'pass' => '',
'db' => '',
'host' => ''
);
require(BASE_DIR.'/lib/dataTables/ssp.class.php');
$result = SSP::simple($_GET, $sql_details, $table, $primaryKey, $columns);
echo json_encode($result);
To use that trick, you also need to edit ssp.class.php and replace all instances of FROM `$table` with FROM $table to remove backticks.
Alternatively, there is github.com/emran/ssp repository for library that extends ssp.class.php allowing GROUP BY, JOIN, aliases.
LINKS
See jQuery DataTables: Using WHERE, JOIN and GROUP BY with ssp.class.php for more information.
I have the following query that selects some records from the database:
$select_person = $this->select()
->setIntegrityCheck(false)
->from(array('a' => 'tableA'),
array(new Zend_Db_Expr('SQL_CALC_FOUND_ROWS a.id'),
'a.cid',
'a.email',
'final' => new Zend_Db_Expr( "concat( '<div
style=\"color:#1569C7; font-weight:bold\">',
a.head , ' ' , a.tail, '</div>')" ),
'a.red_flag'
)
)
->joinLeft(array('b' => 'tableb'), ... blah blah)
->where('blah blah')
->order('a.head ASC')
I want to modify the above query so that it selects a different value for 'final' depending on the value of
a.red_flag.
which can have values - true or false.
I understand I can use the CASE statement of mysql - eg something like the following:
'final' => new Zend_Db_Expr("CASE a.red_flag WHEN 'true' THEN '$concatstr1'
ELSE '$concatstr2' END")
The value of $concatstr1 = "concat( '<div style=\"color:red; font-weight:bold\">', a.head , ' ' , a.tail, '</div>')" ;
The value of $concatstr2 = "concat( '<div style=\"color:blue; font-weight:bold\">', a.head , ' ' , a.tail, '</div>')" ;
However, it throws an error saying
Message: SQLSTATE[42000]: Syntax error or access violation: 1064
You have an error in your SQL syntax; check the manual that
corresponds to your MySQL server version for the right syntax to use
near 'div
style="color:red; font-weight:bold">',
a.head , ' ' , ' at line 1
How can I make this query work?
Any help is greatly appreciated.
Thanks
Personnaly, I don't like to get HTML as an answer from the DB. It gets confusing and harder to debug and change afterwards. Furthermore, you might get some errors due to the confusion with the ' and " and all the reserved characters in MySQL (<, >, ;, ...) I would suggest that you try this:
'final' => new Zend_Db_Expr("CASE a.red_flag WHEN 'true' THEN 1
ELSE 0 END")
Then do a check on the value of a.red_flag;
if($this->final) {
$output .= '<div style=\"color:red; font-weight:bold\">';
} else {
$output .= '<div style=\"color:blue; font-weight:bold\">';
}
$output .= $this->head.' '.$this->tail;
$output .= '</div>';
If the query still doesn't work. Try
echo $select->__toString; exit();
and check the query. Try the output that you got with the __toString on your database and check if it works. It's easier to fix it that way. You could also show the query string here and it'll be easier to debug.
Finally, I found the error in my statement.
The culprit was - I was using quotes in $concatstr1 and $concatstr2 inside the $select_person statement.
The correct query should be formed as follows:
$select_person = $this->select()
->setIntegrityCheck(false)
->from(array('a' => 'tableA'),
array(new Zend_Db_Expr('SQL_CALC_FOUND_ROWS a.id'),
'a.cid',
'a.email',
final' => new Zend_Db_Expr("CASE a.red_flag WHEN 'true' THEN $concatstr1 ELSE $concatstr2 END"),
'a.red_flag'
)
)
->joinLeft(array('b' => 'tableb'), ... blah blah)
->where('blah blah')
->order('a.head ASC');
This is now returning me the appropriate value of 'final' - concatstr1 when red_flag is true otherwise it is returning me concatstr2.