How to catch fails in pdo batch insert - mysql

I have this code to insert data from a CSV file to the database. Since, the CSV might have thousands of records, I am trying to implement a batch insert as follows.
$this->_connection->beginTransaction();
$sql = "INSERT INTO dbtable (col1,col2) VALUES (:value1,:value2)";
$stmt = $this->_connection->prepare($sql);
foreach ($requestArray['csv'] as $data) {
$stmt->bindParam(':value1', $data['csvCol1']);
$stmt->bindParam(':value2', $data['csvCol2']);
$stmt->execute();
}
$this->_connection->commit();
The variable $requestArray['csv'], holds all the record of the CSV post request. This code seems to be working as it should. Though, I am trying to improve it, because I want to let the user know how many records failed to insert on the database and if it is possible to show which records failed. Assume a duplicate key, or invalid data or generally any error that might come from this procedure.

PDOStatement::execute() will return false on failure:
$this->_connection->beginTransaction();
$sql = "INSERT INTO dbtable (col1,col2) VALUES (:value1,:value2)";
$stmt = $this->_connection->prepare($sql);
foreach ($requestArray['csv'] as $data) {
$stmt->bindParam(':value1', $data['csvCol1']);
$stmt->bindParam(':value2', $data['csvCol2']);
if (!$stmt->execute()) {
// error processing goes here.
}
}
$this->_connection->commit();

Related

Delete from two mysql tables at once using join codeigniter

I am trying to delete from two mysql tables at once using this function.
public function deleteBusiness($id)
{
$this->db->from("business");
$this->db->join("opening_hours", "business");
$this->db->where("business", $id);
return $this->db->delete(array("business","opening_hours"));
}
It appears to be working correctly, as the correct data is being deleted from both tables, but when I call it with the following...
if($this->businesses_model->deleteBusiness($id)){
$this->session->set_flashdata('flash_message', 'Business Deleted');
} else {
$this->session->set_flashdata('flash_message', 'There was a problem deleting that business');
}
the failed error message is being displayed.
Replace Above code with Following Code let me know if does not work
public function deleteBusiness($id)
{
$this->db->from("business");
$this->db->join("opening_hours", "business");
$this->db->where("business", $id);
$this->db->delete(array("business","opening_hours"));
return $this->db->affected_rows();
}
if($this->businesses_model->deleteBusiness($id) > 0){
$this->session->set_flashdata('flash_message', 'Business Deleted');
} else {
$this->session->set_flashdata('flash_message', 'There was a problem deleting that business');
}
let me know if it does not work
It looks like you are trying to Delete a row from business table and all the rows in the opening_hours related to that business.
Unfortunately your code will not work as CodeIgniter ignores joins when doing Active Record deletes.
Their Documentation does says that you can pass an array to delete method and it will delete from multiple rows
$tables = array('table1', 'table2', 'table3');
$this->db->where('id', '5');
$this->db->delete($tables);
However, please see that it will delete rows from 3 tables where id = 5. Same way in your case it will delete from business and opening_hours where id is say 2. Obviously this is not what you want.
Only option is to use either 2 queries like below or use a Raw SQL query.
$this->db->from("business");
$this->db->where("id", $id);
$this->db->delete('business');
$this->db->from("opening_hours");
$this->db->where("business_id", $id);
$this->db->delete('opening_hours');

Laravel parameter binding causing occasional MySQL general error

I have an array of integers that need to be inserted as a batch of rows. Each row needs some other data.
$ids = [1,2]
$thing = 1
$now = Carbon::now(); // This is just a timestamp.
$binding_values = trim(str_repeat("({$thing}, ?, '{$now}'),", count($ids)), ',');
The string $binding_values looks like this:
"(1, ?, '2019-01-01 00:00:00'), (1, ?, '2019-01-01 00:00:00')"
Then I prepare my query string and bind the parameters to it. The IGNORE is used because I have a composite unique index on the table. It doesn't seem relevant to the problem though so I've left the details out.
DB::insert("
INSERT IGNORE INTO table (thing, id, created_at)
VALUES {$binding_values}
", $ids);
This works almost all the time but every now and then I get an error SQLSTATE[HY000]: General error: 2031.
Is the way I'm doing this parameter binding some kind of anti-pattern with Laravel? What might the source of this error be?
Because there is no risk of injection in this method and there is no chance that this method would be extended to a use case with a risk of injection, I've modified it to bake in all the parameters and skip parameter binding. I haven't seen any errors so far.
I would still like to know what might cause this behaviour so I can manage it better in the future. I'd be grateful for any insight.
I don't see a big issue with your query other than baking parameters into the query, which is vulnerable to SQL injection.
From what I can see, your problem is that you need INSERT ÌGNORE INTO which is not supported out of the box by Laravel. Have you thought about using a third-party package like staudenmeir/laravel-upsert?
An alternative could be to wrap your query in a transaction and select existing entries first, giving you the chance to not insert them a second time:
$ids = [1, 2];
$thing = 1;
$time = now();
DB::transaction(function () use ($ids, $thing, $time) {
$existing = DB::table('some_table')->whereIn('id', $ids)->pluck('id')->toArray();
$toInsert = array_diff($ids, $existing);
$dataToInsert = array_map(function ($id) use ($thing, $time) {
return [
'id' => $id,
'thing' => $thing,
'created_at' => $time
];
}, $toInsert);
DB::table('some_table')->insert($dataToInsert);
});
This way you will only insert what is not present yet, but you will also stay within the framework capabilities of the query builder. Only downside is that it will be slightly slower due to a second roundtrip to the database.

How do I remove a row from mysql immediately after calling it?

Basically, I have a mysql database where every time I get a row, I want to delete it from the database (after reading its information).
I know I could do something like
$result = mysql_query(" SELECT <some row> FROM table ");
$row = mysql_fetch_array($result);
$id = $row["id"];
mysqli_query(" DELETE FROM table WHERE id=$id");
but now it seems I have two queries going on. Is there a command to tell mysql that I want the row deleted as soon as it gives me the information? I imagine that'd save time and resources.
In my head, it looks like
$result = mysql_query(" SELECT <some row> FROM table THEN DELETE ");
EDIT additional information: I wish to use the SELECTed information after deleting the row. To put it simply, I only want one instance of the information to exist at any give time; it would be as if I were only "moving" a physical copy of the information, so that when it is put on a device/what have you, it is no longer in the table since there is only one copy.
Sorry if my understanding of mysql is rough -- I'm pretty new to it :P
I don't know why you need it but you can use a stored procedure for that:
DELIMITER $$
CREATE PROCEDURE select_and_delete(IN aid INT)
BEGIN
SELECT * FROM table1 WHERE id = aid;
DELETE FROM table1 WHERE id = aid;
END$$
DELIMITER ;
Here is SQLFiddle demo.
mysql_* extension is deprecated, therefore use prepared statements and mysqli or PDO.
Your php code using PDO might look like
$id = 1;
try {
$db = new PDO('mysql:host=localhost;dbname=test;charset=UTF8', 'user', 'userpwd');
$db->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
$db->setAttribute(PDO::ATTR_EMULATE_PREPARES, false);
$query = $db->prepare("CALL select_and_delete(?)");
$query->execute(array($id));
$result = $query->fetchAll(PDO::FETCH_ASSOC);
} catch (PDOException $e) {
echo "Exception: " .$e->getMessage();
$result = false;
}
$query = null;
$db = null;
//do whatever you need to do with your resultset
var_dump($result);
Following a rather simplified table structure (with the only column id) presented in SQL Fiddle example if you call it with $id=1 you'll you'll get in $result:
array(1) {
[0]=>
array(1) {
["id"]=>
int(1)
}
}
You'll need to add a timestamp field (with default as CURRENT_TIMESTAMP) to be able to tell when the row was added.
Then you can run the following MySQL query.
DELETE FROM `table` WHERE `timestamp` > DATE_SUB(NOW(), INTERVAL 0 SECOND);
You will need to run this as a cron job though. AFAIK it can't be done in MySQL alone.

Get ID of multiple inserts in Codeigniter

I'm in the process of learning CI for myself and this came up. If I run multiple INSERTs, whether as transaction or not, is it possible to get the insert ID of each (in the proper order) instead of running $this->db->insert_id() one at a time for each INSERT?
For example
-- Get ID of each
INSERT INTO table VALUES (insert1), (insert2), (insert3)
-- What if they're called separately
INSERT INTO table VALUES (insert1)
INSERT INTO table VALUES (insert2)
INSERT INTO table VALUES (insert3)
Is it still possible to get the ID as an array from all this?
I think you really must insert it between each inserts as far as i know, just store it in an array.
var $insert_id = array();
public function insert()
{
INSERT INTO table VALUES (insert1)
$this->insert_id =$this->db->insert_id();
INSERT INTO table VALUES (insert2)
$this->insert_id =$this->db->insert_id();
INSERT INTO table VALUES (insert3)
$this->insert_id =$this->db->insert_id();
}
everytime a database record is inserted you can still use insert_id(); after each insert then store it to an array. then you coudl make a function to return it.
public function get_inserted_keys()
{
return $insert_id;
}
It is simple
Controller
$ids = array();
$data[0] = array(blah,blah);
$data[1] = array(blah,blah);
$data[2] = array(blah,blah);
for($i=0;$i<3;$i++)
{
$ids[] = $this->my_model->insert($data[$i]);
}
echo '<pre>';
print_r($ids);
Model
public function insert($data)
{
$this->db->insert('table_name',$data);
return $this->db->insert_id()
}

How to insert array into mysql using PDO and bindParam?

I'm using the following code. The code works, but I want to change it so that it uses bindparam
try {
$dbh = new PDO("mysql:host=$hostname;dbname=$dbname", $username, $password);
$stqid=array();
for ($i=0; $i<$array_count; $i++){
$stqid[$i][0]=$lastInsertValue;
$stqid[$i][1]=$qid[$i][0];
$stqid[$i][2]=$qid[$i][1];
}
$values = array();
foreach ($stqid as $rowValues) {
foreach ($rowValues as $key => $rowValue) {
$rowValues[$key] = $rowValues[$key];
}
$values[] = "(" . implode(', ', $rowValues) . ")";
}
$count = $dbh->exec("INSERT INTO qresults(instance, qid, result) VALUES ".implode (', ', $values));
$dbh = null;
}
catch(PDOException $e){
echo $e->getMessage();
}
I replaced the following
$count = $dbh->exec("INSERT INTO qresults(instance, qid, result) VALUES ".implode (', ', $values));
with
$sql = "INSERT INTO qresults (instance, qid, result) VALUES (:an_array)";
$stmt = $dbh->prepare($sql);
$stmt->bindParam(':an_array', implode(',', $values),PDO::PARAM_STR);
$stmt->execute();
but the insert doesn't work anymore (I didn't get any error messages though).
QUESTION: What am I doing wrong? How can I rewrite the code to use bindParam?
You're trying to create a statement and bind a param.
Statement are great because it potentially nullify any kind of SQL injection. And it does it by removing the concept of a query being only seen as a string. The SQL query is seen as a string with a parameter list and an the associated data as binded variables.
So the query is not only text, but text + data.
I mean:
This simple query:
SELECT * FROM A WHERE val="$param"
It is not safe because the query is only viewed as a string. And if $param is not checked, it is a SQLi hole.
But when create a statement, your query becomes:
SELECT * FROM A WHERE val=:param
Then you use bindparam to specify the value a :param. Which mean the value is not appended to the query string, but the query is already parsed and the data is provided.
In your case, you bind to the param :array an imploded array (I assume "data1", "data2", etc..). Which is only one parameter with the value as a string ( "data1, data2, data3..." ), so it will only result in one insert and not multiple insertions.
You can change your statement generation by generating a query with enough parameters to handle your array
$sql = "INSERT INTO qresults (instance, qid, result) VALUES ( :val0, :val1, :val2, ...)";
Then loop on your array and call the bindparam method for each parameters.
$count = 0;
foreach($values as $val)
{
$stmt->bindParam(":val$count", $val,PDO::PARAM_STR);
$count++;
}
This will work.
Edit: This solution show how it works for a one dimensional array, but can be easily extended to your problem by tweaking the statement query generation and modify the bindparam loop.
Your statement should looks like:
$sql = "INSERT INTO qresults (instance, qid, result) VALUES (:val0, :val1, :val2) , (:val3, :val4, :val5), ...";
You just have to count the number of element in your base array.