Related
I am attempting to do a bulk insert into MySQL using
INSERT INTO TABLE (a, b, c) VALUES (?, ?, ?), (?, ?, ?)
I have the general log on, and see that this works splendidly for most cases. However, when the table has a BLOB column, it doesn't work as well.
I am trying to insert 20 records.
Without the BLOB, I see all 20 records in the same query in the general log, 20 records inserted in the same query.
WITH the BLOB, I see only 2 records per query in the general log, it takes 10 queries in total.
Is this a problem with MySQL, the JDBC Driver, or am I missing something else. I would prefer to use a BLOB as I have data in protobufs.
Here is an example table...
CREATE TABLE my_table (
id CHAR(36) NOT NULL,
name VARCHAR(256) NOT NULL,
data BLOB NOT NULL,
PRIMARY KEY (id)
);
Then, create your batch inserts in code...
val ps = conn.prepareStatement(
"INSERT INTO my_table(id, name, data) VALUES (?, ?, ?)")
records.grouped(1000).foreach { group =>
group.foreach { r =>
ps.setString(1, UUID.randomUUID.toString)
ps.setString(2, r.name)
ps.setBlob(3, new MariaDbBlob(r.data))
ps.addBatch()
}
ps.executeBatch()
}
If you run this and inspect the general log, you will see...
"2018-10-12T18:37:55.714825Z 4 Query INSERT INTO my_table(id, name, fqdn, data) VALUES ('b4955537-2450-48c4-9953-e27f3a0fc583', '17-apply-test', _binary '
17-apply-test\"AAAA(?2Pending8?????,J$b4955537-2450-48c4-9953-e27f3a0fc583
1:2:3:4:5:6:7:8Rsystem'), ('480e470c-6d85-4bbc-b718-21d9e80ac7f7', '18-apply-test', _binary '
18-apply-test\"AAAA(?2Pending8?????,J$480e470c-6d85-4bbc-b718-21d9e80ac7f7
1:2:3:4:5:6:7:8Rsystem')
2018-10-12T18:37:55.715489Z 4 Query INSERT INTO my_table(id, name, data) VALUES ('7571a651-0e0b-4e78-bff0-1394070735ce', '19-apply-test', _binary '
19-apply-test\"AAAA(?2Pending8?????,J$7571a651-0e0b-4e78-bff0-1394070735ce
1:2:3:4:5:6:7:8Rsystem'), ('f77ebe28-73d2-4f6b-8fd5-284f0ec2c3f0', '20-apply-test', _binary '
20-apply-test\"AAAA(?2Pending8?????,J$f77ebe28-73d2-4f6b-8fd5-284f0ec2c3f0
As you can see, each INSERT INTO only has 2 records in it.
Now, if you remove the data field from the schema and insert and re-run, you will see the following output (for 10 records)...
"2018-10-12T19:04:24.406567Z 4 Query INSERT INTO my_table(id, name) VALUES ('d323d21e-25ac-40d4-8cff-7ad12f83b8c0', '1-apply-test'), ('f20e37f2-35a4-41e9-8458-de405a44f4d9', '2-apply-test'), ('498f4e96-4bf1-4d69-a6cb-f0e61575ebb4', '3-apply-test'), ('8bf7925d-8f01-494f-8f9f-c5b8c742beae', '4-apply-test'), ('5ea663e7-d9bc-4c9f-a9a2-edbedf3e5415', '5-apply-test'), ('48f535c8-44e6-4f10-9af9-1562081538e5', '6-apply-test'), ('fbf2661f-3a23-4317-ab1f-96978b39fffe', '7-apply-test'), ('3d781e25-3f30-48fd-b22b-91f0db8ba401', '8-apply-test'), ('55ffa950-c941-44dc-a233-ebecfd4413cf', '9-apply-test'), ('6edc6e25-6e70-42b9-8473-6ab68d065d44', '10-apply-test')"
All 10 records are in the same query
I tinkered until I found the fix...
val ps = conn.prepareStatement(
"INSERT INTO my_table(id, name, data) VALUES (?, ?, ?)")
records.grouped(1000).foreach { group =>
group.foreach { r =>
ps.setString(1, UUID.randomUUID.toString)
ps.setString(2, r.name)
//ps.setBlob(3, new MariaDbBlob(r.data))
ps.setBytes(r.data)
ps.addBatch()
}
ps.executeBatch()
Using PreparedStatement.setBytes instead of using MariaDbBlob seemed to do the trick
How to update mysql data in bulk ?
How to define something like this :
UPDATE `table`
WHERE `column1` = somevalues
SET `column2` = othervalues
with somevalues like :
VALUES
('160009'),
('160010'),
('160011');
and othervalues :
VALUES
('val1'),
('val2'),
('val3');
maybe it's impossible with mysql ?
a php script ?
The easiest solution in your case is to use ON DUPLICATE KEY UPDATE construction. It works really fast, and does the job in easy way.
INSERT into `table` (id, fruit)
VALUES (1, 'apple'), (2, 'orange'), (3, 'peach')
ON DUPLICATE KEY UPDATE fruit = VALUES(fruit);
or to use CASE construction
UPDATE table
SET column2 = (CASE column1 WHEN 1 THEN 'val1'
WHEN 2 THEN 'val2'
WHEN 3 THEN 'val3'
END)
WHERE column1 IN(1, 2 ,3);
If the "bulk" data you have is dynamic and is coming from PHP (you did tag it, after all), then the query would look something like this:
INSERT INTO `foo` (id, bar)
VALUES
(1, 'pineapple'),
(2, 'asian pear'),
(5, 'peach')
ON DUPLICATE KEY UPDATE bar = VALUES(bar);
and the PHP to generate this from an existing array (assuming the array is of a format like:
$array = (
somevalues_key => othervalues_value
);
) would look something like this (by no means the best (doesn't address escaping or sanitizing the values, for instance), just an quick example):
$pairs = array();
foreach ($array as $key => $value) {
$pairs[] = "($key, '$value')";
}
$query = "INSERT INTO `foo` (id, bar) VALUES " . implode(', ', $pairs) . " ON DUPLICATE KEY UPDATE bar = VALUES(bar)";
You could try an UPDATE with JOIN as below:
UPDATE table
INNER JOIN (
SELECT 1 column1, 2 column2, 10 new_v1, 20 new_v2, 30 new_v3
UNION ALL SELECT 4 column1, 5 column2, 40 new_v1, 50 new_v2, 60 new_v3
) updates
ON table.column1 = updates.column1
AND table.column2 = updates.column2
SET
table.column1 = updates.new_v1,
table.column2 = updates.new_v2,
table.column3 = updates.new_v3;
As long as you can craft the inner SELECT statements from the updates subquery you would get the benefit of running all these updates in a single statement (which should give you some performance boost on InnoDB depending on your table size).
If you are using a drag & drop tableView or collectionView to sort datas in your app, like allowing users to arrange their photos by drag and drop functionality, send a comma seperated list of ordered ids to the backend after user edits finish.
In your backend, explode ids to the an array like
$new_ranks = array();
$supplied_orders = explode(",", $_POST["supplied_new_order"]); //52,11,6,54,2 etc
$start_order = 99999;
foreach ($supplied_orders as $supplied_row_id) {
//your all validations... make sure supplied_row_id belongs to that user or not etc..
$new_ranks[intval($supplied_row_id)] = $start_order--;
}
now, you can update all new ranks like #Farside recommendation 2.
if (count($new_ranks) > 0) {
$case_sqls = array();
foreach ($new_ranks as $id => $rank) {
$case_sqls[] = "WHEN ".intval($id)." THEN ".intval($rank)."";
}
$case_sql = implode(" ", $case_sqls);
$this->db->query("
UPDATE
service_user_medias
SET
rank = (CASE id ".$case_sql." END)
WHERE
id IN(".implode(",", array_keys($new_ranks)).");
");
}
If you have data in array format then try this
and your query is like "UPDATE table WHERE column1 = ? SET column2 = ?"
then set it like below
foreach($data as $key => $value) {
$query->bind_param('ss', $key, $value);
$query->execute();
}
hope it'll work.
Reference from this.
Presently I am inserting a new entry (copying the data from an existing entry) & then updating this new entry. eg:
$age = 30;
INSERT INTO table (category,sex,age) SELECT category,sex,age FROM table WHERE id = $id
<br>$new_id = mysql_insert_id();
<br>UPDATE sample_table SET age = $age WHERE id = $new_id
However, I would like to save a database interaction by inserting the new entry in one go (without having to get mysql_insert_id / updating the new entry). eg:
$age = 30;
INSERT INTO table (category,sex,$age) SELECT category,sex FROM table WHERE id = $id
Is there a way to insert the above by explicitly defining the age field, or is there another MYSQL command I should be using?
You are looking for this:
INSERT INTO table (category, sex, age)
SELECT category, sex, $age
FROM table
WHERE id = $id ;
You can put constants in the select list.
I want to update a table by getting the last inserted id but it is giving no results.
here is the query :
$quer = mysql_query("UPDATE date
SET d_startdate = '$start', d_enddate = '$end'
WHERE d_id = LAST_INSERT_ID() AND d_sid = $id");
d_id is the primary key and d_sid is a foreign key of another table
I have used INSERT as well as UPDATE operation on my same table and its working fine. You can change this query as per your need.
<?php
$con = mysql_connect("localhost","root","") or die("Could not connect");
mysql_selectdb("test", $con);
$query = 'INSERT INTO item (`name`) VALUES ("DELTaaaA")';
$res = mysql_query($query, $con) or die(mysql_error());
echo "<pre>";
print_r($res);
$query = 'UPDATE item set name="DELTaaaA1" WHERE id = LAST_INSERT_ID()';
$res = mysql_query($query, $con) or die(mysql_error());
print_r($res);
?>
It should return 1 1
use code like this before that
SELECT LAST_INSERT_ID() and assign this to a variable, then use that variable in your code
I don't know the normal syntax, but PDO syntax is quiet simple, you can get last inserted id by the function PDO::lastInsertId() use it as $myPDOobject->lastInsertId() . More information here : http://php.net/manual/en/pdo.lastinsertid.php
LAST_INSERT_ID gives you the id of the most recent insert.
Suppose that you added a row which has d_id=10 (d_id set by auto increment) and d_sid=20 and then another one with d_id=11 (again, auto increment) and d_sid=30.
You then want to look for the most recent insert with d_sid=20, but doing this with LAST_INSERT_ID is not possible, since LAST_INSERT_ID has value 11, and no row matches with that d_id and d_sid=20. You have to keep track by yourself of the most recent d_id for each category, most likely when you insert the new rows.
Do the INSERT
SELECT the LAST_INSERT_ID (name it L_ID)
Store the tuple <d_sid, L_ID> somewhere, so you know that for d_sid your most recent INSERT has value L_ID
UPDATE your table where the d_sid is the one you want and d_id is taken from the tuple
As a side note, mysql_* is deprecated, so you should switch to something else like mysqli or PDO.
In my table I have an userID that is auto-incremented. In the same row I have an idHash. Is it possible to generate the idHash (simply an MD5 sum) from it directly with the same INSERT statement so that I don't have to SELECT the id, and then UPDATE the idHash again?
Problem is: I do not know the userID before it is being generated (auto-incremented) by MySQL.
Thanks
Frank
PS: I'm using PHP.
PPS: This question is all about a SINGLE INSERT. I know that I can use PHP or other languages to manually select the data and then update it.
I don't believe you can do it within a single INSERT statement.
What you probably could do is use an INSERT trigger, that both determines the new ID, hashes it, and then updates the record.
One solution I can recommend is using the last insert ID instead of re-querying the table. Here is a simplified example:
<?php
$mysqli = new mysqli("localhost", "my_user", "my_password", "world");
/* check connection */
if (mysqli_connect_errno()) {
printf("Connect failed: %s\n", mysqli_connect_error());
exit();
}
$query = "INSERT INTO users VALUES (....)";
$mysqli->query($query);
$newUserID = $mysqli->insert_id;
$query = "UPDATE users SET idHash = MD5(userID) WHERE userID = $newUserID";
$mysqli->query($query);
/* close connection */
$mysqli->close();
?>
AFAIK there's no "secure" way for doing this in the same query if you're using auto_increment.
However, if rows are never deleted in your table, you can use this little trick :
insert into mytable (col1, col2, col3, idhash)
values ('', '', '', md5(select max(id) from mytable))
I don't understand why you need to hash the id though, why not use the id directly ?
This seems to work for me:
CREATE TABLE tbl (id INT PRIMARY KEY AUTO_INCREMENT, idHash TEXT);
INSERT INTO tbl (idHash) VALUES (MD5(LAST_INSERT_ID() + 1));
SELECT *, MD5(id) FROM tbl;
Note this will only work on single-row inserts as LAST_INSERT_ID returns the insert ID of the first row inserted.
Performing MD5(column_name) on an auto_increment value does not work as the value has not been generated yet, so it is essentially calling MD5(0).
PHP snippet
<?
$tablename = "tablename";
$next_increment = 0;
$qShowStatus = "SHOW TABLE STATUS LIKE '$tablename'";
$qShowStatusResult = mysql_query($qShowStatus) or die ( "Query failed: " . mysql_error() . "<br/>" . $qShowStatus );
$row = mysql_fetch_assoc($qShowStatusResult);
$next_increment = $row['Auto_increment'];
echo "next increment number: [$next_increment]";
?>
This will get you the next auto-increment and then you can use this in your insert.
Note: This is not perfect (Your method is imperfect as you will effectively have 2 primary keys)
From: http://blog.jamiedoris.com/geek/560/