Related
I came across a method of fast bulk update in mySQL recently:
The slow way:
UPDATE `example` SET `name` = 'Mary', `age` = '20' WHERE id = 2;
UPDATE `example` SET `name` = 'Nancy', `age` = '30' WHERE id = 4;
UPDATE `example` SET `name` = 'Oliver', `age` = '40' WHERE id = 5;
The fast way:
UPDATE `example` SET
`name` = ELT(FIELD(id,2,4,5),'Mary','Nancy','Oliver') ,
`age` = ELT(FIELD(id,2,4,5),'20','30','40')
WHERE id IN (2,4,5);
According to my google searching, the fast way is 50 times faster at large scale than the slow way, so I am wondering whether Laravel support this type of bulk update without the need to execute raw sql generated by myself.
Also if anyone is familiar with mysql and can tell me is this really that much faster or not, and if there is any scenario this method is actually worse I will be very graceful.
Edit(as requested in comment):
My current Laravel code would work like this for update:
foreach ($modelList as $model) {
$model->setConnection($connection);
$model->save();
}
What I would like for bulk update is
DB::connection($connection)->table($table)->bulkUpdate($models);
As I know currently Laravel already supports bulk insert as follows:
foreach ($models as $model) {
$attributes = $model->getAttributes();
$params[] = $attributes;
}
DB::connection($connection)->table($table)->insert($params);
which will generate one insert statement with multiple records instead of multiple insert statements.
If there's no solution, you can still do it raw with prepared statements:
DB::update("
UPDATE `example` SET
`name` = ELT(FIELD(id,2,4,5),?,?,?) ,
`age` = ELT(FIELD(id,2,4,5),?,?,?)
WHERE id IN (?,?,?)
", [
'Mary', 'Nancy', 'Oliver',
'20', '30', '40',
2, 4, 5
]);
Technically, this should also work. Haven't checked it yet:
$sql = DB::table('users')->update([
['name' => DB::raw("ELT(FIELD(id,2,4,5),'Mary','Nancy','Oliver')")],
['age' => DB::raw("ELT(FIELD(id,2,4,5),'20','30','40')")]
])
->whereIn('id', [ 2, 4, 5 ]);
Another approach:
INSERT INTO t
(name, age, id)
VALUES
('Mary', 20, 2),
('Nancy', 30, 4),
('Oliver', 40, 5)
ON DUPLICATE KEY UPDATE name = VALUES(name);
Assuming the ids are already in the table and it has PRIMARY KEY(id), there would be any INSERTs, only UPDATEs -- which is what you want.
The name = VALUES(name) is a dummy no-op. It needs to change to name = NEW.name in MySQL 8.0.
From Comment...
If you must not INSERT "new" rows, then consider this:
INSERT INTO tmp
(name, age, id)
VALUES
('Mary', 20, 2),
('Nancy', 30, 4),
('Oliver', 40, 5);
UPDATE t
JOIN tmp USING(name);
(I am not sure of the syntax; play around with the "multi-table update".)
This second 'solution' requires creating the tmp table and/or having it already available (plus possibly a TRUNCATE).
UPDATE student set roll='12',fee=fee-10 where roll='15'
Use this example for your code. This is easier and fastest way to support any platform for mysql.
How to update mysql data in bulk ?
How to define something like this :
UPDATE `table`
WHERE `column1` = somevalues
SET `column2` = othervalues
with somevalues like :
VALUES
('160009'),
('160010'),
('160011');
and othervalues :
VALUES
('val1'),
('val2'),
('val3');
maybe it's impossible with mysql ?
a php script ?
The easiest solution in your case is to use ON DUPLICATE KEY UPDATE construction. It works really fast, and does the job in easy way.
INSERT into `table` (id, fruit)
VALUES (1, 'apple'), (2, 'orange'), (3, 'peach')
ON DUPLICATE KEY UPDATE fruit = VALUES(fruit);
or to use CASE construction
UPDATE table
SET column2 = (CASE column1 WHEN 1 THEN 'val1'
WHEN 2 THEN 'val2'
WHEN 3 THEN 'val3'
END)
WHERE column1 IN(1, 2 ,3);
If the "bulk" data you have is dynamic and is coming from PHP (you did tag it, after all), then the query would look something like this:
INSERT INTO `foo` (id, bar)
VALUES
(1, 'pineapple'),
(2, 'asian pear'),
(5, 'peach')
ON DUPLICATE KEY UPDATE bar = VALUES(bar);
and the PHP to generate this from an existing array (assuming the array is of a format like:
$array = (
somevalues_key => othervalues_value
);
) would look something like this (by no means the best (doesn't address escaping or sanitizing the values, for instance), just an quick example):
$pairs = array();
foreach ($array as $key => $value) {
$pairs[] = "($key, '$value')";
}
$query = "INSERT INTO `foo` (id, bar) VALUES " . implode(', ', $pairs) . " ON DUPLICATE KEY UPDATE bar = VALUES(bar)";
You could try an UPDATE with JOIN as below:
UPDATE table
INNER JOIN (
SELECT 1 column1, 2 column2, 10 new_v1, 20 new_v2, 30 new_v3
UNION ALL SELECT 4 column1, 5 column2, 40 new_v1, 50 new_v2, 60 new_v3
) updates
ON table.column1 = updates.column1
AND table.column2 = updates.column2
SET
table.column1 = updates.new_v1,
table.column2 = updates.new_v2,
table.column3 = updates.new_v3;
As long as you can craft the inner SELECT statements from the updates subquery you would get the benefit of running all these updates in a single statement (which should give you some performance boost on InnoDB depending on your table size).
If you are using a drag & drop tableView or collectionView to sort datas in your app, like allowing users to arrange their photos by drag and drop functionality, send a comma seperated list of ordered ids to the backend after user edits finish.
In your backend, explode ids to the an array like
$new_ranks = array();
$supplied_orders = explode(",", $_POST["supplied_new_order"]); //52,11,6,54,2 etc
$start_order = 99999;
foreach ($supplied_orders as $supplied_row_id) {
//your all validations... make sure supplied_row_id belongs to that user or not etc..
$new_ranks[intval($supplied_row_id)] = $start_order--;
}
now, you can update all new ranks like #Farside recommendation 2.
if (count($new_ranks) > 0) {
$case_sqls = array();
foreach ($new_ranks as $id => $rank) {
$case_sqls[] = "WHEN ".intval($id)." THEN ".intval($rank)."";
}
$case_sql = implode(" ", $case_sqls);
$this->db->query("
UPDATE
service_user_medias
SET
rank = (CASE id ".$case_sql." END)
WHERE
id IN(".implode(",", array_keys($new_ranks)).");
");
}
If you have data in array format then try this
and your query is like "UPDATE table WHERE column1 = ? SET column2 = ?"
then set it like below
foreach($data as $key => $value) {
$query->bind_param('ss', $key, $value);
$query->execute();
}
hope it'll work.
Reference from this.
I need to set up some initial records in our mysql database.
I insert a new record which gives me the siteID of the last insert I then use this in creating the rows in my allNews_copy table. The following works (syntax is from applicationcraft ide) as I get the correct structure but there is a 60 timeout for the call to the server.
Therefore the script stops after about 270 rows.
for(i3=1; i3<51; i3++) {
//console.log('i3 = '+i3 );
for(i2=1; i2<101; i2++) {
//console.log('i2 = '+i2 );
var isVisible2 = 0;
if(i2 < 6){ isVisible2 = 1;}
cObj.insert('allNews_copy',{
siteID:siteID,
newsIDInt:i2,
catIDInt:i3,
title:'Item title '+i2 + ' in Cat' + i3,
newsDesc:'Item Desc',
visible:isVisible2
});
}
}
The total number of rows would be 5000.
So can I do this by using a mysql loop via a std mysql syntax?
In standard SQL syntax you can insert multiple rows in a single statement:
INSERT INTO table (<list of columns>)
VALUES (<list of values>), (<list of values>), (<list of values>), ...
I don't know how to translate this syntax to the API you're using, though.
I am trying to insert records in 2 different mysql tables. Here's the situation:
Table 1: is_main that contains records of resorts with a primary key called id.
Table 2: is_features that contains a list of features that a resort can have (i.e. beach, ski, spa etc...). Each feature has got a primary key called id.
Table 3: is_i2f to connect each resort id with the feature id. This table has got 2 fields: id_i and id_f. Both fields are primary key.
I have created a form to insert a new resort, but I'm stuck here. I need a proper mysql query to insert a new resort in the is_main table and insert in is_i2f one record for each feature it has, with the id of the resort id id_i and the id of the feature id id_f.
$features = ['beach','relax','city_break','theme_park','ski','spa','views','fine_dining','golf'];
mysql_query("INSERT INTO is_main (inv_name, armchair, holiday, sipp, resort, price, rooms, inv_length, more_info)
VALUES ('$name', '$armchair', '$holiday', '$sipp', '$resort', '$price', '$rooms', '$length', '$more_info')");
$id = mysql_insert_id();
foreach($features as $feature) {
if(isset($_POST[$feature])) {
$$feature = 1;
mysql_query("INSERT INTO is_i2f (id_i, id_f) VALUES (" . $id . ", ?????????????? /missing part here????/ ); }
else {
$$feature = 0; }
}
Thanks.
Please, I'm going CrAzY!!!!!!!!!!!!!!
This may not be relevant to you, but...
Would it not make more sense to leave the link table unpopulated? You can use JOINs to then select what you need to populate the various views etc in your application
i.e. query to get 1 resort with all features:
SELECT
Id,
f.Id,
f.Name
FROM IS_MAIN m
CROSS JOIN IS_FEATURES f
WHERE m.Id = $RequiredResortId
Please find the answer on Mysql insert into 2 tables.
If you want to do multiple insert at a time you can write a SP to fulfill your needs
If I understand you correctly you could concatenate variable amount of to be inserted/selected values into one query. (This is the second query which needs an id from the first.)
//initializing variables
$id = mysql_insert_id();
$qTail = '';
$i = -1;
//standard beginning
$qHead = "INSERT INTO `is_i2f` (`id`,`feature`) VALUES ";
//loop through variable amount of variables
foreach($features] as $key => $feature) {
$i++;
//id stays the same, $feature varies
$qValues[$i] = "('{$id}', '{$feature}')";
//multiple values into one string
$qTail .= $qValues[$i] . ',';
} //end of foreach
//concatenate working query, need to remove last comma from $qTail
$q = $qHead . rtrim($qTail, ',');
Now you should have a usable insert query $q. Just echo it and see how it looks and test if it works.
Hope this was the case. If not, sorry...
Currently I'm manually creating a string where I concatenate all the values in each row in my table. I'm hashing this string for each row to get a hash value for the current values (/status) of the row, which I'm later is using to determine if the row has changed.
Instead of doing this manually, is there an build-in way i mySQL to get a unique hash value for each row?
you could do something like
SELECT MD5(concat(field1, field2, field3, ...)) AS rowhash
but you can't get away from listing which fields you want, as concat(*) is not an option (syntax error).
It's better to use concat_ws(). e.g. two adjacent column: 12,3 => 1,23 .
Sorry, this still has some problems. Think about the null value, empty string, string can contain ',', etc...
A program is required to generate the hash statement, which should replace null to specific value (for null-able columns), and also use the seldom used char/byte as separator.
There are problems with CONCAT, e.g. CONCAT('ab', 'c') vs CONCAT('a', 'bc'). Two different rows, but result is the same. You could use CONCAT_WS(';', 'ab', 'c') to get ab;c but in case of CONCAT_WS(';', ';', '') vs CONCAT_WS(';', '', ';') you still get the same result.
Also CONCAT(NULL, 'c') returns NULL.
I think the best way is to use QUOTE:
SELECT MD5(CONCAT(QUOTE(c1), QUOTE(c2), QUOTE(c3))) AS row_hash FROM t1;
Result of: select (concat(quote('a'), quote('bc'), quote('NULL'), quote(NULL), quote('\''), quote('')));
is: 'a''bc''NULL'NULL'\''''
Also, don't use GROUP_CONCAT() to get hash of table, it has limit: https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_group_concat_max_len
Instead, CHECKSUM TABLE might be better, but you can't skip columns with CHECKSUM TABLE https://dev.mysql.com/doc/refman/5.7/en/checksum-table.html
Well I made a little script that could do excactly what you want, and maybe what others want... so here it goes...for PHP that is...
first you have to make a list of columns of the table, then you make a "case when" statement for each column based on their type and put that in the concat_ws statement and then you hash it with sha1...i've used this method on very large tables (600000+ records) and the speed is quite good when selecting all records. also I think that it is faster to concat the required data in a concat_ws and explode it in php or whatever you are using, but that is just a hunch...
<?
$query= mysql_query("SHOW COLUMNS FROM $table", $linklive);
while ($col = mysql_fetch_assoc($query)) {
$columns[] = mysql_real_escape_string($col['Field']);
if ($col['Key'] == 'PRI') {
$key = mysql_real_escape_string($col['Field']);
}
$columnsinfo[$col['Field']] = $col;
}
$dates = array("date","datetime","time");
$int = array("int","decimal");
$implcols = array();
foreach($columns as $col){
if(in_array($columnsinfo[$col]['Type'], $dates)){
$implcols[] = "(CASE WHEN (UNIX_TIMESTAMP(`$col`)=0 || `$col` IS NULL) THEN '[$col EMPTY]' ELSE `$col` END)";
}else{
list($type, $rest) = explode("(",$columnsinfo[$col]['Type']);
if(in_array($columnsinfo[$col]['Type'], $dates)){
$implcols[] = "(CASE WHEN ( `$col`=0 || `$col` IS NULL ) THEN '[$col EMPTY]' ELSE `$col` END)";
}else{
$implcols[] = "(CASE WHEN ( `$col`='' || `$col` IS NULL ) THEN '[$col EMPTY]' ELSE `$col` END)";
}
}
}
$keyslive = array();
//echo "SELECT $key SHA1(CONCAT_WS('',".implode(",", $columns).")) as compare FROM $table"; exit;
$q = "SELECT $key as `key`, SHA1(CONCAT_WS('',".implode(", ",$implcols).")) as compare FROM $table";
?>