I am attempting to do a bulk insert into MySQL using
INSERT INTO TABLE (a, b, c) VALUES (?, ?, ?), (?, ?, ?)
I have the general log on, and see that this works splendidly for most cases. However, when the table has a BLOB column, it doesn't work as well.
I am trying to insert 20 records.
Without the BLOB, I see all 20 records in the same query in the general log, 20 records inserted in the same query.
WITH the BLOB, I see only 2 records per query in the general log, it takes 10 queries in total.
Is this a problem with MySQL, the JDBC Driver, or am I missing something else. I would prefer to use a BLOB as I have data in protobufs.
Here is an example table...
CREATE TABLE my_table (
id CHAR(36) NOT NULL,
name VARCHAR(256) NOT NULL,
data BLOB NOT NULL,
PRIMARY KEY (id)
);
Then, create your batch inserts in code...
val ps = conn.prepareStatement(
"INSERT INTO my_table(id, name, data) VALUES (?, ?, ?)")
records.grouped(1000).foreach { group =>
group.foreach { r =>
ps.setString(1, UUID.randomUUID.toString)
ps.setString(2, r.name)
ps.setBlob(3, new MariaDbBlob(r.data))
ps.addBatch()
}
ps.executeBatch()
}
If you run this and inspect the general log, you will see...
"2018-10-12T18:37:55.714825Z 4 Query INSERT INTO my_table(id, name, fqdn, data) VALUES ('b4955537-2450-48c4-9953-e27f3a0fc583', '17-apply-test', _binary '
17-apply-test\"AAAA(?2Pending8?????,J$b4955537-2450-48c4-9953-e27f3a0fc583
1:2:3:4:5:6:7:8Rsystem'), ('480e470c-6d85-4bbc-b718-21d9e80ac7f7', '18-apply-test', _binary '
18-apply-test\"AAAA(?2Pending8?????,J$480e470c-6d85-4bbc-b718-21d9e80ac7f7
1:2:3:4:5:6:7:8Rsystem')
2018-10-12T18:37:55.715489Z 4 Query INSERT INTO my_table(id, name, data) VALUES ('7571a651-0e0b-4e78-bff0-1394070735ce', '19-apply-test', _binary '
19-apply-test\"AAAA(?2Pending8?????,J$7571a651-0e0b-4e78-bff0-1394070735ce
1:2:3:4:5:6:7:8Rsystem'), ('f77ebe28-73d2-4f6b-8fd5-284f0ec2c3f0', '20-apply-test', _binary '
20-apply-test\"AAAA(?2Pending8?????,J$f77ebe28-73d2-4f6b-8fd5-284f0ec2c3f0
As you can see, each INSERT INTO only has 2 records in it.
Now, if you remove the data field from the schema and insert and re-run, you will see the following output (for 10 records)...
"2018-10-12T19:04:24.406567Z 4 Query INSERT INTO my_table(id, name) VALUES ('d323d21e-25ac-40d4-8cff-7ad12f83b8c0', '1-apply-test'), ('f20e37f2-35a4-41e9-8458-de405a44f4d9', '2-apply-test'), ('498f4e96-4bf1-4d69-a6cb-f0e61575ebb4', '3-apply-test'), ('8bf7925d-8f01-494f-8f9f-c5b8c742beae', '4-apply-test'), ('5ea663e7-d9bc-4c9f-a9a2-edbedf3e5415', '5-apply-test'), ('48f535c8-44e6-4f10-9af9-1562081538e5', '6-apply-test'), ('fbf2661f-3a23-4317-ab1f-96978b39fffe', '7-apply-test'), ('3d781e25-3f30-48fd-b22b-91f0db8ba401', '8-apply-test'), ('55ffa950-c941-44dc-a233-ebecfd4413cf', '9-apply-test'), ('6edc6e25-6e70-42b9-8473-6ab68d065d44', '10-apply-test')"
All 10 records are in the same query
I tinkered until I found the fix...
val ps = conn.prepareStatement(
"INSERT INTO my_table(id, name, data) VALUES (?, ?, ?)")
records.grouped(1000).foreach { group =>
group.foreach { r =>
ps.setString(1, UUID.randomUUID.toString)
ps.setString(2, r.name)
//ps.setBlob(3, new MariaDbBlob(r.data))
ps.setBytes(r.data)
ps.addBatch()
}
ps.executeBatch()
Using PreparedStatement.setBytes instead of using MariaDbBlob seemed to do the trick
Related
I'm inserting values into a MySQL database using Nodejs mysql2 library.
Here is an example of a prepared statement:
await conn.execute(
'INSERT INTO Friends (id, user, name, gender) VALUES (UUID(), ?, ?, ?)',
[ user, body.name, body.gender ]
);
How can I achieve the above if sometimes the body.gender value is not set? I want several attributes in the http request to be optional and insert all allowed values that have been sent in the http request into the database.
The above code gives an error if I leave body.gender out of the http request.
If there is no some data in body or not sending some data from the client to register in the database, you have to put a null value for that row in that column. You can use this JavaScript feature to do this:
await conn.execute(
'INSERT INTO Friends (id, user, name, gender) VALUES (UUID(), ?, ?, ?)',
[ user || null, body.name || null, body.gender || null ]
);
Using this possibility, in the absence of any of the data sent in the body, its value is undefined and the value of null is placed in the query.
I'm using the mysql-otp driver for Erlang. It seems to be working fine but there is no documentation on using it to insert multiple rows into a table.
simple use case for single row insert:
ok = mysql:query(Pid, "INSERT INTO mytable (id, bar) VALUES (?, ?)", [1, 42]).
But I need to insert multiple values, can I do something like this?
ok = mysql:query(Pid, "INSERT INTO mytable (id, bar) VALUES (?, ?)", [(1, 42),(2, 36), (3,12)]).
Documentation states Params = [term()], so probably not, which is a bummer.
You can certainly do a combination of lists:foldl/3 and lists:join/2 on your arguments to create the desired query format:
L = [[1, 42],[2, 36], [3,12]],
PreparedList = lists:foldl(fun (Params, Inserts) -> Inserts ++ [io_lib:format("(~p,~p)", Params)] end, [], L),
%% Then you need to join these with a comma:
Prepared = lists:flatten(lists:join(",", PreparedList)),
%% this will result in "(1,42),(2,36),(3,12)"
Now you just need to call the mysql insert with this Prepared variable:
ok = mysql:query(Pid, "INSERT INTO mytable (id, bar) VALUES ?", [Prepared]).
%% The query will look like: "INSERT INTO mytable (id, bar) VALUES (1,42),(2,36),(3,12)"
I don't think this driver or mysql can do such kind of things.
I think you should do it likes below
insert_mytable(Data)->
{ok,Ref} = mysql:prepare(Pid,insert_mytable,"INSERT INTO mytable (id, bar) VALUES (?, ?)"),
loop_insert(_Pid,Ref,Data).
loop_insert(_Pid,_Ref,[])-> ok;
loop_insert(Pid,Ref,[H|T])->
ok = mysql:execute(Pid,Ref,H),
loop_insert(Pid,Ref,T).
I'm trying to upsert a database table with loopback. The raw query is
insert into all_inventory (sku, qty, regal, fach, skuRegalFach)
values (?, 1, ?, ?, ?)
on duplicate key update
qty = qty + 1,
regal = values(regal),
fach = values(fach)
Is there any way to do this with loopback?
Currently I'm facing two problems.
I get:
ER_DUP_ENTRY: Duplicate entry '22323' for key
'all_inventory_SkuRegalFach_uindex'
Because loopback doesn't seem to be able to handle the key correctly.
And I have no idea how to tell loopback to add 1 to the qty field instead of just overriding it with the new value.
I have it working with a raw query right now,
let ds = Inventory.dataSource,
values = [sku, regal, fach, sku + regal + fach],
sql = `insert into all_inventory (sku, qty, regal, fach, skuRegalFach) values (?, 1, ?, ?, ?) on duplicate key update qty = qty + 1, regal = values(regal), fach = values(fach)`
ds.connector.query(sql, values, (err, products) => {
if (err) return console.error(err);
cb(null, products);
});
Is there a way to do this with loopback's ORM?
I'd use a find filter. Either findById or find with an appropriate filter.
If you get a result, then it exists. You can modify qty and then store it. If it doesn't exist, then you just create it.
I am trying to execute 2 queries.
First one should insert data (especially "product") or update in case the db already has a row with such title.
Second one should insert new category for product which was inserted\updated from 1st query and ignore any inserts, if table already has such product with such category
Here is my code :
conn = DatabaseConnection.getConnection();
stmt = conn.createStatement();
conn.setAutoCommit(false);
String updateSQL = "INSERT INTO product (title, price, `status`) " +
"VALUES(?, ?, ?)" +
"ON DUPLICATE KEY UPDATE price = ?, `status` = ?;"
PreparedStatement preparedStatement = conn.prepareStatement(updateSQL);
preparedStatement.setString(1, product.getTitle());
preparedStatement.setBigDecimal(2, product.getPrice());
preparedStatement.setInt(3, product.getStatus().ordinal());
preparedStatement.executeUpdate();
updateSQL = "INSERT IGNORE INTO product_categories (product_id, category_id) " +
"VALUES (last_insert_id(), ?);";
preparedStatement = conn.prepareStatement(updateSQL);
preparedStatement.setLong(1, categoryId);
preparedStatement.executeUpdate();
conn.commit();
So, the problem is that I use last_insert_id() which means that i will use incorrect row in 2nd query if 1st query just updated the data.
So, I would like to know how could I synchronize these 2 queries.
Since you don't have access to last_insert_id() in the second query, you'll have to fetch it as in the answers for this question.
Here's an example:
...
preparedStatement.executeUpdate(); // this is the first query
ResultSet rs = preparedStatement.getGeneratedKeys();
if ( rs.next() )
{
long last_insert_id = rs.getLong(1);
updateSQL = "INSERT IGNORE INTO product_categories (product_id, category_id) " +
"VALUES (?, ?);";
preparedStatement = conn.prepareStatement(updateSQL);
preparedStatement.setLong(1, last_insert_id);
preparedStatement.setLong(2, categoryId);
preparedStatement.executeUpdate();
}
conn.commit();
If the first query didn't result in an INSERT, then there isn't enough information to add the product to the product_category, in which case this is skipped all together. This does assume that the product is already in the category. If you're not sure about that, and want to execute the second query regardless, you could query for the product_id:
SELECT id FROM product WHERE title = ?
and then use that id instead of the last_insert_id variable, or, you could change the second query and use title as a key (although I'd stick with an id):
INSERT IGNORE INTO product_categories (product_id, category_id)
VALUES (SELECT id FROM product WHERE title = ?), ?)
I have a simple SQL insertion into a two field MySQL table (records):
Where 'id' is the primary key (int) ... and 'photo' is a mediumblob
$photo = <binaryfile>
$id = recordNo
$MYSQL = "INSERT INTO records (id, photo)
VALUES (?, ?)
ON DUPLICATE KEY UPDATE photo = ?";
$cursor->execute( *what goes in here? I have 3 ?'s but only 2 variables* );
In the execute function you should pass the array like this:
$cursor->execute(array($id,$photo,$photo));
Here $photo name should come two time one is for second one and next is for third ? mark.