I am facing an issue where I am getting frequent request and my code is getting struck for insert query..
Code is first checking whether records exists or not and as per output code is taking decision for insert OR update. But issue is this when my first request is coming it executes create query with transaction and later after 3-4 seconds that transaction is commit. But when second request is coming where first request is stll getting executed and checking if records exists then it is returning null due to which my code is again going into create whereas it should have gone in update case.
let data = await Model.findOne({
where: { code: variantCode }
});
if (!data) {
variant = await Model.create(body, {
transaction
});
} else {
await data.update(body);
}
I have already tried upsert, but that doesnt work.
Related
I'm trying to implement an HTTP event streaming server using MySQL where Users are able to append an event to a stream (a MySQL table) and also define the expected sequence number of the event.
The logic is somewhat simple:
Open transaction
get the next sequence number in the table to insert
verify if the next sequence number matches the expected(if supplied)
insert in database
Here's my code:
public async append(
data: any = {},
expectedSeq?: number
): Promise<void> {
let published_at = $date.create();
try {
await $mysql.transaction(async trx => {
let max = await trx(this.table)
.max({
seq: "seq",
})
.first();
if (!max) {
throw $error.InternalError(`unexpected mysql response`);
}
let next = (max.seq || 0) + 1;
if (expectedSeq && expectedSeq !== next) {
throw $error.ExpectationFailed(
`expected seq does not match current seq`
);
}
await trx(this.table).insert({
published_at,
seq: next,
data: $json.stringify(data),
});
});
} catch (err) {
if (err.code === "ER_DUP_ENTRY") {
return this.append(data, expectedSeq);
}
throw err;
}
}
My problem is this is extremely slow since there are race conditions between parallel requests to append to the same stream.. my laptop inserts/second on one stream went from ~1k to ~75.
Any pointers/suggestions on how to optimize this logic?
CONCLUSION
After consideration from comments, I decided to go with auto increment and reset the auto_increment only if there's an error. It yields around the same writes/sec with expectedSeq but much higher rate if ordering is not required.
Here's the solution:
public async append(data: any = {}, expectedSeq?: number): Promise<Event> {
if (!$validator.validate(data, this.schema)) {
throw $error.ValidationFailed("validation failed for event data");
}
let published_at = $date.create();
try {
let seq = await $mysql.transaction(async _trx => {
let result = (await _trx(this.table).insert({
published_at,
data: $json.stringify(data),
})).shift();
if (!result) {
throw $error.InternalError(`unexpected mysql response`);
}
if (expectedSeq && expectedSeq !== result) {
throw $error.ExpectationFailed(
`expected seq ${expectedSeq} but got ${result}`
);
}
return result;
});
return eventFactory(this.topic, seq, published_at, data);
} catch (err) {
await $mysql.raw(`ALTER TABLE ${this.table} auto_increment = ${this.seqStart}`);
throw err;
}
}
Why does the web page need to provide the sequence number? That is just a recipe for messy code, perhaps even messier than what you sketched out. Simply let the auto_increment value be returned to the User.
INSERT ...;
SELECT LAST_INSERT_ID(); -- session-specific, so no need for transaction.
Return that value to user.
Why not use Apache Kafka, It does all of this natively. With the easy answer out of the way, optimization is always tricky with partial information, however I think you've given us one hint that might enable a suggestion. You said without the order clause it performs much faster, which means that getting the max value is what is taking so long. That tells me a few things, first this value is not the clustered index (which is good news), second that you probably do not have sufficient index support (also good news since it's fixable by creating an index on this column, and sorting the index desc). This sounds like a table with millions or billions of rows in it, and this particular column has no guaranteed order, without the right indexing you could be doing a table scan between inserts to get the max value.
Why not use a GUID for your primary key instead of an auto-incremented integer? Then your client could generate the key and would also be able to insert it every time for sure.
Batch inserts versus singleton inserts
Your latency/performance problem is due to a batch size of 1 - as each send to the the database requires multiple round trips to the rdbms. Rather than inserting one row at a time, with a commit and verification after each row, you should rewrite your code to issue batch sizes of 100 or 1000 at a time, inserting n rows and verifying per batch rather than per row. If the batch insert fails, you can retry one row at a time.
I'm trying to make a post request to a table, using Express, Sequelize, MySQL and Postman to test the request. I'm getting an error with Sequelize not recognizing a field.
The query is not inserting the field IdProduto, I don't know why, I've checked the spelling a million times, checked all the files related and also the database table.
Executing (default):
SELECT `idPedidos`, `idCliente_fk`, `idRestaurante_fk`, `valorCompra`, `horarioChegada`, `statusPedido`, `IdProduto`
FROM `Pedidos`
AS `Pedidos`;
Executing (default):
INSERT INTO `Pedidos` (`idPedidos`,`valorCompra`,`horarioChegada`,`statusPedido`,`createdAt`,`updatedAt`,`idCliente_fk`,`idRestaurante_fk`)
VALUES (DEFAULT,40,30,'Feito','2019-05-29 23:03:41','2019-05-29 23:03:41',14,16);`
Unhandled rejection SequelizeDatabaseError: Field 'IdProduto' doesn't
have a default value
Here is my code that make the post.
exports.post = (req, res, next) => {
models.Pedidos.create({
IdProduto: req.body.IdProduto,
idCliente_fk: req.body.idCliente_fk,
idRestaurante_fk: req.body.idRestaurante_fk,
valorCompra: req.body.valorCompra,
horarioChegada: req.body.horarioChegada,
statusPedido: req.body.statusPedido
}).then((result) => res.json(result))
};
Here is the json I'm using in postman.
{
"idRestaurante_fk":16,
"valorCompra":40,
"horarioChegada":30,
"statusPedido":"Feito",
"idCliente_fk":14,
"IdProduto":2
}
Also I've checked the names in the table, and it's all right. I was thinking about setting the default value in the database myself, but it will not work, because in the console the INSERT query executed shows that the IdProduto is not getting inserted. Not sure what can be wrong here.
I have a function that is supposed to check if a license plate already exists in my MySQL database, but on the then-promise-return the result comes as:
result: [object Object]. How do you actually get the response of this knex query and parse it to check for a license plate existing?
var licensePlateExists = function(licensePlate){
knex.select('plate').from('licensePlates').where({'plate': licensePlate}).limit(1).then(function(result){
console.log("Result: " + result);
if(!result){
return true;
}
return false;
}).catch(function(error) {
console.error(error);
});
}
I think I might have an error related to the query itself, but I tested the raw query string in MySQL CLI and it outputs the row as expected. Maybe ordering matters in the knex query builder?
P.S. This doesn't error, it executes all the way through.
Try with
knex.select('plate').from('licensePlates').where('plate', licensePlate)
Actually using count query would be better
I am writing a simple API which I have been testing using Postman. I have an auto incrementing primary key (CustomerTypeID) for my "customer_type" table stored in MySQL. For practical reasons I need to be able to create records in this table without sending a CustomerTypeID. When I send the following POST request using Postman:
{
"CustomerType": "testing"
}
The updated table shows a new row with CustomerTypeID of 2 and a CustomerType of NULL.
Below is a snippet of code in my Express API which shows this specific query and how the routing for this POST request works.
var db = require('../dbconnection.js');
var CustomerType = {
addCustomerType:function(CustomerType,callback) {
return db.query("INSERT INTO customer_type (CustomerType) VALUES (?)", [CustomerType.CustomerType], callback);
}
};
module.exports = CustomerType;
I know that I could change the query to say
INSERT INTO customer_type (CustomerTypeID, CustomerType) VALUES (?,?);
and that would fill both columns. But, I do not know how to leave out the CustomerTypeID column as it will be a number that the end user will have no way of knowing.
It turns out that the syntax for the last query I gave is correct. I used the same POST request as before:
{
"CustomerType": "testing"
}
And by using the SQL query that includes CustomerTypeID, MySQL knew to just increment the value of CustomerTypeID since it was not given a value in the POST request. When I ran the same POST again with this query I received a new row with both a CustomerTypeID and a CustomerType.
I have made an application in Nodejs that every minute calls an endpoint and gets a json array that has about 100000 elements. I need to upsert this elements into my database such that if the element doesn't exist I insert it with column "Point" value set to 0.
So far I'm having a cron job and simple upsert query. But it's so slow:
var q = async.queue(function (data, done) {
db.query('INSERT INTO stat(`user`, `user2`, `point`) '+data.values+' ON DUPLICATE KEY UPDATE point=point+ 10',function (err, result) {
if (err) throw err;
});
},100000);
//Cron job here Every 1 minute execute the lines below:
var values='' ;
for (var v=0;v<stats.length;v++) {
values = '("JACK","' + stats[v] + '", 0)';
q.push({values: values});
}
How can I do such a task in a very short amount of time. Is using mysql a wrong decision? I'm open to any other architecture or solution. Note that I have to do this every minute.
I fixed this problem by using Bulk Upsert (from documentation)! I managed to Upsert over 24k rows in less than 3 seconds. Basically created the query first then ran it:
INSERT INTO table (a,b,c) VALUES (1,2,3),(4,5,6)
ON DUPLICATE KEY UPDATE c=VALUES(a)+VALUES(b);