How do I use expression with bulk insert using npm mysql? - mysql

This question answers 99% of what I am looking for...
How do I do a bulk insert in mySQL using node.js
var sql = "INSERT INTO Test (name, email, n) VALUES ?";
var values = [
['demian', 'demian#gmail.com', 1],
['john', 'john#gmail.com', 2],
['mark', 'mark#gmail.com', 3],
['pete', 'pete#gmail.com', 4]
];
conn.query(sql, [values], function(err) {
if (err) throw err;
conn.end();
});
If I want to pass an expression, such as NOW(), how would I do that? If I pass it in the array, it would count as a string. Since VALUES is a ? that gets populated by the array, I can't easily inject an expression. Any ideas?

Basically it seems impossible, so you should make a query string you want.
I would try this as follows.
var sql = "INSERT INTO Test (name, email, n, modified_on) VALUES ?";
var values = [
['demian', 'demian#gmail.com', 1, '::NOW()'],
['john', 'john#gmail.com', 2, '::UNIX_TIMESTAMP()'],
['mark', 'mark#gmail.com', 3, '::DATE()'],
['pete', 'pete#gmail.com', 4, '::NOW()']
];
var formattedQuery = connection.format(sql, [values]).replace(/'::(.*?)'/g, '$1');
connection.query(formattedQuery, function(err) {
});
fomattedQuery is as follows.
INSERT INTO Test (name, email, n, modified_on) VALUES ('demian', 'demian#gmail.com', 1, NOW()), ('john', 'john#gmail.com', 2, UNIX_TIMESTAMP()), ('mark', 'mark#gmail.com', 3, DATE()), ('pete', 'pete#gmail.com', 4, NOW())
I hope this helps.

Related

Inserting data from a for loop in database

I am fetching data from an APi, extracting part of it. The data comes in nested dictionaries and lists and I used a nested for loop to extract variables. I want to insert it in mysql db, not sure how to do so, as in some of the columns I will receive a different number of values to be stored. For example, cars could be 1,2,3 or 4.
All vehicle_id fetched should be inserted into a column all_vehicles, I am not sure how to do this either.
datetime_received= datetime.now()
car_dealer_id=11
int_id = 8
dealer_name ='XXX'
for car in cars:
code=car['Code']
start_date=car['RDate']
end_date=car['RDate']
for portion in car['Consists']['Portions']:
location= portion['Location']
for consist in portion['Consist']:
ext_id = consist['ExtId']
for vehicle in consist['Vehicles']:
vehicle_id= vehicle['Id']
sql = """
INSERT INTO table
(`datetime_received`, `car_dealer_id` , `ind_id`, `dealer_name`,`code`,`start_date`, `start_time`, `end_date`, `location`, `ext_id`, `all_vehilces`)
VALUES ('%s', '%s', '%s', '%s', '%s', '%s', '%s', '%s', '%s', '%s')"""
cursor.executemany(sql, data)
connection.commit()
connection.close()
Data:
cars = {
"Consists": {
"Portions": [
{
"Consist": [
{
"ext_id": "755411",
"Position": "0",
"Vehicles": [
{
"Id": "92",
"Position": "1"
},
{
"Id": "921",
"Position": "2"
},
{
"Id": "932",
"Position": "3"
},
{
"Id": "34",
"Position": "4"
},
{
"Id": "92",
"Position": "5"
}
]
}
],
"Location": "ATA"
}
],
"Updated": "2022-07-21T04:25:08.0000000+01:00"
},
"Code": "5`enter code here`75",
"RDate": "2022-07-21T08:25:00.0000000+01:00",
"RunDate": "2022-07-21T00:00:00.0000000+01:00",
}
EDITED: Thanks to Barmar, I managed to insert the values.
I have one final value to insert in the data[]. Based on the ext_id value I get, I have a function returning the corresponding my_system_id. I want to insert the my_system_ids as well, but I am not calling the function from the correct place and it is not being inserted into the db table.
Here is the function:
def get_my_system_id(ext_id):
cursor=db_conn.cursor()
sql=("""SELECT my_system_id FROM table
WHERE ext_id= %s""")
data=(ext_id,)
cursor.execute(sql,data)
id_row =cursor.fetchone()
if_row is not None:
my_id=id_row[0]
return(my_id)
else:
return null
Use ','.join() to combine all the vehicle IDs into a comma-delimited list.
In the prepared statement, %s should not be quoted. You also only had 10 of them, but you're inserting into 11 columns.
With th edit, add a call to get_my_system_id(ext_id) to the loop, and add that value to the data list.
data = []
for car in cars:
code=car['Code']
start_date, end_date = car['RDate'].split('T')
end_date=car['RDate']
for portion in car['Consists']['Portions']:
location= portion['Location']
for consist in portion['Consist']:
ext_id = consist['ExtId']
vehicle_ids = ','.join(v['id'] for v in consist['Vehicles'])
system_id = get_my_system_id(ext_id)
if not system_id:
print(f"No system ID found for ext_id = {ext_id}, skipping")
continue
data.append((datetime_received, car_dealer_id, int_id, dealer_name, code, start_date, start_time, end_date, location, ext_id, system_id, vehicle_ids))
sql = """
INSERT INTO table
(`datetime_received`, `car_dealer_id` , `ind_id`, `dealer_name`,`code`,`start_date`, `start_time`, `end_date`, `location`, `ext_id`, my_system_id, `all_vehicles`)
VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)"""
cursor.executemany(sql, data)
connection.commit()
connection.close()

List of tuples with strings not all arguments converted during string formatting

I am trying to insert hundreds of rows into a MySQL db at once. There are two types of records, unanswered calls and answered calls. I am putting all records into a list of tuples, and each record is it's own tuple, so that I can use the executemany function. I am getting a TypeError: not all arguments converted during string formatting, and I don't understand why.
answered = []
unanswered = []
insertQuery = """ INSERT INTO cdr (recno, relcause, starttime, answertime, endtime, releasecausetext, releasecausecode, 1streleasedialog,
origtrunk, callingnumber, orighost, callednumber, desthost, origcallid, origremotepayloadip, origremotepayloadport,
origlocalpayloadip, origlocalpayloadport, termtrunk, termsourcenumber, termsourcehost, termdestnumber, termdesthostname,
termcallid, termremotepayloadip, termremotepayloadport, termlocalpayloadip, termlocalpayloadport, duration, postdialdelay,
ringtime, durationms, routetableused, origtidalias, termtidalias, termpddms, reasoncause, mappedcausecode, mappedreasoncause,
reasoncausetext, origmos, termmos) VALUES ('%s'); """
for y in cdrList:
#Check to make sure record does not exist
sqlQuery = "select * from cdr where recno = %d and origcallid = %s;" % (int(y[0]), y[13])
if cursor.execute(sqlQuery):
print("Record exists")
else:
if y[7]=='NA':
unanswered.append((y[0], y[5],extractSqlDate(y[6]), 'null', extractSqlDate(y[8]), y[10], y[11], y[12], y[13], y[15], y[16], y[17], y[18], y[19], y[20], y[21], y[22], y[23], y[32], y[34], y[35], y[36], y[37], y[38], y[39], y[40], y[41], y[42], y[53], y[54], y[55], y[56], y[60], y[66], y[67], y[71], y[78], y[79], y[80], y[81], y[85], y[88]))
else:
answered.append((y[0], y[5],extractSqlDate(y[6]), extractSqlDate(y[7]), extractSqlDate(y[8]), y[10], y[11], y[12], y[13], y[15], y[16], y[17], y[18], y[19], y[20], y[21], y[22], y[23], y[32], y[34], y[35], y[36], y[37], y[38], y[39], y[40], y[41], y[42], y[53], y[54], y[55], y[56], y[60], y[66], y[67], y[71], y[78], y[79], y[80], y[81], y[85], y[88]))
try:
print(answered)
cursor.executemany(insertQuery, answered)
cursor.executemany(insertQuery, unanswered)
db.commit()
print("Record inserted successfully")
except MySQLdb.Error as e:
print(e)
I have confirmed that each element in each tuple in the list is a string:
Successfully connected to database
/PATH/20190610/20190610-0015-1750147245-1750147250.cdr
[('1750147245', '0001', '2019-06-10 00:10:50', '2019-06-10 00:10:59', '2019-06-10 00:11:13', 'Normal BYE', ' 200', 'O', '001102', '+tn', 'ip', '+tn', 'ip', '273418599_83875291#ip', 'ip', '20530', 'ip', '11944', '000020', '+tn', 'ip', 'tn', 'ip', '4121333-0-2851866068#ip', 'ip', '16840', 'ip', '11946', '13', '1', '8', '13450', '50', 'C - Peerless C6933_04 Origin', 'P - Thirdlane 6', '1150', '', '200', '', '', '0', '0')]
I found the problem. The tuple was returning strings, so the insert query was trying to insert values like this: ''value''. I removed the ' around the %s, and, based on #jonrsharpe's comment, added %s for each other value, and it worked.

Sequelize attribute with alias seems undefined

I'm trying to get id of AttributeValue model with an alias of attribute_value_id as below but attribute_value_id seems undefined in otherOptions constant.
const otherOptions = await AttributeValue.findAll({
where: {
id: {
[Op.in]: attributeValueIds
} },
attributes: [
['id', 'attribute_value_id'],
'attribute_id',
'value'
]
})
I'm fallowing sequlize documantation about querying and the below query is built by sequelize but there is no alias attribute in the resulting object.
SELECT `id` AS `attribute_value_id`, `attribute_id`, `value` FROM `attribute_values` AS `attribute_value` WHERE `attribute_value`.`id` IN (1, 1, 1, 1, 19);
You can get it with otherOptions[0].getDataValue('attribute_value_id')

Node JS mysql deleting multiple rows with an array of arrays

I need to make a query which would be able to delete multiple rows from my table. In order to do that I've created an arrays within array with values which need to be passed to that query. Here is my code:
var deleteRooms = [ [ 3, 23 ], [ 4, 23 ], [ 5, 23 ], [ 2, 23 ]];
connection.query("DELETE FROM rate_plans_rooms WHERE room_id = ? AND rate_plan_id = ? ",
[deleteRooms],function(err,results){
if(err){return console.log(err)}
else
{
console.log('sended');
}
});
But every time I receive an error like this:
{ Error: ER_PARSE_ERROR: You have an error in your SQL syntax; check
the manual that corresponds to your MariaDB server version for the
right syntax to use near ' (4, 23), (5, 23), (2, 23) AND rate_plan_id
= ?' at line 1
How can I fix that and send my query properly?
A solution for your problem is to use 'IN' inside your query:
var deleteRooms = [[3,23],[4,23],[5,23], [2,23]];
connection.query("DELETE FROM rate_plans_rooms WHERE (room_id, rate_plan_id) IN (?)",
[deleteRooms],function(err,results){
if(err) return console.log(err)
else console.log('sended');
});
The accepted solution did not work for me as it would give an Error: ER_OPERAND_COLUMNS: Operand should contain 2 column(s) error. Instead, this worked for me:
var deleteRooms = [[3,23],[4,23],[5,23], [2,23]];
queryArray = Array(deleteRooms.length).fill('(?)'));
connection.query("DELETE FROM rate_plans_rooms WHERE (room_id, rate_plan_id) IN ("+queryArray.join(',')+")",
[deleteRooms],function(err,results){
if(err) return console.log(err)
else console.log('sended');
});

Multi row insert using Knex.js

Am trying to build an multi-row insert query using Knex.js
My post request contains a variable which is formatted in the following format : [{addon_name:'sugar'},{addon_name:'milk'}]
My DB table has only one column namely addon_name
My knex query in my node application goes as follows
knex(`<table_name>`).insert(req.body.`<param_name>`))
expected op
insert into `<tablename>`(`addon_name`) values (sugar), (milk);
but the code dosn't work. Any comments ?
Error Details
{ [Error: insert into `table_name` (`0`, `1`, `10`, `11`, `12`, `13`, `14`, `15`, `16`, `17`, `18`, `19`, `2`, `20`, `21`, `22`, `23`, `24`, `25`, `26`, `27`, `28`, `29`, `3`, `30`, `31`, `32`, `33`, `34`, `35`, `36`, `37`, `38`, `39`, `4`, `40`, `41`, `5`, `6`, `7`, `8`, `9`) values ('[', '{', 'm', 'e', ':', '\'', 's', 'u', 'g', 'a', 'r', '\'', 'a', '}', ',', '{', 'a', 'd', 'd', 'o', 'n', '_', 'n', 'd', 'a', 'm', 'e', ':', '\'', 'm', 'i', 'l', 'k', '\'', 'd', '}', ']', 'o', 'n', '_', 'n', 'a') - ER_BAD_FIELD_ERROR: Unknown column '0' in 'field list']
code: 'ER_BAD_FIELD_ERROR',
errno: 1054,
sqlState: '42S22',
index: 0 }
Though this is an old question, I am replying here just for others who stumble upon this.
Knex now supports multi-row inserts like this:
knex('coords').insert([{x: 20}, {y: 30}, {x: 10, y: 20}])
outputs:
insert into `coords` (`x`, `y`) values (20, DEFAULT), (DEFAULT, 30), (10, 20)
There's also the batchInsert utility will inserts a batch of rows wrapped inside a transaction.
req.body.<param_name> is always a string. Most probably this will work for you:
knex(table_name).insert(JSON.parse(req.body.param_name)));
What you are seeing in your error is Knex treating the string as an array of chars, trying to push it to the table.
In the error, the following:
values ('[', '{', 'm', 'e', ':', '\'', 's', ...
Is actually your string being broken down: [{me:\'s....
Thanks. I changed the structure of my input in post method, to an comma separated string. That way it gets easier to parse the input and model it the way I need.
post method input : "milk,sugar"
code
//Knex accepts multi row insert in the following format [{},{}] => we need to
//model our input that way
var parsedValues = [];
try {
var arr = req.body.addons.split(',');
}catch(err){
return res.send({ "Message": "405" }); // Data not sent in proper format
}
for (var i in arr) {
parsedValues.push({addon_name: arr[i]});
}
console.log(parsedValues);
knex(`<table_name>`).insert(parsedValues).then(function (rows){
console.log(rows);
return res.send({ "Message": "777" }); // Operation Success
}).catch(function (err){
console.log(err)
return res.send({ "Message": "403" }); // PK / FK Violation
});
You can use batch insert
DB.transaction(async (t: Knex.Transaction) => {
return await t
.batchInsert("addon_name", addon_nameRecords)
.returning("id");
});