how to handle very long SQL INSERT statement in mysql - mysql

I am using the following (python) code to generate a (MySQL) SQL INSERT statement (there are more columns, I left them out for simplicity):
mylist = [('1', '2', '3'),
('4', '5', '6'),
.
.
.
('7', '8', '9')]
sql_statement = "insert into mytable (col1, col2, col3) values "
for i in mylist:
if sql_statement == "insert into mytable (col1, col2, col3) values ":
# append this for the 1st element
sql_statement += "(" + i[0] + ", " + i[1] + ", " + i[2] + ")"
else:
# append this for everything else
sql_statement += ", (" + i[0] + ", " + i[1] + ", " + i[2] + ")"
which results in a string like the following:
sql_statement = "insert into mytable (col1, col2, col3) values (1, 2, 3), (4, 5, 6), ... (7, 8, 9)"
I then use sql_statement to execute the sql statement.
the issue with this approach is that the sql_statement string is getting to long and the insert does not consider all data.
any suggestions how to handle this?
UPDATE: prepared statement is the way to go. with that the (python) code looks like this:
sql_statement = "insert into mytable (col1, col2, col3) values (%s, %s, %s)"
for i in mylist:
cursor.execute(sql_statement, i)


Does the python library you are using supported prepared parameterized queries? I've found the performance difference between multi-value inserts such as this and repeated executions of prepared statements (in .Net at least) to be minimal in all but extreme cases. (In those cases, a mix of the two is optimal.)
Alternatively, just keep track of your query length, execute before it gets too big, and reinitialize the string & continue until all rows are handled.


Create a transaction and do insert one by one. and finally commit it. So only in one call all insert operation commit.

Related

If Concat insert result

I have a problem, I am making an insert using the concat to generate multiple inserts, however I realize that a null field is affecting all the queries, since the records that have null in their field, their insert is not generated correctly , now when integrating an IF() validation in the value of the field where the problem is caused, mysql sends me an error therefore the query is not executed
This is my query insert
SELECT concat ( "INSERT INTO `erp_gamma`.`partner` (`n_partner_id`, `n_activo`, `c_apellido_materno`, `c_apellido_paterno`, `c_correo_electronico`, `n_dias_credito`, `c_identificador`, `c_json`, `d_limite_credito`, `c_nombre`, `n_persona_fisica`, `c_razon_social`, `c_rfc`, `n_status`, `n_corporativo_sucursal_id`, `n_partner_tipo_id`, `c_codigo_postal`, `n_numero_telefonico`, `create_date`, `update_date`, `c_cp_fiscal`, `c_regimen_fiscal`, `c_sistema_id_serie_numerico`, `n_sistema_id_numerico`, `c_id_migracion`) VALUES (NULL, '1', '",partner.contacto_nombre,"', '",partner.contacto_nombre,"', '",partner.contacto_email,"', '",partner.control_dias_credito,"', '11111', '{\"id_calle_cliente_md\":\"",partner.contacto_calle,"\",\"id_colonia_cliente_md\":\"",partner.contacto_colonia,"\",\"id_ciudad_cliente_md\":\"",partner.contacto_ciudad,"\",\"id_estado_cliente_md\":\"",partner.contacto_estado,"\",\"id_cp_cliente_md\":\"",partner.contacto_cp,"\",\"id_telefono2_cliente_md\":\"",partner.contacto_telefono2,"\",\"id_celular_cliente_md\":\"",partner.contacto_telefono3,"\",\"id_fax_cliente_md\":\"",partner.contacto_fax,"\",\"id_web_cliente_md\":\"",partner.contacto_web,"\",\"id_tipoprecio_cliente_md\":\"",partner.control_tipo_precio_default,"\",\"id_calle_rfc_cliente_md\":\"",partner.factura_calle,"\",\"id_colonia_rfc_cliente_md\":\"",partner.factura_colonia,"\",\"id_ciudad_rfc_cliente_md\":\"",partner.factura_ciudad,"\",\"id_estado_rfc_cliente_md\":\"",partner.factura_estado,"\"}', '",partner.control_limite_credito,"', '",partner.contacto_nombre,"', '1', '",partner.factura_nombre,"', '",partner.factura_rfc,"', '1', '122', '1', '",partner.contacto_cp,"', '",partner.contacto_telefono1,"', now(), now(), '",partner.factura_cp,"', "IF(partner.factura_regimen_fiscal IS NULL," ", partner.factura_regimen_fiscal)", 'TP1_20', '20', '",partner.sku,"' );")
FROM partner
WHERE partner.tipo_partner = 1 AND partner.status =1;
I hope the inserts can be generated with the records I need

I post the query as it is as i never can test it
i added a lot of spaces, but i think it is more a problem from mysql workbench, as that big concat, aren't usually done, so the parser will get into trouble
The misstale you made is that you have is
IF(partner.factura_regimen_fiscal IS NULL," ", partner.factura_regimen_fiscal)
must be in commas, as it is ntpart if the string that you are combining
SELECT concat ( "INSERT INTO `erp_gamma`.`partner` (`n_partner_id`, `n_activo`, `c_apellido_materno`, `c_apellido_paterno`, `c_correo_electronico`, `n_dias_credito`, `c_identificador`, `c_json`, `d_limite_credito`, `c_nombre`, `n_persona_fisica`, `c_razon_social`, `c_rfc`, `n_status`, `n_corporativo_sucursal_id`, `n_partner_tipo_id`, `c_codigo_postal`, `n_numero_telefonico`, `create_date`, `update_date`, `c_cp_fiscal`, `c_regimen_fiscal`, `c_sistema_id_serie_numerico`, `n_sistema_id_numerico`, `c_id_migracion`) VALUES (NULL, '1', '"
,partner.contacto_nombre,"', '",partner.contacto_nombre,"', '",partner.contacto_email,"', '"
,partner.control_dias_credito,"', '11111', '{\"id_calle_cliente_md\":\" "
,partner.contacto_calle,"\",\"id_colonia_cliente_md\":\" "
,partner.contacto_colonia," \",\"id_ciudad_cliente_md\":\" ",partner.contacto_ciudad," \",\"id_estado_cliente_md\":\" "
,partner.contacto_estado,"\",\"id_cp_cliente_md\":\" ",partner.contacto_cp," \",\"id_telefono2_cliente_md\":\" "
,partner.contacto_telefono2," \",\"id_celular_cliente_md\":\" ",partner.contacto_telefono3," \",\"id_fax_cliente_md\":\" "
,partner.contacto_fax," \",\"id_web_cliente_md\":\" ",partner.contacto_web," \",\"id_tipoprecio_cliente_md\":\" "
,partner.control_tipo_precio_default," \",\"id_calle_rfc_cliente_md\":\" ",partner.factura_calle," \",\"id_colonia_rfc_cliente_md\":\" "
,partner.factura_colonia," \",\"id_ciudad_rfc_cliente_md\":\" ",partner.factura_ciudad," \",\"id_estado_rfc_cliente_md\":\" "
,partner.factura_estado," \"}', ' ",partner.control_limite_credito,"', '",partner.contacto_nombre
,"', '1', '",partner.factura_nombre,"', '",partner.factura_rfc,"', '1', '122', '1', '"
,partner.contacto_cp,"', '",partner.contacto_telefono1,"', now(), now(), '"
,partner.factura_cp,"', ",IF(partner.factura_regimen_fiscal IS NULL," ", partner.factura_regimen_fiscal),", 'TP1_20', '20', '",partner.sku,"' );")
FROM partner
WHERE partner.tipo_partner = 1 AND partner.status =1;

Try using a case statement formatted something like this;
case when partner.factura_regimen_fiscal IS NULL then "" else
partner.factura_regimen_fiscal end

Error running SQL statement generated in code

String query = "INSERT INTO `new_db2`(`name`, `price`, `add_date`, `image`) " + "VALUES ('"+name+"','"+p_price+"','"+date+"','"image"')";
i have ';' error expected in this Sql query please help me to solve this

I don't know the language you are using (I suppose Java) but may be you should write the statement like this:
String query = "INSERT INTO `new_db2`(`name`, `price`, `add_date`, `image`) " + "VALUES ('"+name+"','"+p_price+"','"+date+"','"+image+"')";
(lack of + before and after image)

Any way to auto add a quote mark around a character variable during implode for mysql?

Here it is..
Foreach ($data as $x) {
$mydata = implode( ", ", $x);
$sql = "INSERT INTO `wp_realty_listingsdb` (`listingsdb_id`, `user_id`,
`class_id`, `MLS`, `DOM`, `Zip`, `Status`) VALUES($id, 1, 1, $mydata);";
echo "$sql<br>";
$id++;
}
Keep in mind this is a simplified example and there will be over 200 fields being imploded for insertion.. so there might be as many as 100+ character variables that will require tick encapsulation so if the implode won't do it then it could get complicated..
End result of echo the resultant sql..
Line 1:
INSERT INTO `wp_realty_listingsdb` (`listingsdb_id`, `user_id`,
`class_id`, `MLS`, `DOM`, `Zip`, `Status`) VALUES(2, 1, 1, 1475566, 626,
89005, Sold);
Line 2:
INSERT INTO `wp_realty_listingsdb` (`listingsdb_id`, `user_id`,
`class_id`, `MLS`, `DOM`, `Zip`, `Status`) VALUES(3, 1, 1, 1485995, 492,
89005, 'Sold');
PROBLEM: To use php insert the character variables require that it have a tick on each side of the variable like 'Sold' as you see in line 1 it will not put the tick on implode.. Line 2 is an example of where i manually added the tick.. and it works fine.. Is there anyway to have the implode add the ticks around any character variables... w/o extensive additional programming.

$xt = array_map(function($x){ return "'$x'";}, $x);
$mydata = implode( ", ", $xt);
Apart from that the code is probably vulnerable to SQL injection.

Perl many insert statments into mysql database

I currently have code for perl that looks like this:
#valid = grep { defined($column_mapping{ $headers[$_] }) } 0 .. $#headers;
...
my $sql = sprintf 'INSERT INTO tablename ( %s ) VALUES ( %s )',
join( ',', map { $column_mapping{$_} } #headers[#valid] ),
join( ',', ('?') x scalar #valid);
my $sth = $dbh->prepare($sql);
...
my #row = split /,/, <INPUT>;
$sth->execute( #row[#valid] );
(Taken from mob's answer to a previous question.)
That is basically dynamically building a sql insert statement from csv data, and only allowing the csv data with proper headers from my column mapping to be picked.
I have been looking for examples on how to do an insert statment with multiple rows of data at once.
My perl script needs to run around a few hundred million insert statments, and doing it one at a time seems really slow, especially since the server I am running it on only has 6gb of ram and a slowish internet connection.
Is there a way I can upload more than 1 row at a time of data? So one insert statment uploads maybe 50 rows, or 100 rows at once? I cant find out how with perl DBI.

my $sql_values = join( ' ', ('(?, ?, ?)') x scalar(#array) );
As said before, then you can just flatten it.

You can insert multiple rows at once with the same syntax as in regular SQL, but you need to build your INSERT statemtent properly with Perl. Perl's slice() may help you:
Suppose you have 7 rows of data and want to insert them in chunks of 3 rows. "Regular" SQL would be like so:
insert into T (col1, col2) values ( 1, 2), ( 3, 4), ( 5, 6);
insert into T (col1, col2) values ( 7, 8), ( 9,10), (11,12);
insert into T (col1, col2) values (13,14);
Let's suppose your perl structure is like this:
my $values = [ [1,2], [3,4], ..., [13,14] ];
If it's not, bring it into this shape. Now:
use constant CHUNKSIZE => 3;
my $stmt = sprintf( 'insert into T (col1, col2) values %s',
join(',', '(?,?)' x CHUNKSIZE) );
# $stmt is now 'insert into T (col1, col2) values (?,?),(?,?),(?,?)'
my $sth = $dbh->prepare($stmt);
while( my #chunk = splice( #{$values}, 0, CHUNKSIZE ) ) {
# #chunk has 3 elements (rows), or less for the last chunk
if (scalar #chunk == CHUNKSIZE) {
$sth->execute( #chunk ); # inserts 3 rows at once
} else {
# build and prepare a new statement for the remaining rows.
# in our sample there is only 1 remaining row.
$stmt = sprintf( 'insert into T (col1, col2) values %s',
join(',', '(?,?)' x scalar #chunk) );
$sth = $dbh->prepare($stmt);
$sth->execute( #chunk ); # inserts the last row
}
}

Inserting data into the mysql database from perl

I am trying to insert data into a MySQL database:
$response = $client->fql->query(
query => '
SELECT name, email, birthday, username, first_name, last_name, pic
FROM user
WHERE uid = me()
',
);
print join "\n Name:", sort map { $_->{name} } #$response;
$dbh->do("
INSERT INTO Users(SNo,Name,Email,Birthday,UserName,FirstName,LastName)
VALUES(1,
sort map { $_->{name} } #$response,
'imm\#gmail.com',
'1987/12/10',
'imm',
'imm',
'Dee')
");
$dbh->disconnect();
used the mysql query in one line.
This above print statement is printing the name correctly but why the above sql insert statement is not working?
I connect the db and after that i am receiving the value and printing in the browser is working.
Why does the mysql statement not accept the value?
When inserting the database is not working?

You should have a look at the official doc
and specially this :
# INSERT some data into 'foo'. We are using $dbh->quote() for
# quoting the name.
$dbh->do("INSERT INTO foo VALUES (1, " . $dbh->quote("Tim") . ")");
# Same thing, but using placeholders
$dbh->do("INSERT INTO foo VALUES (?, ?)", undef, 2, "Jochen");