I'm using MySQLdb and Python to insert data into the local database.
The insert query is
for row in v1:
cur2.execute("""""INSERT INTO s_test (CO1,CO2, CO3, CO4, CO5) VALUES (%s, %s, %s, %s, %s);""""",(row[0],row[1], row[2],row[3], row[4]));
I get the error
IndexError: tuple index out of range
If I use the query below, I get no errors
for row in v1:
cur2.execute("INSERT INTO s_test (CO1) VALUES (%s);",(row[0]));
If I increase the number of columns into which I insert, I get the same error mentioned above.
What is the correct syntax?
I don't remember much about Python, but this sounds like you don't have 5 elements in the row array, more than a SQL error. Can you check that?
Assuming the data's entirely numerical, you could try this:
cur2.execute("""INSERT INTO s_test (CO1, CO2, CO3, CO4, CO5) VALUES ({0}, {1}, {2}, {3}, {4});""".format(row[0], row[1], row[2], row[3], row[4]);
Use 3 quotes instead of 5; replace the comma with a period; use the more modern .format string substitution.
Related
cmd="insert into People(ID,Name,Age,Gender,CriminalRecords)values("+str(Id)+","+str(Name)+","+str(Age)+","+str(Gender)+","+str(Criminal)+")"
The values that are strings need quotes around them.
But it's better to use a query with parameters rather than concatenating variables.
cmd="insert into People(ID,Name,Age,Gender,CriminalRecords)values(%s, %s, %s, %s, %s)"
cursor.execute(cmd, (Id, Name, Age, Gender, Criminal))
conn.commit()
Replace cursor and conn with the actual variables containing your cursor and connection objects.
I am trying to insert data from xlsx file into mysqdl table. I want to insert data in table and if there is a duplicate on primary keys, I want to update the existing data otherwise insert. I have written the script already but I realized it is too much work and using pandas it is quick. How can I achieve it in pandas?
#!/usr/bin/env python3
import pandas as pd
import sqlalchemy
engine_str = 'mysql+pymysql://admin:mypass#localhost/mydb'
engine = sqlalchemy.create_engine(engine_str, echo=False, encoding='utf-8')\
file_name = "tmp/results.xlsx"
df = pd.read_excel(file_name)
I can think of two options, but number 1 might be cleaner/faster:
1) Make SQL decide on the update/insert. Check this other question. You can iterate by rows of your 'df', from i=1 to n. Inside the loop for the insertion you can write something like:
query = """INSERT INTO table (id, name, age) VALUES(%s, %s, %s)
ON DUPLICATE KEY UPDATE name=%s, age=%s"""
engine.execute(query, (df.id[i], df.name[i], df.age[i], df.name[i], df.age[i]))
2) Define a python function that returns True or False when the record exists and then use it in your loop:
def check_existence(user_id):
query = "SELECT EXISTS (SELECT 1 FROM your_table where user_id_str = %s);"
return list(engine.execute(query, (user_id, ) ) )[0][0] == 1
You could iterate over rows and do this check before inserting
Please also check the solution in this question and this one too which might work in your case.
Pangres is the tool for this job.
Overview here:
https://pypi.org/project/pangres/
Use the function pangres.fix_psycopg2_bad_cols to "clean" the columns in the DataFrame.
Code/usage here:
https://github.com/ThibTrip/pangres/wiki
https://github.com/ThibTrip/pangres/wiki/Fix-bad-column-names-postgres
Example code:
# From: <https://github.com/ThibTrip/pangres/wiki/Fix-bad-column-names-postgres>
import pandas as pd
# fix bad col/index names with default replacements (empty string for '(', ')' and '%'):
df = pd.DataFrame({'test()':[0],
'foo()%':[0]}).set_index('test()')
print(df)
test() foo()%
0 0
# clean cols, index w/ no replacements
df_fixed = fix_psycopg2_bad_cols(df)
print(df_fixed)
test foo
0 0
# fix bad col/index names with custom replacements - you MUST provide replacements for '(', ')' and '%':
# reset df
df = pd.DataFrame({'test()':[0],
'foo()%':[0]}).set_index('test()')
# clean cols, index w/ user-specified replacements
df_fixed = fix_psycopg2_bad_cols(df, replacements={'%':'percent', '(':'', ')':''})
print(df_fixed)
test foopercent
0 0
Will only fix/correct some of the bad characters:
Replaces '%', '(' and ')' (characters that won't play nicely or even at all)
But, useful in that it handles cleanup and upsert.
(p.s., I know this post is over 4 years old, but still shows up in Google results when searching for "pangres upsert determine number inserts and updates" as the top SO result, dated May 13, 2020.)
When using Pandas no iteration is needed. Isn't that faster?
df = pd.read_csv(csv_file,sep=';',names=['column'])
df.to_sql('table', con=con, if_exists='append', index=False, chunksize=20000)
My Python program connects to BigQuery and fetching data which I want to insert into a MySQL table. It's successfully fetching the results from BigQuery. It's also successfully connecting to MySQL DB but it's not inserting the data. I see its complaining for the row[1].
What's the right way to insert the values from BigQuery response into MySQL table columns?
query_data = {mybigquery}
query_response = query_request.query(projectId='myprojectid',body=query_data).execute()
for row in query_response['rows']:
cursor.execute ("INSERT INTO database.table VALUES ('row[0]','row[1]','row[2]','row[3]','row[4]');")
Also, I tried to use
cursor.execute ("INSERT INTO database.table VALUES (%s,%s,%s,%s,%s);")
or
cursor.execute ("INSERT INTO database.table VALUES (row[0],row[1],row[2],row[3],row[4]);")
But in all it fails while inserting values in mysql table
String literals
Regarding the original question, the issue lies with quoting your variables. This causes the execute function to treat them as string literals rather than getting the values from them.
As suggested by #Herman, to properly execute the SQL statement with the values I think you intend, you would need something more like this:
query_data = {mybigquery}
statement = 'INSERT INTO database.table VALUE (%s, %s, %s);'
response = query_request.query(projectId='myprojectid', body=query_data).execute()
rows = response['rows']
for row in rows:
values = (row[0], row[1], row[2])
cursor.execute(statement, values)
BigQuery query JSON
Keep in mind however that the above will not work out of the box as row in the code above does not conform to the response received from the BigQuery Job: query API.
In this API, rows is an array of row objects. Each row object has a property f which is an array of fields. Lastly, each field has a property v which is the value of this field.
To get the value of second field in a row, you should use row['f'][1]['v']. Since you require a tuple or list for the params argument of the cursor.execute() method, you could get a list of field values using list comprehension as follows:
for row in rows:
values = [field['v'] for field in row['f]]
Sanitize values before inserting
The TypeError you get once correctly reading the field values may be raised because execute or str cannot convert a value to a string properly. One of the significant differences between BigQuery and MySQL is that a value in BigQuery can be a record with multiple values of its own. To ensure this gets inserted properly, you must sanitize those values yourself prior to inserting them. If the value is a list or dict, it cannot be stored in MySQL without being serialized in some way like with the str method.
Example
def sanitize(value):
if type(value) is list:
return str(value)
if type(value) is dict:
return str(value)
# this may be required for other types
return value
data = {mybigquery}
statement = 'INSERT INTO database.table VALUE (%s, %s, %s);'
response = request.query(projectId='projid', body=data).execute()
for row in response['rows']:
values = [sanitize(field['v']) for field in row['f']]
cursor.execute(statement, values)
This is very basic sanitation. You should really validate all field values and ensure that they will be properly converted to MySQL types and not simply insert an array of values.
What is the error message? It should be something like:
cursor.execute(
"INSERT INTO database.table VALUES (%s, %s, %s, %s, %s)", row[0:5])
I need to insert ~150 simple rows (an id, and a static status of 'discard'). I have a string of the ids:
'123', '234r', '345', '456xyz'...
What's the simplest way to insert rows using this string of ids?
It seems like maybe there's some way to split the string on commas and... create a temp table to ...? I don't know - it just seems like this is the kind of thing that MySQL often manages to pull off in some cool, expedient way.
An example how to do create an INSERT statement with a few lines of PHP:
<?php
// copy your string of ids into this variable
$input = "'123', '234r', '345', '456xyz'";
// modify next line to get your desired filename
$filename = 'insert.sql'
// modify next line to your table name
$insert_statement = "INSERT INTO your_table_name (id, status) VALUES \n" .
'(' . implode(", 'discard')\n(", explode(', ', $input)) . ", 'discard');\n";
file_put_contents($filename, $insert_statement);
?>
Note
This is for this special use case. If the string of ids contains some special characters like single quotes, then this simple approach will fail.
The one way is to create CSV file with appropriate records and upload it at once to mysql.
Please follow this tutorial: http://www.mysqltutorial.org/import-csv-file-mysql-table/
Very similar to this question MySQL Dynamic Query Statement in Python
However what I am looking to do instead of two lists is to use a dictionary
Let's say i have this dictionary
instance_insert = {
# sql column variable value
'instance_id' : 'instnace.id',
'customer_id' : 'customer.id',
'os' : 'instance.platform',
}
And I want to populate a mysql database with an insert statement using sql column as the sql column name and the variable name as the variable that will hold the value that is to be inserted into the mysql table.
Kind of lost because I don't understand exactly what this statement does, but was pulled from the question that I posted where he was using two lists to do what he wanted.
sql = "INSERT INTO instance_info_test VALUES (%s);" % ', '.join('?' for _ in instance_insert)
cur.execute (sql, instance_insert)
Also I would like it to be dynamic in the sense that I can add/remove columns to the dictionary
Before you post, you might want to try searching for something more specific to your question. For instance, when I Googled "python mysqldb insert dictionary", I found a good answer on the first page, at http://mail.python.org/pipermail/tutor/2010-December/080701.html. Relevant part:
Here's what I came up with when I tried to make a generalized version
of the above:
def add_row(cursor, tablename, rowdict):
# XXX tablename not sanitized
# XXX test for allowed keys is case-sensitive
# filter out keys that are not column names
cursor.execute("describe %s" % tablename)
allowed_keys = set(row[0] for row in cursor.fetchall())
keys = allowed_keys.intersection(rowdict)
if len(rowdict) > len(keys):
unknown_keys = set(rowdict) - allowed_keys
print >> sys.stderr, "skipping keys:", ", ".join(unknown_keys)
columns = ", ".join(keys)
values_template = ", ".join(["%s"] * len(keys))
sql = "insert into %s (%s) values (%s)" % (
tablename, columns, values_template)
values = tuple(rowdict[key] for key in keys)
cursor.execute(sql, values)
filename = ...
tablename = ...
db = MySQLdb.connect(...)
cursor = db.cursor()
with open(filename) as instream:
row = json.load(instream)
add_row(cursor, tablename, row)
Peter
If you know your inputs will always be valid (table name is valid, columns are present in the table), and you're not importing from a JSON file as the example is, you can simplify this function. But it'll accomplish what you want to accomplish. While it may initially seem like DictCursor would be helpful, it looks like DictCursor is useful for returning a dictionary of values, but it can't execute from a dict.