I'm really trying hard to execute an INSERT query by passing named placeholders and it doesn't work. This is the error I get: execute() got an unexpected keyword argument 'vid'
Here is my code:
query = """INSERT INTO `index`
(`video_id`, `label`, `img`, `context`, `genre`, `kind`) VALUES
({vid}, {label}, {img}, {context}, {genre}, {kind}) ON DUPLICATE KEY UPDATE
label={label}, img={img}, context={context}, genre={genre}, kind={kind}
"""
self.db.insert(query, {
'vid': video_id,
'label': label,
'img': img_source,
'context': json.dumps(context),
'genre': slugify(self.genre),
'kind': self.kind
})
in my db handler:
def insert(self, query, params):
print(query.format(**params))
try:
self.cursor.execute(query, **params)
self.connection.commit()
except Exception as e:
print(e)
self.connection.rollback()
quit()
Related
Trying to handle a Py4JJavaError that is raised when entering invalid sql code to spark.sql(). My function is as follows:
import py4j
def sql_to_df_and_create_view(sql_string: str, view_name: str):
''' Takes in a SQL command in string format and creates a spark df from the command.
Secondly, creates a temporary view with name specified under view_name parameter. Ensures that
the SQL code is valid and that the view is created. '''
try:
df = spark.sql(sql_string)
except py4j.protocol.Py4JJavaError:
raise Exception(f'Invalid SQL code passed in by {sql_string}.')
return
df.createOrReplaceTempView(view_name)
return df
With the error code being:
org.apache.spark.sql.catalyst.parser.ParseException:
---------------------------------------------------------------------------
Py4JJavaError Traceback (most recent call last)
/databricks/spark/python/pyspark/sql/utils.py in deco(*a, **kw)
62 try:
---> 63 return f(*a, **kw)
64 except py4j.protocol.Py4JJavaError as e:
/databricks/spark/python/lib/py4j-0.10.7-src.zip/py4j/protocol.py in get_return_value(answer,
gateway_client, target_id, name)
327 "An error occurred while calling {0}{1}{2}.\n".
--> 328 format(target_id, ".", name), value)
329 else:
Py4JJavaError: An error occurred while calling o213.sql.
: org.apache.spark.sql.catalyst.parser.ParseException:
mismatched input 'sd' expecting {'(', 'SELECT', 'FROM', 'ADD', 'DESC', 'WITH', 'VALUES',
'CREATE', 'TABLE', 'INSERT', 'DELETE', 'DESCRIBE', 'EXPLAIN', 'SHOW', 'USE', 'DROP', 'ALTER',
'MAP', 'SET', 'RESET', 'START', 'COMMIT', 'ROLLBACK', 'MERGE', 'UPDATE', 'CONVERT', 'REDUCE',
'REFRESH', 'CLEAR', 'CACHE', 'UNCACHE', 'DFS', 'TRUNCATE', 'ANALYZE', 'LIST', 'REVOKE',
'GRANT', 'LOCK', 'UNLOCK', 'MSCK', 'EXPORT', 'IMPORT', 'LOAD', 'OPTIMIZE', 'COPY'}(line 1, pos
0)
== SQL ==
sd
^^^
When I run the function with invalid sql in the sql_string argument, the error is not handled and it still raises the same error code rather than 'Exception(f'Invalid SQL code passed in by {sql_string}.')'. If anyone can figure out why this is not being handled correctly I would be v grateful :)
I try:
cursor = client.cursor()
query = "INSERT INTO t_table (column_1,column_2) VALUES (%s)"
values = ['temp', 'temp']
cursor.execute(query, values)
client.commit()
print(cursor.rowcount, "record inserted")
finally:
client.close()
The error
"MySQLdb._exceptions.ProgrammingError: not all arguments converted
during bytes formatting"
keeps appearing. I have gone through the syntax error and GEOMETRY field error but I'm stuck at this one.
For two values you need also to placeholders
try:
cursor = client.cursor()
query = "INSERT INTO t_table (column_1,column_2) VALUES (%s,%s)"
values = ['temp', 'temp']
cursor.execute(query, values)
client.commit()
print(cursor.rowcount, "record inserted")
finally:
client.close()
When executing the following code in order to insert multiple values through a variable from python into mysql, I get:
'Incorrect number of arguments executing prepared statement ' error after executing 'result = cursor.executemany(sql_insert_query, records_to_insert)'
if i remove 'prepared=True', the error becomes:
'Not all parameters were used in the SQL statement'
import mysql.connector
connection = mysql.connector.connect(host='localhost',
database='majorprediction',
user='root',
password='')
records_to_insert = [ ('x') ,
('y'),
('z') ]
sql_insert_query = """ INSERT INTO majorpred (Major)
VALUES (%s) """
cursor = connection.cursor(prepared=True)
result = cursor.executemany(sql_insert_query, records_to_insert)
connection.commit()
Can anyone specify where is the problem?
You are passing a list of characters instead of tuples. For instance, if you try and run:
for record in records_to_insert:
print(record, isinstance(record, str), isinstance(record, tuple))
You will get:
x True False
y True False
z True False
To create tuples with a single element in python you can do the following:
records_to_insert = [
('x',),
('y',),
('z',)
]
If you have a list of parameters and want to cast all of them to tuple you can do as follows:
list_of_elements = list("list_of_characters")
tuples = [
tuple(e) for e in list_of_elements
]
Hope this helps!
I'm using this scrapy pipeline. If there is any error in the sql in the insert_record function, it fails silently. For example, if a column name is miss-spelled, like this
def _insert_record(self, tx, item):
print "before tx.execute"
result = tx.execute(
""" INSERT INTO table(col_one, col_typo, col_three) VALUES (1,2,3)"""
)
print "after tx.execute"
if result > 0:
self.stats.inc_value('database/items_added')
then nothing is output after "before execute". There is a handle_error method but that's not called either. How can I catch and handle such errors?
Just needed to surround it with try...except
try:
result = tx.execute(
"""INSERT INTO table(col_one, col_typo, col_three) VALUES (1,2,3)"""
)
except Exception,e:
print str(e)
I have a python script which everytime a wait_for_page call is made it writes the time it took to wait for the page to a database. The query is below:
conn = mysql.connector.connect(**config)
connect = conn.cursor()
params = {'build': self.tc.tag, 'page': unicode(self), 'object_id': self.object_id, 'page_header':
self.page_header, 'interval': t.interval, 'timestamp': timestamp}
query = u'INSERT INTO page_load_times (build, page, object_id, page_header, elapsed_time, date_run) ' \
'VALUES (%(build)s, %(page)s, %(object_id)s, %(page_header)s, %(interval)s, %(timestamp)s)'
connect.execute(query, params)
conn.commit()
conn.close()
Occasionally, when this runs, I get an error which says:
"Failed processing pyformat-parameters; %s" % err)
ProgrammingError: Failed processing pyformat-parameters; 'MySQLConverter'
object has no attribute '_list_to_mysql'
I know what is causing this, just uncertain how to go about fixing it. The 'page': unicode(self) param occasionally gets a list as a result.
In an attempt to fix this, I tweaked the above script to extract the list into a string, with the following:
page_list = u'{}'.format(self)
page_results = "('%s')" % "','".join(page_list)
params = {'build': self.tc.tag, 'page': page_results, 'object_id': self.object_id, 'page_header':
self.page_header, 'interval': t.interval, 'timestamp': timestamp}
When I run this, the error I am getting now is that the data is too long for the field. I debug it, to find that my page results has each character parsed out individually looking like so:
u'(\\'A\\',\\'p\\',\\'p\\',\\'M\\',\\'a\\',\\'i\\',\\'n\\',\\'M\\',\\'e\\',\\'n\\',\\'u\\',\\':\\',\\' \\',\\'N\\',\\'o\\',\\'n\\',\\'e\\')'
So the solution was to do the following, which takes the page_header and if it is in the instance of list to make that list a string:
conn = mysql.connector.connect(**config)
connect = conn.cursor()
page_list = u'{}'.format(self)
page_header_list = u'{}'.format(self.page_header)
if isinstance(page_header_list, list):
page_header_list = ', '.join(page_header_list)[0:100]
params = {'build': self.tc.tag, 'page': page_list, 'object_id': self.object_id,
'page_header': page_header_list, 'interval': t.interval, 'timestamp': timestamp}
query = u'INSERT INTO page_load_times (build, page, object_id, page_header, elapsed_time, date_run) ' \
'VALUES (%(build)s, %(page)s, %(object_id)s, %(page_header)s, %(interval)s, %(timestamp)s)'
connect.execute(query, params)
conn.commit()
conn.close()
Thank you #DarthOpto, you gave me lots of light. I solved it by putting srt around the variable:
params = {'build': self.tc.tag, 'page': srt(page_list),'object_id': self.object_id,
'page_header': srt(page_header_list),'interval': t.interval,'timestamp': timestamp}