I am trying to insert the dictionary which is retrieved from a rest API into Snowflake column of variant datatype using the below python script
from sqlalchemy import create_engine
import urllib
import requests
import json
engine = create_engine(
'snowflake://{user}:{password}#{account_identifier}/'.format(
user='UserName',
password='pwd',
account_identifier='account_Identifier'
)
)
jval={"title":"value", "address":[{ "Road":"st xavier's","landmark":"D' Pauls"}]}
try:
connection = engine.connect()
results = connection.execute(
"INSERT INTO db_name.schema_name.sqlalchemy(jval) (select PARSE_JSON('" + json.dumps(jval) + "'))").fetchone()
print(results[0])
finally:
connection.close()
engine.dispose()
But ending up with the following error
File "C:\Users\PycharmProjects\snowflake\venv\lib\site-packages\snowflake\connector\errors.py", line 207, in default_errorhandler
raise error_class(
sqlalchemy.exc.ProgrammingError: (snowflake.connector.errors.ProgrammingError) 001003 (42000): SQL compilation error:
syntax error line 1 at position 133 unexpected 's'.
syntax error line 1 at position 134 unexpected '", "'.
[SQL: INSERT INTO db_name.schema_name.sqlalchemy(jval) (select PARSE_JSON('{"title": "value", "address": [{"Road": "st xavier's", "landmark": "D' Pauls"}]}'))]
(Background on this error at: https://sqlalche.me/e/14/f405)
I take it this is causing due to the single quote (') present in the string. how to handle this error? the same error comes even when using snowflake.connector library as well.
Use parameter binding so that the quoting is handled automatically.
from sqlalchemy import text
...
results = connection.execute(
text(
'INSERT INTO db_name.schema_name.sqlalchemy(jval) (select PARSE_JSON(:some_json))'
),
{'some_json': jval},
).fetchone()
Related
I am a novice when it comes to Python and I am trying to import a .csv file into an already existing MySQL table. I have tried it several different ways but I cannot get anything to work. Below is my latest attempt (not the best syntax I'm sure). I originally tried using ‘%s’ instead of ‘?’, but that did not work. Then I saw an example of the question mark but that clearly isn’t working either. What am I doing wrong?
import mysql.connector
import pandas as pd
db = mysql.connector.connect(**Login Info**)
mycursor = db.cursor()
df = pd.read_csv("CSV_Test_5.csv")
insert_data = (
"INSERT INTO company_calculations.bs_import_test(ticker, date_updated, bs_section, yr_0, yr_1, yr_2, yr_3, yr_4, yr_5, yr_6, yr_7, yr_8, yr_9, yr_10, yr_11, yr_12, yr_13, yr_14, yr_15)"
"VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)"
)
for row in df.itertuples():
data_inputs = (row.ticker, row.date_updated, row.bs_section, row.yr_0, row.yr_1, row.yr_2, row.yr_3, row.yr_4, row.yr_5, row.yr_6, row.yr_7, row.yr_8, row.yr_9, row.yr_10, row.yr_11, row.yr_12, row.yr_13, row.yr_14, row.yr_15)
mycursor.execute(insert_data, data_inputs)
db.commit()
Error Message:
> Traceback (most recent call last): File
> "C:\...\Python_Test\Excel_Test_v1.py",
> line 33, in <module>
> mycursor.execute(insert_data, data_inputs) File "C:\...\mysql\connector\cursor_cext.py",
> line 325, in execute
> raise ProgrammingError( mysql.connector.errors.ProgrammingError: Not all parameters were used in the SQL statement
MySQL Connector/Python supports named parameters (which includes also printf style parameters (format)).
>>> import mysql.connector
>>> mysql.connector.paramstyle
'pyformat'
According to PEP-249 (DB API level 2.0) the definition of pyformat is:
pyformat: Python extended format codes, e.g. ...WHERE name=%(name)s
Example:
>>> cursor.execute("SELECT %s", ("foo", ))
>>> cursor.fetchall()
[('foo',)]
>>> cursor.execute("SELECT %(var)s", {"var" : "foo"})
>>> cursor.fetchall()
[('foo',)]
Afaik the qmark paramstyle (using question mark as a place holder) is only supported by MariaDB Connector/Python.
I am getting this error: DatabaseError: Execution failed on sql 'SELECT name FROM sqlite_master WHERE type='table' AND name=?;': Not all parameters were used in the SQL statement. when trying to convert my dataframe to sql
My connection variable:
con = mysql.connector.connect(
host="****",
port="****",
database="*****",
user="*****",
password="*****"
)
My try to convert it to sql:
df.to_sql('menageiro2',con)
Note: I am using:
import pandas as pd
import sqlalchemy
import mysql.connector
The reference says con: sqlalchemy.engine.(Engine or Connection) or sqlite3.Connection. You appear to be passing in a mysql connection instead of a SQLAlchamy engine (that you connected to MySQL):
con = sqlalchemy.create_engine(
'mysql+mysqlconnector://<user>:<password>#<host>:<port>/<default_db>...')
I'm having a heck of a time getting the mysql.connector module to work. I'd really like to find some accurate documentation on it. By hit and by miss, I have arrived here.
Traceback (most recent call last):
File "update_civicrm_address.py", line 80, in <module>
cursor.execute(mysql_select_query, address_id)
File "/home/ubuntu/.local/lib/python3.6/site-packages/mysql/connector/cursor.py", line 1210, in execute
msg="Incorrect number of arguments " \
mysql.connector.errors.ProgrammingError: 1210: Incorrect number of arguments executing prepared statement
Here is the program (it's a bit messy because I have tried so many things to get it to work). Aside from the fact that the update is not working at all, what is causing the error? There is only one parameter and it is accounted for.
import sys
import mysql.connector
import csv
import os
from mysql.connector import Error
from mysql.connector import errorcode
#Specify the import file
try:
inputCSV = 'geocoded_rhode_island_export.csv'
#Open the file and give it a handle
csvFile = open(inputCSV, 'r')
#Create a reader object for the input file
reader = csv.reader(csvFile, delimiter = ',')
except IOError as e:
print("The input file ", inputCSV, " was not found", e)
exit()
try:
conn = mysql.connector.connect(host='localhost',
database='wordpress',
user='wp_user',
password='secret!',
use_pure=True)
cursor = conn.cursor(prepared=True)
except mysql.connector.Error as error:
print( "Failed to connect to database: {}".format(error))
exit()
try:
record_count = 0
for row in reader:
contact_id,address_id,last_name, first_name, middle_name, longitude, latitude = row
print(row)
#Update single record now
print(address_id)
cursor.execute(
"""
update civicrm_address
set
geo_code_1 = %s,
geo_code_2 = %s
where
id = %s
and
location_type_id = %s
""",
(longitude, latitude, address_id, 6)
)
conn.commit
print(cursor.rowcount)
print("Record updated successfully")
mysql_select_query = """
select
id,
geo_code_1,
geo_code_2
from
civicrm_address
where
id = %s
"""
input = (address_id)
cursor.execute(mysql_select_query, address_id)
record = cursor.fetchone()
print(record)
record_count = record_count + 1
finally:
print(record_count, " records updated")
#closing database connection.
if(conn.is_connected()):
conn.close()
print("connection is closed")
The is an error in the code
conn.commit
should be
conn.commit()
I'm getting the following errors when trying to decode this data, and the 2nd error after trying to compensate for the unicode error:
Error 1:
write.writerows(subjects)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u201c' in position 160: ordinal not in range(128)
Error 2:
with open("data.csv", encode="utf-8", "w",) as writeFile:
SyntaxError: non-keyword arg after keyword arg
Code
import requests
import json
import csv
from bs4 import BeautifulSoup
import urllib
r = urllib.urlopen('https://thisiscriminal.com/wp-json/criminal/v1/episodes?posts=10000&page=1')
data = json.loads(r.read().decode('utf-8'))
subjects = []
for post in data['posts']:
subjects.append([post['title'], post['episodeNumber'],
post['audioSource'], post['image']['large'], post['excerpt']['long']])
with open("data.csv", encode="utf-8", "w",) as writeFile:
write = csv.writer(writeFile)
write.writerows(subjects)
Using requests and with the correction to the second part (as below) I have no problem running. I think your first problem is due to the second error (is a consequence of that being incorrect).
I am on Python3 and can run yours with my fix to open line and with
r = urllib.request.urlopen('https://thisiscriminal.com/wp-json/criminal/v1/episodes?posts=10000&page=1')
I personally would use requests.
import requests
import csv
data = requests.get('https://thisiscriminal.com/wp-json/criminal/v1/episodes?posts=10000&page=1').json()
subjects = []
for post in data['posts']:
subjects.append([post['title'], post['episodeNumber'],
post['audioSource'], post['image']['large'], post['excerpt']['long']])
with open("data.csv", encoding ="utf-8", mode = "w",) as writeFile:
write = csv.writer(writeFile)
write.writerows(subjects)
For your second, looking at documentation for open function, you need to use the right argument names and add the name of the mode argument if not positional matching.
with open("data.csv", encoding ="utf-8", mode = "w") as writeFile:
I tried all possible means. I included backQuotes for the string as suggested some in stack but nothing worked. It repeats the error as usual.
I also tried some queries that worked in other python files still it shows the same. I also tried queries with string without hyphens even it didn't work. I cant find out whats the problem here.
import MySQLdb
import sys
from PyQt4 import QtCore, QtGui, uic
qtCreatorFile = "Studisplay.ui" # Enter file here.
Ui_MainWindow1, QtBaseClass = uic.loadUiType(qtCreatorFile)
class stuDisplay(QtGui.QMainWindow, Ui_MainWindow1,QtGui.QTableWidget):
def __init__(self,ID):
#super(stuDisplay, self).__init__(parent)
QtGui.QMainWindow.__init__(self)
Ui_MainWindow1.__init__(self)
QtGui.QWidget.__init__(self)
self.setupUi(self)
obj = MySQLdb.connect("localhost", "root", "1234567", "python")
#The value of ID here is 14-VEC-244 I also tried `14-VEC-244` but did not work
sql = 'SELECT MEMname FROM Borrowed WHERE MemberID ='+ str(ID)
cursor = obj.cursor()
cursor.execute(sql)
name=cursor.fetchone()
print name
I get this error:
Traceback (most recent call last): File
"/home/gautham/PycharmProjects/LIBALERT/Login.py", line 105, in
pushButton_clicked
self.call = StuSecond.stuDisplay(StuID) File "/home/gautham/PycharmProjects/LIBALERT/StuSecond.py", line 22, in
init
cursor.execute(sql) File "/usr/lib/python2.7/dist-packages/MySQLdb/cursors.py", line 226, in
execute
self.errorhandler(self, exc, value) File "/usr/lib/python2.7/dist-packages/MySQLdb/connections.py", line 36, in
defaulterrorhandler
raise errorvalue
_mysql_exceptions.OperationalError: (1054, "Unknown column 'VEC' in 'where clause'")
You're passing a string into your WHERE clause, so it must be quoted within the query string that gets passed to the database, something like so:
sql = "SELECT MEMname FROM Borrowed WHERE MemberID = '" + str(ID) + "'"
so that the finished string looks like
sql = "SELECT MEMname FROM Borrowed WHERE MemberID = '14-VEC-244'"
(Note that the single quotes are "forward" quotes, not backticks.)
This would also be an excellent application for a prepared statement; unfortunately I am not familiar with pyqt and so cannot advise you there.