I'm not familiar with SQL(or python to be honest), and i was wondering how to add variables into tables. This is what i tried:
import sqlite3
conn = sqlite3.connect('TESTDB.db')
c = conn.cursor()
c.execute('''CREATE TABLE IF NOT EXISTS table(number real, txt text)''')
r=4
c.execute('''INSERT INTO table(r,'hello')''')
conn.commit()
conn.close()
This doesn't work.I get the error, "TypeError: 'str' object is not callable"How do i make the table insert the variable r??
Thanks
You have to bind values to placeholders in a prepared statement. See examples in the documentation.
Trying to build a query string on the fly with unknown values inserted directly in it is a great way to get a SQL injection attack and should be avoided.
Your problem is that you don't inject r properly into your query. This should work:
c.execute('''INSERT INTO table(''' + str(r) + ''','hello')''')
cheers
Related
I'm writing a simple - or it should be simple - script to acquire tweets from Twitter's API (I have developer/app keys and am using the Tweepy interface, not scraping or anything of that sort - I may ditch Tweepy for something closer to the modern API but that is almost certainly not what's causing this issue here).
I have a MySQL instance which I connect to and can query just fine, until it comes time to insert the tweet - which has a lot of special characters, almost inevitably. To be clear, I am using the official Python driver/connector for MySQL.
import mysql.connector
from mysql.connector import errorcode
Now, I'm aware StackOverflow is LITTERED with threads where people get my exact error - simply stating to check the MySQL syntax manual. These threads, which aren't all that old (and I'm not using the latest Python, I use 3.7.9 for compatibility with some NLP libraries) insist the answer is to place the string that has the special characters into an old-style format string WITHIN the cursor.execute method, to enclose string variable placeholders in quotes, and to pass a tuple with an empty second value if, as in my case, only one variable is to be inserted. This is also a solution posted as part of a bug report response on the MySQL website - and yet, I have no success.
Here's what I've got - following the directions on dozens of pages here and the official database website:
for tweet in tweepy.Cursor(twilek.search, q=keyword, tweet_mode='extended').items():
twi_tweet = tweet.full_text
print(twi_tweet)
twi_tweet = twi_tweet.encode('utf8')
requests_total+=1
os.environ['TWITTER_REQUESTS'] = str(requests_total)
requests_total = int(os.environ.get('TWITTER_REQUESTS'))
# insert the archived tweet text into the database table
sql = 'USE hate_tweets'
ms_cur.execute(sql)
twi_tweet = str(twi_tweet)
insert_tweet = re.sub(r'[^A-Za-z0-9 ]+', '', twi_tweet)
ms_cur.execute("INSERT INTO tweets_lgbt (text) VALUES %s" % (insert_tweet,))
cnx.commit()
print(ms_cur.rowcount, "record inserted.")
(twilek is my cursor object because I'm a dork)
expected result: string formatter passes MySQL a modified tweet string that it can process and add as a row to the tweets_lgbt table
actual result: insertion fails on a syntax error for any tweet
I've tried going so far as to use regex to strip everything but alphanumeric and spaces - same issue. I'm wondering if the new string format features of current Python versions have broken compatibility with this connector? I prefer to use the official driver but I'll switch to an ORM if I must. (I did try the newer features like F strings, and found they caused the same result.)
I have these observations:
the VALUES clause requires parentheses VALUES (%s)
the quoting / escaping of values should be delegated to the cursor's execute method, by using unquoted placeholders in the SQL and passing the values as the second argument: cursor.execute(sql, (tweet_text,)) or cursor.executemany(sql, [(tweet_text1,), (tweet_text2,)])
once these steps are applied there's no need for encoding/stringifying/regex-ifying: assuming twi_text is a str and the database's charset/collation supports the full UTF-8 range (for example utf8mb4) then the insert should succeed.
in particular, encoding a str and then calling str on the result is to be avoided: you end up with "b'my original string'"
This modified version of the code in the question works for me:
import mysql.connector
DDL1 = """DROP TABLE IF EXISTS tweets_lgbt"""
DDL2 = """\
CREATE TABLE tweets_lgbt (
`text` VARCHAR (256))
"""
# From https://twitter.com/AlisonMitchell/status/1332567013701500928?s=20
insert_tweet = """\
Particularly pleased to see #SarahStylesAU
quoted in this piece for the work she did
👌
Thrive like a girl: Why women's cricket in Australia is setting the standard
"""
# Older connector releases don't support with...
with mysql.connector.connect(database='test') as cnx:
with cnx.cursor() as ms_cur:
ms_cur.execute(DDL1)
ms_cur.execute(DDL2)
ms_cur.execute("INSERT INTO tweets_lgbt (`text`) VALUES (%s)", (insert_tweet,))
cnx.commit()
print(ms_cur.rowcount, "record inserted.")
This is how you should insert a row to your table,
insert_tweet = "ABCEFg 9 XYZ"
"INSERT INTO tweets_lgbt (text) VALUES ('%s');"%(insert_tweet)
"INSERT INTO tweets_lgbt (text) VALUES ('ABCEFg 9 XYZ');"
Things to note
The arguments to a string formatter is just like the arguments to a
function. So, you cannot add a comma at the end to convert a string
to a tuple there.
If you are trying to insert multiple values at once, you can use cursor.executemany or this answer.
I am importing data into my Python3 environment and then writing it to a MySQL database. However, there is a lot of different data tables, and so writing out each INSERT statement isn't really pragmatic, plus some have 50+ columns.
Is there a good way to create a table in MySQL directly from a dataframe, and then send insert commands to that same table using a dataframe of the same format, without having to actually type out all the col names? I started trying to call column names and format it and concat everything as a string, but it is extremely messy.
Ideally there is a function out there to directly handle this. For example:
apiconn.request("GET", url, headers=datheaders)
#pull in some JSON data from an API
eventres = apiconn.getresponse()
eventjson = json.loads(eventres.read().decode("utf-8"))
#create a dataframe from the data
eventtable = json_normalize(eventjson)
dbconn = pymysql.connect(host='hostval',
user='userval',
passwd='passval',
db='dbval')
cursor = dbconn.cursor()
sql = sqltranslate(table = 'eventtable', fun = 'append')
#where sqlwrite() is some magic function that takes a dataframe and
#creates SQL commands that pymysql can execute.
cursor.execute(sql)
What you want is a way to abstract the generation of the SQL statements.
A library like SQLAlchemy will do a good job, including a powerful way to construct DDL, DML, and DQL statements without needing to directly write any SQL.
I am trying to return a date selected from date picker in to my sql query in my python code. I also tried using encode(utf-8) to remove the unicode string but still, I am getting the error.
I am new to python. Can anyone please help me figure out how to solve this problem? I am using python flask to create the webpage
if request.method=='POST':
dateval2 = request.form['datepick']
dateval = dateval2.encode('utf-8')
result = ("SELECT * FROM OE_TAT where convert(date,Time_IST)='?'",dateval
df = pd.read_sql_query(result,connection)`
Error:
pandas.io.sql.DatabaseError
DatabaseError: Execution failed on sql '("SELECT * FROM OE_TAT where convert(date,Time_IST)='?'", '2015-06-01')': The first argument to execute must be a string or unicode query.
You are providing a tuple to read_sql_query, while the first argument (the query) has to be a string. That's why it gives the error "The first argument to execute must be a string or unicode query".
You can pass the parameter like this:
result = "SELECT * FROM OE_TAT where convert(date,Time_IST)=?"
df = pd.read_sql_query(result, connection, params=(dateval,))
Note that the use of ? depends on the driver you are using (there are different ways to specify parameters, see https://www.python.org/dev/peps/pep-0249/#paramstyle). It is possible you will have to use %s instead of ?.
You could also format the string in beforehand, like result = "SELECT * FROM OE_TAT where convert(date,Time_IST)={0}".format(dateval), however, this is not recommended, see eg here
I am able to do mySQL data insert using following,
from twisted.enterprise.adbapi import ConnectionPool
.
.
self.factory.pool.runOperation ('insert into table ....')
But, somehow unable to figure out how to do a simple select from an adbapi call to mySQL like following,
self.factory.pool.runOperation('SELECT id FROM table WHERE name = (%s)',customer)
How do I retrieve the id value from this partilcar call? I was working OK with plain python but somehow really fuzzed up with the twisted framework.
Thanks.
runOperation isn't for SELECT statements. It is for statements that do not produce rows, eg INSERT and DELETE.
Statements that produce rows are supported by runQuery. For example:
pool = ...
d = pool.runQuery("SELECT id FROM table WHERE name = (%s)", (customer,))
def gotRows(rows):
print 'The user id is', rows
def queryError(reason):
print 'Problem with the query:', reason
d.addCallbacks(gotRows, queryError)
In this example, d is an instance of Deferred. If you haven't encountered Deferreds before, you definitely want to read up about them: http://twistedmatrix.com/documents/current/core/howto/defer.html
What would be the equivalent raw sql for the following:
def index:
Emails.objects.create(email=request.POST['invite_email'])
I have this so far, but I can't quite get the quotations working --
cursor = connection.cursor()
cursor.execute("insert into splash_emails (id, email) values ('0','request.POST[invite_email]')")
transaction.commit_unless_managed()
What would be correct way to write this, and is this the simplest way to perform raw sql?
If you ever want to see the queries django is using you can do:
emails = Emails.objects.create(email=request.POST['invite_email'])
print emails.query
It's a bit verbose, but you'll get the gist.
I think after reading the Django cookbook chapter on Security, you'll have a good idea on how to execute raw sql AND execute it safely.