This question expands on this question
Here, I'm using the custom function created by #Simon.S.A. shown in the answer to this question. I'm attempting to save a tbl_sql object in R to MySQL as a new table without first saving it locally. Here, the database and schema in my MySQL are named "test." The tbl_sql object in R is my_data, and I want to save this is a new table in MySQL labeled "car_data".
library(DBI)
library(tidyverse)
library(dbplyr)
#establish connection and import data from MySQL
con <- DBI::dbConnect(RMariaDB::MariaDB(),
dbname = "test",
host = "127.0.0.1",
user = "user",
password = "password")
my_data <- tbl(con, "mtcars")
my_data <- my_data %>% filter(mpg >= 22)
# write function to save tbl_sql as a new table in SQL
write_to_database <- function(input_tbl, db, schema, tbl_name){
# connection
tbl_connection <- input_tbl$src$con
# SQL query
sql_query <- glue::glue(
"SELECT *\n",
"INTO {db}.{schema}.{tbl_name}\n",
"FROM (\n",
dbplyr::sql_render(input_tbl),
"\n) AS sub_query"
)
result <- dbExecute(tbl_connection, as.character(sql_query))
}
# execute function
write_to_database(my_data, "test", "test", "car_data")
After running final line, I get the following error. I'm not sure how I can fix this.
Error: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '.test.car_data
FROM (
SELECT *
FROM `mtcars`
WHERE (`mpg` >= 22.0)
) AS sub_quer' at line 2 [1064]
12.
stop(structure(list(message = "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '.test.car_data\nFROM (\nSELECT *\nFROM `mtcars`\nWHERE (`mpg` >= 22.0)\n) AS sub_quer' at line 2 [1064]",
call = NULL, cppstack = NULL), class = c("Rcpp::exception",
"C++Error", "error", "condition")))
11.
result_create(conn#ptr, statement, is_statement)
10.
initialize(value, ...)
9.
initialize(value, ...)
8.
new("MariaDBResult", sql = statement, ptr = result_create(conn#ptr,
statement, is_statement), bigint = conn#bigint, conn = conn)
7.
dbSend(conn, statement, params, is_statement = TRUE)
6.
.local(conn, statement, ...)
5.
dbSendStatement(conn, statement, ...)
4.
dbSendStatement(conn, statement, ...)
3.
dbExecute(tbl_connection, as.character(sql_query))
2.
dbExecute(tbl_connection, as.character(sql_query))
1.
write_to_database(my_data, "test", "test", "car_data")
Creating a table with INTO command is an SQL Server (even MS Access) specific syntax and not supported in MySQL. Instead, consider the counterpart statement: CREATE TABLE...SELECT. Also, schema differs between RDBMS's. For MySQL, database is synonymous to schema.
Therefore, consider adjusted version of SQL build:
sql_query <- glue::glue(
"CREATE TABLE {db}.{tbl_name}\n AS \n",
"SELECT * \n",
"FROM (\n",
dbplyr::sql_render(input_tbl),
"\n) AS sub_query"
)
I am fairly new to programming and I've been trying to implement this code in Python, using PyCharm. I'm running the code via a remote server, using PyCharm on my local computer. It was written by a former colleague, and has been giving a lot of encoding issues since we updated the packages like MySQL and the Python interpreter to 3.8. The MySQL version is 8.0, but this is an update. That was not the version installed originally when the code was written.
This is the full error that I am getting:
findBestMatch
*** WARNING: FoundCity[i] = {'Country': 'Austria', 'Page': 'Contact', 'Confidence': 10, 'Mentions': 1}
WriteToDB
Problem <class 'MySQLdb._exceptions.ProgrammingError'>
Traceback (most recent call last):
File "/remotepath/TextMining/NER/FindLocationStoreSQL.py", line 399, in
WriteToDB(c.title(), cn, idProject, 10, "Contact", "v1", cursor, db, database_country)
File "/remotepath/TextMining/NER/FindLocationStoreSQL.py", line 286, in WriteToDB
cursor.execute(sql,values)
File "/home/localhost/.local/lib/python3.8/site-packages/MySQLdb/cursors.py", line 206, in execute
res = self._query(query)
File "/home/localhost/.local/lib/python3.8/site-packages/MySQLdb/cursors.py", line 319, in _query
db.query(q)
File "/home/localhost/.local/lib/python3.8/site-packages/MySQLdb/connections.py", line 259, in query
_mysql.connection.query(self, query)
MySQLdb._exceptions.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'City text mined, Country from datasource'',Confidence='10',FoundWhere=''Contact'' at line 1")
Process finished with exit code -1
The code sample of the function it is trying to run is below:
def findBestMatch(FoundCity,FoundCountry,database_country):
pair_candidates = []
for i in range(0,5):
for j in range(0,5):
if len(FoundCity)>i and len(FoundCountry)>j:
city_i = FoundCity[i].get('City')
country_j = FoundCountry[j].get('Country')
if city_i != None and country_j != None:
#sql = "SELECT City,Country_CountryName,Longitude,Latitude FROM Semanticon.City where city like '{0}' and Country_CountryName like '{1}' and Population>0 order by Population desc".format(FoundCity[i]['City'].encode('utf-8'),FoundCountry[j]['Country'].encode('utf-8'))
sql = "SELECT City,Country_CountryName,Longitude,Latitude FROM Semanticon.City where city like '{0}' and Country_CountryName like '{1}' and Population>0 order by Population desc".format(
FoundCity[i]['City'], FoundCountry[j]['Country'])
try:
cursor.execute(sql)
except:
db = MySQLdb.connect(host, username, password, database, charset='utf8')
db.set_character_set("utf8")
cursor = db.cursor()
cursor.execute(sql)
resul = cursor.fetchall()
if len(resul)>0:
pair_candidates.append({"City":FoundCity[i]['City'],"Country":FoundCountry[j]['Country'],"Score":(FoundCity[i]["Confidence"]+FoundCountry[j]["Confidence"]+0.5*(FoundCity[i]["Mentions"]+FoundCountry[j]["Mentions"]))})
#return FoundCity[i]['City'],FoundCountry[j]['Country'],FoundCity[i]['Confidence']
else:
if city_i == None:
print("*** WARNING: FoundCity[i] = ", FoundCity[i])
else:
print("*** WARNING: FoundCountry[j] = ", FoundCountry[j])
I had to take the encoding out, hence the commented out 'sql' line. The encoding was causing problems and adding an extra 'b' to the string to be read from the database.
The 'WriteToDB' function that it's complaining about is below:
def WriteToDB(City,Country,ProjectId,Confidence,Location,Version,cursor,db,database_country):
sql = None
if database_country!="":
if database_country == Country:
if City != "":
#sql = "UPDATE ProjectLocation SET City='{0}',DataTrace='{1}',Confidence={2},FoundWhere='{3}' WHERE Projects_idProjects={4} and Country='{5}';".format(City," City text mined, Country from datasource",Confidence,Location,ProjectId,original_database_cntry)
sql = "UPDATE ProjectLocation SET City='%s',DataTrace='%s',Confidence='%s',FoundWhere='%s' WHERE Projects_idProjects='%s' and Country='%s';"
values = (City, " City text mined, Country from datasource", Confidence, Location, ProjectId,
original_database_cntry)
if database_country!=Country:
if Country.encode('utf-8') in database_country:
#sql = "UPDATE ProjectLocation SET City='{0}',DataTrace='{1}',Confidence={2},FoundWhere='{3}' WHERE Projects_idProjects={4} and Country='{5}';".format(
# City, " City text mined, Country from datasource", Confidence, Location, ProjectId,
#original_database_cntry)
sql = "UPDATE ProjectLocation SET City='%s',DataTrace='%s',Confidence='%s',FoundWhere='%s' WHERE Projects_idProjects='%s' and Country='%s';"
values = (City, " City text mined, Country from datasource", Confidence, Location, ProjectId,
original_database_cntry)
else:
print("Country conflict in project:"+str(ProjectId))
else:
#sql = "Insert into ProjectLocation (Type,City,Country,Projects_idProjects,Original_idProjects,IsLocationFromDataset,Confidence,FoundWhere,Version,DataTrace)" \
# "Values ('{0}','{1}','{2}',{3},{4},0,{5},'{6}','{7}','{8}')".format("Main",City,Country,ProjectId,ProjectId,Confidence,Location,Version,"Both minded from text v0.1")
sql = "Insert into ProjectLocation (Type,City,Country,Projects_idProjects,Original_idProjects,IsLocationFromDataset,Confidence,FoundWhere,Version,DataTrace)" \
"Values ('%s','%s','%s','%s',0,'%s','%s','%s','%s','%s')"
values = ("Main", City, Country, ProjectId,ProjectId, Confidence, Location, Version, "Both minded from text v0.1")
if sql!=None:
cursor.execute(sql,values)
db.commit()
I commented out the SQL queries as shown and tried to bind them instead, because it was giving a lot of encoding errors. I'm not sure how to get rid of this error and not end up with the encoding errors yet again.
Can someone help?
UPDATE to the question.
I reverted back all the sql queries, and used the queries with the encoding now (previously commented out) and I am getting 'b's in the output.
Any suggestions on how to properly encode these SQL queries so the binary encoding does not come out as b's in the output?
Here is a sample of the output:
ProjectID,ProjectName,FoundCity,FoundCountry,DatabaseCity,DatabaseCountry,Confidence,FoundWhere,Website
2542, Migrantour Country,,b'',b'',b'',10,Contact,link
3938,GeoSmartCity,,,b'',b'',10,Contact,link
I'm looking for a clean way to add multiple values on one row which corresponds to a list of columns in mysql.
Essentially I have two lists:
cols = ['Col_A', 'Col_B','Col_E', 'Col_H, 'Col_n'....]
vals = ['1','56','HEX 00 A0 DB 00', 'Pass', '87'....]
The lists lengths can be 100+ items. Both cols and vals lists will be the same length, so each cols item has a corresponding vals item.
I am using pymysql to connect to an SQL database on a network storage device running MariaDB.
Here's a snippet of my non-working function attempt at passing the two lists:
def add_to_database(cols, vals):
connection = pymysql.connect(host='11.22.33.44',
user='usr',
password='pass',
db='my_db',
charset='utf8mb4',
cursorclass=pymysql.cursors.DictCursor,
autocommit=True)
cursor = connection.cursor()
try:
cursor.execute("CREATE TABLE data_tbl (%s)" % 'id INTEGER PRIMARY KEY')
except:
pass
# Add Column and Values lists to database here
for item in cols:
try:
# First add columns if not already present
cursor.execute("ALTER TABLE data_tbl ADD COLUMN " + str(item))
except:
# Pass column add if already present
pass
cursor.execute("INSERT INTO data_tbl '{0}' VALUES '{1}';".format(cols, vals,))
conn.close()
return
I'm still new to SQL and I've also been playing around with the SQL syntax, so apologies if the code looks a bit odd now.
The common error I get is below:
pymysql.err.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ''('Col_A', 'Col_B', 'Col_E', 'Col_H', 'Col_...' at line 1")
I am trying to commit a sql transaction to MySQL but I get myself past an MySQLSyntaxErrorException.
The code I am using is:
implicit connection =>
SQL("""
start transaction;
insert into projects(id_user, name, description) values({idUser}, {name}, {description});
set #last_id = last_insert_id();
insert into assigned(id_user, id_project) values({idUser}, #last_id);
commit;
""")
.on('idUser -> idUser,
'name -> project.name,
'description -> project.description
).execute()
The exception I get:
[MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'insert into projects(id_user, name, description) values(1, 'First inserted proje' at line 1]
I am starting to think that I can't execute such statements at all with Anorm.
You cannot use transaction that way. You have to understand that anorm is simply a wrapper around existing jdbc libraries. By default, when using the withConnection and SQL :
DB.withConnection { conn =>
SQL("...
}
Your query is transformed using a PreparedStatement. Meaning the ; chars are causing errors.
Thus, if you want to use transaction, you have to use anorm's mecanism for that.
DB.withTransaction { conn =>
SQL("...
}
Engine = create_engine("mysql+mysqldb://blah-blah-blah", encoding="utf-8")
Session = sessionmaker(bind = Engine)
ses = Session()
Meta = MetaData(bind = Engine, reflect = True)
PersonTable = Meta.tables["person"]
class Person(object):
pass
mapper(Person, PersonTable)
APerson = Person("1111", "2222", "1.01.1980")
ses.add(APerson)
ses.commit()
sqlalchemy.exc.ProgrammingError: (ProgrammingError) (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '%s, %s, %s)' at line 1") b'INSERT INTO person (Name, OriginalName, DoB) VALUES (%s, %s, %s)' ('1111', '2222', '25.01.1980')
What is the %s? What do I wrong?
Python 3.1
SQLAlchemy 0.6.5
MySQL 5.1
Windows 7 Ultimate
Thank you.
You sqlalchemy commit is trying to issue an insert query that is not compatible with the schema. near '%s, %s, %s)' means your trying to insert invalid data. I can only speculate that it is because the date format - this is not the proper mysql date format YYYY-MM-DD.
I migrated back to Python 2.7. Now it works fine.