R to MySQL query - mysql

I got a data frame in R querying a SQL Server DB, Now I want to loop on each line and insert it to MySQL DB
Tried with dbwritetable but it didn't work
library(RODBC)
library(odbc)
library(RMySQL)
con <- dbConnect(odbc(),
Driver = "SQL Server",
Server = "XX",
Database = "XX",
UID = "XX",
PWD = "XX",
Port = XX)
mydb = dbConnect(MySQL(), user='XX', password='XX', dbname='YY', host='YYY')
resultset <- dbGetQuery(con, "SET NOCOUNT ON
DECLARE #StartDate DateTime
DECLARE #EndDate DateTime
SET #StartDate = dateadd(d,-1,getdate())
SET #EndDate = getdate()
SET NOCOUNT OFF
SELECT …..
LEFT JOIN ... ON ….
LEFT JOIN …. ON x.Key = y.Key
WHERE temp.StartDateTime >= #StartDate")
nrows <- nrow(resultset)
colnames(resultset) <- c("tagName", "date_inserted","value") `
So in here I got my result, in resultset but I don't know how to insert the resulset in MySQL
dbWriteTable(mydb, name='data', value=resultset[0,],append=TRUE)
dbReadTable(mydb, "data")
I Expect to insert the data, but I don't know should it be a for loop (for each line a query) or how is it done
More details with this images :
This is my data set
This is MySQL DB structure

Try using a parameterized insert using the RODBCext package. I have used the following function in the past.
This will append records into your database
library(RODBC)
library(RODBCext)
First we need to connect to the database using RODBC.
sql.driver = "MySQL ODBC 5.3 ANSI Driver" # need to figure the version out
sql.server = "your_server_here"
sql.port = "3306" # or whatever your port number is
sql.user = "your_user_name_here"
sql.pass = "your_password_name_here"
sql.db = "your_database_name_here"
con.string = paste0("Driver=", sql.driver, ";",
"Server=", sql.server, ";",
"Port=", sql.port, ";",
"Uid=", sql.user, ";",
"Pwd=", sql.pass, ";",
"Database=", sql.db, ";")
ch = odbcDriverConnect(con.string)
Then here is the custom function saveTable(). You will want to run this with your specific inputs, defined below.
saveTable <- function(con, table_name, df) {
# con = the ODBC connection (e.g., ch)
# table_name = the SQL database table to append to
# df = the data.frame() to append
sql_code = paste("INSERT INTO",table_name,"(",paste(colnames(df),collapse=", "),") VALUES (",paste(rep("?",ncol(df)),collapse=","),")")
sqlExecute(con, sql_code, df)
}

Related

Lost connection to MySQL server during query [2013] in shiny app

I have migrated MySQL older version to v8.0.19 . In older version it worked fine, but now facing some issues with pool connection as below:
R SQL connection pool:
library(pool)
library(DBI)
library(RMariaDB)
pool = dbPool(
drv=MariaDB(),
dbname="mydb",
username="root",
password=Sys.getenv("MYSQL_PASSWORD"),
host="localhost",
sslmode = 'require',
port=3306
)
statement = paste0("select * from Employee where Id IN (", ids ,")")
con3 = poolCheckout(pool)
x = dbGetQuery(con3,statement) //Throwing Error here
poolReturn(con3)
But it is working when I changed same code to :
statement = paste0("select * from Employee where Id IN (", ids ,")")
con3 = poolCheckout(pool)
x = dbGetQuery(pool,statement) //Not getting any issue like this
poolReturn(con3)
Is it I am doing in wrong way?
Whats going on there (", ids ,")")?
x = dbGetQuery(con3,statement) not the same as below , you have pool there!

Mysql query works well at workbench but takes too long in r

I have a query to run in R which retrieves data from the database and performs operations on it. When I run it in mysql workbench, it works just fine but in r it takes way too long and may hang the entire system. I also tried to run it in command prompt but got the error:
Error: memory exhausted (limit reached?)
mysql query:
library(DBI)
library(RMySQL)
con <- dbConnect(RMySQL::MySQL(),
dbname ="mydb",
host = "localhost",
port = 3306,
user = "root",
password = "")
pedigree <- dbGetQuery (connection, "SELECT aa.name as person, mother as mom, father as dad
FROM addweight LEFT JOIN aa ON addweight.name2 = aa.name2 or addweight.name = aa.name
LEFT JOIN death ON addweight.name2 = death.name2 or addweight.name = death.name
Where((death.dodeath > curdate() OR aa.name2 NOT IN (SELECT name2 FROM death) OR aa.name NOT IN (SELECT name FROM death) OR aa.name NOT IN (SELECT name FROM death)) AND (dob < curdate() AND domove < curdate()))")
The solution could be to replace dbGetQuery with dbSendQuery and dbFetch call.
The simple steps could be:
library(RMySQL)
# From OP
con <- dbConnect(RMySQL::MySQL(),
dbname ="mydb",
host = "localhost",
port = 3306,
user = "root",
password = "")
# iterationresults is a table in your database. One can replace query with his own
rs = dbSendQuery(con, "select * from iterationresults")
# Fetch first 20 rows and repeat it for all rows
df = dbFetch(rs, n=20)
# For repeated call
while (!dbHasCompleted(rs)){
df<- dbFetch(rs, n=20)
}
# OR Fetch all rows in one go
df = dbFetch(rs, n=-1)
# Free all resources
dbClearResult(rs)
# Close connection
dbDisconnect(con)
# df will contain results i.e.
df
# ID Truck_ID Speed trip_id
#1 11 TTI 039 6 217
#2 12 TTI 039 6 217
# ........

UPDATE in mysql using python

how to do Update in mysql consider Rate is int(8)
k=int(4000);
db=MySQLdb.connect("localhost","user","pass,"databse")
cursor=db.cursor()
sql="UPDATE tablename SET Rate=k WHERE Name='xxx'
cursor=db.cursor()
try:
cursor.execute(sql)
db.commit()
except:
db.rollback()
db.close()
sql should be:
sql = "UPDATE tablename SET Rate = (%s) WHERE Name='xxx'"
and then you should do:
cursor.execute(sql,(k,))
note that k needs to be in a container to be passed correctly to .execute(). in this case it is a tuple.
also int(4000) is redundant. you can just do: k = 4000
if you also want to pass in the name you can do:
sql = "UPDATE tablename SET Rate = (%s) WHERE Name=(%s)"
and then:
cursor.execute(sql,(k,name))

Execute multiple SQL commands at once on R

I am using RMySQL and DBI for the connection between R and MySQL
library(RMySQL)
library(DBI, quietly = TRUE)
Everything is working fine for one command, such as
sql = "select * from clients"
con <- dbConnect(MySQL(),user=user, password=password, dbname=dbname, host=host)
rs <- dbSendQuery(con, sql)
data <- fetch(rs, n=-1)
huh <- dbHasCompleted(rs)
dbClearResult(rs)
on.exit(dbDisconnect(con))
However, when I want to execute multiple commands with ";" between them (such as to set a parameter), it returns error. For example
sql = "SET #LAST_TEN_DAY = DATE_ADD(NOW(), INTERVAL -10 DAY); select * from clients where date > #LAST_TEN_DAY"
con <- dbConnect(MySQL(),user=user, password=password, dbname=dbname, host=host)
rs <- dbSendQuery(con, sql)
data <- fetch(rs, n=-1)
huh <- dbHasCompleted(rs)
dbClearResult(rs)
on.exit(dbDisconnect(con))
Many thanks,
For multiple commands we need to work as follows. The dbSendQuery will save the parameter
sql = "select * from clients where date > #LAST_TEN_DAY"
con <- dbConnect(MySQL(),user=user, password=password, dbname=dbname, host=host)
dbSendQuery(con, 'SET #LAST_TEN_DAY = DATE_ADD(NOW(), INTERVAL -10 DAY)')
rs <- dbSendQuery(con, sql)
data <- fetch(rs, n=-1)
huh <- dbHasCompleted(rs)
dbClearResult(rs)
on.exit(dbDisconnect(con))

How specify join variables with different names in different MySQL tables

I need to join two tables where the common column-id that I want to use has a different name in each table. The two tables have a "false" common column name that does not work when dplyr takes the default and joins on columns "id".
Here's some of the code involved in this problem
library(dplyr)
library(RMySQL)
SDB <- src_mysql(host = "localhost", user = "foo", dbname = "bar", password = getPassword())
# Then reference a tbl within that src
administrators <- tbl(SDB, "administrators")
members <- tbl(SDB, "members")
Here are 3 attempts -- that all fail -- to pass along the information that the common column on the members side is "id" and on the adminisrators side it's "idmember":
sqlq <- semi_join(members,administrators, by=c("id","idmember"))
sqlq <- inner_join(members,administrators, by= "id.x = idmember.y")
sqlq <- semi_join(members,administrators, by.x = id, by.y = idmember)
Here's an example of the kinds of error messages I'm getting:
Error in mysqlExecStatement(conn, statement, ...) :
RS-DBI driver: (could not run statement: Unknown column '_LEFT.idmember' in 'where clause')
The examples I see out there pertain to data tables and data frames on the R side. My question is about how dplyr sends "by" statements to a SQL engine.
In the next version of dplyr, you'll be able to do:
inner_join(members, administrators, by = c("id" = "idmember"))
Looks like this is an unresolved issue:
https://github.com/hadley/dplyr/issues/177
However you can use merge:
❥ admin <- as.tbl(data.frame(id = c("1","2","3"),false = c(TRUE,FALSE,FALSE)))
❥ members <- as.tbl(data.frame(idmember = c("1","2","4"),false = c(TRUE,TRUE,FALSE)))
❥ merge(admin,members, by.x = "id", by.y = "idmember")
id false.x false.y
1 1 TRUE TRUE
2 2 FALSE TRUE
If you need to do left or outer joins, you can always use the ALL.x, or ALL arguments to merge. A thought though... You've got a sql db, why not use it?
❥ con2 <- dbConnect(MySQL(), host = "localhost", user = "foo", dbname = "bar", password = getPassword())
❥ dbGetQuery(con, "select * from admin join members on id = idmember")