I am using RMySQL and DBI for the connection between R and MySQL
library(RMySQL)
library(DBI, quietly = TRUE)
Everything is working fine for one command, such as
sql = "select * from clients"
con <- dbConnect(MySQL(),user=user, password=password, dbname=dbname, host=host)
rs <- dbSendQuery(con, sql)
data <- fetch(rs, n=-1)
huh <- dbHasCompleted(rs)
dbClearResult(rs)
on.exit(dbDisconnect(con))
However, when I want to execute multiple commands with ";" between them (such as to set a parameter), it returns error. For example
sql = "SET #LAST_TEN_DAY = DATE_ADD(NOW(), INTERVAL -10 DAY); select * from clients where date > #LAST_TEN_DAY"
con <- dbConnect(MySQL(),user=user, password=password, dbname=dbname, host=host)
rs <- dbSendQuery(con, sql)
data <- fetch(rs, n=-1)
huh <- dbHasCompleted(rs)
dbClearResult(rs)
on.exit(dbDisconnect(con))
Many thanks,
For multiple commands we need to work as follows. The dbSendQuery will save the parameter
sql = "select * from clients where date > #LAST_TEN_DAY"
con <- dbConnect(MySQL(),user=user, password=password, dbname=dbname, host=host)
dbSendQuery(con, 'SET #LAST_TEN_DAY = DATE_ADD(NOW(), INTERVAL -10 DAY)')
rs <- dbSendQuery(con, sql)
data <- fetch(rs, n=-1)
huh <- dbHasCompleted(rs)
dbClearResult(rs)
on.exit(dbDisconnect(con))
Related
I got a data frame in R querying a SQL Server DB, Now I want to loop on each line and insert it to MySQL DB
Tried with dbwritetable but it didn't work
library(RODBC)
library(odbc)
library(RMySQL)
con <- dbConnect(odbc(),
Driver = "SQL Server",
Server = "XX",
Database = "XX",
UID = "XX",
PWD = "XX",
Port = XX)
mydb = dbConnect(MySQL(), user='XX', password='XX', dbname='YY', host='YYY')
resultset <- dbGetQuery(con, "SET NOCOUNT ON
DECLARE #StartDate DateTime
DECLARE #EndDate DateTime
SET #StartDate = dateadd(d,-1,getdate())
SET #EndDate = getdate()
SET NOCOUNT OFF
SELECT …..
LEFT JOIN ... ON ….
LEFT JOIN …. ON x.Key = y.Key
WHERE temp.StartDateTime >= #StartDate")
nrows <- nrow(resultset)
colnames(resultset) <- c("tagName", "date_inserted","value") `
So in here I got my result, in resultset but I don't know how to insert the resulset in MySQL
dbWriteTable(mydb, name='data', value=resultset[0,],append=TRUE)
dbReadTable(mydb, "data")
I Expect to insert the data, but I don't know should it be a for loop (for each line a query) or how is it done
More details with this images :
This is my data set
This is MySQL DB structure
Try using a parameterized insert using the RODBCext package. I have used the following function in the past.
This will append records into your database
library(RODBC)
library(RODBCext)
First we need to connect to the database using RODBC.
sql.driver = "MySQL ODBC 5.3 ANSI Driver" # need to figure the version out
sql.server = "your_server_here"
sql.port = "3306" # or whatever your port number is
sql.user = "your_user_name_here"
sql.pass = "your_password_name_here"
sql.db = "your_database_name_here"
con.string = paste0("Driver=", sql.driver, ";",
"Server=", sql.server, ";",
"Port=", sql.port, ";",
"Uid=", sql.user, ";",
"Pwd=", sql.pass, ";",
"Database=", sql.db, ";")
ch = odbcDriverConnect(con.string)
Then here is the custom function saveTable(). You will want to run this with your specific inputs, defined below.
saveTable <- function(con, table_name, df) {
# con = the ODBC connection (e.g., ch)
# table_name = the SQL database table to append to
# df = the data.frame() to append
sql_code = paste("INSERT INTO",table_name,"(",paste(colnames(df),collapse=", "),") VALUES (",paste(rep("?",ncol(df)),collapse=","),")")
sqlExecute(con, sql_code, df)
}
I have a query to run in R which retrieves data from the database and performs operations on it. When I run it in mysql workbench, it works just fine but in r it takes way too long and may hang the entire system. I also tried to run it in command prompt but got the error:
Error: memory exhausted (limit reached?)
mysql query:
library(DBI)
library(RMySQL)
con <- dbConnect(RMySQL::MySQL(),
dbname ="mydb",
host = "localhost",
port = 3306,
user = "root",
password = "")
pedigree <- dbGetQuery (connection, "SELECT aa.name as person, mother as mom, father as dad
FROM addweight LEFT JOIN aa ON addweight.name2 = aa.name2 or addweight.name = aa.name
LEFT JOIN death ON addweight.name2 = death.name2 or addweight.name = death.name
Where((death.dodeath > curdate() OR aa.name2 NOT IN (SELECT name2 FROM death) OR aa.name NOT IN (SELECT name FROM death) OR aa.name NOT IN (SELECT name FROM death)) AND (dob < curdate() AND domove < curdate()))")
The solution could be to replace dbGetQuery with dbSendQuery and dbFetch call.
The simple steps could be:
library(RMySQL)
# From OP
con <- dbConnect(RMySQL::MySQL(),
dbname ="mydb",
host = "localhost",
port = 3306,
user = "root",
password = "")
# iterationresults is a table in your database. One can replace query with his own
rs = dbSendQuery(con, "select * from iterationresults")
# Fetch first 20 rows and repeat it for all rows
df = dbFetch(rs, n=20)
# For repeated call
while (!dbHasCompleted(rs)){
df<- dbFetch(rs, n=20)
}
# OR Fetch all rows in one go
df = dbFetch(rs, n=-1)
# Free all resources
dbClearResult(rs)
# Close connection
dbDisconnect(con)
# df will contain results i.e.
df
# ID Truck_ID Speed trip_id
#1 11 TTI 039 6 217
#2 12 TTI 039 6 217
# ........
I'm trying to use dynamic insert statements with my database but it fails on character columns. See code below.
library(dplyr)
library(DBI)
library(pool)
library(RSQLite)
df1 <- data.frame(stringsAsFactors = F, id = 1:4, value = letters[1:4])
df2 <- data.frame(stringsAsFactors = F, id = 1:4, value = 100:103)
con <- dbPool(SQLite(), dbname = "test") %>% poolCheckout()
dbWriteTable(con, "with_text", df1, overwrite = T)
dbWriteTable(con, "no_text", df2, overwrite = T)
db1 <- dbReadTable(con, "with_text")
db2 <- dbReadTable(con, "no_text")
new1 <- db1[1,]
new2 <- db2[1,]
query1 <- sprintf(
"INSERT INTO %s (%s) VALUES (%s);",
"with_text",
paste(names(new1), collapse = ", "),
paste(new1, collapse = ", ")
)
query2 <- sprintf(
"INSERT INTO %s (%s) VALUES (%s);",
"no_text",
paste(names(new2), collapse = ", "),
paste(new2, collapse = ", ")
)
db_query1 <- dbSendStatement(con, query1)#fails
dbClearResult(db_query1)
dbReadTable(con, "with_text")
db_query2 <- dbSendStatement(con, query2)
dbClearResult(db_query2)
dbReadTable(con, "no_text")
The #fails line produces this error:
Error in rsqlite_send_query(conn#ptr, statement) : no such column: a
The value of query1 is:
[1] "INSERT INTO with_text (id, value) VALUES (1, a);"
I realize the issue is the lack of single quotes (') around the text value but there has to be a workaround for that. Any help is appreciated. I tried adding column types but couldn't get it to work.
I know that this is a simplified example so I hope that this solution works for your real use case.
`query1 = sprintf("INSERT INTO %s (%s) VALUES (%s,\'%s\');",
"with_text", paste(names(new1), collapse =
","),new1[1,1],new1[1,2])`
I'm editing to add more of an explanation. If you know your second value is going to be text then you can add single quotes into your sprintf to surround that value but you will need to call these values separately instead of calling the entire row and pasting them together. I added \'%s\' escaped single quotes around your second value and then called the values separately.
i have three variables a,b,c (Actually more than 300 variables in my case)
t<-c(a,b,d)
a<-dbGetQuery(con, "SELECT * FROM a")
b<-dbGetQuery(con, "SELECT * FROM b")
d<-dbGetQuery(con, "SELECT * FROM d")
How can I make a loop to request data from MySQL in R? The existing question does not have the explanation on how to write it into the variable names. I need a,b,c in my environment.
Not tested, but something as below should work.
myTables <- c("a","b","c")
res <- lapply(myTables,
function(myTable){
sqlStatement <- paste("select * from",myTable)
dbGetQuery(con, sqlStatement)
})
names(res) <- myTables
I have a MySQL table I am attempting to access with R using RMySQL.
There are 1690004 rows that should be returned from
dbGetQuery(con, "SELECT * FROM tablename WHERE export_date ='2015-01-29'")
Unfortunately, I receive the following warning messages:
In is(object, Cl) : error while fetching row
In dbGetQuery(con, "SELECT * FROM tablename WHERE export_date ='2015-01-29'", : pending rows
And only receive ~400K rows.
If I break the query into several "fetches" using dbSendQuery, the warning messages start appearing after ~400K rows are received.
Any help would be appreciated.
So, it looks like it was due to a 60 second timeout imposed by my hosting provider (damn Arvixe!). I got around this by "paging/chunking" the output. Because my data has an auto-incrementing primary key, every row returned is in order, allowing me to take the next X rows after each iteration.
To get 1.6M rows I did the following:
library(RMySQL)
con <- MySQLConnect() # mysql connection function
day <- '2015-01-29' # date of interest
numofids <- 50000 # number of rows to include in each 'chunk'
count <- dbGetQuery(con, paste0("SELECT COUNT(*) as count FROM tablename WHERE export_date = '",day,"'"))$count # get the number of rows returned from the table.
dbDisconnect(con)
ns <- seq(1, count, numofids) # get sequence of rows to work over
tosave <- data.frame() # data frame to bind results to
# iterate through table to get data in 50k row chunks
for(nextseries in ns){ # for each row
print(nextseries) # print the row it's on
con <- MySQLConnect()
d1 <- dbGetQuery(con, paste0("SELECT * FROM tablename WHERE export_date = '",day,"' LIMIT ", nextseries,",",numofids)) # extract data in chunks of 50k rows
dbDisconnect(con)
# bind data to tosave dataframe. (the ifelse is avoid an error when it tries to rbind d1 to an empty dataframe on the first pass).
if(nrow(tosave)>0){
tosave <- rbind(tosave, d1)
}else{
tosave <- d1
}
}