I just started using the R package called RMySQL in order to get around some memory limitations on my computer. I am trying to take a matrix with 100 columns in R (called data.df), then make a new table on an SQL database that has "100 choose 2" (=4950) columns, where each column is a linear combination of two columns from the initial matrix. So far I have something like this:
countnumber <- 1
con <- dbConnect(MySQL(), user = "root", password = "password", dbname = "myDB")
temp <- as.data.frame(data.df[,1] - data.df[,2])
colnames(temp) <- paste(pairs[[countnumber]][1], pairs[[countnumber]][2], sep = "")
dbWriteTable(con, "spreadtable", temp, row.names=T, overwrite = T)
for(i in 1:(n-1)){
for(j in (i+1):n){
if(!((i==1)&&(j==2))){ #this part excludes the first iteration already taken care of
temp <- as.data.frame(data.df[,i] - data.df[,j])
colnames(temp) <- "hola"
dbWriteTable(con, "spreadtable", value = temp, append = TRUE, overwrite = FALSE, row.names = FALSE)
countnumber <- countnumber + 1
}
}
}
I've also tried toying around with the "field.types" argument of RMySQL::dbWriteTable(), which was suggested at RMySQL dbWriteTable with field.types. Sadly it hasn't helped me out too much.
Questions:
Is making your own sql database a valid solution to the memory-bound nature of R, even if it has 4950 columns?
Is the dbWriteTable() the proper function to be using here?
Assuming the answer is "yes" to both of the previous questions...why isn't this working?
Thanks for any help.
[EDIT]: code with error output:
names <- as.data.frame(index)
names <- t(names)
#dim(names) is 1 409
con <- dbConnect(MySQL(), user = "root", password = "password", dbname = "taylordatabase")
dbGetQuery(con, dbBuildTableDefinition(MySQL(), name="spreadtable", obj=names, row.names = F))
#I would prefer these to be double types with 8 decimal spaces instead of text
#dim(temp) is 1 409
temp <- as.data.frame(data.df[,1] - (ratios[countnumber]*data.df[,2]))
temp <- t(temp)
temp <- as.data.frame(temp)
dbWriteTable(con, name = "spreadtable", temp, append = T)
The table is created successfully in the database (I will change variable type later), but the dbWriteTable() line produces the error:
Error in mysqlExecStatement(conn, statement, ...) :
RS-DBI driver: (could not run statement: Unknown column 'row_names' in 'field list')
[1] FALSE
Warning message:
In mysqlWriteTable(conn, name, value, ...) : could not load data into table
If I make a slight change, I get a different error message:
dbWriteTable(con, name = "spreadtable", temp, append = T, row.names = F)
and
Error in mysqlExecStatement(conn, statement, ...) :
RS-DBI driver: (could not run statement: Unknown column 'X2011_01_03' in 'field list')
[1] FALSE
Warning message:
In mysqlWriteTable(conn, name, value, ...) : could not load data into table
I just want to use "names" as a bunch of column labels. They were initially dates. The actual data I would like to be "temp."
Having a query with 4950 rows is ok, the problem is that what columns you need.
If you always "select * ", you will eventually exhaust all you system memory (in the case that the table has 100 columns)
Why not give us some error message if you have encountered any problems ?
Related
I am trying to use the following function in R:
heritblup <- function(name) {
library(RMySQL)
library(DBI)
con <- dbConnect(RMySQL::MySQL(),
dbname ="mydab",
host = "localhost",
port = 3306,
user = "root",
password = "")
value1 <- 23;
rss<- paste0 ("INSERT INTO namestable
(myvalue, person)
VALUES ('$value1', '",name,"')")
rs <<- dbGetQuery (con, rss)
}
heritblup("Tommy")
But I keep getting this error:
Error in as.character.default()
: no method for coercing this S4 class to a vector Called from:
as.character.default()
I tried to change the paste function to this:
rss<- paste0 ("INSERT INTO namestable
(myvalue, person)
VALUES ($value1, ",name,")")
the error persists;
I have no idea whats wrong.
Please help
Couple of issues in code. I'm not sure if OP is attempting to insert records in database or fetch from database.
Assuming, based on query that he is expecting to insert data in database table.
The rule is that query should be prepared in R the way it will be executed in MySQL. Value replacement (if any) should be performed in R as MySQL engine will not have any idea about variables from R.
Hence, the query preparation steps should be done as:
rss <- sprintf("INSERT INTO namestable (myvalue, person) VALUES (%d, '%s')", value1, name)
# "INSERT INTO namestable (myvalue, person) VALUES (23, 'test')"
If data insert is goal then dbGetQuery is not right option per R documentation instead dbSendStatement() should be used for data manipulation. The reference from help suggest:
However, callers are strongly encouraged to use dbSendStatement() for
data manipulation statements.
Based on that query execution line should be:
rs <- dbSendStatement(con, rss)
ret_val <- dbGetRowsAffected(rs)
dbClearResult(rs)
dbDisconnect(con)
return(ret_val)
I am using RMySQL package to write (append) data in current table.
I am using R, version 3.3.2.
My code looks like this:
library(RMySQL)
df_final <- some_data
m<-dbDriver("MySQL")
mydb <- dbConnect(m, user='odvjet12_mislav',
password='my_pass',
host='91.234.46.219',
dbname='odvjet12_fina_pn')
dbWriteTable(mydb, value = df_final, name = "fina_pn", append = TRUE, row.names = FALSE)
This code works fine for some time, but in last ten days, it always return an error:
Error in .local(conn, statement, ...) :
could not run statement: The used command is not allowed with this MySQL version
I don't understand how it is possible for code to work for some time and now, it returns an error?
I kindly ask for feedback on this issue.
Best,
Mislav Šagovac
You could also use dbGetQuery from the RMySQL package and iterate over the rows, which was my solution when I reached a similar error for a dataframe I wanted to write to a MySQL DB:
mydb = dbConnect(MySQL(), user='user', password='password', dbname='databasename', host='hostname')
for(i in 1:nrow(df)){
dbGetQuery(mydb,paste0("INSERT INTO MYTABLE (COL1,COL2) VALUES(",df$col1[i],",",df$col2[i],")"))
}
I have a data frame containing columns 'Quarter' having values like "16/17 Q1", "16/17 Q2"... and 'Vendor' having values like "a", "b"... .
I am trying to write this data frame into database using
query <- paste("INSERT INTO cc_demo (Quarter,Vendor) VALUES(dd$FY_QUARTER,dd$VENDOR.x)")
but it is throwing error :
Error in .local(conn, statement, ...) :
could not run statement: Unknown column 'dd$FY_QUARTER' in 'field list'
I am new to Rmysql, Please provide me some solution to write entire dataframe?
To write a data frame to mySQL DB you need to:
Create a connection to your database, you need to specify:
MySQL connection
User
Password
Host
Database name
library("RMySQL")
connection <- dbConnect(MySQL(), user = 'root', password = 'password', host = 'localhost', dbname = 'TheDB')
Using the connection create a table and then export data to the database
dbWriteTable(connection, "testTable", testTable)
You can overwrite an existing table like this:
dbWriteTable(connection, "testTable", testTable_2, overwrite=TRUE)
I would advise against writing sql query when you can actually use very handy functions such as dbWriteTable from the RMySQL package. But for the sake of practice, below is an example of how you should go about writing the sql query that does multiple inserts for a MySQL database:
# Set up a data.frame
dd <- data.frame(Quarter = c("16/17 Q1", "16/17 Q2"), Vendors = c("a","b"))
# Begin the query
sql_qry <- "insert into cc_demo (Quarter,Vendor) VALUES"
# Finish it with
sql_qry <- paste0(sql_qry, paste(sprintf("('%s', '%s')", dd$Quarter, dd$Vendors), collapse = ","))
You should get:
"insert into cc_demo (Quarter,Vendor) VALUES('16/17 Q1', 'a'),('16/17 Q2', 'b')"
You can provide this query to your database connection in order to run it.
I hope this helps.
I am trying to use pool to join two remote tables (City and Country) using below code:
pool <- dbPool(
drv = RMySQL::MySQL(),
dbname = "shinydemo",
host = "shiny-demo.csa7qlmguqrf.us-east-1.rds.amazonaws.com",
username = "guest",
password = "guest"
)
src_pool(pool) %>%
tbl('City') %>%
left_join('Country', by=c('CountryCode'='Code'))
But this is the error I get when run the code:
Error: x and y don't share the same src.
Set copy = TRUE to copy y into x's source (this may be time consuming).
In addition: Warning message:
In force(expr) : You have a leaked pooled object. Destroying it.
Below a working example of the same query using dplyr:
srccon <- src_mysql(
host = "shiny-demo.csa7qlmguqrf.us-east-1.rds.amazonaws.com",
dbname = "shinydemo",
user = "guest",
password = "guest"
)
tbl(srccon, 'City') %>%
left_join(tbl(srccon, 'Country'), by=c('CountryCode'='Code'))
And another example using pool::dbGetQuery
sql <- "SELECT * FROM City LEFT JOIN Country ON (CountryCode=Code)"
dbGetQuery(pool, sql)
I'm not familiar with pool, but the error message:
Error: x and y don't share the same src.
Set copy = TRUE to copy y into x's source (this may be time consuming).
is due to the fact that one of your tables is being manipulated in-database (I'd guess City) and the other is in memory/not in the database ('Country'), but it could be the other way around. In this instance, x and y are your two tables, and the error message is just telling you they're not bothin the same place. By setting copy = TRUE the tibble in y will be (temporarily) copied into wherever x is stored (e.g. either to the database or into RAM) in order to perform the computation.
You could try either bringing both objects into memory using dplyr::collect() which will mean that the join happens in memory; or ensure that both are on your database (in which case the join will occur there instead).
i.e.
# Keep both in RAM
City <- tbl('City') %>% collect()
Country <- tbl('Country') %>% collect()
joined <- City %>% left_join(Country, by = c('CountryCode' = 'Code))
I would guess that the Warning about a leaked pooled object relates to the same issue (one table is in-memory, one is on the database).
I am running an R script that adds data to mySQL database. I usually format the data and add it as as it comes in after couple of hours (data string is not continuous). My first set of data was added properly in MySQL database. The second string of data can not be added properly.
con = dbConnect(MySQL(), user='root', password='xxxxxx', dbname='test', host='localhost')
dbWriteTable(con, 'Tables', value = parseTweets(filterStream(file.name= "", track=c("lebron"), timeout=10, oauth=my_oauth)))
When I rerun the last code (dbWriteTable) again, it gives me following error
Error: Error in .local(conn, statement, ...) :
could not run statement: Table 'tables' already exists
I also used
dbWriteTable(con, 'Tables', value = parseTweets(filterStream(file.name= "", track=c("lebron"), timeout=10, oauth=my_oauth)), append = TRUE)
but it provides the same error
For some reason giving setting append to TRUE is not working. Instead give it a number. See the code below for better understanding
dbWriteTable(con, 'Tables', value = parseTweets(filterStream(file.name= "", track=c("lebron"), timeout=100, oauth=my_oauth)), overwrite = 0, row.names = 0, append = 1)