I have a Shiny application (hosted on shinyapps.io) that records a user's click of certain actionButtons to a MySQL database. I'd love some advice on a few things:
where to put the dbConnect code (i.e. inside or outside the shinyServer function)
when to close the connection (as I was running into the problem of too many open connections)
Each addition to the database just adds a new row, so users aren't accessing and modifying the same elements. The reason I ask this is I was running into problem of multiple users not being able to use the app at the same time (with the error "Disconnected from server") and I wasn't sure if it was from the MySQL connections.
Thank you!
Someone in the comments posted about the pool package, which serves this exact purpose! Here's the relevant parts of my server.R code:
library(shiny)
library(RMySQL)
library(pool)
pool <- dbPool(
drv = RMySQL::MySQL(),
user='username',
password='password',
dbname='words',
host='blahblahblah')
shinyServer(function(input, output) {
## function to write to databse
writeToDB <- function(word, vote){
query <- paste("INSERT INTO word_votes (vote, word) VALUES (", vote, ", '", word, "');", sep="")
conn <- poolCheckout(pool)
dbSendQuery(conn, query)
conn <- poolReturn(conn)
## rest of code
}
I added the poolCheckout and poolReturn to run successfully and prevent leaks.
Related
I have a weird problem:
Using RMySQL from Shiny (running locally) I have no problem to retrieve data from MySQL database (small table, few rows only). But once the app is deployed (shinyapps.io) the query result contains zero rows (but column names are fine). Looking at the shinyapps.io log:
Warning in dbFetch(rs, n = n, ...) : error while fetching rows
What I am doing wrong? The exact same thing was working before and now I can't make it running. MySQL connection seems fine.
library(shiny)
library(DBI)
ui <- fluidPage(
numericInput("nrows", "Enter the number of rows to display:", 5),
tableOutput("tbl")
)
server <- function(input, output, session) {
output$tbl <- renderTable({
conn <- dbConnect(
drv = RMySQL::MySQL(),
dbname = "***",
host = "***",
username = "***",
password = "***")
on.exit(dbDisconnect(conn), add = TRUE)
dbGetQuery(conn, paste0(
"SELECT * FROM datasets LIMIT ", input$nrows, ";"))
})
}
shinyApp(ui, server)
EDIT:
When I use Shiny dummy database (from this example) it is working fine, so looks like some problem with MySQL but can't figure it out what... Any ideas?
dbname = "shinydemo",
host = "shiny-demo.csa7qlmguqrf.us-east-1.rds.amazonaws.com",
username = "guest",
password = "guest")
EDIT2
I tried everything. Create new table, new database (same hosting though), different shinyapps account, fresh R installation with all updated packages, still the same problem. When app is running locally, everything is fine. But from shinyapps - error and zero results (except colnames).
Ok, I have no idea why, but looks like changing table engine fix the issue
ALTER TABLE table_name ENGINE = InnoDB
I would like to use dplyr and RMySQL to work with my large data. There is no issues with dplyr code. The problem (I think) is about exporting data out of MySQL to R. My connection is dropped every time even I am using n=Inf in collect. Theoretically, my data should have more than 50K rows, but I can only get around 15K back. Any suggestions are appreciated.
Approach 1
library(dplyr)
library(RMySQL)
# Connect to a database and select a table
my_db <- src_mysql(dbname='aermod_1', host = "localhost", user = "root", password = "")
my_tbl <- tbl(my_db, "db_table")
out_summary_station_raw <- select(my_tbl, -c(X, Y, AVERAGE_CONC))
out_station_mean_local <- collect(out_summary_station_raw)
Approach 2: using Pool
library(pool)
library(RMySQL)
library(dplyr)
pool <- dbPool(
drv = RMySQL::MySQL(),
dbname = "aermod_1",
host = "localhost",
username = "root",
password = ""
)
out_summary_station_raw <- src_pool(pool) %>% tbl("aermod_final") %>% select(-c(X, Y, AVERAGE_CONC))
out_station_mean_local <- collect(out_summary_station_raw, n = Inf)
Warning message (both approaches):
Warning messages:
1: In dbFetch(res, n) : error while fetching rows
2: Only first 15,549 results retrieved. Use n = Inf to retrieve all.
Update:
Checked log and it looks fine from the server side. For my example, the slow-log said Query_time: 79.348351 Lock_time: 0.000000 Rows_sent: 15552 Rows_examined: 16449696, but collect just could not retrieve the full data. I Am able to replicate the same move using MySQL Bench.
As discussed here
https://github.com/tidyverse/dplyr/issues/1968,
https://github.com/tidyverse/dplyr/blob/addb214812f2f45f189ad2061c96ea7920e4db7f/NEWS.md and https://github.com/tidyverse/dplyr/commit/addb214812f2f45f189ad2061c96ea7920e4db7fthis package issue seems to be addresed.
What version of the dplyr package are you using?
After the most recent RMySQL update, I noticed that I could not collect() data from large views, and I reported it as an issue. Your problem might be related.
One thing to try is to roll back to the last version.
devtools::install_version("RMySQL", version = "0.10.9",
repos = "http://cran.us.r-project.org")
I've made dashboard using shiny, shinydashboard and RMySQL package.
Following is what I wrote in order to refresh data every 10 minutes if any change occured.
In global.R
con = dbConnect(MySQL(), host, user, pass, db)
check_func <- function() {dbGetQuery(con, check_query}
get_func <- function() {dbGetQuery(con, get_query}
In server.R
function(input, output, session) {
# check every 10 minutes for any change
data <- reactivePoll(10*60*1000, session, checkFunc = check_func, valueFunc = get_func)
session$onSessionEnded(function() {dbDisconnect(con)})
However, above code infrequently generates corrupt connection handle error from check_func.
Warning: Error in .local: internal error in RS_DBI_getConnection: corrupt connection handle
Should I put dbConnect code inside server function?
Any better ideas?
link: using session$onsessionend to disconnect rshiny app from the mysql server
"pool" package is the answer: http://shiny.rstudio.com/articles/pool-basics.html
This adds a new level of abstraction when connecting to a database: instead of directly fetching a connection from the database, you will create an object (called a pool) with a reference to that database. The pool holds a number of connections to the database. ... Each time you make a query, you are querying the pool, rather than the database. ... You never have to create or close connections directly: the pool knows when it should grow, shrink or keep steady.
I've got answer from here. -> https://stackoverflow.com/a/39661853/4672289
I created an online experiment with the shiny package for R. Let's say I have 3 reactive values called "toss", "decision" and "rating".
Additionally, I launched a MySQL database on Amazon web service RDS. The version is MySQL 5.6.22.
I successfully managed to to send non-reactive values- like the timestamp- to MySQL database. So I assume the problem is where to locate the code talking to MySQL within the Server.R code. For non-reactive values it works perfectly when the code is outside (before) the reactive server-function. But with reactive values I suppose it should be somewhere within.
I tried this code:
Server.R
library(shiny)
library(RMySQL)
library(DBI)
con <- dbConnect(MySQL(), dbname="db", username="myname", password="mypassword", host="myhost.com", port=xxxx)
function(input, output, session){
sql <- reactive({
paste("insert into scenario1 (toss, dec, rat, timestamp)
values (",input$toss,",",input$decision,",",input$rating,"now())")
})
result<-reactive({dbSendQuery(con, sql())})
}
This way, I do not get an error message. So maybe the error is within the insert into-code.
Additionally, I'm not sure whether the packages that I used are ideal for this purpose. I tried out a lot of things. Whenever I add a reactive value by leaving it out of the SQL-quote it stops working. I'm starting to think that RMySQL is missing that feature. There is nothing about insert into in the manual.
Is anyone able to detect the mistake I made?
Finally, I could make the query run with this code:
writingMarks <- function(input){
con <- dbConnect(MySQL(), dbname="db", username="myname", password="mypassword",
host="myhost.com", port=xxxx)
result <- sprintf<-("insert into scenario1 (toss, dec, timestamp) values (%s,%s,%s)",
input$toss, input$decision, "now()")
dbSendQuery(con, result)
}
I could not make the paste query run. With sprintf there is less confusion with the commas.
And, yes I indeed had to add an isolate(). I inserted it into an observe(). So it looks like this:
observe({
if (input$condition==1){
isolate({
writingMarks(input)
})
}
return()
})
You have a problem in:
paste("insert into scenario1 (toss, dec, rat, timestamp)
values (",input$toss,",",input$decision,",",input$rating,"now())")
The issue is: No , before now():
paste("insert into scenario1 (toss, dec, rat, timestamp)
values (",input$toss,",",input$decision,",",input$rating,",now())")
That should make the query run. Look into prepared statements, that will prevent this kind of (very common everybody makes them) concatenation mistakes.
As per my question earlier today, I suspect I have an issue with unclosed connections that is blocking data from being injected into my MySQL database. Data is being allowed into tables that are not currently being used (hence I suspect many open connections preventing uploading into that particular table).
I am using RMySQL on Ubuntu servers to upload data onto a MySQL database.
I'm looking for a way to a) determine if connections are open b) close them if they are. The command exec sp_who and exec sp_who2 from the SQL command line returns an SQL code error.
Another note: I am able to connect, complete the uploading process, and end the R process successfully, and there is no data on the server (checked via the SQL command line) when I try only that table.
(By the way,: If all else fails, would simply deleting the table and creating a new one with the same name fix it? It would be quite a pain, but doable.)
a. dbListConnections( dbDriver( drv = "MySQL"))
b. dbDisconnect( dbListConnections( dbDriver( drv = "MySQL"))[[index of MySQLConnection you want to close]]). To close all: lapply( dbListConnections( dbDriver( drv = "MySQL")), dbDisconnect)
Yes, you could just rewrite the table, of course you would lose all data. Or you can specify dbWriteTable(, ..., overwrite = TRUE).
I would also play with the other options, like row.names, header, field.types, quote, sep, eol. I've had a lot of weird behavior in RMySQL as well. I can't remember specifics, but it seems like I've had no error message when I had done something wrong, like forget to set row.names. HTH
Close all active connections:
dbDisconnectAll <- function(){
ile <- length(dbListConnections(MySQL()) )
lapply( dbListConnections(MySQL()), function(x) dbDisconnect(x) )
cat(sprintf("%s connection(s) closed.\n", ile))
}
executing:
dbDisconnectAll()
Simplest:
lapply(dbListConnections( dbDriver( drv = "MySQL")), dbDisconnect)
List all connections and disconnect them by lapply
Closing a connection
You can use dbDisconnect() together with dbListConnections() to disconnect those connections RMySQL is managing:
all_cons <- dbListConnections(MySQL())
for(con in all_cons)
dbDisconnect(con)
Check all connections have been closed
dbListConnections(MySQL())
You could also kill any connection you're allowed to (not just those managed by RMySQL):
dbGetQuery(mydb, "show processlist")
Where mydb is..
mydb = dbConnect(MySQL(), user='user_id', password='password',
dbname='db_name', host='host')
Close a particular connection
dbGetQuery(mydb, "kill 2")
dbGetQuery(mydb, "kill 5")
lapply(dbListConnections(MySQL()), dbDisconnect)
In current releases the "dbListConnections" function is deprecated and DBI no longer requires drivers to maintain a list of connections. As such, the above solutions may no longer work. E.g. in RMariaDB the above solutions create errors.
I made with the following alternative that uses the MySQL server's functionality and that should work with current DBI / driver versions:
### listing all open connection to a server with open connection
query <- dbSendQuery(mydb, "SHOW processlist;")
processlist <- dbFetch(query)
dbClearResult(query)
### getting the id of your current connection so that you don't close that one
query <- dbSendQuery(mydb, "SELECT CONNECTION_ID();")
current_id <- dbFetch(query)
dbClearResult(query)
### making a list with all other open processes by a particular set of users
# E.g. when you are working on Amazon Web Services you might not want to close
# the "rdsadmin" connection to the AWS console. Here e.g. I choose only "admin"
# connections that I opened myself. If you really want to kill all connections,
# just delete the "processlist$User == "admin" &" bit.
queries <- paste0("KILL ",processlist[processlist$User == "admin" & processlist$Id != current_id[1,1],"Id"],";")
### making function to kill connections
kill_connections <- function(x) {
query <- dbSendQuery(mydb, x)
dbClearResult(query)
}
### killing other connections
lapply(queries, kill_connections)
### killing current connection
dbDisconnect(mydb)