I obtained this code from R Shiny Gallery here:http://shiny.rstudio.com/gallery/reactive-poll-and-file-reader.html
function(input, output, session) {
# Create a random name for the log file
logfilename <- paste0('logfile',
floor(runif(1, 1e+05, 1e+06 - 1)),
'.txt')
# ============================================================
# This part of the code writes to the log file every second.
# Writing to the file could be done by an external process.
# In this example, we'll write to the file from inside the app.
logwriter <- observe({
# Invalidate this observer every second (1000 milliseconds)
invalidateLater(1000, session)
# Clear log file if more than 10 entries
if (file.exists(logfilename) &&
length(readLines(logfilename)) > 10) {
unlink(logfilename)
}
# Add an entry to the log file
cat(as.character(Sys.time()), '\n', file = logfilename,
append = TRUE)
})
# When the client ends the session, suspend the observer and
# remove the log file.
session$onSessionEnded(function() {
logwriter$suspend()
unlink(logfilename)
})
# ============================================================
# This part of the code monitors the file for changes once per
# 0.5 second (500 milliseconds).
fileReaderData <- reactiveFileReader(500, session,
logfilename, readLines)
output$fileReaderText <- renderText({
# Read the text, and make it a consistent number of lines so
# that the output box doesn't grow in height.
text <- fileReaderData()
length(text) <- 14
text[is.na(text)] <- ""
paste(text, collapse = '\n')
})
# ============================================================
# This part of the code monitors the file for changes once
# every four seconds.
pollData <- reactivePoll(4000, session,
# This function returns the time that the logfile was last
# modified
checkFunc = function() {
if (file.exists(logfilename))
file.info(logfilename)$mtime[1]
else
""
},
# This function returns the content of the logfile
valueFunc = function() {
readLines(logfilename)
}
)
output$pollText <- renderText({
# Read the text, and make it a consistent number of lines so
# that the output box doesn't grow in height.
text <- pollData()
length(text) <- 14
text[is.na(text)] <- ""
paste(text, collapse = '\n')
})
}
If you look closely at the code above, you will see this segment:
# When the client ends the session, suspend the observer and
# remove the log file.
session$onSessionEnded(function() {
logwriter$suspend()
unlink(logfilename)
})
I want to tweak that code in my shiny app to disconnect my MySQL server from my shiny app. Like this:
session$onSessionEnded(function() {
dbDisconnect(con)
})
I want to do this to ensure this warning message: "Unable to connect to database: Too many connections"
Would that ensure connections are freed up for other users? (I had a problem with my app because so many users were on and did not disconnect, leaving no available spot on the server.)
Related
I have a R Shiny Application which uses MySQL as a datasource. When the application loads and the user logs into the app with their username and password, a database connection interface opens up where the user inputs their MySQL credentials. In order to prevent the app from crashing when the user enters the wrong MySQL connection credential, I am trying to use the following error handling.
# run when Load Data button is clicked
datapms <- eventReactive(input$pull_data, {
req(input$db_user,input$db_user,input$db_password,input$db_name,input$db_host,input$db_port)
progress <- Progress$new(session, min=1, max=15)
on.exit(progress$close())
progress$set(message = 'Pulling data from database',
detail = 'This message will disappear once completed.')
# establish a database connection
tryCatch({
con <- RMySQL::dbConnect(
RMySQL::MySQL(),
user = input$db_user,
password = input$db_password,
dbname = input$db_name,
host = input$db_host
)
}, error = function(e) {
debug_msg(e$message)
})
# construct the SQL statement
sql <- "SELECT * FROM pmsanalytics;"
# Fetch data
pmsanalytics <- tryCatch({
pmsanalytics <- dbGetQuery(conn = con, sql)
}, error = function(e) {
debug_msg(e$message)
})
### display debugging message in R (if local)
### and in the console log (if running in shiny)
debug_msg <- function(...) {
is_local <- Sys.getenv('SHINY_PORT') == ""
in_shiny <- !is.null(shiny::getDefaultReactiveDomain())
txt <- toString(list(...))
if (is_local) message(txt)
if (in_shiny) shinyjs::runjs(sprintf("console.debug(\"%s\")", text))
}
Initially this code was working, and the app was not crashing. However, now, when for example one enters the wrong connection credentials, i am getting the following error message:
Warning: Error in as.character: cannot coerce type 'closure' to vector of type 'character'
138: sprintf
136: debug_msg [C:\PMSAnalytics/app.R#107]
135: value[[3L]] [C:\PMSAnalytics/app.R#211]
134: tryCatchOne
133: tryCatchList
132: tryCatch
131: eventReactiveValueFunc [C:\PMSAnalytics/app.R#202]
Basically, the app crashes because there is no data which it is expecting to get, in other words the reactive datapms() data source it is expecting to get is empty.
Kind assist in reviewing my code to prevent app crashing.
Regards,
Chris
For my application, new file uploaded to storage is read and the data is added to a main file. The new file contains 2 lines, one a header and other an array whose values are separated by a comma. The main file will need maximum of 265MB. The new files will have maximum of 30MB.
def write_append_to_ecg_file(filename,ecg,patientdata):
file1 = open('/tmp/'+ filename,"w+")
file1.write(":".join(patientdata))
file1.write('\n')
file1.write(",".join(ecg.astype(str)))
file1.close()
def storage_trigger_function(data,context):
#Download the segment file
download_files_storage(bucket_name,new_file_name,storage_folder_name = blob_path)
#Read the segment file
data_from_new_file,meta = read_new_file(new_file_name, scale=1, fs=125, include_meta=True)
print("Length of ECG data from segment {} file {}".format(segment_no,len(data_from_new_file)))
os.remove(new_file_name)
#Check if the main ecg_file_exists
file_exists = blob_exists(bucket_name, blob_with_the_main_file)
print("File status {}".format(file_exists))
data_from_main_file = []
if ecg_file_exists:
download_files_storage(bucket_name,main_file_name,storage_folder_name = blob_with_the_main_file)
data_from_main_file,meta = read_new_file(main_file_name, scale=1, fs=125, include_meta=True)
print("ECG data from main file {}".format(len(data_from_main_file)))
os.remove(main_file_name)
data_from_main_file = np.append(data_from_main_file,data_from_new_file)
print("data after appending {}".format(len(data_from_main_file)))
write_append_to_ecg_file(main_file,data_from_main_file,meta)
token = upload_files_storage(bucket_name,main_file,storage_folder_name = main_file_blob,upload_file = True)
else:
write_append_to_ecg_file(main_file,data_from_new_file,meta)
token = upload_files_storage(bucket_name,main_file,storage_folder_name = main_file_blob,upload_file = True)
The GCF is deployed
gcloud functions deploy storage_trigger_function --runtime python37 --trigger-resource patch-us.appspot.com --trigger-event google.storage.object.finalize --timeout 540s --memory 8192MB
For the first file, I was able to read the file and write the data to the main file. But after uploading the 2nd file, its giving Function execution took 70448 ms, finished with status: 'connection error' On uploading the 3rd file, it gives the Function invocation was interrupted. Error: memory limit exceeded. Despite of deploying the function with 8192MB memory, I am getting this error. Can I get some help on this.
I have a josn file I'm working with that contains multiple json objects in a single file. R is unable to read the file as a whole. But since each object occurs at regular intervals, I would like to iteratively read a fixed number of lines into R.
There are a number of SO questions on reading single lines into R but I have been unable to extend these solutions to a fixed number of lines. For my problem I need to read 16 lines into R at a time (eg 1-16, 17-32 etc)
I have tried using a loop but can't seem to get the syntax right:
## File
file <- "results.json"
## Create connection
con <- file(description=file, open="r")
## Loop over a file connection
for(i in 1:1000) {
tmp <- scan(file=con, nlines=16, quiet=TRUE)
data[i] <- fromJSON(tmp)
}
The file contains over 1000 objects of this form:
{
"object": [
[
"a",
0
],
[
"b",
2
],
[
"c",
2
]
]
}
With #tomtom inspiration I was able to find a solution.
## File
file <- "results.json"
## Loop over a file
for(i in 1:1000) {
tmp <- paste(scan(file=file, what="character", sep="\n", nlines=16, skip=(i-1)*16, quiet=TRUE),collapse=" ")
assign(x = paste("data", i, sep = "_"), value = fromJSON(tmp))
}
I couldn't create a connection as each time I tried the connection would close before the file had been completely read. So I got rid of that step.
I had to include the what="character" variable as scan() seems to expect a number by default.
I included sep="\n", paste() and collapse=" " to create a single string rather than the vector of characters that scan() creates by default.
Finally I just changed the final assignment operator to have a bit more control over the names of the output.
This might help:
EDITED to make it use a list and Reduce into one file
## Loop over a file connection
data <- NULL
for(i in 1:1000) {
tmp <- scan(file=con, nlines=16, skip=(i-1)*16, quiet=TRUE)
data[[i]] <- fromJSON(tmp)
}
df <- Reduce(function(x, y) {paste(x, y, collapse = " ")})
You would have to make sure that you don't reach further than the end of the file though ;-)
How might I call an R script from the shell (e.g. from Node.js exec) and export results as JSON (e.g. back to Node.js)?
The R code below basically works. It reads data, fits a model, converts the parameter estimates to JSON, and prints them to stdout:
#!/usr/bin/Rscript --quiet --slave
install.packages("cut", repos="http://cran.rstudio.com/");
install.packages("Hmisc", repos="http://cran.rstudio.com/");
install.packages("rjson", repos="http://cran.rstudio.com/");
library(rjson)
library(reshape2);
data = read.csv("/data/records.csv", header = TRUE, sep=",");
mylogit <- glm( y ~ x1 + x2 + x3, data=data, family="binomial");
params <- melt(mylogit$coefficients);
json <- toJSON(params);
json
Here's how I'd like to call it from Node...
var exec = require('child_process').exec;
exec('./model.R', function(err, stdout, stderr) {
var params = JSON.parse(stdout); // FAIL! Too much junk in stdout
});
Except the R process won't stop printing to stdout. I've tried --quiet --slave --silent which all help a little but not enough. Here's what's sent to stdout:
The downloaded binary packages are in
/var/folders/tq/frvmq0kx4m1gydw26pcxgm7w0000gn/T//Rtmpyk7GmN/downloaded_packages
The downloaded binary packages are in
/var/folders/tq/frvmq0kx4m1gydw26pcxgm7w0000gn/T//Rtmpyk7GmN/downloaded_packages
[1] "{\"value\":[4.04458733165933,0.253895751245782,-0.1142272181932,0.153106007464742,-0.00289013062471735,-0.00282580664375527,0.0970325223603164,-0.0906967639834928,0.117150317941983,0.046131890754108,6.48538603593323e-06,6.70646151749708e-06,-0.221173770066275,-0.232262366060079,0.163331098409235]}"
What's the best way to use R scripts on the command line?
Running R --silent --slave CMD BATCH model.R per the post below still results in a lot of extraneous text printed to model.Rout:
Run R script from command line
Those options only stop R's own system messages from printing, they won't stop another R function doing some printing. Otherwise you'll stop your last line from printing and you won't get your json to stdout!
Those messages are coming from install.packages, so try:
install.packages(-whatever-, quiet=TRUE)
which claims to reduce the amount of output. If it reduces it to zero, job done.
If not, then you can redirect stdout with sink, or run things inside capture.output.
I am using RMySQL() to send data from R to a MySQL database. The problem is that the database does not receive any data.... I am using doParallel() since i am running over 4500 iterations.... could it be because i try to send the data to the database in the pullSpread() function?
library(RMySQL)
library(doParallel)
library(stringr)
library(foreach)
makeCluster(detectCores()) # ANSWER = 4
cl <- makeCluster(4, type="SOCK") # also used PSOCK & FORK but receive the same problem
registerDoParrallel(cl)
# Now use foreach() and %dopar% to pull data...
# the apply(t(stock1), 2, pullSpread) works but not "parallelized"
# I have also used clusterApply() but is unsuccessful
system.time(
foreach(a=t(stock1)) %dopar% pullSpread(a)
)
When I look in my working directory, all the files are copied successfully onto a .csv file as it should but when I check MySQL workbench or even call the files from R they do not exist...
Here is the stock1() character vector and the pullSpread() function used...
# This list contains more than 4500 iterations.. so I am only posting a few
stock1<-c(
"SGMS.O","SGNL.O","SGNT.O",
"SGOC.O","SGRP.O", ...)
Important Dates needed for function:
Friday <- Sys.Date()-10
# Get Previous 5 days
Thursday <- Friday - 1
Wednesday <- Thursday -1
Tuesday <- Wednesday -1
Monday <- Tuesday -1
#Make Them readable for NetFonds
Friday <- format(Friday, "%Y%m%d")
Thursday<- format(Thursday, "%Y%m%d")
Wednesday<- format(Wednesday, "%Y%m%d")
Tuesday<- format(Tuesday, "%Y%m%d")
Monday<-format(Monday, "%Y%m%d")
Here is the pullSpread() function:
pullSpread = function (stock1){
AAPL_FRI<- read.delim(header=TRUE, stringsAsFactor=FALSE,
paste(sep="",
"http://www.netfonds.no/quotes/posdump.php?date=",
Friday,"&paper=",stock1,"&csv_format=txt"))
tryit <- try(AAPL_FRI[,c(1:7)])
if(inherits(tryit, "try-error")){
rm(AAPL_FRI)
} else {
AAPL_THURS<- read.delim(header=TRUE, stringsAsFactor=FALSE,
paste(sep="",
"http://www.netfonds.no/quotes/posdump.php?date=",
Thursday,"&paper=",stock1,"&csv_format=txt"))
AAPL_WED<- read.delim(header=TRUE, stringsAsFactor=FALSE,
paste(sep="",
"http://www.netfonds.no/quotes/posdump.php?date=",
Wednesday,"&paper=",stock1,"&csv_format=txt"))
AAPL_TUES<- read.delim(header=TRUE, stringsAsFactor=FALSE,
paste(sep="",
"http://www.netfonds.no/quotes/posdump.php?date=",
Tuesday,"&paper=",stock1,"&csv_format=txt"))
AAPL_MON<- read.delim(header=TRUE, stringsAsFactor=FALSE,
paste(sep="",
"http://www.netfonds.no/quotes/posdump.php?date=",
Monday,"&paper=",stock1,"&csv_format=txt"))
SERIES <- rbind(AAPL_MON,AAPL_TUES,AAPL_WED,AAPL_THURS,AAPL_FRI)
#Write .CSV File
write.csv(SERIES,paste(sep="",stock1,"_",Friday,".csv"), row.names=FALSE)
dbWriteTable(con2,paste0( "",str_sub(stock1, start = 1L, end = -3L),""),paste0(
"~/Desktop/R/",stock1,"_",Friday,".csv"), append=T)
}
}
Retrieve last Friday using something like this:
Friday <- Sys.Date()
while(weekdays(Friday) != "Friday")
{
Friday <- Friday - 1
}
As a matter of good practice, when retrieving data from the internet, separate the act of downloading it with processing it. That way, when the processing fails, you don't waste time and bandwidth redownloading things.
lastWeek <- format(Friday - 0:4, "%Y%m%d")
stockDatePairs <- expand.grid(Stock = stock1, Date = lastWeek)
urls <- with(
stockDatePairs,
paste0(
"http://www.netfonds.no/quotes/posdump.php?date=",
Date,
"&paper=",
Stock,
"&csv_format=txt"
)
)
for(url in urls)
{
# or whatever file name you want
download.file(url, paste0("data from ", make.names(url), ".txt"))
}
Make sure that you know which directory those files are being saved to. (Either provide an absolute path or set your working directory.)
Now try reading and rbinding those files.
If that works, then you can try doing things in parallel.
Also note that many online data services will limit the rate that you can download, unless you are paying for the service. So parallel downloads may just mean that you hit the limit quicker.