I am currently writing a script that downloads a bunch of .csv's from a FTP server, and then puts each .csv in a MySQL database as its own table.
I download the .csv's from the FTP using RCurl and place all of the .csv's in my working directory. To create tables out of each .csv, I am using the sqlSave function from the RODBC package, where the table name is the same name as the .csv. This works fine whenever a .csv name is less than 18 characters, but when it is greater it fails. And by "fails", I mean R crashes. To track down the bug, I called debug on sqlSave.
I found that there are at least two functions that sqlSave calls that cause R to crash. The first is RODBC:::odbcTableExists, which is a non-visible function. Here is the code for the function:
RODBC:::odbcTableExists
function (channel, tablename, abort = TRUE, forQuery = TRUE,
allowDot = attr(channel, "interpretDot"))
{
if (!odbcValidChannel(channel))
stop("first argument is not an open RODBC channel")
if (length(tablename) != 1)
stop(sQuote(tablename), " should be a name")
tablename <- as.character(tablename)
switch(attr(channel, "case"), nochange = {
}, toupper = tablename <- toupper(tablename), tolower = tablename <- tolower(tablename))
isExcel <- odbcGetInfo(channel)[1L] == "EXCEL"
hasDot <- grepl(".", tablename, fixed = TRUE)
if (allowDot && hasDot) {
parts <- strsplit(tablename, ".", fixed = TRUE)[[1]]
if (length(parts) > 2)
ans <- FALSE
else {
res <- if (attr(channel, "isMySQL"))
sqlTables(channel, catalog = parts[1], tableName = parts[2])
else sqlTables(channel, schema = parts[1], tableName = parts[2])
ans <- is.data.frame(res) && nrow(res) > 0
}
}
else if (!isExcel) {
res <- sqlTables(channel, tableName = tablename)
ans <- is.data.frame(res) && nrow(res) > 0
}
else {
res <- sqlTables(channel)
tables <- stables <- if (is.data.frame(res))
res[, 3]
else ""
if (isExcel) {
tables <- sub("^'(.*)'$", "\\1", tables)
tables <- unique(c(tables, sub("\\$$", "", tables)))
}
ans <- tablename %in% tables
}
if (abort && !ans)
stop(sQuote(tablename), ": table not found on channel")
enc <- attr(channel, "encoding")
if (nchar(enc))
tablename <- iconv(tablename, to = enc)
if (ans && isExcel) {
dbname <- if (tablename %in% stables)
tablename
else paste(tablename, "$", sep = "")
if (forQuery)
paste("[", dbname, "]", sep = "")
else dbname
}
else if (ans) {
if (forQuery && !hasDot)
quoteTabNames(channel, tablename)
else tablename
}
else character(0L)
}
This fails here when the table name over 18 characters in length:
res <- sqlTables(channel, tableName = tablename)
I have fixed it by changing this to:
res <- sqlTables(channel, tablename)
I then reassign the function with the same name (odbcTableExists) in the namespace with this code change using assignInNamepace.
RODBC:::odbcTableExists no longer causes an issue. However, R still crashes when sqlwrite is called from within sqlSave(). I called debug on sqlwrite, and I found that RODBC:::odbcColumns (another non-visible function) causes that to crash when tablenames are too long. Unfortunately, I am not sure how to change RODBC:::odbcColumns to avoid the bug like I did before.
I am using R 2.15.1, and the platform is :x86_64-pc-ming32/x64 (64-bit). I should also note that I am trying to run this on a work computer, but if I run the exact same code on my personal computer, R does not crash (no bug). The work computer runs windows 7 professional, and my home computer runs windows 7 home premium with R 2.14.1.
I love this hack (I too have Windows 7 Professional at R 2.15.1 at work), and it does not crash anymore, but it causes another problem after I replaced that line and used assignInNamespace; also for some reason I had to replace odbcValidChannel with RODBC:::odbcValidChannel and quoteTabNames with RODBC:::quoteTabNames
But when I used sqlSave, I got the following error:
Error in odbcUpdate(channel, query, mydata, coldata[m, ], test = test, :
no parameters, so nothing to update
I don't even use odbcUpdate anywhere in the code, and the RODBC::: sqlSave does not have the odbcUpdate call inside.
Any thoughts?
thank you,
-Alex
Related
I'm using renderDataTable in my shiny app to display the contents of a data.table vals$content4table which is a reactiveValues.
It can happen that the vals$content4table is equal to a datatable with no columns.
In that case i have an error while using formatCurrency because it searches for a column that does not exist.
Is there any way to check if the datatable has columns with ifelse to avoid the error?
Here is a piece of my code of my server.
#initialising vals$content4table when launching the app
vals <- reactiveValues(content4table = if(someBoolean) {data.table(NULL)} else {data.table("Column1" = "whatever","Currency" = 1000}
)
output$TableInUI <- DT::renderDataTable(datatable(vals$content4table) %>% ifelse(nrow(vals$content4table)>0,formatCurrency(2, currency = "", interval = 3, mark = ",", digits = 0),fnothing()))
#where fnothing is defined as
fnothing<-function(df) return(df)
The above code doesn't work and gives this error:
Warning: Error in ifelse: unused argument (fnothing())
You could use req:
output$TableInUI <- DT::renderDataTable({
req(isTRUE(ncol(vals$content4table)>0))
vals$content4table
})
I successfully fetch data from my mysql db using r:
library(RMySQL)
mydb = dbConnect(MySQL(), user='user', password='pass', dbname='fib', host='myhost')
rs = dbSendQuery(mydb, 'SELECT distinct(DATE(date)) as date, open,close FROM stocksng WHERE symbol = "FIB7F";')
data <- fetch(rs, n=-1)
dbHasCompleted(rs)
so now I've an object a list:
> print (typeof(data))
[1] "list"
each elements is a tuple(?) like date(charts),open(long),close(long)
ok well now my problem: I want to get a vector of percentuale difference betwen close (x) and next day open (x+1) until the end BUT I can't access properly to the item!
Example: ((open)/close*100)-100)
I try:
for (item in data){
print (item[2])
}
and all possible combination like:
for (item in data){
print (item[][2])
}
but cannot access to right element :! anyone could help?
You have a bigger problem than this in your MySQL query, because you did not specify an ORDER BY clause. Consider using the following query:
SELECT DISTINCT
DATE(date) AS date,
open,
close
FROM stocksng
WHERE
symbol = "FIB7F"
ORDER BY
date
Here we order the result set by date, so that it makes sense to speak of the current and next open or close. Now with a proper query in place if you wanted to get the percentile difference between the current close and the next day open you could try:
require(dplyr)
(lead(open, 1) / close*100) - 100
Or using base R:
(open[2:(length(open)+1)] / close*100) - 100
naif version:
for (row in 1:nrow(data)){
date <- unname (data[row,"date"])
open <- unname (data[row+1,"open"])
close <- unname (data[row,"close"])
var <- abs((close/open*100)-100)
print (var)
}
I'm using the R package monitoR and getting an error message that I can't figure out.
I'm trying to upload a correlation template list ("bithTemps") to a MySQL database ("noh") using the dbUploadTemplate command.
dbUploadTemplate(templates = bithTemps,
uid = "root",
pwd = "****",
db.name = "noh",
analyst = 1,
locationID = "2",
date.recorded = "2017/09/07",
recording.equip = "Unknown",
species.code = "BITH",
type = "COR")
This returns:
Error: $ operator is invalid for atomic vectors
I have confirmed the ODBC connection is working, that the template list is functional (i.e., it works when called to other arguments in the package), and that the SQL database has the required entries for analyst, location, and species code.
It seems that this error was actually triggered by a non-functional ODBC connection. This part of the dbUploadTemplate function
species <- RODBC::sqlQuery(dbCon, paste("SELECT `pkSpeciesID`, `fldSpeciesCode` FROM `tblSpecies` WHERE `fldSpeciesCode` = '",
paste(species.code, sep = "", collapse = "' OR `fldSpeciesCode` = '"),
"'", sep = ""))
queries a table in the SQL database and returns an object called 'species'. If the query fails (e.g., because RODBC can't connect) than 'species' is empty, and the following operation
speciesID <- NULL
for (i in 1:length(species.code)) {
speciesID[i] <- species$pkSpeciesID[species$fldSpeciesCode ==
species.code[i]]
}
triggers the error. Fixing the ODBC connection resolves the error.
I'm trying to save some data in to a table. I get the data from a database and it works ok.
My problem is that the data is not saved in the table. It is a lua table like table = {} and NOT a database table.
Maybe it is saved but it looks like the prints are done before the saving even though I call them after. In fact it seems like my network request is done last in my program even though I call it first.
I would real like to know the reason for this. Any ideas?
Here is the code:
---TESTING!
print("Begin teting!")
--hej = require ( "test2" )
local navTable = {
Eng_Spd = 0,
Spd_Set = 0
}
local changeTab = function()
navTable.Eng_Spd = 2
end
printNavTable = function()
print("navTable innehåller: ")
print(navTable.Eng_Spd)
print(navTable.Spd_Set)
end
require "sqlite3"
local myNewData
local json = require ("json")
local decodedData
local SaveData2 = function()
local i = 1
local counter = 1
local index = "livedata"..counter
local navValue = decodedData[index]
print(navValue)
while (navValue ~=nil) do
--tablefill ="INSERT INTO navaltable VALUES (NULL,'" .. navValue[1] .. "','" .. navValue[3] .."','" .. navValue[4] .."','" .. navValue[5] .."','" .. navValue[6] .."');"
--print(tablefill)
--db:exec(tablefill)
if navValue[3] == "Eng Spd" then navTable.Eng_Spd = navValue[4]
elseif navValue[3] == "Spd Set" then navTable.Spd_Set = navValue[4]
else print("blah")
end
print(navTable.Eng_Spd)
print(navTable.Spd_Set)
counter=counter+1
index = "livedata"..counter
navValue = decodedData[index]
end
end
local function networkListener( event )
if (event.isError) then
print("Network error!")
else
myNewData = event.response
print("From server: "..myNewData)
decodedData = (json.decode(myNewData))
SaveData2()
--db:exec("DROP TABLE IN EXISTS navaltable")
end
end
--function uppdateNavalTable()
network.request( "http://127.0.0.1/firstMidle.php", "GET", networkListener )
--end
changeTab()
printNavTable()
--uppdateNavalTable()
printNavTable()
print("Done!")
And here is the output:
Copyright (C) 2009-2012 C o r o n a L a b s I n c .
Version: 2.0.0
Build: 2012.971
Begin teting!
navTable innehåller:
2
0
navTable innehåller:
2
0
Done!
From server: {"livedata1":["1","0","Eng Spd","30","0","2013-03-15 11:35:48"],"li
vedata2":["1","1","Spd Set","13","0","2013-03-15 11:35:37"]}
table: 008B5018
30
0
30
13
And btw, navTable innehåller means navTable contains.
The answer is that networklistener run parallell with the rest of the code.
I just started using the R package called RMySQL in order to get around some memory limitations on my computer. I am trying to take a matrix with 100 columns in R (called data.df), then make a new table on an SQL database that has "100 choose 2" (=4950) columns, where each column is a linear combination of two columns from the initial matrix. So far I have something like this:
countnumber <- 1
con <- dbConnect(MySQL(), user = "root", password = "password", dbname = "myDB")
temp <- as.data.frame(data.df[,1] - data.df[,2])
colnames(temp) <- paste(pairs[[countnumber]][1], pairs[[countnumber]][2], sep = "")
dbWriteTable(con, "spreadtable", temp, row.names=T, overwrite = T)
for(i in 1:(n-1)){
for(j in (i+1):n){
if(!((i==1)&&(j==2))){ #this part excludes the first iteration already taken care of
temp <- as.data.frame(data.df[,i] - data.df[,j])
colnames(temp) <- "hola"
dbWriteTable(con, "spreadtable", value = temp, append = TRUE, overwrite = FALSE, row.names = FALSE)
countnumber <- countnumber + 1
}
}
}
I've also tried toying around with the "field.types" argument of RMySQL::dbWriteTable(), which was suggested at RMySQL dbWriteTable with field.types. Sadly it hasn't helped me out too much.
Questions:
Is making your own sql database a valid solution to the memory-bound nature of R, even if it has 4950 columns?
Is the dbWriteTable() the proper function to be using here?
Assuming the answer is "yes" to both of the previous questions...why isn't this working?
Thanks for any help.
[EDIT]: code with error output:
names <- as.data.frame(index)
names <- t(names)
#dim(names) is 1 409
con <- dbConnect(MySQL(), user = "root", password = "password", dbname = "taylordatabase")
dbGetQuery(con, dbBuildTableDefinition(MySQL(), name="spreadtable", obj=names, row.names = F))
#I would prefer these to be double types with 8 decimal spaces instead of text
#dim(temp) is 1 409
temp <- as.data.frame(data.df[,1] - (ratios[countnumber]*data.df[,2]))
temp <- t(temp)
temp <- as.data.frame(temp)
dbWriteTable(con, name = "spreadtable", temp, append = T)
The table is created successfully in the database (I will change variable type later), but the dbWriteTable() line produces the error:
Error in mysqlExecStatement(conn, statement, ...) :
RS-DBI driver: (could not run statement: Unknown column 'row_names' in 'field list')
[1] FALSE
Warning message:
In mysqlWriteTable(conn, name, value, ...) : could not load data into table
If I make a slight change, I get a different error message:
dbWriteTable(con, name = "spreadtable", temp, append = T, row.names = F)
and
Error in mysqlExecStatement(conn, statement, ...) :
RS-DBI driver: (could not run statement: Unknown column 'X2011_01_03' in 'field list')
[1] FALSE
Warning message:
In mysqlWriteTable(conn, name, value, ...) : could not load data into table
I just want to use "names" as a bunch of column labels. They were initially dates. The actual data I would like to be "temp."
Having a query with 4950 rows is ok, the problem is that what columns you need.
If you always "select * ", you will eventually exhaust all you system memory (in the case that the table has 100 columns)
Why not give us some error message if you have encountered any problems ?