I have an application that allows users to query MySQL database using input widgets on a shiny app. The queries involve joining tables too. The code becomes too long when employing IF ...ELSE statements to determine if a widget is empty or has some user input like in the code below.
Sample MySQL data can be created as below:
CREATE TABLE quoteauthors (
FirstName VARCHAR(255) ,
LastName VARCHAR(255) ,
authorID VARCHAR(255)
);
CREATE TABLE quotes (
quote VARCHAR(255) ,
authorID VARCHAR(255)
);
INSERT INTO quoteauthors
VALUES ('Albert', 'Einstein', 'a1'),
('Stephen', 'Hawking', 'a2'),
('Isaac', 'Newton', 'a3');
INSERT INTO quotes
VALUES ('Unthinking respect for authority is the greatest enemy of truth.', 'a1'),
('In the middle of difficulty lies opportunity.', 'a1'),
('Intelligence is the ability to adapt to change.', 'a2'),
('Science is not only a disciple of reason but, also, one of romance and passion.', 'a2'),
('If I have seen further it is by standing on the shoulders of Giants.', 'a3'),
('I can calculate the motion of heavenly bodies but not the madness of people', 'a3');
Sample shiny app is as below:
library(shiny)
library(shinydashboard)
library(DBI)
library(RMySQL)
ui <- dashboardPage(
dashboardHeader(),
dashboardSidebar(
sidebarMenu(
menuItem("QUOTE Search", tabName = "Tabs", icon = icon("object-ungroup"))
)
),
dashboardBody(
tabItem(tabName = "Tabs",
fluidRow(
column(width=3,
box(
title="Search ",
solidHeader=TRUE,
collapsible=TRUE,
width=NULL,
textInput("quoteSearch1", " Search Term 1 ", '', placeholder = "Type search term"),
radioButtons("combi", "Logical Operator to Combine Terms:",
c(
"AND" = "AND",
"OR" = "OR"
), inline = TRUE),
textInput("quoteSearch2", " Search Term 2 ", '', placeholder = "Type search term"),
selectInput("authorchoice", "Select AUTHOR", selected = NULL, multiple = T,
choices=c('Albert','Stephen','Isaac')),
submitButton("Search")
)
),
column( width=9,
tabBox(
width="100%",
tabPanel("Search Results",
htmlOutput("quotesearchdetails")
)))))))
server <- function(input, output) {
output$quotesearchdetails <-renderUI({
if(input$quoteSearch1!=""){
con <- dbConnect(MySQL(),
user='XXXXXXXXXXX',
port = 3306, password='XXXXXXXXXXX',
dbname='XXXXXXXXXXX',
host='XXXXXXXXXXX')
dbSendQuery(con, "SET NAMES utf8mb4;")
dbSendQuery(con, "SET CHARACTER SET utf8mb4;")
dbSendQuery(con, "SET character_set_connection=utf8mb4;")
on.exit(dbDisconnect(con), add = TRUE)
quotedetails <- reactive({
if (input$authorchoice == ""){
if (input$quoteSearch2 == ""){
dbGetQuery(con, statement =
paste0(" SELECT q.quote, a.FirstName, a.LastName
FROM quotes q
JOIN quoteauthors a
ON (q.authorID = a.authorID)
WHERE (q.quote LIKE '%",input$quoteSearch1,"%') "))
}else{
if (input$combi == "AND"){
dbGetQuery(con, statement =
paste0("
SELECT q.quote, a.FirstName, a.LastName
FROM quotes q
JOIN quoteauthors a
ON (q.authorID = a.authorID)
WHERE (q.quote LIKE '%",input$quoteSearch1,"%' AND
q.quote LIKE '%",input$quoteSearch2,"%')"))
}else{
dbGetQuery(con, statement =
paste0("
SELECT q.quote, a.FirstName, a.LastName
FROM quotes q
JOIN quoteauthors a
ON (q.authorID = a.authorID)
WHERE (q.quote LIKE '%",input$quoteSearch1,"%'
OR q.quote LIKE '%",input$quoteSearch2,"%')"))
}
}
}else{
if (input$quoteSearch2 == ""){
dbGetQuery(con, statement =
paste0("
SELECT q.quote, a.FirstName, a.LastName
FROM quotes q
JOIN quoteauthors a
ON (q.authorID = a.authorID)
WHERE (q.quote LIKE
'%",input$quoteSearch1,"%'
AND a.FirstName LIKE '%",input$authorchoice,"%') "))
}else {
if (input$combi == "AND"){
dbGetQuery(con, statement =
paste0("
SELECT q.quote, a.FirstName, a.LastName
FROM quotes q
JOIN quoteauthors a
ON (q.authorID = a.authorID)
WHERE (q.quote LIKE '%",input$quoteSearch1,"%' AND
q.quote LIKE '%",input$quoteSearch2,"%') AND
a.FirstName LIKE '%",input$authorchoice,"%' "))
}else{
dbGetQuery(con, statement =
paste0("
SELECT q.quote, a.FirstName, a.LastName
FROM quotes q
JOIN quoteauthors a
ON (q.authorID = a.authorID)
WHERE (q.quote LIKE '%",input$quoteSearch1,"%' OR
q.quote LIKE '%",input$quoteSearch2,"%')
AND
a.FirstName LIKE '%",input$authorchoice,"%' "))
}
}
}
})
outputed=""
quotedetailsreturned <- quotedetails()
if (dim(quotedetailsreturned)[1] > 0){
for(i in seq(from=1,to=dim(quotedetailsreturned)[1])){
outputed<-paste(outputed,
paste("Author's First name: ",quotedetailsreturned[i,"FirstName"]),
sep="<br/><br/>")
outputed<-paste(outputed,
paste("Author's Last name: ",quotedetailsreturned[i,"LastName"]),
sep="<br/><br/>")
outputed<-paste(outputed,
paste("Quote: ",quotedetailsreturned[i,"quote"]),
sep="<br/><br/>")
}
} else { outputed <-"your search yielded no results."}
HTML(outputed)
}else {
paste("Please input a search term at least in the first field")
}
})
}
shinyApp(ui, server)
I am seeking a solution on how to avoid the repetition and long codes using the IF...ELSE statements in my code. What best programming practices I could use to combine the MySQL queries with user input on various shiny widgets including textInput, radioButtons, selectize/selectInput and so on by considering some inputs can be left empty thus should not be considered in the query.
I would first build the query string only, step-by-step, adding each clause at a step according to the chosen settings. After it has been built, execute the query. Makes the code much shorter and easier to read.
Related
I have multiple models in Sparx Enterprise Architect in file-based, i.e. using MS access.
I'm using a custom template to populate a table with data from object's properties, including some with <memo> fields.
This is the query i'm using in the template fragment:
SELECT obj.object_id,
obj.Stereotype,
objp.Property as Prop,
switch(objp.Value = '<memo>', objp.Notes, objp.Value LIKE '{*}',
NULL, 1=1, objp.Value) AS Val,
(SELECT tobj2.ea_guid & tobj2.Name FROM t_object tobj2 WHERE
tobj2.ea_guid = objp.Value) AS [Obj-Hyperlink]
FROM t_object obj
INNER JOIN t_objectproperties objp
ON (obj.object_id = objp.object_id)
WHERE obj.object_id = #OBJECTID# AND obj.Stereotype='Data-
Stream' AND objp.Property NOT IN ('isEncapsulated')
ORDER BY objp.Property ASC;
I found that the when these fields are longer than 249 chars I get an error message when generating the reports and the cell in the generated table is simply empty. This is also noticeable with a query:
The error I'm getting states:
"Error Processing xml document: an invalid character was found in text context"
Is there any workarround to enable including the <memo> fields' data with more than 249 chars in the reports?
Any help is much appreciated.
I've found a workaround for this by joining two queries with a "Union all". The first query will handle the non-memo fields with the switch function and the second one the memo fields without the switch function.
select
obj.object_id,
obj.Stereotype,
objp.Property as Prop,
objp.Notes AS Val,
(
SELECT
tobj2.ea_guid & tobj2.Name
FROM
t_object tobj2
WHERE
tobj2.ea_guid = objp.Value
) AS [Obj-Hyperlink]
from
t_objectproperties objp
left join t_object obj on (obj.object_id = objp.object_ID)
where
obj.object_id = #OBJECTID#
AND obj.Stereotype = 'Data-Stream'
AND objp.Property NOT IN ('isEncapsulated')
AND objp.Value = "<memo>"
UNION ALL
SELECT
obj2.object_id,
obj2.Stereotype,
objp2.Property as Prop,
switch(
objp2.Value LIKE '{*}', NULL, 1 = 1, objp2.Value
) AS Val,
(
SELECT
tobj2.ea_guid & tobj2.Name
FROM
t_object tobj2
WHERE
tobj2.ea_guid = objp2.Value
) AS [Obj-Hyperlink]
FROM
t_object obj2
INNER JOIN t_objectproperties objp2 ON (obj2.object_id = objp2.object_id)
WHERE
obj2.object_id = #OBJECTID#
AND obj2.Stereotype = 'Data-Stream'
AND objp2.Property NOT IN ('isEncapsulated')
and objp2.Value <> "<memo>"
order by
3 asc;
Thanks a lot #geertbellekens for your comment which was crucial to find this solution.
I'm reading the official Django documentation, but I can't find an answer to my question.
Right now I have this query implemented, working with a custom MariaDB connector for Django:
results = []
cursor = get_cursor()
try:
sql="SELECT a.*, COALESCE( NULLIF(a.aa, '.'), NULLIF(a.gen, '.') ) AS variant, b.*, c.* FROM `db-dummy`.sp_gen_data c JOIN `db-dummy`.gen_info a ON a.record_id = c.gen_id JOIN `db-dummy`.sp_data b ON b.record_id = c.sp_id WHERE a.gene_name LIKE concat(?, '%') AND a.report_notation LIKE concat('%', ?, '%') AND b.sp_id LIKE concat(?, '%');"
data = (gen, var, sp)
cursor.execute(sql, data)
except mariadb.Error as e:
print(f"Error: {e}")
columns = [column[0] for column in cursor.description]
results = []
for row in cursor.fetchall():
results.append(dict(zip(columns, row)))
return results
It works, but now I need to adapt this query to the django's default MySQL backend.
What I've tried:
results = []
cursor = get_cursor()
sql="SELECT a.*, COALESCE( NULLIF(a.aa, '.'), NULLIF(a.gen, '.') ) AS variant, b.*, c.* FROM `db-dummy`.sp_gen_data c JOIN `db-dummy`.gen_info a ON a.record_id = c.gen_id JOIN `db-dummy`.sp_data b ON b.record_id = c.sp_id WHERE a.gene_name LIKE %s AND a.report_notation LIKE %s AND b.sp_id LIKE %s;"
data = ('{}%'.format(gen), '{}%'.format(var), '{}%'.format(sp))
cursor.execute(sql, data)
columns = [column[0] for column in cursor.description]
results = []
for row in cursor.fetchall():
results.append(dict(zip(columns, row)))
return results
So basically what I had to change to make it work was LIKE concat(?, '%') for LIKE %s.
The problem is, for the a.report_notation LIKE concat('%', ?, '%') part, I do not know how to convert it to something that Django can interpret.
Any ideas?
Your first query should be fine just adjusted to match the format that Django expects.
First, replace ? with %s to pass parameters to the query
Second, replace % with %% as a single percent is an "escape" character and you need to escape the escape char
DOCS
Here's your original query truncated to show an example of how it could work
sql="... WHERE a.gene_name LIKE concat(%s, '%%') AND a.report_notation LIKE concat('%%', %s, '%%') AND b.sp_id LIKE concat(%s, '%%');"
data = (gen, var, sp)
cursor.execute(sql, data)
I'm trying to use dynamic insert statements with my database but it fails on character columns. See code below.
library(dplyr)
library(DBI)
library(pool)
library(RSQLite)
df1 <- data.frame(stringsAsFactors = F, id = 1:4, value = letters[1:4])
df2 <- data.frame(stringsAsFactors = F, id = 1:4, value = 100:103)
con <- dbPool(SQLite(), dbname = "test") %>% poolCheckout()
dbWriteTable(con, "with_text", df1, overwrite = T)
dbWriteTable(con, "no_text", df2, overwrite = T)
db1 <- dbReadTable(con, "with_text")
db2 <- dbReadTable(con, "no_text")
new1 <- db1[1,]
new2 <- db2[1,]
query1 <- sprintf(
"INSERT INTO %s (%s) VALUES (%s);",
"with_text",
paste(names(new1), collapse = ", "),
paste(new1, collapse = ", ")
)
query2 <- sprintf(
"INSERT INTO %s (%s) VALUES (%s);",
"no_text",
paste(names(new2), collapse = ", "),
paste(new2, collapse = ", ")
)
db_query1 <- dbSendStatement(con, query1)#fails
dbClearResult(db_query1)
dbReadTable(con, "with_text")
db_query2 <- dbSendStatement(con, query2)
dbClearResult(db_query2)
dbReadTable(con, "no_text")
The #fails line produces this error:
Error in rsqlite_send_query(conn#ptr, statement) : no such column: a
The value of query1 is:
[1] "INSERT INTO with_text (id, value) VALUES (1, a);"
I realize the issue is the lack of single quotes (') around the text value but there has to be a workaround for that. Any help is appreciated. I tried adding column types but couldn't get it to work.
I know that this is a simplified example so I hope that this solution works for your real use case.
`query1 = sprintf("INSERT INTO %s (%s) VALUES (%s,\'%s\');",
"with_text", paste(names(new1), collapse =
","),new1[1,1],new1[1,2])`
I'm editing to add more of an explanation. If you know your second value is going to be text then you can add single quotes into your sprintf to surround that value but you will need to call these values separately instead of calling the entire row and pasting them together. I added \'%s\' escaped single quotes around your second value and then called the values separately.
This seems ridiculous, but I just can't get this right - any help much appreciated please!
Basically: I'm using RMySQL to do some simple SQL, in order to get my head around how SQL works. I'd like to chain together a few SQL select queries, as a simple example. This is covered in the RMySQL PDF - but the example therein seems to be the incorrect syntax (http://cran.r-project.org/web/packages/RMySQL/RMySQL.pdf , page 3, example 6).
If I have three queries, say like this:
q1 <- "SELECT db.table FROM table WHERE stuff = 'blah' "
q2 <- "SELECT db.other_table FROM other_table WHERE stuff = 'different blah' "
q3 <- "SELECT db.table2 FROM table2 WHERE table2 = 1000"
and try to paste them as follows:
script <- paste(q1, q2, q3, sep=";")
the result is
> script
[1] "SELECT db.table FROM table WHERE stuff = 'blah' ;SELECT fb.other_table FROM
other_table WHERE stuff = 'different blah' ;SELECT db.table2 FROM table2 WHERE table2 =
'1000'
and so invoking dbSendQuery clearly fails.
I've tried \", but this also doesn't work:
q1 <- "SELECT db.table FROM table WHERE stuff = 'blah' \" "
q2 <- "SELECT db.other_table FROM other_table WHERE stuff = 'different blah' \""
q3 <- "SELECT db.table2 FROM table2 WHERE table2 = 1000 \" "
script <- paste(q1, q2, q3, sep=";")
> script
[1] "SELECT db.table FROM table WHERE stuff = 'blah' \" ; ;SELECT db.other_table FROM
other_table WHERE stuff = 'different blah' \";SELECT db.table2 FROM table2 WHERE table2
= 1000 \" "
Can anyone please point out what I'm doing wrong?
EDIT: just for clarification, executing this via RMySQL as follows:
my.queries <- dbGetQuery(my.con, script, client.flag = CLIENT_MULTI_STATEMENTS)
as per the RMySQL manual, I get
RS-DBI driver: (could not run statement: You have an error in your SQL syntax;
Presumably, this is because the result of the paste function should be:
"SELECT db.table FROM table WHERE stuff = 'blah'" ;"SELECT fb.other_table FROM
other_table WHERE stuff = 'different blah'" ;"SELECT db.table2 FROM table2 WHERE table2
= '1000'"
Each of the individual queries works just fine, so I'm assuming that it's my paste command that's causing the issue.
EDIT: to simplify this: suppose I have two strings, as follows:
t1 <- "the 'stuff'"
t2 <- "more 'stuff'"
paste(t1, t2, sep=";")
[1] "the 'stuff' ; more 'stuff' "
what I'd like is for the result of the paste command to be "the 'stuff'";"more 'stuff'".
You have to pass the argument client.flag = CLIENT_MULTI_STATEMENTS to the function dbConnection, not to dgGetQuery.
Then, your first approach should work:
q1 <- "SELECT db.table FROM table WHERE stuff = 'blah' "
q2 <- "SELECT db.other_table FROM other_table WHERE stuff = 'different blah' "
q3 <- "SELECT db.table2 FROM table2 WHERE table2 = 1000"
script <- paste(q1, q2, q3, sep=";")
I have a function (called "powersearch", the irony!) that searches for a set of strings across a bunch(~ 5) of fields.
The words come in as one string and are separated by spaces.
Some fields can have exact matches, others should have "contains".
(Snipped for brevety)
//Start with all colors
IQueryable<Color> q = db.Colors;
//Filter by powersearch
if (!string.IsNullOrEmpty(searchBag.PowerSearchKeys)){
foreach (string key in searchBag.SplitSearchKeys(searchBag.PowerSearchKeys)
.Where(k=> !string.IsNullOrEmpty(k))){
//Make a local copy of the var, otherwise it gets overwritten
string myKey = key;
int year;
if (int.TryParse(myKey, out year) && year > 999){
q = q.Where(c => c.Company.Name.Contains(myKey)
|| c.StockCode.Contains(myKey)
|| c.PaintCodes.Any(p => p.Code.Equals(myKey))
|| c.Names.Any(n => n.Label.Contains(myKey))
|| c.Company.CompanyModels.Any(m => m.Model.Name.Contains(myKey))
|| c.UseYears.Any(y => y.Year.Equals(year))
);
}
else{
q = q.Where(c => c.Company.Name.Contains(myKey)
|| c.StockCode.Contains(myKey)
|| c.PaintCodes.Any(p => p.Code.Contains(myKey))
|| c.Names.Any(n => n.Label.Contains(myKey))
|| c.Company.CompanyModels.Any(m => m.Model.Name.Equals(myKey))
);
}
}
}
Because the useYear count is rather large, I tried to check for it as little as possible by outruling all numbers that can never be a number that makes sence in this case. Similar checks are not possible on the other fields since they can pretty much contain any thinkable string.
Currently this query takes about 15 secs for a single, non-year string. That's too much.
Anything I can do to improve this?
--Edit--
Profiler shows me the following info for the part where the string is not a year:
exec sp_reset_connection
Audit login
exec sp_executesql N'
SELECT COUNT(*) AS [value]
FROM [dbo].[CLR] AS [t0]
INNER JOIN [dbo].[CO] AS [t1] ON [t1].[CO_ID] = [t0].[CO_ID]
WHERE
([t1].[LONG_NM] LIKE #p0)
OR ([t0].[EUR_STK_CD] LIKE #p1)
OR (EXISTS(
SELECT NULL AS [EMPTY]
FROM [dbo].[PAINT_CD] AS [t2]
WHERE ([t2].[PAINT_CD] LIKE #p2)
AND ([t2].[CLR_ID] = [t0].[CLR_ID])
AND ([t2].[CUSTOM_ID] = [t0].[CUSTOM_ID])
)
)OR (EXISTS(
SELECT NULL AS [EMPTY]
FROM [dbo].[CLR_NM] AS [t3]
WHERE ([t3].[CLR_NM] LIKE #p3)
AND ([t3].[CLR_ID] = [t0].[CLR_ID])
AND ([t3].[CUSTOM_ID] = [t0].[CUSTOM_ID])
)
) OR (EXISTS(
SELECT NULL AS [EMPTY]
FROM [dbo].[CO_MODL] AS [t4]
INNER JOIN [dbo].[MODL] AS [t5] ON [t5].[MODL_ID] = [t4].[MODL_ID]
WHERE ([t5].[MODL_NM] = #p4)
AND ([t4].[CO_ID] = [t1].[CO_ID])
)
)
',N'#p0 varchar(10),#p1 varchar(10),#p2 varchar(10),#p3 varchar(10),#p4 varchar(8)',#p0='%mercedes%',#p1='%mercedes%',#p2='%mercedes%',#p3='%mercedes%',#p4='mercedes'
(took 3626 msecs)
Audit Logout (3673 msecs)
exec sp_reset_connection (0msecs)
Audit login
exec sp_executesql N'
SELECT TOP (30)
[t0].[CLR_ID] AS [Id],
[t0].[CUSTOM_ID] AS [CustomId],
[t0].[CO_ID] AS [CompanyId],
[t0].[EUR_STK_CD] AS [StockCode],
[t0].[SPCL_USE_CD] AS [UseCode],
[t0].[EFF_IND] AS [EffectIndicator]
FROM [dbo].[CLR] AS [t0]
INNER JOIN [dbo].[CO] AS [t1] ON [t1].[CO_ID] = [t0].[CO_ID]
WHERE
([t1].[LONG_NM] LIKE #p0)
OR ([t0].[EUR_STK_CD] LIKE #p1)
OR (EXISTS(
SELECT NULL AS [EMPTY]
FROM [dbo].[PAINT_CD] AS [t2]
WHERE ([t2].[PAINT_CD] LIKE #p2)
AND ([t2].[CLR_ID] = [t0].[CLR_ID])
AND ([t2].[CUSTOM_ID] = [t0].[CUSTOM_ID])
)
)
OR (EXISTS(
SELECT NULL AS [EMPTY]
FROM [dbo].[CLR_NM] AS [t3]
WHERE ([t3].[CLR_NM] LIKE #p3)
AND ([t3].[CLR_ID] = [t0].[CLR_ID])
AND ([t3].[CUSTOM_ID] = [t0].[CUSTOM_ID])
)
)
OR (EXISTS(
SELECT NULL AS [EMPTY]
FROM [dbo].[CO_MODL] AS [t4]
INNER JOIN [dbo].[MODL] AS [t5] ON [t5].[MODL_ID] = [t4].[MODL_ID]
WHERE ([t5].[MODL_NM] = #p4)
AND ([t4].[CO_ID] = [t1].[CO_ID])
)
)'
,N'#p0 varchar(10),#p1 varchar(10),#p2 varchar(10),#p3 varchar(10),#p4 varchar(8)',#p0='%mercedes%',#p1='%mercedes%',#p2='%mercedes%',#p3='%mercedes%',#p4='mercedes'
(took 3368 msecs)
The database structure, sadly, is not under my control. It comes from the US and has to stay in the exact same format for compatibility reasons. Although most of the important fields are indeed indexed, they are indexed in (unnecessary) clustered primary keys. There's verry little I can do about that.
Okay, let's break this down - the test case you're interested in first is a single non-year, so all we've got is this:
q = q.Where(c => c.Company.Name.Contains(myKey)
|| c.StockCode.Contains(myKey)
|| c.PaintCodes.Any(p => p.Code.Contains(myKey))
|| c.Names.Any(n => n.Label.Contains(myKey))
|| c.Company.CompanyModels.Any(m => m.Model.Name.Equals(myKey))
Am I right? If so, what does the SQL look like? How long does it take just to execute the SQL statement in SQL Profiler? What does the profiler say the execution plan looks like? Have you got indexes on all of the appropriate columns?
Use compiled queries.
If you don't, you will lose up to 5-10x times performance, as LINQ-to-SQL will have to generate SQL from query every time you call it.
Things become worse when you use non-constants in LINQ-to-SQL as getting their values is really slow.
This assumes that you already have indexes and sane DB schema.
BTW, I am not kidding about 5-10x part.