Slick flatMap does not execute all queries as a group - mysql

I fount that Scala Slick can not execute multiple queries as plain sql.
For example:
val query = sql"""
SET #referenceTime = '12:00:00';
SELECT * FROM table WHERE dateTime <= #referenceTime;
""".as[ClassResult]
dbConfig.db.run(query)
In this string are 2 queries, and Slick return an error as :
You have an error in your SQL syntax; check the manual .... to use near 'SELECT * FROM
From this, I understand that all queries before "SELECT" (maybe the last query) are ignored. So, I found a solution using flatMap, but is not the perfect.
val timeQuery = sql"""SET #referenceTime = '12:00:00';""".as[String]
val dataQuery = sql"""SELECT * FROM table WHERE dateTime <= #referenceTime;""".as[ClassResult]
val composedAction = for {
timeRes <- timeQuery,
dataRes <- dataQuery
} yield dataRes
dbConfig.db.run(composedAction)
This run and in 99% of cases return the result (a list of items). But, sometimes the list is empty, even if I'm sure that must return some data). So, I think that composedAction doesn't wait and execute both queries as a group every time. How I can do this, because I need in second query the result of the first (is used as a parameter in second)
edit:
another solution is to wait for result of the first query and use it as parameter in second. But it is a good solution/practice? Using sync code.
val timeQuery = sql"""SELECT '12:00:00';""".as[String]
var defaultTime: String = ""
val tempResult = dbConfig.db.run(timeParameterQuery.head).map(res => defaultTime = res)
Await.result(tempResult, Duration.Inf)
val dataQuery = sql"""SELECT * FROM table WHERE dateTime <= $defaultTime;""".as[ClassResult]
dbConfig.db.run(dataQuery)

I guess using SQL variables in slick is problematic, could be multiple db contexts. You can chain the queries with the 'map' rather than awaiting the result. Try something like this (untested)
val timeQuery = sql"""SELECT '12:00:00';""".as[String]
dbConfig.db.run(timeQuery.head).flatMap{res =>
dbConfig.db.run(sql"""SELECT * FROM table WHERE dateTime <= $res;""".as[ClassResult])
}

Related

Pattern matching using regex with Scala Anorm

I'm Using Scala(2.11) and playframework(2.3) and trying to run a query using a helper function to get results through pattern matching. The function is as follows
def resultsfunc() = {
val gradeRegex = "^Class 5\."
val currRegex = "\.NCERT$"
DB.withConnection{ implicit c =>
val filterQuery = SQL(
"""
select * from tbl_graphs
where graph_name REGEXP '{grade_regex}' and
graph_name REGEXP '{curr_regex}' and org_id = 4
""")
.on("grade_regex" -> gradeRegex,
"curr_regex" -> currRegex)
filterQuery().map{ graphRecord =>
new ResultObj(graphRecord[Long]("id"),
graphRecord[String]("name"))
}.toList
}
}
I don't get any errors but I get empty result even though there are multiple records that match the pattern. The same query works if I try to run in mysql workbench and when I tried to print filterQuery the arguments were also mapped correctly.
Should Pattern matching with regex must be carried out differently in Scala Anorm ?
It has absolutely nothing to do specifically with Anorm.
Make sure that executing manually the query with exactly the same data and parameter, you get result.
When using JDBC (even through Anorm), string parameter must not be quoted in the query statement (... '{grade_regex}' ...).
Since a long time, it's recommended to use Anorm interpolation (SQL"SELECT x FROM y WHERE z = ${v}") rather than SQL(..) function.

Hash a Select SQLAlchemy query

I have a SQLAlchemy query that I build, such as :
query_one = User.query.filter(User.id == 1) # Note that I don't call .first() or .all() as I want the "select" instance.
I want to store this Select query in such a way that I can retrieve it by having the same query :
stored_queries = {}
stored_queries[hash(query_one)] = query_one
# ... later on:
query_two = User.query.filter(User.id == 1)
if hash(query_two) in stored_queries:
# Execute custom code because it's the same query
Of course, hash in that case does not work, but is there a SQLAlchemy method that works in the same way?
I thought of str(query_one), but that query only consider the request, without the value. I need both.
Thank you in advance.
You can compile the query to get access to the parameters, and use those as part of your key:
def query_key(query):
statement = query_one.statement.compile()
return str(statement), str(statement.params)
query_key(query_one)
('SELECT user.id, ... FROM user WHERE user.id = :id_1', "{'id_1': 1}")
See https://docs.sqlalchemy.org/en/14/core/selectable.html#sqlalchemy.sql.expression.TableClause.compile

Unable to retrieve data from my sql database using pymysql

I have been trying to retrieve data from my database. I was successful, however, this time inside an if statement. The code looks like:
cur_msql = conn_mysql.cursor(cursor=pymysql.cursors.DictCursor)
select_query = """select x,y,z from table where type='sample' and code=%s"""
cur_msql.execute(select_query, code)
result2 = cur_msql.fetchone()
if(result2==None):
insert_func(code)
select_query = f"""select x,y,z from table where type='sample' and code='{code}'"""
mycur = conn_mysql.cursor(cursor=pymysql.cursors.DictCursor)
print(select_query)
mycur.execute(select_query)
result3 = mycur.fetchone()
if(result2==None):
result2=result3
Now I see that insert_func does successfully insert into the 'table'. However, on trying to fetch that row, immediately after the insertion, it returns None as if the row is absent. On debugging I find that result3 is also None. Nothing looks wrong to me but it's not working.
you donĀ“t execute it in the right way, in the cur_msql.execute, you the to send the query and a tuple of values, and you are sending just a value:
cur_msql = conn_mysql.cursor(cursor=pymysql.cursors.DictCursor)
select_query = "select learnpath_code,learnpath_id,learnpath_name from contentgrail.knowledge_vectors_test where Type='chapters' and code=%s"
cur_msql.execute(select_query, (meta['chapter_code'],))
result2 = cur_msql.fetchone()

Linq to Sql: Join, why do I need to load a collection

I have 2 tables that I need to load together all the time, the both must exist together in the database. However I am wondering why Linq to Sql demands that I have to load in a collection and then do a join, I only want to join 2 single tables where a record where paramid say = 5, example...
var data = _repo.All<TheData>(); //why do I need a collection/IQueryable like this?
var _workflow = _repo.All<WorkFlow>()
.Where(x => x.WFID== paramid)
.Join(data, x => x.ID, y => y.WFID, (x, y) => new
{
data = x,
workflow = y
});
I gues then I need to do a SingleOrDefault()? If the record is not null pass it back?
I Understand the Sql query comes out correctly, is there a better way to write this?
NOTE: I need to search a table called Participants to see if the loggedonuser can actually view this record, so I guess I should leave it as this? (this is main requirement)
var participant = _repo.All<Participants>();
.Any(x=> x.ParticipantID == loggedonuser.ID); //add this to above query...
The line var data = _repo.All<TheData>(); is something like saying 'start building query against the TheData table'.
This function returns you an IQueryable which will contain a definition of the query against your database.
So this doesn't mean you load the whole TheData table data with this line!
The query will be executed the moment you do something like .Count(), .Any(), First(), Single(), or ToList(). This is called deferred execution.
If you would end your query with SingleOrDefault() this will create a sql query that joins the two tables, add the filter and select the top most record or null(or throw an error if there are more!).
You could also use Linq instead of query extension methods.
It would look like:
var data = _repo.All<TheData>();
var _workflow = from w in _repo.All<WorkFlow>()
join t in _repo.All<TheData> on w.Id equals t.WFID
where x.WIFD = paramid
select new
{
data = t,
workflow = x
});

Could not format node 'Value' for execution as SQL

I've stumbled upon a very strange LINQ to SQL behaviour / bug, that I just can't understand.
Let's take the following tables as an example: Customers -> Orders -> Details.
Each table is a subtable of the previous table, with a regular Primary-Foreign key relationship (1 to many).
If I execute the follow query:
var q = from c in context.Customers
select (c.Orders.FirstOrDefault() ?? new Order()).Details.Count();
Then I get an exception: Could not format node 'Value' for execution as SQL.
But the following queries do not throw an exception:
var q = from c in context.Customers
select (c.Orders.FirstOrDefault() ?? new Order()).OrderDateTime;
var q = from c in context.Customers
select (new Order()).Details.Count();
If I change my primary query as follows, I don't get an exception:
var q = from r in context.Customers.ToList()
select (c.Orders.FirstOrDefault() ?? new Order()).Details.Count();
Now I could understand that the last query works, because of the following logic:
Since there is no mapping of "new Order()" to SQL (I'm guessing here), I need to work on a local list instead.
But what I can't understand is why do the other two queries work?!?
I could potentially accept working with the "local" version of context.Customers.ToList(), but how to speed up the query?
For instance in the last query example, I'm pretty sure that each select will cause a new SQL query to be executed to retrieve the Orders. Now I could avoid lazy loading by using DataLoadOptions, but then I would be retrieving thousands of Order rows for no reason what so ever (I only need the first row)...
If I could execute the entire query in one SQL statement as I would like (my first query example), then the SQL engine itself would be smart enough to only retrieve one Order row for each Customer...
Is there perhaps a way to rewrite my original query in such a way that it will work as intended and be executed in one swoop by the SQL server?
EDIT:
(longer answer for Arturo)
The queries I provided are purely for example purposes. I know they are pointless in their own right, I just wanted to show a simplistic example.
The reason your example works is because you have avoided using "new Order()" all together. If I slightly modify your query to still use it, then I still get an exception:
var results = from e in (from c in db.Customers
select new { c.CustomerID, FirstOrder = c.Orders.FirstOrDefault() })
select new { e.CustomerID, Count = (e.FirstOrder != null ? e.FirstOrder : new Order()).Details().Count() }
Although this time the exception is slightly different - Could not format node 'ClientQuery' for execution as SQL.
If I use the ?? syntax instead of (x ? y : z) in that query, I get the same exception as I originaly got.
In my real-life query I don't need Count(), I need to select a couple of properties from the last table (which in my previous examples would be Details). Essentially I need to merge values of all the rows in each table. Inorder to give a more hefty example I'll first have to restate my tabels:
Models -> ModelCategoryVariations <- CategoryVariations -> CategoryVariationItems -> ModelModuleCategoryVariationItemAmounts -> ModelModuleCategoryVariationItemAmountValueChanges
The -> sign represents a 1 -> many relationship. Do notice that there is one sign that is the other way round...
My real query would go something like this:
var q = from m in context.Models
from mcv in m.ModelCategoryVariations
... // select some more tables
select new
{
ModelId = m.Id,
ModelName = m.Name,
CategoryVariationName = mcv.CategoryVariation.Name,
..., // values from other tables
Categories = (from cvi in mcv.CategoryVariation.CategoryVariationItems
let mmcvia = cvi.ModelModuleCategoryVariationItemAmounts.SingleOrDefault(mmcvia2 => mmcvia2.ModelModuleId == m.ModelModuleId) ?? new ModelModuleCategoryVariationItemAmount()
select new
{
cvi.Id,
Amount = (mmcvia.ModelModuleCategoryVariationItemAmountValueChanges.FirstOrDefault() ?? new ModelModuleCategoryVariationItemAmountValueChange()).Amount
... // select some more properties
}
}
This query blows up at the line let mmcvia =.
If I recall correctly, by using let mmcvia = new ModelModuleCategoryVariationItemAmount(), the query would blow up at the next ?? operand, which is at Amount =.
If I start the query with from m in context.Models.ToList() then everything works...
Why are you looking into only the individual count without selecting anything related to the customer.
You can do the following.
var results = from e in
(from c in db.Customers
select new { c.CustomerID, FirstOrder = c.Orders.FirstOrDefault() })
select new { e.CustomerID, DetailCount = e.FirstOrder != null ? e.FirstOrder.Details.Count() : 0 };
EDIT:
OK, I think you are over complicating your query.
The problem is that you are using the new WhateverObject() in your query, T-SQL doesnt know anyting about that; T-SQL knows about records in your hard drive, your are throwing something that doesn't exist. Only C# knows about that. DON'T USE new IN YOUR QUERIES OTHER THAN IN THE OUTER MOST SELECT STATEMENT because that is what C# will receive, and C# knows about creating new instances of objects.
Of course is going to work if you use ToList() method, but performance is affected because now you have your application host and sql server working together to give you the results and it might take many calls to your database instead of one.
Try this instead:
Categories = (from cvi in mcv.CategoryVariation.CategoryVariationItems
let mmcvia =
cvi.ModelModuleCategoryVariationItemAmounts.SingleOrDefault(
mmcvia2 => mmcvia2.ModelModuleId == m.ModelModuleId)
select new
{
cvi.Id,
Amount = mmcvia != null ?
(mmcvia.ModelModuleCategoryVariationItemAmountValueChanges.Select(
x => x.Amount).FirstOrDefault() : 0
... // select some more properties
}
Using the Select() method allows you to get the first Amount or its default value. I used "0" as an example only, I dont know what is your default value for Amount.