I have a mySQL database table t1 with a bigint field called filter which I need to 'and' to a given query parameter in order to filter out entries.
table t1:
filter bigint unsigned
info varchar(200)
As a pure mySQL statement my query would be this:
SELECT info FROM t1 WHERE (filter & 34603933) = filter;
Of course the number 34603933 is a parameter and changes from query to query.
The question is, if loopbackjs supports such a calculus in the where condition. Can someone point me the way, or if it is not possible suggest a workaround?
In the documentation http://docs.strongloop.com/display/public/LB/Where+filter I did not see a possibility to do this, but somehow I can't really believe it, since using references to columns values in the right side of a where comparison is nothing unusual, right?
I do not believe loopback has this support built-in and bitwise and operations are not that widespread (in my experience). You might try the raw SQL interface:
var mysqlDS = app.dataSources.mysqlDS;
var bitWiseAndValue = 34603933;
mysqlDS.connector.query('SELECT info FROM t1 WHERE (filter & '+bitWiseAndValue+') = filter', function(err) {
if (err) {return console.log(err);}
// success actions
});
Related
Suppose I have Post model that has is_verified column with smallint datatype, how can I get all records that is verified? One thing to do this is using this:
Post::where('is_verified', true)->get();
The code above will produce the following query:
select * from `posts` where `posts`.`is_verified` = true
... which will get me all verified Post records; in note that is_verified on all existing records is either 0 or 1.
However, after I get myself curious and try to manually change some is_verified's record value from 1 to another truthy number e.g. 2, the above eloquent query didn't work as expected anymore: records with is_verified value of 2 didn't get retrieved.
I tried to execute the sql query directly from HeidiSQL as well, but it was just the same. Then I tried to change the = in the sql query to is, and now it's working as expected i.e. all records with truthy is_verified get retrieved:
select * from `posts` where `posts`.`is_verified` is true
So my questions are:
Does the above behaviour is correct and expected?
How can I execute the last sql query in eloquent? One thing I can think of is where('is_verified', '!=', 0) but that feels weird in terms of readability especially when the query is pretty long and a bit complicated
As I stated before, the is_verified column is a smallint. Does this affects the behaviour? Because this conversation here states that boolean column datatype is typically tinyint, not smallint.
And that's it. Thank you in advance!
It is not the correct way to handle boolean values, you shouldn't save boolean columns as smallint, you can use the explicit boolean column type as described in the documentation.
Once you setup the boolean field correctly the logic you have in place will work. So Post::where('is_verified', true)->get(); will return the expected results.
Yes, the problem is the smallint column type, if you put tinyint it also should work like the boolean column. You can read more about the differences here.
After doing some deeper digging, I would like to write down the things I've found:
I have updated my mysql to the newest version as of now (v8) and boolean datatype defined in migration results in tinyint(1) in the db. This is happening turns out because in mysql bool or boolean are actually just the synonyms of tinyint(1), so that was a totally normal behaviour, not due to lower-version issues.
I found #dz0nika answer that states that smallint and tinyint results in different behaviour in the query to be quite incorrect. The two datatypes simply differ in terms of byte-size while storing integer value.
As of mysql documentation, it is stated that:
A value of zero is considered false. Nonzero values are considered true.
But also that:
However, the values TRUE and FALSE are merely aliases for 1 and 0, respectively.
Meaning that:
select * from `posts` where `posts`.`is_verified` = true;
Is the same as
select * from `posts` where `posts`.`is_verified` = 1;
Thus the query will only get Post records with is_verified value of 1.
To get Post records with truthy is_verified value, wether 1, or 2, or 3, etc; use is instead of = in the query:
select * from `posts` where `posts`.`is_verified` is true;
You can read more about these informations here and here (look for the "boolean" part)
So, how about the eloquent query? How can we get Post with truthy is_verified using eloquent?
I still don't know what's best. But instead of using where('is_verified', '!=', 0) as I stated in my question, I believe it's better to use whereRaw() instead:
Post::whereRaw('posts.is_verified is true')->get();
If you found this information to be quite missing or incorrect, please kindly reply. Your opinion is much appreciated.
The SQLite JSON1 extension has some really neat capabilities. However, I have not been able to figure out how I can update or insert individual JSON attribute values.
Here is an example
CREATE TABLE keywords
(
id INTEGER PRIMARY KEY,
lang INTEGER NOT NULL,
kwd TEXT NOT NULL,
locs TEXT NOT NULL DEFAULT '{}'
);
CREATE INDEX kwd ON keywords(lang,kwd);
I am using this table to store keyword searches and recording the locations from which the search was ininitated in the object locs. A sample entry in this database table would be like the one shown below
id:1,lang:1,kwd:'stackoverflow',locs:'{"1":1,"2":1,"5":1}'
The location object attributes here are indices to the actual locations stored elsewhere.
Now imagine the following scenarios
A search for stackoverflow is initiated from location index "2". In this case I simply want to increment the value at that index so that after the operation the corresponding row reads
id:1,lang:1,kwd:'stackoverflow',locs:'{"1":1,"2":2,"5":1}'
A search for stackoverflow is initiated from a previously unknown location index "7" in which case the corresponding row after the update would have to read
id:1,lang:1,kwd:'stackoverflow',locs:'{"1":1,"2":1,"5":1,"7":1}'
It is not clear to me that this can in fact be done. I tried something along the lines of
UPDATE keywords json_set(locs,'$.2','2') WHERE kwd = 'stackoverflow';
which gave the error message error near json_set. I'd be most obliged to anyone who might be able to tell me how/whether this should/can be done.
It is not necessary to create such complicated SQL with subqueries to do this.
The SQL below would solve your needs.
UPDATE keywords
SET locs = json_set(locs,'$.7', IFNULL(json_extract(locs, '$.7'), 0) + 1)
WHERE kwd = 'stackoverflow';
I know this is old, but it's like the first link when searching, it deserves a better solution.
I could have just deleted this question but given that the SQLite JSON1 extension appears to be relatively poorly understood I felt it would be more useful to provide an answer here for the benefit of others. What I have set out to do here is possible but the SQL syntax is rather more convoluted.
UPDATE keywords set locs =
(select json_set(json(keywords.locs),'$.**N**',
ifnull(
(select json_extract(keywords.locs,'$.**N**') from keywords where id = '1'),
0)
+ 1)
from keywords where id = '1')
where id = '1';
will accomplish both of the updates I have described in my original question above. Given how complicated this looks a few explanations are in order
The UPDATE keywords part does the actual updating, but it needs to know what to updatte
The SELECT json_set part is where we establish the value to be updated
If the relevant value does not exsit in the first place we do not want to do a + 1 on a null value so we do an IFNULL TEST
The WHERE id = bits ensure that we target the right row
Having now worked with JSON1 in SQLite for a while I have a tip to share with others going down the same road. It is easy to waste your time writing extremely convoluted and hard to maintain SQL in an effort to perform in-place JSON manipulation. Consider using SQLite in memory tables - CREATE TEMP TABLE... to store intermediate results and write a sequence of SQL statements instead. This makes the code a whole lot eaiser to understand and to maintain.
So i do a simple query like so:
connection.query("UPDATE workers SET timestamp='"+thedate+"' WHERE id = " + id,function(err,upres){
connection.release();
if(!err) {
console.log('updated record')
console.log(upres);
}
})
console.log reveals the data format as: 2015-04-02 19:29:14
And if i debug the SQL statement, that turns out to be:
UPDATE workers SET timestamp='2015-04-02 21:31:16' WHERE id = 3;
However, when i list the data, the output is:
[{"id":3,"worker":"John Doe","timestamp":"2015-04-01T22:00:00.000Z","duration":30}]
This is way off compared to the time that is being reported?
What is causing this?
You do not know how MySQL is turning your VARCHAR into a date. There are a lot of configuration options. It would be better to use the STR_TO_DATE function to circumvent all of the assumptions. Here is a link to the docs for STR_TO_DATE().
As a side note, I would strongly recommend using prepared statements as a way to safeguard your application against errors and sql injection.
EDITS:
In regards to your questions, the column could be DATETIME, but your value you are assigning is a VARCHAR
'UPDATE workers SET timestamp = ? WHERE id = ?', ['4/2/2015 3:00:00 PM', 3'], [callBackFunction]
Based on what you said about the conversion not working, I am suspicious about the data type for the timestamp column.
SELECT * FROM INFORMATION_SCHEMA.COLUMNS WHERE NAME = 'workers'
A statement like that should give you all of the information about that column. You could also find this in a GUI, if you have access. There are three different date types in MySQL date, datetime, or timestamp. This is most likely a DATE column, that will not be able to hold the time.
I'm using LINQ to Entities to display paged results. But I'm having issues with the combination of Skip(), Take() and OrderBy() calls.
Everything works fine, except that OrderBy() is assigned too late. It's executed after result set has been cut down by Skip() and Take().
So each page of results has items in order. But ordering is done on a page handful of data instead of ordering of the whole set and then limiting those records with Skip() and Take().
How do I set precedence with these statements?
My example (simplified)
var query = ctx.EntitySet.Where(/* filter */).OrderByDescending(e => e.ChangedDate);
int total = query.Count();
var result = query.Skip(n).Take(x).ToList();
One possible (but a bad) solution
One possible solution would be to apply clustered index to order by column, but this column changes frequently, which would slow database performance on inserts and updates. And I really don't want to do that.
EDIT
I ran ToTraceString() on my query where we can actually see when order by is applied to the result set. Unfortunately at the end. :(
SELECT
-- columns
FROM (SELECT
-- columns
FROM (SELECT -- columns
FROM ( SELECT
-- columns
FROM table1 AS Extent1
WHERE EXISTS (SELECT
-- single constant column
FROM table2 AS Extent2
WHERE (Extent1.ID = Extent2.ID) AND (Extent2.userId = :p__linq__4)
)
) AS Project2
limit 0,10 ) AS Limit1
LEFT OUTER JOIN (SELECT
-- columns
FROM table2 AS Extent3 ) AS Project3 ON Limit1.ID = Project3.ID
UNION ALL
SELECT
-- columns
FROM (SELECT -- columns
FROM ( SELECT
-- columns
FROM table1 AS Extent4
WHERE EXISTS (SELECT
-- single constant column
FROM table2 AS Extent5
WHERE (Extent4.ID = Extent5.ID) AND (Extent5.userId = :p__linq__4)
)
) AS Project6
limit 0,10 ) AS Limit2
INNER JOIN table3 AS Extent6 ON Limit2.ID = Extent6.ID) AS UnionAll1
ORDER BY UnionAll1.ChangedDate DESC, UnionAll1.ID ASC, UnionAll1.C1 ASC
My workaround solution
I've managed to workaround this problem. Don't get me wrong here. I haven't solved precedence issue as of yet, but I've mitigated it.
What I did?
This is the code I've used until I get an answer from Devart. If they won't be able to overcome this issue I'll have to use this code in the end.
// get ordered list of IDs
List<int> ids = ctx.MyEntitySet
.Include(/* Related entity set that is needed in where clause */)
.Where(/* filter */)
.OrderByDescending(e => e.ChangedDate)
.Select(e => e.Id)
.ToList();
// get total count
int total = ids.Count;
if (total > 0)
{
// get a single page of results
List<MyEntity> result = ctx.MyEntitySet
.Include(/* related entity set (as described above) */)
.Include(/* additional entity set that's neede in end results */)
.Where(string.Format("it.Id in {{{0}}}", string.Join(",", ids.ConvertAll(id => id.ToString()).Skip(pageSize * currentPageIndex).Take(pageSize).ToArray())))
.OrderByDescending(e => e.ChangedOn)
.ToList();
}
First of all I'm getting ordered IDs of my entities. Getting only IDs is well performant even with larger set of data. MySql query is quite simple and performs really well. In the second part I partition these IDs and use them to get actual entity instances.
Thinking of it, this should perform even better than the way I was doing it at the beginning (as described in my question), because getting total count is much much quicker due to simplified query. The second part is practically very very similar, except that my entities are returned rather by their IDs instead of partitioned using Skip and Take...
Hopefully someone may find this solution helpful.
I haven't worked directly with Linq to Entities, but it should have a way to hook specific stored procedures into certain locations when needed. (Linq to SQL did.) If so, you could turn this query into a stored procedure, doing exacly what is required, and doing it efficiently.
Assuming from you comment the persisting the values in a List is not acceptable:
There's no way to completely minimize the iterations, as you intended (and as I would have tried too, living in hope). Cutting the iterations down by one would be nice. Is it possible to just get the Count once and cache/session it? Then you could:
int total = ctx.EntitySet.Count; // Hopefully you can not repeat doing this.
var result = ctx.EntitySet.Where(/* filter */).OrderBy(/* expression */).Skip(n).Take(x).ToList();
Hopefully you can cache the Count somehow, or avoid needing it every time. Even if you can't, this is the best you can do.
Could you please create a sample illusrating the problem and send it to us (support * devart * com, subject "EF: Skip, Take, OrderBy")?
Hope we will be able to help you.
You can also contact us using our forums or contact form.
Are you absolutely certain the ordering is off? What does the SQL look like?
Can you reorder your code as follows and post the output?
// Redefine your queries.
var query = ctx.EntitySet.Where(/* filter */).OrderBy(e => e.ChangedDate);
var skipped = query.Skip(n).Take(x);
// let's look at the SQL, shall we?
var querySQL = query.ToTraceString();
var skippedSQL = skipped.ToTraceString();
// actual execution of the queries...
int total = query.Count();
var result = skipped.ToList();
Edit:
I'm absolutely certain. You can check my "edit" to see trace result of my query with skipped trace result that is imperative in this case. Count is not really important.
Yeah, I see it. Wow, that's a stumper. Might even be an outright bug. I note you're not using SQL Server... what DB are you using? Looks like it might be MySQl.
One way:
var query = ctx.EntitySet.Where(/* filter */).OrderBy(/* expression */).ToList();
int total = query.Count;
var result = query.Skip(n).Take(x).ToList();
Convert it to a List before skipping. It's not too efficient, mind you...
I have a few rows of data pulled into business objects via linq-to-sql from large tables.
Now I want to get a few rows that don't match to test my comparison functions.
Using what I thought would work I get a NotSupportedException:
Local sequence cannot be used in LINQ to SQL implementation of query operators except the Contains() operator.
Here's the code:
//This table has a 2 field primary key, the other has a single
var AllNonMatches = from c in dc.Acaps
where !Matches.Rows.Any((row) => row.Key.Key == c.AppId & row.Key.Value == c.SeqNbr)
select c;
foreach (var item in AllNonMatches.Take(100)) //Exception here
{}
The table has a compound primary key: AppId and SeqNbr.
The Matches.Rows is defined as a dictionary of keyvaluepair(appid,seqnbr).
and the local sequence it is referring to appears to be the local dictionary.
Could you provide more information on the structure and the name(s) of the table(s) plz?
Not sure what you're trying to do...
edit:
Ok.. I think I get it now...
It appears you can't merge/join local tables (dictionary) with a SQL table.
If you can, I'm afraid I don't know how to do it.
The simplest solution I can think of is to put those results in a table ("Match" for instance) with foreign keys related to your table "Acaps" and then use linq-to-sql, like:
var AllNonMatches = dc.Acaps.Where(p=>p.Matchs==null).Take(100).ToList();
Sorry I couldn't come up with any better =(
What about this:
var AllNonMatches = from c in dc.Acaps
where !(Matches.Rows.ContainsKey(c.AppId) && Matches.Rows.ContainsValue(c.SeqNbr))
select c;
That will work fine. I have also used a bitwise AND operator (&&) - I think thats the right term to help improve performance over the standard AND operator.