(I work on php and MySQL)
Here's the situation : through an ajax request, I check each 10sec if there's something new for the user in the calendars he's part of.
There are a few things to check, for each calendars (w/e it's a calendar or an other element, but in this case, it's a calendar), so I did a "big" query to return all datas for each calendars, and I treat it with php, and then with js (to pop notifications or w/e).
Three things to check :
is there a "dates" modification since he last visited the calendar ?
is there a new talk in the chat's calendar since he last saw it ?
is there a request to join the calendar which has no answer ?
I did it this way :
SELECT DISTINCT
ci.ci_cid as c_id,
MAX(cc.cc_timestamp) as chat_lastmsg, umchat.um_value as chat_lastview,
MAX(cr.cr_id) as request_id,
cmodif.cm_value as modification_timestamp,
umodif.um_value as modification_view
FROM c_individuals as ci
LEFT JOIN c_chats as cc ON cc.cc_cid = ci.ci_cid
LEFT JOIN u_metas as umchat ON (umchat.um_uid = ci.ci_ciuid AND umchat.um_cid = ci.ci_cid AND umchat.um_name = "chat_last_view")
LEFT JOIN c_requests as cr ON (cr.cr_cid = ci.ci_cid AND cr.cr_answer = "0")
LEFT JOIN c_metas as cmodif ON (cmodif.cm_cid = ci.ci_cid AND cmodif.cm_name = "last_modifications")
LEFT JOIN u_metas as umodif ON (umodif.um_uid = ci.ci_uid AND umodif.um_cid = ci.ci_cid AND umodif.um_name = "last_view")
WHERE ci.ci_uid = :u_id
GROUP BY ci.ci_cid
The c_individuals table has the users who are in a calendar.
The u_metas table has users'metas with um_uid(user id), um_cid (calendar id related to the meta), um_name (name of the meta), um_value (value of the meta)
The c_requests table has requests'informations (u_id, c_id, answer and stuff)
My query gives me what I want, meaning something like :
0 =>
array (size=6)
'c_id' => string '50' (length=2)
'chat_lastmsg' => null
'chat_lastview' => null
'request_id' => null
'modification_timestamp' => string '1488878397.703508' (length=17)
'modification_view' => string '1488877914.048585' (length=17)
1 =>
array (size=6)
'c_id' => string '51' (length=2)
'chat_lastmsg' => string '1485326492' (length=10)
'chat_lastview' => string '1488470529' (length=10)
'request_id' => null
'modification_timestamp' => string '1488878397.703508' (length=17)
'modification_vue' => string '1488878065.811320' (length=17)
But to be honest, I think it's kinda slow. Without treatment, so just to get the results of the query, it takes like 3.5sec to repeat it 1000 times.
It's not a big deal, but this query is repeated a lot, for a potential big number of users.
So my question is : is there any way to improve it, or am I good with this ?
Thank you for your time, and sorry for my bad english.
Don't see anything wrong with your posted query except that you don't need DISTINCT since you are doing a GROUP BY. Also, make sure you have proper index on the columns which are used in JOIN ON condition and WHERE condition.
It's more of a system/database design question rather than a query design.
The query looks adequate enough, and gives the results that you expect, so it's fine (although unsure why you need LEFT joins - but that's down to your data.)
Things to consider in the system design space:
Denormalising the database structure - whenever you enter a c_chat or
c_request, etc. you could look to store the latest timestamp against
the user, so you aren't doing joins - which are very slow.
Could you benefit from a cache closer to the server (e.g. Memcache)?
if you have lots of requests you may want to look into it if the
database is slow. You probably don't need a cache if you're
denormalising the database.
Indexing on anything you use as a filter in the WHERE if you use more than just the Id
For the record,
I finally found something much faster (on my "benchmark test", it's five to seven times faster).
The fact is I don't need to get the informations (dates etc), I just want to know if there is something new since the user saw it for the last time, whenever it was, whenever the new thing was posted.
So I made this (it's in french, plz don't bother) :
SELECT
ci.ci_cid as c_id,
(SELECT COUNT(cc_id)
FROM c_chats
LEFT JOIN u_metas ON um_nom = "chat_derniere_vue"
WHERE cc_cid = ci.ci_cid AND um_uid = ci.ci_uid AND um_cid = ci.ci_cid AND CAST(cc_timestamp as decimal) > CAST(um_valeur as decimal)
) as chat,
(SELECT COUNT(cr_id)
FROM c_requetes
WHERE cr_cid = ci.ci_cid AND ci.ci_ustatut = "titulaire" AND cr_reponse = "0"
) as notifications,
(SELECT COUNT(cm_cid)
FROM c_metas
LEFT JOIN u_metas ON um_nom = "derniere_vue"
WHERE um_uid = ci.ci_uid AND um_cid = ci.ci_cid AND cm_cid = ci.ci_cid AND cm_nom = "derniere_modification_periodes" AND CAST(cm_valeur as decimal) > CAST(um_valeur as decimal)
) as modifications
FROM c_individus as ci
WHERE ci.ci_uid = :u_id
GROUP BY ci.ci_cid
In this way, chat gives the number of chats with a new message, notifications gives the number of unanswer requests, modifications gives the number of calendars with new dates.
And "tada" !
Related
So i'm trying to select unread messages with a query, but the results are blank. I need to get the info from a seperate table read so it will know it was read or not. Im trying to do something like a IF statement so if readed not exists it will be unread, but i can't get it fixed
this is my query:
SELECT * FROM notify
INNER JOIN readed ON readed.acc_ID = '26' AND readed.user_ID = '6'
AND readed.msg_ID = notify.ID AND readed.readed != '1' OR readed.ID IS NULL
WHERE notify.groep = '1'
DB - readed
ID - int
user_ID - int
acc_ID - int
msg_ID - int
readed - enum ('0','1')
DB - notify
ID - int
notfi - text
thumb_src - text
title - text
url - text
groep - int
I hope someone know whats the problem!
The query seems correct to me, except the part with the OR, so I suppose that the problem is with the data. I will first try to show how you could improve the query nevertheless, and then try to show how to debug your data.
First, let's leave away the OR condition since this won't work as expected in a JOIN ON clause.
Second, when comparing integer fields to values, you should not put quotes around the values. This will only worry every person who tries to understand the query because the quotes denote string values, and it will worry (i.e. slow down) MySQL's parser, because it must convert the string values to numbers.
Third, mixing up normal WHERE conditions and JOIN ON conditions is worrying and bad style (IMHO). I always recommend to put only the conditions which actually link the tables into the JOIN ON clause, and other conditions elsewhere.
Following this advice would lead to something like that:
SELECT * FROM
notify INNER JOIN readed ON
readed.msg_ID = notify.ID
WHERE
readed.acc_ID = 26 AND
readed.user_ID = 6 AND
readed.readed != '1' AND
notify.groep = 1
This should do the same as your original query minus the OR part.
Now, since we suspect that there is a problem with the data, we can begin to debug the data. First, leave away the WHERE clause:
SELECT * FROM
notify INNER JOIN readed ON
readed.msg_ID = notify.ID
If this returns data, then you at least know that there are rows in the readed table matching rows in the notify table. If it does not return any data, then there are no rows which fit together, and have found the root of your problem.
Provided that the above returns data, re-add the WHERE clause line by line and test after each step. For example, start with
SELECT * FROM
notify INNER JOIN readed ON
readed.msg_ID = notify.ID
WHERE
readed.acc_ID = 26 AND
and continue with
SELECT * FROM
notify INNER JOIN readed ON
readed.msg_ID = notify.ID
WHERE
readed.acc_ID = 26 AND
readed.user_ID = 6 AND
and so on, testing the query each time.
That way, you hopefully will find out where the problem is. I am convinced that the problem is with the data, not with the query.
I am trying to check if the current user is already following the selected user, and I am doing this like so:
(I know it's not the best way, but as I am new to MYSQL this is as much as I have been able to come up with)
SELECT EXISTS (SELECT 1 FROM Activity WHERE IdOtherUser = 86 AND id = 145)
I am '145' and the user I selected is '86'.
Now that return '0' If I am not following and '1' If I am following that person.
Seems to be working already but it definetly needs improving!
Now what I would like to do is count the followers in the same query.
So count the people I am following and the people following me.
'Activity' is the table where I store the followers and I save them like this:
'id' = me
'idOtherUser' = other user I followed
'type' = type of action "follow"
I have done count's before when calculating the like counts, but I just cannot get my head around this!!
If anyone could spare some time to help me it is much appreciated!
I am sorry if the question is not the best, but I am still learning and trying my best to format them as clear as possible to understand.
Thanks in advance!!
If you trying to count the followers from specific id from table Activity you might do this way:
SELECT COUNT(idOtherUser) AS "I Follow",
(SELECT COUNT(idOtherUser) FROM Activity WHERE idOtherUser = 145 AND type = "follow"
) AS "FOLLOW ME",
(SELECT COALESCE(id,0) FROM Activity WHERE IdOtherUser = 86 AND id = 145 AND type = "follow")
FROM Activity WHERE id = 145 AND type = "follow"
you can use a "correlated subquery" to simplify the query and you might want distinct in the count also (depends on you data). I would avoid using spaces in column aliases too.
SELECT
COUNT(DISTINCT A1.idOtherUser) as i_follow
, (
SELECT
COUNT(DISTINCT A2.id)
FROM Activity A2
WHERE A2.idOtherUser = A1.id
AND A2.type = 'follow'
) as following_me
FROM Activity A1
WHERE A1.id = 145
AND A1.idOtherUser = 86
AND A1.type = 'follow'
Try it with distinct then without, if the result is the same leave distinct out of the query.
CONDENSED VERSION
I'm trying to join a new list with my existing database with no unique identifier -- but I'm trying to figure out a way to do it in one query that's more specific than matching by first name/last name but less specific than by all the fields available (first name/middle name/last name/address/phone).
So my idea was to match solely on first/last name and then try to assign each possible matching field with points to see if anyone who matched had 'zero points' and thus have the first name/last name match stripped from them. Here's what I came up with:
SELECT *,
#MidMatch := IF(LEFT(l.middle,1)=LEFT(d.middle,1),"TRUE","FALSE") MidMatch,
#AddressMatch := IF(left(l.address,5)=left(d.address,5),"TRUE","FALSE") AddressMatch,
#PhoneMatch := IF(right(l.phone,4)=right(d.phone,4),"TRUE","FALSE") PhoneMatch,
#Points := IF(#MidMatch = "TRUE",4,0) + IF(#AddressMatch = "TRUE",3,0) + IF(#PhoneMatch = "TRUE",1,0) Points
FROM list l
LEFT JOIN database d on IF(#Points <> 0,(l.first = d.first AND l.last = d.last),(l.first = d.first AND l.last = d.last AND l.address = d.vaddress));
The query runs fine but it does still match people who's first/last names are identical even if their points are zero (and if their addresses don't match).
Is there a way to do what I'm looking for with this roundabout points system? I've found that it helps me a lot when trying to identify which duplicate to choose, so I'm trying to expand it to the initial match. Or should I do something different?
SPECIFIC VERSION
This is kind of a roundabout idea -- so if somebody has something more straight forward, I'd definitely be willing to bail on this completely and try something else. But basically I have a 93k person table (from a database) that I'm matching against a 92k person table (from a new list). I expect many of them to be the same but certainly not all -- and I'm trying to avoid creating duplicates. Unfortunately, there's no unique identifiers that can be matched, so I'm generally stuck with matching based on some variation of first name, middle name, last name, address, and/or phone number.
The schema for the two tables (list and database) are pretty identical with the fields you see above (first name, middle name, last name, address, phone) -- the only difference is that the database table also has an unique numerical ID that I would use to upload back into the database after this match. Unfortunately the list table has no such ID. Records with the ID would get matched and loaded in on top of the old record and any record without that ID would get loaded as a new record.
What I'm trying to avoid with this question is creating a bunch of different tables and queries that start with a really specific JOIN statement and then eventually get down to just first and last name -- since there's likely some folks who should match but have moved and/or gotten a new phone number since this last list.
I could write a very simple query as a JOIN and do it numerous times, each time taking out another qualifier:
SELECT *
FROM list l
JOIN database d
ON d.first = l.first AND d.last = l.last AND d.middle = l.middle AND d.address = l.address AND d.phone = l.phone;
And I'd certainly feel confident that those people from the new list matched with the existing people in my database, but it'd only return a very small amount of people, then I'd have to go back and loosen the criteria (e.g. drop the middle name restriction, etc.) and continually create tables then merge them all back together at the end along with all the ones that didn't match at all, which I would assume would be the new people.
But is there a way to write the query solely using a first/last name match, then evaluating the other criteria and wiping the match from people who have zero 'points' (below)? Here's what I attempted to do assigning [arbitrary] points to each match:
SELECT *,
#MidMatch := IF(LEFT(l.middle,1)=LEFT(d.middle,1),"TRUE","FALSE") MidMatch,
#AddressMatch := IF(left(l.address,5)=left(d.address,5),"TRUE","FALSE") AddressMatch,
#PhoneMatch := IF(right(l.phone,4)=right(d.phone,4),"TRUE","FALSE") PhoneMatch,
#Points := IF(#MidMatch = "TRUE",4,0) + IF(#AddressMatch = "TRUE",3,0) + IF(#PhoneMatch = "TRUE",1,0) Points
FROM list l
LEFT JOIN database d on IF(#Points <> 0,(l.first = d.first AND l.last = d.last),(l.first = d.first AND l.last = d.last AND l.address = d.vaddress));
The LEFT and RIGHT formulas within the IF statements are just attempting to control for unstandardized data that gets sent. I also would've done something with a WHERE statement, but I still need the NULL values to return so I know who matched and who didn't. So I ended up attempting to use an IF statement in the LEFT JOIN to say that if the Points cell was equal to zero, that the JOIN statement would get really specific and what I thought would hopefully still return the row but it wouldn't be matched to the database even if their first and last name did.
The query doesn't produce any errors, though unfortunately I'm still getting people back who have zeros in their Points column but matched with the database because their first and last names matched (which is what I was hoping the IF/Points stuff would stop).
Is this potentially a way to avoid bad matches, or am I going down the wrong path? If this isn't the right way to go, is there any other way to write one query that will return a full LEFT JOIN along with NULLs that don't match but have it be more specific than just first/last name but less work than doing a million queries based on a new table each time?
Thanks and hopefully that made some sense!
Your first query:
SELECT *,
#MidMatch := IF(LEFT(l.middle,1)=LEFT(d.middle,1),"TRUE","FALSE") MidMatch,
#AddressMatch := IF(left(l.address,5)=left(d.address,5),"TRUE","FALSE") AddressMatch,
#PhoneMatch := IF(right(l.phone,4)=right(d.phone,4),"TRUE","FALSE") PhoneMatch,
#Points := IF(#MidMatch = "TRUE",4,0) + IF(#AddressMatch = "TRUE",3,0) + IF(#PhoneMatch = "TRUE",1,0) Points
FROM list l LEFT JOIN
database d
on IF(#Points <> 0,(l.first = d.first AND l.last = d.last),(l.first = d.first AND l.last = d.last AND l.address = d.vaddress));
This is making a serious mistake with regards to variables. The simplest is the SELECT -- the SELECT does not guarantee the order of calculation of expressions, so they could calculated in any order. And the logic is wrong if #Points is calculated first. This problem is compounded by referring to variables in different clauses. The SQL statement is a logical statement describing the results set, not a programmatic statement of how the query is run.
Let me assume that you have a unique identifier for each row in the database (just to identify the row). Then you can get the match by using a correlated subquery:
select l.*,
(select d.databaseid
from database d
where l.first = d.first and l.last = d.last
order by (4 * (LEFT(l.middle, 1) = LEFT(d.middle, 1) ) +
3 * (left(l.address, 5) = left(d.address, 5)) +
1 * (right(l.phone, 4) = right(d.phone, 4))
)
limit 1
) as did
from list l;
You can join back to the database table to get more information if you need it.
EDIT:
Your comment made it clear. You don't just want the first and last name but something else as well.
select l.*,
(select d.databaseid
from database d
where l.first = d.first and l.last = d.last and
(LEFT(l.middle, 1) = LEFT(d.middle, 1) or
left(l.address, 5) = left(d.address, 5) or
right(l.phone, 4) = right(d.phone, 4)
)
order by (4 * (LEFT(l.middle, 1) = LEFT(d.middle, 1) ) +
3 * (left(l.address, 5) = left(d.address, 5)) +
1 * (right(l.phone, 4) = right(d.phone, 4))
)
limit 1
) as did
from list l;
I would like to do a subquery and then inner join the result of that to produce a query. I want to do this as I have tested an inner join query and it seems to be far more performant on MySql when compared to a straight IN subquery.
Below is a very basic example of the type of sql I am trying to reproduce.
Tables
ITEM
ItemId
Name
ITEMRELATIONS
ItemId
RelationId
Example Sql I would Like to create
Give me the COUNT of RELATIONs for ITEMs having a name of 'bob':
select ir.itemId, count(ir.relationId)
from ItemRelations ir
inner join (select itemId from Items where name = 'bob') sq
on ir.itemId = sq.itemId
group by ir.itemId
The base Nhibernate QueryOver
var bobItems = QueryOver.Of<Item>(() => itemAlias)
.Where(() => itemAlias.Name == "bob")
.Select(Projections.Id());
var bobRelationCount = session.QueryOver<ItemRelation>(() => itemRelationAlias)
.Inner.Join(/* Somehow join the detached criteria here on the itemId */)
.SelectList(
list =>
list.SelectGroup(() => itemRelationAlias.ItemId)
.WithAlias(() => itemRelationCountAlias.ItemId)
.SelectCount(() => itemRelationAlias.ItemRelationId)
.WithAlias(() => itemRelationCountAlias.Count))
.TransformUsing(Transformers.AliasToBean<ItemRelationCount>())
.List<ItemRelationCount>();
I know it may be possible to refactor this into a single query, however the above is merely as simple example. I cannot change the detached QueryOver, as it is handed to my bit of code and is used in other parts of the system.
Does anyone know if it is possible to do an inner join on a detached criteria?
MySql 5.6.5 has addressed the performance issue related to the query structure.
See here: http://bugs.mysql.com/bug.php?id=42259
No need for me to change the output format of my NHibernate queries anymore. :)
I've been trying to write a few little plugins for personal use with WHMCS. Essentially what I'm trying to do here is grab a bunch of information about a certain order(s), and return it as an array in Perl.
The Perl bit I'm fine with, it's the MySQL query I've formed that's giving me stress..
I know it's big and messy, but what I have is:
SELECT tblhosting.id, tblhosting.userid, tblhosting.orderid, tblhosting.packageid, tblhosting.server, tblhosting.domain, tblhosting.username, tblorders.invoiceid, tblproducts.gid, tblservers.ipaddress, tblinvoices.status
FROM tblhosting, tblproducts, tblorders, tblinvoices, tblservers
WHERE tblorders.status = 'Pending'
AND tblproducts.gid = '2'
AND tblservers.id = tblhosting.server
AND tblorders.id = tblhosting.orderid
AND tblinvoices.id = tblorders.invoiceid
AND tblinvoices.status = 'Paid'
I don't know if this /should/ work, but I assume I'm on the right track as it does return what I'm looking for, however it returns everything twice.
For example, I created a new account with the domain 'sunshineee.info', and then in PHPMyAdmin ran the above query.
id userid orderid packageid server domain username invoiceid gid ipaddress status
13 7 17 6 1 sunshineee.info sunshine 293 2 184.22.145.196 Paid
13 7 17 6 1 sunshineee.info sunshine 293 2 184.22.145.196 Paid
Could anyone give me a heads up on where I've gone wrong with this one.. Obvioiusly (maybe not obviously enough) I want this as only one row returned per match.. I've tried it with >1 domain in the database and it returned duplicates for each of the matches..
Any help would be much appreciated
:)
SELECT distinct tblhosting.id, tblhosting.userid, tblhosting.orderid, tblhosting.packageid, tblhosting.server, tblhosting.domain, tblhosting.username, tblorders.invoiceid, tblproducts.gid, tblservers.ipaddress, tblinvoices.status
FROM tblhosting, tblproducts, tblorders, tblinvoices, tblservers
WHERE tblorders.status = 'Pending'
AND tblproducts.gid = '2'
AND tblservers.id = tblhosting.server
AND tblorders.id = tblhosting.orderid
AND tblinvoices.id = tblorders.invoiceid
AND tblinvoices.status = 'Paid'
Well, its near impossible without any table definitions, but you are doing a lot of joins there. You are starting with tblhosting.id and working your way 'up' from there. If any of the connected tables has a double entry, you'll get more hits
You could add a DISTINCT to your query, but that would not fix the underlying issue. It could be a problem with your data: do you have 2 invoices? Maybe you should select everything (SELECT * FROM) and check what is returned, maybe check your tables for double content.
Using DISTINCT is most of the time not a good choice: it means either your query or your data is incorrect (or you don't understand them thoroughly). It might get you the right result for now, but can get you in trouble later.
A guess about the reason this happens:
You do not connect the products table to the chain of id's. So you are basically adding a '2' to your result as far as I can see. You join on products, and the only thing that limits that table is that "gid" should be 2. So if you add a product with gid 2 you get another result. Either join it (maybe tblproduct.orderid = tblorders.id ? just guessing here) or just remove it, as it does nothing as far as I can see.
If you want to make your query a bit clearer, try not implicitly joining, but do it like this. So you can actually see what's happening
SELECT tblhosting.id, tblhosting.userid, tblhosting.orderid, tblhosting.packageid, tblhosting.server, tblhosting.domain, tblhosting.username, tblorders.invoiceid, tblproducts.gid, tblservers.ipaddress, tblinvoices.status
FROM tblhosting
JOIN tblproducts ON /*you're missing something here!*/
JOIN tblorders ON tblorders.id = tblhosting.orderid
JOIN tblinvoices ON tblinvoices.id = tblorders.invoiceid
JOIN tblservers ON tblservers.id = tblhosting.server
WHERE
tblorders.status = 'Pending'
AND tblproducts.gid = '2'
AND tblinvoices.status = 'Paid'
I don't see in your query JOIN to tblproducts, it seems to be a reason.