My head is already spinning from this and I need your help.
MY DATABASE
imported CSV file: 22 columns and 11k rows
2 tables with the same data (both created from the CSV)
Added ID as PRIMARY KEY to both
All VARCHAR(60) Some columns are empty strings ' '
DB:
PID | CODE 1 | CODE 2 | CODE 3 | CODE 4 | CODE 5 | CODE X (up to 9) | ID
-------------------------------------------------------------------------
1 | a | b | c | | | | 1
2 | a | | b | d | | | 2
3 | x | | | | | y | 3
DB has 22 columns but I'm only including CODE columns (up to 9)
in which I might be interested in terms of SQL statement.
It'll be only read table - MyISAM engine then?
WHAT I'D LIKE TO DO
select PID = 1 from first table
and retrieve all PIDs from second table
IF
selected PID's column CODE 1
or
selected PID's column CODE 2 (which is b) etc (up to 9).
= any PID's CODE X
So I should get only PID 2.
edit: PID is not a ID, it's just an example code, it could be string: '002451' and I'm looking for other PIDs with the same CODES (e.g PID1 has code = a so it should find PID2 becasue one of its CODE columns contains a)
MY ATTEMPT
SELECT a.* FROM `TABLE1` a WHERE
(
SELECT * FROM `TABLE2` b WHERE b.`PID` = 1
AND
(
( b.`CODE 1` NOT IN ('') AND IN (a.`CODE 1`,a.`CODE 2`, A.`CODE 3`...) ) OR
( b.`CODE 2` NOT IN ('') AND (a.`CODE 1`,a.`CODE 2`, A.`CODE 3`...) ) OR...
I'd end up with large query - over 81 conditions. In terms of performance... well, it doesn't work.
I intuitively know that I should:
use INDEXES (on CODE 1 / CODE 2 / CODE 3 etc.?)
use JOIN ON (but I'm too stupid) - that's why I created 2 tables (let's assume I don't want TEMP. TABLES)
How to write the SQL / design the DB efficently?
The correct data structure is one row per pid and code. The simplest way is:
create table PCodes (
pid int not null,
code varchar(255),
constraint fk_PCodes_pid references p(pid)
);
Then you have the values in a single column and it is much simpler to check for matching codes.
In practice, you should have three tables:
create table Codes (
CodeId int not null auto_increment primary key,
Code varchar(255)
);
create table PCodes (
pid int not null,
codeid int not null,
constraint fk_PCodes_pid references p(pid),
constraint fk_PCodes_codeid references codes(codeid);
);
If the ordering of the codes is important for each "p", then include a priority or ordering column in the PCodes table.
Related
I have 12 fixed tables (group, local, element, sub_element, service, ...), each table with different numbers of rows.
The columns 'id_' in all table is a primary key (int). The others columns are of datatype varchar(20). The maximum number of rows in these tables are 300.
Each table was created in this way:
CREATE TABLE group
(
id_G int NOT NULL,
name_group varchar(20) NOT NULL,
PRIMARY KEY (id_G)
);
|........GROUP......| |.......LOCAL.......| |.......SERVICE.......|
| id_G | name_group | | id_L | name_local | | id_S | name_service |
+------+------------+ +------+------------+ +------+--------------+
| 1 | group1 | | 1 | local1 | | 1 | service1 |
| 2 | group2 | | 2 | local2 | | 2 | service2 |
And I have one table that combine all these tables depending on user selects.
The 'id_' come from fixed tables selected by the user are recorded into this table.
This table was crate in this way:
CREATE TABLE group
(
id_E int NOT NULL,
event_name varchar(20) NOT NULL,
id_G int NOT NULL,
id_L int NOT NULL,
...
PRIMARY KEY (id_G)
);
The tables (event) look like this:
|....................EVENT.....................|
| id_E | event_name | id_G | id_L | ... |id_S |
+------+-------------+------+------+-----+-----+
| 1 | mater1 | 1 | 1 | ... | 3 |
| 2 | master2 | 2 | 2 | ... | 6 |
This table get greater each day, an now it has about thousunds of rows.
Column id_E is the primary key (int), event_name is varchar(20).
This table has, in addition of id_E and event_name columns, 12 other columns the came from the fixed tables.
Every time than I need to retrieve information on the event table, to turn more readable, I need to do about 12 joins.
My query look like this where i need to retrieve all columns from table event:
SELECT event_name, name_group, name_local ..., name_service
FROM event
INNER JOIN group on event.id_G = group.id_G
INNER JOIN local on event.id_L = local.id_L
...
INNER JOIN service on event.id_S = service.id_S
WHERE event.id_S = 7 (for example)
This slows down my system performance. Is there a way to reduce the number of joins? I've heard about using Natural Keys, but I think this is not a good idea to form my case thinking in future maintenance.
My queries are taking about 7 seconds and I need to reduce this time.
I changed the WHERE clause and this caused not affect. So, I am sure that the problem is that the query has so many joins.
Could someone give some help? thanks a lot...
MySQL has a great keyword of "STRAIGHT_JOIN" and might be what you are looking for. First, each of your lookup tables (id/description) I have to assume already have an index on the ID column since that is primary key.
Your event table is the one you are querying as the primary basis of the details and joining to the lookups per their respective IDs. As long as your WHERE clause applicable to the EVENT table is optimized, such as the ID you are looking for, it SHOULD be virtually instantaneous.
If it is not, then it might be that MySQL is trying to think for you and take one of the secondary lookup tables and make it a primary basis of the query for whatever reason, such as much lower record count. In this case, add the keyword and try it..
SELECT STRAIGHT_JOIN ... rest of your query
This tells MySQL to do the query in the order you gave it, thus the Event table first and it's where clause on the ID. It should find that one thing, then grab all the corresponding lookup descriptions from the other tables.
Create indexes, concretely use compound indexes, for instance, start creating a compound index for event and groups:
on table events create one for (event id, group id).
then, on the group table create another one for the next relation (group id, local id).
on local do the same with service, and so on...
Disclaimer: I'm using MySQL with 2 tables. So far I've found solutions to my issue when individual groups are queried one at a time using IN() but nothing that allows me to do the whole table at once without looping over multiple queries.
I have two tables:
CREATE TABLE WordGroups (
wgId int NOT NULL AUTO_INCREMENT,
groupId int NOT NULL,
word varchar(255) NOT NULL,
PRIMARY KEY (wgId)
);
Which keeps track of groups of keywords, word to groupId, and
CREATE TABLE ArticleWords (
awId int NOT NULL AUTO_INCREMENT,
articleId int NOT NULL,
word varchar(255) NOT NULL,
PRIMARY KEY (awId)
);
which keeps track of the key words within an article.
I am attempting to build a single query which can take the groups of words, and return for each group all articles which contain AT LEAST all of those words.
I realize if I look for one group at a time in a single query this is very simple, however I can't seem to figure out how to make a single query result in the set of all matching subsets.
For example imagine that the two tables have the following data:
WordGroups
groupId | word
-----------------
1 | B
1 | A
2 | C
2 | E
3 | F
ArticleWords
articleId | word
-----------------
1 | A
1 | C
1 | B
2 | C
3 | A
3 | B
3 | F
4 | C
4 | E
4 | F
The resulting query would return:
groupId | articleId
1 | 1
1 | 3
2 | 4
3 | 3
3 | 4
Since those articles contain at least all the words from those groups.
I've attempted an intersection of the two tables using an inner join but that matches incomplete groups of words resulting in the row:
groupId | articleId
2 | 2
Showing up in the result all because Article 2 contains the word "C". I'm open to ideas as I've dabbled in less serious MySQL but this has been eluding me all week.
Any help is much appreciated. I'm at the point where I'm wondering if I'm trying to make SQL do something it isn't meant to do. I have a very long query which works for WordGroups up to 6 words but it is very exact and not scalable, this query would need to work for any size WordGroup to be feasible.
Thank you for reading!
Here is one method, uses group_concat() for the comparison:
select wg.groupId, aw.articleId
from articlewords aw join
wordgroups wg
on wg.word = aw.word join
(select wg.groupId, group_concat(wg.word order by word) as words
from word_groups wg
group by wg.groupId
) wgw
on wgw.groupId = wg.groupid
group by aw.articleid, wgw.words
having group_concat(aw.word order by aw.word) = wgw.words;
Here is a SQL Fiddle.
I'm running with some troubles on a query. I'm trying to retrieve some data of a big database where 3 tables are involved.
These tables contain data about adds where, in a backend website, the administrator can manage which local adds he wants to be displayed, position and etc... These are organized in 3 tables, 1 of them, contains all the data that are relevant to adds info (Name, date of avaliability, date of expiration, etc...). Then, there's another 2 tables which contain some extra info, but just about views, or clicks.
So I have only 15 adds, that have multiple clicks and multiple views.
Each click and view table, register a new row for every click. So, when a click is registered, it will add a new row where addid_views is a register(click), and addid is addid from adds_table. So for instance, add (1) will have 2 views and 2 clicks while add (2) will have 1 view and 1 click.
My idea is to get for each add, how many clicks and views had in total.
I have 3 tables like these:
adds_table adds_clicks_table adds_views_table
+-------+-----------+ +-------------+------+ +-------------+------+
| addid | name | | addid_click |addid | | addid_views |addid |
+-------+-----------+ +-------------+------+ +-------------+------+
| 1 | add_name1 | | 1 | 1 | | 1 | 1 |
+-------+-----------+ +-------------+------+ +-------------+------+
| 2 | add_name2 | | 2 | 2 | | 2 | 1 |
+-------+-----------+
| 3 | add_name3 | | 3 | 1 | | 3 | 2 |
+-------+-----------+ +-------------+------+ +-------------+------+
CREATE TABLE `bwm_adds` (
`addid` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(100) NOT NULL,
...
PRIMARY KEY (`addid`)
) ENGINE=InnoDB AUTO_INCREMENT=16 DEFAULT CHARSET=utf8
CREATE TABLE `bwm_adds_clicks` (
`add_clickid` int(19) NOT NULL AUTO_INCREMENT,
`addid` int(11) NOT NULL,
...
PRIMARY KEY (`add_clickid`)
) ENGINE=InnoDB AUTO_INCREMENT=3374 DEFAULT CHARSET=utf8
CREATE TABLE `bwm_adds_views` (
`add_viewsid` int(19) NOT NULL AUTO_INCREMENT,
`addid` int(11) NOT NULL,
...
PRIMARY KEY (`add_viewsid`)
) ENGINE=InnoDB AUTO_INCREMENT=2078738 DEFAULT CHARSET=utf8
The result would be a single table where I retrieved, per each add (addid), how many clicks and how many views it had.
I need to get all a query where I get something like this:
+-------+---------+-----------+
| addid | clicks | views |
+-------+---------+-----------+
| 1 | 123123 | 235457568 |
+-------+---------+-----------+
| 2 | 5124123 | 435345234 |
+-------+---------+-----------+
| 3 | 123541 | 453563623 |
+-------+---------+-----------+
I tried to execute a query but it get's stuck and loading for undefined time... I 'm pretty sure that my query is failing cause if I remove one of the counts, displays some data very fast.
SELECT a.addid, COUNT(ac.addid_clicks) as 'clicks', COUNT(av.addid_views) as 'views'
FROM `adds_table` a
LEFT JOIN `adds_clicks_table` ac ON a.addid = ac.addid_click
LEFT JOIN `adds_views_table` av ON ac.addid_click = av.addid_views
GROUP BY a.addid
Mysql gets loading all the time, any idea to help know what I'm missing?
By the way, I found this post where treats almost the same problem I have, you can see I have the query very similar to the first answer, but I get the Loading message all the time. No errors, just Loading.
Edit: I missplaced the numbers and got confused. Now the tables are fixed and I added some explanation about it.
Edit2: I updated the post with SHOW CREATE TABLES DEFINITIONS.
Edit3: Is there any way to optimise this query? It seems it retrieves the result I want but the mysql database cancels the query because it gets more than 30 seconds to execute.
SELECT a.addid,
(SELECT COUNT(addid) FROM add_clicks where addid = a.addid) as clicks,
(SELECT COUNT(addid) FROM add_views where addid = a.addid) as views
FROM adds a ORDER BY a.addid;
If those are really your tables (one column, plus an auto_inc), then there is no meaningful information justifying having 3 tables instead of 1:
CREATE TABLE `bwm_adds` (
`addid` int(11) NOT NULL AUTO_INCREMENT,
`name` varchar(100) NOT NULL,
clicks INT UNSIGNED NOT NULL,
views INT UNSIGNED NOT NULL,
PRIMARY KEY (`addid`)
) ENGINE=InnoDB AUTO_INCREMENT=16 DEFAULT CHARSET=utf8
and then UPDATE ... SET views = views + 1 (etc) rather than inserting into the other tables.
If you have an old version,
SELECT a.addid,
( SELECT COUNT(addid_clicks)
FROM `adds_clicks_table`
WHERE addid = a.addid
) AS 'clicks',
( SELECT COUNT(addid_clicks)
FROM `adds_views_table`
WHERE addid = a.addid
) AS 'views'
FROM adds_table AS a
For 5.6 and later, this might be faster:
SELECT a.addid, c.clicks, v.views
FROM `adds_table` a
LEFT JOIN ( SELECT addid, COUNT(addid_clicks) FROM addid_clicks ) AS c USING(addid)
LEFT JOIN ( SELECT addid, COUNT(addid_views) FROM addid_views ) AS v USING(addid)
If you get NULLs but prefer 0s, then wrap the value in IFNULL(..., 0).
If you need to discuss further, please provide SHOW CREATE TABLE and EXPLAIN SELECT ...
I ended with a solution to my problem. The table I was trying to reach was too big cause of the bad engineered database, where in adds_views_table, for each view, a new row would be added. Ending with almost 3 millions of rows and with a table that weights almost the 35% of the entire database (326MB).
When phpmyadmin tried to execute a query, loaded for ever and never showed a result because a timeout limit applied to mysql. Changing this value would help but wasn't viable to retrieve that data and display it on a website (that implies the website or data wouldn't load until the query its executed).
That problem was fixed thanks to creating an index of addid in adds_table. Also, the query it's faster if subquery's are used for some reason. The query ended like this:
SELECT a.addid,
(SELECT COUNT(addid) FROM adds_clicks_table WHERE addid = a.addid) AS 'clicks',(SELECT COUNT(addid) FROM adds_views_table WHERE addid = a.addid) AS 'views'
FROM adds_table a
ORDER BY a.addid;
Thanks to #Rick James who posted a similar query and I ended modifying it to get the data I needed
forgive my horrible english
I am generating a mySQL query from PHP.
Part of the query re-orders a table based on some variables (which do not include the primary key).
The code doesn't produce errors, however the table is not sorted.
I echo'd out the SQL code, and it looks correct, I tried running it directly in phpMyAdmin, and it runs also without error, but the table is still not sorted as requested.
alter table anavar order by dset_name, var_id;
I am pretty sure that this has to do with the fact that I have a primary key variable (UID) which is not present in the sort.
Both prior and post running the query the table remains ordered by UID. Deleting UID and re-running the query results in a correctly sorted table, but this seems like an overkill solution.
Any suggestions?
create table t2
( id int auto_increment primary key,
someInt int not null,
thing varchar(100) not null,
theWhen datetime not null,
key(theWhen) -- creates an index on theWhen
);
-- my table now has 2 indexes on it
-- see it by running `show indexes from t2`
-- truncate table t2;
insert t2(someInt,thing,theWhen) values
(17,'chopstick','2016-05-08 13:00:00'),
(14,'alligator','2016-05-01'),
(11,'snail','2016-07-08 19:00:00');
select * from t2; -- returns in physical order (the primary key `id`)
select * from t2 order by thing; -- returns via thing, which has no index anyway
select * from t2 order by theWhen,thing; -- partial index use
note that indexes aren't even used until you have a significant number of rows in a db anyway
Edit (new data comes in)
insert t2 (someInt,thing,theWhen) values (777,'apple',now());
select t2.id,t2.thing,t2.theWhen,#rnk:=#rnk+1 as rank
from t2
cross join (select #rnk:=0) xParams
order by thing;
+----+-----------+---------------------+------+
| id | thing | theWhen | rank |
+----+-----------+---------------------+------+
| 2 | alligator | 2016-05-01 00:00:00 | 1 |
| 4 | apple | 2016-09-04 15:04:50 | 2 |
| 1 | chopstick | 2016-05-08 13:00:00 | 3 |
| 3 | snail | 2016-07-08 19:00:00 | 4 |
+----+-----------+---------------------+------+
Focus on the fact that you can maintain your secondary indices and generate a rank on the fly whenever you want.
I have a table like below:
ID|Prototype_A|Prototype_B|Prototype_C|Prototype_D|
---------------------------------------------------
1 |Fast381A |Blue4812 | Green7181 | White4812 |
---------------------------------------------------
2 |Slow841C |Orange8312 | null | null |
---------------------------------------------------
3 |Plane281K | null | null | null |
---------------------------------------------------
I need my query to return all non null prototypes for that ID.
so for example:
1 : Fast381A,Blue4812,Green7181,White4812
2 : Slow841C,Orange8312
3 : Plane281K
Is there a way to wildcard select all columns like select(Prototype_*) or should I setup my table in a different format?
For example I've been taught this type of structure is bad practice:
ID|Prototypes|
---------------------------------------------------
1 |Fast381A,Blue4812,Green7181,White4812
---------------------------------------------------
2 |Slow841C,Orange8312
---------------------------------------------------
3 |Plane281K
---------------------------------------------------
A SQL query returns a fixed set of columns. If you want to combine the non-NULL values into a single column, I would recommend concat_ws():
select id,
concat_ws(',', Prototype_A, Prototype_B, Prototype_C, Prototype_D)
from t;
This ignores the NULL values. The query returns two columns, one is a list of prototypes.
And, the answer to your question is "Yes". You should consider changing your data structure. Having multiple columns storing the same thing, with just an index identifying them usually means that you want a separate table, with one row per id and per prototype.
EDIT:
You want a table like this:
create table ModelPrototypes (
ModelProtypeId int primary key auto_increment,
ModelId int not null,
ProtoTypeChar char(1),
Prototype varchar(255)
);
Then you would populate it with values like:
1 A Fast381A
1 B Blue4812
1 C Green7181
1 D White4812
I'm not sure if PrototypeChar is really needed, but the information is in your table.
There's no way to wildcard select columns.
What you could do:
Setup your table as
ID, Prototype_type, Prototype_name
Then use GROUP_CONCAT:
SELECT id, GROUP_CONCAT(Prototype_name SEPARATOR ',')
FROM table GROUP BY Prototype_name
"Should I setup my table in a different format?"
Yes. Your table might look as follows:
ID Prototype_Code Prototype
------------------------------
1 A Fast381A
1 B Blue4812
1 C Green7181
1 D White4812
2 A Slow841C
2 B Orange8312
3 A Plane281K