I have two tables one with names and telnumbers the second with calls
addressbook name(VARCHAR) number(VARCHAR)
calls date(DATE) number(VARCHAR) name(VARCHAR)
I want to update the names column in the calls table with the entries in the addressbook for the respective
UPDATE calls
SET name = ( SELECT name FROM addressbook WHERE number = calls.number )
WHERE DATE = "2020.01.01"
ORDER BY DATE
And I get Uncaught mysqli_sql_exception: Subquery returns more than 1 row but there are no doublette in the addressbook I checked it several times.
The only way your update statement can fail with
Subquery returns more than 1 row
is if there is at least one calls row whose number appears more than once in addressbook. You can find them with this query:
select number, count(*)
from addressbook
group by number
having count(*) > 1;
Let's say you have these two rows in addressbook:
name number
------ ------
fred 123
barney 123
And let's say this is the row in calls:
date name number
---------- ---- ------
2020.01.01 null 123
When you execute Stefano's update statement, the limit clause is not deterministic because there's no associated order by clause in the subquery. Nor is there any attribute common to calls and addressbook that would make it meaningful. The order by clause on the update is irrelevant. Therefore, you cannot guarantee which name will be assigned to the calls row. This is the point I was trying to make in my comment to Stefano's answer.
If the design of the system is to allow a number to be owned by multiple people over time (which they are of course), then your schema is not complete. And if that's true, then addressbook needs an effective date for the owner of the number.
If the design of the system is not to allow a number to be owned by multiple people over time, then you must delete the duplicate rows.
In either case, you need to do two things:
employ declarative referential integrity constraints so you don't run afoul again
stop updating calls: either insert (not update) the name or remove the column entirely
If I were to implement the tables of a telephony system, I would start with something like this:
create table PERSON (
PERSON_ID integer not null primary key,
NAME varchar(100) not null /*lots of other columns*/);
create table PERSON_PHONE (
PERSON_PHONE_ID integer not null primary key,
PERSON_ID integer not null,
PHONE_NUM varchar(30) not null,
CONTRACTED date not null, /*lots of other columns*/
unique (PERSON_ID, PHONE_NUM, CONTRACTED),
foreign key (PERSON_ID) references PERSON(PERSON_ID));
create table PHONE_CALL (
START_DATE date not null,
END_DATE date not null,
PERSON_PHONE_ID integer not null,
primary key (PERSON_PHONE_ID, START_DATE),
foreign key (PERSON_PHONE_ID) references PERSON_PHONE(PERSON_PHONE_ID));
It is true that sometimes, for the sake of making queries finish faster using fewer resources, people will sometimes denormalize a schema to decrease the number of join operations that would otherwise be required. Denormalization requires careful consideration.
The error is self explanatory, the sub query returns more than one row, a quick solution is:
SELECT name FROM addressbook WHERE number = calls.number LIMIT 1
if this solve the issue than the query return more than a row. If you need to returns just a row without using LIMIT 1 you should review your query adding more constraints or define a primary key for the addressbook table and continuing use your subquery as it is. This is on you.
Using IN to select a dynamic list of software names that a computer must have to be matched, and the data exists in the database for a known value, the result when using:
GROUP BY id
HAVING COUNT(DISTINCT column_name) = count_of_list_values;
returns an empty result set when it should be returning the value expected.
NOTE it works when you are searching for a single list item, this issue is when that list expands to more than one.
Plenty of Google searches, resulting in nothing useful... other than that using phpmyadmin, i have been trying to figure out why it is failing.
Isolated the area where it seems to choking and trying different things with the query to try to get it to work.
this is the section of the query that i am using.
SELECT
am_software_archive.asset_name
FROM
am_software_archive
WHERE
LOWER(am_software_archive.sw_name) IN('nodejs', 'visio 2013')
GROUP BY
am_software_archive.id
HAVING
COUNT(
DISTINCT am_software_archive.sw_name
) = 2
The data in this table exists and is valid so it should work...
The table definition
CREATE TABLE IF NOT EXISTS am_software_archive(
id BIGINT NOT NULL UNIQUE,
asset_name VARCHAR(10) NOT NULL,
sw_name VARCHAR(150) NOT NULL,
sw_developer BIGINT NOT NULL,
sw_key VARCHAR(50) DEFAULT NULL,
sw_osver VARCHAR(15) DEFAULT NULL,
CONSTRAINT PK_software PRIMARY KEY(id, asset_name),
INDEX idx_sw_name_asset(asset_name,sw_name),
INDEX idx_sw_key_asset_name(asset_name,sw_key),
INDEX idx_sw_name_sw_key(sw_name,sw_key),
INDEX idx_osver_sw_name(sw_name,sw_osver),
INDEX idx_osver_asset_name(asset_name,sw_osver)
)
In my database the expected result would be:
"ABX50269"
Actual results well are empty, but shouldn't be.
You group by the id, which is unique so there cannot ever be two or more (different) sw_name per id.
As asset_name is the column in the list for SELECT, I think you actually want to group by asset_name.
In mysql how can I write a query that will fetch ALL business data, and at the same time (or not if it is better another way) check if user is following that business? I have the following relationship table to determine if a user is following a business (status=1 would mean that person is following):
CREATE TABLE IF NOT EXISTS `Relationship_User_Follows_Business` (
`user_id` int(10) unsigned NOT NULL,
`business_id` int(10) unsigned NOT NULL,
`status` tinyint(3) unsigned NOT NULL DEFAULT '0' COMMENT '1=following, 0=not following'
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
ALTER TABLE `Relationship_User_Follows_Business`
ADD UNIQUE KEY `unique_user_business_id` (`user_id`,`business_id`);
Assume business table just holds data on different businesses like name, phone number, etc. I would want to return all of the business data in my query (Business.*). I want to append the status (0 or 1) to the end of each business row to determine if the user is following that business. I have tried the following query but it does not work because it is narrowing the results to only show a business if there is a relationship row. I wish to show ALL businesses regardless if a relationship row exists or not because I only create the relationship row if a user follows:
SELECT Business.*, Relationship_User_Follows_Business.status FROM Business, Relationship_User_Follows_Business WHERE 104=Relationship_User_Follows_Business.user_id AND Business.id=Relationship_User_Follows_Business.business_id
Note that I am using 104 as a test user id. The user id would normally be dependent on user, not a static 104.
You are looking for a LEFT JOIN and not an INNER JOIN which keeps all the records from the master table and all the matching rows from the details table . Also, avoid using implicit join syntax(comma separated) and use the proper syntax of a join :
SELECT Business.*, Relationship_User_Follows_Business.status
FROM Business
LEFT JOIN Relationship_User_Follows_Business
ON Business.id = Relationship_User_Follows_Business.business_id
AND Relationship_User_Follows_Business.user_id = 104
The previous table this data was stored in approached 3-4gb, but the data wasn't compressed before/after storage. I'm not a DBA so I'm a little out of my depth with a good strategy.
The table is to log changes to a particular model in my application (user profiles), but with one tricky requirement: we should be able to fetch the state of a profile at any given date.
Data (single table):
id, username, email, first_name, last_name, website, avatar_url, address, city, zip, phone
The only two requirements:
be able to fetch a list of changes for a given model
be able to fetch state of model on a given date
Previously, all of the profile data was stored for a single change, even if only one column was changed. But to get a 'snapshot' for a particular date was easy enough.
My first couple of solutions in optimising the data structure:
(1) only store changed columns. This would drastically reduce data stored, but would make it quite complicated to get a snapshot of data. I'd have to merge all changes up to a given date (could be thousands), then apply that to a model. But that model couldn't be a fresh model (only changed data is stored). To do this, I'd have to first copy over all data from current profiles table, then to get snapshot apply changes to those base models.
(2) store whole of data, but convert to a compressed format like gzip or binary or whatnot. This would remove ability to query the data other than to obtain changes. I couldn't, for example, fetch all changes where email = ''. I would essentially have a single column with converted data, storing the whole of the profile.
Then, I would want to use relevant MySQL table options, like ARCHIVE to further reduce space.
So my question is, are there any other options which you feel are a better approach than 1/2 above, and, if not, which would be better?
First of all, I wouldn't worry at all about a 3GB table (unless it grew to this size in a very short period of time). MySQL can take it. Space shouldn't be a concern, keep in mind that a 500 GB hard disk costs about 4 man-hours (in my country).
That being said, in order to lower your storage requirements, create one table for each field of the table you want to monitor. Assuming a profile table like this:
CREATE TABLE profile (
profile_id INT PRIMARY KEY,
username VARCHAR(50),
email VARCHAR(50) -- and so on
);
... create two history tables:
CREATE TABLE profile_history_username (
profile_id INT NOT NULL,
username VARCHAR(50) NOT NULL, -- same type as profile.username
changedAt DATETIME NOT NULL,
PRIMARY KEY (profile_id, changedAt),
CONSTRAINT profile_id_username_fk
FOREIGN KEY profile_id_fkx (profile_id)
REFERENCES profile(profile_id)
);
CREATE TABLE profile_history_email (
profile_id INT NOT NULL,
email VARCHAR(50) NOT NULL, -- same type as profile.email
changedAt DATETIME NOT NULL,
PRIMARY KEY (profile_id, changedAt),
CONSTRAINT profile_id_fk
FOREIGN KEY profile_id_email_fkx (profile_id)
REFERENCES profile(profile_id)
);
Everytime you change one or more fields in profile, log the change in each relevant history table:
START TRANSACTION;
-- lock all tables
SELECT #now := NOW()
FROM profile
JOIN profile_history_email USING (profile_id)
WHERE profile_id = [a profile_id]
FOR UPDATE;
-- update main table, log change
UPDATE profile SET email = [new email] WHERE profile_id = [a profile_id];
INSERT INTO profile_history_email VALUES ([a profile_id], [new email], #now);
COMMIT;
You may also want to set appropriate AFTER triggers on profile so as to populate the history tables automatically.
Retrieving history information should be straightforward. In order to get the state of a profile at a given point in time, use this query:
SELECT
(
SELECT username FROM profile_history_username
WHERE profile_id = [a profile_id] AND changedAt = (
SELECT MAX(changedAt) FROM profile_history_username
WHERE profile_id = [a profile_id] AND changedAt <= [snapshot date]
)
) AS username,
(
SELECT email FROM profile_history_email
WHERE profile_id = [a profile_id] AND changedAt = (
SELECT MAX(changedAt) FROM profile_history_email
WHERE profile_id = [a profile_id] AND changedAt <= [snapshot date]
)
) AS email;
You can't compress the data without having to uncompress it in order to search it - which is going to severely damage the performance. If the data really is changing that often (i.e. more than an average of 20 times per record) then it would be more efficient to for storage and retrieval to structure it as a series of changes:
Consider:
CREATE TABLE profile (
id INT NOT NULL autoincrement,
PRIMARY KEY (id);
);
CREATE TABLE profile_data (
profile_id INT NOT NULL,
attr ENUM('username', 'email', 'first_name'
, 'last_name', 'website', 'avatar_url'
, 'address', 'city', 'zip', 'phone') NOT NULL,
value CARCHAR(255),
starttime DATETIME DEFAULT CURRENT_TIME,
endtime DATETIME,
PRIMARY KEY (profile_id, attr, starttime)
INDEX(profile_id),
FOREIGN KEY (profile_id) REFERENCES profile(id)
);
When you add a new value for an existing record, set an endtime in the masked record.
Then to get the value at a date $T:
SELECT p.id, attr, value
FROM profile p
INNER JOIN profile_date d
ON p.id=d.profile_id
WHERE $T>=starttime
AND $T<=IF(endtime IS NULL,$T, endtime);
Alternately just have a start time, and:
SELECT p.id, attr, value
FROM profile p
INNER JOIN profile_date d
ON p.id=d.profile_id
WHERE $T>=starttime
AND NOT EXISTS (SELECT 1
FROM prodile_data d2
WHERE d2.profile_id=d.profile_id
AND d2.attr=d.attr
AND d2.starttime>d.starttime
AND d2.starttime>$T);
(which will be even faster with the MAX concat trick).
But if the data is not changing with that frequency then keep it in the current structure.
You need a slow changing dimension:
i will do this only for e-mail and telephone so you understand (pay attention to the fact of i use two keys, 1 as unique in the table, and another that is unique to the user that it concerns. This is, the table key identifies the the record, and the user key identifies the user):
table_id, user_id, email, telephone, created_at,inactive_at,is_current
1, 1, mario#yahoo.it, 123456, 2012-01-02, , 2013-04-01, no
2, 2, erik#telecom.de, 123457, 2012-01-03, 2013-02-28, no
3, 3, vanessa#o2.de, 1234568, 2012-01-03, null, yes
4, 2, erik#telecom.de, 123459, 2012-02-28, null, yes
5, 1, super.mario#yahoo.it, 654321,2013-04-01, 2013-04-02, no
6, 1, super.mario#yahoo.it, 123456,2013-04-02, null, yes
most recent state of the database
select * from FooTable where inactive_at is null
or
select * from FooTable where is_current = 'yes'
All changes to mario (mario is user_id 1)
select * from FooTable where user_id = 1;
All changes between 1 jan 2013 and 1 of may 2013
select * from FooTable where created_at between '2013-01-01' and '2013-05-01';
and you need to compare with the old versions (with the help of a stored procedure, java or php code... you chose)
select * from FooTable where incative_at between '2013-01-01' and '2013-05-01';
if you want you can do a fancy sql statement
select f1.table_id, f1.user_id,
case when f1.email = f2.email then 'NO_CHANGE' else concat(f1.email , ' -> ', f2.email) end,
case when f1.phone = f2.phone then 'NO_CHANGE' else concat(f1.phone , ' -> ', f2.phone) end
from FooTable f1 inner join FooTable f2
on(f1.user_id = f2.user_id)
where f2.created_at in
(select max(f3.created_at) from Footable f3 where f3.user_id = f1.user_id
and f3.created_at < f1.created_at and f1.user_id=f3.user_id)
and f1.created_at between '2013-01-01' and '2013-05-01' ;
As you can see a juicy query, to compare the user_with the previews user row...
the state of the database on 2013-03-01
select * from FooTable where table_id in
(select max(table_id) from FooTable where inactive_at <= '2013-03-01' group by user_id
union
select id from FooTable where inactive_at is null group by user_id having count(table_id) =1 );
I think this is the easiest way of implement what you want... you could implement a multi-million tables relational model, but then it would be a pain in the arse to query it
Your database is not big enough, I work everyday with one even bigger. Now tell me is the money you save in a new server worthy the time you spend on a super-complex relational model?
BTW if the data changes too fast, this approach cannot be used...
BONUS: optimization:
create indexes on created_at, inactive_at, user_id and the pair
perform partition (both horizontal and vertical)
if you try and put all occurring changes in different tables and later if you require an instance on some date you join them along and display by comparing dates, for example if you want an instance at 1st of july you can run a query with condition where date is equal or less than 1st of july and order it in asc ordering limiting the count to 1. that way the joins will produce exactly the instance it was at 1st of july. in this manner you can even figure out the most frequently updated module.
also if you want to keep all the data flat try range partitioning on the basis of month that way mysql will handle it pretty easily.
Note: by date i mean storing unix timestamp of the date its pretty easier to compare.
I'll offer one more solution just for variety.
Schema
PROFILE
id INT PRIMARY KEY,
username VARCHAR(50) NOT NULL UNIQUE
PROFILE_ATTRIBUTE
id INT PRIMARY KEY,
profile_id INT NOT NULL FOREIGN KEY REFERENCES PROFILE (id),
attribute_name VARCHAR(50) NOT NULL,
attribute_value VARCHAR(255) NULL,
created_at DATETIME NOT NULL DEFAULT GETTIME(),
replaced_at DATETIME NULL
For all attributes you are tracking, simply add PROFILE_ATTRIBUTE records when they are updated, and mark the previous attribute record with the DATETIME it was replaced at.
Select Current Profile
SELECT *
FROM PROFILE p
LEFT JOIN PROFILE_ATTRIBUTE pa
ON p.id = pa.profile_id
WHERE p.username = 'username'
AND pa.replaced_at IS NULL
Select Profile At Date
SELECT *
FROM PROFILE p
LEFT JOIN PROFIILE_ATTRIBUTE pa
ON p.id = pa.profile_id
WHERE p.username = 'username'
AND pa.created_at < '2013-07-01'
AND '2013-07-01' <= IFNULL(pa.replaced_at, GETTIME())
When Updating Attributes
Insert the new attribute
Update the previous attribute's replaced_at value
It would probably be important that the created_at for a new attribute match the replaced_at for the corresponding old attribute. This would be so that there is an unbroken timeline of attribute values for a given attribute name.
Advantages
Simple two-table architecture (I personally don't like a table-per-field approach)
Can add additional attributes with no schema changes
Easily mapped into ORM systems, assuming an application lives on top of this database
Could easily see the history for a certain attribute_name over time.
Disadvantages
Integrity is not enforced. For example, the schema doesn't restrict on multiple NULL replaced_at records with the same attribute_name... perhaps this could be enforced with a two-column UNIQUE constraint
Let's say you add a new field in the future. Existing profiles would not select a value for the new field until they save a value to it. This is opposed to the value coming back as NULL if it were a column. This may or may not be an issue.
If you use this approach, be sure you have indexes on the created_at and replaced_at columns.
There may be other advantages or disadvantages. If commenters have input, I'll update this answer with more information.
Problem Description:
A tag (tags) can be associated with arbitrary objects through a junction table (tagged_as). For a specific object type (specific_object), select the union or intersection of all of the objects associated with a series of tags, order the results by a numeric column on the object and limit the results for pagination purposes.
Contrived Schema:
CREATE TABLE tags (
id INT NOT NULL AUTO_INCREMENT,
name VARCHAR(45) NOT NULL,
PRIMARY KEY (id)
);
CREATE TABLE specific_object(
id INT NOT NULL AUTO_INCREMENT,
name VARCHAR(45) NOT NULL,
vote_sum INT NOT NULL DEFAULT 0,
PRIMARY KEY (id)
);
CREATE TABLE tagged_as(
id INT NOT NULL AUTO_INCREMENT,
tag_id INT NOT NULL,
content_type_id INT NOT NULL,
object_id INT NOT NULL,
PRIMARY KEY (id)
);
For the purposes of this example, I am omitting many other columns in the specific_object table.
Table Row Counts:
tags: 12,297
tagged_as: 46,642,064
specific_object: 2,444,944
Naive MySQL Solution:
SELECT
specific_object.*
FROM
specific_object
JOIN
tagged_as
ON
specific_object.id = tagged_as.object_id
AND
tagged_as.content_type_id = <SPECIFIC_OBJECT_CONTENT_TYPE_ID>
WHERE
tagged_as.tag_id = <TAG_ONE_ID>
AND
tagged_as.tag_id = <TAG_TWO_ID>
...
ORDER BY specific_object.vote_sum DESC
LIMIT 50
The problem with this solution is that MySQL cannot utilize an index to resolve the ORDER BY clause because the "key used to fetch the rows is not the same as the one used in the ORDER BY" (http://dev.mysql.com/doc/refman/5.0/en/order-by-optimization.html). Execution time: 20+ seconds
Naive Redis Solution:
for each specific object: SET specfic_object:<ID> <ID>
for each tagged as: SADD tag:<TAG ID> specific_object:<ID>
specific_object_ids = SUNION tag:<TAG_ONE_ID> tag:<TAG_TWO_ID> ...
specific_object_ids = SINTER tag:<TAG_ONE_ID> tag:<TAG_TWO_ID> ...
SELECT * FROM specific_object WHERE id IN (<specific_object_ids>) ORDER BY vote_sum DESC
The problem with this solution is that the ORDER BY still has to been done by MySQL. Also, a tag could potentially be associated with hundreds of thousands of specific objects which is a lot of data to move around. Execution Time: 20+ seconds for larger tags
Possible Solutions I Haven't Tried Yet
Denormalize
Perhaps move the vote_sum column into the tagged_as table. Remove the need for the join to do the order by. This might have the same issue as the naive solution.
Redis Sorted Sets
for each specific object: SET specific_object:<ID> <ID>
for each specific object: SET specific_object_weight:<ID> <VOTE_SUM>
for each tagged as: SADD tag:<TAG_ID> specific_object:<ID>
SINTERSTORE result:<timestamp> <TAG_ONE_ID> <TAG_TWO_ID> ...
SORT result:<timestamp> BY specific_object_weight_* LIMIT 0 50
specific_object_ids = SMEMBERS result:<timestamp>
DEL result:<timestamp>
SELECT * FROM specific_object WHERE id IN (<specific_object_ids>)
Move all of the sorting into Redis. This add extra complexity because now you have to maintain the vote_sum values in Redis as well. Not sure if this would be fast enough.
Question:
Are either of the possible solutions viable? Are there other solutions or different technologies that would help? I am open to pretty significant changes to solve this problem.
When the problem has been performance of a DESC sort, what I've done in the past is to solve the problem is to store the value of -1*vote_sum in a separate column, and then ORDER BY that column ASC. I've been able to get MySQL to use an index to do the sort on that column.
You could either store a redundant column (both vote_sum and neg_vote_sum, or you could just store the negative value, and just multiply it by -1 when you need to return it as a positive value.
But I'm suspicious that the source of your performance issue is the sort operation. How does the performance of the statement compare, as a test, when you do an ORDER BY vote_sum ASC ?