BigQuery and NOT IN - mysql

Question

Sorry for my bad English. Hopefully you understand, what I want.
I want something like a Pivot table (hopefully it's the right word)
For example I have a table with two columns: userid and domain
UserID Domain
1 | A
1 | B
1 | C
2 | A
2 | B
3 | A
2 | C
What I want. I want a table like the following which extracts the differences row-wise
A B C
A 0 1 1
B 0 0 0
C 0 0 0
How the read the output?
For example the first row (0,1,1)
Imagine all users which visited domain A (in our case user 1, user 2 and user 3).... All of domain A visitors were on domain A (I guess that's clear). The also visited domain B? No, one user (in our case user 3) was not on domain B. So we have a 1. Now we check if all domain A visitors were on domain C! And here we have also on user which was not on domain C. User 1 and 2 were on domain C but user 3 was not on domain C but on domain A. So we have to write a 1 again....
Second row - Check which users where on Domain B.
User 1 and user 2 were on domain B. Where they also on domain A? Yes... Both... So we have to write down a 0. User 1 and user 2 were on domain B? Yes... So 0. And on domain C? Yes... Both.. So we have to write a zero again.
Third row - To check the domain C
On domain C we have the visitors 1 and 2. Both also visited domain A so we have a zero... Both visited domain B? Yes, also zero and the last entry is clear since they came from domain C.....
To keep the long story short: I want to extract all exclusive visitors of each domain compared to the other domains...
I am struggling since 2 days with left joins and case when and so on... Nothing works out.
Is there anybody out their with suggestions? Would be really helpful. And yes, I have more than 3 domains. I have around 200!

Answer 1

very very big query :) , but it's working
DROP PROCEDURE IF EXISTS dowhile;
CREATE PROCEDURE dowhile()
BEGIN
SELECT #domain_arr := CONCAT(GROUP_CONCAT(domain SEPARATOR ','),',') AS domain_arr FROM ( SELECT t1.domain FROM user_domain t1 WHERE 1 GROUP BY t1.domain ) AS tt;
DROP table IF EXISTS temp_table;
create temporary table temp_table (
domain VARCHAR(100) not NULL
);
SET #domain_arr_table= #domain_arr;
WHILE LOCATE(',', #domain_arr_table) > 0 DO
SET #domain = SUBSTRING(#domain_arr_table,1,LOCATE(',',#domain_arr_table) - 1);
SET #domain_arr_table= SUBSTRING(#domain_arr_table, LOCATE(',',#domain_arr_table) + 1);
SET #s= CONCAT('ALTER TABLE temp_table ADD COLUMN ',#domain,' TINYINT DEFAULT 0');
PREPARE stmt3 FROM #s;
EXECUTE stmt3;
END WHILE;
WHILE LOCATE(',', #domain_arr) > 0 DO
SET #domain = SUBSTRING(#domain_arr,1,LOCATE(',',#domain_arr) - 1);
SET #domain_arr= SUBSTRING(#domain_arr, LOCATE(',',#domain_arr) + 1);
SELECT #user_count := COUNT(*) FROM user_domain WHERE domain=#domain;
INSERT INTO temp_table (domain) VALUES (#domain);
SELECT #domains_should_be_1 := CONCAT(GROUP_CONCAT(domain SEPARATOR ','),',') FROM (SELECT domain FROM user_domain WHERE user_id IN (SELECT user_id FROM user_domain WHERE domain=#domain) GROUP BY domain HAVING COUNT(*) < #user_count) AS tt2;
WHILE LOCATE(',', #domains_should_be_1) > 0 DO
SET #domain_sb_1 = SUBSTRING(#domains_should_be_1,1,LOCATE(',',#domains_should_be_1) - 1);
SET #domains_should_be_1= SUBSTRING(#domains_should_be_1, LOCATE(',',#domains_should_be_1) + 1);
SET #s= CONCAT("UPDATE temp_table SET ",#domain_sb_1,"='1' WHERE domain='",#domain,"'");
SELECT #s;
PREPARE stmt3 FROM #s;
EXECUTE stmt3;
END WHILE;
END WHILE;
END;
call dowhile();
SELECT * FROM temp_table;

Answer 2

There are really two questions here
I want to extract all exclusive visitors of each domain compared to the other domains...
I want something like a Pivot table
Let me answer your questions one by one
So,
How extract all exclusive visitors of each domain compared to the other domains...
Below is for BigQuery Standard SQL and produces flattened version of your matrix
#standardSQL
WITH `project.dataset.your_table` AS (
SELECT 1 userid, 'A' domain UNION ALL
SELECT 1, 'B' UNION ALL
SELECT 1, 'C' UNION ALL
SELECT 2, 'A' UNION ALL
SELECT 2, 'B' UNION ALL
SELECT 3, 'A' UNION ALL
SELECT 2, 'C'
), temp AS (
SELECT DISTINCT userid, domain
FROM `project.dataset.your_table`
)
SELECT
a.domain domain_a,
b.domain domain_b,
COUNT(DISTINCT a.userid) - COUNTIF(a.userid = b.userid) count_of_not_in
FROM temp a
CROSS JOIN temp b
GROUP BY a.domain, b.domain
-- HAVING count_of_not_in > 0
This will result with
Row domain_a domain_b count_of_not_in
1 A A 0
2 A B 1
3 A C 1
4 B A 0
5 B B 0
6 B C 0
7 C A 0
8 C B 0
9 C C 0
I think in real life you will not have many zeroes in this data so if you want to compress that flattened version - just uncomment line with HAVING ... , so you will get "compact" version
Row domain_a domain_b count_of_not_in
1 A B 1
2 A C 1
For the sake of exercising and having fun, check out another approach below that produces exactly same result but in totally different way
#standardSQL
WITH `project.dataset.your_table` AS (
SELECT 1 userid, 'A' domain UNION ALL
SELECT 1, 'B' UNION ALL
SELECT 1, 'C' UNION ALL
SELECT 2, 'A' UNION ALL
SELECT 2, 'B' UNION ALL
SELECT 3, 'A' UNION ALL
SELECT 2, 'C'
), domains AS (
SELECT domain, ARRAY_AGG(DISTINCT userid) users
FROM `project.dataset.your_table`
GROUP BY domain
)
SELECT
a.domain domain_a, b.domain domain_b,
ARRAY_LENGTH(a.users) -
(SELECT COUNT(1)
FROM UNNEST(a.users) user_a
JOIN UNNEST(b.users) user_b
ON user_a = user_b
) count_of_not_in
FROM domains a
CROSS JOIN domains b
-- ORDER BY a.domain, b.domain
Now,
How to pivot above result, to produce actual matrix?
Ideally, pivoting should be done outside of BigQuery in whatever visualization tool you are usually using. But if for whatever reason you want to have it done within BigQuery - it is doable and there is enormous amount of questions here in SO related to this. One of the most recent that I have posted answer for is - https://stackoverflow.com/a/50300387/5221944 .
It shows how to generate/produce pivot query to achieve desired matrix
It is relatively easy and can be done either manually as two step process (step 1 - generate pivot query and step 2 - run generated query) or can be implemented using any client of your choice

Answer 3

You cannot (easily) express this as a matrix. But you can express this as a table with three columns: , , count.
with t as ( -- may not be necessary if the rows are already unique
select distinct userid, domain
from tab
)
select t1.domain as domain1, t2.domain as domain2, count(*)
from t t1 join
t t2
on t1.userid = t2.userid
group by t1.domain, t2.domain;
You cannot easily pivot the results in BigQuery into columns, unless you explicitly know the domains that you care about. You can aggregate them into columns, if you like.
For a given set of domains as columns, you can use conditional aggregation:
with t as ( -- may not be necessary if the rows are already unique
select distinct userid, domain
from tab
)
select t1.domain as domain1,
sum(case when t2.domain = 'amazon.com' then 1 else 0 end) as amazon,
sum(case when t2.domain = 'ebay.com' then 1 else 0 end) as ebay,
sum(case when t2.domain = 'yahoo.com' then 1 else 0 end) as yahoo
from t t1 join
t t2
on t1.userid = t2.userid
group by t1.domain, t2.domain;