Bulk apply alias to table columns in MYSQL - mysql

I'm working with a 3rd party MYSQL database over which I have no control except I can read from it. It contains 51 tables with identical column structure but slightly different names. They hold daily summaries for a different data source. Example Table:
CREATE TABLE `archive_day_?????` (
`dateTime` int(11) NOT NULL,
`min` double DEFAULT NULL,
`mintime` int(11) DEFAULT NULL,
`max` double DEFAULT NULL,
`maxtime` int(11) DEFAULT NULL,
`sum` double DEFAULT NULL,
`count` int(11) DEFAULT NULL,
`wsum` double DEFAULT NULL,
`sumtime` int(11) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
where ????? changes to indicate the type of data held.
The dateTime field is mirrored across all tables being midnight of every day since the system has been running.
I want to produce a single data set across all tables using an inner join on the dateTime. But to avoid writing
SELECT ad1.maxtime as ad1_maxtime, ad2.maxtime as ad2_maxtime...
51 times for 9 fields is there a way I can bulk create aliases e.g
ad1.* as ad_*, ad2.* as ad_* and so on.
I have looked at Create Aliases In Bulk? but this doesn't seem to work for MySQL. Ultimatly the data is being used by a Django ORM.
EDIT: Unfortunately Union doesn't uniquely identify the fields or group them together e.g.
SELECT * FROM `archive_day_ET` UNION ALL SELECT * FROM `archive_day_inTemp`
results in:

To generate a string with all the field names from those tables, you could query information_schema.columns
For example:
SELECT
GROUP_CONCAT(CONCAT(TABLE_NAME,'.`',column_name,'` AS `',column_name,'_',replace(TABLE_NAME,'archive_day_',''),'`') SEPARATOR ',\r\n')
FROM information_schema.columns
WHERE TABLE_NAME like 'archive_day_%'
A test on db<>fiddle here
And to generate the JOIN's then you could use information_schema.tables
For example:
SELECT CONCAT('FROM (\r\n ',GROUP_CONCAT(CONCAT('SELECT `dateTime` FROM ',TABLE_NAME) SEPARATOR '\r\n UNION\r\n '),'\r\n) AS dt \r\nLEFT JOIN ',
GROUP_CONCAT(CONCAT(TABLE_NAME,' ON ',
TABLE_NAME,'.`dateTime` = dt.`dateTime`') SEPARATOR '\r\nLEFT JOIN ')) as SqlJoins
FROM information_schema.tables
WHERE TABLE_NAME like 'archive_day_%'
A test on db<>fiddle here
For the 2 example tables they would generate
archive_day_ET.`dateTime` AS `dateTime_ET`,
archive_day_ET.`min` AS `min_ET`,
archive_day_ET.`mintime` AS `mintime_ET`,
archive_day_ET.`max` AS `max_ET`,
archive_day_ET.`maxtime` AS `maxtime_ET`,
archive_day_ET.`sum` AS `sum_ET`,
archive_day_ET.`count` AS `count_ET`,
archive_day_ET.`wsum` AS `wsum_ET`,
archive_day_ET.`sumtime` AS `sumtime_ET`,
archive_day_inTemp.`dateTime` AS `dateTime_inTemp`,
archive_day_inTemp.`min` AS `min_inTemp`,
archive_day_inTemp.`mintime` AS `mintime_inTemp`,
archive_day_inTemp.`max` AS `max_inTemp`,
archive_day_inTemp.`maxtime` AS `maxtime_inTemp`,
archive_day_inTemp.`sum` AS `sum_inTemp`,
archive_day_inTemp.`count` AS `count_inTemp`,
archive_day_inTemp.`wsum` AS `wsum_inTemp`,
archive_day_inTemp.`sumtime` AS `sumtime_inTemp`
And
FROM (
SELECT `dateTime` FROM archive_day_ET
UNION
SELECT `dateTime` FROM archive_day_inTemp
) AS dt
LEFT JOIN archive_day_ET ON archive_day_ET.`dateTime` = dt.`dateTime`
LEFT JOIN archive_day_inTemp ON archive_day_inTemp.`dateTime` = dt.`dateTime`

Related

Woes using UPDATE with a CTE in MySQL (MariaDB)

I'm going crazy trying to get UPDATE to work with a CTE in MySQL.
Here's a simplified schema of sa_general_journal:
CREATE TABLE `sa_general_journal` (
`ID` int(10) unsigned NOT NULL AUTO_INCREMENT,
`Transaction_ID` int(10) unsigned DEFAULT NULL COMMENT 'NULL if not split, same as ID for split records',
`Date` timestamp NOT NULL DEFAULT current_timestamp(),
…
`Statement_s` int(10) unsigned DEFAULT NULL,
…
`Name` varchar(255) DEFAULT NULL,
…
PRIMARY KEY (`ID`),
…
) ENGINE=InnoDB AUTO_INCREMENT=25929 DEFAULT CHARSET=utf8;
Some records are "split," for example, a credit card statement amount might have a sales tax amount that is split out. In such cases, both parts of the split record have the same ID in the Transaction_ID field.
When records are imported in bulk, they can't refer to last_insert_ID in order to fill in the Transaction_ID field, thus the need to go clean these up afterward.
This was my first, naive attempt, which said I had an error near UPDATE. Well duh.
WITH cte AS (
SELECT
ID,
MIN(ID) OVER(PARTITION BY `Date`, `Name`, Statement_s) AS Trans,
Transaction_ID
FROM sa_general_journal
WHERE Transaction_ID = 0)
UPDATE cte
SET Transaction_ID = Trans
The CTE itself seems to work, as I can follow it with SELECT * FROM cte and get what I expected.
So I started searching StackOverflow, and discovered that CTEs are not updatable, but that you need to join them to what you want to update. "No problem!" I think, as I code this up:
WITH cte AS (
SELECT
ID,
MIN(ID) OVER(PARTITION BY `Date`, `Name`, Statement_s) AS Trans,
Transaction_ID
FROM sa_general_journal
WHERE Transaction_ID = 0)
UPDATE sa_general_journal gj, cte
SET gj.Transaction_ID = cte.Trans
WHERE gj.ID = cte.ID
No joy. Same error message.
My understanding is that in MySQL, you don't need a column list, but I did also try this using the column list (a, b, c), with the proper columns referenced in the UPDATE statement, but it still said I had a problem near UPDATE.
There are incredibly few examples of using UPDATE with WITH on the Internet! I found one, from Percona, which I used to create my attempt above, and then found another very similar example from MySQL itself.
Thanks in advance for any help offered!
CTE is a part of subquery definition, not a part of the whole query. The query must be specified after CTE. CTE cannot be used itself. So
UPDATE sa_general_journal gj
JOIN (WITH cte AS ( SELECT
ID,
MIN(ID) OVER(PARTITION BY `Date`, `Name`, Statement_s) AS Trans,
Transaction_ID
FROM sa_general_journal
WHERE Transaction_ID = 0)
SELECT * FROM cte) subquery ON gj.ID = subquery.ID
SET gj.Transaction_ID = subquery.Trans
CTEs work with UPDATE in MySQL 8, but not MariaDB 10.x.

Using SQL to select records with a single "true" bit field from several bit fields

If I have a table like this:
CREATE TABLE `Suppression` (
`SuppressionId` int(11) NOT NULL AUTO_INCREMENT,
`Address` varchar(255) DEFAULT NULL,
`BooleanOne` bit(1) NOT NULL DEFAULT '0',
`BooleanTwo` bit(1) NOT NULL DEFAULT '0',
`BooleanThree` bit(1) NOT NULL DEFAULT '0',
PRIMARY KEY (`SuppressionId`),
)
Is there a set-based way in which I can select all records which have exactly one of the three bit fields = 1 without writing out the field names?
For example given:
1 10 Pretend Street 1 1 1
2 11 Pretend Street 0 0 0
3 12 Pretend Street 1 1 0
4 13 Pretend Street 0 1 0
5 14 Pretend Street 1 0 1
6 14 Pretend Street 1 0 0
I want to return records 4 and 6.
You could "add them up":
where cast(booleanone as unsigned) + cast(booleantwo as unsigned) + cast(booleanthree as unsigned) = 1
Or, use tuples:
where ( (booleanone, booleantwo, booleanthree) ) in ( (0b1, 0b0, 0b0), (0b0, 0b1, 0b0), (0b0, 0b0, 0b1) )
I'm not sure what you mean by "set-based".
If your number of booleans can vary over time and you don't want to update your code, I suggest you make them lines and not columns.
For example:
CREATE TABLE `Suppression` (
`SuppressionId` int(11) NOT NULL AUTO_INCREMENT,
`Address` varchar(255) DEFAULT NULL,
`BooleanId` int(11) NOT NULL,
`BooleanValue` bit(1) NOT NULL DEFAULT '0',
PRIMARY KEY (`SuppressionId`,`BooleanId`),
)
So with 1 query and a 'group by' you can check all values of your booleans, however numerous they are. Of course, this makes your tables bigger.
EDIT: Just came out with another idea: why don't you have a checksum column added, whose value would be the sum of all your bits? So you would update it at every write into your table, and just check this one in your select
If you
must use this denormalized way of representing these flags, and you
must be able to add new flag columns to your table in production, and you
cannot rewrite your queries by hand when you add columns,
then you must figure out how to write a program to write your queries.
You can use this query to retrieve a result set of boolean-valued columns, then you can use that result set in a program to write a query involving all those columns.
SELECT COLUMN_NAME
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME = 'Suppression'
AND COLUMN_NAME LIKE 'Boolean%'
AND DATA_TYPE = 'bit'
AND NUMERIC_PRECISION=1
The approach you have proposed here will work exponentially more poorly as you add columns, unfortunately. Any time a software engineer says "exponential" it's time to run away screaming. Seriously.
A much more scalable approach is to build a one-to-many relationship between your Suppression rows and your flags. Add this table.
CREATE TABLE SuppressionFlags (
SuppressionId int(11) NOT NULL,
FlagName varchar(31) NOT NULL,
Value bit(1) NOT NULL DEFAULT '0',
PRIMARY KEY (SuppressionID, FlagName)
)
Then, when you want to insert a row with some flag variables, do this sequence of queries.
INSERT INTO Suppression (Address) VALUES ('some address');
SET #SuppressionId := LAST_INSERT_ID();
INSERT INTO SuppressionFlags (SuppressionId, FlagName, Value)
VALUES (#SuppressionId, 'BooleanOne', 1);
INSERT INTO SuppressionFlags (SuppressionId, FlagName, Value)
VALUES (#SuppressionId, 'BooleanTwo', 0);
INSERT INTO SuppressionFlags (SuppressionId, FlagName, Value)
VALUES (#SuppressionId, 'BooleanThree', 0);
This gives you one Suppression row with three flags set in the SuppressionFlags table. Note the use of #SuppressionId to set the Id values in the second table.
Then to find all rows with just one flag set, do this.
SELECT Suppression.SuppressionId, Suppression.Address
FROM Suppression
JOIN SuppressionFlags ON Suppression.SuppressionId = SuppressionFlags.SuppressionId
GROUP BY Suppression.SuppressionId, Suppression.Address
HAVING SUM(SuppressionFlags.Value) = 1
It gets a little trickier if you want more elaborate combinations. For example, if you want all rows with BooleanOne and either BooleanTwo or BooleanThree set, you need to do something like this.
SELECT S.SuppressionId, S.Address
FROM Suppression S
JOIN SuppressionFlags A ON S.SuppressionId=A.SuppressionId AND A.FlagName='BooleanOne'
JOIN SuppressionFlags B ON S.SuppressionId=B.SuppressionId AND B.FlagName='BooleanTwo'
JOIN SuppressionFlags C ON S.SuppressionId=C.SuppressionId AND C.FlagName='BooleanThree'
WHERE A.Value = 1 AND (B.Value = 1 OR C.Value = 1)
This common database pattern is called the attribute / value pattern. Because SQL doesn't easily let you use variables for column names (it doesn't really have reflection) this kind of way of naming your attributes is your best path to extensibility.
It's a little more SQL. But you can add as many new flags as you need, in production, without rewriting queries or getting a combinatorial explosion of flag-matching. And SQL is built to handle this kind of query.

MySQL - get date column from a table

I have a MySQL db with a MappingTable which consists of two columns. First column is a date column and another is ID - Autoincrement int column. I created this table for mapping dates and the ID's. When I query the date column with dates to retrieve the ID, no rows are getting selected. Any reason?
I tried
date_format in the SELECT query
str_to_date while checking in the WHERE clause
Compared like current_date > "2016-07-12" AND current_date <= "2016-07-12"
IfI compare LIKE "2016-07-1%" I'm getting matching rows but if I select "2016-07-12%" though there are matching rows, it is giving 0 rows.
I defined my column as DATE only.
Anything I'm missing here?
CREATE TABLE `mapping_table` (
`Current_date` date DEFAULT NULL,
`id` int(11) NOT NULL AUTO_INCREMENT, PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=26 DEFAULT CHARSET=utf8;
My question is, I want to select something like this.
select id from mapping_table where current_date="2016-07-12";
I tried with all approaches as mentioned above, but no rows are not retrieving.
use back tick on columns and table names so it wont be read/parse as keyword.
select `id` from `mapping_table` where `current_date` = "2016-07-12";
In the sample you provided you should use a date_format
select id from mapping_table where current_date= DATE_FORMAT("2016-07-12",'%Y-%d-%m') ;
or use a range
select id from mapping_table where current_date
BETWEEN DATE_FORMAT("2016-07-12",'%Y-%d-%m')
and DATE_FORMAT("2016-07-10",'%Y-%d-%m')

A procedure or a loop

I've a table named messages where users of my local hub store their messages(kind of like a web-forums). Currently, a majority of users are participating and I get nearly 30 to 50 new entries to my table everyday.
Since this has been going on for past few years, we've got nearly 100,000 rows of data in table. The table structure is kind of like this. Where fid is the PRIMARY and ip and id(nickname) are just INDEX.
I was using this kind of query uptil now; and then iterating the resultset in luasql as shown in this link. This, according to me, consumes a lot of time and space(in buffers).
`msg` VARCHAR(280) NOT NULL,
`id` VARCHAR(30) NOT NULL,
`ctg` VARCHAR(10) NOT NULL,
`date` DATE NOT NULL COMMENT 'date_format( %m/%d/%y )',
`time` TIME NOT NULL COMMENT 'date_format( %H:%i:%s )',
`fid` BIGINT(20) NOT NULL AUTO_INCREMENT,
`ip` CHAR(39) NOT NULL DEFAULT '127.0.0.1'
My problem is that now-a-days, we've switched to new API of PtokaX and the number of requests to read and write have increased dramatically. Since, I recently read about MySQL procedures, I was thinking if these procedures are a faster or safer way of dealing with this situation.
SELECT *
FROM ( SELECT *
FROM `messages`
ORDER BY `fid` DESC
LIMIT 50 ) AS `temp`
ORDER BY `fid` ASC;
P.S.
We get around one request to read one message every 7 to 10 seconds on average. On weekends, it rises to around one every 3 seconds.
Please let me know if anything more is required.
TO SUM UP
Is their a way that I can call a stored procedure and get the final result in a smaller time. Current query(and method) takes it nearly 3 seconds to fetch and organize the data.
Few things regarding your query:
SELECT *
FROM ( SELECT *
FROM `messages`
ORDER BY `fid` DESC
LIMIT 50 ) AS `temp`
ORDER BY `fid` ASC;
Never SELECT * (all); always specify a column list (what you need)
Subqueries typically cost more (for sorting & storage)
If you are trying to fetch the bottom '50', trying using a BETWEEN clause instead
You can always see what you're query is doing by using EXPLAIN. I would try the following query:
SELECT `msg`, `id`, `ctg`, `date`, `time`, `fid`, `ip` FROM `messages`
WHERE `fid` > (SELECT MAX(`fid`)-50 FROM `messages`)
ORDER BY `fid`

mysql group_concat one table to another

i would like to have a query that will solve my problem in native sql.
i have a table named "synonym" which holds words and the words' synonyms.
id, word, synonym
1, abandon, forsaken
2, abandon, desolate
...
As you can see words are repeated in this table lots of times and this makes the table unnecessarily big. i would like to have a table named "words" which doesn't have duplicate words like:
id, word, synonyms
1, abandon, 234|90
...
note: "234" and "90" here are the id's of forsaken and desolate in newly created words table.
so i already created a new "words" table with unique words from word field at synonym table. what i need is an sql query that will look at the synonym table for each word's synonyms then find their id's from words table and update the "synonyms" field with vertical line seperated ids. then i will just drop the synonym table.
just like:
UPDATE words SET synonyms= ( vertical line seperated id's (id's from words table) of the words at the synonyms at synonym table )
i know i must use group_concat but i couldn't achieved this.
hope this is clear enough. thanks for the help!
Your proposed schema is plain horrible.
Why not use a many-to-many relationship ?
Table words
id word
1 abandon
234 forsaken
Table synonyms
wid sid
1 234
1 90
You can avoid using update and do it using the queries below:
TRUNCATE TABLE words;
INSERT INTO words
SELECT (#rowNum := #rowNum+1),
a.word,
SUBSTRING(REPLACE(a.syns, a.id + '|', ''), 2) syns
FROM (
SELECT a.*,group_concat(id SEPARATOR '|') syns
FROM synonyms a
GROUP BY word
) a,
(SELECT #rowNum := 0) b
Test Script:
CREATE TABLE `ts_synonyms` (
`id` INT(11) NULL DEFAULT NULL,
`word` VARCHAR(20) NULL DEFAULT NULL,
`synonym` VARCHAR(2000) NULL DEFAULT NULL
);
CREATE TABLE `ts_words` (
`id` INT(11) NULL DEFAULT NULL,
`word` VARCHAR(20) NULL DEFAULT NULL,
`synonym` VARCHAR(2000) NULL DEFAULT NULL
);
INSERT INTO ts_synonyms
VALUES ('1','abandon','forsaken'),
('2','abandon','desolate'),
('3','test','tester'),
('4','test','tester4'),
('5','ChadName','Chad'),
('6','Charles','Chuck'),
('8','abandon','something');
INSERT INTO ts_words
SELECT (#rowNum := #rowNum+1),
a.word,
SUBSTRING(REPLACE(a.syns, a.id + '|', ''), 2) syns
FROM (
SELECT a.*,
GROUP_CONCAT(id SEPARATOR '|') syns
FROM ts_synonyms a
GROUP BY word
) a,
(SELECT #rowNum := 0) b;
SELECT * FROM ts_synonyms;
SELECT * FROM ts_words;