How to check missing timestamp in MySql - mysql

In my SQL database (MySql), I want to record the price history of an asset.
I have a table with a timestamp as a primary key and price as the value. It has only two column timestamp / price
There should be one price point per second recorded.
Sometimes, there are missing price points. (When the server goes down)
Here is an example of the timestamp column.
**timestamp**
1581431400
1581431401
1581431402
1581431403
1581431405
1581431406 //missing 4 rows price points after this
1581431410
1581431411
1581431412
1581431413
1581431414
1581431415 //missing 3 rows price points after this
1581431418
1581431419
1581431420
Given two timestamps, how to run a SQL query that will fetch the timestamp ranges where the data exists without querying the entire database?
For example, I let's say the two timestamp in UNIX are 1 and 2000000000
What is the SQL query I should run to return the following ranges:
[
[1581431400,1581431406],
[1581431410,1581431415],
[1581431418,1581431420]
]

Here is my answer (Hack). You can use a query like this.
SELECT CONCAT( '[',GROUP_CONCAT('\n',
'[', res.missing_from, '],'
,'[', res.missing_to -1,']') , '\n]') AS missing
FROM (
SELECT m.ts+1 AS missing_from,
(SELECT ts FROM mytable WHERE ts > m.ts ORDER BY ts LIMIT 1 ) as missing_to
FROM mytable m
LEFT JOIN mytable mf ON m.ts+1 = mf.ts
WHERE
mf.ts IS NULL
) AS res
WHERE res.missing_to - res.missing_from > 0;
SAMPLE
mysql> SELECT * FROM mytable;
+------------+
| ts |
+------------+
| 1581431400 |
| 1581431401 |
| 1581431402 |
| 1581431403 |
| 1581431405 |
| 1581431406 |
| 1581431410 |
| 1581431411 |
| 1581431412 |
| 1581431413 |
| 1581431414 |
| 1581431415 |
| 1581431418 |
| 1581431419 |
| 1581431420 |
+------------+
15 rows in set (0.00 sec)
TEST
mysql> SELECT CONCAT( '[',GROUP_CONCAT('\n',
'[', res.missing_from, '],'
,'[', res.missing_to -1,']') , '\n]') AS missing
FROM (
SELECT m.ts+1 AS missing_from,
(SELECT ts FROM mytable WHERE ts > m.ts ORDER BY ts LIMIT 1 ) as missing_to
FROM mytable m
LEFT JOIN mytable mf ON m.ts+1 = mf.ts
WHERE
mf.ts IS NULL
) AS res
WHERE res.missing_to - res.missing_from > 0;
+-------------------------------------------------------------------------------------+
| missing |
+-------------------------------------------------------------------------------------+
| [
[1581431404],[1581431404],
[1581431407],[1581431409],
[1581431416],[1581431417]
] |
+-------------------------------------------------------------------------------------+
1 row in set (0.01 sec)

I would simply use window functions:
select min(timestamp), max(timestamp)
from (select timestamp, row_number() over (order by timestamp) as seqnum
from t
) t
group by (timestamp - seqnum);
I'm not sure what "without querying the entire database?" is supposed to mean. This reads the table -- as any such query would need to -- but does not need to query anything else in the database.
This illustrates what happens:
timestamp seqnum diff
1581431400 1 1581431399
1581431401 2 1581431399
1581431402 3 1581431399
1581431403 4 1581431399
1581431405 5 1581431400
1581431406 6 1581431400
1581431410 7 1581431403
1581431411 8 1581431403
The last column is identifying adjacent timestamps that differ by "1". That is what is aggregated in the outer query.

Related

Is it possible get data with search data between values

I am having data in table USER_TABLE as following :
+-----------------+
| USERS_RANGE |
+-----------------+
| 5-98 |
| 9854-98666620 |
| 54-986 |
| 1-20 |
| 10000-122222220 |
| 10-1222 |
+-----------------+
My requirement hear:
Ex:- if i search "7" it should display as :
+-------------+
| USERS_RANGE |
+-------------+
| 5-98 |
| 1-20 |
+-------------+
I tried
ex:- select * from USER_TABLE WHERE USERS_RANGE <= '7';
and some more mathamatical things in java
Is there any query to get data like this.
We can get the required data using subquery approach and SUBSTRING_INDEX() function.
/* select all inside subquery and filter with WHERE*/
SELECT USERS_RANGE FROM (
/* select all range and split the range between '-' */
SELECT
USERS_RANGE
, SUBSTRING_INDEX(USERS_RANGE, '-', 1) AS `from`
, SUBSTRING_INDEX(USERS_RANGE, '-', -1) AS `to`
FROM USERS_RANGE_TABLE
) A
WHERE A.from <= 7 AND A.to >= 7;
The subquery will select all USER_RANGE data, split the range and create a new column with alias from and to,
and the main query will select all inside subquery and filter the result with where by column from and to.
Hope this help answer your question.
Reference:
https://dev.mysql.com/doc/refman/8.0/en/string-functions.html#function_substring-index
https://dev.mysql.com/doc/refman/8.0/en/subqueries.html

Inserting random data from a list

These are my table columns:
ID || Date || Description || Priority
My goal is to insert random test data of 2000 rows with date ranging between (7/1/2019 - 7/1/2020) and randomize the priority from list (High, Medium, Low).
I know how to insert random numbers but I am stuck with the date and the priority fields.
If I need to write code, any pointers on how do I do it?
Just want to be clear - I have issue with randomizing and inserting from a given list
CREATE TABLE mytable (
id SERIAL PRIMARY KEY,
date DATE NOT NULL,
description TEXT,
priority ENUM('High','Medium','Low') NOT NULL
);
INSERT INTO mytable (date, priority)
SELECT '2019-07-01' + INTERVAL FLOOR(RAND()*365) DAY,
ELT(1+FLOOR(RAND()*3), 'High', 'Medium', 'Low')
FROM DUAL;
The fake table DUAL is a special keyword. You can select from it, and it always returns exactly one row. But it has no real columns with data, so you can only select expressions.
Do this INSERT a few times and you get:
mysql> select * from mytable;
+----+------------+-------------+----------+
| id | date | description | priority |
+----+------------+-------------+----------+
| 1 | 2019-10-20 | NULL | Medium |
| 2 | 2020-05-17 | NULL | High |
| 3 | 2020-06-25 | NULL | Low |
| 4 | 2020-05-06 | NULL | Medium |
| 5 | 2019-09-30 | NULL | High |
| 6 | 2019-08-06 | NULL | Low |
| 7 | 2020-02-21 | NULL | High |
| 8 | 2019-11-10 | NULL | High |
| 9 | 2019-07-30 | NULL | High |
+----+------------+-------------+----------+
Here's a trick to use the number of rows in the table itself to insert the same number of rows, basically doubling the number of rows:
INSERT INTO mytable (date, priority)
SELECT '2019-07-01' + INTERVAL FLOOR(RAND()*365) DAY,
ELT(1+FLOOR(RAND()*3), 'High', 'Medium', 'Low')
FROM mytable;
Just changing FROM DUAL to FROM mytable I change from selecting one row, to selecting the current number of rows from the table. But the values I insert are still random expressions, not the values already in those rows. So I get new rows with new random values.
Then repeat this INSERT as many times as you want to double the number of rows.
Read also about the ELT() function.
You seem to be looking for something like this. A basic random sample is:
select t.*
from t
where date >= '2019-07-01' and date < '2020-07-01'
order by random()
fetch first 2000 rows only;
Of course, the function for random() varies by database, as does the logic for limiting rows. This should get about the same distribution of priorities as in the original data.
If you want the rows to come by priority first, then use:
select t.*
from t
where date >= '2019-07-01' and date < '2020-07-01'
order by (case when priority = 'High' then 1 when priority = 'Medium' then 2 else 3 end),
random()
fetch first 2000 rows only;

MYSQL Using multiple selects in insert query is returning Column count doesn't match value count at row 1

UPDATE
Sometime, when a family is being inactivated from a system, it may contain more than 1 individual. In my case show at the sql fiddle, the family with household_id=12 has 3 individuals.
I need to insert the data of these 3 individuals as the same from indiviudal table to individual_history table and just changing the ind_action field into the following message HH has been inactivated.
Here is a sample data:
| individual_id | household_id | family_relation_id | marital_status_id | ind_lmms_id | ind_un_id | head_of_hh | ind_first_name_ar | ind_last_name_ar | ind_first_name_en | ind_last_name_en | ind_gender | dob | ind_status | ind_date_added | user_id | system_date |
|---------------|--------------|--------------------|-------------------|-------------|-----------|------------|-------------------|------------------|-------------------|------------------|------------|------------|------------|----------------------|---------|----------------------|
| 1 | 12 | 3 | 1 | 321 | (null) | no | u | x | (null) | (null) | Male | 2012-01-01 | Active | 2018-07-19T00:00:00Z | 1 | 2018-07-19T00:00:00Z |
| 2 | 12 | 1 | 2 | 123 | (null) | no | x | y | (null) | (null) | Female | 1998-03-05 | Active | 2015-03-05T00:00:00Z | 1 | 2015-03-05T00:00:00Z |
| 3 | 12 | 3 | 1 | 1234 | (null) | no | x | z | (null) | (null) | Female | 2004-04-05 | Active | 2018-04-11T00:00:00Z | 1 | 2018-04-11T00:00:00Z |
All 3 fields should be inserted to the table individual_history and ind_action is set to the note I added above.
I need to insert into a table called individual_history values of a SELECT query from table individual.
Here is the query:
INSERT INTO individual_history
(individual_id,
household_id,
family_relation_id_history,
marital_status_id_history,
ind_lmms_id_history,
ind_un_id_history,
head_of_hh_history,
ind_first_name_ar_history,
ind_last_name_ar_history,
ind_first_name_en_history,
ind_last_name_en_history,
ind_gender_history,
dob_history,
ind_status_history,
ind_action,
ind_date_changed,
user_id,
system_date)
VALUES ((SELECT i.individual_id,
i.household_id,
i.family_relation_id,
i.marital_status_id,
i.ind_lmms_id,
i.ind_un_id,
i.head_of_hh,
i.ind_first_name_ar,
i.ind_last_name_ar,
i.ind_first_name_en,
i.ind_last_name_en,
i.ind_gender,
i.dob,
i.ind_status
FROM individual i
WHERE i.household_id = :hid),
'HH Status Changed to inactive',
(SELECT i.ind_date_added,
i.user_id
FROM individual i
WHERE i.household_id = :hid),
:systemDate)
As you can see from the query, I am splitting the SELECT statement into 2 parts, as I want to insert a specific ind_action message, then I will continue by getting the other 2 fields date added and user_id.
The systemDate is the just the now() function result.
I tried to run this query using 12 as hid and I received the following error:
1136 - Column count doesn't match value count at row 1
After doing few searches, I found that I should add parenthesis for each of the values. So I changed the query to:
INSERT INTO individual_history
(individual_id,
household_id,
family_relation_id_history,
marital_status_id_history,
ind_lmms_id_history,
ind_un_id_history,
head_of_hh_history,
ind_first_name_ar_history,
ind_last_name_ar_history,
ind_first_name_en_history,
ind_last_name_en_history,
ind_gender_history,
dob_history,
ind_status_history,
ind_action,
ind_date_changed,
user_id,
system_date)
VALUES ((SELECT i.individual_id,
i.household_id,
i.family_relation_id,
i.marital_status_id,
i.ind_lmms_id,
i.ind_un_id,
i.head_of_hh,
i.ind_first_name_ar,
i.ind_last_name_ar,
i.ind_first_name_en,
i.ind_last_name_en,
i.ind_gender,
i.dob,
i.ind_status
FROM individual i
WHERE i.household_id = 12),
( 'HH Status Changed to inactive' ),
(SELECT i.ind_date_added,
i.user_id
FROM individual i
WHERE i.household_id = 12),
( NOW() ))
But still got the same error.
I tried to count the number of fields I am inserting compared to the ones I am selecting, and they are the same (18 fields).
UPDATE
I changed the query by removing the VALUES clause:
INSERT INTO individual_history
(
individual_id,
household_id,
family_relation_id_history,
marital_status_id_history,
ind_lmms_id_history,
ind_un_id_history,
head_of_hh_history,
ind_first_name_ar_history,
ind_last_name_ar_history,
ind_first_name_en_history,
ind_last_name_en_history,
ind_gender_history,
dob_history,
ind_status_history,
ind_action,
ind_date_changed,
user_id,
system_date
)
SELECT i.individual_id,
i.household_id,
i.family_relation_id,
i.marital_status_id,
i.ind_lmms_id,
i.ind_un_id,
i.head_of_hh,
i.ind_first_name_ar,
i.ind_last_name_ar,
i.ind_first_name_en,
i.ind_last_name_en,
i.ind_gender,
i.dob,
i.ind_status
FROM individual i
WHERE i.household_id=12,
'HH Status Changed to inactive',
(
SELECT i.ind_date_added,
i.user_id
FROM individual i
WHERE i.household_id=12),
now()
And I got the following error:
1064 - You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use
near '
'HH Status Changed to inactive',
' at line 10
Please note that the datatype of fields are exactly the same in both tables, and individual_history table contain an auto-increment primary key.
HERE IS AN SQL FIDDLE to check with sample data.
You don't need two SELECTs for what you're trying to do. If you want to use some specific value for ind_action, simply replace it in your select, same as you did with the now() function:
INSERT INTO targetTable (col1, col2, col3, col4, colTime)
SELECT colA, colB, 'my specific string', colD, now()
FROM sourceTable WHERE colA = 12;
Here, col3 gets the string, colTime the now().
#Marting Hennings, I am a bit too late ... but this query should work:
INSERT INTO individual_history
(individual_id,
household_id,
family_relation_id_history,
marital_status_id_history,
ind_lmms_id_history,
ind_un_id_history,
head_of_hh_history,
ind_first_name_ar_history,
ind_last_name_ar_history,
ind_first_name_en_history,
ind_last_name_en_history,
ind_gender_history,
dob_history,
ind_status_history,
ind_action,
ind_date_changed,
user_id,
system_date)
SELECT individual_id,
household_id,
family_relation_id,
marital_status_id,
ind_lmms_id,
ind_un_id,
head_of_hh,
ind_first_name_ar,
ind_last_name_ar,
ind_first_name_en,
ind_last_name_en,
ind_gender,
dob,
ind_status,
'HH Status Changed to inactive',
ind_date_added,
user_id,
now()
FROM individual
WHERE individual.household_id = 12

Grouping MySQL results by 7 day increments

Hoping someone might be able to assist me with this.
Assume I have the table listed below. Hosts can show up multiple times on the same date, usually with different backupsizes.
+------------------+--------------+
| Field | Type |
+------------------+--------------+
| startdate | date |
| host | varchar(255) |
| backupsize | float(6,2) |
+------------------+--------------+
How could I find the sum total of backupsize for 7 day increments starting with the earliest date, through the last date? I don't mind if the last few days get cut off because they don't fall into a 7 day increment.
Desired output (prefered):
+------------+----------+----------+----------+-----
|Week of | system01 | system02 | system03 | ...
+------------+----------+----------+----------+-----
| 2014/07/30 | 2343.23 | 232.34 | 989.34 |
+------------+----------+----------+----------+-----
| 2014/08/06 | 2334.7 | 874.13 | 234.90 |
+------------+----------+----------+----------+-----
| ... | ... | ... | ... |
OR
+------------+------------+------------+------
|Host | 2014/07/30 | 2014/08/06 | ...
+------------+------------+------------+------
| system01 | 2343.23 | 2334.7 | ...
+------------+------------+------------+-------
| system02 | 232.34 | 874.13 | ...
+------------+------------+------------+-------
| system03 | 989.34 | 234.90 | ...
+------------+------------+------------+-------
| ... | ... | ... |
Date format is not a concern, just as long as it gets identified somehow. Also, the order of the hosts is not a concern either. Thanks!
The simplest way is to get the earliest date and just count the number of days:
select x.minsd + interval floor(datediff(x.minsd, lb.startdate) / 7) day as `Week of`,
host,
sum(backupsize)
from listedbelow lb cross join
(select min(startdate) as minsd from listedbelow lb) x
group by floor(datediff(x.minsd, lb.startdate) / 7)
order by 1;
This produces a form with week of and host on each row. You can pivot the results as you see fit.
I'll assume that what you want is the sum of bakcupsize grouped by host and that seven-day interval you are talking about.
My solution would be something like this:
You need to define the first date, and then "create" a column with the date you want (the end of the seven-day period)
Then I would group it.
I think temporary tables and little tricks with temp variables are the best way to tackle this, so:
drop table if exists temp_data;
create temporary table temp_data
select a.*
-- The #d variable will have the date that you'll use later to group the data.
, #d := case
-- If the current "host" value is the same as the previous one, then...
when #host_prev = host then
-- ... if #d is not null and is within the seven-day period,
-- then leave the value of #d intact; in other case, add 7 days to it.
case
when #d is not null or a.startdate <= #d then #d
-- The coalesce() function will return the first not null argument
-- (just as a precaution)
else dateadd(coalesce(#d, a.startdate), interval +7 day)
end
-- If the current "host" value is not the same as the previous one,
-- then take the current date (the first date of the "new" host) and add
-- seven days to it.
else #d = dateadd(a.startdate, interval +7 day)
end as date_group
-- This is needed to perform the comparisson in the "case" piece above
, #host_prev := a.host as host2
from
(select #host_prev = '', #d = null) as init -- Initialize the variables
, yourtable as a
-- IMPORTANT: This will only work if you order the data properly
order by a.host, a.startdate;
-- Add indexes to the temp table, to make things faster
alter table temp_data
add index h(host),
add index dg(date_group)
-- OPTIONAL: You can drop the "host2" column (it is no longer needed)
-- , drop column host2
;
Now, you can get the grouped data:
select a.host, a.date_group, sum(a.bakcupsize) as backupsize
from temp_data as a
group by a.host, a.date_group;
This will give you the unpivoted data. If you want to build a pivot table with it, I recommend you take a look to this article, and/or read this question and its answers. In short, you'll have to build a "dynamic" sql instruction, prepare a statement with it and execute it.
Of course, if you want to group this by week, there's a simpler approach:
drop table if exists temp_data2;
create temporary table temp_data2
select a.*
-- The following will give you the end-of-week date
, dateadd(a.startdate, interval +(6 - weekday(a.startdate)) day) as group_date
from yourtable as a;
alter table temp_data
add index h(host),
add index dg(date_group);
select a.host, a.date_group, sum(a.bakcupsize) as backupsize
from temp_data as a
group by a.host, a.date_group;
I leave the pivot part to you.
So I was able to determine a solution that fit my needs using a procedure I created by putting together concepts from your recommended solutions as well as some other other solutions I found on this site. The procedure SUM's by 7 day increments as well as does a pivot.
DELIMITER $$
CREATE PROCEDURE `weekly_capacity_by_host`()
BEGIN
SELECT MIN(startdate) into #start_date FROM testtable;
SET #SQL = NULL;
SELECT
GROUP_CONCAT(DISTINCT
CONCAT(
'SUM(if(host=''',host,''', backupsize, 0)) as ''',host,''''
)
) INTO #SQL
FROM testtable;
SET #SQL = CONCAT('SELECT 1 + DATEDIFF(startdate, ''',#start_date,''') DIV 7 AS week_num
, ''',#start_date,''' + INTERVAL (DATEDIFF(startdate, ''',#start_date,''') DIV 7) WEEK AS week_start,
', #SQL,'
FROM testtable group by week_num'
);
PREPARE stmt FROM #SQL;
EXECUTE stmt;
DEALLOCATE PREPARE stmt;
END$$
DELIMITER ;
Output appears as follows:
mysql> call weekly_capacity_by_host;
+----------+------------+----------+----------+----------+----------+
| week_num | week_start | server01 | server02 | server03 | server04 |
+----------+------------+----------+----------+----------+----------+
| 1 | 2014-06-11 | 1231.08 | 37.30 | 12.04 | 68.17 |
| 2 | 2014-06-18 | 1230.98 | 37.30 | 11.76 | 68.13 |
| 3 | 2014-06-25 | 1243.12 | 37.30 | 8.85 | 68.59 |
| 4 | 2014-07-02 | 1234.73 | 37.30 | 11.77 | 67.80 |
| 5 | 2014-07-09 | 341.32 | 0.04 | 0.14 | 4.94 |
+----------+------------+----------+----------+----------+----------+
5 rows in set (0.03 sec)

Selecting all fields with values greater than a current field value

I have a table that looks like this.
| path_id | step | point_id | delay_time | stand_time | access |
| 202 | 1 | 111 | 0 | 0 | 7 |
Which lists point_id's in step order.
E.g.: 111 - step 1, 181 - step 2, etc.
I need to write a query that would take point_id, select ALL values which have higher step within ALL path_id's that have a given value and return a grouped set of point_id's.
I am currently using this query
SELECT DISTINCT `pdb`.`point_id` AS `id`
FROM `path_detail` AS `pda` INNER JOIN
`path_detail` AS `pdb` ON pda.path_id = pdb.path_id
AND pda.step < pdb.step
WHERE
(pda.point_id = 111)
GROUP BY `pdb`.`path_id`
Which doesn't seem to work too reliably.
Any suggestions?
Try:
SELECT Distinct `pdb`.`point_id` AS `id`
FROM `path_detail` AS `pda`, `path_detail` AS `pdb`
WHERE
pda.point_id = 111
AND pda.path_id = pdb.path_id
AND pda.step < pdb.step
Order by `pdb`.`point_id` ASC