Working with MySQL / MySQLi data - daily, weekly and monthly graphs - mysql

I'm developing a dashboard with graphs.
What's the problem?
Let's say, that I have a table with the folowing sctructure:
+-------+------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------+------+------+-----+---------+-------+
| total | int | NO | | NULL | |
| new | int | NO | | NULL | |
| date | date | YES | | NULL | |
+-------+------+------+-----+---------+-------+
where total stands for Total Members and new for New Members (date is a date of course - in format: yyyy-mm-dd).
Example of columns:
+-------+-------+------------+
| total | new | date |
+-------+-------+------------+
| 3450 | 21 | 2021-11-06 |
| 3650 | 200 | 2021-11-07 |
| 3694 | 34 | 2021-11-08 |
| 3520 | 26 | 2021-11-09 |
| 3399 | -321 | 2021-11-10 |
| 3430 | 31 | 2021-11-11 |
| 3450 | 20 | 2021-11-12 |
| 3410 | -40 | 2021-11-13 |
| 3923 | 513 | 2021-11-14 |
| 4019 | 96 | 2021-11-15 |
| 4119 | 100 | 2021-11-16 |
| 4000 | -119 | 2021-11-17 |
| 3000 | -1000 | 2021-11-18 |
| 3452 | 452 | 2021-11-19 |
| 3800 | 348 | 2021-11-20 |
| 3902 | 102 | 2021-11-21 |
| 4050 | 148 | 2021-11-22 |
+-------+-------+------------+
And there are a few options, where the dashboard user can select between 2 dates and type of graphs (daily, weekly, monthly).
Image, that describes the Setting options.
The Point
I need to take these 2 dates and somehow get all data from the database between the given term. Well, but that's not all. The Daily, Weekly and Monthly option means, that graphs will be showing average newcoming and total members per every week (so if I will grab 7 days from the database, I need to create an average - and do this between all these days / weeks / months in a term), if it's weekly, etc. So the final graph will be showing something like:
250 new 20 new 31 new
1000 total 1020 total 1051 total
Nov 7 Nov 14 Nov 21
etc...
More informations:
Ubuntu: 21.04
MySQL: 8.0.27
PHP: 7.4.23
Apache: 2.4.46
Does anyone have any ideas?

I don't get where your numbers come from
But your query would go like this.
For the month you need to group by MONTHof course
CREATE TABLE members (
`total` INTEGER,
`new` INTEGER,
`date` date
);
INSERT INTO members
(`total`, `new`, `date`)
VALUES
('3450', '21', '2021-11-06'),
('3650', '200', '2021-11-07'),
('3694', '34', '2021-11-08'),
('3520', '26', '2021-11-09'),
('3399', '-321', '2021-11-10'),
('3430', '31', '2021-11-11'),
('3450', '20', '2021-11-12'),
('3410', '-40', '2021-11-13'),
('3923', '513', '2021-11-14'),
('4019', '96', '2021-11-15'),
('4119', '100', '2021-11-16'),
('4000', '-119', '2021-11-17'),
('3000', '-1000', '2021-11-18'),
('3452', '452', '2021-11-19'),
('3800', '348', '2021-11-20'),
('3902', '102', '2021-11-21'),
('4050', '148', '2021-11-22');
SELECT `new`,sumtotal, `date` FROM members m
INNER JOIN (SELECT SUM(`new`) sumtotal, MIN(`date`) mindate FROM members GROUP BY WEEK(`date`)) t1
ON m.`date`= t1.mindate
WHERE m.`date` BETWEEN '2021-11-07' AND '2021-11-22'
new | sumtotal | date
--: | -------: | :---------
200 | -50 | 2021-11-07
513 | 390 | 2021-11-14
102 | 250 | 2021-11-21
db<>fiddle here

Related

To calculate the difference in two different merged queries - SQL

i am having some problems creating a query of my project.
I have created a software that cyclically (every minute) interrogates 3 or 4 devices and saves the collected data (people's incoming and outgoing) on a MySQL database.
This is the structure of my table:
```
+----+----------+----------+----------+---------------------+---------+
| id | DeviceId | ValueIn | ValueOut | DateTime | Status |
+----+----------+----------+----------+---------------------+---------+
| 1 | 1 | 00000429 | 00000327 | 2020-02-20 11:59:23 | online |
| 2 | 2 | 00000350 | 00000278 | 2020-02-28 12:54:12 | online |
| 3 | 1 | 00000000 | 00000000 | 2020-01-25 11:12:13 | offline |
| 4 | 2 | 00000300 | 00000200 | 2020-01-10 08:12:54 | online |
| 5 | 2 | 00000330 | 00000250 | 2020-02-01 09:00:54 | online |
| 6 | 1 | 00000400 | 00000300 | 2020-01-24 18:05:54 | online |
| 7 | 3 | 00000600 | 00000700 | 2020-02-10 13:05:54 | online |
| 8 | 3 | 00000000 | 00000000 | 2020-01-03 15:07:21 | offline |
| 9 | 4 | 00000111 | 00000111 | 2020-02-15 17:20:01 | online |
+----+----------+----------+----------+---------------------+---------+
```
My Query should return one row for each month (grouped by month) and each device with the most recent day of the month that have been collected by the software.
The same line should show also the ValueIn and ValueOut of the previous month: I will need it to work out the difference and calculate the monthly inputs/outputs.
What the table should show:
```
+----------+-----------+----------+---------------------+----------+----------+---------------------+
| ValueIn | ValueOut | DeviceId | DateTime | OLD_IN | OLD_OUT | OLD_DateTime |
+----------+-----------+----------+---------------------+----------+----------+---------------------+
| 00000350 | 000000278 | 2 | 2020-02-28 12:54:12 | 00000300 | 00000200 | 2020-01-10 08:12:54 |
| 00000429 | 000000327 | 1 | 2020-02-20 11:59:23 | 00000400 | 00000300 | 2020-01-25 11:12:13 |
| 00000600 | 000000700 | 3 | 2020-02-10 13:05:54 | 00000000 | 00000000 | 2020-01-03 15:07:21 |
| 00000111 | 000000111 | 4 | 2020-02-15 17:20:01 | 00000000 | 00000000 | null |
+----------+-----------+----------+---------------------+----------+----------+---------------------+
```
So, 50 people entered my DeviceId 2 in February and 78 left.
In DeviceId1 incoming: 429 outgoing: 327
Is it possible ??
I did some tests: if I have no < DateTime rows the query returns no results.
This is my starting query.
SELECT
TodayAccess.ValueIn,
TodayAccess.ValueOut,
TodayAccess.DateTime,
devices.Name,
MAX( OldAccess.ValueIn ),
MAX( OldAccess.ValueOut )
FROM peoplecounter AS TodayAccess
INNER JOIN devices ON TodayAccess.DeviceId = devices.Id
RIGHT JOIN peoplecounter AS OldAccess ON TodayAccess.DeviceId = OldAccess.DeviceId
WHERE TodayAccess.Status = 'online'
AND TodayAccess.DateTime BETWEEN '2020-06-01'
AND '2020-08-31'
AND OldAccess.DateTime < '2020-06-01'
GROUP BY TodayAccess.DeviceId, MONTHNAME(TodayAccess.DateTime)
I thank all those who will help me!

Displaying conditional mysql values in additional column

In a sample project i wanted to display data in such a way that based on dates the records for same student comes in additional columns.
mysql> desc sch_student;
+----------------+--------------+
| Field | Type |
+----------------+--------------+
| s_first_name | varchar(128) |
| s_last_name | varchar(128) |
| rollcode | int(8) |
| regnum | int(8) |
| in_time | datetime |
| out_time | datetime |
| total_time | int(8) |
+----------------+--------------+
for below query i am getting sample output like below , my expected output is something i am unable to get. I tried Sample join but it didn't work.
mysql> select * from sch_student;
+-------------------+---------------+--------------+-----------+---------------------+---------------------+----------------+
| s_first_name | s_last_name | rollcode | regnum | in_time | out_time | total_time |
+-------------------+---------------+--------------+-----------+---------------------+---------------------+----------------+
| Suzan | Matsuo | 8900 | 2897 | 2017-12-02 22:30:11 | 2017-12-02 22:30:11 | 00:17:00 |
| Scottie | Ogletree | 5624 | 5627 | 2017-12-02 16:40:01 | 2017-12-02 16:40:05 | 00:26:04 |
| Cynthia | Zimmerman | 3107 | 6348 | 2017-12-02 16:35:01 | 2017-12-02 16:35:01 | 00:59:89 |
| Ricardo | Shurtliff | 3072 | 261 | 2017-12-02 15:33:01 | 2017-12-02 15:33:01 | 00:16:55 |
| Elizabeth | Milligan | 4722 | 3233 | 2017-12-02 15:06:00 | 2017-12-02 15:10:33 | 00:14:33 |
+-------------------+---------------+--------------+-----------+---------------------+---------------------+----------------+
Expected output is something like below
+-------------------+---------------+--------------+-----------+---------------------+---------------------+----------------+--------------+-----------+---------------------+---------------------+----------------+
| s_first_name | s_last_name | Today's Meeting | Day Before Yesterday's Meeting |
| | rollcode | regnum | in_time | out_time | total_time | rollcode | regnum | in_time | out_time | total_time |
+-------------------+---------------+--------------+-----------+---------------------+---------------------+----------------+--------------+-----------+---------------------+---------------------+----------------+
| Suzan | Matsuo | 8900 | 2897 | 2017-12-02 22:30:11 | 2017-12-02 22:30:11 | 00:17:00 | 8900 | 2897 | 2017-11-30 12:30:11 | 2017-11-30 12:50:11 | 00:17:00 |
| Scottie | Ogletree | 5624 | 5627 | 2017-12-02 16:40:01 | 2017-12-02 16:40:05 | 00:26:04 | 5624 | 5627 | 2017-11-30 18:40:01 | 2017-11-30 19:33:05 | 00:26:04 |
| Cynthia | Zimmerman | 3107 | 6348 | 2017-12-02 16:35:01 | 2017-12-02 16:35:01 | 00:59:89 | 3107 | 6348 | 2017-11-30 13:35:01 | 2017-11-30 14:15:01 | 00:59:89 |
| Ricardo | Shurtliff | 3072 | 261 | 2017-12-02 15:33:01 | 2017-12-02 15:33:01 | 00:16:55 | 3072 | 261 | 2017-11-30 19:33:01 | 2017-11-30 20:33:01 | 00:16:55 |
| Elizabeth | Milligan | 4722 | 3233 | 2017-12-02 15:06:00 | 2017-12-02 15:10:33 | 00:14:33 | 4722 | 3233 | 2017-11-30 18:06:00 | 2017-11-30 19:10:33 | 00:14:33 |
+-------------------+---------------+--------------+-----------+---------------------+---------------------+----------------+--------------+-----------+---------------------+---------------------+----------------+
I tried below join and it's not returning expected output. Is it possible to display conditional column from table?
select * from
(
(select s_first_name,s_last_name,rollcode,regnum,in_time from sch_student where sch_student.in_time BETWEEN CURDATE()- INTERVAL 1 DAY AND CURDATE() ) As TD,
(select s_first_name,s_last_name,rollcode,regnum,in_time from sch_student where sch_student.in_time BETWEEN CURDATE()- INTERVAL 3 DAY AND CURDATE() ) As DBYS
) ;
I think this is what you need. I haven't tested it. Basically the query gets todays data LEFT joins to the day before yesterday's data. I assumed regnum and rollcode makes your primary key. Change if that isnt the case.
SELECT TD.* , DBYS.*
FROM (
SELECT s_first_name
,s_last_name
,rollcode
,regnum
,in_time
FROM sch_student
WHERE sch_student.in_time BETWEEN CURDATE() - INTERVAL 1 DAY
AND CURDATE()) AS TD
LEFT JOIN (
SELECT s_first_name
,s_last_name
,rollcode
,regnum
,in_time
FROM sch_student
WHERE sch_student.in_time BETWEEN CURDATE() - INTERVAL 3 DAY
AND CURDATE() - INTERVAL 2 DAY) AS DBYS
ON (TD.regnum = DBYS.regnum AND
TD.rollcode = DBYS.rollcode);
If you want to get info for today's meeting and the "day-before-yesterday's" meeting, try using a LEFT JOIN instead:
SELECT s_first_name, s_last_name, rollcode, regnum, in_time
FROM sch_student AS sch_today
LEFT JOIN sch_student AS sch_daybeforeyesterday ON
sch_today.<PK_FIELD> = sch_daybeforeyesterday.<PK_FIELD> AND
sch_daybeforeyesterday.in_time BETWEEN CURDATE()- INTERVAL 3 DAY AND CURDATE() - INTERVAL 2 DAY
WHERE sch_student.in_time BETWEEN CURDATE()- INTERVAL 1 DAY AND CURDATE()
This will give you all rows with "in_time" within the last 0-24 hours. For each of those rows, it will return any corresponding rows with "in_time" within the 48-72 hours.

Group by hour and date

I hate to ask a question here, as I know it's been done (slightly), however my problem is slightly different because of the field formats. I've tried a bunch of things on joins and hour selections and others, and I'm stuck. Would love a quick hand.
I've got a MySQL table like so:
CREATE TABLE `VAT` (
`VATid` int(11) NOT NULL AUTO_INCREMENT,
`VATDate` date DEFAULT NULL,
`VATTime` varchar(10) DEFAULT NULL,
`VATPledgeAmount` float DEFAULT NULL,
PRIMARY KEY (`VATid`),
UNIQUE KEY `VATid_UNIQUE` (`VATid`)
) ENGINE=MyISAM AUTO_INCREMENT=6531 DEFAULT CHARSET=latin1;
The problem that is that the VATDate fields have dates in the following format YYYY-MM-DD and the VATTime fields are a HHMM with leading zeros, as shown in the following table with some example data.
Sample Table Data:
VATid | VATDate | VATTime | VATPledgeAmount |
--------------------------------------------------
1 | 2016-06-30 | 0730 | 100 |
2 | 2016-06-30 | 0733 | 20 |
3 | 2016-06-30 | 0840 | 70 |
4 | 2016-06-30 | 0943 | 100 |
5 | 2016-06-30 | 0730 | 50 |
6 | 2016-07-01 | 2113 | 50 |
7 | 2016-07-01 | 2302 | 300 |
8 | 2016-07-02 | 0416 | 10 |
9 | 2016-07-02 | 0417 | 10 |
10 | 2016-07-02 | 0418 | 10 |
What I want to do is have the MySQL server do the calculations for me, so that when the data comes out the dates are still shown, but the total values from the PledgeAmount for each hour are summed up and grouped.
Counts | g_VATDate | g_VATTime | Total_VATPledgeAmount |
--------------------------------------------------
2 | 2016-06-30 | 0700 | 120 |
1 | 2016-06-30 | 0800 | 70 |
1 | 2016-06-30 | 0900 | 100 |
1 | 2016-07-01 | 2100 | 50 |
1 | 2016-07-01 | 2300 | 300 |
3 | 2016-07-02 | 0400 | 30 |
I hope this makes sense. I'm planning on using this for a JS graph, but as I don't need or want the individual pledges and their amount values, I just want the amounts and totals for each hour.
You can use concat(left(VATTime, 2), '00') to get the hour:
select count(VATid) counts, VATDate g_VATDate,
concat(left(VATTime, 2), '00') g_VATTime,
sum(VATPledgeAmount) Total_VATPledgeAmount
from VAT
group by g_VATDate, g_VATTime;

Using a calendar table to interpolate values across a date range

BACKGROUND
I am working on a project where I need to capture the 30 day average of values for some id# then use this average to determine if some new value is anomalous. For the purposes of this question, we can assume I only need a 10-day average since the solutions are probably similar. I currently have two tables: history which holds the actual values that I have recorded for specific id# numbers by day but can have some missing days and calendar a date table that has all of the days that I need in my 30 day average.
create table history (
day date not null,
id bigint not null,
category int not null,
value int not null default '0',
primary key (day, id, category),
key category (category)
);
create table calendar (
day date not null primary key
);
I would like to take the existing data that I have in the history table and fill in the missing data by either copying forward a previous value or copying back a forward value. E.g given this data in the history table:
+------------+-----------+----------+-------+
| day | id | category | value |
+------------+-----------+----------+-------+
| 2015-02-19 | 159253663 | 364 | 212 |
| 2015-02-20 | 159253663 | 364 | 211 |
| 2015-02-22 | 159253663 | 364 | 199 |
| 2015-02-23 | 159253663 | 364 | 192 |
| 2015-02-24 | 159253663 | 364 | 213 |
+------------+-----------+--------+---------+
Note: there is no entry for 2015-02-21
I would like to fill in enough data so that I can compute the 10-day average i.e. copy the oldest value (2015-02-19) back to the beginning of my 10-day range then fill in the missing 2015-02-21 value with the previous day's value. The result would be this (stars mark the newly added rows):
+------------+-----------+----------+-------+
| day | id | category | value |
+------------+-----------+----------+-------+
| 2015-02-14 | 159253663 | 364 | 212 | *
| 2015-02-15 | 159253663 | 364 | 212 | *
| 2015-02-16 | 159253663 | 364 | 212 | *
| 2015-02-17 | 159253663 | 364 | 212 | *
| 2015-02-18 | 159253663 | 364 | 212 | *
| 2015-02-19 | 159253663 | 364 | 212 |
| 2015-02-20 | 159253663 | 364 | 211 |
| 2015-02-21 | 159253663 | 364 | 211 | *
| 2015-02-22 | 159253663 | 364 | 199 |
| 2015-02-23 | 159253663 | 364 | 192 |
| 2015-02-24 | 159253663 | 364 | 213 |
+------------+-----------+--------+---------+
ATTEMPT
My initial thought was to left join to a calendar table that has the date ranges I need, when I do that I get something like this:
select c.day, h.id, h.value
from calendar c
left join history h using (day)
where c.day between curdate() - interval 10 day and curdate();
+------------+-----------+----------+-----------+
| day | id | category | value |
+------------+-----------+----------+-----------+
| 2015-02-14 | NULL | NULL | NULL |
| 2015-02-15 | NULL | NULL | NULL |
| 2015-02-16 | NULL | NULL | NULL |
| 2015-02-17 | NULL | NULL | NULL |
| 2015-02-18 | NULL | NULL | NULL |
| 2015-02-19 | 159253663 | 364 | 212 |
| 2015-02-19 | 159253690 | 364 | 222 |
| 2015-02-20 | 159253663 | 364 | 211 |
| 2015-02-20 | 159253690 | 364 | 221 |
| 2015-02-21 | NULL | NULL | NULL |
| 2015-02-22 | 159253663 | 364 | 199 |
| 2015-02-22 | 159253690 | 364 | 209 |
| 2015-02-23 | 159253663 | 364 | 192 |
| 2015-02-23 | 159253690 | 364 | 202 |
| 2015-02-24 | 159253663 | 364 | 213 |
| 2015-02-24 | 159253690 | 364 | 213 |
+------------+-----------+----------+-----------+
I am not sure where to proceed from this point, because I need an entry for each day for each distinct id#. This join only returns a single day if they are missing. I am looking for a better approach. I would like to push as much of the work as possible on the MySQL server, but can do some things programmaticaly. Any/all ideas or suggestions are welcome.
Here is a SQLFiddle that has the DDL definitions I am testing with: http://sqlfiddle.com/#!2/cc206/2
The following uses an # variable and in-statement assignments to roll backward the value (and id):
SET #lastval = 0, #lastid = 0;
SELECT c.day, #lastid := COALESCE(h.id,#lastid) id, #lastval := COALESCE(h.value,#lastval) VALUE, h.id id1,h.value v1
FROM (SELECT DISTINCT c.day,h.id FROM history h, calendar c) c
LEFT JOIN history h ON h.day = c.day AND h.id = c.id
WHERE c.day BETWEEN CURDATE() - INTERVAL 10 DAY AND CURDATE()
ORDER BY COALESCE(h.id,#lastid),c.day DESC
The sub-query seems to be necessary, never been too sure why (some do, some don`t).
If it looks like the results are in the wrong order you might have to add :
SET optimizer_switch='block_nested_loop=off';
before the statement as the block nested loop optimisation can mess with the Order mysql uses when collecting the rows.

Displaying records from database by quarter in mysql

I have a table named nca_totals.
Table: nca_totals
+----------+-----------+------------+--------------+
| total_id | nca_total | nca_date | account_type |
+----------+-----------+------------+--------------+
| 13 | 10450 | 2015-01-21 | DBP-TRUST |
| 14 | 5000 | 2015-02-05 | DBP-TRUST |
| 15 | 7000 | 2015-04-02 | DBP-TRUST |
| 16 | 4000 | 2015-05-02 | DBP-TRUST |
+----------+-----------+------------+--------------+
Now I want to display all the data by quarter base on its date. Let's say I want to display all the records who belong to 1st Quarter like this:
+----------+-----------+------------+--------------+
| total_id | nca_total | nca_date | account_type |
+----------+-----------+------------+--------------+
| 13 | 10450 | 2015-01-21 | DBP-TRUST |
| 14 | 5000 | 2015-02-05 | DBP-TRUST |
+----------+-----------+------------+--------------+
This date belongs to the 1st quarter of the year (Jan, Feb, March). I only have this query to select the date and return its quarter number as:
SELECT QUARTER('2015-01-11'); /* returns 1 */
How can I combine that query to display all the records by quarter ? Can anyone help ? Thanks.
select *
from nca_totals
where QUARTER(nca_date) = 1
SELECT
CEIL(MONTH(`nca_date`) / 3) AS `quarter`
FROM `nca_totals`;