A procedure or a loop - mysql

I've a table named messages where users of my local hub store their messages(kind of like a web-forums). Currently, a majority of users are participating and I get nearly 30 to 50 new entries to my table everyday.
Since this has been going on for past few years, we've got nearly 100,000 rows of data in table. The table structure is kind of like this. Where fid is the PRIMARY and ip and id(nickname) are just INDEX.
I was using this kind of query uptil now; and then iterating the resultset in luasql as shown in this link. This, according to me, consumes a lot of time and space(in buffers).
`msg` VARCHAR(280) NOT NULL,
`id` VARCHAR(30) NOT NULL,
`ctg` VARCHAR(10) NOT NULL,
`date` DATE NOT NULL COMMENT 'date_format( %m/%d/%y )',
`time` TIME NOT NULL COMMENT 'date_format( %H:%i:%s )',
`fid` BIGINT(20) NOT NULL AUTO_INCREMENT,
`ip` CHAR(39) NOT NULL DEFAULT '127.0.0.1'
My problem is that now-a-days, we've switched to new API of PtokaX and the number of requests to read and write have increased dramatically. Since, I recently read about MySQL procedures, I was thinking if these procedures are a faster or safer way of dealing with this situation.
SELECT *
FROM ( SELECT *
FROM `messages`
ORDER BY `fid` DESC
LIMIT 50 ) AS `temp`
ORDER BY `fid` ASC;
P.S.
We get around one request to read one message every 7 to 10 seconds on average. On weekends, it rises to around one every 3 seconds.
Please let me know if anything more is required.
TO SUM UP
Is their a way that I can call a stored procedure and get the final result in a smaller time. Current query(and method) takes it nearly 3 seconds to fetch and organize the data.

Few things regarding your query:
SELECT *
FROM ( SELECT *
FROM `messages`
ORDER BY `fid` DESC
LIMIT 50 ) AS `temp`
ORDER BY `fid` ASC;
Never SELECT * (all); always specify a column list (what you need)
Subqueries typically cost more (for sorting & storage)
If you are trying to fetch the bottom '50', trying using a BETWEEN clause instead
You can always see what you're query is doing by using EXPLAIN. I would try the following query:
SELECT `msg`, `id`, `ctg`, `date`, `time`, `fid`, `ip` FROM `messages`
WHERE `fid` > (SELECT MAX(`fid`)-50 FROM `messages`)
ORDER BY `fid`

Related

MySQL - Select only the rows that have not been selected in the last read

Problem description
I have a table, say trans_flow:
CREATE TABLE trans_flow (
id BIGINT(20) AUTO_INCREMENT PRIMARY KEY,
card_no VARCHAR(50) DEFAULT NULL,
money INT(20) DEFAULT NULL
)
New data is inserted into this table constantly.
Now, I want to fetch only the rows that have not been fetched in the last query. For example, at 5:00, id ranges from 1 to 100, and I read the rows 80 - 100 and do some processing. Then, at 5:01, the id comes to 150, and I want to get exactly the rows 101 - 150. Otherwise, the processing program will read in old and already processed data. Note that such queries are committed continuously. From a certain perspective, I want to implement "streaming process" on MySQL.
A tentative idea
I have a simple but maybe ugly solution. I create an auxiliary table query_cursor which stores the beginning and end ids of one query:
CREATE TABLE query_cursor (
task_id VARCHAR(20) PRIMARY KEY COMMENT 'Specify which task is reading this table',
first_row_id BIGINT(20) DEFAULT NULL,
last_row_id BIGINT(20) DEFAULT NULL
)
During each query, I first update the query range stored in this table by:
UPDATE query_cursor
SET first_row_id = (SELECT last_row_id + 1 FROM query_cursor WHERE task_id = 'xxx'),
last_row_id = (SELECT MAX(id) FROM trans_flow)
WHERE task_id = 'xxx'
And then, doing query on table trans_flow using stored cursors:
SELECT * FROM trans_flow
WHERE id BETWEEN (SELECT first_row_id FROM query_cursor WHERE task_id = 'xxx')
AND (SELECT last_row_id FROM query_cursor WHERE task_id = 'xxx')
Question for help
Is there a simpler and more elegant implementation that can achieve the same effect (the best if no need to use an auxiliary table)? The version of MySQL is 5.7.

Woes using UPDATE with a CTE in MySQL (MariaDB)

I'm going crazy trying to get UPDATE to work with a CTE in MySQL.
Here's a simplified schema of sa_general_journal:
CREATE TABLE `sa_general_journal` (
`ID` int(10) unsigned NOT NULL AUTO_INCREMENT,
`Transaction_ID` int(10) unsigned DEFAULT NULL COMMENT 'NULL if not split, same as ID for split records',
`Date` timestamp NOT NULL DEFAULT current_timestamp(),
…
`Statement_s` int(10) unsigned DEFAULT NULL,
…
`Name` varchar(255) DEFAULT NULL,
…
PRIMARY KEY (`ID`),
…
) ENGINE=InnoDB AUTO_INCREMENT=25929 DEFAULT CHARSET=utf8;
Some records are "split," for example, a credit card statement amount might have a sales tax amount that is split out. In such cases, both parts of the split record have the same ID in the Transaction_ID field.
When records are imported in bulk, they can't refer to last_insert_ID in order to fill in the Transaction_ID field, thus the need to go clean these up afterward.
This was my first, naive attempt, which said I had an error near UPDATE. Well duh.
WITH cte AS (
SELECT
ID,
MIN(ID) OVER(PARTITION BY `Date`, `Name`, Statement_s) AS Trans,
Transaction_ID
FROM sa_general_journal
WHERE Transaction_ID = 0)
UPDATE cte
SET Transaction_ID = Trans
The CTE itself seems to work, as I can follow it with SELECT * FROM cte and get what I expected.
So I started searching StackOverflow, and discovered that CTEs are not updatable, but that you need to join them to what you want to update. "No problem!" I think, as I code this up:
WITH cte AS (
SELECT
ID,
MIN(ID) OVER(PARTITION BY `Date`, `Name`, Statement_s) AS Trans,
Transaction_ID
FROM sa_general_journal
WHERE Transaction_ID = 0)
UPDATE sa_general_journal gj, cte
SET gj.Transaction_ID = cte.Trans
WHERE gj.ID = cte.ID
No joy. Same error message.
My understanding is that in MySQL, you don't need a column list, but I did also try this using the column list (a, b, c), with the proper columns referenced in the UPDATE statement, but it still said I had a problem near UPDATE.
There are incredibly few examples of using UPDATE with WITH on the Internet! I found one, from Percona, which I used to create my attempt above, and then found another very similar example from MySQL itself.
Thanks in advance for any help offered!
CTE is a part of subquery definition, not a part of the whole query. The query must be specified after CTE. CTE cannot be used itself. So
UPDATE sa_general_journal gj
JOIN (WITH cte AS ( SELECT
ID,
MIN(ID) OVER(PARTITION BY `Date`, `Name`, Statement_s) AS Trans,
Transaction_ID
FROM sa_general_journal
WHERE Transaction_ID = 0)
SELECT * FROM cte) subquery ON gj.ID = subquery.ID
SET gj.Transaction_ID = subquery.Trans
CTEs work with UPDATE in MySQL 8, but not MariaDB 10.x.

MySQL Variable Returning Incorrect Value

The Issue
I have a stored proc in a DB server that's bringing back a value of 5064803 when that record does not exist and the value should be 5064800 as per the query that builds the value of the variable.
I'm not sure if this is an issue with the value being of the FLOAT data type and the value in the record of the table ending in a double-zero or what but I cannot figure it out easily.
The table data types match those from the sensors that are set but this particular value from this sensor never actually gets set to a data type and it's usually always either a 1-8 digit INT with no decimal but I'd like to keep the data types the same as the correlated sensor just in case.
I've broke down the proc and I'm able to recreate the problem easily so I will post the detail below for those that may be able to help me figure out the issue and any workaround, etc.
The SQL Data
Create Table
delimiter $$
CREATE TABLE `number` (
`TimeInt` varchar(10) NOT NULL,
`TimeStr` datetime NOT NULL,
`IsInitValue` int(11) NOT NULL,
`Value` float NOT NULL,
`IQuality` int(11) NOT NULL,
UNIQUE KEY `uk_Times` (`TimeInt`,`TimeStr`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8$$
Insert Data
INSERT INTO `Number` (`TimeInt`,`TimeStr`,`IsInitValue`,`Value`,`IQuality`) VALUES ('1502618950','2017-08-13 10:09:10',1,5064800,0);
INSERT INTO `Number` (`TimeInt`,`TimeStr`,`IsInitValue`,`Value`,`IQuality`) VALUES ('1502618796','2017-08-13 10:06:36',0,5064800,3);
INSERT INTO `Number` (`TimeInt`,`TimeStr`,`IsInitValue`,`Value`,`IQuality`) VALUES ('1502617167','2017-08-13 09:39:27',1,5063310,0);
INSERT INTO `Number` (`TimeInt`,`TimeStr`,`IsInitValue`,`Value`,`IQuality`) VALUES ('1502613355','2017-08-13 08:35:55',0,5063310,3);
INSERT INTO `Number` (`TimeInt`,`TimeStr`,`IsInitValue`,`Value`,`IQuality`) VALUES ('1502612814','2017-08-13 08:26:54',1,0,0);
INSERT INTO `Number` (`TimeInt`,`TimeStr`,`IsInitValue`,`Value`,`IQuality`) VALUES ('1502609015','2017-08-13 07:23:35',0,0,3);
The SQL Query Breakdown
SET #bStartTime = '2017-08-13 09:24:16';
SET #bEndTime = '2017-08-13 10:06:31';
SET #LastNumber = (SELECT Value FROM Number ORDER BY TimeStr DESC LIMIT 1);
SET #NowNumber = (SELECT Value FROM Number WHERE TimeStr BETWEEN #bStartTime AND #bEndTime ORDER BY TimeStr DESC LIMIT 1);
SELECT #NowNumber;
SELECT #LastNumber;
Recreating the Issue
So based on The SQL Query Breakdown above, once all the data is in the table and then I run the queries within the SELECT queries alone within the #NowNumber and/or #LastNumber variables, I get the correct result of 5064800. However, if I run the entire SET statements for both of those to have it set the query and then just do a SELECT of those variable, it brings back the wrong result of 5064803.
So for example if I run SELECT Value FROM Number ORDER BY TimeStr DESC LIMIT 1 then the correct value is returned. If I run SET #LastNumber = (SELECT Value FROM Number ORDER BY TimeStr DESC LIMIT 1); and then run SELECT #LastNumber; I get the incorrect value returned.
Server System Specs
This particular MySQL Server is running the x86 version of 5.5.50 on Windows Server 2008 with 144 GB of RAM for some quick specs.
Question
I'd like to know what is causing this, and if there is a workaround to the problem either with or without changing the data type of the column assuming that's the issue when it's returned as a variable rather than just a straight query result.
I'll be happy to disclose more technical specs of the environment if needed but I've included what I think it important for the question. Perhaps this is a version bug or there's something obvious that causes this that I cannot see easily so I'm hoping someone can help me with this or explain why this is or is not possible with MySQL.
Sorry, declares can only be used in stored procedures in MySQL. I found this article which may help. It explains how MySQL rounds when storing digits and recommends using doubles. Try changing your floats to doubles.
MySql FLOAT datatype and problems with more then 7 digit scale

How to select the most popular rows from today

I've been looking around but couldn't find my answer anywhere.
I have a MySQL table which looks something like this:
CREATE TABLE IF NOT EXISTS `g_video_watch` (
`vwatch_id` int(11) NOT NULL,
`vwatch_user` int(11) NOT NULL,
`vwatch_track` int(11) NOT NULL,
`vwatch_ip` varchar(255) NOT NULL,
`vwatch_date` timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP
) ENGINE=MyISAM DEFAULT CHARSET=utf8 AUTO_INCREMENT=36 ;
A page where the user can watch a video inserts everytime a page opens a row with the information from who the user is (vwatch_user) what track/video he or she opens (vwatch_track) and what the IP is. Also the current timestamp is inserted for the goal I'm trying to achieve.
And that goal is to show the tracks/videos which are most watched today. So in other words, select TODAY, GROUP all tracks and check how many rows per group is returned to order in COUNT where views is the highest...
Anybody an idea?
To get 100 most viewed tracks try this:
SELECT vwatch_track, COUNT(*)
FROM g_video_watch
WHERE DATE(vwatch_date) = DATE(NOW())
GROUP BY vwatch_track
ORDER BY COUNT(*) DESC
LIMIT 100
SELECT vwatch_track , count(*)
FROM g_video_watch
WHERE vwatch_date >= CURRENT_DATE
GROUP BY vwatch_track

MySQL: SUM in WHERE clause

I've got this table
CREATE TABLE `subevents` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`title` varchar(150) DEFAULT NULL,
`content` text,
`class` tinyint(4) NOT NULL DEFAULT '1',
PRIMARY KEY (`id`)
) ENGINE=MyISAM
Each row can have a different value in the 'class' field.
I'd like to select any number of rows, ordered randomly, as long as the sum of the values in the 'class' field is equal to 100.
How could I accomplish it directly in the MySQL query without doing it later in PHP?
Thanks everybody!
By "ordered randomly" I assume you mean that the order of the rows doesn't matter but no row can be used more than once. So you are looking for a combination of rows in which the sum of class equals 100. Use the brute force method. Randomly generate possible solutions until you find one that works.
delimiter //
CREATE PROCEDURE subsetsum(total)
BEGIN
DECLARE sum INTEGER;
REPEAT
CREATE OR REPLACE VIEW `solution`
AS SELECT * FROM `subevents`
WHERE 0.5 <= RAND();
SELECT SUM(`class`) INTO sum FROM `solution`;
UNTIL sum = total END REPEAT;
END
//
delimiter ;
CALL subsetsum(100); /* For example */
SELECT * FROM `solution`;
I have tested this with tables having a TINYINT column of random values and it is actually reasonably fast. The only problem is that there is no guarantee that subsetsum() will ever return.
I don't think this is possible with only SQL...the only thing which comes to my mind is to redo a the sql query as long the sum isn't 100
But I have no clue how to select a random number of rows at once.