It would be absolutely great if the following thing was possible:
Let's say I've got a 'document' mysql table, with a document 'id' and a few other columns:
CREATE TABLE document(id INT AUTO_INCREMENT NOT NULL, ....);
There can be quite a lot of document, but let's say that for now I've got only 2 millions.
I'd like to get the result of this query in my programming language space quickly:
SELECT id FROM document WHERE ... whatever ...;
The clause 'whatever' is potentially empty, so the set can contain the id's of all documents.
So my question is: is there a way to get the result of this query as bit vector BLOB of size 2 million bits (~ 250k of data) instead of potentially 2 millions of stringified numbers ( ~ 14Mo .. not great).
Extra kudos for blob compression in case of sparse sets :)
While performance will be horrendous, this stored procedure will give you the result you requested:
CREATE PROCEDURE ex12688666(whatever TEXT)
DETERMINISTIC
READS SQL DATA
SQL SECURITY INVOKER
COMMENT ''
proc: BEGIN
DECLARE not_found BOOL DEFAULT FALSE;
DECLARE max BIGINT UNSIGNED DEFAULT 0;
DECLARE len BIGINT UNSIGNED;
DECLARE i BIGINT UNSIGNED;
DECLARE pos BIGINT UNSIGNED;
DECLARE result LONGBLOB DEFAULT '';
DECLARE cur1 CURSOR FOR
SELECT id FROM ids WHERE id RLIKE whatever ORDER BY id;
DECLARE CONTINUE HANDLER FOR NOT FOUND SET not_found = TRUE;
SELECT MAX(id) INTO max FROM ids;
IF (max > 0) THEN
SET len = FLOOR((max + 7) / 8);
SET result = REPEAT("\0", len);
OPEN cur1;
loop1: LOOP
FETCH cur1 INTO i;
IF not_found THEN
LEAVE loop1;
END IF;
SET pos = FLOOR(i / 8) + 1;
SET result = CONCAT(
SUBSTRING(result, 1, pos - 1),
CHAR(ASCII(SUBSTRING(result, pos, 1)) | (1 << (i MOD 8))),
SUBSTRING(result, pos + 1)
);
END LOOP;
CLOSE cur1;
END IF;
SELECT HEX(result) AS result;
END;
We're returning HEX(result) for illustrative purposes. In practice, one could replace
SELECT HEX(result) AS result;
with simply
SELECT result;
or if you want zlib compression of the result:
SELECT COMPRESS(result) AS result;
which should get the extra kudos you mentioned.
See http://sqlfiddle.com/#!2/6f5c0/1 for an interactive demo.
Related
Can someone please tell me why MySQL does not recognize SPACE character when assigned to a CHAR variable?
The sample code below simply echoes back the input char by char into a variable just to demonstrate.
When called using my_test('1 2'), which is 6 chars total input, containing 4 spaces, the result is only '12' two chars returned.
If I change the definition of ch to CHAR(2), CHAR(4), CHAR(10) ...same result.
If I change the definition of ch to VARCHAR(1) it works as I would expect, returning all 6 original chars.
To me, this seems like a bug with CHAR handling on MySQL. The same code using CHAR(1), with minor syntax changes, works fine on Oracle and SQL Server.
Using MySQL 5.7.21
/*
select my_test('1 2'); -- Total len = 6 (contains 4 spaces)
*/
DROP FUNCTION IF EXISTS my_test;
DELIMITER //
CREATE FUNCTION my_test( original VARCHAR(100) )
RETURNS VARCHAR(100)
DETERMINISTIC
BEGIN
DECLARE ch CHAR(1); -- Will only recognize SPACE if changed to VARCHAR!
DECLARE idx INT DEFAULT 1;
DECLARE len INT;
DECLARE result VARCHAR(100) DEFAULT '';
SET len = CHAR_LENGTH(original);
WHILE idx <= len DO
SET ch = SUBSTR(original, idx, 1);
SET result = CONCAT(result, ch);
SET idx = idx + 1;
END WHILE;
RETURN result;
END;
The issue you are seeing can be resolved by ensuring that the function was created with sql_mode set to pad_char_to_full_length.
Solution
DROP FUNCTION IF EXISTS my_test;
SET sql_mode = 'PAD_CHAR_TO_FULL_LENGTH';
DELIMITER //
CREATE FUNCTION my_test( original VARCHAR(100) )
RETURNS VARCHAR(100)
DETERMINISTIC
BEGIN
DECLARE ch CHAR(1);
DECLARE idx INT DEFAULT 1;
DECLARE len INT;
DECLARE result VARCHAR(100) DEFAULT '';
SET len = CHAR_LENGTH(original);
WHILE idx <= len DO
SET ch = SUBSTR(original, idx, 1);
SET result = CONCAT(result, ch);
SET idx = idx + 1;
END WHILE;
RETURN result;
END//
delimiter ;
SET sql_mode = '';
Result
select my_test('1 2');
+-----------------+
| my_test('1 2') |
+-----------------+
| 1 2 |
+-----------------+
Explanation
If you have CHAR(5) and you stored A in it, MySQL will internally store it as A. However, when you select from that CHAR(5), right-padded spaces aren't retrieved UNLESS pad_char_to_full_length sql_mode is set. By default, your sql_mode is likely to not have pad_char_to_full_length.
According to documentation:
The length of a CHAR column is fixed to the length that you declare
when you create the table. The length can be any value from 0 to 255.
When CHAR values are stored, they are right-padded with spaces to the
specified length. When CHAR values are retrieved, trailing spaces are
removed unless the PAD_CHAR_TO_FULL_LENGTH SQL mode is enabled.
So, SET ch = SUBSTR(original, idx, 1); stores empty space in ch. However, when extracting information from ch MySQL will truncate spaces from the end resulting in empty string.
sql_mode
You can see your current sql_mode by typing select ##sql_mode. Also see How can I see the specific value of the sql_mode?.
You can review documentation of sql_mode in MySQL's doc.
Solution without using sql_mode of pad_char_to_full_length
Instead of doing this:
SET ch = SUBSTR(original, idx, 1);
SET result = CONCAT(result, ch);
Do this instead:
SET result = CONCAT(result, SUBSTR(original, idx, 1));
This single line assignment will avoid the CHAR trickery and give you intended results even with sql_mode does not have pad_char_to_full_length.
I need to sort below cell values using mysql
Example:
cell contain red,blue,green
But I want that in alphabetic order.
Steps to do this,
1.First you need to make a procedure call for sorting values
2.Call your procedure then
Here is the code to create mysql procedure
-- sort comma separated substrings with unoptimized bubble sort
DROP FUNCTION IF EXISTS sortString;
DELIMITER |
CREATE FUNCTION sortString(inString TEXT) RETURNS TEXT
BEGIN
DECLARE delim CHAR(1) DEFAULT ','; -- delimiter
DECLARE strings INT DEFAULT 0; -- number of substrings
DECLARE forward INT DEFAULT 1; -- index for traverse forward thru substrings
DECLARE backward INT; -- index for traverse backward thru substrings, position in calc. substrings
DECLARE remain TEXT; -- work area for calc. no of substrings
-- swap areas TEXT for string compare, INT for numeric compare
DECLARE swap1 TEXT; -- left substring to swap
DECLARE swap2 TEXT; -- right substring to swap
SET remain = inString;
SET backward = LOCATE(delim, remain);
WHILE backward != 0 DO
SET strings = strings + 1;
SET backward = LOCATE(delim, remain);
SET remain = SUBSTRING(remain, backward+1);
END WHILE;
IF strings < 2 THEN RETURN inString; END IF;
REPEAT
SET backward = strings;
REPEAT
SET swap1 = SUBSTRING_INDEX(SUBSTRING_INDEX(inString,delim,backward-1),delim,-1);
SET swap2 = SUBSTRING_INDEX(SUBSTRING_INDEX(inString,delim,backward),delim,-1);
IF swap1 > swap2 THEN
SET inString = TRIM(BOTH delim FROM CONCAT_WS(delim
,SUBSTRING_INDEX(inString,delim,backward-2)
,swap2,swap1
,SUBSTRING_INDEX(inString,delim,(backward-strings))));
END IF;
SET backward = backward - 1;
UNTIL backward < 2 END REPEAT;
SET forward = forward +1;
UNTIL forward + 1 > strings
END REPEAT;
RETURN inString;
END |
DELIMITER ;
To make procedure call just you have to use,
-- example call:
SET #Xstr = "red,blue,green"; // for query purpose only you need to write within (SQL Query here for that row)
SELECT sortString(#Xstr) AS s1
Please see the documentation guide map
Click here to read
Also there is an alternative way to do if you are interested to study is that about FIND_IN_SET, please you can find some idea from one of the question from stackoverflow. Click here to read
You can create a function which sorts the items in the column:
create function f_comma_list_order ( t text )
returns text
begin
declare v_c int;
drop temporary table if exists tmp;
create temporary table tmp ( v text );
set v_c = 1;
while( v_c > 0 ) do
select locate(',', t) into v_c;
if (v_c>0) then
insert into tmp select substr(t, 1, v_c-1);
set t = substr(t, v_c+1);
else
insert into tmp values (t);
end if;
end while;
select group_concat(v order by v) into t
from tmp;
return t;
end
and then call the function:
select f_comma_list_order('red,green,blue')
I'm migrating the database engine an application from MySql to SAP HANA.
I found a little trouble. I have a query like this:
Select SUBSTRING_INDEX(id, "-", -2) as prod_ref From products;
I don't know how to "translate" the function substring_index, because the initial part of the id has a variable length.
Thanks.
This can be done using a regex:
select substr_regexpr( '.*-([^-]*-[^-]*)$' in 'varia-ble---part-part1-part2' group 1) from dummy;
select substr_regexpr( '.*-([^-]*-[^-]*)$' in 'variable-part-part1-part2' group 1) from dummy;
According to the HANA 2.0 SP0 doc you could use locate with a negative offset (and then using right()), but this does not work on my system ("...feature isn't supported...")
If you execute such queries on a regular basis on lots of records I would recommend extracting the part you are interested in during ETL into a separate field. Or, alternatively fill a separate field using " GENERATED ALWAYS AS...".
I have seen it more than once, that people calculate a field like this in complex SQL queries or complex CalcViews, and then wonder why performance is bad when selecting 100 million records and filtering on the calculated field etc... Performance is usually no problem when you have aggregated your intermediate result set to a reasonable size and then apply "expensive" functions.
I don't think there is any direct function like SUBSTRING_INDEX in SAP HANA. But you have a work around alternative by creating a function to pass the input string and delimiter.
And I am assuming that -2 in SUBSTRING_INDEX and providing the solution
Reverse the string and get the position of the second delimiter, '-' in your case, into "obtainedPosition"
Now subtract that "obtainedPosition" from the length of the string.
obtainedPosition = LENGTH(id) - obtainedPosition
Using that value in the inbuilt substring function you can get the required string and return it from the function.
SELECT SCHEMA.FN_SUBSTRING_INDEX(id,obtainedPosition) INTO ReturnValue FROM DUMMY;
CREATE FUNCTION FN_SUBSTRING_INDEX
(
id VARCHAR(500),
delim VARCHAR(2)
)
RETURNS SplitString VARCHAR(500)
LANGUAGE SQLSCRIPT AS
BEGIN
DECLARE reversedString VARCHAR(500);
DECLARE charString VARCHAR(2);
DECLARE i INT := LENGTH(:id);
DECLARE len INT := LENGTH(:id);
DECLARE obtainedPosition INT := 0;
DECLARE flag INT := 0;
reversedString := '';
--loop to reverse the inputstring
WHILE :i > 0
DO
reversedString = CONCAT(:reversedString, SUBSTRING(:id,:i,1));
i := :i - 1;
END WHILE;
--loop to get the second delimiter position
i := 1;
WHILE :i <= :leng
DO
charString := '';
charString := SUBSTRING(:reversedString,i,1);
IF((:charString = :delim ) AND (:flag < 2)) THEN
BEGIN
obtainedPosition := :i;
flag := :flag + 1;
END;
END IF;
i := :i + 1;
END WHILE;
--IF condition to check if at least 2 delimiters are available, else print complete string
IF(flag = 2) THEN
obtainedPosition := :len - :obtainedPosition + 2; --2 is added to avoid the character at that position and '-' from printing
ELSE
obtainedPosition := 1;
END IF;
--SplitString contains the string's splitted return value
SELECT SUBSTRING(:id,:obtainedPosition) INTO SplitString FROM DUMMY;
END;
The above function is modified from http://www.kodyaz.com/sap-abap/sqlscript-reverse-string-function-in-sap-hana.aspx
For string functions in SAP HANA refer to this: http://www.sapstudent.com/hana/sql-string-functions-in-sap-hana/3
You can use anonymous block in SAP HANA to call and check the function
DO
BEGIN
DECLARE id VARCHAR(500) := 'Test-sam-ple-func';
DECLARE delim VARCHAR(2) := '-';
SELECT SCHEMA.FN_SUBSTRING_INDEX(id,delim) AS "SplitStringIndex" FROM DUMMY;
END;
I would be glad to know reason for a downvote. :)
I have developed a function for split string in tsql but mysql don't have some built in functions. I needed to function in MYSQL as i am new in mysql. Function should accept 2 parameters
1. String to be split
2. separator (',' or whatever)
Kindly reply me.
i had found solution on the internet you can into that.
DELIMITER //
DROP FUNCTION IF EXISTS `splitAndTranslate` //
CREATE FUNCTION splitAndTranslate(str TEXT, delim VARCHAR(124))
RETURNS TEXT
DETERMINISTIC
BEGIN
DECLARE i INT DEFAULT 0; -- total number of delimiters
DECLARE ctr INT DEFAULT 0; -- counter for the loop
DECLARE str_len INT; -- string length,self explanatory
DECLARE out_str text DEFAULT ''; -- return string holder
DECLARE temp_str text DEFAULT ''; -- temporary string holder
DECLARE temp_val VARCHAR(255) DEFAULT ''; -- temporary string holder for query
-- get length
SET str_len=LENGTH(str);
SET i = (LENGTH(str)-LENGTH(REPLACE(str, delim, '')))/LENGTH(delim) + 1;
-- get total number delimeters and add 1
-- add 1 since total separated values are 1 more than the number of delimiters
-- start of while loop
WHILE(ctr<i) DO
-- add 1 to the counter, which will also be used to get the value of the string
SET ctr=ctr+1;
-- get value separated by delimiter using ctr as the index
SET temp_str = REPLACE(SUBSTRING(SUBSTRING_INDEX(str, delim, ctr), LENGTH(SUBSTRING_INDEX(str, delim,ctr - 1)) + 1), delim, '');
-- query real value and insert into temporary value holder, temp_str contains the exploded ID
SELECT <real_value_column> INTO temp_val FROM <my_table> WHERE <table_id>=temp_str;
-- concat real value into output string separated by delimiter
SET out_str=CONCAT(out_str, temp_val, ',');
END WHILE;
-- end of while loop
-- trim delimiter from end of string
SET out_str=TRIM(TRAILING delim FROM out_str);
RETURN(out_str); -- return
END//
reference http://www.slickdev.com/2008/09/15/mysql-query-real-values-from-delimiter-separated-string-ids/
In mysql they they dont support some functionality like sqlserver. so spliting will be difficult in mysql
SELECT e.`studentId`, SPLIT(",", c.`courseNames`)[e.`courseId`]
FROM ..
SELECT TRIM(SUBSTRING_INDEX(yourcolumn,',',1)), TRIM(SUBSTRING_INDEX(yourcolumn,',',-1)) FROM yourtable
CREATE FUNCTION [dbo].[SplitString]
(
#RowData nvarchar(2000),
#SplitOn nvarchar(5)
)
RETURNS #RtnValue table
(
--Id int identity(1,1),
Data nvarchar(100)
)
AS
BEGIN
Declare #Cnt int
Set #Cnt = 1
While (Charindex(#SplitOn,#RowData)>0)
Begin
Insert Into #RtnValue (data)
Select
Data = ltrim(rtrim(Substring(#RowData,1,Charindex(#SplitOn,#RowData)-1)))
Set #RowData = Substring(#RowData,Charindex(#SplitOn,#RowData)+1,len(#RowData))
Set #Cnt = #Cnt + 1
End
Insert Into #RtnValue (data)
Select Data = ltrim(rtrim(#RowData))
Return
END
I want to limit my SELECT results in mySQL by sum.
For Example, this is my table:
(id, val)
Data Entries:
(1,100),
(2,300),
(3,50),
(4,3000)
I want to select first k entries such that the sum of val in those entries is just enough to make it to M.
For example, I want to find entries such that M = 425.
The result should be (1,100),(2,300),(3,50).
How can I do that in a mysql select query?
Try this variant -
SET #sum = 0;
SELECT id, val FROM (
SELECT *, #sum:=#sum + val mysum FROM mytable2 ORDER BY id
) t
WHERE mysum <= 450;
+------+------+
| id | val |
+------+------+
| 1 | 100 |
| 2 | 300 |
| 3 | 50 |
+------+------+
This stored procedure might help:
DELIMITER ;;
CREATE PROCEDURE selectLimitBySum (IN m INT)
BEGIN
DECLARE mTmp INT DEFAULT 0;
DECLARE idTmp INT DEFAULT 0;
DECLARE valTmp INT DEFAULT 0;
DECLARE doneLoop SMALLINT DEFAULT 0;
DECLARE crsSelect CURSOR FOR SELECT id, val FROM test3;
DECLARE CONTINUE HANDLER FOR NOT FOUND SET doneLoop = 1;
OPEN crsSelect;
aloop: LOOP
SET idTmp = 0;
SET valTmp = 0;
FETCH crsSelect INTO idTmp, valTmp;
if doneLoop THEN
LEAVE aloop;
END IF;
SELECT idTmp, valTmp;
SET mTmp = mTmp + valTmp;
if mTmp > m THEN
LEAVE aloop;
END IF;
END LOOP;
CLOSE crsSelect;
END ;;
DELIMITER ;
Please feel free to change the table names or variable names as per your needs.
from mysql reference manual:
The LIMIT clause can be used to constrain the number of rows returned by the SELECT statement. LIMIT takes one or two numeric arguments, which must both be nonnegative integer constants (except when using prepared statements).
So you cannot use limit the way you proposed. To achieve what you want you need to use your application (java, c, php or whatever else), read the result set row by row, and stop when your condition is reached.
or you can use a prepared statement, but anyway you cant have conditional limit (it must be a constant value) and it is not exactly what you asked for.
create table #limit(
id int,
val int
)
declare #sum int, #id int, #val int, #m int;
set #sum=0;
set #m=250; --Value of an entry
declare limit_cursor cursor for
select id, val from your_table order by id
open limit_cursor
fetch next from limit_cursor into #id, #val
while(##fetch_status=0)
begin
if(#sum<#m)
begin
set #sum = #sum+#val;
INSERT INTO #limit values (#id, #val);
fetch next from limit_cursor into #id, #val
end
else
begin
goto case1;
end
end
case1:
close limit_cursor
deallocate limit_cursor
select * from #limit
truncate table #limit