I have this blob field in a MySQL database and its quite lengthy and I'm needing to split up the values every 4 bytes, the data is displayed in hex variables.
7A080040950507000100000000000000320900420200000002000000C04D032E1841712CFFFFFFFF4E0000000100000000000000AD95014202000000020000004040032E78FD712CFFFFFFFFA89C0B0001000000000000003209004202000000020000004040032E1841712C96080040FFFFFFFF01000000F4B55D0CA79501420200000002000000C04D032E10E8712CFFFFFFFF7F4310000100000000000000AD950142020000000200000040CBFA2D78FD682CFFFFFFFF0000000001000000000000003F090042020000000200000040CBFA2D401F6F2CFFFFFFFF0000000000000000000000000000000000000000000000000000000000000000FFFFFFFF0000000000000000000000000000000000000000000000000000000000000000FFFFFFFF0000000000000000000000000000000000000000000000000000000000000000FFFFFFFF0000000000000000000000000000000000000000000000000000000000000000FFFFFFFF0000000001000000000000004E06004202000000F4011C10C0C7B82EF8A9652CFFFFFFFF000000000100000000000000AA06004202000000020000004040032E4873682CFFFFFFFF000000000100000000000000AA060042020000000200000040CBFA2D20805F2CFFFFFFFF000000000100000000000000360600420
This is a sample of the data and I'm just wanting to split it up to look like 7A08 0040 9505 0700 0100 0000 0000 0000 3209 0042 and so on to place into their own columns.
I've done a lot of searching but I've not been able to find anything that will allow me to do what I'm asking and any help would be appreciated. I need to be able to do this in MySQL only.
If you just need to split up the data you can use Substring('Text',start,length).
However to assign values to an unspecified number of columns, is not how SQL normally work. I would suggest you make a subtable to contain the substrings and relate the main table to the subtable with af key.
DECLARE #text NVARCHAR(1000)
DECLARE #text_Sub NVARCHAR(10)
DECLARE #i int -- integration variable
DECLARE #foreignKey int --relation key to main table
SET #foreignKey = 1 -- Must be adjusted for each string you want to pass
SET #text = '0x7A080040950507000100000000000000320900420200000002000000C04D032E1841712CFFFFFFFF4E0000000100000000000000AD95014202000000020000004040032E78FD712CFFFFFFFFA89C0B0001000000000000003209004202000000020000004040032E1841712C96080040FFFFFFFF01000000F4B55D0CA79501420200000002000000C04D032E10E8712CFFFFFFFF7F4310000100000000000000AD950142020000000200000040CBFA2D78FD682CFFFFFFFF0000000001000000000000003F090042020000000200000040CBFA2D401F6F2CFFFFFFFF0000000000000000000000000000000000000000000000000000000000000000FFFFFFFF0000000000000000000000000000000000000000000000000000000000000000FFFFFFFF0000000000000000000000000000000000000000000000000000000000000000FFFFFFFF0000000000000000000000000000000000000000000000000000000000000000FFFFFFFF0000000001000000000000004E06004202000000F4011C10C0C7B82EF8A9652CFFFFFFFF000000000100000000000000AA06004202000000020000004040032E4873682CFFFFFFFF000000000100000000000000AA060042020000000200000040CBFA2D20805F2CFFFFFFFF000000000100000000000000360600420'
-- this should be permanent table
CREATE TABLE #TempTable(
Id INT IDENTITY(1,1),
ForeignKey int,
Text NVARCHAR(10)
)
-- loop over text and insert into af subtable
SET #i = 0
SET #text_Sub = SUBSTRING(#text,#i,10)
WHILE (LEN(#text_Sub) > 0)
BEGIN
INSERT INTO #TempTable
( ForeignKey,Text)
VALUES
( #foreignKey,#text_Sub)
SET #i = #i +10
SET #text_Sub = SUBSTRING(#text,#i,10)
END
--Test the subtable have been filled
SELECT COUNT( *),MAX(Id)
FROM #TempTable
-- Assume you have a table called Table insert the relationkey/foreignKey
-- INSERT INTO Table
-- (ForeignKey)
-- VALUES
-- (#foreignKey)
-- WHERE 'SomeIdentifier'
--Clean up the temp table
DROP TABLE #TempTable
WITH RECURSIVE
cte AS ( SELECT UNHEX(LEFT(HEX(val), 8)) part,
UNHEX(SUBSTRING(HEX(val) FROM 9)) slack
FROM test
UNION ALL
SELECT UNHEX(LEFT(HEX(slack), 8)),
UNHEX(SUBSTRING(HEX(slack) FROM 9))
FROM cte
WHERE slack != '' )
SELECT part
FROM cte;
fiddle
That would work except I'm having to use MySQL 5.6 for the program to work properly.
SELECT /* UNHEX(SUBSTRING(HEX(val) FROM 1+8*(num1.num*100+num2.num*10+num3.num) FOR 8)) part */
SUBSTRING(HEX(val) FROM 1+8*(num1.num*100+num2.num*10+num3.num) FOR 8) part
FROM test
JOIN (SELECT 0 num UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9) num1
JOIN (SELECT 0 num UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9) num2
JOIN (SELECT 0 num UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9) num3
HAVING part != ''
ORDER BY num1.num*100+num2.num*10+num3.num
The query assumes that max. length of BLOB value is 4000 bytes. If the length is greater then add proper numN tables count.
Related
Is it possible to create sql file which can get two number parameters and use them in a loop, that in each iteration we do replace into directive using the two parameters, and increment them at the end of the loop?
Can someone show me how to do so?
Edit: Consider I want to update table named zip code, I want to insert new codes in this way:
You get two parameters which are numbers.
The first is the a start code for example: 1000
The second is number of sequential codes to add , lets say 5.
So you will update the table with 1000, 1001... 1004
SQL query cannot do loops, but you can "emulate" them by generating some data and then describing what you want to do with them in declarative way:
-- your input variables
set #start = 1000;
set #count = 5;
select val as zip from (
-- generate some numbers starting with the value of #start
select #start + (a.a + (10 * b.a) + (100 * c.a)) as val
from (
-- this creates cross join of 3 tables of numbers 0-9
-- so the select up there gets rows with values 0-999
-- you can add another cross join and 1000*d.a to get 0-9999
select 0 as a union all select 1 union all select 2 union all select 3 union all select 4 union all select 5 union all select 6 union all select 7 union all select 8 union all select 9) as a
cross join (select 0 as a union all select 1 union all select 2 union all select 3 union all select 4 union all select 5 union all select 6 union all select 7 union all select 8 union all select 9) as b
cross join (select 0 as a union all select 1 union all select 2 union all select 3 union all select 4 union all select 5 union all select 6 union all select 7 union all select 8 union all select 9) as c
) tmp
-- we should generate enough numbers to always cover the needs
-- then this condition will filter only currently needed values
where (val >= #start) and (val < #start + #count)
See it at http://sqlfiddle.com/#!9/9eecb7d/17037
I used this in few cases to generate all days between some dates (mostly to fill gaps in data for reporting) and it was surprisingly fast even if I tried big numbers. But if you know what the maximal value of #count can be, you can just use that much.
Is there a simple way to compare a list of numbers in my query to a column in a table to return the ones that are NOT in the db?
I have a comma separated list of numbers (1,57, 888, 99, 76, 490, etc etc) that I need to compare to the number column in a table in my DB. SOME of those numbers are in the table, some are not. I need the query to return those that are in my comma separated list, but are NOT in the DB...
I would put the list of numbers to be checked in a table of their own, then use WHERE NOT EXISTS to check whether they exist in the table to be queried. See this SQLFiddle demo for an example of how this might be accomplished:
If you're comfortable with this syntax, you can even avoid putting into a temp table:
SELECT * FROM (
SELECT 1 AS mycolumn
UNION
SELECT 2
UNION
SELECT 3
UNION
SELECT 4
UNION
SELECT 5
UNION
SELECT 6
UNION
SELECT 7
) a
WHERE NOT EXISTS ( SELECT 1 FROM mytable b
WHERE b.mycolumn = a.mycolumn )
UPDATE per comments from OP
If you can insert your very long list of numbers into a table, then query as follows to get the numbers that are not found in the other table:
SELECT mynumber
FROM mytableof37000numbers a
WHERE NOT EXISTS ( SELECT 1 FROM myothertable b
WHERE b.othernumber = a.mynumber)
Alternately
SELECT mynumber
FROM mytableof37000numbers a
WHERE a.mynumber NOT IN ( SELECT b.othernumber FROM myothertable b )
Hope this helps.
May be this is what you are looking for.
Convert your CSV to rows using SUBSTRING_INDEX. Use NOT IN operator to find the values which is not present in DB
Then Convert the result back to CSV using Group_Concat.
select group_concat(value) from(
SELECT SUBSTRING_INDEX(SUBSTRING_INDEX(t.a, ',', n.n), ',', -1) value
FROM csv t CROSS JOIN
(
SELECT a.N + b.N * 10 + 1 n
FROM
(SELECT 0 AS N UNION ALL SELECT 1 UNION ALL SELECT 2 UNION ALL SELECT 3 UNION ALL SELECT 4 UNION ALL SELECT 5 UNION ALL SELECT 6 UNION ALL SELECT 7 UNION ALL SELECT 8 UNION ALL SELECT 9) a
,(SELECT 0 AS N UNION ALL SELECT 1 UNION ALL SELECT 2 UNION ALL SELECT 3 UNION ALL SELECT 4 UNION ALL SELECT 5 UNION ALL SELECT 6 UNION ALL SELECT 7 UNION ALL SELECT 8 UNION ALL SELECT 9) b
ORDER BY n
) n
WHERE n.n <= 1 + (LENGTH(t.a) - LENGTH(REPLACE(t.a, ',', '')))) ou
where value not in (select a from db)
SQLFIDDLE DEMO
CSV TO ROWS referred from this ANSWER
You could use the 'IN' clause of MySQL. Maybe check this out IN clause tutorial
Edit 1 the code is just an example, I do not have suburb data, my real data is inherited and messy and could be fixed by creating code out of the database or creating reference data. The question should have been something like does anyone have a good cell sort function or other solution that can be reused?
Table
CREATE TABLE postcode (
`id` int NOT NULL AUTO_INCREMENT,
`suburbs` varchar(2000) DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE = InnoDB
DEFAULT CHARACTER SET = utf8
COLLATE = utf8_unicode_ci;
Insert
INSERT INTO postcode (`suburbs`)
VALUES ( 'Zackville;Astor;Mary-town;Jackson' );
Want field to be sorted an result to be
Astor;Jackson;Mary-town;Zackville
All comments have very valid points and you should try to avoid working with delimited values in RDBMS.
That being said, if you're stuck with the existing database and you for some reason want to do it on the database level rather than with client code you can leverage tally(number) table and SUBSTRING_INDEX() to split delimited values into rows, then GROUP_CONCAT() to pivot data back in the ordered manner.
One of the several ways to create a tally table:
CREATE TABLE tally (n int not null primary key);
INSERT INTO tally (n)
SELECT a.N + b.N * 10 + 1 n
FROM
(SELECT 0 AS N UNION ALL SELECT 1 UNION ALL SELECT 2 UNION ALL SELECT 3 UNION ALL SELECT 4 UNION ALL SELECT 5 UNION ALL SELECT 6 UNION ALL SELECT 7 UNION ALL SELECT 8 UNION ALL SELECT 9) a
,(SELECT 0 AS N UNION ALL SELECT 1 UNION ALL SELECT 2 UNION ALL SELECT 3 UNION ALL SELECT 4 UNION ALL SELECT 5 UNION ALL SELECT 6 UNION ALL SELECT 7 UNION ALL SELECT 8 UNION ALL SELECT 9) b
ORDER BY n
Number of rows in the tally table should be greater or equal to the maximum possible number of delimited values.
Now to resort delimited values:
UPDATE postcode p JOIN
(
SELECT id, GROUP_CONCAT(suburbs ORDER BY suburbs SEPARATOR ';') suburbs
FROM
(
SELECT t.id, SUBSTRING_INDEX(SUBSTRING_INDEX(t.suburbs, ';', n.n), ';', -1) suburbs
FROM postcode t CROSS JOIN tally n
WHERE n.n <= 1 + (LENGTH(t.suburbs) - LENGTH(REPLACE(t.suburbs, ';', '')))
) q
GROUP BY id
) s
ON p.id = s.id
SET p.suburbs = s.suburbs;
Output:
| ID | SUBURBS |
|----|-----------------------------------|
| 1 | Astor;Jackson;Mary-town;Zackville |
Here is a SQLFiddle demo
I have a database which contains data saved every 30 minutes (+/- ~3 seconds). There are about 20 000 records. Now I want to get all the datetimes when there isn't a record saved: For example, I don't want to get 2012-11-22 16:30 as a result because it exists in the database. But I want to get 2012-11-22 16:00 as one because the database doesn't contain an entry with that date.
Remember that the seconds part may vary. Usually it's exactly at the minute but sometimes it can be 2012-05-10 10:00:03 or so.
How do I do such a query?
If you're able to use stored procedures, then you can use this stored procedure to generate a range of date-times between the highest and lowest dates in the system.
If you can't be certain about the to-the-minute granularity of your timestamps, then you may need to use seconds as the interval instead of minutes.
A left-join against this table should reveal the dates and times when data hasn't been saved.
If you are looking for gaps, an easier query would be to find all times for which the next later time isn't within 30 minutes 6 seconds.
It is possible to do it in a single query for a specific total length of time. The following will check for missing times in a given range using an ad-hoc table of 65536 even 30 minute times from 2010 on (about 3.7 years of times):
select t
from (select date_add('2010-01-01', interval (a+4*b+16*c+64*d+256*e+1024*f+4096*g+16384*h)*30 minute) t from (select 0 a union select 1 union select 2 union select 3) a, (select 0 b union select 1 union select 2 union select 3) b, (select 0 c union select 1 union select 2 union select 3) c, (select 0 d union select 1 union select 2 union select 3) d, (select 0 e union select 1 union select 2 union select 3) e, (select 0 f union select 1 union select 2 union select 3) f, (select 0 g union select 1 union select 2 union select 3) g, (select 0 h union select 1 union select 2 union select 3) h order by t) ad_hoc_times
left join ( your_table, (select -3 t_adj union select -2 union select -1 union select 0 union select 1 union select 2 union select 3) t_adj )
on your_timestamp=date_add(t, interval t_adj second)
where t between '2010-07-01' and '2012-07-01'
and your_table.your_timestamp is null;
(Your timestamp field must be indexed.)
I created one table to show my stored procedure. Table creation query is given below
CREATE TABLE `testtable1` (
`id` INT(11) NULL DEFAULT NULL,
`timecol` DATETIME NULL DEFAULT NULL
)
Table contain data as given below
To meet your requirement i created following stored procedure
DELIMITER $$
CREATE PROCEDURE proc1(fromtime DATETIME,totime DATETIME)
BEGIN
DECLARE a INT Default 1;
DECLARE temptime DATETIME;
DECLARE ini,diff,nos int;
DECLARE temp1,temp6 datetime;
drop table if exists mytemptable;
CREATE TEMPORARY TABLE IF NOT EXISTS mytemptable ( `missing_dates` DATETIME NULL DEFAULT NULL);
if(minute(fromtime)>30) then
set diff=60-(minute(fromtime));
else
set diff=30-(minute(fromtime));
end if;
set temptime=ADDTIME(fromtime,concat('00:',diff,':00'));
while((unix_timestamp(totime)-unix_timestamp(temptime))>0) DO
set temp1=SUBTIME(temptime,'00:00:03');
set temp6=ADDTIME(temptime,'00:00:03');
select count(*) into nos from testtable1 where timecol>=temp1 and timecol<=temp6;
if(nos=0) then
insert into mytemptable (missing_dates) values (temptime);
end if;
set temptime=ADDTIME(temptime,'00:30:00');
END WHILE;
select * from mytemptable;
END $$
To get your required result just call above stored procedure with 'from time' and 'to time'. For example
call proc1('2013-01-01 14:00:00','2013-01-01 17:00:00')
Result is given below
I'm going to ask a question that has been asked in very abstract terms, with (understandably) no concrete answers provided:
From the MySQL prompt, how do I create and populate a table, rand_numbers, with one column, number INT, and 1111 rows, where the number column holds a random number between 2222 and 5555?
Something like:
CREATE TABLE rand_numbers(number INT);
#run following line 1111 times
INSERT INTO rand_numbers (number) VALUES (2222 + CEIL( RAND() * 3333));
This question has been asked, but either relies on external languages for the loop or is far too general. I would like to know if it's possible to do something this simple from a typical Linux MySQL prompt.
To create the table use:
CREATE TABLE rand_numbers (
number INT NOT NULL
) ENGINE = MYISAM;
Then to populate it with random values, you can define a stored procedure (which supports looping):
DELIMITER $$
CREATE PROCEDURE InsertRand(IN NumRows INT, IN MinVal INT, IN MaxVal INT)
BEGIN
DECLARE i INT;
SET i = 1;
START TRANSACTION;
WHILE i <= NumRows DO
INSERT INTO rand_numbers VALUES (MinVal + CEIL(RAND() * (MaxVal - MinVal)));
SET i = i + 1;
END WHILE;
COMMIT;
END$$
DELIMITER ;
CALL InsertRand(1111, 2222, 5555);
Then you can reuse that procedure to insert more random values based on different parameters.. say 600 rows with random values between 1200 and 8500:
CALL InsertRand(600, 1200, 8500);
Without creating a stored procedure, one technique I've applied is to use the table itself to add the columns. First seed it with a value...
INSERT INTO rand_numbers ( number ) VALUES ( rand() * 3333 );
Then insert again, selecting from this table to double the rows each time...
INSERT INTO rand_numbers ( number ) SELECT number * rand() FROM rand_numbers;
You don't need to run the second query that many times to get quite a few random rows. Not as "neat" as using a stored procedure of course, just proposing an alternative.
As pointed out by mohamed23gharbi, you can run into duplicates if your test mass is too large. You can use INSERT IGNORE to skip duplicates if that is a problem.
The task can be done also this way:
-- scale from 0 to MAX
UPDATE `table` SET `column` = 1000 * RAND() WHERE 1;
-- scale from MIN to MAX
UPDATE `table` SET `column` = MIN + (MAX - MIN) * RAND() WHERE 1;
You can also use math function like FLOOR(), CEIL(), etc. in the expression..
I have always used this -
insert into rand_numbers ( number ) select rand() from (
select 0 as i
union select 1 union select 2 union select 3
union select 4 union select 5 union select 6
union select 7 union select 8 union select 9
) as t1, (
select 0 as i
union select 1 union select 2 union select 3
union select 4 union select 5 union select 6
union select 7 union select 8 union select 9
) as t2, (
select 0 as i
union select 1 union select 2 union select 3
union select 4 union select 5 union select 6
union select 7 union select 8 union select 9
) as t3;
Inserts 1000 random numbers. On-the-fly tables t1, t2, t3 are cross joined so we get 10x10x10 rows.
So, for like a million rows, just add 3 more of
(select 0 as i union select 1 ...) as statements. This seems convenient to me, since there's not much effort copy-pasting a few lines a bunch of times.
Hope this helps,
If you are lazy and you have the query for creating the table, try http://filldb.info//