How do I populate a MySQL table with many random numbers? - mysql

I'm going to ask a question that has been asked in very abstract terms, with (understandably) no concrete answers provided:
From the MySQL prompt, how do I create and populate a table, rand_numbers, with one column, number INT, and 1111 rows, where the number column holds a random number between 2222 and 5555?
Something like:
CREATE TABLE rand_numbers(number INT);
#run following line 1111 times
INSERT INTO rand_numbers (number) VALUES (2222 + CEIL( RAND() * 3333));
This question has been asked, but either relies on external languages for the loop or is far too general. I would like to know if it's possible to do something this simple from a typical Linux MySQL prompt.

To create the table use:
CREATE TABLE rand_numbers (
number INT NOT NULL
) ENGINE = MYISAM;
Then to populate it with random values, you can define a stored procedure (which supports looping):
DELIMITER $$
CREATE PROCEDURE InsertRand(IN NumRows INT, IN MinVal INT, IN MaxVal INT)
BEGIN
DECLARE i INT;
SET i = 1;
START TRANSACTION;
WHILE i <= NumRows DO
INSERT INTO rand_numbers VALUES (MinVal + CEIL(RAND() * (MaxVal - MinVal)));
SET i = i + 1;
END WHILE;
COMMIT;
END$$
DELIMITER ;
CALL InsertRand(1111, 2222, 5555);
Then you can reuse that procedure to insert more random values based on different parameters.. say 600 rows with random values between 1200 and 8500:
CALL InsertRand(600, 1200, 8500);

Without creating a stored procedure, one technique I've applied is to use the table itself to add the columns. First seed it with a value...
INSERT INTO rand_numbers ( number ) VALUES ( rand() * 3333 );
Then insert again, selecting from this table to double the rows each time...
INSERT INTO rand_numbers ( number ) SELECT number * rand() FROM rand_numbers;
You don't need to run the second query that many times to get quite a few random rows. Not as "neat" as using a stored procedure of course, just proposing an alternative.
As pointed out by mohamed23gharbi, you can run into duplicates if your test mass is too large. You can use INSERT IGNORE to skip duplicates if that is a problem.

The task can be done also this way:
-- scale from 0 to MAX
UPDATE `table` SET `column` = 1000 * RAND() WHERE 1;
-- scale from MIN to MAX
UPDATE `table` SET `column` = MIN + (MAX - MIN) * RAND() WHERE 1;
You can also use math function like FLOOR(), CEIL(), etc. in the expression..

I have always used this -
insert into rand_numbers ( number ) select rand() from (
select 0 as i
union select 1 union select 2 union select 3
union select 4 union select 5 union select 6
union select 7 union select 8 union select 9
) as t1, (
select 0 as i
union select 1 union select 2 union select 3
union select 4 union select 5 union select 6
union select 7 union select 8 union select 9
) as t2, (
select 0 as i
union select 1 union select 2 union select 3
union select 4 union select 5 union select 6
union select 7 union select 8 union select 9
) as t3;
Inserts 1000 random numbers. On-the-fly tables t1, t2, t3 are cross joined so we get 10x10x10 rows.
So, for like a million rows, just add 3 more of
(select 0 as i union select 1 ...) as statements. This seems convenient to me, since there's not much effort copy-pasting a few lines a bunch of times.
Hope this helps,

If you are lazy and you have the query for creating the table, try http://filldb.info//

Related

Splitting string in mysql every n value

I have this blob field in a MySQL database and its quite lengthy and I'm needing to split up the values every 4 bytes, the data is displayed in hex variables.
7A080040950507000100000000000000320900420200000002000000C04D032E1841712CFFFFFFFF4E0000000100000000000000AD95014202000000020000004040032E78FD712CFFFFFFFFA89C0B0001000000000000003209004202000000020000004040032E1841712C96080040FFFFFFFF01000000F4B55D0CA79501420200000002000000C04D032E10E8712CFFFFFFFF7F4310000100000000000000AD950142020000000200000040CBFA2D78FD682CFFFFFFFF0000000001000000000000003F090042020000000200000040CBFA2D401F6F2CFFFFFFFF0000000000000000000000000000000000000000000000000000000000000000FFFFFFFF0000000000000000000000000000000000000000000000000000000000000000FFFFFFFF0000000000000000000000000000000000000000000000000000000000000000FFFFFFFF0000000000000000000000000000000000000000000000000000000000000000FFFFFFFF0000000001000000000000004E06004202000000F4011C10C0C7B82EF8A9652CFFFFFFFF000000000100000000000000AA06004202000000020000004040032E4873682CFFFFFFFF000000000100000000000000AA060042020000000200000040CBFA2D20805F2CFFFFFFFF000000000100000000000000360600420
This is a sample of the data and I'm just wanting to split it up to look like 7A08 0040 9505 0700 0100 0000 0000 0000 3209 0042 and so on to place into their own columns.
I've done a lot of searching but I've not been able to find anything that will allow me to do what I'm asking and any help would be appreciated. I need to be able to do this in MySQL only.
If you just need to split up the data you can use Substring('Text',start,length).
However to assign values to an unspecified number of columns, is not how SQL normally work. I would suggest you make a subtable to contain the substrings and relate the main table to the subtable with af key.
DECLARE #text NVARCHAR(1000)
DECLARE #text_Sub NVARCHAR(10)
DECLARE #i int -- integration variable
DECLARE #foreignKey int --relation key to main table
SET #foreignKey = 1 -- Must be adjusted for each string you want to pass
SET #text = '0x7A080040950507000100000000000000320900420200000002000000C04D032E1841712CFFFFFFFF4E0000000100000000000000AD95014202000000020000004040032E78FD712CFFFFFFFFA89C0B0001000000000000003209004202000000020000004040032E1841712C96080040FFFFFFFF01000000F4B55D0CA79501420200000002000000C04D032E10E8712CFFFFFFFF7F4310000100000000000000AD950142020000000200000040CBFA2D78FD682CFFFFFFFF0000000001000000000000003F090042020000000200000040CBFA2D401F6F2CFFFFFFFF0000000000000000000000000000000000000000000000000000000000000000FFFFFFFF0000000000000000000000000000000000000000000000000000000000000000FFFFFFFF0000000000000000000000000000000000000000000000000000000000000000FFFFFFFF0000000000000000000000000000000000000000000000000000000000000000FFFFFFFF0000000001000000000000004E06004202000000F4011C10C0C7B82EF8A9652CFFFFFFFF000000000100000000000000AA06004202000000020000004040032E4873682CFFFFFFFF000000000100000000000000AA060042020000000200000040CBFA2D20805F2CFFFFFFFF000000000100000000000000360600420'
-- this should be permanent table
CREATE TABLE #TempTable(
Id INT IDENTITY(1,1),
ForeignKey int,
Text NVARCHAR(10)
)
-- loop over text and insert into af subtable
SET #i = 0
SET #text_Sub = SUBSTRING(#text,#i,10)
WHILE (LEN(#text_Sub) > 0)
BEGIN
INSERT INTO #TempTable
( ForeignKey,Text)
VALUES
( #foreignKey,#text_Sub)
SET #i = #i +10
SET #text_Sub = SUBSTRING(#text,#i,10)
END
--Test the subtable have been filled
SELECT COUNT( *),MAX(Id)
FROM #TempTable
-- Assume you have a table called Table insert the relationkey/foreignKey
-- INSERT INTO Table
-- (ForeignKey)
-- VALUES
-- (#foreignKey)
-- WHERE 'SomeIdentifier'
--Clean up the temp table
DROP TABLE #TempTable
WITH RECURSIVE
cte AS ( SELECT UNHEX(LEFT(HEX(val), 8)) part,
UNHEX(SUBSTRING(HEX(val) FROM 9)) slack
FROM test
UNION ALL
SELECT UNHEX(LEFT(HEX(slack), 8)),
UNHEX(SUBSTRING(HEX(slack) FROM 9))
FROM cte
WHERE slack != '' )
SELECT part
FROM cte;
fiddle
That would work except I'm having to use MySQL 5.6 for the program to work properly.
SELECT /* UNHEX(SUBSTRING(HEX(val) FROM 1+8*(num1.num*100+num2.num*10+num3.num) FOR 8)) part */
SUBSTRING(HEX(val) FROM 1+8*(num1.num*100+num2.num*10+num3.num) FOR 8) part
FROM test
JOIN (SELECT 0 num UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9) num1
JOIN (SELECT 0 num UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9) num2
JOIN (SELECT 0 num UNION SELECT 1 UNION SELECT 2 UNION SELECT 3 UNION SELECT 4 UNION SELECT 5 UNION SELECT 6 UNION SELECT 7 UNION SELECT 8 UNION SELECT 9) num3
HAVING part != ''
ORDER BY num1.num*100+num2.num*10+num3.num
The query assumes that max. length of BLOB value is 4000 bytes. If the length is greater then add proper numN tables count.

Creating dynamic mysql queries

Is it possible to create sql file which can get two number parameters and use them in a loop, that in each iteration we do replace into directive using the two parameters, and increment them at the end of the loop?
Can someone show me how to do so?
Edit: Consider I want to update table named zip code, I want to insert new codes in this way:
You get two parameters which are numbers.
The first is the a start code for example: 1000
The second is number of sequential codes to add , lets say 5.
So you will update the table with 1000, 1001... 1004
SQL query cannot do loops, but you can "emulate" them by generating some data and then describing what you want to do with them in declarative way:
-- your input variables
set #start = 1000;
set #count = 5;
select val as zip from (
-- generate some numbers starting with the value of #start
select #start + (a.a + (10 * b.a) + (100 * c.a)) as val
from (
-- this creates cross join of 3 tables of numbers 0-9
-- so the select up there gets rows with values 0-999
-- you can add another cross join and 1000*d.a to get 0-9999
select 0 as a union all select 1 union all select 2 union all select 3 union all select 4 union all select 5 union all select 6 union all select 7 union all select 8 union all select 9) as a
cross join (select 0 as a union all select 1 union all select 2 union all select 3 union all select 4 union all select 5 union all select 6 union all select 7 union all select 8 union all select 9) as b
cross join (select 0 as a union all select 1 union all select 2 union all select 3 union all select 4 union all select 5 union all select 6 union all select 7 union all select 8 union all select 9) as c
) tmp
-- we should generate enough numbers to always cover the needs
-- then this condition will filter only currently needed values
where (val >= #start) and (val < #start + #count)
See it at http://sqlfiddle.com/#!9/9eecb7d/17037
I used this in few cases to generate all days between some dates (mostly to fill gaps in data for reporting) and it was surprisingly fast even if I tried big numbers. But if you know what the maximal value of #count can be, you can just use that much.

Search for the existance of many objects in a database

Say I have a database with 5 million users, with the columns
id (unsigned int, auto-increment), facebook_id (unsigned int), and name (varchar)
In a program, I have a list of a variable amount of users from a person's facebook friend list (generally ranging from 500-1200 different facebook ids).
What's the most efficient way to send a query to my database that returns the facebook_id's of all of the users where that same facebook_id exists in the database?
Pseudo-code:
$friends = array(12345, 22345, 32345, 42345, 52345, ... ~1000 more);
$q = mysql_query("SELECT * FROM users ...");
$friendsAlreadyUsingApp = parseQuery($q);
This is a topic of almost an endless number of articles, blogs, Q&As etc; and the essence of this problem is that it looks really simple - but isn't.
The heart of the problem is that the parameters looks like it should work using WHERE field IN() BUT it does not do that because the parameter is a single string that just happens to have lots of commas in it.
So, when that parameter is passed to SQL it is necessary to process that single string into multiple parts so that the field can be compared to each part. This is where it gets a little complex as not all database types have all the same features to handle this. MySQL for example does not have a table variable that MS SQL Server provides.
So. A simple method, for MySQL is this:
SET #param := '105,110,125,135,145,155,165,175,185,195,205';
SELECT
*
FROM Users
WHERE FIND_IN_SET(facebook_id, #param) > 0
;
FIND_IN_SET Return the index position of the first argument
within the second argument
Just how well this scales in your database I cannot tell, it might not be acceptable for parameters containing 1000+ id's.
So if text processing like FIND_IN_SET is too slow, then each id needs to be broken out from the parameter and inserted into a table. That way the resulting table can be used through an INNER JOIN to filter the users; but this requires a table and inserts which take time, and there may be concurrency issues if more than one user is attempting to use that table at the same time.
Using the following sets-up a table of 10,000 integers (1 to 10,000)
/* Create a table called Numbers */
CREATE TABLE `Numbers`
(
`Number` int PRIMARY KEY
);
/* use cross joins to create 10,000 integers from 1 & store into table */
INSERT INTO Numbers (Number)
select 1 + (a.a + (10 * b.a) + (100 * c.a) + (1000 * d.a)) as N
from (select 0 as a union all select 1 union all select 2 union all select 3 union all select 4 union all select 5 union all select 6 union all select 7 union all select 8 union all select 9) as a
cross join (select 0 as a union all select 1 union all select 2 union all select 3 union all select 4 union all select 5 union all select 6 union all select 7 union all select 8 union all select 9) as b
cross join (select 0 as a union all select 1 union all select 2 union all select 3 union all select 4 union all select 5 union all select 6 union all select 7 union all select 8 union all select 9) as c
cross join (select 0 as a union all select 1 union all select 2 union all select 3 union all select 4 union all select 5 union all select 6 union all select 7 union all select 8 union all select 9) as d
;
This "utility table" can then be used to divide a comma separated parameter into a derived table of the individual integers, and this then used in an INNER JOIN to your users table will provide the wanted result.
SET #param := '105,110,125,135,145,155,165,175,185,195,205';
SET #delimit := ',';
SELECT
users.id
, users.facebook_id
, users.name
FROM users
INNER JOIN (
SELECT
CAST(SUBSTRING(iq.param, n.number + 1, LOCATE(#delimit, iq.param, n.number + 1) - n.number - 1) AS UNSIGNED INTEGER) AS itemID
FROM (
SELECT
concat(#delimit, #param, #delimit) AS param
) AS iq
INNER JOIN Numbers n
ON n.Number < LENGTH(iq.param)
WHERE SUBSTRING(iq.param, n.number, 1) = #delimit
) AS derived
ON users.facebook_id = derived.itemID
;
This query can be used as the basis for a stored procedure which might be easier for you to call from PHP.
See this SQLFiddle demo

SQL command to Insert a range of years into a table

I am very new toSQL and am creating a basic database for a Car Dealership. I have a table "Years" that I need to hold a range of years (1950 - 2014).
Instead of
INSERT INTO 'Years' ('year') ("1950");
INSERT INTO 'Years' ('year') ("1951");
INSERT INTO 'Years' ('year') ("1952");
etc, etc...
Is there an easier way to populate the table?
Apart from the disscussing the necessity of such a table for your application the task of pre-populating tables on MySQL side can be solved in several ways.
First and fastest is to do it in one statement utilizing a tally(numbers) table that can be persisted in your db or generate it on the fly as in following example
INSERT INTO years (`year`)
SELECT 1950 + q.n - 1
FROM
(
SELECT a.N + b.N * 10 + 1 n
FROM
(SELECT 0 AS N UNION ALL SELECT 1 UNION ALL SELECT 2 UNION ALL SELECT 3 UNION ALL SELECT 4 UNION ALL SELECT 5 UNION ALL SELECT 6 UNION ALL SELECT 7 UNION ALL SELECT 8 UNION ALL SELECT 9) a
,(SELECT 0 AS N UNION ALL SELECT 1 UNION ALL SELECT 2 UNION ALL SELECT 3 UNION ALL SELECT 4 UNION ALL SELECT 5 UNION ALL SELECT 6 UNION ALL SELECT 7 UNION ALL SELECT 8 UNION ALL SELECT 9) b
ORDER BY n
) q
WHERE q.n <= 2014 - 1950 + 1
The subquery generates a sequence of numbers from 1 to 100. If you need ranges that span more than 100 years then you can adjust it or if you do a lot such queries then you can substitute it with a persisted tally(numbers) table.
Here is SQLFiddle demo
Second approach is to create a stored procedure that uses LOOP and uses prepared statement
DELIMITER $$
CREATE PROCEDURE populate_years(IN _start_year INT, IN _end_year INT)
BEGIN
PREPARE stmt FROM 'INSERT INTO years (year) VALUES(?)';
SET #y = _start_year;
START TRANSACTION;
WHILE #y <= _end_year DO
EXECUTE stmt USING #y;
SET #y = #y + 1;
END WHILE;
COMMIT;
DEALLOCATE PREPARE stmt;
END$$
DELIMITER ;
Here is SQLFiddle demo
You should rethink your code, you will never need a table like that... whatever you are doing with that table surely could be done without it.
If you still want to create the table you have to use a loop.
For example if you are using php:
for ($i = 1950; $i <= 2014; $i++){
mysqli->query("INSERT INTO 'Years' ('year') (".$i.")");
}

Selecting missing entries in MySQL

I have a database which contains data saved every 30 minutes (+/- ~3 seconds). There are about 20 000 records. Now I want to get all the datetimes when there isn't a record saved: For example, I don't want to get 2012-11-22 16:30 as a result because it exists in the database. But I want to get 2012-11-22 16:00 as one because the database doesn't contain an entry with that date.
Remember that the seconds part may vary. Usually it's exactly at the minute but sometimes it can be 2012-05-10 10:00:03 or so.
How do I do such a query?
If you're able to use stored procedures, then you can use this stored procedure to generate a range of date-times between the highest and lowest dates in the system.
If you can't be certain about the to-the-minute granularity of your timestamps, then you may need to use seconds as the interval instead of minutes.
A left-join against this table should reveal the dates and times when data hasn't been saved.
If you are looking for gaps, an easier query would be to find all times for which the next later time isn't within 30 minutes 6 seconds.
It is possible to do it in a single query for a specific total length of time. The following will check for missing times in a given range using an ad-hoc table of 65536 even 30 minute times from 2010 on (about 3.7 years of times):
select t
from (select date_add('2010-01-01', interval (a+4*b+16*c+64*d+256*e+1024*f+4096*g+16384*h)*30 minute) t from (select 0 a union select 1 union select 2 union select 3) a, (select 0 b union select 1 union select 2 union select 3) b, (select 0 c union select 1 union select 2 union select 3) c, (select 0 d union select 1 union select 2 union select 3) d, (select 0 e union select 1 union select 2 union select 3) e, (select 0 f union select 1 union select 2 union select 3) f, (select 0 g union select 1 union select 2 union select 3) g, (select 0 h union select 1 union select 2 union select 3) h order by t) ad_hoc_times
left join ( your_table, (select -3 t_adj union select -2 union select -1 union select 0 union select 1 union select 2 union select 3) t_adj )
on your_timestamp=date_add(t, interval t_adj second)
where t between '2010-07-01' and '2012-07-01'
and your_table.your_timestamp is null;
(Your timestamp field must be indexed.)
I created one table to show my stored procedure. Table creation query is given below
CREATE TABLE `testtable1` (
`id` INT(11) NULL DEFAULT NULL,
`timecol` DATETIME NULL DEFAULT NULL
)
Table contain data as given below
To meet your requirement i created following stored procedure
DELIMITER $$
CREATE PROCEDURE proc1(fromtime DATETIME,totime DATETIME)
BEGIN
DECLARE a INT Default 1;
DECLARE temptime DATETIME;
DECLARE ini,diff,nos int;
DECLARE temp1,temp6 datetime;
drop table if exists mytemptable;
CREATE TEMPORARY TABLE IF NOT EXISTS mytemptable ( `missing_dates` DATETIME NULL DEFAULT NULL);
if(minute(fromtime)>30) then
set diff=60-(minute(fromtime));
else
set diff=30-(minute(fromtime));
end if;
set temptime=ADDTIME(fromtime,concat('00:',diff,':00'));
while((unix_timestamp(totime)-unix_timestamp(temptime))>0) DO
set temp1=SUBTIME(temptime,'00:00:03');
set temp6=ADDTIME(temptime,'00:00:03');
select count(*) into nos from testtable1 where timecol>=temp1 and timecol<=temp6;
if(nos=0) then
insert into mytemptable (missing_dates) values (temptime);
end if;
set temptime=ADDTIME(temptime,'00:30:00');
END WHILE;
select * from mytemptable;
END $$
To get your required result just call above stored procedure with 'from time' and 'to time'. For example
call proc1('2013-01-01 14:00:00','2013-01-01 17:00:00')
Result is given below