Check if IP is in CIDR netmask (range) - mysql

I have 2 tables ip and cidr.
In the first one I store IP's. (2 column table, id, ip), here is an example (the values are fictional):
id | ip
---+-------------
1 | 172.922.2.10
---+-------------
2 | 194.22.10.13
In the second one I store CIDR netmask's (2 column table, id, cidr), here is an example (the values are fictional):
id | cidr
---+-------------
1 | 26.232.49.0/20
---+---------------
2 | 14.44.182.0/24
Is there any way to make a mysql query to check whether the ip's from the first table are in the range of any of my cidr netmasks?
Note: To convert a cidr netmask to a range of ip's click here

I'd personally recommend using postgres as it has a CIDR data-type and powerful functions to go with it, but there's an interesting discussion on doing similar things in MySQL, too.
http://planet.mysql.com/entry/?id=29283

This has come up in a related project of mine, and this appears to be the top google result for the question, so, you get the answer!
create function get_lowest_ipv4(cidr char(18)) returns bigint deterministic return INET_ATON(SUBSTRING_INDEX(cidr, '/', 1));
create function get_highest_ipv4(cidr char(18)) returns bigint deterministic return get_lowest_ipv4(cidr) + (0x100000000 >> SUBSTRING_INDEX(cidr,'/', -1)) - 1;
You can then do ... from ip_map where INET_ATON("ip.add.re.ss") between get_lowest_ipv4(ip) AND get_highest_ipv4(ip)
Because you declare the functions as deterministic, it'll get cached inside mysql and the calculation will only need to be run once. Then it'll just be 'is integer greater than y and less than x', which will be effectively instant.
MySQL [astpp]> set #cidr="10.11.0.0/16";
Query OK, 0 rows affected (0.00 sec)
MySQL [astpp]> select get_lowest_ipv4(#cidr), get_highest_ipv4(#cidr), INET_NTOA(get_lowest_ipv4(#cidr)), INET_NTOA(get_highest_ipv4(#cidr));
+------------------------+-------------------------+-----------------------------------+------------------------------------+
| get_lowest_ipv4(#cidr) | get_highest_ipv4(#cidr) | INET_NTOA(get_lowest_ipv4(#cidr)) | INET_NTOA(get_highest_ipv4(#cidr)) |
+------------------------+-------------------------+-----------------------------------+------------------------------------+
| 168493056 | 168558591 | 10.11.0.0 | 10.11.255.255 |
+------------------------+-------------------------+-----------------------------------+------------------------------------+
1 row in set (0.00 sec)
MySQL [astpp]> set #cidr="10.11.12.1/32";
Query OK, 0 rows affected (0.00 sec)
MySQL [astpp]> select get_lowest_ipv4(#cidr), get_highest_ipv4(#cidr), INET_NTOA(get_lowest_ipv4(#cidr)), INET_NTOA(get_highest_ipv4(#cidr));
+------------------------+-------------------------+-----------------------------------+------------------------------------+
| get_lowest_ipv4(#cidr) | get_highest_ipv4(#cidr) | INET_NTOA(get_lowest_ipv4(#cidr)) | INET_NTOA(get_highest_ipv4(#cidr)) |
+------------------------+-------------------------+-----------------------------------+------------------------------------+
| 168496129 | 168496129 | 10.11.12.1 | 10.11.12.1 |
+------------------------+-------------------------+-----------------------------------+------------------------------------+
1 row in set (0.01 sec)
MySQL [astpp]>
The only important caveat is that you insert VALID CIDRs. For example, 10.11.12.13/24 is not valid. That's an IP Address INSIDE the 10.11.12.0/24 network.
If you are unable to validate the CIDRs before inserting them (for some crazy reason), you could change get_lowest_ipv4 to do a bitwise comparison on the source, but that's much less elegant.
INET_ATON(SUBSTRING_INDEX(`ip`, '/', 1)) & 0xffffffff ^((0x1 <<(32 - SUBSTRING_INDEX(`ip`, '/', -1))) -1 )
Is an (untested) stab at matching invalid CIDRs.

Related

How to insert the default value in temporal tables in MySQL?

I want to create a temporal table from a SELECT statement in MySQL. It involves several JOINs, and it can produce NULL values that I want MySQL to take as zeroes. It sounds like an easy problem (simply default to zero), but MySQL (5.6.12) fails to elicit the default value.
For example, take the following two tables:
mysql> select * from TEST1;
+------+------+
| a | b |
+------+------+
| 1 | 2 |
| 4 | 25 |
+------+------+
2 rows in set (0.00 sec)
mysql> select * from TEST2;
+------+------+
| b | c |
+------+------+
| 2 | 100 |
| 3 | 100 |
+------+------+
2 rows in set (0.00 sec)
A left join gives:
mysql> select TEST1.*,c from TEST1 left join TEST2 on TEST1.b=TEST2.b;
+------+------+------+
| a | b | c |
+------+------+------+
| 1 | 2 | 100 |
| 4 | 25 | NULL |
+------+------+------+
2 rows in set (0.00 sec)
Now, if I want to save these values in a temporal table (changing NULL for zero), this is the code I would use:
mysql> create temporary table TEST_JOIN (a int, b int, c int default 0 not null)
select TEST1.*,c from TEST1 left join TEST2 on TEST1.b=TEST2.b;
ERROR 1048 (23000): Column 'c' cannot be null
What am I doing wrong? The worst part is that this code used to work before I did a system-wide upgrade (I don't remember which version of MySQL I had, but surely it was lower than my current 5.6). It used to produce the behavior I would expect: if it's NULL, use the default, not the frustrating error I'm getting now.
From the documentation of 5.6 (unchanged since 4.1):
Inserting NULL into a column that has been declared NOT NULL. For
multiple-row INSERT statements or INSERT INTO ... SELECT statements,
the column is set to the implicit default value for the column data
type. This is 0 for numeric types, the empty string ('') for string
types, and the “zero” value for date and time types. INSERT INTO ...
SELECT statements are handled the same way as multiple-row inserts
because the server does not examine the result set from the SELECT to
see whether it returns a single row. (For a single-row INSERT, no
warning occurs when NULL is inserted into a NOT NULL column. Instead,
the statement fails with an error.)
My current workaround is to store the NULL values in the temporal table, and then replace them by zeroes, but it seems rather cumbersome with many columns (and terribly inefficient). Is there a better way to do it?
BTW, I cannot simply ignore some columns in the query (as suggested for another question), because it's a multirow query.
IFNULL(`my_column`,0);
That would set NULLs to 0. Other values stay as is.
Just wrap your values/column names with IFNULL and it will convert them to whatever default value you put into the function. E.g. 0. Or "european swallow", or whatever you want.
Then you can keep strict mode on and still handle NULLs gracefully.

MySQL: how uuid_short is generated?

I have MySQL 5.5.24. Сalling uuid_short() few times I get just an incremented values:
mysql> select uuid_short();
+-------------------+
| uuid_short() |
+-------------------+
| 22851044396498953 |
+-------------------+
1 row in set (0.00 sec)
mysql> select uuid_short();
+-------------------+
| uuid_short() |
+-------------------+
| 22851044396498954 |
+-------------------+
1 row in set (0.00 sec)
But manual says:
The UUID_SHORT() return value is constructed this way:
(server_id & 255) << 56
+ (server_startup_time_in_seconds << 24)
+ incremented_variable++;
Seems like neither "server_startup_time_in_seconds" or "server_id" changes. (I changed ##global.server_id system variable and it took no effect).
Does anyone know why?
Seems to me like the function does return what's specified. The server Id and the start up time are server specific values and (usually) do not change as long as the server is running. The least significant part of the short UUID is an incremented value.
So I assume after the server start up a seed for UUIDs is created taken the left-shifted server id and the left-shifted start up time. This value is incremented and returned on every creation of an UUID. This explains why changing the ##global.server_id variable has no effect.

MySql: using #variable in select statment takes hundreds times longer

I'm trying to understand a huge performance difference that I'm seeing in equivalent code. Or at least code I think is equivalent.
I have a table with about 10 million records on it. It contains a field, which is indexed defined as:
USPatentNum char(8)
If I set a variable withing MySql to a value, it takes over 218 seconds. The exact same query with a string literal takes under 1/4 of a second.
In the code below, the first select statement (with where USPatentNum = #pn;) takes forever, but the second, with the literal value
(where USPatentNum = '5288812';) is nearly instant
mysql> select #pn := '5288812';
+------------------+
| #pn := '5288812' |
+------------------+
| 5288812 |
+------------------+
1 row in set (0.00 sec)
mysql> select patentId, USPatentNum, grantDate from patents where USPatentNum = #pn;
+----------+-------------+------------+
| patentId | USPatentNum | grantDate |
+----------+-------------+------------+
| 306309 | 5288812 | 1994-02-22 |
+----------+-------------+------------+
1 row in set (3 min 38.17 sec)
mysql> select #pn;
+---------+
| #pn |
+---------+
| 5288812 |
+---------+
1 row in set (0.00 sec)
mysql> select patentId, USPatentNum, grantDate from patents where USPatentNum = '5288812';
+----------+-------------+------------+
| patentId | USPatentNum | grantDate |
+----------+-------------+------------+
| 306309 | 5288812 | 1994-02-22 |
+----------+-------------+------------+
1 row in set (0.21 sec)
Two questions:
Why is the use of the #pn so much slower?
Can I change the select statement so that the performance will be the same?
Declare #pn as char(8) before setting its value.
I suspect it will be a varchar as you do it now. If so, the performance loss is because MySql can't mach the index with your variable.
It doesn't matter whether you use constant or #var. You get different result because the second time MySQL gets results from cache. If you execute once again your scenario but trade places queries with const and with #var you will get them same results (but with another value). First will be slowed, second will be fast.
Hope it helps

In MySQL, should I quote numbers or not?

For example - I create database and a table from cli and insert some data:
CREATE DATABASE testdb CHARACTER SET 'utf8' COLLATE 'utf8_general_ci';
USE testdb;
CREATE TABLE test (id INT, str VARCHAR(100)) TYPE=innodb CHARACTER SET 'utf8' COLLATE 'utf8_general_ci';
INSERT INTO test VALUES (9, 'some string');
Now I can do this and these examples do work (so - quotes don't affect anything it seems):
SELECT * FROM test WHERE id = '9';
INSERT INTO test VALUES ('11', 'some string');
So - in these examples I've selected a row by a string that actually stored as INT in mysql and then I inserted a string in a column that is INT.
I don't quite get why this works the way it works here. Why is string allowed to be inserted in an INT column?
Can I insert all MySQL data types as strings?
Is this behavior standard across different RDBMS?
MySQL is a lot like PHP, and will auto-convert data types as best it can. Since you're working with an int field (left-hand side), it'll try to transparently convert the right-hand-side of the argument into an int as well, so '9' just becomes 9.
Strictly speaking, the quotes are unnecessary, and force MySQL to do a typecasting/conversion, so it wastes a bit of CPU time. In practice, unless you're running a Google-sized operation, such conversion overhead is going to be microscopically small.
You should never put quotes around numbers. There is a valid reason for this.
The real issue comes down to type casting. When you put numbers inside quotes, it is treated as a string and MySQL must convert it to a number before it can execute the query. While this may take a small amount of time, the real problems start to occur when MySQL doesn't do a good job of converting your string. For example, MySQL will convert basic strings like '123' to the integer 123, but will convert some larger numbers, like '18015376320243459', to floating point. Since floating point can be rounded, your queries may return inconsistent results. Learn more about type casting here. Depending on your server hardware and software, these results will vary. MySQL explains this.
If you are worried about SQL injections, always check the value first and use PHP to strip out any non numbers. You can use preg_replace for this: preg_replace("/[^0-9]/", "", $string)
In addition, if you write your SQL queries with quotes they will not work on databases like PostgreSQL or Oracle.
Check this, you can understand better ...
mysql> EXPLAIN SELECT COUNT(1) FROM test_no WHERE varchar_num=0000194701461220130201115347;
+----+-------------+------------------------+-------+-------------------+-------------------+---------+------+---------+--------------------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+------------------------+-------+-------------------+-------------------+---------+------+---------+--------------------------+
| 1 | SIMPLE | test_no | index | Uniq_idx_varchar_num | Uniq_idx_varchar_num | 63 | NULL | 3126240 | Using where; Using index |
+----+-------------+------------------------+-------+-------------------+-------------------+---------+------+---------+--------------------------+
1 row in set (0.00 sec)
mysql> EXPLAIN SELECT COUNT(1) FROM test_no WHERE varchar_num='0000194701461220130201115347';
+----+-------------+------------------------+-------+-------------------+-------------------+---------+-------+------+-------------+
| id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra |
+----+-------------+------------------------+-------+-------------------+-------------------+---------+-------+------+-------------+
| 1 | SIMPLE | test_no | const | Uniq_idx_varchar_num | Uniq_idx_varchar_num | 63 | const | 1 | Using index |
+----+-------------+------------------------+-------+-------------------+-------------------+---------+-------+------+-------------+
1 row in set (0.00 sec)
mysql>
mysql>
mysql> SELECT COUNT(1) FROM test_no WHERE varchar_num=0000194701461220130201115347;
+----------+
| COUNT(1) |
+----------+
| 1 |
+----------+
1 row in set, 1 warning (7.94 sec)
mysql> SELECT COUNT(1) FROM test_no WHERE varchar_num='0000194701461220130201115347';
+----------+
| COUNT(1) |
+----------+
| 1 |
+----------+
1 row in set (0.00 sec)
AFAIK it is standard, but it is considered bad practice because
- using it in a WHERE clause will prevent the optimizer from using indices (explain plan should show that)
- the database has to do additional work to convert the string to a number
- if you're using this for floating-point numbers ('9.4'), you'll run into trouble if client and server use different language settings (9.4 vs 9,4)
In short: don't do it (but YMMV)
This is not standard behavior.
For MySQL 5.5. this is the default SQL Mode
mysql> select ##sql_mode;
+------------+
| ##sql_mode |
+------------+
| |
+------------+
1 row in set (0.00 sec)
ANSI and TRADITIONAL are used more rigorously by Oracle and PostgreSQL. The SQL Modes MySQL permits must be set IF AND ONLY IF you want to make the SQL more ANSI-compliant. Otherwise, you don't have to touch a thing. I've never done so.
It depends on the column type!
if you run
SELECT * FROM `users` WHERE `username` = 0;
in mysql/maria-db you will get all the records where username IS NOT NULL.
Always quote values if the column is of type string (char, varchar,...) otherwise you'll get unexpected results!
You don't need to quote the numbers but it is always a good habit if you do as it is consistent.
The issue is, let's say that we have a table called users, which has a column called current_balance of type FLOAT, if you run this query:
UPDATE `users` SET `current_balance`='231608.09' WHERE `user_id`=9;
The current_balance field will be updated to 231608, because MySQL made a rounding, similarly if you try this query:
UPDATE `users` SET `current_balance`='231608.55' WHERE `user_id`=9;
The current_balance field will be updated to 231609

mysql, prepared statements, and automatic type conversion

I am getting different results performing the exact same query using regular statements and prepared statements, and I think it's a type conversion bug.
mysql> show columns from server where field = "vlan";
+-------------+--------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+-------------+--------+------+-----+---------+-------+
| vlan | int(5) | YES | MUL | NULL | |
+-------------+--------+------+-----+---------+-------+
mysql> select hostname from server where `vlan` = '184.182' limit 1;
Empty set (0.00 sec)
mysql> prepare stupid from "select hostname from server where `vlan` = ? limit 1";
Query OK, 0 rows affected (0.00 sec)
Statement prepared
mysql> set #vlan = '184.182';
Query OK, 0 rows affected (0.00 sec)
mysql> execute stupid using #vlan;
+-------------------+
| hostname |
+-------------------+
| web20.servers.com |
+-------------------+
1 row in set (0.00 sec)
the real value of vlan is 184
it looks like the way mysql is handling type conversions is different for prepared statements and regular statements? does that make sense? how do i fix this?
The expected data type of prepared statement parameters is determined upon statement preparation, and type conversion to that data type takes place prior to statement execution.
In your example, an integer parameter is expected; therefore the provided string is cast to an integer (184) before the statement is executed, and the comparison between the integer column vlan and the parameter is successful for the matching record.
The "regular" statement, by contrast, compares the integer column with a string; therefore the arguments are compared as floating point numbers, and no record has a matching vlan.
To avoid this situation, ensure that the data type cannot be determined upon preparation (or that the determined data type does not lose any information) - for example:
prepare not_so_stupid from
"select hostname from server where `vlan` = CAST(? AS CHAR) limit 1"
;