As has been requested in "ipv6-capable inet_aton and inet_ntoa functions needed", there is currently no MySQL function for storing IPv6 addresses. What would be the recommended data type/function for storing/inserting? (I don't intend to store them as a string). I also don't want to separate the IPv6 address into 2 INT's.
How about:
BINARY(16)
That should be effective enough.
Currently there is no function to convert textual IPv6 addresses from/to binary in the MySQL server, as noted in that bug report. You either need to do it in your application or possibly make a UDF (User-Defined Function) in the MySQL server to do that.
UPDATE:
MySQL 5.6.3 has support for IPv6 addresses, see the following: "INET6_ATON(expr)".
The data type is VARBINARY(16) instead of BINARY(16) as I suggested earlier. The only reason for this is that the MySQL functions work for both IPv6 and IPv4 addresses. BINARY(16) is fine for storing only IPv6 addresses and saves one byte. VARBINARY(16) should be used when handling both IPv6 and IPv4 addresses.
An implementation for older versions of MySQL and MariaDB, see the following: "EXTENDING MYSQL 5 WITH IPV6 FUNCTIONS".
No one has posted a full working answer (and lots of examples use the Windows ::1 which can be very misleading for live (or "production") environments) any where (at least that I can find) so here is:
The format to store with.
Example INSERT query using a reasonably complex IPv6 IP address.
Example SELECT query that you will be able to echo the IPv6 IP address back to the client.
Troubleshooting to ensure you haven't missed any legacy code.
I changed all the column names to ipv6 to reflect that they properly support IPv6 (and that allows you to keep the old column ip intact). It is possible to store the ip column in the ipv6 column and then just DROP the ip column once you're certain the conversion has worked; when I actually have time I'll add that to this post.
IPv6 Data Type
As has been mentioned VARBINARY 16 is the desirable way to go until AMD blesses us with 128 bit CPUs and the databases are updated to support 128 bit integers (did I say that correctly?). IPv6 is 128 bit, not 64 bit.
CREATE TABLE `example`
(
`id` INT(11) NOT NULL AUTO_INCREMENT,
`ipv6` VARBINARY(16) NOT NULL,
PRIMARY KEY (`id`)
)
COLLATE='utf8mb4_unicode_520_ci'
ENGINE=InnoDB;
IPv6 INSERT Query
We obviously need to store an IPv6 IP address before we can SELECT it; here is the PHP / SQL code:
$ipv6 = mysqli_real_escape_string($db,'FE80:0000:0000:0000:0202:B3FF:FE1E:8329');
$query1 = "INSERT INTO example (ipv6) VALUES (INET6_ATON('$ipv6'));";
IPv6 SELECT Query
The PHP / SQL code:
$ipv6 = mysqli_real_escape_string($db,'FE80:0000:0000:0000:0202:B3FF:FE1E:8329');
$query2 = "SELECT INET6_NTOA(ipv6) AS ipv6 FROM example WHERE ipv6=INET6_ATON('$ipv6');";
This will return fe80::202:b3ff:fe1e:8329; no, not the full IPv6 (which is FE80:0000:0000:0000:0202:B3FF:FE1E:8329), it's a condensed / shorthand version. There is code to make it the formal full-length version but this is to save myself and others time because this Q/A is the one that keeps coming up.
Important: just because some IPv6 addresses look like they'd fit in to bigint does not imply two minutes later someone with a larger IPv6 address won't stop by and wreak havoc.
Hopefully this will save some folks from the insanity of opening another two dozen tabs. When I have time in the future I'll add the extra PHP code that extends the condensed IPv6 to the full formal format.
Troubleshooting
If for some reason storing and/or retrieving IPv6 addresses is not working for you then grab yourself a copy of Advanced Find and Replace (works faster in Wine than Linux's native grep); use this predominantly for finding, not replacing. Ensure that your code is consistent everywhere in your software.
All $ip variables must be converted to $ipv6 so you know you've got that bit covered.
Do not forget to remove the ending ) for the next four steps:
Search for all instances of PHP inet_pton( functions and remove them.
Search for all instances of PHP inet_ntop( functions and remove them.
Search for all instances of SQL INET_ATON( functions and remove them.
Search for all instances of SQL INET_NTOA( functions and remove them.
Search for all instances of $ipv6 and ensure that all IP-IN-TO-SQL instances use INET6_ATON('$ipv6') and that all instances where IP-FROM-SQL use INET6_NTOA(ipv6) AS ipv6.
Search for all instances of $row1['ip'] and replace them with $row1['ipv6'].
Ensure that all instances of $ipv6 = use the following code (with your database object reference changed): $ipv6 = (isset($_SERVER['REMOTE_ADDR']) && strlen($_SERVER['REMOTE_ADDR']) > 0) ? mysqli_real_escape_string($db,$_SERVER['REMOTE_ADDR']) : mysqli_real_escape_string($db,getenv('REMOTE_ADDR'));.
Ensure that your tests use freshly tested IP addresses instead of potentially botched versions if you are aware that there was something wrong before you started debugging.
Excellent example however I noted the following.
That only works if one's version of mysql has as function for INET6_ATON.
Otherwise the error message might be something like: That FUNCTION DOES NOT EXIST.
Related
Apologies in advance for what may be a silly question, but I am working on building a little "journal" website, where users can type in daily thoughts in a private way. I'm currently storing this information in a MEDIUMTEXT datatype in a MySQL database.
My question is: is there a way to store this so that individuals (like myself) who have access to the database are not able to read the field to secure users privacy, similar to how I might hash a password?
Thanks in advance
The better way is to encrypt data before the DBMS (with whatever programming language you are using on the website), store it already encrypted in MySQL/MariaDB, then read the encrypted data back and decrypt it again inside the software outside of the DBMS. That way your website software will do the encryption/decryption and MySQL/MariaDB will just store the data.
If you need to do it in SQL, here is a simple (not very secure!) way to do this by using the AES_ENCRYPT and AES_DECRYPT functions. First I will create a simple test table
CREATE TABLE encrypt(
col MEDIUMBLOB
);
When I insert the data, I will use the encrypt function:
-- do this if you are unsure if the general log is enabled systemwide
SET sql_log_off = 'ON';
INSERT INTO encrypt(col) VALUES
(AES_ENCRYPT("blah","mysecretpassword"));
To read the data back, you must use AES_DECRYPT:
SELECT CONVERT(AES_DECRYPT(col, "mysecretpassword") USING utf8) AS col
FROM encrypt;
+------+
| col |
+------+
| blah |
+------+
-- now you can turn the log back on if you need it
-- and it is not disabled globally in my.cnf
SET sql_log_off = 'OFF';
Supplying wrong password will give a NULL value. Note that while it will probably work flawless, MEDIUMTEXT is not the best datatype - when you store the encrypted data it is all binary so BLOB is better (it does not care for collations, encodings, etc).
Why it is not recommended? First you should be careful for log files (like general log or log-slow-queries). If they are enabled, the DBMS may log your secret key in plaintext and it will be easy to recover for people who have administrative access to the machine. So if you are going to use that way, you must definitely disable logging! In the example above I showed how you can use the sql_log_off variable which will disable the plaintext general query log only for the current session (it will not disable it serverwide that way so other queries will be logged).
But that's not the whole story about logging - there is also a binary log for transactions. If enabled, it will log changes to data (like all INSERT, UPDATE and DELETE statements in particular). There is mysqlbinlog utility using which people with admin rights on the DBMS machine will be able to recover data with it... and they may eventually recover your secret key from the INSERT statement above as well. If you want to prevent the binlog too, you must do this before executing the INSERT statement:
SET sql_log_bin = 'OFF';
And of course enable it back on after it. Note that it cannot happen inside a transaction - the bin log is needed for transaction management. Also this makes your database a bit crash unsafe. If the system crashes in the middle of the insert statement when your bin log is disabled, it may corrupt the data in the table. So final conclusion here - it is definitely better to do encryption/decryption in the application and do not do it in the DB. It will save you a lot of hassle with the logging issue.
Second the example above is not secure encryption because (by default) it is using the unsecure ECB mode. Briefly - all data is separated in blocks and each block is being encrypted the same way with the same key. That way equal plaintext blocks will result in the same encrypted blocks - this may leak patterns. Therefore it is better to use some block-chaining mode with initialization vector - it's a much stronger encryption. Unfortunately if you are using MariaDB, you should stop here as it does not support anything other than ECB yet (they are working to add other modes in future). If you are using MySQL, continue reading to improve the sql solution...
With MySQL you should consider switching the default encryption mode and start using the third IV (initialization vector) parameter of the encrypt/decrypt functions which must be 16 bytes (it should be a random value which is not a secret and it can be stored directly in the DB). Here is how:
First change the block_encryption_mode system variable to something different than ECB (check here). You can use CBC for example.
Then change the above queries like this:
CREATE TABLE encrypt(
col MEDIUMBLOB,
iv BINARY(16)
);
-- Run this each time you encrypt/decrypt
-- if you cannot guarantee that it is
-- set properly in my.cnf
SET block_encryption_mode="aes-256-cbc";
SET sql_log_off = 'ON';
INSERT INTO encrypt(iv, col)
VALUES(RANDOM_BYTES(16), AES_ENCRYPT("blah", "mysecretpassword", iv));
SELECT CONVERT(AES_DECRYPT(col, "mysecretpassword", iv) USING utf8) AS col
FROM encrypt;
SET sql_log_off = 'OFF';
I have one table where I store ip address. For faster access I store ip int also. This is the table structure.
CREATE TABLE `ipv6_test` (
`ip_string` varchar(20) DEFAULT NULL,
`ip_int` double DEFAULT null
) ;
I have used following method to get integer version of ip address.
new BigInteger(1, com.google.common.net.InetAddresses.forString(ip).getAddress());
Everything was working fine for ipv4 addresses, but my code started failing while storing ipv6 addresses.
How to store BigInteger in mysql database?
I'm thinking to use VARBINARY. Can I create index on VARBINARY column?
How is the VARBINARY's performance compared to index on integer?
INET6_ATON converts to a varbinary(16) for IPv4 + IPv6 storage.
Yes you can use VARBINARY as indexes. Its effectively a VARCHAR without character encoding. Its probably a bit slower than integers but ints can't do the same job. You probably won't notice it so don't prematurely optimize. Its only 16 characters maximum so its not much.
If you want to keep an text version for easy retrieval create a generated column converting the varbinary back.
I have a database where I store MAC addresses. There is quite some discussion about how to store a MAC address in a MySQL database, I decided to store it as an unsigned BIGINT to save a few bytes per row and achieve faster searches.
This would work well, except that when I retrieve an entity from the table the mac address in the entity is limited to 2147483647, e.i. the PHP MAX_INT value for a 32 bit system. So the mac address will not be correct if it exceeds 2147483647 (which will happen for 99% of addresses).
It seems that somewhere in the process from database to entity the value is parsed as int, which breaks the values.
Is there some way of avoiding this? Would it be possible to force the entity property to be a string instead of int?
Similar question that didn't receive an answer: Cakephp 3 Bigint issue - Not receiving the same contact number
It seems the PHP deployment on my Windows 10 laptop runs in 32 bit. A solution would trying to make my PHP + Apache (XAMPP) run in 64 bit, but I would rather have a general solution.
The conversion happens in the type layer (\Cake\Database\Type), it's triggered by the query object when it fetches rows.
By default big integers are mapped to the \Cake\Database\Type\IntegerType. If you need to run this on a 32-bit system, then you could for example remap the biginteger type to \Cake\Database\Type\StringType. In your config/bootstrap.php:
Type::map('biginteger', \Cake\Database\Type\StringType::class);
See also
Cookbook > Database Access & ORM > Database Basics > Adding Custom Types
I have the following two columns in a MySQL database table named "ip_test":
ipv4 as INT(11) (example value for 127.0.0.1: 2130706433).
ipv6 as VARBINARY (example value for 127.0.0.1: 0xFE800000000000000202B3FFFE1E8329).
What is the simplest manner to effectively convert the data (for a visual demonstration of the idea: UPDATE ip_test SET ipv6=ipv4;) though properly?
My goal was to avoid any server side scripting and to achieve this purely in MySQL. This seems correct from my initial tests:
UPDATE ip_test SET ipv6 = INET6_ATON(INET_NTOA(ipv4));
I am storing IPv6 addresses as a varbinary with PHP function inet_pton($_SERVER['REMOTE_ADDR']). When I run an SQL query such as:
SELECT ip_address FROM some_table WHERE id='some_id';
I wouldn't be able to read the ip_address because it is in a binary format.
I notice that there is a corresponding MySQL function INET6_NTOA(expr) in MySQL version 5.6 to revert it back to readable format. Unfortunately, I am using MySQL version 5.5, which doesn't have that function.
Is there any other way I can read the IPv6 addresses without going back to PHP to do the conversion? I can easily read off the hexadecimal notation of the binary string with the editor in the Workbench as shown in the image attached, so I thought there should be an easy way to do this.
You can write user-defined functions to do the conversion for you and call them in the select clause. See this link for more info.