I am storing IPv6 addresses as a varbinary with PHP function inet_pton($_SERVER['REMOTE_ADDR']). When I run an SQL query such as:
SELECT ip_address FROM some_table WHERE id='some_id';
I wouldn't be able to read the ip_address because it is in a binary format.
I notice that there is a corresponding MySQL function INET6_NTOA(expr) in MySQL version 5.6 to revert it back to readable format. Unfortunately, I am using MySQL version 5.5, which doesn't have that function.
Is there any other way I can read the IPv6 addresses without going back to PHP to do the conversion? I can easily read off the hexadecimal notation of the binary string with the editor in the Workbench as shown in the image attached, so I thought there should be an easy way to do this.
You can write user-defined functions to do the conversion for you and call them in the select clause. See this link for more info.
Related
I am migrating a database from PostgresSql to MySql.
We were saving files in the database as PostgreSQL bytea columns. I wrote a script to export the bytea data and then insert the data into a new MySql database as a blob. The data is inserting into Mysql fine, but it is not working at the application level. However, the application should not care, as the data is exactly the same. I am not sure what is wrong, but I feel like it is some difference between MySql vs. PostgreSQL. Any help would be greatly appreciated.
This could really be a number of issues, but I can provide some tips in regards to converting binary data between sql vendors.
The first thing you need to be aware of is that each sql database vendor uses different escape characters. I suspect that your binary data export is using hex and you most likely have unwanted escape characters when you import to your new database.
I recently had to do this. The exported binary data was in hex and vendor specific escape characters were included.
In your new database, check if the text value of the binary data starts with an 'x' or unusual encoding. If it does you need to get rid of this. Since you already have the data inserting properly, to test, you can just write an sql script to remove any unwanted vendor specific escape characters from each imported binary data record in your new database. Finally, you may need to unhex each each new record.
So, something like this worked for me:
UPDATE my_example_table
SET my_blob_column = UNHEX(SUBSTRING(my_blob_column, 2, CHAR_LENGTH(my_blob_column)))
Note: The 2 in the SUBSTRING function is because the export script
was using hex and prepending '\x' as a vendor specific escape character.
I am not sure that will work for you, but it maybe worth a try.
I am migrating a database with geometric data from MySQL to PostgreSQL. I've run into a snag because the POINT datatypes are incompatible between the two databases.
In MySQL, the POINT values are single hex strings like 0xE21D4B40. However, I cannot insert these into PostgreSQL because PostgreSQL's POINT datatype expects coordinates in the form of (x,y). What is the best way to handle this conversion?
#tadman's comment ended up being my solution: unpacking the hex string to a pair of values on the MySQL side. In my case the hex string was a column called location in the city table, so I used the query SELECT ST_AsText(location) FROM city;.
This returned rows like POINT(-133.03531 54.23346) which PostgreSQL was happy to accept.
I am building a reporting database and get data in CSV as well as from MS SQL Server. These are mostly personal records, tied together by SSN (well, not really SSN but something very similar). For security reasons, we are not storing the SSN, but rather the SHA2-256 hash of the SSN.
On MySql 5.6, I can simply use the built-in function:
sha2( string, 256).
For MS SQL Express (SQL Server 2008 running on Windows 7) I used this link (http://geekswithblogs.net/hroggero/archive/2009/09/19/strong-password-hashing-with-sql-server.aspx) to write an external UDF.
Each function should return the same string for the same argument .. and it does not. Even more puzzling is that the binary representation seems to be quite different:
MySql Output: bfd6b995588ec54ce16871bc82a7ac86dd43a2c22309ea68e479a50043683937
MSSQL Output: 0x1B4F27012B7F6E7DA6563376E3CB560FCB07FDE2E33C6C3241A5D53885ABCF71
The MSSQL output is clearly a hex encoding (0-9, A-F) of the low and high 4 bit portions of each byte. But how does MySql represent the binary characters?
Ran the MySql queries both through SqlYog and through the DOS command-line ... no difference.
I have checked the MySql documentation and everything else I could find on the web, no luck.
Actually, MySql returns the same hex codes as MSSQL, just in lowercase. I did correct my C# code and now both functions return the same string. Hurrayyyyyyy
I want to store IP addresses (v4 and v6) in my rails application. I have installed an extension to MySQL adds functions to convert ip strings to binary which will allow me to query by IP range easily.
I can use unescaped sql statements for SELECT type queries, thats easy.
The hard part is that I also need a way to overwrite the way the field is escaped for insert/update statements.
This ActiveRecord statement
new_ip = Ip.new
new_ip.start = '1.2.3.4'
new_ip.save
Should generate the following SQL statement
INSERT INTO ips(start) VALUES(inet6_pton('1.2.3.4'));
Is there a way to do this? I tried many things, including overriding ActiveRecord::Base#arel_attributes_values, without luck : the generated sql is always converted to binary (if that matters, my column is a MySQL VARBINARY(16)).
ActiveRecord::Base.connection.execute("INSERT INTO ips(start) VALUES(inet6_pton('1.2.3.4'))")
As has been requested in "ipv6-capable inet_aton and inet_ntoa functions needed", there is currently no MySQL function for storing IPv6 addresses. What would be the recommended data type/function for storing/inserting? (I don't intend to store them as a string). I also don't want to separate the IPv6 address into 2 INT's.
How about:
BINARY(16)
That should be effective enough.
Currently there is no function to convert textual IPv6 addresses from/to binary in the MySQL server, as noted in that bug report. You either need to do it in your application or possibly make a UDF (User-Defined Function) in the MySQL server to do that.
UPDATE:
MySQL 5.6.3 has support for IPv6 addresses, see the following: "INET6_ATON(expr)".
The data type is VARBINARY(16) instead of BINARY(16) as I suggested earlier. The only reason for this is that the MySQL functions work for both IPv6 and IPv4 addresses. BINARY(16) is fine for storing only IPv6 addresses and saves one byte. VARBINARY(16) should be used when handling both IPv6 and IPv4 addresses.
An implementation for older versions of MySQL and MariaDB, see the following: "EXTENDING MYSQL 5 WITH IPV6 FUNCTIONS".
No one has posted a full working answer (and lots of examples use the Windows ::1 which can be very misleading for live (or "production") environments) any where (at least that I can find) so here is:
The format to store with.
Example INSERT query using a reasonably complex IPv6 IP address.
Example SELECT query that you will be able to echo the IPv6 IP address back to the client.
Troubleshooting to ensure you haven't missed any legacy code.
I changed all the column names to ipv6 to reflect that they properly support IPv6 (and that allows you to keep the old column ip intact). It is possible to store the ip column in the ipv6 column and then just DROP the ip column once you're certain the conversion has worked; when I actually have time I'll add that to this post.
IPv6 Data Type
As has been mentioned VARBINARY 16 is the desirable way to go until AMD blesses us with 128 bit CPUs and the databases are updated to support 128 bit integers (did I say that correctly?). IPv6 is 128 bit, not 64 bit.
CREATE TABLE `example`
(
`id` INT(11) NOT NULL AUTO_INCREMENT,
`ipv6` VARBINARY(16) NOT NULL,
PRIMARY KEY (`id`)
)
COLLATE='utf8mb4_unicode_520_ci'
ENGINE=InnoDB;
IPv6 INSERT Query
We obviously need to store an IPv6 IP address before we can SELECT it; here is the PHP / SQL code:
$ipv6 = mysqli_real_escape_string($db,'FE80:0000:0000:0000:0202:B3FF:FE1E:8329');
$query1 = "INSERT INTO example (ipv6) VALUES (INET6_ATON('$ipv6'));";
IPv6 SELECT Query
The PHP / SQL code:
$ipv6 = mysqli_real_escape_string($db,'FE80:0000:0000:0000:0202:B3FF:FE1E:8329');
$query2 = "SELECT INET6_NTOA(ipv6) AS ipv6 FROM example WHERE ipv6=INET6_ATON('$ipv6');";
This will return fe80::202:b3ff:fe1e:8329; no, not the full IPv6 (which is FE80:0000:0000:0000:0202:B3FF:FE1E:8329), it's a condensed / shorthand version. There is code to make it the formal full-length version but this is to save myself and others time because this Q/A is the one that keeps coming up.
Important: just because some IPv6 addresses look like they'd fit in to bigint does not imply two minutes later someone with a larger IPv6 address won't stop by and wreak havoc.
Hopefully this will save some folks from the insanity of opening another two dozen tabs. When I have time in the future I'll add the extra PHP code that extends the condensed IPv6 to the full formal format.
Troubleshooting
If for some reason storing and/or retrieving IPv6 addresses is not working for you then grab yourself a copy of Advanced Find and Replace (works faster in Wine than Linux's native grep); use this predominantly for finding, not replacing. Ensure that your code is consistent everywhere in your software.
All $ip variables must be converted to $ipv6 so you know you've got that bit covered.
Do not forget to remove the ending ) for the next four steps:
Search for all instances of PHP inet_pton( functions and remove them.
Search for all instances of PHP inet_ntop( functions and remove them.
Search for all instances of SQL INET_ATON( functions and remove them.
Search for all instances of SQL INET_NTOA( functions and remove them.
Search for all instances of $ipv6 and ensure that all IP-IN-TO-SQL instances use INET6_ATON('$ipv6') and that all instances where IP-FROM-SQL use INET6_NTOA(ipv6) AS ipv6.
Search for all instances of $row1['ip'] and replace them with $row1['ipv6'].
Ensure that all instances of $ipv6 = use the following code (with your database object reference changed): $ipv6 = (isset($_SERVER['REMOTE_ADDR']) && strlen($_SERVER['REMOTE_ADDR']) > 0) ? mysqli_real_escape_string($db,$_SERVER['REMOTE_ADDR']) : mysqli_real_escape_string($db,getenv('REMOTE_ADDR'));.
Ensure that your tests use freshly tested IP addresses instead of potentially botched versions if you are aware that there was something wrong before you started debugging.
Excellent example however I noted the following.
That only works if one's version of mysql has as function for INET6_ATON.
Otherwise the error message might be something like: That FUNCTION DOES NOT EXIST.