Does the mysql CLI tool provide a way to display binary data in a console-friendly manner? - mysql

I have a MySQL database containing a table with a binary-typed column. I'd like to be able to project that column without having to run it through, e.g., HEX(). Does the mysql CLI tool have a configuration option or other means to display a representation of binary data in a manner that won't output arbitrary bytes for my console to interpret in hilarious/annoying ways?

Start MySQL CLI with param --binary-as-hex
Example:
mysql --binary-as-hex

Set mysql client options in /etc/my.cnf works for me:
[client]
binary-as-hex = true
[mysql]
binary-as-hex = true

Since you want to look at the table mostly for convenience, create a view:
CREATE OR REPLACE VIEW myview AS
SELECT col1, HEX(col2) AS col2, col3, etc....
FROM table;
Then, all you have to do is reference myview instead of table:
SELECT * FROM myview;

The behavior of the MySQL command line client when viewing result sets with binary data has always been an annoyance to me, in fact I found this page because I was once again annoyed by the MySQL command line client (dumping binary data into my terminal when looking at a result set with binary UUID columns) and I wanted to solve the issue once and for all :-)
Creating views really isn't an option for me (I'm looking at dozens of tables with binary UUID columns) and I also found that it's really annoying to switch from SELECT * to typing out all of the column names instead (just so HEX() can be applied to the value of one column).
Eventually I came up with a creative hack that provides inspiration for alternative solutions to this annoyance: Using a custom pager command to sanitize output for terminal rendering. Here's how it works:
Create an executable (chmod +x) Python script with the following contents:
#!/usr/bin/python
import binascii, string, sys
for line in sys.stdin:
line = line.rstrip()
column, _, value = line.partition(': ')
if any(c not in string.printable for c in value):
sys.stdout.write("%s: %s\n" % (column, binascii.hexlify(value)))
else:
sys.stdout.write("%s\n" % line)
Start the MySQL command line client as follows:
$ mysql --pager=/home/peter/binary-filter.py --vertical ...
Change the pathname of the Python script as applicable. You can also put the script in your $PATH, in that case you can just pass the name to the --pager option (similar to how you would use less as a pager for the MySQL client).
Now when you SELECT ..., any line that shows a column whose value contains non-printable characters is rewritten so that the complete value is rendered as hexadecimal characters, similar to the results of MySQL's HEX() function.
Disclaimer: This is far from a complete solution, for example the Python snippet I showed expects SELECT ... \G format output (hence the --vertical option) and I tested it for all of five minutes so it's bound to contain bugs.
My point was to show that the problem can be solved on the side of the MySQL command line client, because that's where the problem is! (this is why it feels backwards for me to define server side views - only to make a command line client more user friendly :-P)

For me there is no problem with database size, so I will use two different column in every table, one as binary(16), and the second as char(32) without indexing. both of them will have the same value.
when I need to search I will use binary column, and when I need to read I will use char(32).
is there any problem with this scenario?

Related

BCP CHAR value to Snowflake

I am trying to create a BCP file with | delimiter and then load it to a snowflake table.
Issue:
in SQL server there are columns defined as CHAR(4) and have values "sss"
so when i do BCP the its being padded to length of 4 "sss " and being loaded to snowflake
due to which our reports are failing because they do something like where column="SSS" but due to trailing space in snowflake the correct columns are not showing up.
we do not want to change our reports. So, is there a way that BCP can handle the padding or trimming of these columns?
note that there 24 tables and each have around 130+ columns so i cant go and put Trim functions on each char column
If your BCP file is maintaining the trailing space, then Snowflake will retain it, too, as long as the field is being FIELD_OPTIONALLY_ENCLOSED_BY a " or '. You may also want to make sure your TRIM_SPACE option is correctly set on your format definition for your COPY INTO command.
If your BCP file isn't maintaining the space and you can't figure out how to get that to work, you could force the space back in during the COPY INTO command with some string functions in your SELECT, or you could create a view for your report that does the same set of string functions to force the space for your report to work from.
So, is there a way that BCP can handle the padding or trimming of these columns?
Yes, but not by some switch or option. The correct way to handle this is to set your datatypes up front. As someone mentioned in comments to your question, your query that is creating BCP output should use VARCHAR(4) instead of CHAR(4). BCP is giving you what you asked of it. They way to avoid whitespace is to use varchar.
Seems like a fairly quick "find and replace" against scripted out query objects would work fine but you know your situation best.
Additionally, "trim" wont work - FYI. Even if the value of the field was only "SSS" (as in your example); if the result/column is defined as CHAR(4) you will get 4 bytes of data and a blank in the 4th place since you only had 3 bytes of data. Trim will work during the query... the padded " " you are getting is placed there by the copy out. The way to correct this is to set your data types as you need up front.
Unless someone knows of a better way in snowflake (im not familiar with it) the only other option is to manipulate the file inbetween SQL and Snowflake. replace " |" with "|"... but... blech.
This is a known "issue" with BCP. The "solution" is to use the queryout option, which means you must include a query with every export. But the data are the way they are.
Eg: https://social.msdn.microsoft.com/Forums/sqlserver/en-US/88c258fe-d1a6-4f3a-9dac-40388d04e9c7/remove-space-in-columns-on-bcp-out?forum=transactsql
But this is really a Snowflake problem, because Snowflake has its own default CHAR semantics.
You get a warning in the documentation String & Binary Data Types but that doesn't tell the whole truth.
The following executed on Oracle (and apparently MSSQL? MySQL?) will select the aaa line:
CREATE TABLE C AS SELECT CAST('aaa ' AS CHAR(4)) t FROM DUAL;
SELECT * FROM C WHERE t = 'aaa';
but won't on Snowflake, unless you create the column with COLLATION:
CREATE OR REPLACE TABLE C (t CHAR(4) COLLATE 'en_US-rtrim');
INSERT INTO C VALUES('aaa ');
SELECT * FROM C WHERE t = 'aaa';
Unfortunately, you can't ALTER the collation after creation, which would have been convenient after a COPY INTO <table>.
PS: Mike Walton's answer is better, TRIM_SPACE is much cleaner than COLLATE.

MySQLAdmin replace text in a field with percent in text

Using MySQLAdmin. Moved data from Windows server and trying to replace case in urls but not finding the matches. Need slashes as I don't want to replace text in anything but the urls (in post table). I think the %20 are the problem somwhow?
UPDATE table_name SET field = replace(field, '/user%20name/', '/User%20Name/')
The actual string is more like:
https://www.example.com/forum/uploads/user%20name/GFCI%20Stds%20Rev%202006%20.pdf
In a case you are using MariaDB you have REGEXP_REPLACE() function.
But best approach is to dump the table into the file. Open it in a Notepad ++
and run regex replace like specified on a pic:
Pattern is: (https:[\/\w\s\.]+uploads/)(\w+)\%20(\w+)((\/.*)+)
Replace with: $1\u$2\%20\u$3$4
Then import the table again
Hope this help
If its MariaDB, you can do the following:
UPDATE table_name SET field = REGEXP_REPLACE(field, '\/user%20name\/', '\/User%20Name\/');
First, please check, what is actually stored in the database: %20 is a html-entity which represents a whitespace. Usually, when you are storing this inside the database, it will be represented as an actual whitespace (converted before you store it) -> Hence your replace doesn't match the actual data.
The second option that might be possible - depending on what you want to do: You are seeing the URL containing %20, therefore you created your database records (which you would like to fetch) with that additional %20 - And when you now try to query your results based on the actual url, the %20 is replaced with an "actual" whitespace (before your query) and hence it doesn't match your stored data.

MySQL is outputting binary hex strings that somehow resolve into text. How does this work?

I'm using a GUI MySQL manager called Sequel Pro on my Mac.
Historically, when I'd right-click a row and click "Copy as SQL Insert", it would copy a standard-looking MySQL insert statement containing the row contents in plain text.
I noticed the other day when doing this, the output now looks like this:
INSERT INTO `config` (`key`, `value`)
VALUES
(X'68656C6C6F5F737461636B5F6F766572666C6F77', X'5468616E6B7320666F7220616E73776572696E67206D79207175657374696F6E');
What is with the random letter/number strings beginning with X? Starting a string with an X doesn't look like standard SQL syntax. Somehow though, when I run the SQL command it enters a valid new record with the data I would expect into the table.
What feature is this, what benefit does it serve, and why did it suddenly change since last week?
It's a SQL standard to escape binary literals in the format:
x'hexstring'
Like the following:
x'68656C6C6F5F737461636B5F6F766572666C6F77'
According to a comment on the documentation for the BINARY and VARBINARY Types, MySQL also supports the ODBC standard:
0x68656C6C6F5F737461636B5F6F766572666C6F77

Getting 2 different lengths for a text field in perl from a DBI query

I have encrypted data in a mysql table stored as a text field.
Everything was originally written in Windows perl and that still works without issue.
My problem is that I am running the same code on Linux and when I query the table the text result in perl tells me it is longer (which causes my decryption to blow up since it is too long).
This happens running the same script so I know there is not a code difference.
Mysql server is 5.1.63 running on OpenSuSE Linux 11.4 x64.
Linux perl is v5.12.3
Windows perl is 5.10.1
The field in question is defined as text, utf8_general_ci and when I access it via JDBC the data reports 128 bytes,
the SQL in question is simple (pruned down to just what matters here)
my $gatherSQL = "select
table.encryptedText from action.theTable table
where table.custno=" . $dbHandle->quote($custno)
my $getHandle = $dbHandle->prepare($gatherSQL);
$getHandle->execute();
my $arrayRef = $getHandle->fetchall_arrayref();
foreach my $myrow (#$arrayRef)
{
$type = $$myrow[0];
}
$getHandle->finish();
#DB Handle is opened with a simple
my $workSQLhandle = DBI->connect("dbi:mysql:$dataDB:$dataServer:$dataPort", $userToUse, $pwToUse);
return($workSQLhandle);
When I run the code in Windows (through a samba share) I get a length of the field of 128 (which decrypts)
Same code on the same machine run from a command prompt tells me the same return string is 193 chars long (and won't decrypt)
I did a compare of the results coming back and they are identical but perl tells me one is longer than the other.
Any thoughts on how I can address this and what the root cause is?
check if perhaps mysql/perl are doing some translations on the text. e.g select length(table.encryptedText) to see what mysql thinks the length is. encrypted text tends to look like binary garbage, and if you're storing it in a TEXT-type field, it WILL be subject to automatic charset translation. encrypted data should go into BLOB fields, which are otherwise identical to TEXT, but are NOT charset-sensitive.

How can I directly view blobs in MySQL Workbench

I'm using MySQL Workbench CE 5.2.30 CE / Rev 6790 . When execute the following statement:
SELECT OLD_PASSWORD("test")
I only get back a nice BLOB icon, I need to left-click to select the cell, right-click and choose "Open Value in viewer" and select the "Text" tab.
Using the same with phpMyAdmin, I get directly back the value of the OLD_PASSWORD call. It's just an example, but is there a way to directly see such results in the output?
In short:
Go to Edit > Preferences
Choose SQL Editor
Under SQL Execution, check Treat BINARY/VARBINARY as nonbinary character string
Restart MySQL Workbench (you will not be prompted or informed of this requirement).
In MySQL Workbench 6.0+
Go to Edit > Preferences
Choose SQL Queries
Under Query Results, check Treat BINARY/VARBINARY as nonbinary character string
It's not mandatory to restart MySQL Workbench (you will not be prompted or informed of this requirement).*
With this setting you will be able to concatenate fields without getting blobs.
I think this applies to versions 5.2.22 and later and is the result of this MySQL bug.
Disclaimer: I don't know what the downside of this setting is - maybe when you are selecting BINARY/VARBINARY values you will see it as plain text which may be misleading and/or maybe it will hinder performance if they are large enough?
I'm not sure if this answers the question but if if you right click on the "blob" icon in the field (when viewing the table) there is an option to "Open Value in Editor". One of the tabs lets you view the blob. This is in ver. 5.2.34
Perform three steps:
Go to "WorkBench Preferences" --> Choose "SQL Editor" Under "Query Results": check "Treat BINARY/VARBINARY as nonbinary character string"
Restart MySQL WorkBench.
Now select SELECT SUBSTRING(<BLOB_COLUMN_NAME>,1,2500) FROM <Table_name>;
casting works, but it is a pain, so I would recommend using spioter's method unless you are using a lot of truly blob data.
SELECT CAST(OLD_PASSWORD("test") AS CHAR)
You can also cast as other types, and even restrict the size, but most of the time I just use CHAR:
http://dev.mysql.com/doc/refman/5.5/en/cast-functions.html#function_cast
select CONVERT((column_name) USING utf8) FROM table;
In my case, Workbench does not work. so i used the above solution to show blob data as text.
Doesn't seem to be possible I'm afraid, its listed as a bug in workbench:
http://bugs.mysql.com/bug.php?id=50692
It would be very useful though!
had the same problem, according to the MySQL documentation, you can select a Substring of a BLOB:
SELECT id, SUBSTRING(comment,1,2000) FROM t
HTH, glissi
I pieced a few of the other posts together, as the workbench 'preferences' fix did not work for me. (WB 6.3)
SELECT CAST(`column` AS CHAR(10000) CHARACTER SET utf8) FROM `table`;
Work bench 6.3
Follow High scoring answer then use UNCOMPRESS()
(In short:
1. Go to Edit > Preferences
2. Choose SQL Editor
3. Under SQL Execution, check Treat BINARY/VARBINARY as nonbinary character string
4. Restart MySQL Workbench (you will not be prompted or informed of this requirement).)
Then
SELECT SUBSTRING(UNCOMPRESS(<COLUMN_NAME>),1,2500) FROM <Table_name>;
or
SELECT CAST(UNCOMPRESS(<COLUMN_NAME>) AS CHAR) FROM <Table_name>;
If you just put UNCOMPRESS(<COLUMN_NAME>) you can right click blob and click "Open Value in Editor".
there is few things that you can do
SELECT GROUP_CONCAT(CAST(name AS CHAR))
FROM product
WHERE id IN (12345,12346,12347)
If you want to order by the query you can order by cast as well like below
SELECT GROUP_CONCAT(name ORDER BY name))
FROM product
WHERE id IN (12345,12346,12347)
as it says on this blog
http://www.kdecom.com/mysql-group-concat-blob-bug-solved/
NOTE: The previous answers here aren't particularly useful if the BLOB is an arbitrary sequence of bytes; e.g. BINARY(16) to store 128-bit GUID or md5 checksum.
In that case, there currently is no editor preference -- though I have submitted a feature request now -- see that request for more detailed explanation.
[Until/unless that feature request is implemented], the solution is HEX function in a query: SELECT HEX(mybinarycolumn) FROM mytable.
An alternative is to use phpMyAdmin instead of MySQL Workbench - there hex is shown by default.