How to Convert Hex String to Unicode character byte string in Tcl - tcl

Question

Suppose I have a hex string like 905082.
Now I want convert it it corresponding unicode character in Tcl. I have used following code:
set charstring ""
set hexstring 905082
set len [string length $hexstring]
for {set i 0} {$i < [expr $len / 2]} {incr i} {
set j [expr 2 * $i]
set char [string range $hexstring $j [expr $j + 1]]
set charstring $charstring[format %c [format %i 0x$char]]
}
puts $charstring
but it is not working... Probably it contains such hex value which represents non-printable unicode character. So how can I do this so that I can convert the hex string to unicode byte string so that I can pass it to C function using SWIG?

Answer 1

binary format formatString ?arg arg ...? (link) is your friend.
The binary format command generates a binary string whose layout is specified by the formatString >and whose contents come from the additional arguments. The resulting binary value is returned.
In your case you would be feeding hex characters to the binary format function, so your formatString would be h* or H*, depending on whether your MSB is the rightmost character (H*) or your leftmost character (h*)
binary format H* 905082 will return three bytes containing the raw data 0x905082 even if the string representation may not always be printable.
Store the command output in a variable and you're good to go.
PS: an alternative would be to retrieve the integer value of the string you want to parse with scan (link).
scan $hexstring %x intvalue and the integer value would be stored in $intvalue

Answer 2

U+905082 isn't a Unicode character. The Unicode spec explicitly states that the limit is U+10FFFF. It's also not a UTF-8 encoding of a single character (since the byte 0x50 always encodes a single character in UTF-8 by itself, a P). So whatever you're after, it's not a single character.
So what might we have open to us? Well, we can convert to a sequence of bytes:
set bytes [binary format H* "905082"]
Well, that's it! We're now in Unicode! The bytes are now converted the same sequence of characters as you'd get with "\u0090\u0050\u0082". But maybe you wanted them in a specific encoding? Well, that's where you might use encoding convertto to change to a different sequence of bytes;
set bytesTurnedToUtf8 [encoding convertto utf-8 $bytes]
If we knew that they were in some other encoding, we could use encoding convertfrom to do the reverse operation. Mind you, most of the time you don't need to spend a lot of time thinking about encodings in Tcl; the Tcl runtime manages string encodings for you and you hardly ever have to think about things other than as sequences of characters.