Trying to read 32-bit unsigned integer in Tcl using binary scan - tcl

Question

I'm sure I'm doing something very stupid but can't seem to get this simple item figured out. I'm using the native-messaging API in a browser extension to pass a JSON string to a Tcl script. I have this working in C using uint32_t but can't get the 32 bit unsigned integer that is prefixed to the message read in Tcl. I know that it is reading something because I wrote the $len to file and I'm pretty sure it sits and waits for the procedure coread to read more bytes than there are in the message. I've tried Iu and iu in addition to nu in the binary scan command. Would you please tell me what I'm doing wrong?
Should it matter if the messages include multi-byte characters such as Hebrew and Greek?
Added later: I'm pretty sure it's the multi-byte characters because the code works until they are included. But why would that be? Do not the browser and Tcl both count the length of the message in bytes? Whatever is happening, the length determined in Tcl exceeds the length that Tcl reads.
Thank you.
Doc. reads:
Each message is serialized using JSON, UTF-8 encoded and is preceded
with an unsigned 32-bit value containing the message length in native
byte order.
In C the form for listening for incoming message on stdin is:
int listen( void ) {
uint32_t msg_len;
while ( fread( &msg_len, sizeof msg_len, 1, stdin ) == 1 ) {
char *buf = malloc( msg_len );
if ( !buf ) {
}
else if ( fread( buf, sizeof *buf, msg_len, stdin ) != msg_len ) {
}
else {
// Read the full messsage.
}
fflush( stdout );
free( buf );
}
return 0;
}
In Tcl I'm been trying:
proc coread {reqBytes} {
set ::extMsg {}
set remBytes $reqBytes
while {![chan eof stdin]} {
yield
append ::extMsg [set data [read stdin $remBytes]]
set remBytes [expr {$remBytes - [string length $data]}]
if {$remBytes == 0} {
return $reqBytes
}
}
throw {COREAD EOF} "Unexpected EOF"
}
proc ExtRead {} {
chan event stdin readable coro_stdin
while {1} {
if { [coread 4] != 4 || [binary scan $::extMsg nu len] != 1 } {
exit
}
set ::extMsg {}
if { [coread $len] != $len } {
exit
}
# Read full message.
}
}
proc Listen {} {
# Listening on stdin
set ::forever 1
coroutine coro_stdin ExtRead
vwait forever
}
set extMsg {}
Listen

Answer 1

You nearly figured it out. The problem is almost certainly due to multi-byte characters. With the default stdin encoding setting, Tcl counts each character as 1, while the length you got is probably indicating the bytes, not the characters. If the JSON string includes crlf line endings, that would be another cause for discrepancies.
The solution is to configure stdin to binary and then apply the utf-8 decoding later:
fconfigure stdin -translation binary
coread 4
binary scan $::extMsg nu len
coread $len
puts [encoding convertfrom utf-8 $::extMsg]
Not using binary translation could also mess up your length. For example: If the length of the message is 269, it would come out as 266. This is because the \r gets turned into a \n, changing 0x10d into 0x10a.