Why does the channel event readable keep firing in this code? - tcl

Could you tell me what in this code would cause the chan event $sock readable [list ReadLine $sock] to repeatedly be fired after a socket has been reset?
I'm trying to keep the socket open on an extremely simple local server by resetting the socket to read again at the first line.
I'm not sure if it is the browser or the Tcl code. I flush the socket before sending each response using chan flush $sock; so, I don't think there should be any data left in the input buffer to fire the readable event.
Everything works fine until I stop using the application for a few minutes, and then procedure ReadLine is repeatedly invoked with a state of 1 but no data.
I included procedure GetLexi at the end because its the one I've been testing when this started occurring and, perhaps, I'm doing something wrong there and the browser doesn't know the response is complete. I should add also that it is called from with procedure GetHandler and should return back to the switch block and be reset. I did test that the socket is reset after an invocation of GetLexi.
Thank you for any guidance you may be able to provide.
proc ResetSock {sock} {
global state
set state($sock) {1}
chan configure $sock -encoding iso8859-1 -translation crlf
}; #close ResetSock
proc ClientConnect {sock client_ip client_port} {
global state
if {![info exists state($sock)]} {
set state($sock) {1}; # 1 for first line; 2 for any other header line.
chan configure $sock -buffering line -blocking 0 -encoding iso8859-1 -translation crlf
}
chan event $sock readable [list ReadLine $sock]
}; #close ClientConnect
proc ReadLine {sock} {
global state
set sptr state($sock)
set row [lindex [set $sptr] 0]
if {[catch {gets $sock line} len]} {
# Handle this error.
return
}
if {$len == 0} {
#According to Tclhttpd, in between requests, a row 1 and len 0
#combination can occur. There, it is ignored.
if {$row == 2 } {
switch [lindex [set $sptr] 1] {
"POST" {
set post [PostHandler $sock [lindex [set $sptr] 3]]
puts stdout "Posted: $post"
ResetSock $sock
}
"GET" {
GetHandler $sock [lindex [set $sptr] 2]
ResetSock $sock
}
default { CloseSock $sock }
}
}
} elseif {$len > 0} {
switch $row {
1 {
# First row of request.
lassign [split [string trim $line]] op arg rest
lappend $sptr $op $arg
lset $sptr 0 0 2
}
2 {
# Read headers.
}
default { }
}
} else {
# Only other option for $len is a negative value;
# thus, an error to be handled.
}
}; #close ReadLine
proc GetLexi { nbr sock } {
chan flush $sock
set sql { select img_base64 from lexi_raw where strongs_no = $nbr }
dbws eval $sql {
set lexi [binary format a* "{\"lexi\":\"$img_base64\"}"]
}
set headers ""
append headers "HTTP/1.1 200 OK\n"
append headers "Content-Type: application/json; charset: utf-8\n"
append headers "Content-length: [string length $lexi]\n"
append headers "Connection: Keep-Alive\n"
puts $sock $headers
chan configure $sock -translation binary
puts $sock $lexi
}; #close GetLexi
set state(x) {}
if [catch {set listener [socket -server ClientConnect -myaddr 127.0.0.1 8000]}] {
error "couldn't open listening socket"
}
vwait forever
catch {close $listener}

It's possible for the amount of data received in a packet to be not enough to complete a line. TCP hides most of the details, of course, but it remains entirely possible for the readable event to fire when a complete line is not available. As you've put the socket in non-blocking mode, that means that gets will do a zero length read (empty string written to line, len becomes 0); if the socket was blocking, the gets would block the thread until a complete line was available. You can also get zero-length reads if the channel is closed; detection of closure isn't very reliable with TCP (because networking is like that) but may happen. When the socket is closed, all reading from it in non-blocking mode results in zero-length results.
How do you distinguish these cases?
First, we check for end-of-stream:
if {[chan eof $sock]} {
# It's gone; there's not much else you can do at this point except log it I guess
close $sock
return
}
Then we need to see if things are blocked, and if so, how much is buffered:
if {[chan blocked $sock]} {
set bufferSize [chan pending input $sock]
if {$bufferSize > 4096} { # 4k is enough for most things
# Line is too long; client not well-behaved…
# You *might* send an error here.
close $sock
return
}
}
If it's neither of those cases, we've actually read a line.
if {$len == 0} {
# Empty line; end of HTTP header
} else {
# etc.
}

Related

Reading Hebrew in JSON via native-messaging API and writing to file but Hebrew looks wrong in the file?

I'm using this code to read messages received from a browser extension via the native-messaging API. Fortunately, it works and reads all the data. Unfortunately, I'm doing something wrong or failing to do something right (if there is a difference there) and when the data is written to the file, it appears that the multi-byte characters (Hebrew, at the moment) are being written as individual bytes.
Would you please tell me what needs to be done before writing [dict get $msg doc] to the file? $msg is a big string of HTML. Am I configuring stdin incorrectly or does the data being written to file need to be encoded?
Instead of seeing Hebrew, the file shows items like this ×”Ö¸×ָֽרֶץ×.
According to the MDN document the incoming message is:
On the application side, you use standard input to receive messages
and standard output to send them.
Each message is serialized using JSON, UTF-8 encoded and is preceded
with an unsigned 32-bit value containing the message length in native
byte order.
Thank you.
#!/usr/bin/env tclsh
package require json
proc coread {reqBytes} {
set ::extMsg {}
set remBytes $reqBytes
while {![chan eof stdin]} {
yield
append ::extMsg [set data [read stdin $remBytes]]
set remBytes [expr {$remBytes - [string length $data]}]
if {$remBytes == 0} {
return $reqBytes
}
}
throw {COREAD EOF} "Unexpected EOF"
}
proc ExtRead {} {
chan event stdin readable coro_stdin
while {1} {
set ::extMsg {}
chan puts $::fp_log "starting coread 4"
if { [coread 4] != 4 || [binary scan $::extMsg iu len] != 1 } {
exit
}
if { [coread $len] != $len } {
exit
}
set msg [::json::json2dict $::extMsg]
set book [dict get $msg book]
set path "../${book}.html"
set fp [open $path w]
chan puts $fp [dict get $msg doc]
close $fp
}
}
proc CloseConnect {} {
set ::forever 0
}
proc Listen {} {
# Listening on stdin
set ::forever 1
chan configure stdin -buffering full -blocking 0 -encoding iso8859-1 -translation crlf
coroutine coro_stdin ExtRead
vwait forever
}
set extMsg {}
Listen
I don't really want to answer my own question but this is too much for a comment; and I'll be happy to delete it if someone provides a better one.
As #Abra pointed out, I had the encoding configured wrong on stdin. I thought all that needed done then was to use [string bytelength $data] instead of [string length $data] but that only worked for a few iterations. I do not understand why [coread 4] was ending up reading more than 4 bytes such that $remBytes was negative. It resulted in misreading the data and the coroutine waited for far more data than there was. So, I moved the configuring of stdin into proc extRead and changed it from binary to utf-8 for each part of the read. Binary for the 4 bytes indicating the length of the message, and utf-8 for the message. And, I had to use [string length $data] for the binary read of 4 bytes and [string bytelength $data] for reading the message; and this resulted in $remBytes always equaling zero upon completion of reading both the length and the message, such that if {$remBytes == 0} works rather than needing the earlier change mentioned in my comment of if {$remBytes <= 0}.
It works and the Hebrew now appears correctly. But it bothers me that I don't understand why [string bytelength $data] would not be 4 when reading 4 bytes. I write the values out to a log/debugging file at various steps to verify this.
I should add, here, that I had a similar question in the past which #Schelte Bron answered but I don't think I followed his advice correctly or unknowingly reverted to an old version of my code. He recommended:
The solution is to configure stdin to binary and then apply the utf-8
decoding later:
I don't know why I didn't start with that but he probably provided the answer three months ago.
#!/usr/bin/env tclsh
package require json
proc coread {reqBytes readType} {
set ::extMsg {}
set remBytes $reqBytes
while {![chan eof stdin]} {
yield
append ::extMsg [set data [read stdin $remBytes]]
if { $readType } {
set remBytes [expr {$remBytes - [string bytelength $data]}]
} else {
set remBytes [expr {$remBytes - [string length $data]}]
}
if {$remBytes == 0} {
return $reqBytes
}
}
throw {COREAD EOF} "Unexpected EOF"
}
proc ExtRead {} {
chan event stdin readable coro_stdin
while {1} {
chan configure stdin -buffering full -blocking 0 -encoding binary
set ::extMsg {}
if { [coread 4 0] != 4 || [binary scan $::extMsg iu len] != 1 } {
exit
}
chan configure stdin -buffering full -blocking 0 -encoding utf-8 -translation crlf
if { [coread $len 1] != $len } {
exit
}
set msg [::json::json2dict $::extMsg]
set book [dict get $msg book]
set path "../${book}.html"
set fp [open $path w]
chan puts $fp [dict get $msg doc]
close $fp
}
}
proc CloseConnect {} {
set ::forever 0
}
proc Listen {} {
# Listening on stdin
set ::forever 1
coroutine coro_stdin ExtRead
vwait forever
}
set extMsg {}
Listen

Check abnormal connection drop before writing

I am new to TCL scripting and writing a production code to open a socket to our server and write command and then read its output. Below is my code:
set chan [socket 123.345.45.33 23]
fconfigure $chan -buffering line
foreach item [dict keys $command] {
set cmd [dict get $command $item]
set res [Data_write $chan "get data $cmd"]
}
Where Data_write procedure is mentioned below:
proc Data_write { channel data } {
if {[eof $channel]} {
close $channel
ST_puts "Data_write: Error while writing to chanel"
return -1
} else {
puts $channel $data
return 0
}
}
I am not sure that how can we achive the validations below:
set chan [socket 123.345.45.33 23] - socket connection open is success
fconfigure on the channel is success
How to know before any write that any abnormal connection drop has happen on channel?
set chan [socket 123.345.45.33 23] - socket connection open is success
fconfigure on the channel is success
These are simple enough: if there's a failure, you get a Tcl error, which is a sort of exception. You can use catch or try to trap the error if you want:
try {
set chan [socket 123.345.45.33 23]
fconfigure $chan -buffering line
} on error msg {
puts "a serious problem happened: $msg"
# Maybe also exit here...
}
How to know before any write that any abnormal connection drop has happen on channel?
The bad news is that you can't know this. The OS itself won't really know this until you do one of two things: write to the channel, or read from the channel. (There are sometimes hints available, such as fileevent events firing, but they're not certain at all.) Instead, you need to trap the error when actually you do the write. See above for the general pattern.
Remember: Tcl operations throw errors when they fail, and EOF is not an error when reading, but is an error when writing.
Use socket -async and the readable and writeable fileevents to make the whole connection process event oriented.
In the writable event you can check the status of the connection using fconfigure $channel -error. If something failed in the connection, the socket is made writable and the error condition presented on the error property. If this is empty then you can configure the readable event and start processing data from the socket. In any readable event handler you should check for eof after reading and disable the readable event handler or close the socket once eof is seen as a socket in eof state becomes constantly readable.
This roughly works out to be:
proc OnWritable {chan} {
set err [fconfigure $chan -error]
if {$err ne "" } {
puts stderr "ERROR: $err"
exit 1
}
fconfigure $chan -blocking 0 -buffering none -encoding binary -translation lf
fileevent $chan writable {}
fileevent $chan readable [list OnReadable $chan]
}
proc OnReadable {chan} {
set data [read $chan]
puts "[string length $data]: $data"
if {[eof $chan]} {
fileevent $chan readable {}
puts "closed $chan"
set ::forever "closed"
}
}
set sock [socket -async $address $port]
fileevent $sock writable [list OnWriteable $sock]
vwait ::forever

Read socket is blocked

I'm writing a socket utility to communicate a client to a server. When input to the socket from the client side, the server is receiving it fine. However, when input to the socket from the server, the client can't read. When checking for fblocked $channel. It is 1. I've tried everything including adding new line, ...
Please help.
Below is my code
proc read_command { sock } {
variable self
global connected
set len [gets $sock line]
set bl [fblocked $sock]
puts "Characters Read: $len Fblocked: $bl"
if {$len < 0} {
if {$bl} {
puts "Input is blocked"
} else {
set connected 1
puts "The socket was closed - closing my end"
close $sock
}
} else {
if {!$bl} {
puts "Read $len characters: $line"
catch {uplevel #0 $line} output
puts "1==>$output<=="
puts $sock "$output"
puts $sock "\n"
flush $sock
}
}
}
proc client { host port } {
variable self
set s [socket $host $port]
set self(csock) $s
set self($s,addr) $host
set self($s,port) $port
fconfigure $s -buffering line -blocking 0
return $s
}
proc prun { sock args} {
variable self
set result [list]
set cmd $args
set cmd [regsub -all {(^\s*\{)||(\}\s*$)} $cmd ""]
set cmd [string trimleft $cmd]
set o1 [eval $cmd]
#catch {uplevel #0 $cmd} o1
puts "1_$sock ==> $o1"
lappend result $o1
#--------------
puts $sock $cmd
flush $sock
set bl [fblocked $sock]
set file [read $sock]
set bl [fblocked $sock]
puts "Fblocked: $bl"
puts "Output: $file"
puts "2_$Comm::self(csock) ==> $file ==> $bl"
lappend result $file
return $result
}
Here is how I run it.
I call server on 1 of the terminal. It will echo the ip address and the port.
Then I call client using the address and the port above to get back the client socket
Then I call prun on the client shell to get back a pair of values, one with the value of the cmd call on the client, and the other the value of the cmd call on the server. Basically I would like to get the pair of values so I can use them for correlation between the 2 set of data.
Below is the code:
1.
On server shell
$ server
2.
On client shell
$ set s [client $addr $port]
3.
Call a proc to get the value from the client shell, then send the command to the server to get the value from the server shell, and return that value back to the client.
$ set res [prun $s {set val [get_attribute [get_nets mynet] pin_capacitance_max]}]
You wrote:
puts "2_$Comm::self(csock) ==> $file ==> $bl"
and defined self with variable. Are you working with packages?. May be you forgot something related to it.
For test you can use just global but using arrays would be a little more complicated.

Socket timeout after some time

If the server went into a infinite loop, how we can close the server connection after sometime?
Here is the code in which I am trying:
set s [socket $host $port]
fconfigure $s -blocking 1 -buffering line
after 2000 set end 1
vwait end
if { $s != "" } {
puts -nonewline $s "$msg\n.\n"
flush $s
fileevent $s readable [set answer [read $s]]
puts "$answer"
if {[catch {close $s}]} {
puts "Server hanged"
}
This above code is working if the answer was given by the server without any problem. If the server went into infinite loop, it is keep on hanging in read $s. Please help on how to handle this read socket in a non-blocking mode as like in fconfigure.
If you're using blocking sockets, you'll have this problem: putting the channel in non-blocking mode is the fix (together with using after to write a timeout). This does mean that you'll have to deal with all the complexity of asynchronous programming, but that's the tradeoff that you need here.
The two places where things can hang are in connection establishment and in production of the data. You would therefore use asynchronous connection and non-blocking retrieval.
set s [socket -async $host $port]
fconfigure $s -blocking 0
fileevent $s writeable [list connected $s]
proc connected {s} {
global msg
fileevent $s writeable {}
puts -nonewline $s "$msg\n.\n"
flush $s
fileevent $s readable [list accumulateBytes $s]
}
set accumulate ""
proc accumulateBytes {s} {
global accumulate end
append accumulate [read $s]
if {[eof $s]} {
set end 0
}
}
# Longer *total* time to connect and communicate.
after 5000 set end 1
vwait end
catch {close $s}
if {$end} {puts "timed out"}
puts "received message: $accumulate"

TCL gets command with kind of -nohang option?

Here is a code which just implements an interactive TCL session with command prompt MyShell >.
puts -nonewline stdout "MyShell > "
flush stdout
catch { eval [gets stdin] } got
if { $got ne "" } {
puts stderr $got
}
This code prompts MyShell > at the terminal and waits for the enter button to be hit; while it is not hit the code does nothing. This is what the gets command does.
What I need, is some alternative to the gets command, say coolget. The coolget command should not wait for the enter button, but register some slot to be called when it is hit, and just continue the execution. The desired code should look like this:
proc evaluate { string } \
{
catch { eval $string } got
if { $got ne "" } {
puts stderr $got
}
}
puts -nonewline stdout "MyShell > "
flush stdout
coolgets stdin evaluate; # this command should not wait for the enter button
# here goes some code which is to be executed before the enter button is hit
Here is what I needed:
proc prompt { } \
{
puts -nonewline stdout "MyShell > "
flush stdout
}
proc process { } \
{
catch { uplevel #0 [gets stdin] } got
if { $got ne "" } {
puts stderr $got
flush stderr
}
prompt
}
fileevent stdin readable process
prompt
while { true } { update; after 100 }
I think you need to look at the fileevent, fconfigure and vwait commands. Using these you can do something like the following:
proc GetData {chan} {
if {[gets $chan line] >= 0} {
puts -nonewline "Read data: "
puts $line
}
}
fconfigure stdin -blocking 0 -buffering line -translation crlf
fileevent stdin readable [list GetData stdin]
vwait x
This code registers GetData as the readable file event handler for stdin, so whenever there is data available to be read it gets called.
Tcl applies “nohang”-like functionality to the whole channel, and it's done by configuring the channel to be non-blocking. After that, any read will return only the data that is there, gets will only return complete lines that are available without waiting, and puts (on a writable channel) will arrange for its output to be sent to the OS asynchronously. This depends on the event loop being operational.
You are recommended to use non-blocking channels with a registered file event handler. You can combine that with non-blocking to implement your coolget idea:
proc coolget {channel callback} {
fileevent $channel readable [list apply {{ch cb} {
if {[gets $ch line] >= 0} {
uplevel [lappend cb $line]
} elseif {[eof $ch]} {
# Remove handler at EOF: important!
fileevent $ch readable {}
}
}} $channel $callback]
}
That will then work just fine, except that you've got to call either vwait or update to process events (unless you've got Tk in use too; Tk is special) as Tcl won't process things magically in the background; magical background processing causes more trouble than it's worth…
If you're getting deeply tangled in asynchronous event handling, consider using Tcl 8.6's coroutines to restructure the code. In particular, code like Coronet can help a lot. However, that is very strongly dependent on Tcl 8.6, as earlier Tcl implementations can't support coroutines at all; the low-level implementation had to be rewritten from simple C calls to continuations to enable those features, and that's not backport-able with reasonable effort.