I need redirect that I found in one of my searches:
redirect -tee LogFile.log {include ../RunDemoTests.tcl}
Where include is a TCL proc and ../RunDemoTests.tcl is a parameter to the proc. Is there a library I need to be able to use redirect or is this not general tcl?
I am working in an EDA tool environment that runs under both Windows and Linux, so I need a solution that is just TCL and does not rely on something from the OS.
I have tried numerous variations of:
set ch [open |[list include "../OsvvmLibraries/UART/RunDemoTests.tcl"] r]
set lf [open build.log w]
puts "Starting"
puts $lf "Starting"
while {[gets $ch line]>=0} {
puts $line
puts $lf $line
}
close $lf
However, this only seems to work when the command is something from the OS environment, such as:
set ch [open |[list cat ../tests.pro] r]
Printing from this can be a significant number of lines, buffering is ok, but not collecting the whole file and then printing as the files can be long (180K lines).
In response to a question on comp.lang.tcl a while ago, I created a small Tcl module to provide tee-like functionality in Tcl. I have now published the code on the Tcl wiki.
You would use it like this:
package require tee
tee stdout build.log
try {
puts "Starting"
include ../OsvvmLibraries/UART/RunDemoTests.tcl
} finally {
# Make sure the tee filter is always popped from the filter stack
chan pop stdout
}
This assumes the include RunDemoTests.tcl command produces output to stdout.
Related
I am using tee from https://wiki.tcl-lang.org/page/Tee to redirect file output from my procedures. I need to redirect both stdout and stderr to the file.
Using the input from Redirecting output of tcl proc to file and output (like tee) I arrived at doing the following:
set LogFile [open ${LogFileName} w]
tee channel stderr $LogFile
tee channel stdout $LogFile
set BuildErrorCode [catch {LocalBuild $BuildName $Path_Or_File} BuildErrMsg]
set BuildErrorInfo $::errorInfo
# Restore stdout and stderr
chan pop stdout
chan pop stderr
# Handle errors from Build ...
I am testing this on three different EDA tools and I have three different issues.
When I run from tclsh (on MSYS2 running on Windows 10) and run either the open source simulator GHDL, ModelSim, or QuestaSim, all the even characters are the NUL character.
If I run ModelSim or QuestaSim from the GUI, I miss the output of each command. Shouldn't that be going to either stdout or stderr?
In Riviera-PRO, I am getting extraneous characters that were previously printed. They are generally the second half of a word.
Am I doing something wrong? I tested out the above code using:
set LogFile [open test_tee.log w]
tee channel stderr $LogFile
tee channel stdout $LogFile
puts "Hello, World!"
puts stderr "Error Channel"
puts stdout "Output Channel"
chan pop stdout
chan pop stderr
And this works well.
I am hoping to find something that works in the general case for all tools rather than having to write a different handler for each tool.
============ Update =============
For #1 above, with #Shawn's suggestion, I tried the following and it did not work.
set LogFile [open ${LogFileName} w]
chan configure $LogFile -encoding ascii
. . .
I also tried the following and it did not work.
set LogFile [open ${LogFileName} w]
fconfigure $LogFile -encoding ascii
. . .
Then I tried updating the write in tee to the following and it did not work:
proc tee::write {fd handle buffer} {
puts -nonewline $fd [encoding convertto ascii $buffer]
return $buffer
}
Any other hints solutions appreciated
============ Update2 =============
I have successfully removed the nul characters by doing the following, except now I have an extra newline. Still not a solution.
proc tee::write {fd handle buffer} {
puts -nonewline $fd [regsub -all \x00 $buffer ""]
return $buffer
}
The extra NUL bytes are probably because the stdout ahd steer channels are being written in UTF-16 (the main use for that encoding is the console on Windows). The tee interceptors you are using come after the data being written is encoded. There's a few ways to fix it, but the easiest is to open the file with the right encoding when reading it.
The output of the commands is not necessarily written to those channels. Code written in C or C++ is entirely free to write directly, and Tcl code cannot see that; it's all happening behind our back. Command results can be intercepted using execution traces, but that cannot see anything that the commands internally print that aren't routed via the Tcl library somehow. (There are a few more options on Unix due to the different ways that the OS handles I/O.)
Don't know what's happening with the extra characters. I can tell you that you are getting what goes through the channel, but there are too many tricks (especially in interactive use!) for a useful guess on that front.
What is wrong in this code
set fid [open "file_name" a+]
while {[gets $fid line] > -1} {
lappend short_keys_list [lindex $line 5]
puts $line
}
close $fid
# while loop not working
Since you have opened the file with a+ mode, the file pointer is set to the end of the file. Due to this, the call to gets $fid line is returning -1 as it has nothing to read from the file and causing the while loop to terminate.
Try this
set fid [open "file_name" r]
Reference : open
You've been answered with a couple of hints to what isn't working. The best solution to what you seem to be doing, however, is this:
foreachLine line file_name {
lappend short_keys_list [lindex $line 5]
puts $line
}
This invocation takes care of opening and closing of the file, reading each line of the file and storing it in the variable name you've provided (line in this case) and calling your script once for each line. For this to work you first need to get the fileutil package ready:
package require fileutil
namespace import ::fileutil::*
The fileutil package contains a lot of useful commands to easily deal with issues that are a major bother when using low-level filehandling, as I'm usually quick to point out #heyhoodiecrowwhydontyoumarryfileutilalready.
Documentation: fileutil package, lappend, lindex, namespace, package, puts
In text file, I have some Tcl expressions like:
# DO NOT EDIT!!
# [exec whoami] - [exec date]
# Generated config file
Basically, this text file is a part of a config file. I want to read the file in and replace those expressions with actual values.
The subst command is designed for this purpose.
set f [open $theFileName]
set unsubstituted [read $f]
close $f
set substituted [subst $unsubstituted]
# Now use the contents...
Be aware that this can change all sorts of things inside your script, as set is one of the commands that might be there! To limit the potential damage a bit, run in a sub-interpreter:
set f [open $theFileName]
set unsubstituted [read $f]
close $f
set i [interp create]
set substituted [interp eval $i [list subst $unsubstituted]]
interp delete $i
The script will still be able to exit (or exec rm -rf *; it's genuinely as unsafe as running an arbitrary program) but at least won't be able to change your internal configuration unexpectedly.
The info that whoami produces is available directly in Tcl as $tcl_platform(user).
The info that exec date produces is available directly in Tcl as clock format [clock seconds]. Well, with slightly different field order for me by default it seems…
I'm trying to implement a tcl script which reads a text file, and masks all the sensitive information (such as passwords, ip addresses etc) contained it and writes the output to another file.
As of now I'm just substituting this data with ** or ##### and searching the entire file with regexp to find the stuff which I need to mask. But since my text file can be 100K lines of text or more, this is turning out to be incredibly inefficient.
Are there any built in tcl functions/commands I can make use of to do this faster? Do any of the add on packages provide extra options which can help get this done?
Note: I'm using tcl 8.4 (But if there are ways to do this in newer versions of tcl, please do point me to them)
Generally speaking, you should put your code in a procedure to get best performance out of Tcl. (You have got a few more related options in 8.5 and 8.6, such as lambda terms and class methods, but they're closely related to procedures.) You should also be careful with a number of other things:
Put your expressions in braces (expr {$a + $b} instead of expr $a + $b) as that enables a much more efficient compilation strategy.
Pick your channel encodings carefully. (If you do fconfigure $chan -translation binary, that channel will transfer bytes and not characters. However, gets is not be very efficient on byte-oriented channels in 8.4. Using -encoding iso8859-1 -translation lf will give most of the benefits there.)
Tcl does channel buffering quite well.
It might be worth benchmarking your code with different versions of Tcl to see which works best. Try using a tclkit build for testing if you don't want to go to the (minor) hassle of having multiple Tcl interpreters installed just for testing.
The idiomatic way to do line-oriented transformations would be:
proc transformFile {sourceFile targetFile RE replacement} {
# Open for reading
set fin [open $sourceFile]
fconfigure $fin -encoding iso8859-1 -translation lf
# Open for writing
set fout [open $targetFile w]
fconfigure $fout -encoding iso8859-1 -translation lf
# Iterate over the lines, applying the replacement
while {[gets $fin line] >= 0} {
regsub -- $RE $line $replacement line
puts $fout $line
}
# All done
close $fin
close $fout
}
If the file is small enough that it can all fit in memory easily, this is more efficient because the entire match-replace loop is hoisted into the C level:
proc transformFile {sourceFile targetFile RE replacement} {
# Open for reading
set fin [open $sourceFile]
fconfigure $fin -encoding iso8859-1 -translation lf
# Open for writing
set fout [open $targetFile w]
fconfigure $fout -encoding iso8859-1 -translation lf
# Apply the replacement over all lines
regsub -all -line -- $RE [read $fin] $replacement outputlines
puts $fout $outputlines
# All done
close $fin
close $fout
}
Finally, regular expressions aren't necessarily the fastest way to do matching of strings (for example, string match is much faster, but accepts a far more restricted type of pattern). Transforming one style of replacement code to another and getting it to go really fast is not 100% trivial (REs are really flexible).
Especially for very large files - as mentioned - it's not the best way to read the whole file into a variable. As soon as your system runs out of memory you can't prevent your app crashes. For data that is separated by line breaks, the easiest solution is to buffer one line and process it.
Just to give you an example:
# Open old and new file
set old [open "input.txt" r]
set new [open "output.txt" w]
# Configure input channel to provide data separated by line breaks
fconfigure $old -buffering line
# Until the end of the file is reached:
while {[gets $old ln] != -1} {
# Mask sensitive information on variable ln
...
# Write back line to new file
puts $new $ln
}
# Close channels
close $old
close $new
I can't think of any better way to process large files in Tcl - please feel free to tell me any better solution. But Tcl was not made to process large data files. For real performance you may use a compiled instead of a scripted programming language.
Edit: Replaced ![eof $old] in while loop.
A file with 100K lines is not that much (unless every line is 1K chars long :) so I'd suggest you read the entire file into a var and make the substitution on that var:
set fd [open file r+]
set buf [read $fd]
set buf [regsub -all $(the-passwd-pattern) $buf ****]
# write it back
seek $fd 0; # This is not safe! See potrzebie's comment for details.
puts -nonewline $fd $buf
close $fd
I need to store some logs in a file that can grow with every execution. A logical way would be to use a+ option when opening because using w+ would truncate the file. However, with the a+ option (Tcl 8.4) I cannot write anywhere in the file. seek works fine. I can verify that the pointer was moved using tell. But the output is always done at the tail end of the file.
Is there any way to resolve this? I.e. having the ability to seek and write in any place and also preserve the old file at the open.
In Tcl 8.5, the behavior of Tcl on Unix was changed so that the O_APPEND flag is passed to the open() system call. This causes the OS to always append the data to the file, and is inherited when the FD is passed to subprocesses; for logs, it is exactly the right thing. (In 8.4 and before, and in all versions on Windows, the behavior is simulated inside Tcl's file channel implementation, which will internally seek() to the end immediately before the write(); that obviously is subject to potential problems with race conditions when there are multiple processes logging to the same file and is definitely unsafe when the FD is passed to subprocesses.) You can manage truncation of the opened file with chan truncate (new in 8.5), which works just fine on a+-opened files.
If you do not want the seek-to-end behavior, you should not use a+ (or a). Try r+ or some combination of flags, like this:
set f [open $filename {RDWR CREAT}]
For comparison, the a+ option is now exactly the same as the flags RDWR CREAT APPEND, and not all combinations of longer flags can be described by short form flag specifiers. If you're not specifying APPEND, you'll need to do the seek $f 0 end yourself (and watch out for problems with multiple processes if you're appending to logs; that's when APPEND becomes required and exceptionally hard to correctly simulate any other way).
Open with r+ - it opens in read mode (thus not turncating the file) but allows writing as well.
See the documentation of open for more info: http://www.tcl.tk/man/tcl8.5/TclCmd/open.htm
I have verified that using the a+ option allow me to read/write anywhere in the file. However, by writing in the middle (or at the beginning) of a file, I overwrite the data there, not inserting. The following code illustrate that point:
#!/usr/bin/env tclsh
# Open the file, with truncation
set f [open foo w]
puts $f "one"
puts $f "two"
close $f
# Open again, with a+ ==> read/write/append
set f [open foo a+]
puts $f "three" ;# This goes to the end of the file
seek $f 4 ;# Seek to the beginning of the word "two"
puts $f "2.0" ;# Overwrite the word "two"
close $f
# Open and verify the contents
set f [open foo r]
puts [read $f]
close $f
Output:
one
2.0
three
If you are looking to insert in the middle of the file, you might want to look at the fileutil package, which contains the ::fileutil::insertIntoFile command.