I would like a way to limit what packages can be loaded into tcl.
I can't just remove them from the path, as it has some packages that should be loaded.
Is there a way to handle this without separating the packages to different directories ?
Also, is there a way to add a trace to every package being loaded ? So I can see after the fact which packages have been loaded ?
Thanks
Tcl loads packages in several stages, which are described in a reasonable amount of detail here:
If you ask for a package it does not already known about, it calls the handler registered with package unknown to search the package path (with various origins of the bits of the path) for package index files (pkgIndex.tcl) which provide descriptors of the packages that can be loaded. These descriptors are files that call package ifneeded to give a script to call to load a particular version of a particular package name. If it still doesn't know about the package after doing the loading, you get an error message.
Most pkgIndex.tcl scripts are incredibly simple, though they tend to be dynamically generated during the package installation process. A couple of minor points though; they are not evaluated in the global context, but rather are inside a scope that defines a local dir variable that says the name of the directory containing the pkgIndex.tcl script, and they are always assumed to be UTF-8 encoded on all platforms (I recommend only using ASCII characters in them anyway, which tends to be pretty easy to do precisely because the script is entirely under programmer control).
If you ask for a package it does know about, because the package is either already loaded or there is a descriptor present for it, it checks the version numbers for a match and if the package isn't actually loaded, it evaluates the script provided through package ifneeded in the global context to actually load the package. That will usually source Tcl scripts and/or load DLLs. Some packages do more complicated things than that, but it's common to not put too much complexity in the descriptor scripts themselves: complicated stuff goes into the code they load in, which can be as complex as the package needs it to be.
Now, there is another package subcommand that is useful to you, package forget, which makes Tcl forget the package descriptor and whether the package is loaded. It does not forget the implementation of the package if it is actually loaded but if you can forget the descriptor before stage two, you deny the Tcl interpreter the ability to load the package. If you've got a whitelist of allowed package names and versions, all you need to do is intercept the package unknown handler, delegate to the old implementation, and then strip out anything you don't want before returning. Fortunately, the unknown package handler system is designed to be interceptable like this.
# These packages are to not be version-checked
set builtins {zlib TclOO tcl::tommath Tcl}
# These are the external packages we want to support filtering; THIS IS A WHITELIST!
set whitelist {
Thread {2.8.1}
yaml {0.3.1}
http {2.8.11}
}
proc filterPackages {wrappedCall args} {
# Do the delegation first
uplevel 1 [list {*}$wrappedCall {*}$args]
# Now filter out anything unexpected
global builtins whitelist
foreach p [package names] {
# Check if this is a package we should just tolerate first
if {$p in $builtins} continue
set scripts {}
foreach v [package versions $p] {
if {[dict exists $whitelist $p] && $v in [dict get $whitelist $p]} {
# This is a version we want to keep
lappend scripts [list \
package ifneeded $p $v [package ifneeded $p $v]]
# Everything else is going to be forgotten in a few moments
}
}
# Forget all those package descriptors...
package forget $p
# ...but put back the ones we want.
foreach s $scripts { eval $s }
}
}
# Now install our filter
package unknown [list filterPackages [package unknown]]
As long as install this filter early in your script, before any package require commands have been done (except of the small list of true builtins), you'll have a system which can only package require the packages that you want to allow.
Related
I worked on a TCL script that takes a bit of time to execute. In order to to run it faster, I would like to use multi-threading using the Thread package. The problem is that I execute my TCL script in an engineering software called Hypermesh. It supports TCL scripting quite well since everything that can be done in the software has an associated TCL command.
For example if the user was to use Hypermesh to create a node at coordinate (x, y, z) then the same result can be obtained using the command *createnode $x $y $z.
Back to the multi-thread problem: I can create threads just fine but inside the thread, I cannot use the command *createnode because it is not recognized. I am guessing it is because some packages are not loaded inside the thread. I would like to load those missing packages with "package require" but no matter how I tried, it could not find any package.
I found out that inside a thread, the auto_path variable does not exist. I tried to create it, but it still would not work.
It looks like my thread can execute basic TCL buit-in commands and nothing else.
On the other hand, when I created a Thread using wish, I did not have this problem at all. All the packages "outside" the thread were also directly loaded "inside".
Here is how my code looks like:
package require Thread
package require Ttrace
set scriptToSend {
package require Trf
return 0
}
set t1 [thread::create]
thread::send -async $t1 $scriptToSend result
puts $result
Trf is just a random package that can be loaded just fine when "package require" is called outside of t1
Is there any way I could setup the thread to have all the commands that exist outside the thread? For example by making the use of package require possible?
EDIT:
I spent some more time on this problem and tried to use tpool instead of thread. The threads created in tpool seem "better". Unlike with thread, they do have an auto_path variable and I am able to use it to successfully load packages when using threads. However now I have more of an Hypermesh issue since it does not let me load all the necessary packages to use procs such as *createnode.
I will try to go around the problem by making my threads do as much as they can without using any Hypermesh function. It will still speed up the script quite a bit.
Tcl uses a limited version of the auto_path global in threads by default because it doesn't set it up in quite the same way (e.g., it doesn't evaluate your ~/.tclshrc). To use the same packages as in the main thread, you'll need to transfer a copy of that variable over.
thread::send $otherThread [list set ::auto_path $::auto_path]
Do that before sending other scripts to the subthread in order to synch things up.
On my system I'd see this sort of thing too:
bash$ tclsh8.6
% puts $::auto_path
/opt/local/lib/tcl8.6 /opt/local/lib /opt/local/lib/tcllib1.19
% package require Thread
2.8.4
% set otherThread [thread::create]
tid0x7000053ab000
% puts [thread::send $otherThread {set ::auto_path}]
/opt/local/lib/tcl8.6 /opt/local/lib
There's also the module path (controllable via the tcl::tm::path command) but that's the same for me in the subthread as in the main thread.
For my summer programming course I need to compile and test several functions from an ocaml file (a .ml file) our teacher gave us. I've never programmed in ocaml before, but I downloaded it on my ubuntu vm and not I can't figure out how to compile it and test the functions from the terminal.
I'm also curious if I need to add print statements to the code in order to test them, since ocaml doesn't require main methods. If so, how do I print the return value of a function?
Thank you for your help, I apologize if this is a newbie question.
Make sure that you have installed OCaml properly, by running the OCaml interactive toplevel (don't type $ this is a prompt from your shell):
$ ocaml
It should show something like this:
OCaml version 4.07.0
#
The # symbol is a prompt, you can type OCaml definitions there and send them to the interpreter using the ;; terminating sequence (again don't type # it is also a prompt symbol), e.g.,
# print_endline "Hello, world";;
Hello, world
- : unit = ()
#
Hint: to enable history install rlwrap with sudo apt install rlwrap and run the ocaml toplevel as
$ rlwrap ocaml
Now, we are ready to compile our first program. In OCaml, like in Python, all top-level definitions in your program are evaluated in order of their appearance, therefore you don't need to have a special main function. Despite this fact, I'm usually defining one, using the following idiom
let main () =
print_endline "Hello, world"
(* the entry point - just call the main function *)
let () = main ()
Now, create a new folder (make sure it is empty)
$ mkdir my-first-ocaml-program
$ cd my-first-ocaml-program
and put the OCaml code above into a file named test.ml (the filename doesn't have any special meaning for the compiler toolchain, but I will reference this name in the shell commands below).
let's test that everything is correct, by printing the contents of the test.ml file
$ cat test.ml
and the output should be
let main () =
print_endline "Hello, world"
(* the entry point - just call the main function *)
let () = main ()
Now, let's compile and run at the same time,
$ ocamlbuild test.native --
And we should see the "Hello, world",
Finished, 4 targets (4 cached) in 00:00:00.
Hello, world
The first line is the output from the compiler (ignore it, unless it is different). Starting from the second line it is our output. Here are some explanations on the build one-liner, ocamlbuild test.native. It uses ocamlbuild, an easy to use but powerful OCaml build tool. The test.native tells ocamlbuild that you want to build a native (machine code) binary and that test.ml is the main source file. You can also ask to build a bytecode binary, e.g., test.byte. This is called a target in ocamlbuild parlance. The -- is optional, and it tells ocamlbuild to run the built target as soon as it is ready. Any argument past -- is passed to your program as command line arguments.
What about larger programs or programs with dependencies? The good news is that you can put your code in several files in the same folder, and ocamlbuild will be clever enough to build them in the proper order (it will do the dependency analysis, compiling, and linking for you - all seamlessly). If you need to use some external package, then you can specify it via the -pkg option. For example, let's assume that we're using the fileutils package to implement our version of the ls program1. Let's update our test.ml so that it now is having the following contents:
$ cat test.ml
let main () =
FileUtil.ls "." |> List.iter print_endline
(* the entry point - just call the main function *)
let () = main ()
and, as usual, build and print in one
$ ocamlbuild -pkg fileutils test.native --
Finished, 4 targets (4 cached) in 00:00:00.
./test.native
./_build
./test.ml
1) How to install a package is system dependent. E.g., if you're using apt to install OCaml then you can do sudo apt install libfileutils-ocaml-dev. If you're using opam (the recommended way), then it is just opam install fileutils. In any case, package installation is out of the scope of this question, I'm assuming that in your course you would be using some packages pre-installed. We use fileutils here just as an example, which you can easily adapt to your own purposes.
i have many package requires which is used inside the tcl code as below:
package require tlautils 1.0
package require profilemanager
package require tla::log 1.2
package require res_mgr 1.1
package require resource 1.2
package require sysexits
package require utils 1.5
package require testutil 1.0
package require profilemgr
package require leakChecker
package require testutil
what is the alternative to use instead of using so many package requires? this is taking time and i am in search of any other alternatives for package require whcih increases the time in seconds/miliseconds
The package require lines don't really take much longer than the load and source calls that they delegate to (all the packages are doing is stopping you from having to hard-code paths to everything, taking care of all the details of versions and so on). However, when you do package require of a package whose name is not already known, Tcl has to actually search for the pkgIndex.tcl files that describe how to actually load the packages. It does this by calling the code that you can look up (or replace if necessary) using package unknown, and that's actually really quite slow. Depending on the TCLLIBPATH environment variable's contents, it could be extremely slow.
But we can “compile” that so that we will be able to source a single file and be able to load these specific packages on this machine quickly.
To do that, we need the above package requires and a bit of extra wrapping code:
package require tlautils 1.0
package require profilemanager
package require tla::log 1.2
package require res_mgr 1.1
package require resource 1.2
package require sysexits
package require utils 1.5
package require testutil 1.0
package require profilemgr
package require leakChecker
package require testutil
set f [open "pkgdefs.tcl" w]
foreach pkg [package names] {
# Skip the packages built directly into Tcl 8.6
if {$pkg in {zlib TclOO tcl::tommath Tcl}} continue
# Get the actual loaded version; known but absent would be an error
if {[catch {package present $pkg} version]} continue
# Get the actual script that we'd run to load the package.
set script [package ifneeded $pkg $version]
# Add to the file
puts $f [list package ifneeded $pkg $version $script]
}
close $f
Once you've run that, you'll have a script in pkgdefs.tcl that you can source. If in future runs you source it before doing any of those package require calls that you listed, those package require calls will be fast. (This also includes any packages that are dependencies of the ones you list.) However, if you ever install new packages to use in the list, or update the versions of the packages, or move them around, you'll need to rebuild the list: it makes your code quite a lot more inflexible, which is why we don't do it by default.
I need help here.
I have list of directory/file path and my program will read through every one.
Somehow one of the directory is unmount/bad disks and cause my program hang over there when I'm try to open the file using command below.
catch {set directory_fid [open $filePath r]}
So, how can I check the directory status before I'm reading/open the file? I want to skip that file if no response for certain time and continue to read next file.
*file isdir $dir is not working as well
*There is no response when i'm using ls -dir in Unix also.
Before you start down this path, I would review your requirements and see if there's any easier way to handle this. It would be better to fix the mounts so that they don't cause a hang condition if an access attempt is made.
The main problem is that for the directories you are checking, you need to know the corresponding mount point. If you don't know the mount point, it's hard to tell whether the directory you want to check will cause any hangs when you try to access it.
First, you would have to parse /etc/fstab and get a list of possible filesystem mount points (Assumption, Linux system -- if not Linux, there will be an equivalent file).
Second, to see what is currently mounted you need the di Tcl extension (wiki page) (or main page w/download links). (*). Using this extension, you can get a list of mounted filesystems.
# the load only needs to be done once...
set ext [info sharedlibextension]
set lfn [file normalize [file join [file dirname [info script]] diskspace$ext]]
load $lfn
# there are various options that can be passed to the `diskspace`
# command that will change which filesystems are listed.
set fsdata [diskspace -f {}]
set fslist [dict keys $fsdata]
Now you have a list of possible mount points, and you know which are mounted.
Third, you need to figure out which mount point corresponds to the directory you want to check. For example, if you have:
/user/bll/source/stuff.c
You need to check for /user/bll/source, then /user/bll, then /user, then / as possible mount points.
There's a huge assumption here that the file or any of its parent directories are not symlinked to another place.
Once you determine the probable mount point, you can check if it is mounted:
if { $mountpoint in $fslist } {
...
} else {
# better skip this one, the probable mount point is not mounted.
}
As you can see, this is a lot of work. It's fragile.
Better to fix the mounts so they don't hang.
(*) I wrote di and the di Tcl extension. This is a portable solution. You can of course use exec to run df or mount, but there are other issues (parsing, portability, determining which filesystems to use) if you use the more manual method.
Run tclsh command without any tcl file, the interpreter will go into interactive mode.
Can I simply disable this feature by modifying the tclsh source code ?
I can't imagine why you would want to bother doing this, given that supplying any script file will turn off interactive mode. The script you supply will have full access to the additional arguments passed in (a list in the global argv variable) and the standard IO channels (stdin, stdout and stderr). It can exit when it is done. Literally anything you want can be done at that point; you've just got to write a script to do it.
If you're including Tcl in your own program, the behaviour of tclsh is implemented in the C function Tcl_Main. If you never call that — instead just using Tcl_FindExecutable, Tcl_CreateInterp and Tcl_Eval/Tcl_EvalFile — then you never get any of that interactive behaviour. While theoretically you could modify the Tcl source itself to do what you want — it's all open source — why would you bother when you could just not call that code in the first place?