i have many package requires which is used inside the tcl code as below:
package require tlautils 1.0
package require profilemanager
package require tla::log 1.2
package require res_mgr 1.1
package require resource 1.2
package require sysexits
package require utils 1.5
package require testutil 1.0
package require profilemgr
package require leakChecker
package require testutil
what is the alternative to use instead of using so many package requires? this is taking time and i am in search of any other alternatives for package require whcih increases the time in seconds/miliseconds
The package require lines don't really take much longer than the load and source calls that they delegate to (all the packages are doing is stopping you from having to hard-code paths to everything, taking care of all the details of versions and so on). However, when you do package require of a package whose name is not already known, Tcl has to actually search for the pkgIndex.tcl files that describe how to actually load the packages. It does this by calling the code that you can look up (or replace if necessary) using package unknown, and that's actually really quite slow. Depending on the TCLLIBPATH environment variable's contents, it could be extremely slow.
But we can “compile” that so that we will be able to source a single file and be able to load these specific packages on this machine quickly.
To do that, we need the above package requires and a bit of extra wrapping code:
package require tlautils 1.0
package require profilemanager
package require tla::log 1.2
package require res_mgr 1.1
package require resource 1.2
package require sysexits
package require utils 1.5
package require testutil 1.0
package require profilemgr
package require leakChecker
package require testutil
set f [open "pkgdefs.tcl" w]
foreach pkg [package names] {
# Skip the packages built directly into Tcl 8.6
if {$pkg in {zlib TclOO tcl::tommath Tcl}} continue
# Get the actual loaded version; known but absent would be an error
if {[catch {package present $pkg} version]} continue
# Get the actual script that we'd run to load the package.
set script [package ifneeded $pkg $version]
# Add to the file
puts $f [list package ifneeded $pkg $version $script]
}
close $f
Once you've run that, you'll have a script in pkgdefs.tcl that you can source. If in future runs you source it before doing any of those package require calls that you listed, those package require calls will be fast. (This also includes any packages that are dependencies of the ones you list.) However, if you ever install new packages to use in the list, or update the versions of the packages, or move them around, you'll need to rebuild the list: it makes your code quite a lot more inflexible, which is why we don't do it by default.
Related
I want to compile and link some cython (and cython_freeze) generated c files into an executable (on Windows and Linux)
I can do this by setting up (a machine specifc) environment (eg what C compiler, where is it installed (path), what include directories, what lib directories...) and calling cl directly (with some os specific options).
But cythonize is is able to compile cython created c files into dynamic libraries without me explicitly setting any machine or os specific settings (just using distutils).
Is there a simple way of using distutils / setuptools to do that (ie compile a c file into a object and link multiple objects into a executable)?
NB: I don't want any of the complex setup logic normally in setup.py. Just the plain compile and link primitives for my development machines.
There is a quite simple solution (that is not really documented):
import sys
import sysconfig
#from distutils.ccompiler import new_compiler # distutils is currently deprecated
from setuptools._distutils.ccompiler import new_compiler
compiler = new_compiler()
compiler.add_include_dir(sysconfig.get_path("include"))
compiler.add_library_dir(sysconfig.get_config_var("installed_platbase") + "/libs")
if sys.platform == "linux":
compiler.add_library_dir(sysconfig.get_config_var("LIBPL"))
compiler.add_library_dir(sysconfig.get_config_var("LIBDIR"))
compiler.add_library(sysconfig.get_config_var("LIBRARY")[3:-2])
for lib in sysconfig.get_config_var("LIBS").split():
compiler.add_library(lib[2:])
objectFiles = compiler.compile(["source1.c", "source2.c", "test.rc"])
compiler.link_executable(objectFiles, "test")
I worked on a TCL script that takes a bit of time to execute. In order to to run it faster, I would like to use multi-threading using the Thread package. The problem is that I execute my TCL script in an engineering software called Hypermesh. It supports TCL scripting quite well since everything that can be done in the software has an associated TCL command.
For example if the user was to use Hypermesh to create a node at coordinate (x, y, z) then the same result can be obtained using the command *createnode $x $y $z.
Back to the multi-thread problem: I can create threads just fine but inside the thread, I cannot use the command *createnode because it is not recognized. I am guessing it is because some packages are not loaded inside the thread. I would like to load those missing packages with "package require" but no matter how I tried, it could not find any package.
I found out that inside a thread, the auto_path variable does not exist. I tried to create it, but it still would not work.
It looks like my thread can execute basic TCL buit-in commands and nothing else.
On the other hand, when I created a Thread using wish, I did not have this problem at all. All the packages "outside" the thread were also directly loaded "inside".
Here is how my code looks like:
package require Thread
package require Ttrace
set scriptToSend {
package require Trf
return 0
}
set t1 [thread::create]
thread::send -async $t1 $scriptToSend result
puts $result
Trf is just a random package that can be loaded just fine when "package require" is called outside of t1
Is there any way I could setup the thread to have all the commands that exist outside the thread? For example by making the use of package require possible?
EDIT:
I spent some more time on this problem and tried to use tpool instead of thread. The threads created in tpool seem "better". Unlike with thread, they do have an auto_path variable and I am able to use it to successfully load packages when using threads. However now I have more of an Hypermesh issue since it does not let me load all the necessary packages to use procs such as *createnode.
I will try to go around the problem by making my threads do as much as they can without using any Hypermesh function. It will still speed up the script quite a bit.
Tcl uses a limited version of the auto_path global in threads by default because it doesn't set it up in quite the same way (e.g., it doesn't evaluate your ~/.tclshrc). To use the same packages as in the main thread, you'll need to transfer a copy of that variable over.
thread::send $otherThread [list set ::auto_path $::auto_path]
Do that before sending other scripts to the subthread in order to synch things up.
On my system I'd see this sort of thing too:
bash$ tclsh8.6
% puts $::auto_path
/opt/local/lib/tcl8.6 /opt/local/lib /opt/local/lib/tcllib1.19
% package require Thread
2.8.4
% set otherThread [thread::create]
tid0x7000053ab000
% puts [thread::send $otherThread {set ::auto_path}]
/opt/local/lib/tcl8.6 /opt/local/lib
There's also the module path (controllable via the tcl::tm::path command) but that's the same for me in the subthread as in the main thread.
I would like a way to limit what packages can be loaded into tcl.
I can't just remove them from the path, as it has some packages that should be loaded.
Is there a way to handle this without separating the packages to different directories ?
Also, is there a way to add a trace to every package being loaded ? So I can see after the fact which packages have been loaded ?
Thanks
Tcl loads packages in several stages, which are described in a reasonable amount of detail here:
If you ask for a package it does not already known about, it calls the handler registered with package unknown to search the package path (with various origins of the bits of the path) for package index files (pkgIndex.tcl) which provide descriptors of the packages that can be loaded. These descriptors are files that call package ifneeded to give a script to call to load a particular version of a particular package name. If it still doesn't know about the package after doing the loading, you get an error message.
Most pkgIndex.tcl scripts are incredibly simple, though they tend to be dynamically generated during the package installation process. A couple of minor points though; they are not evaluated in the global context, but rather are inside a scope that defines a local dir variable that says the name of the directory containing the pkgIndex.tcl script, and they are always assumed to be UTF-8 encoded on all platforms (I recommend only using ASCII characters in them anyway, which tends to be pretty easy to do precisely because the script is entirely under programmer control).
If you ask for a package it does know about, because the package is either already loaded or there is a descriptor present for it, it checks the version numbers for a match and if the package isn't actually loaded, it evaluates the script provided through package ifneeded in the global context to actually load the package. That will usually source Tcl scripts and/or load DLLs. Some packages do more complicated things than that, but it's common to not put too much complexity in the descriptor scripts themselves: complicated stuff goes into the code they load in, which can be as complex as the package needs it to be.
Now, there is another package subcommand that is useful to you, package forget, which makes Tcl forget the package descriptor and whether the package is loaded. It does not forget the implementation of the package if it is actually loaded but if you can forget the descriptor before stage two, you deny the Tcl interpreter the ability to load the package. If you've got a whitelist of allowed package names and versions, all you need to do is intercept the package unknown handler, delegate to the old implementation, and then strip out anything you don't want before returning. Fortunately, the unknown package handler system is designed to be interceptable like this.
# These packages are to not be version-checked
set builtins {zlib TclOO tcl::tommath Tcl}
# These are the external packages we want to support filtering; THIS IS A WHITELIST!
set whitelist {
Thread {2.8.1}
yaml {0.3.1}
http {2.8.11}
}
proc filterPackages {wrappedCall args} {
# Do the delegation first
uplevel 1 [list {*}$wrappedCall {*}$args]
# Now filter out anything unexpected
global builtins whitelist
foreach p [package names] {
# Check if this is a package we should just tolerate first
if {$p in $builtins} continue
set scripts {}
foreach v [package versions $p] {
if {[dict exists $whitelist $p] && $v in [dict get $whitelist $p]} {
# This is a version we want to keep
lappend scripts [list \
package ifneeded $p $v [package ifneeded $p $v]]
# Everything else is going to be forgotten in a few moments
}
}
# Forget all those package descriptors...
package forget $p
# ...but put back the ones we want.
foreach s $scripts { eval $s }
}
}
# Now install our filter
package unknown [list filterPackages [package unknown]]
As long as install this filter early in your script, before any package require commands have been done (except of the small list of true builtins), you'll have a system which can only package require the packages that you want to allow.
So for a particular CGI perl script I have included JSON like this to handle some .json files:
use lib "./modules/JSON/lib";
use JSON;
This works fine and well. The web directory holds the files required in the modules folder.
However, the JSON module is very slow. I read that JSON:XS can be much, much faster but I can't seem to simply use it as so:
use lib "./modules/JSON-XS";
use JSON::XS;
There is no lib folder in the JSON-XS files, i've tried combinations of use (ie, using both folders and etc) but it didn't work.
And no I cannot simply install the module for this particular project.
Any help is appreciated.
And no I cannot simply install the module for this particular project.
You can't use a module without installing it. You've just been getting away with doing a half-assed job of it. That won't work for JSON::XS, though. The reason it's fast is because it's written in C, so you'll need to compile the C code. The easiest way by far to do this is to use the provided installer instead of reinventing the wheel.
(You do know you can install a module into any directory, and that this does not require special permissions, right?)
Perl distributions are usually usable in an uninstalled state. What you just need to do is to call perl Makefile.PL && make (or for a Module::Build-based distribution: perl Build.PL && ./Build). This will do all necessary compilations (if it's an XS module) and copy the library files into the blib subdirectory. In your script instead of use lib you would write use blib:
use blib "/path/to/JSON-XS";
Note that if a module has dependencies, then you have to resolve it yourself and add that many use blib statements. JSON::XS does not have that many dependencies, but it will be really inconvenient for other modules. In this case you should probably seek another solution, e.g. using CPAN.pm together with local::lib.
Okay this finally worked for me:
I did this process to all the dependencies (in the order of no dependencies to more dependencies)
export PERL5LIB = ~/path/to/modules/perl5
perl Makefile.PL PREFIX=$PERL5LIB LIB=$PERL5LIB
make
make test
make install
This installed all modules into a directory I called perl5. It also means that when you try to install other modules locally the dependencies issue does not appear due to the PREFIX/LIB additions.
Then all I did was add this to my perl CGI script:
use lib "./modules/perl5";
use JSON::XS;
PS: JSON::XS is so much faster!
:D
does tcl have a standard way of doing NSS lookups (getpwnam, setpwent,...)
Tcl doesn't expose those as APIs (it doesn't really use them internally either) but the TclX extension package does support exactly what you want I believe. For example:
package require TclX
set uid [id convert user $tcl_platform(user)]
puts "Your userid is $uid and you are a member of these groups: [id groups]"
If you're using ActiveTcl, you've definitely got the TclX package available (either already installed or available from the teapot repository).