This is a very old installation of octave, 3.05, but it is standard on centos-5, which still exists in many production environments. When I strace octave, I see that it is constantly calling stat on the same files over and over. It must be spending half its time just stat-ing the same files. Here is a sample:
stat("/usr/local/share/octave/3.0.5/m/path", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/usr/local/share/octave/3.0.5/m/plot", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/usr/local/share/octave/3.0.5/m/pkg", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/usr/local/share/octave/3.0.5/m/sparse", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/usr/local/share/octave/3.0.5/m/control", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/usr/local/share/octave/3.0.5/m/control/obsolete", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
stat("/usr/local/share/octave/3.0.5/m/control/hinf", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
That doesn't really give the full flavor because its many more and the same ones over and over. It happens in bursts. Why is it doing this and how can I make it stop? Or at least make it do it once instead of constantly.
I figured it out! Octave is stat-ing the files to check the modification timestamp to see if they have changed since the last time it stat-ed them. This can be disabled and should improve performance, regardless of the octave version.
From http://www.gnu.org/software/octave/doc/interpreter/Function-Files.html:
If you know that your own function files will not change while you are running Octave, you can improve performance by calling ignore_function_time_stamp ("all"), so that Octave will ignore the time stamps for all function files. Passing "system" to this function resets the default behavior.
Related
I have a strange problem with my windows .bat files a 0 is coming before < while executing. I don't know where its getting it from. Below is the contents of the batch file date1.bat
set mysql="C:\Program Files\MySQL\MySQL Server 8.0\bin\mysql.exe"
set progDir="D:\BigData\14.Nodejs\3.Firebase"
set dataDir=D:\BigData\14.Nodejs\3.Firebase\data
%mysql% -ualpha -pbeta test < "%dataDir%\LatestData - Q -201811 - INSERT DMLs.sql"
Issue I am referring to comes in the line
%mysql% -ualpha -pbeta test < "%dataDir%\LatestData - Q -201811 - INSERT DMLs.sql"
Below is the output
D:\BigData\14.Nodejs\3.Firebase>date1
D:\BigData\14.Nodejs\3.Firebase>set mysql="C:\Program Files\MySQL\MySQL Server 8.0\bin\mysql.exe"
D:\BigData\14.Nodejs\3.Firebase>set progDir="D:\BigData\14.Nodejs\3.Firebase"
D:\BigData\14.Nodejs\3.Firebase>set dataDir=D:\BigData\14.Nodejs\3.Firebase\data
D:\BigData\14.Nodejs\3.Firebase>"C:\Program Files\MySQL\MySQL Server 8.0\bin\mysql.exe" -ualpha -pbeta test 0<"D:\BigData\14.Nodejs\3.Firebase\data\LatestData - Q -201811 - INSERT DMLs.sql"
In the last time you can see a "0<" not sure where its getting that 0 from. Is there a way to avoid it.
I am just trying to run DMLs in multiple files via windows batch.
0 means standard input. 0< myfile means send the contents of myfile to standard input. < myfile is shorthand for 0< myfile. The 0 is doing no harm and you don't need to get rid of it.
What you are viewing is the echo of commands as how the interpreter evaluates the code.
Handle 0 is Stdin which < redirection is interpreted as from handle 0<.
Handle 1 is Stdout which > redirection is interpreted as from handle 1> or to handle >&1.
Handle 2 is Stderr which 2> redirection is interpreted as from handle 2> or to handle>&2.
Handles 3 to 9 are auxiliary handles unique to batch-file.
I have a code that creates a subfolder but first removes the subfolder if it already existed. I am using Octave3.6.4_gcc4.6.2 for MinGW on a Win7 pro machine. I noticed that mkdir fails if the subfolder existed and contained several files. It seems like rmdir has not completed in the background before the next lines of code are executed. Below is a sample of the test code.
parentDir = 'C:\Temp\rmDir';
childDir = fullfile(parentDir, 'output');
if (exist(childDir, 'dir') ~= 0)
[status] = rmdir(childDir, 's');
disp(status);
end;
[status] = mkdir(parentDir, 'output');
disp(status);
disp(exist(childDir, 'dir'));
Below is the Octave result for when the subfolder does not exist. This works as expected.
octave:5> testrmdir
1
7
Below is the Octave result for when a subfolder exists and is empty. This works as expected.
octave:6> testrmdir
1
1
7
Below is the Octave result for when a subfolder exists and contains 3 PNG files with a total size of 349 KB. Status is 1 for both mkdir and rmdir. However, the exist function reports that the folder does not exist. I confirm from windows explorer that the subfolder is deleted. My guess is that when mkdir executes, the files are still being deleted by the prior rmdir function. So mkdir reports success because the subfolder has not been deleted by rmdir yet. However, by the time exist is executed rmdir has completed and so the subfolder no longer exists.
octave:7> testrmdir
1
1
0
I tried different file types with the following results:
2 PNG files, 232 KB total - pass
4 PNG files, 465 KB total - fail
3 PNG files, 349 KB total - fail
3 csv files, 518 KB total - pass
5 csv files, 777 KB total - fail
The behavior is the same when I run Octave from the command line. I have used the same code on MATLAB in the past without any noticeable issues. For now, I had to switch to Octave for test automation on a different machine.
Does this make sense? Any suggestions on how to make this code work regardless of the subfolder contents or size?
Not sure if this is important, but I have the following setting in the resource file: confirm_recursive_rmdir(false).
I changed the if statement to a while loop and this fixed the problem (i.e. all I did was replace "if" with "while"). Then I added a counter in the while loop and saw that rmdir was successful on the first iteration. Therefore, I cannot explain why the code does not work with an if statement. See expanded code with new counter below. But like I said, the code also works if I simply replace "if" in the original code with "while".
parentDir = 'C:\Temp\rmDir';
childDir = fullfile(parentDir, 'output');
count = 0;
while (exist(childDir, 'dir') ~= 0)
%if (exist(childDir, 'dir') ~= 0)
count++
[status] = rmdir(childDir, 's');
disp(status);
disp(count);
end;
[status] = mkdir(parentDir, 'output');
disp(status);
disp(exist(childDir, 'dir'));
When mftraining is executed on my training files, I get the following error message:
PS > mftraining -F font_properties -U unicharset -O lang.unicharset .\eng.ds-digita
l.exp0.box.tr .\eng.ds-digitalb.exp0.box.tr .\eng.ds-digitali.exp0.box.tr
Warning: No shape table file present: shapetable
Reading .\eng.ds-digital.exp0.box.tr ...
Reading .\eng.ds-digitalb.exp0.box.tr ...
Reading .\eng.ds-digitali.exp0.box.tr ...
Font id = -1/0, class id = 1/12 on sample 0
font_id >= 0 && font_id < font_id_map_.SparseSize():Error:Assert failed:in file ..\..\classify\trainingsampleset.cpp, li
ne 622
A dialog from Windows also appears stating "feature training for Tesseract has stopped working". There are several posts around the net adressing this issue, but none of them (That I have tried so far) seems have any solutions to make my data-set go through.
The folder where the mftraining command is executed at contains the following files:
eng.ds-digital.exp0.box
eng.ds-digital.exp0.box.tr
eng.ds-digital.exp0.box.txt
eng.ds-digital.exp0.tif
eng.ds-digitalb.exp0.box
eng.ds-digitalb.exp0.box.tr
eng.ds-digitalb.exp0.box.txt
eng.ds-digitalb.exp0.tif
eng.ds-digitali.exp0.box
eng.ds-digitali.exp0.box.tr
eng.ds-digitali.exp0.box.txt
eng.ds-digitali.exp0.tif
font_properties
unicharset
And the font_properties has the following content (It also ends with a newline as the documentation states):
ds-digital 0 0 0 0 0
ds-digitalb 0 1 0 0 0
ds-digitali 1 0 0 0 0
I've also tried different naming conventions on the font-name on the font_properties (althought the documentation is quite clear it is the font name of the file and not the file name, but some people around the net seems to claim otherwise), and renaming the files so the .tr-files follows the pattern eng.ds-digital*.exp0.tr without anvil.
Edit: I am running on Tesseract 3.02
I was getting same issue and resolved by checking Font name in eng.ds-digital.exp0.box.tr should be same as you given in font_properties file.
Example:
echo "ds-digital 0 0 0 0 0" > font_properties
then eng.ds-digital.exp0.box.tr should have ds-digital font name.
another easy way to train tesseract link.
i properly asking a question that do not have an answer, but i am out of resources and have to try any mean that possible to have a way to solve this problem. first off. i ain't programmer, i just an online store staff that assign to deal with the server and stuff like that. i have basic knowledge to cpanel, whm, but no experience in programming.
we see our server is very very slow recently, i go in whm and see the cpu usage constantly stay like 200% up to 2000% (yes, 2000%) from time to time, then i looked into the process manager and see one particular line that border me:
pid991 - this use like at least 100% of the cpu most of the time i look and the command is
/usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --plugin-dir=/usr/lib64/mysql/plugin --user=mysql --log-error=/var/lib/mysql/www.(domain).com.err --pid-file=/var/lib/mysql/www.(domain).com.pid
and if i go into trace, i see this code repeat again and again
>fcntl(12, F_GETFL) = 0x2 (flags O_RDWR)
>fcntl(12, F_SETFL, O_RDWR|O_NONBLOCK) = 0
>accept(12, {sa_family=AF_FILE, NULL}, [2]) = 95
>fcntl(12, F_SETFL, O_RDWR) = 0
>getsockname(95, {sa_family=AF_FILE, path="/var/lib/mysql/mysql.sock"}, [28]) = 0
>gettimeofday({1380692546, 638961}, NULL) = 0
>fcntl(95, F_SETFL, O_RDONLY) = 0
>fcntl(95, F_GETFL) = 0x2 (flags O_RDWR)
>setsockopt(95, SOL_SOCKET, SO_RCVTIMEO, "\36\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 16) = 0
>setsockopt(95, SOL_SOCKET, SO_SNDTIMEO, "<\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0", 16) = 0
>fcntl(95, F_SETFL, O_RDWR|O_NONBLOCK) = 0
>setsockopt(95, SOL_IP, IP_TOS, [8], 4) = -1 EOPNOTSUPP (Operation not supported)
>gettimeofday({1380692546, 641571}, NULL) = 0
>clone(child_stack=0x7ff0ead58ff0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7ff0ead599d0, tls=0x7ff0ead59700, child_tidptr=0x7ff0ead599d0) = 6590
>poll([{fd=10, events=POLLIN}, {fd=12, events=POLLIN}], 2, -1) = 1 ([{fd=12, revents=POLLIN}])
with only difference in the number string and may be some ids. i actually have a programmer here, but he is a freelancer and keep tell us he doesn't know what happen, so...
can we know what this is doing with only these repeat lines of ode?
there is a kill button in process manager, is it safe to simply kill the process?
thank you for your help!
The trace messages do not identify the cause of your performance problem. See https://serverfault.com/questions/374120/setsockopt-eopnotsupp-operation-not-supported
It's not safe to simply kill that process; that is your MySQL database, so I expect that if you kill the process, your website will go down.
It's quite likely that you have long-running queries or just a huge volume of database activity on that server, so you'll need to investigate in that direction rather than in the trace. Start with SHOW PROCESSLIST in the MySQL console to see what's currently running and for how long, and that should point you in the direction of statements and connections to look at.
The Octave documentation on this subject is both intimidating and sparse.
I did not know where else to document the solution I found, so I am posting here. I apologize if that's inappropriate, but I want to help the next guy.
The following solution is for a simple windows distributable.
Use Case:
A solution is developed in Octave 3.2.4, and needs to be distributed to end-users with few computer skills. Installing and explaining Octave is impossible, the solution must be "one-click" or "brain-dead-simple."
Known Issues:
imread fails in 3.2.4 because file_in_path.m is wrong. You will need to update the file file_in_path.m to the following (just replace it):
function name=file_in_path(p,file)
idx=[1 findstr(p,pathsep) length(p)+1];
for i=1:length(idx)-1
if idx(i+1)-idx(i)<=1
dir=strcat(pwd,"/");
else
dir=p(idx(i)+1:idx(i+1)-1);
end
name = fullfile(dir, file);
fid = fopen(name,"r");
if fid >= 0
fclose(fid);
return
end
end
fid = fopen(file,"r");
if fid >= 0,
fclose(fid);
name=file;
return
end
name=[];
Solution: Create a distributable exe using mkoctfile, and package this exe with the core Octave files, and other .oct and .m files as necessary.
Step 1: Create a stand-alone executable.
You can see code that works here:
http://www.gnu.org/software/octave/doc/interpreter/Standalone-Programs.html
Particularly the file "embedded.cc".
I have simplified that file as follows:
#include <iostream>
#include <octave/oct.h>
#include <octave/octave.h>
#include <octave/parse.h>
int
main (int argc, char *argvc[])
{
string_vector argv (2);
argv(0) = "embedded";
argv(1) = "-q";
octave_main (2, argv.c_str_vec(), 1);
octave_value_list in = octave_value (argvc[1]);
octave_value_list out = feval ("your_custom_m_file", in);
if (!error_state && out.length () > 0)
{
}
else
{
std::cout << "invalid\n";
}
return 0;
}
Build this file with the command
mkoctfile --link-stand-alone embedded.cc -o embedded
It may throw warnings, but as long as it throws no errors, you should be fine. The file embedded.exe will be built, and can be run. The only issue is that it will lack all the goodies that make octave awesome. You will have to provide those.
Step 2: Create a distribution folder
You will need to create a copy of many of the Octave files. I suggest a directory specifically for this. At a minimum, you will need a copy of all or most of the DLLs in \bin. Additionally, place your distributable executable in this directory.
Step 3: Other files whack-a-mole
You will now need to find out what other files will be necessary to run your .m script. You can simplify this step by copying \oct\i686-pc-mingw32*.oct and \share\octave\3.2.4\m\*\*.m to the distribution directory, although this will be overkill, and will not actually prevent the whack-a-mole step.
Now, you must play whack-a-mole or the time-honored tradition of "where my includes be at, yo?"
Open a cmd prompt and navigate to your distribution folder.
Get rid of any useful PATH strings. Your customers won't have them.
Attempt to run the program embedded.exe. You will get an error such as the following:
embedded.exe
error: `max' undefined near line 83 column 22
error: evaluating argument list element number 1
error: evaluating argument list element number 1
error: called from:
error: T:\sms\Development\research\c2\disttest\strcat.m at line 83, column 3
error: T:\sms\Development\research\c2\disttest\file_in_path.m at line 5, column 10
error: T:\sms\Development\research\c2\disttest\imread.m at line 50, column 6
A Search in your Octave installation for "max". It will either be a .oct or a .m file. In this case, it is a .oct file, max.oct. Copy it to your distribution directory.
B You search for something obvious like "min", and get no results. This is because the Loadable Function "min" is in the .oct file "max.oct". Make a copy of max.oct, and rename it to min.oct. It will work now. How do you know where the functions are? I'm not sure. Most of them are in obvious places like "max.oct" for min, and "fft2.oct" for "ifft2.oct". Good luck with all that.
Repeat until your executable runs.
Just to add that if you want to run a script instead of an m function, then the line of the embedded.cc:
octave_value_list out = feval ("your_custom_m_file", in);
should be:
octave_value_list out = feval ("your_custom_m_script");
Also use 'which' to find where the missing functions are packed. For example for the min function:
octave:22> which min
min is a function from the file C:\Octave\Octave3.6.2_gcc4.6.2\lib\octave\3.6.2\oct\i686-pc-mingw32\max.oct
Something I found when linking my custom m file into an Octave standalone:
Needed #include <octave/toplev.h>
Replace return 0; (as above) with clean_up_and_exit(0);
Without these steps my program repeatedly crashed on exit.
Run mkoctfile --link-stand-alone embedded.cc -o embedded
from the octave solution and not from a batch file.
Just saved you half day (-;
In the above solution in bullet 4 B:
B You search for something obvious like "min", and get no results.
This is because the Loadable Function "min" is in the .oct file
"max.oct". Make a copy of max.oct, and rename it to min.oct. It will
work now.
This might not work if some function is being called from #folder function.m and also to avoid unnecessary duplicated files, just add the following code somewhere in your m file outside #folder
autoload ("min", "max.oct");
Likewise, it can be removed via
autoload ("min", "max.oct", "remove");
Ensure that the path to max.oct is provided here.
The above understanding is based on a file PKG_ADD and PKG_DEL in the communications package located at \Octave-4.0.1\lib\octave\packages\communications-1.2.1\i686-w64-mingw32-api-v50+\
Check out Stratego Octave Compiler.
(I've not tested it yet, but plan to do so in the next few days.)
I had that very same requirement (one-click, brain-dead-simple), so I made a setup that contained only curl.exe, the batch file below, an exe which was a .bat in disguise (simply calling the batch file below) and the .vbs script below (not writen by me). And of course my m-file.
This will download Octave 4.2.1 as a portable program (32 bit, otherwise we'dd have to download again if the system turns out to be 32 bit), unpack using the vbs script, move the contents to the same folder as the batch file and run it in GUI mode. Every next time the same script is called, it will only check if octave.bat is still there.
Of course this results in a huge waste of disk space, downloading the 280MB zip, which unpacks to over 1GB (which I make even worse by not deleting the zip afterwards), and you're stuck with a cmd window that is not easy to hide.
But it does offer the simplest solution I could find. It is also less likely to break in the future (either with an update of your own, or an update from Octave). Some glorious day, mkoktfile will actually be easy to use and will solve dependencies on its own, but until that day this remains the least headache-inducing solution I could find. And aspirins are more expensive than someone else's disk space.
::this file will test if the octave portable is downloaded and unpacked
#ECHO OFF
SET my_m_file=your_mfile.m
SET name_of_this_script=run_me.bat
::if the file exists, skip to the actual running.
IF EXIST "octave.bat" goto OctaveIsExtracted
IF EXIST "octave-4.2.1-w32.zip" goto OctaveIsDownloaded
ECHO The runtime (Octave portable 4.2.1) will now be downloaded.
ECHO This may take a long time, as it is about 280MB.
ECHO .
ECHO If this download restarts multiple times, you can manually download the octave-4.2.1-w32.zip from the GNU website. Make sure to unpack the contents.
::if this errors, you can uncomment the line with archive.org (which doesn't report total size during download)
curl http://ftp.gnu.org/gnu/octave/windows/octave-4.2.1-w32.zip > octave-4.2.1-w32.zip
::curl http://web.archive.org/web/20170827205614/https://ftp.gnu.org/gnu/octave/windows/octave-4.2.1-w32.zip > octave-4.2.1-w32.zip
:OctaveIsDownloaded
::check to see if the file size is the correct size to assume a successful download
::if the file size is incorrect, delete the file, restart this script to attempt a new download
::file size should be 293570269 bytes
call :filesize octave-4.2.1-w32.zip
IF /I "%size%" GEQ "293560000" goto OctaveIsDownloadedSuccessfully
del octave-4.2.1-w32.zip
::start new instance and exit and release this one
start %name_of_this_script%
exit
:OctaveIsDownloadedSuccessfully
IF EXIST "octave.bat" goto OctaveIsExtracted
::unzip and move those contents to the current folder
ECHO Unzipping octave portable, this may take a moment.
cscript //B j_unzip.vbs octave-4.2.1-w32.zip
SET src_folder=octave-4.2.1
SET tar_folder=%cd%
for /f %%a IN ('dir "%src_folder%" /b') do move %src_folder%\%%a %tar_folder%
pause
:OctaveIsExtracted
octave.bat %my_m_file%
goto :eof
:filesize
set size=%~z1
exit /b 0
And j_unzip.vbs
' j_unzip.vbs
'
' UnZip a file script
'
' By Justin Godden 2010
'
' It's a mess, I know!!!
'
' Dim ArgObj, var1, var2
Set ArgObj = WScript.Arguments
If (Wscript.Arguments.Count > 0) Then
var1 = ArgObj(0)
Else
var1 = ""
End if
If var1 = "" then
strFileZIP = "example.zip"
Else
strFileZIP = var1
End if
'The location of the zip file.
REM Set WshShell = CreateObject("Wscript.Shell")
REM CurDir = WshShell.ExpandEnvironmentStrings("%%cd%%")
Dim sCurPath
sCurPath = CreateObject("Scripting.FileSystemObject").GetAbsolutePathName(".")
strZipFile = sCurPath & "\" & strFileZIP
'The folder the contents should be extracted to.
outFolder = sCurPath
'original line: outFolder = sCurPath & "\"
WScript.Echo ( "Extracting file " & strFileZIP)
Set objShell = CreateObject( "Shell.Application" )
Set objSource = objShell.NameSpace(strZipFile).Items()
Set objTarget = objShell.NameSpace(outFolder)
intOptions = 256
objTarget.CopyHere objSource, intOptions
WScript.Echo ( "Extracted." )