I am trying to add parallel computation option to an R (netresponse) package based on doMC and multicore. The script works ok, but only on the second trial.
To reproduce the bug, start R and run the script below. It gets stuck on the last line. After interrupting with ctrl-c I get a few messages of "select: Interrupted system call". Then, running the same script again will give the expected result without problems.
Is some further initialization needed to get this work properly already on the first run? Or any other tips?
thanks for your support,
- L
require(netresponse)
require(multicore)
require(doMC)
registerDoMC(3)
print(getDoParWorkers())
res <- foreach(i = 1:100, .combine = cbind,
.packages = "netresponse") %dopar% netresponse::vdp.mixt(matrix(rnorm(1000), 100, 10))
Heres the list of dependencies from the help page for package netresponse: "Depends: methods, igraph, graph, minet". I suspect that you are not getting all of them to the workers by just listing "netresponse" on the .packages argument.
Quick fix for problem with foreach %dopar% is to reinstall these packages:
install.packages("doSNOW")
install.packages("doParallel")
install.packages("doMPI")
As mentioned in various threads at StackOverflow, these are responsible for parallelism in R. Bug which existed in old versions of these packages is now removed. I should mention that it will most likely help even though you are not using these packages in your project/package.
Related
Say I have inside my JRuby program the following loop:
loop do
x=foo()
break if x
bar()
end
and I want to collect profiling information just for the invocations of bar. How to do this? I got so far:
pd = []
loop do
x=foo()
break if x
pd << JRuby::Profiler.profile { bar() }
end
This leaves me with an array pd of profile data objects, one for each invocation of bar. Is there a way to create a "summary" data object, by combining all the pd elements? Or even better, have a single object, where profile would just add to the existing profiling information?
I googled for a documentation of the JRuby::Profiler API, but couldn't find anything except a few simple examples, none of them covering my case.
UPDATE : Here is another attempt I tried, which does not work either.
Since the profile method initially clears the profile data inside the Profiler, I tried to separate the profiling steps from the data initializing steps, like this:
JRuby::Profiler.clear
loop do
x=foo()
break if x
JRuby::Profiler.send(:current_thread_context).start_profiling
bar()
JRuby::Profiler.send(:current_thread_context).stop_profiling
end
profile_data = JRuby::Profiler.send(:profile_data)
This seems to work at first, but after investigation, I found that profile_data then contains the profiling information from the last (most recent) execution of bar, not of all executions collected together.
I figured out a solution, though I have the feeling that I'm using a ton of undocumented features to get it working. I also must add that I am using (1.7.27), so later JRuby versions might or might not need a different approach.
The problem with profiling is that start_profiling (corresponding to the Java method startProfiling in the class Java::OrgJrubyRuntime::ThreadContext) not only turns on the profiling flag, but also allocates a fresh ProfileData object. What we want to do, is to reuse the old object. stop_profiling OTOH only toggles the profiling switch and is uncritical.
Unfortunately, ThreadContext does not provide a method to manipulate the isProfiling toggle, so as a first step, we have to add one:
class Java::OrgJrubyRuntime::ThreadContext
field_writer :isProfiling
end
With this, we can set/reset the internal isProfiling switch. Now my loop becomes:
context = JRuby::Profiler.send(:current_thread_context)
JRuby::Profiler.clear
profile_data_is_allocated = nil
loop do
x=foo()
break if x
# The first time, we allocate the profile data
profile_data_is_allocated ||= context.start_profiling
context.isProfiling = true
bar()
context.isProfiling = false
end
profile_data = JRuby::Profiler.send(:profile_data)
In this solution, I tried to keep as close as possible to the capabilities of the JRuby::Profiler class, but we see, that the only public method still used is the clear method. Basically, I have reimplemented profiling in terms of the ThreadContext class; so if someone comes up with a better way to solve it, I will highly appreciate it.
I tried to use remotecall() in julia to distribute work to specific processor. The function I like to run does not have any return but it will output something. I can not make it work as there is no output file after running the code.
This is the test code I am creating:
using DelimitedFiles
addprocs(4) # add 4 processors
#everywhere function test(x) # Define the function
print("hi")
writedlm(string("test",string(x),".csv"), [x], ',')
end
remotecall(test, 2, 2) # To run the function on process 2
remotecall(test, 3, 3) # To run the function on process 3
This is the output I am getting:
Future(3, 1, 67, nothing)
And there is no output file (csv), or "hi" shown
I wonder if anyone can help me with this or I did anything wrong. I am fairly new to julia and have never used parallel processing.
The background is I need to run a big simulation (A big function with bunch of includes, but no direct return outputs) lots of times, and I like to split the work to different processors.
Thanks a lot
If you want to use a module function in a worker, you need to import that module locally in that worker first, just like you have to do it in your 'root' process. Therefore your using DelimitedFiles directive needs to occur "#everywhere" first, rather than just on the 'root' process. In other words:
#everywhere using DelimitedFiles
Btw, I am assuming you're using a relatively recent version of Julia and simply forgot to add the using Distributed directive in your example.
Furthermore, when you perform a remote call, what you get back is a "Future" object, which is a way of allowing you to obtain the 'future results of that computation' from that worker, once they're finished. To get the results of that 'future computation', use fetch.
This is all very simplistic and general information, since you haven't provided a specific example that can be copy / pasted and answered specifically. Have a look at the relevant section in the manual, it's fairly clearly written: https://docs.julialang.org/en/v1/manual/parallel-computing/#Multi-Core-or-Distributed-Processing-1
I'm trying to add some lua functionality to my existing conky setup so that repetitive "code" in my conky text can be cleaned up. For example, I have information for each mounted FS, each core, etc. where each row displayed in my panel differs ONLY by one parameter.
My first skeletal, attempt at using lua functions for this seems to run but displays nothing in my panel. I've only found very simple examples to base this on, so I may have made a simple error, but I don't even know how to diagnose it. My code here is modeled after what I HAVE been able to find regarding writing functions, such as this How to implement a basic Lua function in Conky? , but that's about all the depth I've found on the topic except for drawing and cairo examples.
Here's the code added to my conky config, as well as the contents of my functions.lua file
conky.config = {
...
lua_load = '/home/conky-manager/MyConky/functions.lua',
};
conky.text = [[
...
${voffset 5}${lua conky_test 'test'}
...
]]
file - functions.lua
function conky_test(parm1)
return 'result text'
end
What I would expect is to see is "result text" displayed in my panel at the location where that function call appears, but nothing shows.
Is there a log created by conky as it runs, or a way to provide some debug output? Even if I'd made a simple error here, I'd still like to have the ability to diagnose things as my code gets more complex.
Success!
After cobbling info from several articles together, I figured out my basic flaws -
1. Missing a 'conky_main' function,
2. Missing a 'lua_draw_hook_post' to invoke it, and
3. Realizing that if I invoke conky from a terminal, print statements in lua would appear there.
So, for anyone who sees this question and has the same issues, here's the corrected code.
conky.config = {
...
lua_load = '/home/conky-manager/MyConky/functions.lua',
lua_draw_hook_post = "main",
};
conky.text = [[
...
${lua conky_test 'test'}
...
]]
and the proper basics in my functions.lua file
function conky_test(parm1)
return 'result text'
end
function conky_main()
if conky_window == nil then
return
end
end
A few notes:
I still haven't determined if using 'lua_draw_hook_pre' instead of 'lua_draw_hook_post' makes any difference, but it doesn't seem to in this example.
Also, some examples showed actually calling this 'test' function instead of writing a 'main', but the 'main' seemed to have value in checking to see if conky_window existed.
Some examples seemed to state that naming functions with the prefix 'conky_' was required, but then showed examples of calling those functions without the prefix, so I assume the prefix is inferred during the call.
a major note: you should run conky from the directory containing the lua scripts.
I have written a programme which merges two 1D arrays containing names. I print the list of arr1, arr2 and arr3.
I am using Lazarus Free Pascal v. 1.0.14 . I was wondering if anyone knows how to break the results in the dos-like window because the list is so long that I can only see the last few names in the returned results. The rest go by too fast to read.
I know I can save the resuls to file and I also use the delay command, but would like to know if there is a way to somehow break the results or slow them down or even edit the output console?
I appreciate your help.
This isn't really a programming question, because your console application should output the values without pause. Otherwise your program would become useless if you ever wanted it to run as part of another pipeline in an automated fashion.
Instead you need a tool that you wrap around your program to paginate the output if, and when, you so desire. Such tools are known as terminal pagers and the basic one that ships with Windows is called more. You execute your program and pipe the output to the more program. Like this:
C:\SomeDir>MyProject.exe <input_args> | more
You can change the code of your loop in the following way:
say you print the results by the followng loop:
for i:=0 to 250 do
WriteLn(ArrUnited[i]);
you can replace it with:
for i:=0 to 250 do
begin
WriteLn(ArrUnited[i]);
if (i mod 25) = 24 then //the code will wait for the user pressing Enter every 25 rows
ReadLn;
end;
For the future please! post MCVE in your questions otherwise everyone has to guess what your code is.
So I'm doing some hobby-related stuff which involves taking Fourier transforms of large real arrays which barely fit in memory, and was curious to see if there was an in-place version of rfft and irfft that saved RAM, since RAM consumption is important to me. These transforms are possible despite the input-vs-output-type mismatch, and require an extra row of padding.
In Implement in-place rfft! and irfft!, Tim Holy said he was working on an in-place rfft! and irfft! that made use of a buffer-containing RCpair object, but then Steven Johnson said that he was implementing something equivalent using A_mul_B!(y, plan, x), which he elaborated on here.
Things get a little weird from then on. In the documentation for both 0.3.0 and 0.4.0 there is no mention of A_mul_B!, although A_mul_B is listed. But when I try entering them into Julia, I get
A_mul_B!
A_mul_B! (generic function with 28 methods)
A_mul_B
ERROR: A_mul_B not defined
which suggests that the situation is actually the opposite of what the documentation currently describes.
So since A_mul_B! seems to exist, but isn't documented anywhere, I tried to guess how to test it in-place as follows:
A = rand(Float32, 10, 10);
p = plan_rfft(A);
A_mul_B!(A,p,A)
which resulted in
ERROR: `A_mul_B!` has no method matching A_mul_B!(::Array{Float32,2}, ::Function, ::Array{Float32,2})
So...
Are in-place real FFTs still a work in progress? Or am I using A_mul_B! wrong?
Is there a mismatch between the 0.3.0 documentation and 0.3.0's function library?
That pull request from Steven Johnson is listed as open, not merged; that means the work hasn't been finished yet. The one from me is closed, but if you want the code you can grab it by clicking on the commits.
The docs indeed omit mention of A_mul_B!. A_mul_B is equivalent to A*B, and so isn't exported independently now. A_mul_B! would be used like this: instead of C = A*B, you could say A_mul_B!(C, A, B).
Can you please edit the docs to fix these issues? (You can edit files here in your webbrowser.)