So I'm doing some hobby-related stuff which involves taking Fourier transforms of large real arrays which barely fit in memory, and was curious to see if there was an in-place version of rfft and irfft that saved RAM, since RAM consumption is important to me. These transforms are possible despite the input-vs-output-type mismatch, and require an extra row of padding.
In Implement in-place rfft! and irfft!, Tim Holy said he was working on an in-place rfft! and irfft! that made use of a buffer-containing RCpair object, but then Steven Johnson said that he was implementing something equivalent using A_mul_B!(y, plan, x), which he elaborated on here.
Things get a little weird from then on. In the documentation for both 0.3.0 and 0.4.0 there is no mention of A_mul_B!, although A_mul_B is listed. But when I try entering them into Julia, I get
A_mul_B!
A_mul_B! (generic function with 28 methods)
A_mul_B
ERROR: A_mul_B not defined
which suggests that the situation is actually the opposite of what the documentation currently describes.
So since A_mul_B! seems to exist, but isn't documented anywhere, I tried to guess how to test it in-place as follows:
A = rand(Float32, 10, 10);
p = plan_rfft(A);
A_mul_B!(A,p,A)
which resulted in
ERROR: `A_mul_B!` has no method matching A_mul_B!(::Array{Float32,2}, ::Function, ::Array{Float32,2})
So...
Are in-place real FFTs still a work in progress? Or am I using A_mul_B! wrong?
Is there a mismatch between the 0.3.0 documentation and 0.3.0's function library?
That pull request from Steven Johnson is listed as open, not merged; that means the work hasn't been finished yet. The one from me is closed, but if you want the code you can grab it by clicking on the commits.
The docs indeed omit mention of A_mul_B!. A_mul_B is equivalent to A*B, and so isn't exported independently now. A_mul_B! would be used like this: instead of C = A*B, you could say A_mul_B!(C, A, B).
Can you please edit the docs to fix these issues? (You can edit files here in your webbrowser.)
Related
While finalizing my upcoming Raku Advent Calendar post on sigils, I decided to double-check my understanding of the type constraints that sigils create. The docs describe sigil type constraints with the table
below:
Based on this table (and my general understanding of how sigils and containers work), I strongly expected this code
my %percent-sigil is List = 1,2;
my #at-sigil is Map = :k<v>;
to throw an error.
Specifically, I expected that is List would attempt to bind the %-sigiled variable to a List, and that this would throw an X::TypeCheck::Binding error – the same error that my %h := 1,2 throws.
But it didn't error. The first line created a List that seemed perfectly ordinary in every way, other than the sigil on its variable. And the second created a seemingly normal Map. Neither of them secretly had Scalar intermediaries, at least as far as I could tell with VAR and similar introspection.
I took a very quick look at the World.nqp source code, and it seems at least plausible that discarding the % type constraint with is List is intended behavior.
So, is this behavior correct/intended? If so, why? And how does that fit in with the type constraints and other guarantees that sigils typically provide?
(I have to admit, seeing an %-sigiled variable that doesn't support Associative indexing kind of shocked me…)
I think this is a grey area, somewhere between DIHWIDT (Docter, It Hurts When I Do This) and an oversight in implementation.
Thing is, you can create your own class and use that in the is trait. Basically, that overrides the type with which the object will be created from the default Hash (for %) and Array (for # sigils). As long as you provide the interface methods, it (currently) works. For example:
class Foo {
method AT-KEY($) { 42 }
}
my %h is Foo;
say %h<a>; # 42
However, if you want to pass such an object as an argument to a sub with a % sigil in the signature, it will fail because the class did not consume the Associatve role:
sub bar(%) { 666 }
say bar(%h);
===SORRY!=== Error while compiling -e
Calling bar(A) will never work with declared signature (%)
I'm not sure why the test for Associative (for the % sigil) and Positional (for #) is not enforced at compile time with the is trait. I would assume it was an oversight, maybe something to be fixed in 6.e.
Quoting the Parameters and arguments section of the S06 specification/speculation document about the related issue of binding arguments to routine parameters:
Array and hash parameters are simply bound "as is". (Conjectural: future versions ... may do static analysis and forbid assignments to array and hash parameters that can be caught by it. This will, however, only happen with the appropriate use declaration to opt in to that language version.)
Sure enough the Rakudo compiler implemented some rudimentary static analysis (in its AOT compilation optimization pass) that normally (but see footnote 3 in this SO answer) insists on binding # routine parameters to values that do the Positional role and % ones to Associatives.
I think this was the case from the first official Raku supporting release of Rakudo, in 2016, but regardless, I'm pretty sure the "appropriate use declaration" is any language version declaration, including none. If your/our druthers are static typing for the win for # and % sigils, and I think they are, then that's presumably very appropriate!
Another source is the IRC logs. A quick search quickly got me nothing.
Hmm. Let's check the blame for the above verbiage so I can find when it was last updated and maybe spot contemporaneous IRC discussion. Oooh.
That is an extraordinary read.
"oversight" isn't the right word.
I don't have time tonight to search the IRC logs to see what led up to that commit, but I daresay it's interesting. The previous text was talking about a PL design I really liked the sound of in terms of immutability, such that code could become increasingly immutable by simply swapping out one kind of scalar container for another. Very nice! But reality is important, and Jonathan switched the verbiage to the implementation reality. The switch toward static typing certainty is welcome, but has it seriously harmed the performance and immutability options? I don't know. Time for me to go to sleep and head off for seasonal family visits. Happy holidays...
I am using Octave version 4.2.2, but I think the question applies to previous versions as well.
I want to know if the following behavior is well-known and caused by my ignorance or if it is a bug that should be addressed, and also if there is a workaround. Note that I am focusing on parallelization in situations where vectorization is not possible, and where copy by values are not an option.
Basically, my problem is that functions such as pararrayfun or parcellfun seem to violate the principle that the properties of handles are passed by reference.
As an example, suppose we define
classdef data_class < handle
properties
data
end
end
and that we want to take each element from the .data property of one object input_arr from this class, apply some stochastic function fancy_func to them, and copy the result to the corresponding index of the .data property of another object arr of this class. The .data property is just a matrix of size 12*1, and we want to have 4 processes that each process 3 elements.
elem_per_process=3;
num_processes=4;
start_indexes={1,4,7,10};
function []=fill_arr(start_idx,num_elems,in_arr_handle,arr_to_fill_handle)
for i=1:num_elems
arr_to_fill_handle.data(start_idx+i-1,:)=fancy_func(in_arr_handle.data(start_idx+i-1,:));
end
end
filler=#(start_idx)fill_arr(start_idx,elem_per_process,input_arr,arr);
cellfun(filler,start_indexes);%works fine
parcellfun(num_processes,filler,start_indexes);% PROBLEM! Nothing is copied
So, this is actually worse that what I thought:
parcellfun copies object handle properties by value
It breaks code that works with cellfun !
A quick look at
<octave_dir>/packages/parallel-3.1.1/parcellfun.m
seems to indicate that the parallelization relies on processes instead of threads (also note pararrayfun uses parcellfun at its core), so this is expected. What bothers me is that it does so without a single warning, while going against the fundamental properties that many Octave users expect to hold.
So, to summarize:
Is this as a (minor) bug?
Is there any way to parallelize using threads instead of processes (for situations where vectorization is not possible, and where copy by value is unacceptable)?
I am wondering which version is the best one to implement.
The parameters are states that have 2 possible values.
This is an abstract example of the actual problem.
I am programming in a language that is procedural (without classes) and does not have typed variable.
I just read an article stating that version 1 is bad for readability and the caller. Personally I don't like version 2 either. Maybe there is a better option?
Version 1:
doSth(par1, par2)
Not redundant +
Single Method for a task +
More complex implementation -
Wrong parameters can be passed easily -
Version 2:
doSthWithPar1Is1AndPar2Is1()
doSthWithPar1Is1AndPar2Is2()
doSthWithPar1Is2AndPar2Is1()
doSthWithPar1Is2AndPar2Is2()
Redundant -
Too many methods (especially with more parameters) -
Long Method Names -
Simple implementation +
No parameters that could be passed wrong +
Given that you already have considered V1 feasible tells me, that the different argument value combinations have something in common with regards to how the values are to be processed.
In V2 you simply have to type and read more, which I'd say is the single most frequent reason for introducing errors/incorrectness and lose track of your requirements.
In V2 you have to repeat what is common in the individual implementations and if you make a mistake, the overall logic will be inconsistent at best. And if you want to fix it, you probably have to fix it in several places.
But, you can optimize code safety based on V1: choose a more "verbose" name for the procedure, like
doSomethingVerySpecificWithPar1OfTypeXAppliedToPar2OfTypeY(par1, par2)
(I am exaggerating a bit...) so you see immediately what you have originally intended.
You could even take the best out of V2 and introduce the individual functions, which simply redirect to the common function of V1 (so you avoid the redundancy). The gain in clarity almost always outweighs the slight loss of efficiency.
doSthWithPar1Is1AndPar2Is1()
{
doSomethingVerySpecificWithPar1OfTypeXAppliedToPar2OfTypeY(1, 1);
}
Always remember David Wheeler: "All problems in computer science can be solved by another level of indirection".
Btw: I don't consider long method names a problem but rather a benefit (up to a certain length of course).
I'm working on some lisp code that munges a CVS repository in order to get it into shape for a git conversion. As such, when something goes wrong it might not be a bug in my code but might instead be an inaccuracy in the input data which needs fixing.
Deep down in the bowels of my code, I'll have (say) some function that merges two date ranges by taking an intersection. If that intersection turns out to be empty, I'll raise an error. This is all good, but my "merge dates" function is missing lots of the information that the user (me!) needs in order to figure out what went wrong. For example, which CVS master file ("foo,v") was I working on? What branch was I thinking about? Etc. etc.
My partial solution to this problem is to handle and re-raise the error. For example, this sort of code:
(handler-case
(do-something)
(unclear-graft-point (c)
(setf (slot-value c 'master) master)
(setf (slot-value c 'branch) branch)
(error c)))
which sets some extra slots with useful info before re-raising the error.
The report function for the condition then checks whether those slots have been set and, if so, will use them to give a more helpful error message.
Unfortunately, the backtrace that I get stops at the top-level re-raise. That makes sense: it's the code I wrote. But it's not really what I want...
Is there a way in Common Lisp to "annotate" a condition as it hurtles past, without re-raising it and, hence, partially unwinding the stack without reporting that in the back trace?
I gave a reasonable amount of background info about what I'm doing in the hope that, if this isn't possible, someone will say "Ah, what you should do in this situation is...": am I just going about this the wrong way?
Yes, there is a Common Lisp technology applicapble to your case - it is called HANDLER-BIND. You can find a detailed explanation and use cases for it alongside other condition-handling machinery in Peter Seibel's Practical Common Lisp Chapter 19. Conditions and Restarts.
So I'm creating a vim script that needs to load and parse a JSON file into a local object graph. I searched and I couldn't find any native way to process a JSON file, and I don't want to add any dependencies to the script. So I wrote my own function to parse the JSON string (gotten from the file), but it's really slow. At the moment, I iterate through each character in the file like so:
let len = strlen(jsonString) - 1
let i = 0
while i < len
let c = strpart(jsonString, i, 1)
let i += 1
" A lot of code to process file....
" Note: I've tried short cutting the process by searching for enclosing double-quotes when I come across the initial double quotes (also taking into account escaping '\' character. It doesn't help
endwhile
I've also tried this method:
for c in split(jsonString, '\zs')
" Do a lot of parsing ....
endfor
For reference, a file with ~29,000 characters takes about 4 seconds to process, which is unacceptable.
Is there a better way to iterate over a string in vim script?
Or better yet, have I missed a native function to parse JSON?
Update:
I asked for no dependencies because I:
Didn't want to deal with them
Genuinely wanted some ideas for best way to do this without someone else's work.
Sometimes I just like to do things manually even though the problem has already been solved.
I'm not against plugins or dependencies at all, it's just that I'm curious. Thus the question.
I ended up creating my own function to parse the JSON file. I was creating a script that could parse the package.json file associated with node.js modules. Because of this, I could rely on a fairly consistent format and quit the processing whenever I'd retrieved the information I needed. This usually cut out large chunks of the file since most developers put the largest chunk of the file, their "readme" section, at the end. Because the package.json file is strictly defined, I left the process somewhat fragile. It assumed a root dictionary { } and actively looks for certain entries. You can find the script here: https://github.com/ahayman/vim-nodejs-complete/blob/master/after/ftplugin/javascript.vim#L33.
Of course, this doesn't answer my own question. It's only the solution to my unique problem. I'll wait a few days for new answers and pick the best one before the bounty ends (already set an alarm on my phone).
The simplest solution with the least dependencies is just using the json_decode vim function.
let dict = json_decode(jsonString)
Even though Vim's origin dates back a lot it happens that its internal string() eval() representation is that close to JSON that its likely to work unless you need special characters.
You can lookup the implementation here which even supports true/false/null if you want:
https://github.com/MarcWeber/vim-addon-json-encoding
Better use that library (vim-addon-manager allows to install dependencies easily).
Now it depends on your data whether this is good enough.
Now Benjamin Klein posted your question to vim_use which is why I'm replying.
Best and fast replies happen if you subscribe to the Vim mailinglist.
Goto vim.sf.net and follow the community link.
You cannot expect the Vim community to scrape stackoverflow.
I've added the keyword "json" and "parsing" to that little code that it can be found easier.
If this solution does not work for you you can try the many :h if_* bindings or write an external script which extracts the information you're looking for, or turns JSON into Vim's dictionary representation which can be read by eval() escaping special characters you care about correctly.
If you seek for completely correct solution omitting dependencies is one of the worst thing you can do. The eval() variant mentioned by #MarcWeber is one of the fastest, but it has its disadvantages:
Using solution for securing eval I mentioned in comment makes it no longer the fastest. In fact after you use this it makes eval() slower by more then an order of magnitude (0.02s vs 0.53s in my test).
It does not respect surrogate pairs.
It cannot be used to verify that you have correct JSON: it accepts some strings (e.g. "\<C-o>") that are not JSON strings and it allows trailing commas.
It fails to give normal error messages. It fails badly if you use vam#VerifyIsJSON I mentioned in p.1.
It fails to load floating point values like 1e10 (vim requires numbers to look like 1.0e10, but numbers like 1e10 are allowed: note “and/or” in the first paragraph).
. All of the above (except for the first) statements also apply to vim-addon-json-encoding mentioned by #MarcWeber because it uses eval. There are some other possibilities:
Fastest and the most correct is using python: pyeval('json.loads(vim.eval("varname"))'). Not faster then eval, but fastest among other possibilities. (0.04 in my test: approximately two times slower then eval())
Note that I use pyeval() here. If you want solution for vim version that lacks this functionality it will no longer be one of the fastest.
Use my json.vim plugin. It has an advantages of slightly better error reporting compared to failed vam#VerifyIsJSON, slightly worse compared to eval() and it correctly loads floating-point numbers. It can be used for verification of strings (it does not accept "\<C-a>"), but it loads lists with trailing comma just fine. It does not support surrogate pairs. It is also very slow: in the test I used (it uses 279702 character long strings) it takes 11.59s to load. Json.vim tries to use python if possible though.
For the best error reporting you can take yaml.vim and purge YAML support out of it leaving only JSON (I once have done the same thing for pyyaml, though in python: see markedjson library used in powerline: it is pyyaml minus YAML stuff plus classes with marks). But this variant is even slower then json.vim and should only be used if the main thing you need is error reporting: 207 seconds for loading the same 279702 character long string.
Note that the only variant mentioned that satisfies both requirements “no dependencies” and “no python” is eval(). If you are not fine with its disadvantages you have to throw away one or both of these requirements. Or copy-paste code. Though if you take speed into account only two candidates are left: eval() and python: if you want to parse json fast you really must use C and only these solutions spend most time in functions written in C.
Most other interpreters (ruby/perl/TCL) do not have pyeval() equivalent so they will be slower even if their JSON implementation is written in C. Some other (lua/racket (mzscheme)) have pyeval() equivalent, but e.g. luaeval('{}') is zero meaning that you will have to add additional step explicitly and recursively converting objects into vim dictionaries and lists (e.g. luaeval('vim.dict({})')) which will impact performance. Cannot say anything about mzeval(), but I have never heard about anybody actually using racket (mzscheme) with vim.