I am reading this docs. I have doubt about the content of the vector of the sections in binary. For eg Table Section here. How many items of type tabletype can be present in that vector<tabletype>. Is it 0,1 and more or 0(not present) and 1 (only one present)?
The table section is made of vector of tables: vec(table). Each table has one tabletype. From here:
Vectors are encoded with their u32 length followed
by the encoding of their element sequence.
This means that a module might have [0..2^32) tables, each with its owntabletype. That said, it is expected, that some "reasonable" limitation is placed on the number of tables (and other objects) that a program is willing to load.
If there are no tables, the section might be omitted:
Every section is optional; an omitted section is equivalent
to the section being present with empty contents.
This makes a total of three ways to have no tables: no section; empty section; section with a zero length tables vector.
As usual, when de-serializing (parse) one might expect all cases, but the when serializing (generate) it is better to have the minimum number of bytes.
Related
I am reading this docs to study wasm binary format. I am finding it very tough to understand the composition of element section. Can someone please give me an example / explanation about it ? Maybe similar to one given here
The element segments section
The idea of this section is to fill the WebAssembly.Table objects with content. Initially there was only one table, and its only possible content were indexes/ids of functions. You could write:
(elem 0 (offset (i32.const 1)) 2)
It means: during the instantiation of the instance fill index 1 of table 0 with a value of 2, like tables[0][1] = 2;. Here 2 is the index of the function the table will store.
The type of element segment above is called active nowadays, and after the instantiation it will no longer be accessible by the application (they are "dropped"). From the specs:
An active element segment copies its elements into a table during instantiation, as specified by a table index and a constant expression defining an offset into that table
So far so good. But it was recognized that there is a need for a more powerful element segment section. Introduced were the passive and the declarative element segments.
The passive segment is not used during the instantiation and it is always available at runtime (until is not dropped, by the application itself, with elem.drop). There are instructions (from the Bulk memory and table instructions proposal, already integrated into the standard) that can be used to do operations with tables and element segments.
A declarative element segment is not available at runtime but merely serves to forward-declare references that are formed in code with instructions like ref.func.
Here is the test suite, where you can see many examples of element segments (in a text format).
The binary format
Assuming that you parser the code, you read one u32, and based on its value you expect the format from specification:
0 means an active segment, as the one above, for an implicit table index of 0, and a vector of func.refs.
1 means a passive segment, the elemkind (0x00 for func.ref at this time), followed by a vector of the respective items (func.refs).
2 is an active segment.
3 means a declarative segment.
4 is an active segment where the values in the vector are expressions, not just plain indexes (so you could have (i32.const 2) in the above example, instead of 2).
5 passive with expressions
6 active with table index and expressions
7 declarative with expressions
For this reason the specs says that from this u32 [0..7] you can use its three lower bits to detect what is the format that you have to parse. For example, the 3th bit signifies "is the vector made of expressions?".
Now, all that said, it seems that the reference types proposal is not (yet) fully integrated into the specification's binary format (but seems to be in the text one). When it is you will be able to have other then 0x00 (func.ref) for an elemkind.
It is visible that some of this formats overlap, but the specification evolves, and for backward compatibility reasons with the earliest versions the format is like this today.
I need to anonymyze personal data in our MySql database. The problem is that I still need to be able to link two persons together after they have been anonymized.
I thought this could be done by hashing their social security number or e-mail address, which lead to my question:
When hashing two equal strings (s1 and s1) I get two hash values (h1 and h2), how sure can I be that:
1) the hashed value is equal (h1 = h2)
2) no not equal (s3 = s1) will produce the same hash value
1) Same strings will always produce equal hash values
2) Different strings theoretically might produce same hash if you choose small hash length compared to data volume. But using default hash lengths (32 or 40) wont cause such problems.
1) (h1 = h2) is always true for equal strings (s1 and s2) per definition, when using a correct hash function.
2) Two different strings can have the same hash value. This is called a "collsison". The probability depends on the hash function used and the length of the resulting hash. For MD5 for example there are websites and tables for finding collisions, which is quite interesting.
I'm not sure what you mean by linking persons together or what your requirements are, so I cannot help you with that. But you could link two persons together with their ids.
The question
What are the ways of coercing octave to create a real copy of whatever object? Structures are the main interest.
My underlying problem
In my problem I'm obtaining a rather large structure from another function in a loop but for the current task only a few pieces of it are needed. For example:
for i=1:many
res=solver(params);
store1{i}=res.string1;
store2{i}=res.arr(:,1);
end
res is a sizable chunk of data and due to lazy-copy those store-s are references to tiny portions of bytes in that chunk. After I store those tiny portions, I don't need res itself, however, since middle of that chunk is referenced by store, the memory area is unfit for res obtained on the next iteration (they are of the same size) and thus another sizable piece of memory is allocated, which is then again crossed by few tiny links an so on.
Without storing parts of res, the program successfully keeps the memory consumption same after first couple of iterations.
So how do I make a complete copy of structure field?
I've tried using struct-related functions like rmfield but those keep references instead of their own objects.
I've tried to wrap the assignment of in its own function:
new_struct=copy( rmfield(old_struct,"bigdata"));
function c=copy(a);
c=a;
end;
This by the way doesn't work even for arrays.
I'm interested in method applicable to any generic variable.
Minimal working example of the problem
a=cell(3,1);
for i=1:length(a);
r=rand(100000,1000);
a{i}=r(1:100,end);
whos; fflush(stdout);
pause(2);
end;
The above code will cause memory usage to gradually grow by far more than 8.08 kb reported by whos due to references stored by a{i} blocking bigger memory block than they actually need. If you force the proper copy, the problem is not present.
Numerical arrays
For numeric types addition of zero is enough to warrant a new array.
c=a+0;
Strings
For string which is 1 x n char array, something along the following lines will work:
c=[a "a"](1:end-1);
Multidimensional char arrays will require concatenation with a column:
c=[a true(size(a,1),1)](:,1:end-1);
Here true is used to generate dummy array of size compatible with char. (There seems to be no procedural method of generating char array of arbitrary size) char(zeros(size(a,1),1)) and char(true(size(a,1),1)) caused excess memory usage during their creation on some calls.
Note that empty concatenation c=[a ""]; will not result in a copying. Also it is possible to do c=[a+0 ""]; which will result in a copying due to +0 but that one infers type conversions to and from double which is 8 times larger in size. (char(zeros( doesn't seem to cause that)
Other types
In general you can use casting for the types allowed by it in order to not tailor the expressions manually as I had to do above:
typelist={"double","single","char"}; %full list of supported types is available in the link
class_of_a = typelist{ isa(a,typelist) };
c=typecast( [typecast(a,'single'); single(1)] (1:end-1), class_of_a);
Single is seemingly smallest datatype available in octave.
Note that logical is not supported by this method.
Copying structures
Apparently you'd have to write your own function to go around struct fields, copy them with above methods and recursively go to substructs.
(As it doesn't involve complexities relevant here, I'd rather leave that to be done by those who actually needs that, my own problem being solved by +0's.)
I am learning CUDA from the Udacity's course on parallel programming. In a quiz, they have a given a problem of sorting a pre-ranked variable(player's height). Since, it is a one-one correspondence between input and output array, should it not be a Map communication pattern instead of a Scatter?
CUDA makes no canonical definition of these terms, that I know of. Therefore my answer is merely a suggestion of how it might be or have been interpreted.
"Since, it is a one-one correspondence between input and output array"
This statement doesn't appear to be supported by the diagram, which shows gaps in the output array, which have no corresponding input point associated with them.
If a smaller set of values are distributed into a larger array (with resultant gaps in the output array, therefore, in which no input value corresponds to the gap location(s)), then a scatter might be used to describe that operation. Both scatters and maps have maps which describe where the input values go, but it might be that the instructor has defined scatter and map in such a way as to differentiate between these two cases, such as the following plausible definitions:
Scatter: one-to-one relationship from input to output (ie. unidirectional relationship). Every input location has a corresponding output location, but not every output location has a corresponding input location.
Map: one-to-one relationship between input and output (ie. bidirectional relationship). Every input location has a corresponding output location, and every output location has a corresponding input location.
Gather: one-to-one relationship from output to input (ie. unidirection relationship). Every output location has a corresponding input location, but not every input location has a corresponding output location.
The definition of each communication pattern (map, scatter, gather, etc.) varies slightly from one language/environment/context to another, but since I have followed that same Udacity course I'll try to explain that term as I understand it in the context of the course:
The Map operation calculates each output element as a function of its corresponding input element, i.e.:
output[tid] = foo(input[tid]);
The Gather pattern calculates each output element as a function of one or more (usually more) input elements, not necessarily the corresponding one (typically these are elements from a neighborhood). For example:
output[tid] = (input[tid-1] + input[tid+1]) / 2;
Lastly, the Scatter operation has each input element contribute to one or more (again, usually more) output elements. For instance,
atomicAdd( &(output[tid-1]), input[tid]);
atomicAdd( &(output[tid]), input[tid]);
atomicAdd( &(output[tid+1]), input[tid]);
The example given in the question is clearly not a Map, because each output is calculated from an input at a different location.
Also, it is hard to see how the same example can be a scatter, because each input element only causes one write to the output, but it is indeed a scatter because each input causes a write to an output whose location is determined by the input.
In other words, each CUDA thread processes an input element at the location associated with its tid(thread ID number), and calculates where to write the result. More usually a scatter would write on several places instead of only one, so this is a particular case that might as well be named differently.
Each player has 3 properties (name, height, rank).
So I think scatter is correct, because we should consider these three things to make output.
If player has only one property like rank,
then Map is correct I think.
reference: Parallel Communication Patterns Recap in this lecture
reference: map/reduce/gather/scatter with image
I trying to parse GIF format and have one problem with reading image data.
This data represented like bit array, containing variable-length values.
ex:
0010-1010-0010-0000-00111-10000-11111...
Sometimes length of the code increases, but I can't understand how I can detect this increasing.
I have just initial code size (length of the first code ex. 4).
Standart says only:
Appendix F. Variable-Length-Code LZW Compression.
...
The Variable-Length-Code aspect of the algorithm is based on an initial code size
(LZW-initial code size), which specifies the initial number of bits used for
the compression codes. When the number of patterns detected by the compressor
in the input stream exceeds the number of patterns encodable with the current
number of bits, the number of bits per LZW code is increased by one.
...
When parsing a GIF file, the Image Descriptor includes the bit width of the unencoded symbols (example: 8 bits).
As you probably already know, the initial code size of the compressed data is one bit wider than the bit width of the unencoded symbols (example: 9 bits).
Also, as you probably already know, the possible compressed code values in a GIF file gradually increase in size,
up to a maximum of 0xFFF == 4095 which requires 12 bits to store.
For every code that the decompressor pulls from the compressed data,
the decompressor adds a new item to its dictionary.
For example, if the first two 9-bit codes the decompressor reads are 0x028 0x0FF,
the decompressor adds a two-byte sequence to its dictionary.
Later if the decompressor ever reads the code 0x102,
the decompressor decodes that 0x102 code to the two 8-bit bytes 0x28 0xFF.
The next item the decompressor adds to the dictionary is assigned the code 0x103.
The next item the decompressor adds to the dictionary is assigned the code 0x104. ...
Eventually the decompressor adds an item to the dictionary that is assigned the code 0x1FF.
That is the largest number that fits into 9 bits.
After storing that item into the dictionary,
the decompressor starts reading 10-bit codes.
The next item the decompressor adds to the dictionary is assigned the code 0x200.
There isn't any special "command" in the data sequence that tells the decompressor to increment the code width.
The decompressor must keep track of how many items the dictionary contains so far (which often can be conveniently re-used as the index of where to write the next item into the dictionary).
When the decompressor adds item 0x1ff to the dictionary, it's time for the decompressor to start reading 10-bit codes.
When the decompressor adds item 0x3ff to the dictionary, it's time for the decompressor to start reading 11 bit codes.
Wikipedia: Graphics Interchange Format
Wikipedia: Lempel–Ziv–Welch with variable-length codes
Look at this example first, it may be clearer to understand LZW than looking at the standard.
And this may also be useful.