I am working on Octave with a brain MRI dataset.
The data is in the form of hundreds of .mat fils.
I load the data file using: x = load("filename.mat") and the images are stored in the form of an array from 0 to 256.
Then when I try to imshow(x.image) a window pops up with 512 x 512 graph but is completely black.
It also throws a warning saying: unsupported type for cdata (= int16 matrix). Valid types are uint8, uint16, double, single, and bool.
I have also tried casting the image array using: imshow(cast(x, "double")) and all the other data types mentioned above, but the result is almost the same.
Help me out here.
You can use the limits parameter of imshow. The image is appearing black because it only contains values in the range [0,256], while int16 contains values up to 32767.
You can set the limits manually:
imshow(x.image, [0, 256])
Or use
imshow(x.image, [])
For automatic limit detection.
Note: on Octave, this will only work with supported types. Since your image doesn't contain negative values, you can convert the image to uint16, which will fit your values. Or you can use double, for a more general approach:
imshow(uint16(x.image), [])
imshow(double(x.image), [])
In your case, if you don't mind the pixels with value 256 being saturated to 255, you can convert the image to uint8:
imshow(uint8(x.image))
Related
I've been using strings to represent decoded JSON integers larger than 32 bits. It seems the string_of_int is capable of dealing with large integer inputs. So a decoder, written (in the Json.Decode namespace):
id: json |> field("id", int) |> string_of_int, /* 'id' is string */
is succefully dealing with integers of at least 37 bits.
Encoding, on the other hand, is proving troublesome for me. The remote server won't accept a string representation, and is expecting an int64. Is it possible to make bs-json support the int64 type? I was hoping something like this could be made to work:
type myData = { id: int64 };
let encodeMyData = (data:myData) => Json.Encode.(object_([("id", int64(myData.id)]))
Having to roll my own encoder is not nearly as formidable as a decoder, but ... I'd rather not.
You don't say exactly what problem you have with encoding. The int encoder does literally nothing except change the type, trusting that the int value is actually valid. So I would assume it's the int_of_string operation that causes problems. But that begs the question, if you can successfully decode it as an int, why are you then converting it to a string?
The underlying problem here is that JavaScript doesn't have 64 bit integers. The max safe integer is 253 - 1. JavaScript doesn't actually have integers at all, only floats, which can represent a certain range of integers, but can't efficiently do integer arithmetic unless they're converted to either 32-bit or 64-bit ints. And so for whatever reason, probably consistent overflow handling, it was decided in the EcmaScript specification that binary bitwise operations should operate on 32-bit integers. And so that opened the possibility for an internal 32-bit representation, a notation for creating 32-bit integers, and the possibility of optimized integer arithmetic on those.
So to your question:
Would it be "safe" to just add external int64 : int64 -> Js.Json.t = "%identity" to the encoder files?
No, because there's no 64-bit integer representation in JavaScript, int64 values are represented as an array of two Numbers I believe, but is also an internal implementation detail that's subject to change. Just casting it to Js.Json.t will not yield the result you expect.
So what can you do then?
I would recommend using float. In most respects this will behave exactly like JavaScript numbers, giving you access to its full range.
Alternatively you can use nativeint, which should behave like floats except for division, where the result is truncated to a 32-bit integer.
Lastly, you could also implement your own int_of_string to create an int that is technically out of range by using a couple of lightweight JavaScript functions directly, though I wouldn't really recommend doing this:
let bad_int_of_string = str =>
str |> Js.Float.fromString |> Js.Math.floor_int;
I'm trying to read variable length 1-D inputs into a Tensorflow CNN.
I have previously implemented reading fixed length inputs by first constructing a CSV file (where the first column is the label and the remaining columns are the input values - flattened spectrogram data all padded/truncated to the same length) using tf.TextLineReader().
This time I have a directory full of files each one containing a line of data I want to use as input (flattened spectrogram data again but I do not want to force them to the same dimensions), and the line lengths are not fixed. I'm getting an error trying to use the previous approach of compiling a CSV first. I looked into the documentation of tf.TextLineReader() and it specifies that all CSV rows must be the same shape, so I am stuck! Any help would be much appreciated, thanks :)
I'm assuming that the data isn't changing shape when you have a longer or shorter sample right? By that I mean that if you trained your network on arrays of 1000 pixels for example, with a kernel of say [5,1] size. That [5,1] kernel needs to see the same patterns in the variable length data as it did in the training data. If your data is stretched or shrunk, then the correct solution is to interpolate the data to the same size as the training data so the shapes/patterns match.
Assuming you just want variable length inputs, then in theory you should be able to do this by setting your batch size to 1 and varying the 1st dimension of the data.
So your input placeholder would look like:
X = tf.placeholder(dtype, shape=[1,None,1,1])
The 4 shape arguments are: 1=batch size; None=unknown first dimension size; 1=unused because it's a 1D dataset, 1=one channel images, again unused but necessary for tf.conv2d to receive the expected 4D image.
This is not very different from configuring tensorflow to support variable batch sizes. So you should review this link below and understand that process.
get the size of a variable batch dimension
Note that you can't use a batch size more than 1 here because you wouldn't be able to construct a matrix with missing values in the 2nd dimension. I expect the convolution operations to work with this variable dimension (though I haven't actually tried this).
Another option to deal with this problem would be to pad your inputs with 0's so they all have a common length, but that will need to have been trained into the model up front.
The function make is describe like this:
func make(Type, size IntegerType) Type
When we use make for slicing sometimes it shows like:
make([]int, 0, 10)
So my question is:
How can the make function take three parameters? The size IntegerType is not Vararg. I'm confused...
The make function is one of a bunch of built-in functions that are allowed to do things that you cannot achieve (at least not cleanly and easily) in your Go code.
It has a number of overloaded forms for creating maps, channels and slices (see
https://golang.org/ref/spec#Making_slices_maps_and_channels) :
Your confusion probably stems from https://golang.org/pkg/builtin/#make which shows make as having the signature func make(Type, size IntegerType) Type.
If you look closer in that section, you would also see an indication that make can have a third argument:
Slice: The size specifies the length. The capacity of the slice is
equal to its length. A second integer argument may be provided to
specify a different capacity; it must be no smaller than the length,
so make([]int, 0, 10) allocates a slice of length 0 and capacity 10.
You can also notice that make can also be used without its integer argument:
Map: An initial allocation is made according to the size but the
resulting map has length 0. The size may be omitted, in which case a
small starting size is allocated.
Channel: The channel's buffer is initialized with the specified buffer capacity. If zero, or the size is omitted, the channel is unbuffered.
The make() function is not a regular function, is a builtin function being part of the language specification. What you see in the builtin package (builtin.make()) is only for documentation purposes. That is not the actual signature of the function. The 3rd optional parameter is the capacity, which may only be provided when you're creating a slice.
It's described in the spec: Making slices, maps and channels:
make(T, n) slice slice of type T with length n and capacity n
make(T, n, m) slice slice of type T with length n and capacity m
And also mentioned at Slice types:
A new, initialized slice value for a given element type T is made using the built-in function make, which takes a slice type and parameters specifying the length and optionally the capacity. A slice created with make always allocates a new, hidden array to which the returned slice value refers. That is, executing
make([]T, length, capacity)
produces the same slice as allocating an array and slicing it, so these two expressions are equivalent:
make([]int, 50, 100)
new([100]int)[0:50]
The function make is a builtin function. The function has several features not available to other functions. One is that it takes a variable number arguments as you noted. Another is that the first argument is a type.
The function definition func make(Type, size IntegerType) Type is for documentation purposes only. It is not the actual definition of the function.
The question
What are the ways of coercing octave to create a real copy of whatever object? Structures are the main interest.
My underlying problem
In my problem I'm obtaining a rather large structure from another function in a loop but for the current task only a few pieces of it are needed. For example:
for i=1:many
res=solver(params);
store1{i}=res.string1;
store2{i}=res.arr(:,1);
end
res is a sizable chunk of data and due to lazy-copy those store-s are references to tiny portions of bytes in that chunk. After I store those tiny portions, I don't need res itself, however, since middle of that chunk is referenced by store, the memory area is unfit for res obtained on the next iteration (they are of the same size) and thus another sizable piece of memory is allocated, which is then again crossed by few tiny links an so on.
Without storing parts of res, the program successfully keeps the memory consumption same after first couple of iterations.
So how do I make a complete copy of structure field?
I've tried using struct-related functions like rmfield but those keep references instead of their own objects.
I've tried to wrap the assignment of in its own function:
new_struct=copy( rmfield(old_struct,"bigdata"));
function c=copy(a);
c=a;
end;
This by the way doesn't work even for arrays.
I'm interested in method applicable to any generic variable.
Minimal working example of the problem
a=cell(3,1);
for i=1:length(a);
r=rand(100000,1000);
a{i}=r(1:100,end);
whos; fflush(stdout);
pause(2);
end;
The above code will cause memory usage to gradually grow by far more than 8.08 kb reported by whos due to references stored by a{i} blocking bigger memory block than they actually need. If you force the proper copy, the problem is not present.
Numerical arrays
For numeric types addition of zero is enough to warrant a new array.
c=a+0;
Strings
For string which is 1 x n char array, something along the following lines will work:
c=[a "a"](1:end-1);
Multidimensional char arrays will require concatenation with a column:
c=[a true(size(a,1),1)](:,1:end-1);
Here true is used to generate dummy array of size compatible with char. (There seems to be no procedural method of generating char array of arbitrary size) char(zeros(size(a,1),1)) and char(true(size(a,1),1)) caused excess memory usage during their creation on some calls.
Note that empty concatenation c=[a ""]; will not result in a copying. Also it is possible to do c=[a+0 ""]; which will result in a copying due to +0 but that one infers type conversions to and from double which is 8 times larger in size. (char(zeros( doesn't seem to cause that)
Other types
In general you can use casting for the types allowed by it in order to not tailor the expressions manually as I had to do above:
typelist={"double","single","char"}; %full list of supported types is available in the link
class_of_a = typelist{ isa(a,typelist) };
c=typecast( [typecast(a,'single'); single(1)] (1:end-1), class_of_a);
Single is seemingly smallest datatype available in octave.
Note that logical is not supported by this method.
Copying structures
Apparently you'd have to write your own function to go around struct fields, copy them with above methods and recursively go to substructs.
(As it doesn't involve complexities relevant here, I'd rather leave that to be done by those who actually needs that, my own problem being solved by +0's.)
I am using the imread function in Octave to load an image:
image = imread ("data/images/image1.jpg")(:);
This is apparently loading the image as a matrix of integers with values 0-255.
I want to load it as matrix of doubles with values 0.0-1.0. I can convert it like this.
doubleImage = double(image) / 255.0;
However, converting it is pretty slow, especially for a lot of images. Is there any way to load the image directly as a matrix of doubles?
No, there is no way to directly read it as doubles. It doesn't make sense anyway, because the image is an integer in the file, so integers will always have to be read first. If a conversion into another type is to be done, it makes sense that it does done separated. Or maybe, use a file format that stores images in double floating point precision.
However, there is a better way to do what you are doing to convert into a double.
pkg load image;
img = imread ("image1.jpg");
img = im2double (img);
Using im2double won't make it faster (the operation it performs is the same as yours) but it will save you if in the future the image that is read is uint16, and even if the image already is of class double.
Also, I don't see how the conversion to double is slow. This a really fast operation.