I have 40x tiff images and i want to use them to build 20x, 10x levels. How can I do that? - tiff

My problem is that I have TIFF images with multiple levels (40x,20x,10x,5x) but some only have 40x level and I need all of them. My question is if there is any way to get 20x or other levels from 40x level images. I read about a library called vips but I don't understand how to use it to my specific problem.

Yes, libvips can compute the missing levels for you. For example:
$ vips copy k2.tif x.tif[pyramid]
The [pyramid] is an option to the libvips TIFF writer to enable pyramid output. You can check the result with tiffinfo:
$ tiffinfo x.tif
TIFF Directory at offset 0x9437192 (900008)
Image Width: 1450 Image Length: 2048
Tile Width: 128 Tile Length: 128
Resolution: 72.009, 72.009 pixels/inch
Bits/Sample: 8
Sample Format: unsigned integer
Compression Scheme: None
Photometric Interpretation: RGB color
Orientation: row 0 top, col 0 lhs
Samples/Pixel: 3
Planar Configuration: single image plane
TIFF Directory at offset 0x11797866 (b4056a)
Subfile Type: reduced-resolution image (1 = 0x1)
Image Width: 725 Image Length: 1024
Tile Width: 128 Tile Length: 128
Resolution: 72.009, 72.009 pixels/inch
Bits/Sample: 8
Compression Scheme: None
Photometric Interpretation: RGB color
Orientation: row 0 top, col 0 lhs
Samples/Pixel: 3
Planar Configuration: single image plane
TIFF Directory at offset 0x12388198 (bd0766)
Subfile Type: reduced-resolution image (1 = 0x1)
Image Width: 362 Image Length: 512
Tile Width: 128 Tile Length: 128
Resolution: 72.009, 72.009 pixels/inch
Bits/Sample: 8
Compression Scheme: None
Photometric Interpretation: RGB color
Orientation: row 0 top, col 0 lhs
Samples/Pixel: 3
Planar Configuration: single image plane
TIFF Directory at offset 0x12585098 (c0088a)
Subfile Type: reduced-resolution image (1 = 0x1)
Image Width: 181 Image Length: 256
Tile Width: 128 Tile Length: 128
Resolution: 72.009, 72.009 pixels/inch
Bits/Sample: 8
Compression Scheme: None
Photometric Interpretation: RGB color
Orientation: row 0 top, col 0 lhs
Samples/Pixel: 3
Planar Configuration: single image plane
TIFF Directory at offset 0x12634494 (c0c97e)
Subfile Type: reduced-resolution image (1 = 0x1)
Image Width: 90 Image Length: 128
Tile Width: 128 Tile Length: 128
Resolution: 72.009, 72.009 pixels/inch
Bits/Sample: 8
Compression Scheme: None
Photometric Interpretation: RGB color
Orientation: row 0 top, col 0 lhs
Samples/Pixel: 3
Planar Configuration: single image plane
You can see it's written a five level pyramid, with each level being a tiled TIFF with 128 x 128 pixel tiles. That might or might not be correct for your application, you'd need to give a lot more information. For example:
$ vips copy k2.tif x.tif[pyramid,compression=jpeg,Q=85,tile-width=256,tile-height=256]
Might be better. Check the docs.

Related

Why is the spatial resolution of image from WorldView 3 like this?

The output of this image from gdalinfo 20141030-wv03.tif is like this:
Driver: GTiff/GeoTIFF
Files: 20141030-wv03.tif
Size is 16484, 15253
Coordinate System is:
GEOGCRS["WGS 84",
DATUM["World Geodetic System 1984",
ELLIPSOID["WGS 84",6378137,298.257223563,
LENGTHUNIT["metre",1]]],
PRIMEM["Greenwich",0,
ANGLEUNIT["degree",0.0174532925199433]],
CS[ellipsoidal,2],
AXIS["geodetic latitude (Lat)",north,
ORDER[1],
ANGLEUNIT["degree",0.0174532925199433]],
AXIS["geodetic longitude (Lon)",east,
ORDER[2],
ANGLEUNIT["degree",0.0174532925199433]],
ID["EPSG",4326]]
Data axis to CRS axis mapping: 2,1
Origin = (113.959353776485997,23.091020758099145)
Pixel Size = (0.000002966620901,-0.000002966620901)
Metadata:
AREA_OR_POINT=Area
DataType=Generic
Image Structure Metadata:
COMPRESSION=LZW
INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left ( 113.9593538, 23.0910208) (113d57'33.67"E, 23d 5'27.67"N)
Lower Left ( 113.9593538, 23.0457709) (113d57'33.67"E, 23d 2'44.78"N)
Upper Right ( 114.0082556, 23.0910208) (114d 0'29.72"E, 23d 5'27.67"N)
Lower Right ( 114.0082556, 23.0457709) (114d 0'29.72"E, 23d 2'44.78"N)
Center ( 113.9838047, 23.0683958) (113d59' 1.70"E, 23d 4' 6.22"N)
Band 1 Block=128x128 Type=Byte, ColorInterp=Red
NoData Value=256
Band 2 Block=128x128 Type=Byte, ColorInterp=Green
NoData Value=256
Band 3 Block=128x128 Type=Byte, ColorInterp=Blue
NoData Value=256
The spatial resolution is (0.000002966620901,-0.000002966620901), how to understand this value?
I also check another image from WorldView 2, ths output is:
Driver: GTiff/GeoTIFF
Files: 20150708.tif
Size is 9984, 10132
Coordinate System is:
PROJCRS["WGS 84 / UTM zone 50N",
BASEGEOGCRS["WGS 84",
DATUM["World Geodetic System 1984",
ELLIPSOID["WGS 84",6378137,298.257223563,
LENGTHUNIT["metre",1]]],
PRIMEM["Greenwich",0,
ANGLEUNIT["degree",0.0174532925199433]],
ID["EPSG",4326]],
CONVERSION["UTM zone 50N",
METHOD["Transverse Mercator",
ID["EPSG",9807]],
PARAMETER["Latitude of natural origin",0,
ANGLEUNIT["degree",0.0174532925199433],
ID["EPSG",8801]],
PARAMETER["Longitude of natural origin",117,
ANGLEUNIT["degree",0.0174532925199433],
ID["EPSG",8802]],
PARAMETER["Scale factor at natural origin",0.9996,
SCALEUNIT["unity",1],
ID["EPSG",8805]],
PARAMETER["False easting",500000,
LENGTHUNIT["metre",1],
ID["EPSG",8806]],
PARAMETER["False northing",0,
LENGTHUNIT["metre",1],
ID["EPSG",8807]]],
CS[Cartesian,2],
AXIS["(E)",east,
ORDER[1],
LENGTHUNIT["metre",1]],
AXIS["(N)",north,
ORDER[2],
LENGTHUNIT["metre",1]],
USAGE[
SCOPE["unknown"],
AREA["World - N hemisphere - 114°E to 120°E - by country"],
BBOX[0,114,84,120]],
ID["EPSG",32650]]
Data axis to CRS axis mapping: 1,2
Origin = (291153.100000000034925,2705938.760000000242144)
Pixel Size = (0.510000000000000,-0.510000000000000)
Metadata:
AREA_OR_POINT=Area
TIFFTAG_XRESOLUTION=1
TIFFTAG_YRESOLUTION=1
Image Structure Metadata:
INTERLEAVE=BAND
Corner Coordinates:
Upper Left ( 291153.100, 2705938.760) (114d56'22.85"E, 24d27'10.88"N)
Lower Left ( 291153.100, 2700771.440) (114d56'25.58"E, 24d24'22.98"N)
Upper Right ( 296244.940, 2705938.760) (114d59'23.60"E, 24d27'13.32"N)
Lower Right ( 296244.940, 2700771.440) (114d59'26.26"E, 24d24'25.41"N)
Center ( 293699.020, 2703355.100) (114d57'54.57"E, 24d25'48.15"N)
Band 1 Block=9984x1 Type=Byte, ColorInterp=Red
Band 2 Block=9984x1 Type=Byte, ColorInterp=Green
Band 3 Block=9984x1 Type=Byte, ColorInterp=Blue
The spatial resolution is (0.510000000000000,-0.510000000000000). How do I understand their difference between them? Thanks.
Your images are in two different coordinate systems.
Your second file 20150708.tif is in an UTM projection (UTM 50N to be exact) which has map units in meters - that's why the pixel resolutions is in meters (0.51m).
Your first file 20141030-wv03.tif is in a geographic coordinate system, the widely used World Geodetic System 1984 (or WGS84) which has map units in degrees, giving you the pixel resolution also in (decimal) degrees. On the equator 0.00001 degrees is around 1.11 meters so both images have likely the same resolution.
For more info on WGS84 vs UTM, this post on GIS stackexchange might be interesting.

Slicing an input data layer in caffe - unknown blob input

I am trying to do pixel-wise classification with caffe, so need to provide a ground truth image the size of the input image. There is several ways of doing this, and I decided to set up my input as a 4-channel LMDB (according to the 2nd point of this answer). This requires me to add a Slice layer after my input, which is also outlined in the same answer.
I keep getting Unknown blob input data_lmdb to layer 0 as an error message (data_lmdb is supposed to be my very bottom input layer). I found that unknown blob (be it top or bottom) error is mostly caused by forgetting to define something in one of the TRAIN / TEST phases while defining it in the other (e.g. this question, or this one). But, I am using a combination of train.prototxt, inference.prototxt and solver.prototxt files that I have previously used, just replacing the input layers from HD5 to LMDB (for a bit of practice), so everything should be defined.
Can anybody see why I am getting the Unknown blob input data_lmdb to layer 0 error? From the train log files I can see that it crashes as soon as it reads the train.prototxt file (it doesn't even reach the Creating layer part).
My prototxt files are as follows:
solver.prototxt
net: "train.prototxt" # Change this to the absolute path to your model file
test_initialization: false
test_iter: 1
test_interval: 1000000
base_lr: 0.01
lr_policy: "fixed"
gamma: 1.0
stepsize: 2000
display: 20
momentum: 0.9
max_iter: 10000
weight_decay: 0.0005
snapshot: 100
snapshot_prefix: "set_snapshot_name" # Absolute path to output solver snapshots
solver_mode: GPU
train.prototxt (first two layers only; they are followed by a LNR normalization layer and then a Convolution layer):
name: "my_net"
layer {
name: "data_lmdb"
type: "Data"
top: "slice_input"
data_param {
source: "data/train"
batch_size: 4
backend: LMDB
}
}
layer{
name: "slice_input"
type: "Slice"
bottom: "data_lmdb" # 4-channels = rgb+truth
top: "data"
top: "label"
slice_param {
axis: 1
slice_point: 3
}
}
The first few layer definitions in inference.prototxt are identical to train.prototxt (which shouldn't matter anyway as it is not used in training) except the following:
in data_lmdb the source path is different (data/test)
in data_lmdb layer uses batch_size: 1
Please do let me know if I need to include any more information or layers. I was trying to keep it brief, which didn't really work out in the end.
The message Unknown blob input points on non-existent blob that some layer wants to have as input. Your slice_input layer specified data_lmdb as input blob, but there is no such a blob in your network. Instead, you have a layer with such a name. Blob names are defined by the top field, which is slice_input in this case.
You shoud either change top: "slice_input" to top: "data_lmdb" in your data_lmdb layer, or use bottom: "slice_input" # 4-channels = rgb+truth.
However, for more clear naming I would offer you the following:
name: "my_net"
layer {
name: "data"
type: "Data"
top: "data_and_label"
data_param {
source: "data/train"
batch_size: 4
backend: LMDB
}
}
layer{
name: "slice_input"
type: "Slice"
bottom: "data_and_label" # 4-channels = rgb+truth
top: "data"
top: "label"
slice_param {
axis: 1
slice_point: 3
}
}

Using im2col layer, reshape layer and inner product layer to implement locally connected layer?

I am using caffe and it doesn't have a locally connected layer. So any example on how to use im2col layer, reshape layer and inner product layer to implement locally connected layer? Thanks
Personal View of Point:
I have also tried to use Crop, Im2col, Reshape and InnerProduct layer to implement locally connected layer but failed.
Because when I want to implement a convolution operation using InnerProduct layer, I find that in InnerProductLayer<Dtype>::Forward_cpu() function:
caffe_cpu_gemm<Dtype>(CblasNoTrans, transpose_ ? CblasNoTrans : CblasTrans,
M_, N_, K_, (Dtype)1.,
bottom_data, weight, (Dtype)0., top_data);
and in BaseConvolutionLayer<Dtype>::forward_cpu_gemm() function:
caffe_cpu_gemm<Dtype>(CblasNoTrans, CblasNoTrans, conv_out_channels_ /
group_, conv_out_spatial_dim_, kernel_dim_,
(Dtype)1., weights + weight_offset_ * g, col_buff + col_offset_ * g,
(Dtype)0., output + output_offset_ * g);
the weight(s), which should be used as convolution kernels, are passed to different arguments of caffe_cpu_gemm().
So I can't implement a convolution operation using InnerProductLayer<Dtype>::Forward_cpu() function and thus can't implement a local connected layer(I mean local convolution here) using Crop, Im2col, Reshape and InnerProduct layers.
My solution:
However, I implemented a local convolution layer here and its idea is to divide input feature maps into N*N grid(even with overlap) and performs convolution on each of the grid using different kernels. For example, the input feature maps have a shape (2, 3, 8, 8) and you want to divide the spatial feature map 8*8 into 16 2*2 local regions and then perform convolution on each local region with different bank of kernels, you can write a prototxt like this:
layer {
name: "local_conv"
type: "LocalConvolution"
bottom: "bottom" # shape (2,3,8,8)
top: "top"
param {
lr_mult: 1
decay_mult: 1
}
param {
lr_mult: 2
decay_mult: 0
}
local_conv_param {
local_region_number_h: 4
local_region_number_w: 4
local_region_ratio_h: 0.3 # determin the height/width of local regions
local_region_ratio_w: 0.3 # local_region_size = floor(local_region_ratio * input_size)
local_region_step_h: 2 # step between local regions on the top left part
# and other regions will lie in the axial symmetry positions
# automatically
local_region_step_w: 2
num_output: 5
kernel_h: 3
kernel_w: 1
stride: 1
pad: 0
weight_filler {
type: "xavier"
}
bias_filler {
type: "constant"
}
}
}
You can easily add this layer to your caffe and the related files are:
include/caffe/layers/local_conv_layer.hpp
src/caffe/layers/local_conv_layer.cpp(cu)
and you should also add message LocalConvolutionParameter, optional LocalConvolutionParameter local_conv_param from src/caffe/proto/caffe.proto to your caffe.proto.
.

Binwalk - Compressed data is corrupt

root#kali:~/ROUTER# binwalk new-firmware.bin
DECIMAL HEXADECIMAL DESCRIPTION
--------------------------------------------------------------------------------
84 0x54 uImage header, header size: 64 bytes, header CRC: 0xE52A7F50, created: 2012-02-10 07:27:12, image size: 819799 bytes, Data Address: 0x80002000, Entry Point: 0x801AC9F0, data CRC: 0x6A10D412, OS: Linux, CPU: MIPS, image type: OS Kernel Image, compression type: lzma, image name: "Linux Kernel Image"
148 0x94 LZMA compressed data, properties: 0x5D, dictionary size: 8388608 bytes, uncompressed size: 2386252 bytes
917588 0xE0054 Squashfs filesystem, little endian, version 4.0, compression:lzma, size: 2588426 bytes, 375 inodes, blocksize: 16384 bytes, created: 2016-02-05 02:05:56
root#kali:~/ROUTER# dd if=new-firmware.bin of=uboot.lzma skip=148 bs=1
3735488+0 records in
3735488+0 records out
3735488 bytes (3.7 MB, 3.6 MiB) copied, 4.16712 s, 896 kB/s
root#kali:~/ROUTER# dd if=new-firmware.bin of=kernel.squash skip=917588 bs=1
2818048+0 records in
2818048+0 records out
2818048 bytes (2.8 MB, 2.7 MiB) copied, 3.46517 s, 813 kB/s
All looks to have worked so far but then I try the following:
root#kali:~/ROUTER# lzma -d uboot.lzma
lzma: uboot.lzma: Compressed data is corrupt
root#kali:~/ROUTER# unsquashfs kernel.squash
Parallel unsquashfs: Using 4 processors
lzma uncompress failed with error code 0
read_block: failed to read block #0x277af0
read_fragment_table: failed to read fragment table index
FATAL ERROR:failed to read fragment table
Any idea why it may be happening? As the prompt says, I am on Kali Linux. ALso get this
jj#ubuntu:~/Firmware$ binwalk -e new-firmware.bin
DECIMAL HEXADECIMAL DESCRIPTION
--------------------------------------------------------------------------------
84 0x54 uImage header, header size: 64 bytes, header CRC: 0xE52A7F50, created: 2012-02-10 07:27:12, image size: 819799 bytes, Data Address: 0x80002000, Entry Point: 0x801AC9F0, data CRC: 0x6A10D412, OS: Linux, CPU: MIPS, image type: OS Kernel Image, compression type: lzma, image name: "Linux Kernel Image"
148 0x94 LZMA compressed data, properties: 0x5D, dictionary size: 8388608 bytes, uncompressed size: 2386252 bytes
WARNING: Extractor.execute failed to run external extractor 'sasquatch -p 1 -le -d '%%squashfs-root%%' '%e'': [Errno 2] No such file or directory
WARNING: Extractor.execute failed to run external extractor 'sasquatch -p 1 -be -d '%%squashfs-root%%' '%e'': [Errno 2] No such file or directory
917588 0xE0054 Squashfs filesystem, little endian, version 4.0, compression:lzma, size: 2588426 bytes, 375 inodes, blocksize: 16384 bytes, created: 2016-02-05 02:05:56
Try
lzma -d < uboot.lzma > uboot
The error you found is due to LZMA Utils having been superseded by XV Utils, which don't ignore the trailing garbage. If you use it as a stream (like I gave you), you'll still have the corruption error message in the end, but you'll get your file alright.
Source (search for "corrupt" and you'll find other related comments)
Try
binwalk -e new-firmware.bin
you will end up with 2 file one being extracted
In the extracted one you will get 2 one unzippped and a zipped one
you can use dd at this point and unlzma should work at least it did work out for me
this is what i mean
root#ROOT20:~/Desktop/project/dlink/dlink2# binwalk dcs932l_v1.14.04.bin
DECIMAL HEXADECIMAL DESCRIPTION
106352 0x19F70 U-Boot version string, "U-Boot 1.1.3"
106816 0x1A140 CRC32 polynomial table, little endian
124544 0x1E680 HTML document header
124890 0x1E7DA HTML document footer
124900 0x1E7E4 HTML document header
125092 0x1E8A4 HTML document footer
125260 0x1E94C HTML document header
125953 0x1EC01 HTML document footer
327680 0x50000 uImage header, header size: 64 bytes, header CRC: 0x88345E96, created: 2016-09-09 13:52:27, image size: 3804958 bytes, Data Address: 0x80000000, Entry Point: 0x803B8000, data CRC: 0x531E94DE, OS: Linux, CPU: MIPS, image type: OS Kernel Image, compression type: lzma, image name: "Linux Kernel Image"
327744 0x50040 LZMA compressed data, properties: 0x5D, dictionary size: 33554432 bytes, uncompressed size: 6558763 bytes
root#ROOT20:~/Desktop/project/dlink/dlink2# dd if=dcs932l_v1.14.04.bin skip=327744 bs=1 of=kernel.lzma
3866560+0 records in
3866560+0 records out
3866560 bytes (3.9 MB, 3.7 MiB) copied, 9.61104 s, 402 kB/s
root#ROOT20:~/Desktop/project/dlink/dlink2# unlzma kernel.lzma
unlzma: kernel.lzma: Compressed data is corrupt
root#ROOT20:~/Desktop/project/dlink/dlink2# binwalk -e dcs932l_v1.14.04.bin
DECIMAL HEXADECIMAL DESCRIPTION
106352 0x19F70 U-Boot version string, "U-Boot 1.1.3"
106816 0x1A140 CRC32 polynomial table, little endian
124544 0x1E680 HTML document header
124890 0x1E7DA HTML document footer
124900 0x1E7E4 HTML document header
125092 0x1E8A4 HTML document footer
125260 0x1E94C HTML document header
125953 0x1EC01 HTML document footer
327680 0x50000 uImage header, header size: 64 bytes, header CRC: 0x88345E96, created: 2016-09-09 13:52:27, image size: 3804958 bytes, Data Address: 0x80000000, Entry Point: 0x803B8000, data CRC: 0x531E94DE, OS: Linux, CPU: MIPS, image type: OS Kernel Image, compression type: lzma, image name: "Linux Kernel Image"
327744 0x50040 LZMA compressed data, properties: 0x5D, dictionary size: 33554432 bytes, uncompressed size: 6558763 bytes
root#ROOT20:~/Desktop/project/dlink/dlink2# ls
dcs932l_v1.14.04.bin _dcs932l_v1.14.04.bin.extracted kernel.lzma
root#ROOT20:~/Desktop/project/dlink/dlink2# cd _dcs932l_v1.14.04.bin.extracted/
root#ROOT20:~/Desktop/project/dlink/dlink2/_dcs932l_v1.14.04.bin.extracted# ls
50040 50040.7z
root#ROOT20:~/Desktop/project/dlink/dlink2/_dcs932l_v1.14.04.bin.extracted# binwalk 50040
DECIMAL HEXADECIMAL DESCRIPTION
3145804 0x30004C Linux kernel version 2.6.21
3175792 0x307570 SHA256 hash constants, little endian
3389960 0x33BA08 Unix path: /usr/gnemul/irix/
3392244 0x33C2F4 Unix path: /usr/lib/libc.so.1
3393940 0x33C994 Unix path: /dev/vc/0
3408260 0x340184 Copyright string: "Copyright (c) 2010 Alpha Networks Inc."
3491536 0x3546D0 Unix path: /etc/Wireless/RT2860STA/RT2860STA.dat
3573187 0x3685C3 Neighborly text, "neighbor %.2x%.2x.%.2x:%.2x:%.2x:%.2x:%.2x:%.2x lost on port %d(%s)(%s)"
3807776 0x3A1A20 CRC32 polynomial table, little endian
4038656 0x3DA000 LZMA compressed data, properties: 0x5D, dictionary size: 1048576 bytes, uncompressed size: 8072704 bytes
root#ROOT20:~/Desktop/project/dlink/dlink2/_dcs932l_v1.14.04.bin.extracted# dd if=50040 skip=4038656 bs=1 of=kernel2.lzma
2520107+0 records in
2520107+0 records out
2520107 bytes (2.5 MB, 2.4 MiB) copied, 8.23342 s, 306 kB/s
root#ROOT20:~/Desktop/project/dlink/dlink2/_dcs932l_v1.14.04.bin.extracted# unlzma kernel2.lzma
root#ROOT20:~/Desktop/project/dlink/dlink2/_dcs932l_v1.14.04.bin.extracted#
root#ROOT20:~/Desktop/project/dlink/dlink2/_dcs932l_v1.14.04.bin.extracted# mkdir cpio; cd cpio
root#ROOT20:~/Desktop/project/dlink/dlink2/_dcs932l_v1.14.04.bin.extracted/cpio# cpio -idm --no-absolute-filenames < ../kernel2
cpio: Removing leading `/' from member names
15767 blocks
root#ROOT20:~/Desktop/project/dlink/dlink2/_dcs932l_v1.14.04.bin.extracted/cpio# ls
bin dev etc etc_ro home init lib media mnt mydlink proc sbin sys tmp usr var

meaning of index in texture atlas files

I am following libgdx tutuorial on texture atlases. This is an excerpt from a texture atlas file. What is the meaning of index parameter and in what situations it is useful to the programmer? All texture region have it and it is the same namely -1, in all of them.
prehistoric.png
format: RGBA8888
filter: Nearest,Nearest
repeat: none
background
rotate: false
xy: 2, 2
size: 1280, 720
orig: 1280, 720
offset: 0, 0
index: -1
trex
rotate: false
xy: 1286, 479
size: 179, 243
orig: 179, 243
offset: 0, 0
index: -1
caveman
rotate: false
xy: 1286, 319
size: 83, 156
orig: 83, 156
offset: 0, 0
index: -1
From the Javadoc:
The number at the end of the original image file name, or -1 if none.
When sprites are packed, if the original file name ends with a number, it is stored >as the index and is not considered as part of the sprite's name.
I should also add, that this index is also used on findRegion(String name, int index) method which returns the first region found with the specified name and index.
It's typically used for animations. You can append frame numbers to the file names of each frame of animation before you pack them into the atlas, i.e. run0.png, run1.png, run2.png, etc. During texture packing, the number is removed from the sprite's name and used as its index. Then you can load the animation all at once:
animation = new Animation(0.1f, atlas.findRegions("run"));
The index is -1 when the original file name did not end in a number.