What methods can be used to enlarge spartial dimension of the layer output blob?
As far as I can see from documentation it can be:
UpSampling2D
Does it only upsample with power of 2?
Also it Repeats the rows and columns of the data by size[0] and size[1] respectively. and it's not very smart.
Conv2DTranspose
Can it have arbitary output size (not power of 2 upsapmple)?
How bilinear interpolation upscale with arbitary dimensions can be done (it can be Conv2DTranspose with fixed weights?)
What other options can be used to enlarge spartial dimension of the layer output blob?
Expanding on the answer from y300, here is a complete example of wrapping TensorFlow bilinear image resizing in a Keras Lambda layer:
from keras import Sequential
from keras.layers import Lambda
import tensorflow as tf
def UpSampling2DBilinear(size):
return Lambda(lambda x: tf.image.resize_bilinear(x, size, align_corners=True))
upsampler = Sequential([UpSampling2DBilinear((256, 256))])
upsampled = upsampler.predict(images)
Note that align_corners=True to get similar performance to other bilinear image upsampling algorithms, as described in this post.
To use bicubic resampling, create a new function and replace resize_bilinear with resize_bicubic.
For an implementation even more similar to UpSampling2D, try this:
from keras import backend as K
def UpSampling2DBilinear(stride, **kwargs):
def layer(x):
input_shape = K.int_shape(x)
output_shape = (stride * input_shape[1], stride * input_shape[2])
return tf.image.resize_bilinear(x, output_shape, align_corners=True)
return Lambda(layer, **kwargs)
This will allow you to use name='', input_shape='' and other arguments for Lamba, and allows you to pass an integer stride/upsample amount.
You can define your own resize Layer:
from keras import layers, models, utils
from keras.backend import tf as ktf
class Interp(layers.Layer):
def __init__(self, new_size, **kwargs):
self.new_size = new_size
super(Interp, self).__init__(**kwargs)
def build(self, input_shape):
super(Interp, self).build(input_shape)
def call(self, inputs, **kwargs):
new_height, new_width = self.new_size
resized = ktf.image.resize_images(inputs, [new_height, new_width],
align_corners=True)
return resized
def compute_output_shape(self, input_shape):
return tuple([None, self.new_size[0], self.new_size[1], input_shape[3]])
def get_config(self):
config = super(Interp, self).get_config()
config['new_size'] = self.new_size
return config
Related
Here we have an abstract of this problem:
Assuming that we have two models: ResNet and EfficienNet, respectively.
The first model is as follow (ResNet):
def __init__(self, in_channels, out_channels, num_classes):
super().__init__()
self.conv1_0 = _conv3x3(3, 32, stride=2)
self.bn1_0 = _bn(32)
self.conv1_1 = _conv3x3(32, 32, stride=1)
self.bn1_1 = _bn(32)
self.conv1_2 = _conv3x3(32, 64, stride=1)
self.relu = nn.ReLU()
self.pad = torch.nn.ReplicationPad2d(padding=(0, 0, 1, 1))
self.maxpool = nn.MaxPool2d(kernel_size=3, stride=2)
self.end_point = _fc(225, num_classes)
def forward(self, x):
x = self.conv1(x)
#and so on...
out = self.end_point(x)
return out
while the second model has been downloaded as follow (Efficient):
efficientNet = models.efficientnet_b5().to(device)
So, we have two models but the first is developed to scratch and the second one is imported by torchvision.models library.
Now, we want the EfficientNet to take the output of ResNet, then make an end-to-end optimization from EfficientNet’s output to first layer of ResNet.
I know an easier way, which consist of changing the codes and directly using ResNet’s result as an input of Efficient.forward(). However, Efficient model is too complex, making such a change is difficult.
For mathematical reasons, assuming that we have antoher easy model between the ResNet and EfficientNet, that it's called gumbel_model.
So, in conclusion we have 3 models but we got just one target labels about the last one of them, then we can only calculate the loss of the last model (Efficent Net).
When we calculate the loss of the last model, we actually write the three rows as follow:
optimizer.zero_grad()
loss.backforward()
optimizer.step()
Where the optimizer is as follow:
optimizer = optim.SGD([dict(params=efficientNet.parameters(), lr=LR)])
Is it correct that we are backpropagationing only the last model?
To backpropagation end to end models, is it necessary to add parameters of ResNet like the first arg of optim.SGD? If Yes, how could we get params by ResNet? (see above)
I have tried using some codes during one epoch, as follow :
efficientNet = get_efficient_trained(device, out_features=1, in_features=3,path_model)
predictor = ResNet(ResidualBlockBase, layer_config, num_classes=num_cl)
for i, imgs in enumerate(dataloader):
inputs, labels = imgs
inputs, labels = inputs.to(device), labels.to(device)
predictor_output = predictor(inputs)
predictor_gumbel_output = gumbel(predictor_output)
optimizer.zero_grad()
outputs = efficientnet(predictor_gumbel_output).torch.squeeze(outputs, 1)
loss = loss_fn(outputs, labels)
loss.backward()
optimizer.step()
return model
But from the results I have the apprehension that only EfficientNet is just trained.
Are there any ways to let the back-optimization reach to ResNet?
How could I solve this type of problem?*
Waiting for responses.
I really thank you all in advance.
The params argument of the optimizer specifies all the parameters you want to optimize. Here, as you are only passing the parameters of EfficientNet, only those get optimized, as you suspect.
To optimize for all parameters end-to-end, simply pass them all when initializing the optimizer. This can be done as:
optimizer = optim.SGD(list(efficientNet.parameters()) + list(gumbel.parameters()) + list(predictor.parameters()), lr=LR)
what is the difference between "torch.utils.data.TensorDataset" and "torch.utils.data.Dataset" - the docs are not clear about that and I could not find any answers on google.
The Dataset class is an abstract class that is used to define new types of (customs) datasets. Instead, the TensorDataset is a ready to use class to represent your data as list of tensors.
You can define your custom dataset in the following way:
class CustomDataset(torch.utils.data.Dataset):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
# Your code
self.instances = your_data
def __getitem__(self, idx):
return self.instances[idx] # In case you stored your data on a list called instances
def __len__(self):
return len(self.instances)
If you just want to create a dataset that contains tensors for input features and labels, then use the TensorDataset directly:
dataset = TensorDataset(input_features, labels)
Note that input_features and labels must match on the length of the first dimension.
This might be a stupid question as nobody on fastai is trying to answer it. If it is one you can go ahead and tell me but also please tell me the answer, because right now I am completely lost.
I am currently working on a U-Net model for the segmentation of cells in microscopy images. Due to class imbalances and to amplify the importance of cell boundaries, I calculated a pixelwise weightmap for each image that I pass into fastai. Therefore I created a new ItemBase class to save labels and weights together:
class WeightedLabels(ItemBase):
"""
Custom ItemBase to store and process labels and pixelwise weights together.
Also handling the target_size of the labels.
"""
def __init__(self, lbl: Image, wgt: Image, target_size: Tuple = None):
self.lbl, self.wgt = lbl, wgt
self.obj, self.data = (lbl, wgt), [lbl.data, wgt.data]
self.target_size = target_size
...
I use extensive augmentation, like elastic deformation, mirroring and rotations on both weights and labels, as well as the original image. I determine the Loss with a custom Cross-entropy loss function that uses the weights to get the weighted loss for each pixel and averages them.
My problem is, that I do not get a very good performace. I have the feeling that might be because of fastai trying to predict the weights as well. My questions are:
Am I right to assume my model tries to predict both?
If so, how do I tell the learner what to use for updating the layers and to only predict part of my labels, while still applying augmentation to both?
Here's the code for how I implemented my custom LabelList and my custom ItemList:
class CustomSegmentationLabelList(ImageList):
"'Item List' suitable for WeightedLabels containing labels and pixelweights"
_processor = vision.data.SegmentationProcessor
def __init__(self,
items: Iterator,
wghts = None,
classes: Collection = None,
target_size: Tuple = None,
loss_func=CrossEntropyFlat(axis=1),
**kwargs):
super().__init__(items, **kwargs)
self.copy_new.append('classes')
self.copy_new.append('wghts')
self.classes, self.loss_func, self.wghts = classes, loss_func, wghts
self.target_size = target_size
def open(self, fn):
res = io.imread(fn)
res = pil2tensor(res, np.float32)
return Image(res)
def get(self, i):
fn = super().get(i)
wt = self.wghts[i]
return WeightedLabels(fn, self.open(wt), self.target_size)
def reconstruct(self, t: Tensor):
return WeightedLabels(Image(t[0]), Image(t[1]), self.target_size)
class CustomSegmentationItemList(ImageList):
"'ItemList' suitable for segmentation with pixelwise weighted loss"
_label_cls, _square_show_res = CustomSegmentationLabelList, False
def label_from_funcs(self, get_labels: Callable, get_weights: Callable,
label_cls: Callable = None, classes=None,
target_size: Tuple = None, **kwargs) -> 'LabelList':
"Get weights and labels from two functions. Saves them in a CustomSegmentationLabelList"
kwargs = {}
wghts = [get_weights(o) for o in self.items]
labels = [get_labels(o) for o in self.items]
if target_size:
print(
f'Masks will be cropped to {target_size}. Choose \'target_size \\= None \' to keep initial size.')
else:
print(f'Masks will not be cropped.')
y = CustomSegmentationLabelList(
labels, wghts, classes, target_size, path=self.path)
res = self._label_list(x=self, y=y)
return res
Also here the part, where I initiate my databunch object:
data = (CustomSegmentationItemList.from_df(img_df,IMG_PATH, convert_mode=IMAGE_TYPE)
.split_by_rand_pct(valid_pct=(1/N_SPLITS),seed=SEED)
.label_from_funcs(get_labels, get_weights, target_size=MASK_SHAPE, classes = array(['background','cell']))
.transform(tfms=tfms, tfm_y=True)
.databunch(bs=BATCH_SIZE))
I use the regular learner, where I pass in my U-Net model, the data, my loss function and some additional arguments that shouldn't really matter here. When trying to apply my model, after training it, it gives me two identical output tensors. I assume that one is probably for the labels and the other for the weights. Both have the following dimensions: (WxHxC). I do not understand why this is happening, because my model is supposed to only have one output in form of (WxHxC). If this happens during prediction, this probably also happens during training. How can I overcome this?
I do have a keras CNN model ready which expects [None,20,20,3] arrays as input. (20 is image size here...) On the other side I do have a CSV with 1200 (20*20*3) columns ready in my cloud storage.
I want to write an ETL pipeline with tensorflow to obtain a [20,20,3] shape tensor for each row in the csv.
My code so far:
I spent days of work already and feel confident, that this approoach might work out in the end.
import tensorflow as tf
BATCH_SIZE = 30
tf.enable_eager_execution()
X_csv_path = 'gs://my-bucket/dataX.csv'
X_dataset = tf.data.experimental.make_csv_dataset(X_csv_path, BATCH_SIZE, column_names=range(1200) , header=False)
X_dataset = X_dataset.map(lambda x: tf.stack(list(x.values())))
iterator = X_dataset.make_one_shot_iterator()
image = iterator.get_next()
I would expect to have a [30,1200] shape but I still get 1200 tensors of shape [30] instead. My idea is to read every line into a [1200] shaped tensor and then reshape the line to a [20,20,3] tensor to feed my model with. Thanks for your time!
tf.data.experimental.make_csv_dataset creates a OrderedDict of column arrays. For your task I'd use tf.data.TextLineDataset.
def parse(filename):
string = tf.strings.split([filename], sep=',').values
return string
dataset = tf.data.TextLineDataset('sample.csv').map(parse).batch(BATCH_SIZE)
for i in dataset:
print(i)
This will output tensor of shape (BATCH_SIZE, row_length), where row_length is a row from csv file. You can apply any additional preprocessing, depending on your task
A Keras introduction to Seq2Seq model have been published a few weeks ago that can be found here. I do not really understand one part of this code:
decoder_lstm = LSTM(latent_dim, return_sequences=True, return_state=True)
decoder_outputs, _, _= decoder_lstm(decoder_inputs,initial_state=encoder_states)
decoder_dense = Dense(num_decoder_tokens, activation='softmax')
decoder_outputs = decoder_dense(decoder_outputs)
Here the decoder_lstm is defined. It is a layer of dimension latent_dim. We use the states of the encoder as initial_state for the decoder.
What I do not understand is why a dense layer is then added after the LSTM layer and why it is working?
The decoder is supposed to return all the sequence because of return_sequences = True, so how is it possible that adding a dense layer after is working?
I guess I miss something here.
Although the common cases use 2D data (batch,dim) as inputs for dense layers, in newer versions of Keras you can use 3D data (batch,timesteps,dim).
If you don't flatten this 3D data, your Dense layer will behave as if it would be applied to each of the time steps. And you will get outputs like (batch,timesteps,dense_units)
You can check these two little models below and confirm that independently of the time steps, both Dense layers have the same number of parameters, showing its parameters are suited only for the last dimension.
from keras.layers import *
from keras.models import Model
import keras.backend as K
#model with time steps
inp = Input((7,12))
out = Dense(5)(inp)
model = Model(inp,out)
model.summary()
#model without time steps
inp2 = Input((12,))
out2 = Dense(5)(inp2)
model2 = Model(inp2,out2)
model2.summary()
The result will show 65 (12*5 + 5) parameters in both cases.