I want to use the embedding layer as the input for encoder, however I got an error as follow. My input y is a time series data with shape of 1*84. Could you please help me with that?
import numpy
import torch.nn as nn
r_input = torch.nn.Embedding(84, 10)
activation = nn.functional.relu
mu_r = nn.Linear(10, 6)
log_var_r = nn.Linear(10, 6)
y = np.random.rand(1,84)
def encode_r(y):
y = torch.reshape(y, (-1, 1, 84)) # torch.Size([batch_size, 1, 84])
hidden = torch.flatten(activation(r_input(y)), start_dim = 1)
z_mu = mu_r(hidden)
z_log_var = log_var_r(hidden)
return z_mu, z_log_var```
Error: RuntimeError: Expected tensor for argument #1 'indices' to have one of the following scalar types: Long, Int; but got torch.cuda.FloatTensor instead (while checking arguments for embedding)
According to this thread: https://discuss.pytorch.org/t/expected-tensor-for-argument-1-indices-to-have-scalar-type-long-but-got-cpufloattensor-instead-while-checking-arguments-for-embedding/32441/4, it seems that one possible solution would be to ensure embeddings have integer values and not float values in them (by embeddings we mean the lookup table not an actual embedding vector).
Related
I have built an XGboost model and trying to write a custom function such as:
cols_when_model_builds = model.get_booster().feature_names
def get_prediction(parameter_1, parameter_2, parameter_3):
all_columns = ["parameter_1", "parameter_2", "parameter_3"]
lst = [parameter_1, parameter_2, parameter_3]
df = pd.DataFrame([lst], columns=all_columns)
X = pd.get_dummies(df, columns=["parameter_1"], drop_first=False)
X2 = pd.DataFrame(columns = cols_when_model_builds)
X = X.reindex(labels=X2.columns,axis=1)
result = model.predict(X)
Thus, "parameter_3" is a categorical feature and the other two parameters are integer features (numeric). However, when I pass a value to "parameter_3" it return an error:
ValueError: DataFrame.dtypes for data must be int, float, bool or category. When
categorical type is supplied, DMatrix parameter `enable_categorical` must
be set to `True`. Invalid columns:"parameter_2"
It should be noted that the code doesn't have problems with the "parameter_1" even though it is numeric as well
Setting "enable_categorical" won't work because the trouble making parameter is not categorical.
I am trying to create a Dropout Layer for my neural network using nn.Sequential() like this:
class DropoutLayer(nn.Module):
def __init__(self, p):
super().__init__()
self.p = p
def forward(self, input):
if self.training:
u1 = (np.random.rand(*input.shape)<self.p)
u1 *= input
return u1
else:
input *= self.p
model = nn.Sequential(Flatten(),DropoutLayer(p = 0.7),nn.LogSoftmax(dim = -1))
opt = torch.optim.Adam(modelDp.parameters(), lr=0.005)
train(modelDp, opt, 5)
But I get this error:
ValueError: optimizer got an empty parameter list
First, there is what I assume to be a small typo : you declare model = nn.Sequential(...) but then use modelDp.parameters(). I assume you just made a small copypaste mistake and these are actually the same thing.
This error is yielded because no layer in your model has trainable parameters, i.e parameters that will be affected by the gradient backpropagation step. Actually, this "network" cannot learn anything at all.
To get rid of the error and get an actual working neural network, you need to include the learning layers, which according to the previous error you had reported, are linear layers. That would be something like :
model = nn.Sequential(nn.Linear(784, 10), Flatten(), DropoutLayer(0.7), nn.LogSoftMax(dim=-1))
Now a couple additional remarks :
You may want to use the pytorch random tensors instead of Numpy's. It will be easier to deal with the devicewhen you will eventually want to move your network on GPU.
This code is going to yield another error as soon as you try it in eval mode because your second conditional branch in the forward method does not return anything. You may want to replace the instruction with return input * self.p
This might be a stupid question as nobody on fastai is trying to answer it. If it is one you can go ahead and tell me but also please tell me the answer, because right now I am completely lost.
I am currently working on a U-Net model for the segmentation of cells in microscopy images. Due to class imbalances and to amplify the importance of cell boundaries, I calculated a pixelwise weightmap for each image that I pass into fastai. Therefore I created a new ItemBase class to save labels and weights together:
class WeightedLabels(ItemBase):
"""
Custom ItemBase to store and process labels and pixelwise weights together.
Also handling the target_size of the labels.
"""
def __init__(self, lbl: Image, wgt: Image, target_size: Tuple = None):
self.lbl, self.wgt = lbl, wgt
self.obj, self.data = (lbl, wgt), [lbl.data, wgt.data]
self.target_size = target_size
...
I use extensive augmentation, like elastic deformation, mirroring and rotations on both weights and labels, as well as the original image. I determine the Loss with a custom Cross-entropy loss function that uses the weights to get the weighted loss for each pixel and averages them.
My problem is, that I do not get a very good performace. I have the feeling that might be because of fastai trying to predict the weights as well. My questions are:
Am I right to assume my model tries to predict both?
If so, how do I tell the learner what to use for updating the layers and to only predict part of my labels, while still applying augmentation to both?
Here's the code for how I implemented my custom LabelList and my custom ItemList:
class CustomSegmentationLabelList(ImageList):
"'Item List' suitable for WeightedLabels containing labels and pixelweights"
_processor = vision.data.SegmentationProcessor
def __init__(self,
items: Iterator,
wghts = None,
classes: Collection = None,
target_size: Tuple = None,
loss_func=CrossEntropyFlat(axis=1),
**kwargs):
super().__init__(items, **kwargs)
self.copy_new.append('classes')
self.copy_new.append('wghts')
self.classes, self.loss_func, self.wghts = classes, loss_func, wghts
self.target_size = target_size
def open(self, fn):
res = io.imread(fn)
res = pil2tensor(res, np.float32)
return Image(res)
def get(self, i):
fn = super().get(i)
wt = self.wghts[i]
return WeightedLabels(fn, self.open(wt), self.target_size)
def reconstruct(self, t: Tensor):
return WeightedLabels(Image(t[0]), Image(t[1]), self.target_size)
class CustomSegmentationItemList(ImageList):
"'ItemList' suitable for segmentation with pixelwise weighted loss"
_label_cls, _square_show_res = CustomSegmentationLabelList, False
def label_from_funcs(self, get_labels: Callable, get_weights: Callable,
label_cls: Callable = None, classes=None,
target_size: Tuple = None, **kwargs) -> 'LabelList':
"Get weights and labels from two functions. Saves them in a CustomSegmentationLabelList"
kwargs = {}
wghts = [get_weights(o) for o in self.items]
labels = [get_labels(o) for o in self.items]
if target_size:
print(
f'Masks will be cropped to {target_size}. Choose \'target_size \\= None \' to keep initial size.')
else:
print(f'Masks will not be cropped.')
y = CustomSegmentationLabelList(
labels, wghts, classes, target_size, path=self.path)
res = self._label_list(x=self, y=y)
return res
Also here the part, where I initiate my databunch object:
data = (CustomSegmentationItemList.from_df(img_df,IMG_PATH, convert_mode=IMAGE_TYPE)
.split_by_rand_pct(valid_pct=(1/N_SPLITS),seed=SEED)
.label_from_funcs(get_labels, get_weights, target_size=MASK_SHAPE, classes = array(['background','cell']))
.transform(tfms=tfms, tfm_y=True)
.databunch(bs=BATCH_SIZE))
I use the regular learner, where I pass in my U-Net model, the data, my loss function and some additional arguments that shouldn't really matter here. When trying to apply my model, after training it, it gives me two identical output tensors. I assume that one is probably for the labels and the other for the weights. Both have the following dimensions: (WxHxC). I do not understand why this is happening, because my model is supposed to only have one output in form of (WxHxC). If this happens during prediction, this probably also happens during training. How can I overcome this?
I'm trying to create an autoencoder in keras with bucketing where the input and the output have different time steps.
model = Sequential()
#encoder
model.add(Embedding(vocab_size, embedding_size, mask_zero=True))
model.add(LSTM(units=hidden_size, return_sequences=False))
#decoder
model.add(RepeatVector(max_out_length))
model.add(LSTM(units=hidden_size, return_sequences=True))
model.add(TimeDistributed(Dense(num_class, activation='softmax')))
For the input there is no problem as the network can accept different length inputs as long as the whole batch has the same length. However the problem is with the output size as its determined by the RepeatVector length and there is not easy way to change it.
Is there a solution for such a problem?
If you mean "inputs with variable lengths" and "outputs with the same lengths as the inputs", you can do this:
Warning: this solution must work with batch size = 1
You will need to create an external loop and pass each sample as a numpy array with the exact length
You cannot use masking in this solution, and the right output depends on the correct length of the input
This is a working code using Keras + Tensorflow:
Imports:
from keras.layers import *
from keras.models import Model
import numpy as np
import keras.backend as K
from keras.utils.np_utils import to_categorical
Custom functions to use in Lambda layers:
#this function gets the length from the original input
#and stores it in the final output of the encoder
def storeLength(x):
inputTensor = x[0]
storeInto = x[1] #the final output
length = K.shape(inputTensor)[1]
length = K.cast(length,K.floatx())
length = K.reshape(length,(1,1))
#will put length as the first element in the final output
return K.concatenate([length,storeInto])
#this function expands the length of the input in the decoder
def expandLength(x):
#lenght is the first element in the encoded input
length = K.cast(x[0,0],'int32') #or int64 if necessary
#the remaining elements are the actual data to be decoded
data = x[:,1:]
#a tensor with shape (length,)
length = K.ones_like(K.arange(0,length))
#make both length tensor and data tensor 3D and with paired dimensions
length = K.cast(K.reshape(length,(1,-1,1)),K.floatx())
data = K.reshape(data,(1,1,-1))
#this automatically repeats the elements based on the paired shapes
return data*length
Creating the models:
I assumed the output is equal to the input, but since you're using an Embedding, I made "num_classes" equal to the number of words.
For this solution, we use a branching, thus I had to use the functional API Model. Which will be way better later, because you will want to train with autoencoder.train_on_batch and then just encode with encoder.predict() or just decode with decoder.predict().
vocab_size = 100
embedding_size = 7
num_class=vocab_size
hidden_size = 3
#encoder
inputs = Input(batch_shape = (1,None))
outputs = Embedding(vocab_size, embedding_size)(inputs)
outputs = LSTM(units=hidden_size, return_sequences=False)(outputs)
outputs = Lambda(storeLength)([inputs,outputs])
encoder = Model(inputs,outputs)
#decoder
inputs = Input(batch_shape=(1,hidden_size+1))
outputs = Lambda(expandLength)(inputs)
outputs = LSTM(units=hidden_size, return_sequences=True)(outputs)
outputs = TimeDistributed(Dense(num_class, activation='softmax'))(outputs)
decoder = Model(inputs,outputs)
#autoencoder
inputs = Input(batch_shape=(1,None))
outputs = encoder(inputs)
outputs = decoder(outputs)
autoencoder = Model(inputs,outputs)
#see each model's shapes
encoder.summary()
decoder.summary()
autoencoder.summary()
Just an example with fake data and the method that should be used for training:
inputData = []
outputData = []
for i in range(7,10):
inp = np.arange(i).reshape((1,i))
inputData.append(inp)
outputData.append(to_categorical(inp,num_class))
autoencoder.compile(loss='mse',optimizer='adam')
for epoch in range(1):
for inputSample,outputSample in zip(inputData,outputData):
print(inputSample.shape,outputSample.shape)
autoencoder.train_on_batch(inputSample,outputSample)
for inputSample in inputData:
print(autoencoder.predict(inputSample).shape)
Let's say I have some data z=[1,2,3,4]
I am trying to fit this data to a model which is known, so the exercise is simply to find the value of an unknown parameter D
My log likelihood function looks like this
l(D|z)= \sum(\sqrt(z^2 + D^2))
I am trying to define this log likelihood function, z is the data which is a list and theta is the parameter vector which in this case is 1 dimensional
import scipy.optimize as op
import numpy as np
D_true = some given value
def f(z,theta):
D=theta
z2=[x**2 for x in z]
return np.sqrt(np.sum(z2 + D**2))
result = op.minimize(f, D_true,args=(z))
print result.x
But I am getting the error message unsupported operand type(s) for ** or pow(): 'list' and 'int'
and pointing towards return np.sqrt(np.sum(z2 + D**2))
Can anyone help me solve this issue?
as they say, "we have eyes, if we only but see." What I mean is, it's telling you where to look
"But I am getting the error message unsupported operand type(s) for ** or pow(): 'list' and 'int'
and pointing towards return np.sqrt(np.sum(z2 + D**2))"
and it's even telling you the problem: you cannot add a list object to an int object. By the way you wrote it, I think you are assuming Python will broadcast, but it will not do that unless at least one of the objects is a numpy (ndarray) object.
You probably wanted the value D ** 2 to be added to each entry of z2. One option is to write
return np.sqrt(np.sum(z2 + D **2 * np.ones_like(z2)))
another option is
return np.sqrt(np.sum(np.array(z2) + D ** 2))
It's awkward to me to see lists being used with numpy functions, you may consider working exclusively with numpy arrays. For example, you wouldn't have run into this problem if you opted to do that from the start.
If you had a numpy array, you can use ufuncs instead of list comprehensions.
z = np.arange(1,5)
z2 = z ** 2
is the same as
z2 = [x ** 2 for x in z]
but returns a numpy array instead of a list.
I think you should replace D = theta by D, = theta since theta is a list. This way we unpack it on the fly.