Keras provides categorical_accuracy, sparse_categorical_accuracy and top_k_categorical_accuracy metrics.
I would like to define a sparse version of top_k_categorical_accuracy.
My tentative implementation, is as follows:
def top_k_sparse_categorical_accuracy(y_true, y_pred, z=5):
return K.mean(K.in_top_k(y_pred, K.max(y_true, axis=-1), z), axis=-1)
This is modeled after top_k_categorical_accuracy and sparse_categorical_accuracy
def sparse_categorical_accuracy(y_true, y_pred):
return K.cast(K.equal(K.max(y_true, axis=-1),
K.cast(K.argmax(y_pred, axis=-1), K.floatx())),
K.floatx())
def top_k_categorical_accuracy(y_true, y_pred, k=5):
return K.mean(K.in_top_k(y_pred, K.argmax(y_true, axis=-1), k), axis=-1)
However, this does not work because the second argument of K.in_top_k should be a tensor of integers, and here I have a tensor of floats, and K.cast can only cast toward floats.
Is there a way to perform a cast to int in the Keras backend? Or is there another way to implement this metric?
Related
i would like to ask a question regarding the nn.BatchNorm1d in PyTorch.
I have one main tensor, which has shape [B, 3, N]. Then, i have two additional tensors which have shape [B, 3, V1] and [B, 3, V2]. I will concatenate the main tensor with the two tensors separately, to construct new tensors [B, 3, N+V1] and [B, 3, N+V2].
I pass my tensors to a plain MLP (consists of conv1d and batchnorm1d). Ideally, i want to predict something "point-wise", like no matter what the number of dimension 2, it has some consistent prediction only given the value. However, the batchnorm1d will have different results given input [B, 3, N+V1] and [B, 3, N+V2], while i am only focusing on first N points in 2nd dimension.
import torch
import torch.nn as nn
# nn.BatchNorm1d
B=2
dim=64
N=40000
V1=1000
v2=2000
torch.manual_seed(0)
x = torch.rand(B, dim, N) # here imgs are flattened from 28x28
v1 = torch.rand(B, dim, V1)
v2 = torch.rand(B, dim, v2)
layer = nn.BatchNorm1d(dim) # batch norm is done on channels
out2 = layer(torch.cat((x, v1), dim=2))
out3 = layer(torch.cat((x, v2), dim=2))
torch.equal(out2[:, :, :N], out3[:, :, :N])
Is there any possible way to have consistent prediction of first N points?
Is this more along the lines of what you're looking for? Normalizing just across the channels?
out2 = torch.cat((x, v1), dim=2) / torch.linalg.norm(torch.cat((x, v1), dim=2), dim=1, keepdim=True)
out3 = torch.cat((x, v2), dim=2) / torch.linalg.norm(torch.cat((x, v2), dim=2), dim=1, keepdim=True)
torch.equal(out2[:, :, :N], out3[:, :, :N])
# True
I think if you want to do something like this within pytorch nn libraries you'll need to transpose your channels and feature dimensions that way you can use LayerNorm1d or InstanceNorm. See here for a nice visual example of the different normalization techniques
Update answer:
In case you want to use an nn module specifically. InstanceNorm or GroupNorm could also get you the response. However the number of channels now differs between the two so you'll need two distinct layers.
layer1 = nn.GroupNorm(V1+N, V1+N)
layer2 = nn.GroupNorm(V2+N, V2+N)
out2 = layer1(torch.cat((x, v1), dim=2).transpose(1,2))
out3 = layer2(torch.cat((x, v2), dim=2).transpose(1,2))
torch.equal(out2[:, :N, :], out3[:, :N, :])
True
I am trying to implement a custom loss function in a Pytorch Autoencoder.
The loss function tries to maximize the cosine similarity between a given output tensor U (a vector) and 100 random vectors J where both U and J have the same dimension of [300]. This is repeated for each batch.
Suppose we have 30 items per batch, then the output tensor is
train_Y.shape = [30,300]
Random_vectors.shape = [30,100,300]
I can implement the loss function in two ways:
All_Y =[]
for Y,z_r in zip(train_y, random_vectors):
Y_cosine_list =[]
for z in z_r:
cosi = torch.dot(Y,z) / (torch.norm(Y)*torch.norm(z))
Y_cosine_list.append(cosi)
All_Y.append(Y_cosine_list)
All_Y = torch.tensor(All_Y).to(device)
train_loss = torch.sum(torch.abs(All_Y))/dim_0
train_loss = torch.tensor(train_loss.data, requires_grad = True)
or
train_Y = torch.zeros([dim_0, 100])
for i, (Y,z_r) in enumerate(zip(train_Y, random_vectors)):
for j,z in enumerate(z_r):
train_Y[i,j] = cos(Y,z)
train_Y = train_Y.to(device)
train_loss = torch.sum(torch.abs(train_Y))/dim_0
The second one is more elegant and to the point. However it is giving a "Cuda illegal memory access error". I have checked that the memory is not exceeded in either case. Is there anything wrong with the second implementation?
The first implementation is inelegant and I am not sure that it makes sense from a neural net optimization perspective. But it does not give errors and am able to complete training for all the epochs.
Ps: I have tried encapsulating this code block in a loss_fn method but I get the same illegal memory access error.
I have tried everything that I could find for the illegal memory access error - changing GPUs, removing a torch.stack block etc. But I can't seem to get rid of the problem.
Here is a vectorized way to do it
class CosineLoss(nn.Module):
def __init__(self, ):
super().__init__()
pass
def forward(self, x, y):
"""
Args:
x (torch.tensor): [batchsize, N, M] - tensor.
y (torch.tensor): [batchsize, M] - tensor.
Returns:
torch.tensor: scalar mean cosine loss
"""
# dot product along dimension 'm' i.e multiply and sum along 'm'.
dotp = torch.einsum("bm, bnm -> bn", y, x)
# L2 norm along dimension 'm' and multiply by broadcasting
length = torch.norm(y, dim=-1)[:, None]*torch.norm(x, dim=-1)
# cosine = dotproduct of unit vectors
cos = dotp/length
return cos.mean()
def test():
b, n, m = 30, 100, 300
train_Y = torch.randn(b, m, device='cuda')
random_vectors = torch.randn(b, n, m, requires_grad=True, device='cuda')
print(f'{random_vectors.grad = }')
cosineloss = CosineLoss()
loss = cosineloss(random_vectors, train_Y)
print(f'{loss = }')
loss.backward()
print(f'{random_vectors.grad.shape = }')
References:
einsum
broadcasting
julia language deep learning framework,
This is a quick start for Knet.jl,
https://denizyuret.github.io/Knet.jl/latest/tutorial/#Tutorial
ENV ["COLUMNS"] = 72
using Knet, MLDatasets, IterTools
struct Conv; w; b; f; end
(c :: Conv) (x) = c.f. (pool (conv4 (c.w, x). + C.b))
Conv (w1, w2, cx, cy, f = relu) = Conv (param (w1, w2, cx, cy), param0 (1,1, cy, 1), f);
The complex type Conv has three fields, w, b, and f.
The Conv type c (x) function broadcasts the next function with the f function.
The inner product of the w matrix and the x matrix is ​​calculated with conv4 (c.w, x), and the addition with c.b is performed with. +.
I don't know what the pool is looking for in that matrix.
This (pool (conv4 ...)) is passed through the relu activation function.
At the last Conv (w1, w2, cx, cy, f = relu) = Conv (param (w1, w2, cx, cy), param0 (1,1, cy, 1), f);
I don't know what I'm trying to do.
This is the situation of understanding.
What are you trying to do, especially in the pool?
Why are there two params on the 5th line?
I do not know.
Actually, the layer does a convolution followed by max pooling:
Pooling layers reduce the dimensions of the data by combining the outputs of neuron clusters at one layer into a single neuron in the next layer. Local pooling combines small clusters, typically 2 x 2. Global pooling acts on all the neurons of the convolutional layer. There are two common types of pooling: max and average. Max pooling uses the maximum value of each cluster of neurons at the prior layer, while average pooling instead uses the average value. (source: https://en.wikipedia.org/wiki/Convolutional_neural_network#Pooling_layers)
There are two params on the 5th line, because a convolutional layers has two trainable parameters: the kernel weights w and the bias b. The function param (and param0) initialize them with the correct size and mark them as trainable parameters that will be updated during the optimization.
To learn neural networks, I found these examples: linear regression and a simple feed-forward network (multilayer perceptron) quite useful.
If x is a torch.Tensor of dtype torch.float then are the operations x.item() and float(x) exactly the same?
The operations x.item() and float(x) are not the same.
From the documentation of item(), it can be used to get the value of tensor as a Python number(only from tensors containing a single value). It basically returns the value of the tensor as it is. It does not make any modifications to the tensor.
Where as float() is for converting its input to a floating point number, when possible. Find the documentation here.
To see the difference, consider another Tensor y of dtype int64:
import torch
y = torch.tensor(2)
print(y, y.dtype)
>>> tensor(2) torch.int64
print('y.item(): {}, float(y): {}'.format(y.item(), float(y)))
>>> y.item(): 2, float(y): 2.0
print(type(y.item()), type(float(y)))
>>> <class 'int'> <class 'float'>
Note that float(y) does not convert the type in-place. You would need to assign it in case you need that change. Like:
z = float(y)
print('y.dtype: {}, type(z): {}'.format(y.dtype, type(z)))
>>> y.dtype: torch.int64, type(z): <class 'float'>
We can see that z is not a torch.Tensor. It is simply a floating point number.
The float() operation is not to be confused with self.float(). This operation performs the Tensor dtype conversion (not in-place, needs assignment).
print('y.float(): {},\n y.float().dtype: {},\n y: {},\n y.dtype'.format(y.float(), y.float().dtype, y, y.dtype))
y.float(): 2.0,
y.float().dtype: torch.float32,
y: 2,
y.dtype: torch.int64
The Keras docs for the softmax Activation states that I can specify which axis the activation is applied to. My model is supposed to output an n by k matrix M where Mij is the probability that the ith letter is symbol j.
n = 7 # number of symbols in the ouput string (fixed)
k = len("0123456789") # the number of possible symbols
model = models.Sequential()
model.add(layers.Dense(16, activation='relu', input_shape=((N,))))
...
model.add(layers.Dense(n * k, activation=None))
model.add(layers.Reshape((n, k)))
model.add(layers.Dense(output_dim=n, activation='softmax(x, axis=1)'))
The last line of code doesn't compile as I don't know how to correctly specify the axis (the axis for k in my case) for the softmax activation.
You must use an actual function there, not a string.
Keras allows you to use a few strings for convenience.
The activation functions can be found in keras.activations, and they're listed in the help file.
from keras.activations import softmax
def softMaxAxis1(x):
return softmax(x,axis=1)
.....
......
model.add(layers.Dense(output_dim=n, activation=softMaxAxis1))
Or even a custom axis:
def softMaxAxis(axis):
def soft(x):
return softmax(x,axis=axis)
return soft
...
model.add(layers.Dense(output_dim=n, activation=softMaxAxis(1)))