Heatmap with 2d smoothing using levelplot function from latticeExtra - heatmap

I have been trying to reproduce this plot using the code from here
My last attempts got me pretty close but I can't find the appropiate way to make the graph look like I want.
In my data, z is the numerical result of a simulation carried out under x and y conditions and I want to map that relationship, like you could do simply by a scatterplot but prettier.
My df is pretty large but it has this aspect:
x -> continous variable ranging from 120 - 300
y -> continous variable ranging from 0.2 - 1.8
z -> continous variable ranging from -0.0001 to 3000
This is my First attempt generated with this code
levelplot(z ~ x * y,
data,
panel = panel.levelplot.points,
cex = 0.7,
col.regions = rocket(25, alpha = .8, direction = -1),
colorkey = list(at = (breaks = c(1, 20, 50, 150, 500, Inf))),
scales = list(x=list(at = c(120, 140, 160, 180, 200, 220, 240, 260, 280, 299))),
xlab = "x",
ylab = "y"
) +
layer_(panel.2dsmoother(..., n = 100))
The problem is that, while points are a fair representation of my data, the 2d layer is not. I want to specially stress the difference between <1 and >=1 so my breaks start by 1, the rest would be the scale depending on z values for different experiments (this one reaches over 3000, other just up to 30)
I have also attempted to sepparate both graphs and you can take a look at the scatter plot and the 2dSmooth panel. Here is the code for both:
## Scatter
levelplot(z ~ x * y,
data,
panel = panel.levelplot.points,
cex = 0.7,
at = c(-Inf, 1, 30, 100, 500, Inf),
col.regions = rocket(25, alpha = .8, direction = -1),
colorkey = list(
labels = c("", "1", "30", "100", "500", "")
),
scales = list(x=list(at = c(120, 140, 160, 180, 200, 220, 240, 260, 280, 299))),
xlab = "x",
ylab = "y" )
## 2dSmooth
levelplot(z ~ x * y,
data,
panel = panel.2dsmoother,
n = 200,
cuts = 5,
col.regions = rocket(25, alpha = .8, direction = -1),
colorkey = T,
scales = list(x=list(at = c(120, 140, 160, 180, 200, 220, 240, 260, 280, 299))),
xlab = "x",
ylab = "y"
)
Note I am having troubles with CUT and AT.
When I pass the AT argument to my 2dSmooth graph, the result is absurd and I have no idea why.
However, the CUT result from the 2dSmooth plot are addequate, the problem is that I cannot label them as I don't know the limits the function has taken. If there would be a way of correctly labelling that graph, that would be it. Otherwise, I need to combine both.
¿Any idea where I am messing up?
¿Any possibility of having my 2dsmooth graph done with other libraries? So far I haven't found anything similar to levelplot() with panel.2dsmoother
Thank you for your help

Your problem seems very specific to your data, so it's difficult to answer without having access to it. But generally speaking,
I'm a bit surprised that you are getting a different set of cut points with your 2dSmooth graph, because when you don't specify at, the default values should only depend on the data (which remain unchanged) and not on the panel function. I'm not sure what's happening.
panel.2dsmoother() basically just fits a regression surface to your data, by default using loess(). If your goal is to remove the influence of "outliers" by tuning your at values to the fitted surface instead of the data, you would need to do that externally. You can mimic the code of panel.2dsmoother() to do this, e.g.,
n <- 100
mod <- loess(z ~ x * y, data)
xseq <- seq(120, 300, length = n)
yseq <- seq(0.2, 1.8, length = n)
grid <- expand.grid(x = xseq, y = yseq)
fit <- predict(mod, grid)
You could now visualize this surface using a standard levelplot() call using
(p <- levelplot(fit ~ x + y, grid))
and the corresponding at values should be roughly
do.breaks(range(fit), 5)
You can get the exact values using
trellis.panelArgs(p, 1)$at

Related

Why is my segmentation output appear as a boundary convolutional filter instead of an output mask

So I am building a neural network to look at medical CT images and try to create a segmentation mask of the heart within chest CT imaging. Ive trainied the neural network with a DICE score around 80%. The inputs and masks are all standardized and appropriate, but the output of my model appears to be making a mask that is able to create a silhoute of all soft tissue within the body, instead of just the heart alone. Does anyone know where I may be messing up?
Images:
Input Data ex.
Input Mask ex.
Prediction of model on an image that was used to train the model:
model.predict() output
Model code:
def dense_unet_256(inputs, filters, depth):
#Model Creation
# Define kwargs dictionary
kwargs = {
'kernel_size': (1, 3, 3),
'padding': 'same',
'bias_initializer': 'zeros'
}
# Define lambda functions#
conv = lambda x, filters, strides: layers.Conv3D(filters=int(filters), strides=(1, strides, strides), **kwargs)(x)
norm = lambda x: layers.BatchNormalization()(x)
relu = lambda x: layers.LeakyReLU()(x)
# Define stride-1, stride-2 blocks#
conv1 = lambda filters, x: relu(norm(conv(x, filters, strides=1)))
conv2 = lambda filters, x: relu(norm(conv(x, filters, strides=2)))
# Define single transpose#
tran = lambda x, filters, strides: layers.Conv3DTranspose(filters=int(filters), strides=(1, strides,strides),**kwargs)(x)
# Define transpose block#
tran2 = lambda filters, x: relu(norm(tran(x, filters, strides=2)))
concat = lambda a, b: layers.Concatenate()([a, b])
# Define Dense Block#
def dense_block(filters, input, DB_depth):
ext = 2 + DB_depth
outside_layer = input
for _ in range(0, int(ext)):
inside_layer = conv1(filters, outside_layer)
outside_layer = concat(outside_layer, inside_layer)
return outside_layer
def td_block(conv1_filters, conv2_filters, input, DB_depth):
TD = conv1(conv1_filters, conv2(conv2_filters, input))
DB = dense_block(conv1_filters, TD, DB_depth)
return DB
def tu_block(conv1_filters, tran2_filters, input, td_input, DB_depth):
TU = conv1(conv1_filters, tran2(tran2_filters, input))
C = concat(TU, td_input)
DB = dense_block(conv1_filters, C, DB_depth)
return DB
TD1 = td_block(filters * 1, filters * 1, inputs, depth)
TD2 = td_block(filters * 1.5, filters * 1, TD1, 1+depth)
TD3 = td_block(filters * 2, filters * 1.5, TD2, 2+depth)
TD4 = td_block(filters * 2.5, filters * 2, TD3, 3+depth)
TD5 = td_block(filters * 3, filters * 2.5, TD4, 4+depth)
TU1 = tu_block(filters * 2.5, filters * 3, TD5, TD4, 4+depth)
TU2 = tu_block(filters * 2, filters * 2.5, TU1, TD3, 3+depth )
TU3 = tu_block(filters * 1.5, filters * 2, TU2, TD2, 2+depth)
TU4 = tu_block(filters * 1, filters * 1.5, TU3, TD1, 1+depth)
TU5 = tran2(filters * 1, TU4)
logits = {}
logits['lbl'] = layers.Conv3D(filters=2, name = 'lbl', **kwargs)(TU5)
model = Model(inputs=inputs, outputs=logits['lbl'])
return model
Ive tried training with different loss and metric functions without much change, I've confirmed that the image and masks going into training only identify the heart. I have done some hyperparamter tuning with epochs and learning rate, which with some tweaking have made the output mask of the heart a slightly different value when compared to chest wall tissue, but the ranges are not standardized between images so using a clip to create the mask wont work. I personally feel like it has something to do with the activation of the loaded model, but I am unsure exactly how to prove that. Any advice would be apprecaited!
Merry Christmas and Happy Holidays everyone!

Linear Regression using sklearn

I have a model fitted with data but having trouble using the predict function.
d = {'df_Size': [1, 3, 5, 8, 10, 15, 18], 'RAM': [3676, 6532, 9432, 13697, 16633, 23620, 27990]}
df = pd.DataFrame(data=d)
df
X = np.array(df['df_Size']).reshape(-1, 1)
y = np.array(df['RAM']).reshape(-1, 1)
model = LinearRegression()
model.fit(X, y)
print(regr.score(X, y))
then when I try to predict on
X_Size = 25
X_Size
prediction = model.predict(X_Size)
I get the following error
ValueError: Expected 2D array, got scalar array instead:
array=25.
Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.
I think I am passing the 25 in the wrong format but would just like some help on getting the response for Ram considering the 25 rows.
Thanks,
You need to pass the predictor in the same shape (basically 1 column):
X.shape
Out[11]: (7, 1)
You can do:
model.predict(np.array(25).reshape(1,1))

is there a way to store list values inside a loop in python

I am trying to detect object using yolov3 and OpenCV with python
I want to store the coordinate of the detected object in JSON format but I am just getting the last coordinate how can I store all of my value in my list
for i in range(len(boxes)):
if i in indexes:
x, y, w, h = boxes[i]
label = str(classes[class_ids[i]])
labelSize, baseLine = cv2.getTextSize(label, cv2.FONT_HERSHEY_SIMPLEX, 0.5, 1)
y = max(y, labelSize[1])
cv2.rectangle(img, (x, y - round(1.5 * labelSize[1])), (x + round(1.5 * labelSize[0]), y + baseLine),
(255, 255, 255), cv2.FILLED)
cv2.rectangle(img, (x, y), (x+w, y+h), (255, 255, 255), 1)
# print ("x1=",(left,right),"y:",top,bottom,label)
cv2.putText(img, label, (x, y), cv2.FONT_HERSHEY_SIMPLEX, 0.75, (0, 0, 0), 1)
book = {
"frame_url": "frame",
"frame_width": width,
"frame_height": height,
"objeler": [
{
"tur": label,
"x0": x,
"y0": y,
"x1": x+w,
"y1": y+h
}]
}
s = json.dumps(book, indent=2)
with open("f.json", "w")as f:
f.write(s)
cv2.imshow("Image", img)
cv2.waitKey(0)
cv2.destroyAllWindows()
And this is the output I am getting
{
"frame_url": "frame",
"frame_width": 768,
"frame_height": 360,
"objeler": [
{
"tur": "arac",
"x0": 273,
"y0": 256,
"x1": 299,
"y1": 269
}
]
}
You should create a list outside of the for loop. Then, in the loop, you append to that list. When the loop ends you convert the list to json.
A general example:
import json
# create an empty list
books = []
for x in range(10):
# add to the list
books.append({"key":x})
print(json.dumps(books))
Edit: question in comments.
You can use glob to get all filenames like this:
import glob
# location of images
folder = ""
# print all filenames of .jpg in folder
for filename in glob.glob(folder + "*.jpg"):
image = cv.imread(folder+filename)
# Process image...

define a function in which min() is used

I am trying to define a function in which I want a part of the function limited. I try to do this by using min() but it returns
The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
My code:
def f(x, beta):
K_w = (1+((0.5*D)/(0.5*D+x))**2)**2
K_c = min(11,(3.5*(x/D)**(-0.5))) # <-- this is what gives me the problem. It should limit K_c to 11, but that does not work.
K_tot = (K_c**2+K_w**2+2*K_c*K_w*np.cos(beta))**0.5
return K_tot
x = np.linspace(0, 50, 100)
beta = np.linspace(0, 3.14, 180)
X, Y = np.meshgrid(x, beta)
Z = f(X, Y)
fig = plt.figure()
ax = plt.axes(projection='3d')
ax.contour3D(X, Y, Z, 100, cmap = 'viridis')
ax.set_xlabel('x')
ax.set_ylabel('y')
ax.set_zlabel('z');
I expected K_c to be limited to 11, but it gave a
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
I might be making a rookie mistake, but help is much appreciated!
Consider using np.clip of which its references can be found here.
np.clip(3.5*(x/D)**(-0.5), None, 11)
for your case.
For example,
>>> import numpy as np
>>> np.clip([1, 2, 3, 15], None, 11)
array([ 1, 2, 3, 11])
The problem with your code is that min is comparing a number with a list of which this is not expected.
Alternatively, here is a list comprehension approach:
A = [1, 2, 3, 15]
B = [min(11, a) for a in A]
print(B)

Insert graph from function call into subplots

I have a function which produces exactly the graphs I need:
def graph(x, y, yName, dataset, colour, residuals = False, calc_tau = False):
"""
Takes as an argument two arrays, one for x and one y, a string for the
name of the y-variable, and two colours (which could be strings, values
or iterators).
Plots a scatter graph of the array against log(z), along with the best fit
line, its equation and its regression coefficient, as well as Kendall's tau
and the total number of points plotted.
"""
arrayresults, arrayresids, arrayparams, arrayfit, arrayequation, arraytau = computeLinearStats(x,
y,
yName, calc_tau)
count = np.count_nonzero(~np.logical_or(np.isnan(x), np.isnan(y)))
# arrayequation = 'r\'' + yName +arrayequation[arrayequation.index('='):]
plt.scatter(x, y,
# label = arrayequation,
s = 10,
alpha = 0.5,
c = colour)
if calc_tau: #if calc_tau is set to True, display the value
#in the legend along with equation, r and n
plt.plot(x, arrayfit,
label = r'''%s
r=%g
$\tau$=%g
n=%d'''%(arrayequation, arrayparams[2], arraytau[0], count),
c = colour)
else: #otherwise just display equation, r and n
plt.plot(x, arrayfit,
label = r'''%s
$r=%g$
$n=%d$
$r^2n=%.2f$'''%(arrayequation, arrayparams[2], count, arrayresults.nobs*arrayparams[2]**2),
c = colour)
legendfont = 16
labelfont = 20
plt.xlabel('$log_{10}(z)$', fontsize = labelfont)
plt.ylabel('Magnitude combination, %s dataset'%dataset, fontsize = labelfont)
plt.legend(fontsize = legendfont)
plt.xticks(fontsize = labelfont)
plt.yticks(fontsize = labelfont)
plt.grid(True, which = 'both')
# plt.title(r'The three best high-$r$ combinations in both Hunstead and MgII', fontsize = labelfont)
if residuals:
plotResids(x, y, yName, dataset, colour)
plt.show()
return arrayresults, arrayresids, arrayparams, arrayfit, arrayequation, arraytau
which I can call multiple times to produce, for example:
and
It's kind of an ad hoc function, but it produces the graphs how I want them to look. Is there an easy way of combining the graphs output by multiple calls to the function into a set of subplots? I've tried something like
x = sources['z']
y1 = I-W2
y2 = W3-U
fig,ax = plt.subplots(2,1, sharex=True, sharey=False, gridspec_kw={'hspace': 0})
fig.suptitle('Graphs')
ax[0].scatter(x, y1, c='red', s = 10, alpha = 0.3)
ax[1].scatter(x, y2, c='purple', s = 10, alpha = 0.3)
but I'd like them with all the accoutrements. Ideally, I'd like ax1 and ax2 to call my graphing function and display the output in subplots. Is this possible?
EDIT: thanks to a comment, I'm able to use
x = sources['z']
y1 = I-W2
y2 = W3-U
fig,ax = plt.subplots(2,1, sharex=True, sharey=True, gridspec_kw={'hspace': 0})
ax1 = qf.graph(x, y1, 'I-W2', dataset, 'blue', ax[0], residuals = False, calc_tau = False) #2 rows, 1 column, first plot
ax2 = qf.graph(x, y2, 'W3-U', dataset, 'red', ax[1], residuals = False, calc_tau = False) #2 rows, 1 column, second plot
fig.suptitle('Graphs')
to produce
but how can I get one graph into the top position (ax[0])?