Related
I'm trying to run an Cox-regression using the Firth penalized likelihood. When I run the univariate analysis as a separate analysis I got a different result than when I run that into a function to apply to several columns. Summarizing my data:
OS_months <- c(33.6, 52.8, 7.2, 18, 80.4, 109.2, 16.8, 108, 2.4, 51.6, 33.6,
38.4, 7.2, 12, 144, 2.4, 68.4, 58.8, 52.8, 142.8, 42, 8.4, 115.2,
10.8, 103.2, 66, 18, 141.6, 66, 144, 13.2, 64.8, 38.4, 20.4,
81.6, 86.4, 105.6, 75.6, 51.6, 115.2, 21.6, 201.6, 62.4, 81.6,
157.2, 109.2, 170.4, 44.4, 108, 40.8)
OS_status <- c(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1,
0, 0, 0, 0, 0, 0, 1, 0, 0)
TSPAN6 <- c(0, 0, 0, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1,
1, 0, 1, 1, 0, 0, 1, 0, 1, 1, 1, 0, 0, 0, 0, 0, 1, 1, 0, 1, 1,
0, 0, 1, 0, 1, 0, 1, 0, 0)
HR <- data.frame(OS_months, OS_status, TSPAN6)
Running the analysis as separate analysis.
fit=coxphf(Surv(OS_months,OS_status)~TSPAN6,data=HR)
fit
Result
coxphf(formula = Surv(OS_months, OS_status) ~ TSPAN6, data = HR)
Model fitted by Penalized ML
Confidence intervals and p-values by Profile Likelihood
coef se(coef) exp(coef) lower 0.95 upper 0.95 Chisq p
TSPAN6 1.801352 1.051748 6.057834 1.078865 62.16828 4.200166 0.04042002
Likelihood ratio test=4.200166 on 1 df, p=0.04042002, n=50
Now into a created function.
gene<-function(x){
fit=coxphf(Surv(OS_months,OS_status)~x,data=HR)
}
a<-lapply(HR[3],FUN=gene)
a
Now the result is:
$TSPAN6
coxphf(formula = Surv(OS_months, OS_status) ~ x, data = HR)
Model fitted by Penalized ML
Confidence intervals and p-values by Profile Likelihood
coef se(coef) exp(coef) lower 0.95 upper 0.95 Chisq p
x 0.7413234 0.9106665 2.098711 0.4052701 12.6725 0.8079772 0.368719
Likelihood ratio test=0.8079772 on 1 df, p=0.368719, n=50
Can you telling me what is happening here? I will thank you very much.
I am also new to this field, and I tried your code. I thought the problem occurs in the origin dataset instead of running in a loop. Since
1) coxphf(formula = Surv(OS_months, OS_status) ~ TSPAN6, data = HR)
and
2) coxphf(formula = Surv(OS_months, OS_status) ~ HR$TSPAN6, data = HR)
give different results. 2) returns the same result as your loop result.
Also, I tried the example data provided by coxphf():
time<-c(1,2,3,4)
cens<-c(1,1,1,0)
x<-c(1,1,0,1)
sim<-cbind(time,cens,x)
sim<-data.frame(sim)
coxphf(sim, formula=Surv(time,cens)~x)
gene <- function(var){
+ fit=coxphf(Surv(time,cens)~var, data=sim)
+ return(fit)
+ }
gene(var=x)
The results return same as listed below:
coxphf(formula = Surv(time, cens) ~ x, data = sim)
Model fitted by Penalized ML
Confidence intervals and p-values by Profile Likelihood
coef se(coef) exp(coef) lower 0.95 upper 0.95 Chisq p
x -0.05065785 1.223583 0.9506039 0.1182198 10.9519 0.002226143 0.9623682
Likelihood ratio test=0.002226143 on 1 df, p=0.9623682, n=4
coxphf(formula = Surv(time, cens) ~ var, data = sim)
Model fitted by Penalized ML
Confidence intervals and p-values by Profile Likelihood
coef se(coef) exp(coef) lower 0.95 upper 0.95 Chisq p
var -0.05065785 1.223583 0.9506039 0.1182198 10.9519 0.002226143 0.9623682
Likelihood ratio test=0.002226143 on 1 df, p=0.9623682, n=4
I found a few SO posts on related issues which were unhelpful. I finally figured it out and here's how to read the contents of a .json file. Say the path is /home/xxx/dnns/test/params.json, I want to turn the dictionary in the .json into a Prolog dictionary:
{
"type": "lenet_1d",
"input_channel": 1,
"output_size": 130,
"batch_norm": 1,
"use_pooling": 1,
"pooling_method": "max",
"conv1_kernel_size": 17,
"conv1_num_kernels": 45,
"conv1_stride": 1,
"conv1_dropout": 0.0,
"pool1_kernel_size": 2,
"pool1_stride": 2,
"conv2_kernel_size": 12,
"conv2_num_kernels": 35,
"conv2_stride": 1,
"conv2_dropout": 0.514948804688646,
"pool2_kernel_size": 2,
"pool2_stride": 2,
"fcs_hidden_size": 109,
"fcs_num_hidden_layers": 2,
"fcs_dropout": 0.8559119274655482,
"cost_function": "SmoothL1",
"optimizer": "Adam",
"learning_rate": 0.0001802763794651928,
"momentum": null,
"data_is_target": 0,
"data_train": "/home/xxx/data/20180402_L74_70mm/train_2.h5",
"data_val": "/home/xxx/data/20180402_L74_70mm/val_2.h5",
"batch_size": 32,
"data_noise_gaussian": 1,
"weight_decay": 0,
"patience": 20,
"cuda": 1,
"save_initial": 0,
"k": 4,
"save_dir": "DNNs/20181203090415_11_created/k_4"
}
To read a JSON file with SWI-Prolog, query
?- use_module(library(http/json)). % to enable json_read_dict/2
?- FPath = '/home/xxx/dnns/test/params.json', open(FPath, read, Stream), json_read_dict(Stream, Dicty).
You'll get
FPath = 'DNNs/test/k_4/model_params.json',
Stream = <stream>(0x7fa664401750),
Dicty = _12796{batch_norm:1, batch_size:32, conv1_dropout:0.
0, conv1_kernel_size:17, conv1_num_kernels:45, conv1_stride:
1, conv2_dropout:0.514948804688646, conv2_kernel_size:12, co
nv2_num_kernels:35, conv2_stride:1, cost_function:"SmoothL1"
, cuda:1, data_is_target:0, data_noise_gaussian:1, data_trai
n:"/home/xxx/Downloads/20180402_L74_70mm/train_2.h5", data
_val:"/home/xxx/Downloads/20180402_L74_70mm/val_2.h5", fcs
_dropout:0.8559119274655482, fcs_hidden_size:109, fcs_num_hi
dden_layers:2, input_channel:1, k:4, learning_rate:0.0001802
763794651928, momentum:null, optimizer:"Adam", output_size:1
30, patience:20, pool1_kernel_size:2, pool1_stride:2, pool2_
kernel_size:2, pool2_stride:2, pooling_method:"max", save_di
r:"DNNs/20181203090415_11_created/k_4", save_initial:0, type
:"lenet_1d", use_pooling:1, weight_decay:0}.
where Dicty is the desired dictionary.
If you want to define this as a predicate, you could do:
:- use_module(library(http/json)).
get_dict_from_json_file(FPath, Dicty) :-
open(FPath, read, Stream), json_read_dict(Stream, Dicty), close(Stream).
Even DEC10 Prolog released 40 years ago could handle JSON just as a normal term . There should be no need for a specialized library or parser for JSON because Prolog can just parse it directly .
?- X={"a":3,"b":"hello","c":undefined,"d":null} .
X = {"a":3, "b":"hello", "c":undefined, "d":null}.
?-
I am trying to fit a curve to a set of data points but would like to preserve certain characteristics.
Like in this graph I have curves that almost end up being linear and some of them are not. I need a functional form to interpolate between the given data points or past the last given point.
The curves have been created using a simple regression
def func(x, d, b, c):
return c + b * np.sqrt(x) + d * x
My question now is what is the best approach to ensure a positive slope past the last data point(s) ??? In my application a decrease in costs while increasing the volume doesn't make sense even if the data says so.
I would like to keep the order as low as possible maybe ˆ3 would still be fine.
The data used to create the curve with the negative slope is
x_data = [ 100, 560, 791, 1117, 1576, 2225,
3141, 4434, 6258, 8834, 12470, 17603,
24848, 35075, 49511, 69889, 98654, 139258,
196573, 277479, 391684, 552893, 780453, 1101672,
1555099, 2195148, 3098628, 4373963, 6174201, 8715381,
12302462, 17365915]
y_data = [ 7, 8, 9, 10, 11, 12, 14, 16, 21, 27, 32, 30, 31,
38, 49, 65, 86, 108, 130, 156, 183, 211, 240, 272, 307, 346,
389, 436, 490, 549, 473, 536]
And for the positive one
x_data = [ 100, 653, 950, 1383, 2013, 2930,
4265, 6207, 9034, 13148, 19136, 27851,
40535, 58996, 85865, 124969, 181884, 264718,
385277, 560741, 816117, 1187796, 1728748, 2516062,
3661939, 5329675, 7756940, 11289641, 16431220, 23914400,
34805603, 50656927]
y_data = [ 6, 6, 7, 7, 8, 8, 9, 10, 11, 12, 14, 16, 18,
21, 25, 29, 35, 42, 50, 60, 72, 87, 105, 128, 156, 190,
232, 284, 347, 426, 522, 640]
The curve fitting is simple done by using
popt, pcov = curve_fit(func, x_data, y_data)
For the plot
plt.plot(xdata, func(xdata, *popt), 'g--', label='fit: a=%5.3f, b=%5.3f, c=%5.3f' % tuple(popt))
plt.plot(x_data, y_data, 'ro')
plt.xlabel('Volume')
plt.ylabel('Costs')
plt.show()
A simple solution might just look like this:
import matplotlib.pyplot as plt
import numpy as np
from scipy.optimize import least_squares
def fit_function(x, a, b, c, d):
return a**2 + b**2 * x + c**2 * abs(x)**d
def residuals( params, xData, yData):
diff = [ fit_function(x, *params ) - y for x, y in zip( xData, yData ) ]
return diff
fit1 = least_squares( residuals, [ .1, .1, .1, .5 ], loss='soft_l1', args=( x1Data, y1Data ) )
print fit1.x
fit2 = least_squares( residuals, [ .1, .1, .1, .5 ], loss='soft_l1', args=( x2Data, y2Data ) )
print fit2.x
testX1 = np.linspace(0, 1.1 * max( x1Data ), 100 )
testX2 = np.linspace(0, 1.1 * max( x2Data ), 100 )
testY1 = [ fit_function( x, *( fit1.x ) ) for x in testX1 ]
testY2 = [ fit_function( x, *( fit2.x ) ) for x in testX2 ]
fig = plt.figure()
ax = fig.add_subplot( 1, 1, 1 )
ax.scatter( x1Data, y1Data )
ax.scatter( x2Data, y2Data )
ax.plot( testX1, testY1 )
ax.plot( testX2, testY2 )
plt.show()
providing
>>[ 1.00232004e-01 -1.10838455e-04 2.50434266e-01 5.73214256e-01]
>>[ 1.00104293e-01 -2.57749592e-05 1.83726191e-01 5.55926678e-01]
and
It just takes the parameters as squares, therefore ensuring positive slope. Naturally, the fit becomes worse if following the decreasing points at the end of data set 1 is forbidden. Concerning this I'd say those are just statistical outliers. Therefore, I used least_squares, which can deal with this with a soft loss. See this doc for details. Depending on how the real data set is, I'd think about removing them. Finally, I'd expect that zero volume produces zero costs, so the constant term in the fit function doesn't seem to make sense.
So if the function is only of type a**2 * x + b**2 * sqrt(x) it look like:
where the green graph is the result of leastsq, i.e. without the f_scale option of least_squares.
I need to develop shell script that will use the below data and do the summation of the attributes attribute_x_count, attribute_y_count and attribute_z_count only. I am new to shell scripting.
Could you please provide steps that required for creating shell script to
obtain summation of these attributes?
{
"data_type": "dummy",
"data": {
"start_time": "2016-05-20",
"attribute_a": [0, 0, 0, 0],
"attribute_b": [0.0, 0.0, 0.0, 0.0],
"attribute_x_count": [0, 0, 1, 1],
"attribute_y_count": [0, 0, 2, 3],
"attribute_d": {
"sub_attribute_p": [0, 0, 0, 0],
"sub_attribute_q": [0, 0, 0, 0],
"sub_attribute_r": [0, 0, 0, 0],
"sub_attribute_s": [0, 0, 0, 0]
},
"attribute_e": [0, 0, 0, 0],
"attribute_f": [0, 0, 0, 0],
"end_time": "2016-05-21",
"attribute_z_count": [0, 0, 2, 2],
"attribute_g": [0, 0, 0, 0],
"attribute_i": [0, 0, 0, 0],
"attribute_j": [0, 0, 0, 0],
"attribute_k": [0, 0, 0, 0],
"attribute_i": [3, 0, 0, 0]
}
}
Expected output:-
Sum (attribute_x/y/z_count) = [0, 0, 1, 1] + [0, 0, 2, 3] + [0, 0, 2,
2]
= 2 + 5 + 4 = 11
The data is in JSON format, which means that we should be using a JSON parser to work with it. No, you should not write your own parser, that would be silly.
I'm using jq, which is a "Command-line JSON processor".
In JSON, if you have multiple attributes with the same name (attribute_c_count is an attribute of data, which occurs three times (if this is unintentional, whatever code generated the JSON has a bug and needs to be fixed)), the value of the last one is what counts.
So we can extract the array that need so be summed up like this:
$ jq ".data.attribute_c_count" <indata.json
[
0,
0,
2,
2
]
This can be summed up by applying the add operation to it:
$ jq ".data.attribute_c_count | add" <indata.json
4
EDIT: The question was modified, and now asks to sum the values of attribute_x_count, attribute_y_count and attribute_z_count.
This will extract the tree arrays:
$ jq ".data.attribute_x_count,.data.attribute_y_count,.data.attribute_z_count" <indata.json
[
0,
0,
1,
1
]
[
0,
0,
2,
3
]
[
0,
0,
2,
2
]
I'm using add to concatenate them into one array, and then add again to sum them:
$ jq "[.data.attribute_x_count,.data.attribute_y_count,.data.attribute_z_count]|add|add" <indata.json
11
(The extra [...] is to get an array for the second add, otherwise I would just get numbers (try it))
If sum of values [0, 0, 2, 3] i.e. 5 is required, then below command can be used, Initially attribute_c_count attribute sum was required.
grep attribute_c_count input.txt|cut -d: -f2|sed -e 's/\[//g' -e 's/\],//g'|tr ',' '\n'|awk '{sum+=$1} END {print sum}
EDIT
After different requirement posted by OP.Edited answer. now attribute_x_count, attribute_y_count and attribute_z_count attributes sum is required.
grep -e attribute_x_count -e attribute_y_count -e attribute_z_count input.txt|cut -d: -f2|sed -e 's/\[//g' -e 's/\],//g'|tr ',' '\n'|awk '{sum+=$1} END {print sum}'
Sorry, it's not quite clear, do you expect to print all the arrays before summing up, or is it the only total value would be sufficient.
In the latter case, using awk, it can be something like
gawk -F '[][,]' '/attribute_[xyz]_count/ { for(i = 2; i < NF; ++i) sum += $i }
END { print sum }'
Simple question but I can't find the answer.
I want to combine a ListLinePlot and a regular Plot (of a function) onto one plot. How do I do this?
Thanks.
Use Show, e.g.
Show[Plot[x^2, {x, 0, 3.5}], ListPlot[{1, 4, 9}]]
Note, if plot options conflict Show uses the first plot's option, unless the option is specified in Show. I.e.
Show[Plot[x^2, {x, 0, 3.5}, ImageSize -> 100],
ListPlot[{1, 4, 9}, ImageSize -> 400]]
shows a combined plot of size 100.
Show[Plot[x^2, {x, 0, 3.5}, ImageSize -> 100],
ListPlot[{1, 4, 9}, ImageSize -> 400], ImageSize -> 300]
Shows a combined plot of size 300.
An alternative to using Show and combining two separate plots, is to use Epilog to add the data points to the main plot. For example:
data = Table[{i, Sin[i] + .1 RandomReal[]}, {i, 0, 10, .5}];
Plot[Sin[x], {x, 0, 10}, Epilog -> Point[data], PlotRange -> All]
or
Plot[Sin[x], {x, 0, 10}, Epilog -> Line[data], PlotRange -> All]