Statsmodels: How can I select different confidence intervals for regressions - regression

I want to run a regression with 99% confidence interval instead of the default 95% using statsmodels.
I looked at the documentation if there is an argument in the fit() method but I didn't notice something. I tried also the conf_int method but I am confused from the output.
import pandas as pd
import math
import statsmodels.formula.api as sm
df = pd.read_excel(r'C:\TestData.xlsx')
df['LogBalance'] = df['Balance'].map(lambda x: math.log(x))
est = sm.ols(formula= 'LogBalance ~ N + Rate',
data=df).fit(cov_type='HAC',cov_kwds={'maxlags':1})
print(est.summary())
print(est.conf_int(alpha=0.01, cols=None))
Since I am new in Python can you tell me if and how can I perform a regression in statsmodels with adjusted confidence intervals if possible in the initial regression output?
Thanks

You can specify the confidence interval in .summary() directly Please consider the following example:
import statsmodels.formula.api as smf
import seaborn as sns
# load a sample dataset
df = sns.load_dataset('tips')
# run model
formula = 'tip ~ size + total_bill'
results = smf.ols(formula=formula, data=df).fit()
# use 95 % CI (default setting)
print(results.summary())
OLS Regression Results
==============================================================================
Dep. Variable: tip R-squared: 0.468
Model: OLS Adj. R-squared: 0.463
Method: Least Squares F-statistic: 105.9
Date: Fri, 21 Jun 2019 Prob (F-statistic): 9.67e-34
Time: 21:42:09 Log-Likelihood: -347.99
No. Observations: 244 AIC: 702.0
Df Residuals: 241 BIC: 712.5
Df Model: 2
Covariance Type: nonrobust
==============================================================================
coef std err t P>|t| [0.025 0.975]
------------------------------------------------------------------------------
Intercept 0.6689 0.194 3.455 0.001 0.288 1.050
size 0.1926 0.085 2.258 0.025 0.025 0.361
total_bill 0.0927 0.009 10.172 0.000 0.075 0.111
==============================================================================
Omnibus: 24.753 Durbin-Watson: 2.100
Prob(Omnibus): 0.000 Jarque-Bera (JB): 46.169
Skew: 0.545 Prob(JB): 9.43e-11
Kurtosis: 4.831 Cond. No. 67.6
==============================================================================
# use 99 % CI
print(results.summary(alpha=0.01))
OLS Regression Results
==============================================================================
Dep. Variable: tip R-squared: 0.468
Model: OLS Adj. R-squared: 0.463
Method: Least Squares F-statistic: 105.9
Date: Fri, 21 Jun 2019 Prob (F-statistic): 9.67e-34
Time: 21:45:57 Log-Likelihood: -347.99
No. Observations: 244 AIC: 702.0
Df Residuals: 241 BIC: 712.5
Df Model: 2
Covariance Type: nonrobust
==============================================================================
coef std err t P>|t| [0.005 0.995]
------------------------------------------------------------------------------
Intercept 0.6689 0.194 3.455 0.001 0.166 1.172
size 0.1926 0.085 2.258 0.025 -0.029 0.414
total_bill 0.0927 0.009 10.172 0.000 0.069 0.116
==============================================================================
Omnibus: 24.753 Durbin-Watson: 2.100
Prob(Omnibus): 0.000 Jarque-Bera (JB): 46.169
Skew: 0.545 Prob(JB): 9.43e-11
Kurtosis: 4.831 Cond. No. 67.6
==============================================================================

Related

Mathematical equation of binomial probit gam (mgcv) with tensor product interactions?

I have the following binomial (probit) gam using mgcv, which includes y (0 or 1), two continuous predictors (xa, xb) plus the ‘ti’ interactions of a third covariate (xc) with these two predictors.
mygam <- gamV(y ~ s(xa, k=10, bs="cr") + s(xb, k=10, bs="cr") +
ti(xc, xa, bs = c("cr", "cr"), k = c(5, 5)) +
ti(xc, xb, bs = c("cr", "cr"), k = c(5, 5)),
data = df, method = "ML", family = binomial(link = "probit"))
Using default k=10 for main effects and k=c(5,5) for interactions, the intercept and 50 coefficients are the following:
terms <- c("Intercept", "s(xa).1", "s(xa).2", "s(xa).3", "s(xa).4", "s(xa).5", "s(xa).6", "s(xa).7", "s(xa).8", "s(xa).9", "s(xb).1", "s(xb).2", "s(xb).3", "s(xb).4", "s(xb).5", "s(xb).6", "s(xb).7", "s(xb).8", "s(xb).9", "ti(xc,xa).1", "ti(xc,xa).2", "ti(xc,xa).3", "ti(xc,xa).4", "ti(xc,xa).5", "ti(xc,xa).6", "ti(xc,xa).7", "ti(xc,xa).8", "ti(xc,xa).9", "ti(xc,xa).10", "ti(xc,xa).11", "ti(xc,xa).12", "ti(xc,xa).13", "ti(xc,xa).14", "ti(xc,xa).15", "ti(xc,xa).16", "ti(xc,xb).1", "ti(xc,xb).2", "ti(xc,xb).3", "ti(xc,xb).4", "ti(xc,xb).5", "ti(xc,xb).6", "ti(xc,xb).7", "ti(xc,xb).8", "ti(xc,xb).9", "ti(xc,xb).10", "ti(xc,xb).11", "ti(xc,xb).12", "ti(xc,xb).13", "ti(xc,xb).14", "ti(xc,xb).15", "ti(xc,xb).16")
coefs <- c(-0.0702421404106311, 0.0768316292916553, 0.210036768213672, 0.409025596435604, 0.516554288252813, 0.314600352165584, -0.271938137725695, -1.1169186662112, -1.44829172827383, -2.39608336269616, 0.445091855160863, 0.119747299507175, -0.73508332280573, -1.3851857008194, -1.84125850675114, -1.77797283303084, -1.45118023146655, -1.56696555281429, -2.55103708393941, 0.0505422263407052, -0.110361707609838, -0.168897589312596, -0.0602318423244818, 0.095385784704545, -0.20818521830706, -0.318650042681766, -0.113613570916751, 0.123559386280642, -0.269467853796075, -0.412476320830133, -0.147039497705579, 0.189416535823022, -0.412990646359733, -0.632158143648671, -0.225344249076957, 0.0237165469278517, 0.0434926950921869, 0.080572361088243, 0.397397459143317, 0.0453636001566695, 0.0831126054198634, 0.153350111096294, 0.75009880522662, 0.0583689328419794, 0.107001374561518, 0.197852239031467, 0.970623037721609, 0.0894562434842868, 0.163989821269297, 0.303175057387294, 1.48718228468607)
df_coefs <- data.frame(terms, coefs)
I would like the mathematical equation of this model, which would allow to determine the probability of y given known covariates. Given as example from my dataset (n > 70000), the predicted probability ‘prob’ (type = “response”) obtained with xa = 7.116, xb = 2.6, and xc = 19 was prob = 0.76444141, which is the result to be determined with the expected mathematical equation.
Is this possible?
Thanks for your help and time.
Below, the summary(mygam)
Parametric coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept) -0.07024 0.00709 -9.907 <2e-16 ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Approximate significance of smooth terms:
edf Ref.df Chi.sq p-value
s(xa) 8.007 8.548 5602.328 < 2e-16 ***
s(xb) 8.448 8.908 16282.793 < 2e-16 ***
ti(xc,xa) 1.004 1.007 10.278 0.00138 **
ti(xc,xb) 1.021 1.042 7.718 0.00627 **
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
R-sq.(adj) = 0.52 Deviance explained = 45.6%
-ML = 29379 Scale est. = 1 n = 77870
If you set type="terms" in the predict function, you get the contributions of the individual components to the linear predictor. However, these are not on the scale of outcome probability, but on that of the linear predictor.
Because of the non-linear transformation of the linear predictor -- in your case with the probit link -- attributing the predicted probability to the individual components requires attribution methods that come with additional assumptions.
An example of such an attribution method is Shapley values.

How to use HuggingFace nlp library's GLUE for CoLA

I've been trying to use the HuggingFace nlp library's GLUE metric to check whether a given sentence is a grammatical English sentence. But I'm getting an error and is stuck without being able to proceed.
What I've tried so far;
reference and prediction are 2 text sentences
!pip install transformers
from transformers import BertTokenizer
tokenizer = BertTokenizer.from_pretrained('bert-large-uncased')
reference="Security has been beefed across the country as a 2 day nation wide curfew came into effect."
prediction="Security has been tightened across the country as a 2-day nationwide curfew came into effect."
import nlp
glue_metric = nlp.load_metric('glue',name="cola")
#Using BertTokenizer
encoded_reference=tokenizer.encode(reference, add_special_tokens=False)
encoded_prediction=tokenizer.encode(prediction, add_special_tokens=False)
glue_score = glue_metric.compute(encoded_prediction, encoded_reference)
Error I'm getting;
ValueError Traceback (most recent call last)
<ipython-input-9-4c3a3ce7b583> in <module>()
----> 1 glue_score = glue_metric.compute(encoded_prediction, encoded_reference)
6 frames
/usr/local/lib/python3.6/dist-packages/nlp/metric.py in compute(self, predictions, references, timeout, **metrics_kwargs)
198 predictions = self.data["predictions"]
199 references = self.data["references"]
--> 200 output = self._compute(predictions=predictions, references=references, **metrics_kwargs)
201 return output
202
/usr/local/lib/python3.6/dist-packages/nlp/metrics/glue/27b1bc63e520833054bd0d7a8d0bc7f6aab84cc9eed1b576e98c806f9466d302/glue.py in _compute(self, predictions, references)
101 return pearson_and_spearman(predictions, references)
102 elif self.config_name in ["mrpc", "qqp"]:
--> 103 return acc_and_f1(predictions, references)
104 elif self.config_name in ["sst2", "mnli", "mnli_mismatched", "mnli_matched", "qnli", "rte", "wnli", "hans"]:
105 return {"accuracy": simple_accuracy(predictions, references)}
/usr/local/lib/python3.6/dist-packages/nlp/metrics/glue/27b1bc63e520833054bd0d7a8d0bc7f6aab84cc9eed1b576e98c806f9466d302/glue.py in acc_and_f1(preds, labels)
60 def acc_and_f1(preds, labels):
61 acc = simple_accuracy(preds, labels)
---> 62 f1 = f1_score(y_true=labels, y_pred=preds)
63 return {
64 "accuracy": acc,
/usr/local/lib/python3.6/dist-packages/sklearn/metrics/_classification.py in f1_score(y_true, y_pred, labels, pos_label, average, sample_weight, zero_division)
1097 pos_label=pos_label, average=average,
1098 sample_weight=sample_weight,
-> 1099 zero_division=zero_division)
1100
1101
/usr/local/lib/python3.6/dist-packages/sklearn/metrics/_classification.py in fbeta_score(y_true, y_pred, beta, labels, pos_label, average, sample_weight, zero_division)
1224 warn_for=('f-score',),
1225 sample_weight=sample_weight,
-> 1226 zero_division=zero_division)
1227 return f
1228
/usr/local/lib/python3.6/dist-packages/sklearn/metrics/_classification.py in precision_recall_fscore_support(y_true, y_pred, beta, labels, pos_label, average, warn_for, sample_weight, zero_division)
1482 raise ValueError("beta should be >=0 in the F-beta score")
1483 labels = _check_set_wise_labels(y_true, y_pred, average, labels,
-> 1484 pos_label)
1485
1486 # Calculate tp_sum, pred_sum, true_sum ###
/usr/local/lib/python3.6/dist-packages/sklearn/metrics/_classification.py in _check_set_wise_labels(y_true, y_pred, average, labels, pos_label)
1314 raise ValueError("Target is %s but average='binary'. Please "
1315 "choose another average setting, one of %r."
-> 1316 % (y_type, average_options))
1317 elif pos_label not in (None, 1):
1318 warnings.warn("Note that pos_label (set to %r) is ignored when "
ValueError: Target is multiclass but average='binary'. Please choose another average setting, one of [None, 'micro', 'macro', 'weighted'].
However, I'm able to get results (pearson and spearmanr) for 'stsb' with the same workaround as given above.
Some help and a workaround for(cola) this is really appreciated. Thank you.
In general, if you are seeing this error with HuggingFace, you are trying to use the f-score as a metric on a text classification problem with more than 2 classes. Pick a different metric, like "accuracy".
For this specific question:
Despite what you entered, it is trying to compute the f-score. From the example notebook, you should set the metric name as:
metric_name = "pearson" if task == "stsb" else "matthews_correlation" if task == "cola" else "accuracy"

How to get the prediction value in LSTM model?

This model is trying to classify the emails whether they are spam or ham.
The training is finished, but how should I input the new email text and get the value that can tell me if the email is spam or ham ?
This is the example code in the text book:
import sklearn
from sklearn.metrics import precision_recall_fscore_support as score
# prediction on test data
predicted_blstm=model.predict(test_data)
predicted_blstm
# model evaluation
from sklearn.metrics import precision_recall_fscore_support as score
precision, recall, fscore, support = score(labels_test, predicted_blstm.round())
print('precision: {}'.format(precision))
print('recall: {}'.format(recall))
print('fscore: {}'.format(fscore))
print('support: {}'.format(support))
print("############################")
print(sklearn.metrics.classification_report(labels_test, predicted_blstm.round()))
and the result:
precision: [0.98782961 0.95348837]
recall: [0.99387755 0.91111111]
fscore: [0.99084435 0.93181818]
support: [980 135]
############################
precision recall f1-score support
0 0.99 0.99 0.99 980
1 0.95 0.91 0.93 135
micro avg 0.98 0.98 0.98 1115
macro avg 0.97 0.95 0.96 1115
weighted avg 0.98 0.98 0.98 1115
samples avg 0.98 0.98 0.98 1115
I had tested the code
mode.predict()
but the result is like:
array([[9.9973804e-01, 2.6198191e-04],
[9.9988401e-01, 1.1600493e-04],
[9.9996233e-01, 3.7628190e-05],
[9.9998081e-01, 1.9162568e-05],
[9.9998498e-01, 1.5043216e-05],
[9.9907982e-01, 9.2014833e-04],
...
[9.9996233e-01, 3.7628190e-05],
[9.9996233e-01, 3.7628190e-05]], dtype=float32)
What does this number means ?
Can I get the answer from this array by showing the message "spam" or "ham" ?
I don't know the features in your train and test data, but if your model is trained only on the email text feature, then you could the following:
1) Convert the email text, for example "This is the email I want to test", into a vector using the vectorizer used on the training data.
2) If your vector is stored in a variable 'vec'. Then you can predict if the email is ham or spam using
prediction = model.predict(vec)
The varaible 'prediction' will hold your answer.

Running Python Code on Knime's Python Script Node

I am fairly new to Knime and Python and I was checking some codes in Python that I would like to run on Python Script Node through Knime. The script executes fine through the node but not outside it, plus, the data does not show correctly on output panel for "Poisson Model" attribute on following script:
import pandas as pd
import matplotlib.pyplot as plt
import numpy as np
import seaborn
from scipy.stats import poisson,skellam
import statsmodels.api as sm
import statsmodels.formula.api as smf
epl_1617 = input_table
poisson_model = smf.glm(formula="input_table['GOALS'] ~
input_table['HOME'] + input_table['HomeTeam'] +
input_table['AwayTeam']", data=epl_1617,
family=sm.families.Poisson()).fit()
poisson_model.summary()
When I run the above code on Python it gives the result in form of a table; however, in Knime the output is shown as "poisson_model:
"
Note: the input file can be downloaded from:
http://www.football-data.co.uk/mmz4281/1617/E0.csv
Any idea on how to run the following code on Knime?
I have tried assigning the result to a DataFrame using the following:
output_table = pd.DataFrame(poisson_model.summary())
However, I have been getting the following error:
output_table = pd.DataFrame(poisson_model.summary())
Traceback (most recent call last):
File "C:\Program Files\KNIME\plugins\org.knime.python2_3.6.0.v201807061638\py\PythonKernelBase.py", line 278, in execute
exec(source_code, self._exec_env, self._exec_env)
File "<string>", line 24, in <module>
File "C:\Python27\lib\site-packages\pandas\core\frame.py", line 422,
in __init__
raise ValueError('DataFrame constructor not properly called!')
ValueError: DataFrame constructor not properly called!
and here is the result if I run the same code using Python IDE
poisson_model.summary()
Out[3]:
<class 'statsmodels.iolib.summary.Summary'>
"""
Generalized Linear Model Regression Results
==============================================================================
Dep. Variable: GOALS No. Observations: 740
Model: GLM Df Residuals: 700
Model Family: Poisson Df Model: 39
Link Function: log Scale: 1.0000
Method: IRLS Log-Likelihood: -1107.5
Date: Wed, 27 Mar 2019 Deviance: 906.20
Time: 19:29:22 Pearson chi2: 784.
No. Iterations: 5 Covariance Type: nonrobust
==============================================================================================
coef std err z P>|z| [0.025 0.975]
----------------------------------------------------------------------------------------------
Intercept 0.4226 0.193 2.190 0.029 0.044 0.801
HomeTeam[T.Bournemouth] 0.1726 0.188 0.917 0.359 -0.196 0.542
HomeTeam[T.Burnley] -0.1755 0.208 -0.845 0.398 -0.582 0.232
HomeTeam[T.Chelsea] 0.2356 0.187 1.259 0.208 -0.131 0.602
HomeTeam[T.Crystal Palace] -0.0903 0.201 -0.450 0.652 -0.483 0.303
HomeTeam[T.Everton] 0.0608 0.192 0.317 0.752 -0.316 0.437
HomeTeam[T.Hull] 0.0676 0.195 0.347 0.729 -0.315 0.450
HomeTeam[T.Leicester] 0.0599 0.196 0.306 0.760 -0.324 0.444
HomeTeam[T.Liverpool] 0.1453 0.191 0.760 0.447 -0.229 0.520
HomeTeam[T.Man City] 0.0060 0.196 0.031 0.975 -0.378 0.390
HomeTeam[T.Man United] -0.3528 0.218 -1.616 0.106 -0.781 0.075
HomeTeam[T.Middlesbrough] -0.3185 0.212 -1.505 0.132 -0.733 0.096
HomeTeam[T.Southampton] -0.3345 0.217 -1.545 0.122 -0.759 0.090
HomeTeam[T.Stoke] -0.1255 0.202 -0.623 0.533 -0.521 0.269
HomeTeam[T.Sunderland] -0.0894 0.199 -0.448 0.654 -0.480 0.302
HomeTeam[T.Swansea] 0.1140 0.193 0.592 0.554 -0.264 0.491
HomeTeam[T.Tottenham] 0.0301 0.194 0.155 0.877 -0.350 0.410
HomeTeam[T.Watford] -0.0352 0.201 -0.175 0.861 -0.429 0.358
HomeTeam[T.West Brom] -0.1100 0.200 -0.549 0.583 -0.503 0.283
HomeTeam[T.West Ham] -0.0738 0.200 -0.370 0.712 -0.465 0.317
AwayTeam[T.Bournemouth] -0.0941 0.182 -0.517 0.605 -0.451 0.263
AwayTeam[T.Burnley] -0.3271 0.190 -1.722 0.085 -0.699 0.045
AwayTeam[T.Chelsea] -0.3465 0.192 -1.801 0.072 -0.724 0.031
AwayTeam[T.Crystal Palace] -0.0285 0.177 -0.161 0.872 -0.376 0.319
AwayTeam[T.Everton] -0.3461 0.195 -1.777 0.076 -0.728 0.036
AwayTeam[T.Hull] -0.1969 0.184 -1.071 0.284 -0.557 0.163
AwayTeam[T.Leicester] -0.1790 0.183 -0.978 0.328 -0.537 0.180
AwayTeam[T.Liverpool] -0.1381 0.181 -0.762 0.446 -0.493 0.217
AwayTeam[T.Man City] -0.0412 0.179 -0.230 0.818 -0.392 0.309
AwayTeam[T.Man United] -0.3990 0.194 -2.061 0.039 -0.778 -0.020
AwayTeam[T.Middlesbrough] -0.5297 0.206 -2.574 0.010 -0.933 -0.126
AwayTeam[T.Southampton] -0.2731 0.187 -1.463 0.143 -0.639 0.093
AwayTeam[T.Stoke] -0.2855 0.190 -1.502 0.133 -0.658 0.087
AwayTeam[T.Sunderland] -0.3859 0.198 -1.950 0.051 -0.774 0.002
AwayTeam[T.Swansea] -0.1941 0.184 -1.056 0.291 -0.554 0.166
AwayTeam[T.Tottenham] -0.2569 0.190 -1.351 0.177 -0.630 0.116
AwayTeam[T.Watford] -0.2025 0.184 -1.102 0.270 -0.563 0.158
AwayTeam[T.West Brom] -0.3953 0.198 -1.998 0.046 -0.783 -0.008
AwayTeam[T.West Ham] -0.0867 0.180 -0.480 0.631 -0.440 0.267
HOME 0.2962 0.063 4.695 0.000 0.173 0.420
==============================================================================================
"""
Your help is already appreciated!

Different Sum Sq and MSS using lme4::lmer and lmerTest::lmer

I get sums of squares and mean sums of squares 10x higher when I use anova on lmerTest:: lmer compared to lme4:: lmer objects. See the R log file below. Note the warning that when I attach the lmerTest package, the stats::sigma function overrides the lme4::sigma function, and I suspect that it is this that leads to the discrepancy. In addition, the anova report now says that it is a Type III anova rather than the expected Type I. Is this a bug in the lmerTest package, or is there something about use of the Kenward-Roger approximation for degrees of freedom that changes the calculation of SumSQ and MSS and specification of the anova Type that I don't understand?
I would append the test file, but it is confidential clinical trial information. If necessary I can see if I can cobble up a test case.
Thanks in advance for any advice you all can provide about this.
> library(lme4)
Loading required package: Matrix
Attaching package: ‘lme4’
The following object is masked from ‘package:stats’:
sigma
> test100 <- lmer(log(value) ~ prepost * lowhi + (1|CID/LotNo/rep),
REML = F, data = GSIRlong, subset = !is.na(value))
> library(lmerTest)
Attaching package: ‘lmerTest’
The following object is masked from ‘package:lme4’:
lmer
The following object is masked from ‘package:stats’:
step
Warning message:
replacing previous import ‘lme4::sigma’ by ‘stats::sigma’ when loading
‘lmerTest’
> test200 <- lmer(log(value) ~ prepost * lowhi + (1|CID/LotNo/rep),
REML = F, data = GSIRlong, subset = !is.na(value))
> anova(test100)
Analysis of Variance Table
Df Sum Sq Mean Sq F value
prepost 1 3.956 3.956 18.4825
lowhi 1 130.647 130.647 610.3836
prepost:lowhi 1 0.038 0.038 0.1758
> anova(test200, ddf = 'Ken')
Analysis of Variance Table of type III with Kenward-Roger
approximation for degrees of freedom
Sum Sq Mean Sq NumDF DenDF F.value Pr(>F)
prepost 37.15 37.15 1 308.04 18.68 2.094e-05 ***
lowhi 1207.97 1207.97 1 376.43 607.33 < 2.2e-16 ***
prepost:lowhi 0.35 0.35 1 376.43 0.17 0.676
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Update: Thanks, Ben. I did a little code archeology on lmerTest to see if I could dope out an explanation for the above anomalies. First, it turns out that lmerTest::lmer just submits the model to lme4::lmer and then relabels the result as an "mermodLmerTest" object. The only effect of this is to invoke versions of summary() and anova() from the lmerTest package rather than the usual defaults from base and stats. (These lmerTest functions are compiled, and I have not yet gone farther to look at the C++ code.) lmerTest::summary just adds three columns to the base::summary result, giving df, t value, and Pr. Note that lmerTest::anova, by default, computes a type III anova rather than a type I as in stats::anova. (Explanation of my second question above.) Not a great choice if one's model includes interactions. One can request a type I, II, or III anova using the type = 1/2/3 option.
However there are other surprises using the nlmeTest versions of summary and anova, as shown in the R console file below. I used lmerTest's included sleepstudy data so that this code should be replicable.
a. Note that "sleepstudy" has 180 records (with 3 variables)
b. The summaries of fm1 and fm1a are identical except for the added Fixed effects columns. But note that in the lmerTest::summary the ddfs for the intercept and Days are 1371 and 1281 respectively; odd given that there are only 180 records in "sleepstudy."
c. Just as in my original model above, the nlm4 anad nlmrTest versions of anova give very different values of Sum Sq and Mean Sq. (30031 and 446.65 respectively).
d: Interestingly, the nlmrTest versions of anova using Satterthwaite and Kenward-Rogers estimates of the DenDF are wildly different (5794080 and 28 respecitvely). The K-R value seems more reasonable.
Given the above issues, I am reluctant at this point to depend on lmerTest to give reliable p-values. Based on your (Doug Bates's) blog entry (https://stat.ethz.ch/pipermail/r-help/2006-May/094765.html), I am using now (and recommending) the method from the posting by Dan Mirman (http://mindingthebrain.blogspot.ch/2014/02/three-ways-to-get-parameter-specific-p.html) in the final bit of code below to estimate a naive t-test p-value (assuming essentially infinite degrees of freedom) and a Kenward-Rogers estimate of df (using the R package 'pbkrtest' -- the same package used by lmerTest). I couldn't find R code to compute the Satterthwaite estimates. The naive t-test p-value is clearly anti-conservative, but the KR estimate is thought to be pretty good. If the two give similar estimates of p, then I think that one can feel comfortable with a p-value in the range of [naive t-test, KR estimate].
> library(lme4); library(lmerTest); library(pbkrtest);
dim(sleepstudy)
[1] 180 3
>
> fm1 <- lme4::lmer(Reaction ~ Days + (Days|Subject), sleepstudy)
> fm1a <- lmerTest::lmer(Reaction ~ Days + (Days|Subject), sleepstudy)
>
> summary(fm1)
Linear mixed model fit by REML ['lmerMod']
Formula: Reaction ~ Days + (Days | Subject)
Data: sleepstudy
REML criterion at convergence: 1743.6
Scaled residuals:
Min 1Q Median 3Q Max
-3.9536 -0.4634 0.0231 0.4634 5.1793
Random effects:
Groups Name Variance Std.Dev. Corr
Subject (Intercept) 612.09 24.740
Days 35.07 5.922 0.07
Residual 654.94 25.592
Number of obs: 180, groups: Subject, 18
Fixed effects:
Estimate Std. Error t value
(Intercept) 251.405 6.825 36.84
Days 10.467 1.546 6.77
Correlation of Fixed Effects:
(Intr)
Days -0.138
> summary(fm1a)
Linear mixed model fit by REML t-tests use Satterthwaite approximations to
degrees of freedom [lmerMod]
Formula: Reaction ~ Days + (Days | Subject)
Data: sleepstudy
REML criterion at convergence: 1743.6
Scaled residuals:
Min 1Q Median 3Q Max
-3.9536 -0.4634 0.0231 0.4634 5.1793
Random effects:
Groups Name Variance Std.Dev. Corr
Subject (Intercept) 612.09 24.740
Days 35.07 5.922 0.07
Residual 654.94 25.592
Number of obs: 180, groups: Subject, 18
Fixed effects:
Estimate Std. Error df t value Pr(>|t|)
(Intercept) 251.405 6.825 1371.100 302.06 <2e-16 ***
Days 10.467 1.546 1281.700 55.52 <2e-16 ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Correlation of Fixed Effects:
(Intr)
Days -0.138
Warning message:
In deviance.merMod(object, ...) :
deviance() is deprecated for REML fits; use REMLcrit for the REML
criterion or deviance(.,REML=FALSE) for deviance calculated at the REML fit
>
> anova(fm1)
Analysis of Variance Table
Df Sum Sq Mean Sq F value
Days 1 30031 30031 45.853
> anova(fm1a, ddf = 'Sat', type = 1)
Analysis of Variance Table of type I with Satterthwaite
approximation for degrees of freedom
Sum Sq Mean Sq NumDF DenDF F.value Pr(>F)
Days 446.65 446.65 1 5794080 45.853 1.275e-11 ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Warning message:
In deviance.merMod(object, ...) :
deviance() is deprecated for REML fits; use REMLcrit for the REML criterion or deviance(.,REML=FALSE) for deviance calculated at the REML fit
> anova(fm1a, ddf = 'Ken', type = 1)
Analysis of Variance Table of type I with Kenward-Roger
approximation for degrees of freedom
Sum Sq Mean Sq NumDF DenDF F.value Pr(>F)
Days 446.65 446.65 1 27.997 45.853 2.359e-07 ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Warning message:
In deviance.merMod(object, ...) :
deviance() is deprecated for REML fits; use REMLcrit for the REML criterion or deviance(.,REML=FALSE) for deviance calculated at the REML fit
>
> # t.test
> coefs <- data.frame(coef(summary(fm1)))
> coefs$p.z <- 2 * (1 - pnorm(abs(coefs$t.value)))
> coefs
Estimate Std..Error t.value p.z
(Intercept) 251.40510 6.824556 36.838311 0.000000e+00
Days 10.46729 1.545789 6.771485 1.274669e-11
>
> # Kenward-Rogers
> df.KR <- get_ddf_Lb(fm1, fixef(fm1))
> df.KR
[1] 25.89366
> coefs$p.KR <- 2 * (1 - pt(abs(coefs$t.value), df.KR))
> coefs
Estimate Std..Error t.value p.z p.KR
(Intercept) 251.40510 6.824556 36.838311 0.000000e+00 0.0000e+00
Days 10.46729 1.545789 6.771485 1.274669e-11 3.5447e-07