How to set a fixed random seed for the inner loop of nested hyper-parameter tuning? - mlr

Question

I'm trying to do hyper-parameter tuning with nested crossvalidation. This is my inner loops for two learners lrn1 and lrn2:
inner = makeResampleDesc("CV", iters = 3L)
tune_lrn1 <- makeTuneWrapper(lrn1, resampling = inner, par.set = ps, control = ctrl)
tune_lrn2 <- makeTuneWrapper(lrn2, resampling = inner, par.set = ps, control = ctrl)
Is there any way to set a fixed value of the random seed everytime before "inner" is instantiated, so that the two learners always use exactly the same data partitions for hyper-parameter evaluation?

Answer 1

There are two things that you can do but they might not satisfiy your needs completely.
Fix the resampling for a given task
...or at least a given n. In this example we use the task iris.task.
inner_fixed = makeResampleInstance(inner, iris.task)
tune_lrn1 <- makeTuneWrapper(lrn1, resampling = inner_fixed, par.set = ps, control = ctrl)
tune_lrn2 <- makeTuneWrapper(lrn2, resampling = inner_fixed, par.set = ps, control = ctrl)
If you want to apply this for multiple tasks you have to solve it programmatically.
Setting the seed can fail!
The following setting is already the default
ctrl = makeTuneControl*(same.resampling.instance = TRUE, ...)
This means, that all tuning evaluations are evaluated on the same train/test split. In other words: makeResampleInstance is called at the very beginning of tune(). Now we can go with #pat-s answer, which does not always work because the RNG is used during train for some learners and accordingly the ongoing train/test-splits will "diverge":
library(mlr)
inner = makeResampleDesc("CV", iters = 3L)
task = iris.task
lrn1 = makeLearner("classif.rpart")
lrn2 = makeLearner("classif.svm")
ctrl = makeTuneControlRandom(same.resampling.instance = TRUE, budget = 4)
library(mlrHyperopt)
ps1 = getDefaultParConfig(lrn1)$par.set
ps2 = getDefaultParConfig(lrn2)$par.set
tune_lrn1 = makeTuneWrapper(lrn1, resampling = inner, par.set = ps1, control = ctrl)
tune_lrn2 = makeTuneWrapper(lrn2, resampling = inner, par.set = ps2, control = ctrl)
set.seed(1)
r1 = resample(tune_lrn1, resampling = cv10, task = iris.task, models = TRUE)
set.seed(1)
r2 = resample(tune_lrn2, resampling = cv10, task = iris.task, models = TRUE)
sapply(1:10, function(i) {
identical(r2$models[[i]]$learner.model$opt.result$resampling$train.inds, r1$models[[i]]$learner.model$opt.result$resampling$train.inds)
})
# [1] TRUE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE FALSE