When modifying some of the attributes of a duplicated SpatRaster, the original is also modified:
library(terra)
r <- rast(ncol=2, nrow=2, vals=c(5.3, 7.1, 3, 1.2))
#class : SpatRaster
#dimensions : 2, 2, 1 (nrow, ncol, nlyr)
#resolution : 180, 90 (x, y)
#extent : -180, 180, -90, 90 (xmin, xmax, ymin, ymax)
#coord. ref. : +proj=longlat +datum=WGS84 +no_defs
#source : memory
#name : lyr.1
#min value : 1.2
#max value : 7.1
xmin(r)
#[1] -180
t <- r # duplication
xmin(t) <- -300 # xmin modification of the duplicated SpatRaster
xmin(r) # the original SpatRaster has also been modified
#[1] -300
Is it an error or a choice? It only occurs for some attributes, not all. If it is a choice, what is the way to create an 'independant" copy, or how to break the link?
This happened because a SpatRaster is just a wrapper around a C++ object. That makes x (below) a shallow copy (i.e. pointing to the same object in memory)
library(terra)
r <- rast()
x <- r
It only matters in some cases, when using replacement methods (your example is no longer affected with current terra). I have also added a copy method that returns a deep copy, that is, a SpatRaster pointing to a different (deep copied) C++ object.
For information, function add has the same problem (in terra 1.0.11):
logo <- rast(system.file("ex/logo.tif", package="terra"))
nlyr(logo)
[1] 3
r <- logo
add(r) <- r[[1]]
nlyr(logo)
#[1] 4
There may be a misunderstanding here, especially from me. The fact that add(r) <- r[[1]] changes r is absolutely normal, as you point out. But the fact that it also changes logo is not at all routine (the add function is supposed to add a layer to r, not to any other object unrelated to the current script line, according to the documentation).
If I understood correctly, and as you explained before, it is because r <- logo doesn't duplicate logo (deep copy) but only creates a pointer (shallow copy) to logo. This choice has important consequences on the use of terra objects since later modifications of r will also modify the logo object (a "side effect", from the user's point of view). I see at least 3 points here:
Unusual behaviour. Perhaps the user should be warned.
"In R semantics, objects are copied by value. This means that modifying the copy leaves the original object intact.": deep or lazy copies are a basic rule of R (https://rlang.r-lib.org/reference/duplicate.html). The user must therefore be well aware that simple assignments in terra (r <- logo) do not copy by value, but by reference, otherwise the risk of error in the scripts is high. However, this unusual behavior is currently not explained in the documentation of terra.
Shallow copies are more a programmer-oriented tool.
When users will undersand that terra copies are shallow copies, they will probably most often prefer to make deep copies, because the use of shallow copies for terra users is probably restricted and uncommon. I don't see too many cases where this behaviour would be useful. Deep or lazy copies are much more common for users.
Heterogeneous behaviour according to functions
But that's not the main question. The main problem in the fact that, for the moment, some operations affect both the original object and the copied object, and others don't. add modifies the original object (side effect), but res doesn't change it :
logo <- rast(system.file("ex/logo.tif", package="terra"))
nlyr(logo)
# [1] 3
res(logo)
# [1] 1 1
r <- logo
add(r) <- r[[1]]
nlyr(logo)
# side effect of add function on the number of layers
# [1] 4
res(r) <- c(10,10)
res(logo)
# no side effect of res function on the resolution
# [1] 1 1
If all these conjectures are right, it would be useful to explain which functions, after an assignment, do or do not affect the original object, otherwise one cannot program reliably.
Related
I use torch.nn.Embedding to embed my model’s categorical input features, however, I face problems when I set the max_norm parameter to not None.
There is a note on the pytorch docs page that explains how to use max_norm parameter through the following example:
n, d, m = 3, 5, 7
embedding = nn.Embedding(n, d, max_norm=True)
W = torch.randn((m, d), requires_grad=True)
idx = torch.tensor(\[1, 2\])
a = embedding.weight.clone() # W.t() # weight must be cloned for this to be differentiable
b = embedding(idx) # W.t() # modifies weight in-place
out = (a.unsqueeze(0) + b.unsqueeze(1))
loss = out.sigmoid().prod()
loss.backward()
I can’t easily understand this example from the docs. What is the purpose of having both ‘a’ and ‘b’ and why ‘out’ is defined as, out = (a.unsqueeze(0) + b.unsqueeze(1))?
Do we need to first clone the entire embedding tensor as in ‘a’, and then finding the embeddings for our desired indices as in ‘b’? Then how do ‘a’ and ‘b’ need to be added?
In my code, I don’t have W explicitly, I am assuming that W is representative of the weights applied by the torch.nn.Linear layers. So, I just need to prepare the input (which includes the embeddings for categorical features) that goes into my network.
I greatly appreciate any instructions on this, as understanding this example would help me adapt my code accordingly.
Because W in the line computing a requires gradients, we must save embedding.weight to compute those gradients in the backward pass. However, in the line computing b, executing embedding(idx) will scale embedding.weight by max_norm - in place. So, without cloning it in line a, embedding.weight will be modified when line b is executed - changing what was saved for the backward pass to update W. Hence the requirement to clone embedding.weight - to save it before it gets scaled in line b.
If you don't use embedding.weight outside of the normal forward pass, you don't need to worry about all this.
If you get an error, post it (and your code).
Referring to the original paper on CycleGAN i am confused about this line
The optimal G thereby translates the domain X to a domain Yˆ
distributed identically to Y . However, such a translation does not
guarantee that an individual input x and output y are paired up in a
meaningful way – there are infinitely many mappings G that will induce
the same distribution over yˆ.
I understand there are two sets of images and there is no pairing between them so when generator will taken one image lets say x from set X as input and try to translate it to an image similar to the images in Y set then my question is that there are many images present in the set Y so which y will our x be translated into? There are so many options available in set Y. Is that what is pointed out in these lines of the paper that i have written above? And is this the reason we take cyclic loss to overcome this problem and to create some type of pairing between any two random images by converting x to y and then converting y back to x?
The image x won't be translated to a concrete image y but rather to a "style" of the domain Y. The input is fed to the generator, which tries to produce a sample from the desired distribution (the other domain), the generated image then goes to the discriminator, which tries to predict if the sample is from the actual distribution or produced by the generator. This is just the normal GAN workflow.
If I understand it correctly, in the lines you quoted, authors explain the problems that arise with adversarial loss. They say it again here:
Adversarial training can, in theory, learn mappings G and F that produce outputs identically distributed as target domains Y and X respectively. However, with large enough capacity, a network can map the same set of input images to any random permutation of images in the target domain, where any of the learned mappings can induce an output distribution that matches the target distribution. Thus, an adversarial loss alone cannot guarantee that the learned function can map an individual input x_i to a desired output y_i.
This is one of the reasons for introducing the concept of cycle-consistency to produce meaningful mappings, reduce the space of possible mapping functions (can be viewed as a form of regularization). The idea is not to create a pairing between 2 random images which already are in the dataset (the dataset stays unpaired), but to make sure, that if you map a real image from the domain X to the domain Y and then back again, you get the original image back.
Cycle consistency encourages generators to avoid unnecessary changes and thus to generate images that share structural similarity with inputs, it also prevents generators from excessive hallucinations and mode collapse.
I hope that answers your questions.
I am a beginner python user who is trying to get a feel for computer science, I've been learning how to use it by studying concepts/subjects I'm already familiar with, such as Computation Fluid Mechanics & Finite Element Analysis. I got my degree in mechanical engineering, so not much CS background.
I'm studying a series by Lorena Barba on jupyter notebook viewer, Practical Numerical Methods, and i'm looking for some help, hopefully someone familiar with the subjects of CFD & FEA in general.
if you click on the link below and go to the following output line, you'll find what i have below. Really confused on this block of code operated within the function that is defined.
Anyway. If there is anyone out there, with any suggestions on how to tackle learning python, HELP
In[9]
rho_hist = [rho0.copy()]
rho = rho0.copy() **# im confused by the role of this variable here**
for n in range(nt):
# Compute the flux.
F = flux(rho, *args)
# Advance in time using Lax-Friedrichs scheme.
rho[1:-1] = (0.5 * (rho[:-2] + rho[2:]) -
dt / (2.0 * dx) * (F[2:] - F[:-2]))
# Set the value at the first location.
rho[0] = bc_values[0]
# Set the value at the last location.
rho[-1] = bc_values[1]
# Record the time-step solution.
rho_hist.append(rho.copy())
return rho_hist
http://nbviewer.jupyter.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/03_wave/03_02_convectionSchemes.ipynb
The intent of the first two lines is to preserve rho0 and provide copies of it for the history (copy so that later changes in rho0 do not reflect back here) and as the initial value for the "working" variable rho that is used and modified during the computation.
The background is that python list and array variables are always references to the object in question. By assigning the variable you produce a copy of the reference, the address of the object, but not the object itself. Both variables refer to the same memory area. Thus not using .copy() will change rho0.
a = [1,2,3]
b = a
b[2] = 5
print a
#>>> [1, 2, 5]
Composite objects that themselves contain structured data objects will need a deepcopy to copy the data on all levels.
Numpy array values changed without being aksed?
how to pass a list as value and not as reference?
I want to generate small-world networks using igraph, but not using "rewiring" as implemented in the watts.strogatz.game(). In the Newman variation all local links are fixed but a fixed number of random links are lifted and dropped randomly on the network at a fixed rate (basically adding "long-range" connections). I thought I could simply generate a lattice (e.g. g <- graph.lattice(length=20, dim=1, circular=TRUE)) and then put a classical random graph on top of that. However, I do not know how to do this using a graph as input argument. Or maybe it is possible to add random edges at a specified probability?
Any help highly appreciated.
Thanks a lot!
Use graph.lattice to generate a lattice, then erdos.renyi.game with the same number of vertices and a fixed probability to generate a random graph. Then you can combine the two graphs using the %u% (union) operator. There is a small chance for multi-edges if the same edge happens to be the part of the lattice and the random graph as well, so you should also call simplify() on the union if you don't want that.
This seems to do the trick, in case anyone is interested. Just have to create a function to do this "rewiring" over and over again. Many thanks again, Tamas!
library(igraph)
g <- graph.lattice(length=100, dim=1, circular=TRUE)
g2 <- erdos.renyi.game(100, 1/100)
g3 <- g %u% g2
g3 <- simplify(g3)
plot.igraph(g3, vertex.size = 1,vertex.label = NA, layout=layout_in_circle)
I'm having a problem generating simulations from a 3 level glmer model when conditioning on the random effects (I'm actually using predict via bootMer but the problem is the same).
This works:
library(lme4)
fit1 = glmer(cbind(incidence, size - incidence) ~ period + (1 | herd),
data = cbpp, family = binomial)
simulate(fit1, re.form=NULL)
This fails:
cbpp$bigherd = rep(1:7, 8)
fit2 = glmer(cbind(incidence, size - incidence) ~ period + (1 | bigherd / herd),
data = cbpp, family = binomial)
simulate(fit2, re.form=NULL)
Error: No random effects terms specified in formula
Many thanks for any ideas.
Update
Ben, many thanks for your help below, really appreciate it. I wonder if I can impose on you again.
What I want to do is simulate predictions on the response scale and I'm not sure if I can use your work around? Or if there is an alternative to what I'm doing. Thank you!
This works as expected, but is not conditional on random effects:
FUN = function(.){
predict(., type="response")
}
bootMer(fit2, FUN, nsim=3)$t
This doesn't work, as would be expected given above problem:
bootMer(fit2, FUN, nsim=3, use.u=TRUE)$t
As far as I can see, I can't pass re.form to bootMer.
Does the alternative below result in simulated predictions conditional on random effects without passing use.u to bootMer?
FUN = function(.){
predict(., type="response", re.form=~(1|herd:bigherd) + (1|bigherd))
}
bootMer(fit2, FUN, nsim=10)$t
I'm not sure what's going on yet, but here are two workarounds that do work:
simulate(fit2, re.form=lme4:::reOnly(formula(fit2)))
simulate(fit2, re.form=~(1|herd:bigherd) + (1|bigherd))
There must be something going wrong with the expansion of the "slash" term, because this doesn't work:
simulate(fit2, re.form=~(1|bigherd/herd))
I've posted this as an lme4 issue
These workarounds don't work for bootMer (which only takes the use.u argument, not re.form) in the current CRAN release (1.1-9).
It is fixed in the development version on Github (1.1-10): devtools::install_github("lme4/lme4") will install it, if you have compilation tools installed.
In the meantime you could just go ahead and implement your own parametric bootstrap (for parametric bootstrapping, bootMer is actually a very thin wrapper around simulate()/[refit()orupdate()]/FUN`). Much of the complication has to do with parallel computation (you'd have to add some of it back in if you want parallel computation in your own PB implementation).
This is the outline of a hand-rolled parametric bootstrap:
nboot <- 10
nresp <- length(FUN(orig_fit))
res <- matrix(NA,nboot,nresp)
for (i in 1:nboot) {
res[i,] <- FUN(update(orig_fit,data=simulate(orig_fit,...)))
## or use refit() for LMMs
## ... are options to simulate()
}
t(apply(res,2,quantile,c(0.025,0.975)))