LibGDX - overlay texture above another texture using shader - libgdx

Question

I'm trying to mix two different textures(scene and clouds) which are obtained from FBO and draw them on quad.
uniform sampler2D u_texture;
uniform sampler2D u_texture2;
uniform vec2 u_res;
void main(void)
{
vec2 texCoord = gl_FragCoord.xy / u_res.xy;
vec4 sceneColor = texture2D(u_texture, texCoord);
vec4 addColor = texture2D(u_texture2, texCoord);
gl_FragColor = sceneColor+addColor;
}
glBlendFunc is
Gdx.gl20.glBlendFunc(GL20.GL_SRC_ALPHA, GL20.GL_ONE_MINUS_SRC_ALPHA);
I tried all combination of glBlendFunc and the combination above was the best one.
Creating FBOs:
fbClouds = new FrameBuffer(Format.RGBA8888, Gdx.graphics.getWidth(), Gdx.graphics.getHeight(), true);
fbScene = new FrameBuffer(Format.RGB565, Gdx.graphics.getWidth(), Gdx.graphics.getHeight(), true);
fbMix = new FrameBuffer(Format.RGB565, Gdx.graphics.getWidth(), Gdx.graphics.getHeight(), true);
creating clouds:
fbClouds.begin();
Gdx.gl.glClearColor(0, 0, 0, 0); // to make it transparent
Gdx.gl.glClear(GL20.GL_COLOR_BUFFER_BIT | GL20.GL_DEPTH_BUFFER_BIT);
modelBatch.begin(cam);
for (Integer e : drawOrder) {
if(isVisible(cam, clouds[e])){
drawLightning(e, modelBatch);
modelBatch.render(clouds[e], cloudShader);
modelBatch.flush();
}
}
modelBatch.end();
fbClouds.end();
render code:
Gdx.gl20.glDisable(GL20.GL_BLEND);
//Gdx.gl20.glEnable(GL20.GL_BLEND);
//Gdx.gl20.glBlendFunc(GL20.GL_SRC_ALPHA, GL20.GL_ONE_MINUS_SRC_ALPHA);
fbMix.begin();
Gdx.gl.glClear(GL20.GL_COLOR_BUFFER_BIT | GL20.GL_DEPTH_BUFFER_BIT);
mixShader.begin();
fbScene.getColorBufferTexture().bind(1);
mixShader.setUniformi("u_texture", 1);
fbClouds.getColorBufferTexture().bind(0);
mixShader.setUniformi("u_texture2", 0);
mixShader.setUniformf("u_res", Gdx.graphics.getWidth(), Gdx.graphics.getHeight());
quad.render(mixShader, GL20.GL_TRIANGLES);
mixShader.end();
fbMix.end();
So, I get unexpected result(clouds have absolutely white color, though they should be grey):
In case if I use modelbatch to render clouds the result is as should be:
What is the right way to mix two textures without losing color?

Answer 1

The blend function you use to draw the two FBO's to the screen should be irrelevant, because nothing shows through them, right? So blending should be turned off before you draw the FBO's, or you're wasting GPU cycles mixing the FBO's with your clear color.
The reason it turns white is that you are adding gray to blue without darkening the blue first. Normally when you draw a transparent object to the screen, you use a blend function like this: GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA. That means you are multiplying the sprite's color by its transparency (effectively darkening transparent colors) and you multiply the background by the inverse of the sprite's alpha (thereby darkening the pixels that will be added to the sprite's opaque pixels so they won't be too bright).
In your case, you want to emulate the same thing inside your fragment shader, since you are trying to blend two textures inside your shader before outputing them to the screen.
So if your cloud FBO had an alpha channel, you could do this in your fragment shader and you'd be good to go:
void main()
{
vec2 texCoord = gl_FragCoord.xy / u_res.xy;
vec4 sceneColor = texture2D(u_texture, texCoord);
vec4 addColor = texture2D(u_texture2, texCoord);
gl_FragColor = addColor*addColor.a + sceneColor*(1-addColor.a);
}
However, your cloud's FBO does not have an alpha channel so you need to change something.
One thing you could do is make your FBO color texture use RGBA4444 so it has an alpha channel, and then carefully draw your clouds so they also write to the alpha channel. This would be kind of complicated, because you'd have to use a separated blend function, where you select two different blend functions for the RGB and the A channels separately. I haven't done this before. Although it should be possible, I haven't even tried this method before because I think the 4-bit colors would look pretty lousy.
Alternatively, if your clouds are all going to be monochrome, you can encode your alpha information into one of the color channels. To do this you will need to customize the fragment shader you use to draw the clouds to the FBO. It would look something like this:
vec4 textureColor = texture2D(u_texture, v_texCoord);
gl_FragColor = vec4(textureColor.r * textureColor.a, textureColor.a, 0, textureColor.a);
What this does is put the cloud's monochrome color in the R channel with alpha pre-multiplied, and it puts the alpha in the G channel. We want to pre-multiply the alpha so we can simply add the encoded cloud sprite onto the scene. This is because when you draw something in front of an already-drawn sprite in an area that was translucent in the already-drawn sprite, you want to brighten the G-encoded alpha to make the pixel more opaque in the final FBO image. Since we are using pre-multiplied alpha, draw the clouds using the blend function GL_ONE, GL_ONE_MINUS_SRC_ALPHA.
(This is a slight approximation because the G-encoded alpha of the destination is getting darkened a bit by the second part of the blend function, but I looked at the math and it seems acceptable. The approximation results in slightly more transparent clouds.)
So now the cloud FBO would look like a bunch of yellow if you drew it to screen as is. We just need to make a slight adjustment to our fragment shader above to use the encoded data:
void main()
{
vec2 texCoord = gl_FragCoord.xy / u_res.xy;
vec4 sceneColor = texture2D(u_texture, texCoord);
vec4 addColor = texture2D(u_texture2, texCoord);
gl_FragColor = vec4(addColor.r*addColor.g) + sceneColor*(1-addColor.g);
}
If you want to tint your clouds something other than pure gray, you can add a uniform color tint:
void main()
{
vec2 texCoord = gl_FragCoord.xy / u_res.xy;
vec4 sceneColor = texture2D(u_texture, texCoord);
vec4 addColor = texture2D(u_texture2, texCoord);
gl_FragColor = u_cloudTint*vec4(addColor.r*addColor.g) + sceneColor*(1-addColor.g);
}