Cocos2dx: Sprite3D won't render to texture - cocos2d-x

Question

I use RenderTexture to render a layer with all its nodes to a texture then apply an OpenGL shader on that texture to create post-process effects. It works all fine except with Sprite3D and Billboard nodes. It has been asked on their forums a few times without any response. I wonder if anyone got this to work.
Here is an example:
Layer* gameLayer = Layer::create();
this->addChild(gameLayer, 0);
auto dir = Director::getInstance()->getWinSize();
Camera *camera = Camera::createPerspective(60, (GLfloat)dir.width / dir.height, 1, 1000);
camera->setPosition3D(Vec3(0, 100, 100));
camera->lookAt(Vec3(0, 0, 0), Vec3(0, 1, 0));
gameLayer->addChild(camera); //add camera to the scene
// You'll get a NULL camera inside BillBoard::calculateBillbaordTransform() function
// if you call visit()
/*auto billboard = BillBoard::create("cocos2d-x.png", BillBoard::Mode::VIEW_POINT_ORIENTED);
billboard->setPosition(Vec2(VisibleRect::center().x, VisibleRect::center().y));
gameLayer->addChild(billboard, 100);*/
// This one won't render into the texture
Sprite3D* sprite3D = Sprite3D::create("blend_test/character_3_animations_test.c3b");
sprite3D->setScale(5.0f); //sets the object scale in float
sprite3D->setRotation3D(Vec3(0.0f, 0.0f, 0.0f));
//sprite3D->setPosition3D(Vec3(VisibleRect::center().x, VisibleRect::center().y, 0.0f)); //sets sprite position
sprite3D->setPosition(Vec2(VisibleRect::center().x, VisibleRect::center().y));
gameLayer->addChild(sprite3D, 1); //adds sprite to scene, z-index: 1
// This one works just fine and appears black and white as expected
// in the resulting texture
Sprite* sprite2D = Sprite::create("cocos2d-x.png");
sprite2D->setPosition(Vec2(VisibleRect::center().x, VisibleRect::center().y));
gameLayer->addChild(sprite2D);
// Black and white OpenGL shader
GLProgram* glProgram = GLProgram::createWithFilenames("shaders/gray.vert", "shaders/gray.frag");
glProgram->bindAttribLocation(GLProgram::ATTRIBUTE_NAME_COLOR, GLProgram::VERTEX_ATTRIB_POSITION);
glProgram->bindAttribLocation(GLProgram::ATTRIBUTE_NAME_POSITION, GLProgram::VERTEX_ATTRIB_COLOR);
glProgram->bindAttribLocation(GLProgram::ATTRIBUTE_NAME_TEX_COORD, GLProgram::VERTEX_ATTRIB_TEX_COORD);
glProgram->bindAttribLocation(GLProgram::ATTRIBUTE_NAME_TEX_COORD1, GLProgram::VERTEX_ATTRIB_TEX_COORD1);
glProgram->bindAttribLocation(GLProgram::ATTRIBUTE_NAME_TEX_COORD2, GLProgram::VERTEX_ATTRIB_TEX_COORD2);
glProgram->bindAttribLocation(GLProgram::ATTRIBUTE_NAME_TEX_COORD3, GLProgram::VERTEX_ATTRIB_TEX_COORD3);
glProgram->bindAttribLocation(GLProgram::ATTRIBUTE_NAME_NORMAL, GLProgram::VERTEX_ATTRIB_NORMAL);
glProgram->bindAttribLocation(GLProgram::ATTRIBUTE_NAME_BLEND_WEIGHT, GLProgram::VERTEX_ATTRIB_BLEND_WEIGHT);
glProgram->bindAttribLocation(GLProgram::ATTRIBUTE_NAME_BLEND_INDEX, GLProgram::VERTEX_ATTRIB_BLEND_INDEX);
glProgram->link();
glProgram->updateUniforms();
RenderTexture* renderTexture = RenderTexture::create(VisibleRect::width(), VisibleRect::height());
renderTexture->retain();
Sprite* ppSprite = Sprite::createWithTexture(renderTexture->getSprite()->getTexture());
ppSprite->setTextureRect(Rect(0, 0, ppSprite->getTexture()->getContentSize().width,
ppSprite->getTexture()->getContentSize().height));
ppSprite->setAnchorPoint(Point::ZERO);
ppSprite->setPosition(Point::ZERO);
ppSprite->setFlippedY(true);
ppSprite->setGLProgram(glProgram);
this->addChild(ppSprite, 100);
renderTexture->beginWithClear(0.0f, 0.0f, 0.0f, 0.0f);
auto renderer = _director->getRenderer();
auto& parentTransform = _director->getMatrix(MATRIX_STACK_TYPE::MATRIX_STACK_MODELVIEW);
gameLayer->visit(renderer, parentTransform, true);
//gameLayer->visit();
renderTexture->end();
ppSprite->setTexture(renderTexture->getSprite()->getTexture());

Answer 1

Cocos2d-x v3.11.1 (current as of this post) and below don't properly support RenderTextures with Sprite3D because of a clear depth buffer bug.
There is a GitHub issue on the bug. But a workaround now exists:
...
sprite3D->setForce2DQueue(true); // puts your Sprite3D on same render queue as the RenderTexture. More info below.
...
auto rt = RenderTexture::create(1280, 720, Texture2D::PixelFormat::RGBA8888, GL_DEPTH24_STENCIL8); // By default a depth buffer isn't created
rt->setKeepMatrix(true); // required
...
...
rt->beginWithClear(0, 0, 0, 0, 1); // required, clears the depth buffer
Also, changes need to be made to RenderTexture.cpp. This fixes the clear depth buffer bug in Cocos2d-x.
void RenderTexture::onClear()
{
// save clear color
GLfloat oldClearColor[4] = {0.0f};
GLfloat oldDepthClearValue = 0.0f;
GLint oldStencilClearValue = 0;
GLboolean oldDepthWrite = GL_FALSE;
// backup and set
if (_clearFlags & GL_COLOR_BUFFER_BIT)
{
glGetFloatv(GL_COLOR_CLEAR_VALUE, oldClearColor);
glClearColor(_clearColor.r, _clearColor.g, _clearColor.b, _clearColor.a);
}
if (_clearFlags & GL_DEPTH_BUFFER_BIT)
{
glGetFloatv(GL_DEPTH_CLEAR_VALUE, &oldDepthClearValue);
glClearDepth(_clearDepth);
glGetBooleanv(GL_DEPTH_WRITEMASK, &oldDepthWrite);
glDepthMask(true);
}
if (_clearFlags & GL_STENCIL_BUFFER_BIT)
{
glGetIntegerv(GL_STENCIL_CLEAR_VALUE, &oldStencilClearValue);
glClearStencil(_clearStencil);
}
// clear
glClear(_clearFlags);
// restore
if (_clearFlags & GL_COLOR_BUFFER_BIT)
{
glClearColor(oldClearColor[0], oldClearColor[1], oldClearColor[2], oldClearColor[3]);
}
if (_clearFlags & GL_DEPTH_BUFFER_BIT)
{
glClearDepth(oldDepthClearValue);
glDepthMask(oldDepthWrite);
}
if (_clearFlags & GL_STENCIL_BUFFER_BIT)
{
glClearStencil(oldStencilClearValue);
}
}
See the issue for more details. I also made an example gist of the workaround. Screenshot below.
I'm not sure about billboards, but this workaround might fix it too.
Info on Cocos2d-x render queues:
The Sprite3D needs to be on the same render queue as the RenderTexture. Cocos2d-x (as of v3.7 or so) now has 5 render queues:
Global Z Order < 0
3D Opaque
3D Transparent
Global Z Order == 0 (default for 2D)
Global Z Order > 0
You can put the Sprite3D and the RenderTexture on the last queue with setGlobalZOrder(1) or just put the Sprite3D in the 2D queue with sprite3D->setForce2DQueue(true).

Answer 2

unlike cocos2d RenderTexture the following worked fine for 3D screen capture (or anything i imagine)!
Sprite * CcGlobal::getScreenAsSprite(void) {
Size screenSize = Director::getInstance()->getWinSize();
int width = screenSize.width;
int height = screenSize.height;
std::shared_ptr<GLubyte> buffer(new GLubyte[width * height * 4], [](GLubyte* p) { CC_SAFE_DELETE_ARRAY(p); });
glPixelStorei(GL_PACK_ALIGNMENT, 1);
glReadPixels(0, 0, width, height, GL_RGBA, GL_UNSIGNED_BYTE, buffer.get());
Image* image = new (std::nothrow) Image;
image->initWithRawData(buffer.get(), width * height * 4, width, height, 8);
Texture2D *texture = new (std::nothrow) Texture2D();
texture->initWithImage(image);
SpriteFrame *spriteFrame = SpriteFrame::createWithTexture(texture, Rect(Vec2(0, 0), screenSize));
Sprite *sprite = Sprite::createWithSpriteFrame(spriteFrame);
sprite->setFlippedY(true);
delete image;
return sprite;
}
===================================================