Stage3D AGAL Diffuse Light Shader, Dynamic Object Questions - actionscript-3

Question

There are quite a few ActionScript Stage3D tutorial examples 'out there'. I am thankful to all those who attempt to provide us newbies with working code. That said I don't believe I have yet found an example which I think is correctly handling the Lambertian reflectance as outlined by the Wikipedia entry, but I hesitate to add that, as a newbie, perhaps I just have failed to understand the implementations.
Here's what I think is the basic requirement of any implementation -- that it be able to compare the orientation of the light source [I am purposely limiting this discussion to the simpler case of a 'directional light' mimicking the Sun, rather than a 'spot light'.] to the orientation of the normal to the face to be illuminated.
And here is what I think is the heart of the problem I am seeing -- that the computation of the normal of the face, in almost every case, is being performed when the geometry of the object is being created. So, the value of the normal being passed to the shader is expressed in terms of the local object space.
Now I know that the way this wonderful forum works, you would prefer that I just give some sample code, so someone could identify a specific mistake in either the setup code used on the CPU or the shader code used on the GPU. Because I can find examples of this problem using many different frameworks, I think what I [and I imagine many others] need, is not a specific coded solution, but some specific clarification of what is actually required to get a photo-realistic rendering of an object in the simple, base case of fixed camera view point, a non-moving light source, and ignoring considerations of specularity, etc.
So, when performing the dot product of the two vectors:
A. Should the Vector3D value representing the normal to the triangular face to be illuminated be calculated using the object space values of the three vertices or the world space values after transformation?
B. If world space values are required, as I believe, should the dynamically-calculated normal workload be performed each render cycle on the CPU or the GPU?
Thank you.

Answer 1

Terry I ran into the same problem recently, I suspect you have solved it or moved on. But I thought I would answer the question anyways.
I choose to transform the normal in the vertex shader.
var vertexShader:Array = [
"m44 vt0, va0, vc0", // transform vertex positions (va0) by the world camera data (vc0)
// this result has the camera angle as part of the matrix, which is not good when calculating light
"mov op, vt0", // move the transformed vertex data (vt0) in output position (op)
"add v0, va1, vc12.xy", // add in the UV offset (va1) and the animated offset (vc12) (may be 0 for non animated), and put in v0 which holds the UV offset
"mov v1, va3", // pass texture color and brightness (va3) to the fragment shader via v1
"m44 v2, va2, vc4", // transform vertex normal, send to fragment shader
// the transformed vertices without the camera data
"m44 v3, va0, vc8", // the transformed vertices with out the camera data, works great for default AND for translated cube, rotated cube broken still
in the fragment shader
// normalize the light position
"sub ft1, v3, fc0", // subtract the light position from the transformed vertex postion
"nrm ft1.xyz, ft1", // normalize the light position (ft1)
// non flat shading
"dp3 ft2, ft1, v2", // dot the transformed normal with light direction
"sat ft2, ft2", // Clamp dot between 1 and 0, put result in ft2.
Last but not least, what data you pass in! Took quite a bit of experimenting for me to find the magic.
// mvp is the camera data mixed with each object world matrix
_context.setProgramConstantsFromMatrix(Context3DProgramType.VERTEX, 0, mvp, true); // aka vc0
// $model is the model to be drawn
var invmat:Matrix3D = $model.modelMatrix.clone();
invmat.transpose();
_context.setProgramConstantsFromMatrix(Context3DProgramType.VERTEX, 4, invmat, true); // aka vc4
var wsmat:Matrix3D = $model.worldSpaceMatrix.clone();
_context.setProgramConstantsFromMatrix(Context3DProgramType.VERTEX, 8, wsmat, true); // aka vc8
I hope this helps someone in the future.