Normal Map Workflow with Xnormal is not working.
Using the Unreal 4 Documentation found here **HERE**, I am unable to bake a good normal map using the documentation's settings. I have to change the settings quite drastically to achieve a bake that works, and even then, the bake is not quite perfect, with some surface shading errors still occuring. My workflow: High / Low / Cage / and Triangulate in 3Dsmax 2013 and 2014(tried both), 1 smoothing group for the mesh. Reset xforms on meshes. Export triangulated .fbx 2013 files to xnormal with same exact settings as documentation, baked using exported normals and cage. Import same triangulated .fbx into UE4, broken normals appear in the model previewer and in viewport.
I should note, the hard line that appears at the edge facing the camera, is present in the high poly. That is working as intended in the bake. The shading errors with the red X's are the problem areas.
Hello Quack and Everyone Else -
I have been trying to get a really good reproduction of this particular reported issue. From my investigation, this appears to be functioning exactly as intended in UE4 and in all the modeling programs. Before everyone roars in outrage, let me explain what I am seeing and how I got there so we can talk from the same space.
To begin, I think everyone understands how the vertex normal is derived, but to clarify the vertex normal is the unit vector that is perpendicular to the surface (for our discussion this is an ok definition, though vertex normals are slightly different than typical surface normals). From my investigation I cannot see where the vertex normal either derived or calculated comes into the engine as incorrect (I should note here that I am using 4.5.1 for my tests). If anyone is having Vertex Normals computing or derived (imported) coming into the engine as incorrect please post pictures of the vertex normal in your modeling program and the vertex normal in the static mesh editor and, if possible, upload the fbx of your mesh.
The problem that people are having, it seems from reading responses here and over on Polycount, is that the tangents are seeming to import correctly and are calculated incorrectly. In either case the math behind this is very simple to checkout, the tangent of the vertex should run along the + U unwrap starting from the vertex, or more technically, it is the vector which shows increasing U space along each surface. This is very important to understand as no matter what we do to our unwraps the UV space is always perpendicular to each other and always runs 0 to 1.
When you are calculating the tangents, the engine does not look at your smoothing information from Blender, Modo, Max or Maya because you are telling it not to look at it. All that it has to go off of is the unwrap and the vertex itself. Often times this will cause the engine to treat the vertex as multiple vertices because of how you have unwrapped that mesh. Ultimately though the engine looks at the vertex in teh context of your unwrap and assigns as many normals and tangents as it needs.
As a case study of this I used a Pyramid, like JedtheKrampus does over at Polycount. I am going to be showing you Maya images but I confirmed this behavior in both Max(2013) and Blender(2.72). To begin here is my mesh, it shows you hard edges:
And here is the same mesh completely Smoothed in maya:
Now the unwrap for the mesh:
So the smoothed mesh brought into UE4 with imported Normals and Tangents looks like this:
As you can see the Normals and tangents are identical between Maya and the engine. Now lets look at the Calculated Normals:
Looking at the unwrap shown with the Calculated normals and you can see the tangents is based on vertex and unwraps planes connected to the vertex that are not merely triangulated quads, so the bottom plane counts in calculation terms as one plane.
So from the examples I have seen, the tangents appear correct when calculated based on the various unwraps and meshes.
If you feel I've missed something or if you have evidence that the tangents are not calculating as I describe above, please let us know with screenshots and uploading an FBX.
Regarding the initial question from Quack - I would ask as I did earlier but I believe I did not address you directly and I apologize for that, did you triangulate your mesh before running normals? Often time triangulating your mesh before running a normals calculation will result in a normal where the triangulation is running an opposing direction which will cause the original triangulation to show up as a render error. But we have worked on and continue to work on updating the documentation so if you feel some workflow we are using is incorrect, please let us know.
answered Oct 27 '14 at 08:09 PM
Lovecraft_K ♦♦ STAFF
Follow this question
Once you sign in you will be able to subscribe for any updates here