Tessellation gives very rough results.

So, I’m trying to build a corrugated metal material. I have this working reasonably well with a normal and parallax displacement map, but I’m now trying to get a better outline using tessellation. I am having issues, though, because my tessellation is producing extremely rough results:

To get this, I’m feeding a heightmap, which is a simple vertical gradient, into the tessellation distance, as shown:

16819-tessellation+issue+graph.png

The height map I’m using is simply identical, vertical gradient lines, as you can see. There is no variation along their length (I was also using this same texture for the parallax map, which produced correct results).

My tessellation settings are: PN Triangles, Crack Free Tessellation enabled, and Adaptive Tessellation enabled. Changing any of these produces no change to this effect, unless I turn tessellation off entirely.

Can anyone tell me what I’m doing wrong, or how to achieve smooth tessellation? I’m sure it’s not a matter of not having enough triangles, as there are thousands, and the effect does not seem to lessen as I increase the resolution of the base mesh.

EDIT: What did reduce the effect, however, is reducing the tiling of the material. This seems to give the expected results, so perhaps it is something to do with the sampling of the triangles being somewhat off.

Okay, it seems tessellation works perfectly fine on curved meshes, but struggles on flat ones like cubes or planes.

Hey Hoeloe -

Are you scaling the mesh that the tessellation is being applied to? The Texture that you are using in World Displacement, I would suggest unchecking sRGB in the texture editor and setting it to a Vector Displacement map.

Let me know and I will continue to test from our end -

Eric Ketchum

Interestingly, I’ve just had a look at this, and discovered that, with those settings put up correctly, tessellation does indeed produce very smooth results. However, this is not the case when I apply it to a blueprint I made, which is designed to make applying materials to meshes of different sizes simpler. It creates a dynamic material instance in the construction script, and tweaks parameters of the material based on settings in the blueprint details panel. When applying the tessellated material directly to a mesh, unless I scale the UVs, it appears to work correctly.

It also looks very odd in the preview instance in the material editor, though I suspect this is just because of the low resolution of the cube mesh.

Essentially, it seems that the problem is fixed by increasing the triangle resolution, or decreasing the height map resolution, which is what you’d expect from tessellation that isn’t subdivided enough. However, it seems to require an obscene amount of triangles (the effect above is seen with 8x tessellation multiplier) to get anything like a smooth effect, which just doesn’t seem right.

And yes, I am resizing the mesh, but the effect appears when I have the mesh at its native scale, too.

This is a cube at its native scale, with the UVs in the range 0-6 and 8x tessellation. This cube is specifically designed for tessellating materials, with 40 triangles on each side with no tessellation applied in the material.

It doesn’t appear to be related to the blueprint, because I made an instance and applied it directly to the mesh, with the same result.

However, I noted that it gives this “peak and valley” effect, with the peaks being correct, and the valleys being incorrect. I just counted the peaks - exactly 40, the number of triangles in the mesh. It seems that the tessellation is not shifting any points along the original edges in the mesh - only between them (except on the border edges). Is this intended behaviour, and if so, can I do anything about it?

I have somehow managed to get the effect to be small enough to not be noticeable, for now. I would still like to know the cause, and how to fix it, if possible.

Hey Hoeloe -

Can you send me your Height Displacement map for testing? I asked about the scale because we did have an issue in a previous versions with scaling tessellations causing issues and I wanted to make sure that this had not been a regression.

Thank You

Eric Ketchum

Certainly. Give me a moment to upload the asset. I don’t think it’s a regression, as this also occurs on meshes at their native scale. Interestingly, I’ve just discovered that effect seems to be worse when the height map in question is set as a vector displacement map, and somewhat smoother if it’s just set at the default. The difference is only slight though.

EDIT: Here is the asset file that gives acceptable, but still flawed, results: https://dl.dropboxusercontent.com/u/30847069/H_1Gradient.uasset

And here is the one that gives unacceptable results: https://dl.dropboxusercontent.com/u/30847069/H_3Gradient.uasset

The first requires the UVs to be in the range of about 0-18 before the issue becomes a problem, while the second needs just 0-6, though I suspect this is more to do with the image content than anything else.

Here’s some more info. I tried the tessellation on a mesh with no UV scaling, and no resizing of the mesh, to see if I could still spot the issue. What I found instead was this:

This looks to me like the border vertices are remaining in their original positions, and not being shifted by the tessellation, causing sudden changes in the shape of the mesh. While it doesn’t quite look the same as the above issue, it has many of the same characteristics.

Interestingly, this only occurs along two edges that I can see. One side at the top, and the opposite at the bottom (which also appears to have issues with crack-free tessellation):

EDIT: This also occurs along 2 of the other edges, but both on the same side (positive y) this time. The other side appears to be correct.

Hey Hoeloe -

After some tests, I have been able to clean those edges with smoothing groups. The issue that you are seeing is related to the vertices along the edge. In the engine when their are no smoothing groups assigned between faces, the engine will render two vertices to keep the hard edge. The material that is mapped to the mesh is then trying to move two verts that are essentially needing to go in two different directions.

Try adding smoothing groups (Max) or Soften Edges (Maya) on your cube and you should see relatively clean and some nice tessellation along the edges. The texture itself has a limit to the resolution and has variation within the gradient which will always yield some up down variation (You can see this by going into Photoshop and bringing up Levels and notice the many spikes and valleys. There is no great way to fix and in most cases its not an issue.)

Your initial problem with the scaled UVs and having that effecting your tessellation is related to the texture sample itself. Scaling it upwards will exacerbate the texture’s highs and lows.

For the most part I think the smoothing groups and a good refined texture would yield the results you are looking for.

Thank You

Eric Ketchum

Ah, that sounds likely. I was aware that hard edged meshes used multiple vertices, but it didn’t occur to me that it might be causing an issue. Of course, I did figure the results were never going to be perfect because of the finite resolution (and this makes sense that the results greatly improved when I not only used a texture with 1 gradient instead of 3, butwas also gaussian blurred, smoothing the texture. The aliasing effect did seem a little strong for the number of triangles, though (8 subdivision levels and 3200 native triangles should be ample), but I suppose the scaling of the UVs produced aliasing into the samples. I expect the results could be improved by blurring the resized texture, now I think about it, but the performance cost of doing that is unlikely to be worth it. In any case, thanks for all your help!

Hmm… Well, I tweaked the base mesh so it didn’t have any hard edges, and while this did close up the gaps, it also made the join between the edges extremely ugly:

Note how in the broken version, those areas that did work had very cleanly connected the UVs from the side face to the tessellated faces on top. This does not happen here, and in fact produces a result that is worse than the old one. The broken tessellation from before did not occur unless the camera was very close to the mesh, while this occurs all the time. Is there something I can do about it?

EDIT: I believe this is because the effect here is being produced from the tessellation directly, while the previous effect was being calculated from the “crack-free tessellation” option. To be perfectly honest, for the most part, I prefer the look of the older one, as I don’t actually need to get that close in most cases, though it is somewhat unreliable, so I would like to know if there is a way to sort this out properly.

Yes, I’m using the same material on all sides. My material uses a dot product between the vertical (0,0,1) and the vertex normal to determine the face angle, and changes the tessellation height based on this. It does not, however, change the tessellation multiplier, as I could understand that causing problems. My intention was to create a material that could have corrugation along vertical faces, and be smooth along horizontal ones.

Hey Hoeloe -

What are the material setup on your cube? are your sides and top and bottom using the same material? I have been able to reproduce the error and it is centered around the use of the Crack-Free Tessellation and the top trying to remain connected to the side. I will pass this along to our engineers for further testing, but I still feel that there should be a setup which corrects this error.

Thank You

Eric Ketchum

Hey Hoeloe -

I want to confirm, as you using this material in a level using the LPV Dynamic Illumination system? If so, can you test in a level without and see if you still get the same results. You should but I want to make sure as we have a known issue with tessellation and LPVs causing some rendering conflicts.

Thank You

Eric Ketchum

Yes I am (or I’m attempting to, I don’t seem to be able to generate the LPVs, but that’s a separate issue). I’m copying the files over to another project now.

EDIT: I copied everything into a project that uses precomputed lighting, but there’s no change to the result.

The only thing that seems to fix the issue is smoothing the edges in Maya, but as I showed above, this makes the UVs extremely ugly, and does not give the effect I need. As for a more tesaellated asset, I find it hard to believe that a 40x40 quad mesh (triangulated of course) is not tessellated enough, when the mesh itself is only 100 units along each side.

Hey Hoeloe -

After some investigation, the spikes you are seeing (the sawtooth pattern along the edges) are a result of the two mesh sides trying to pull the same verts in opposing directions. This effect can be mitigated by having an asset that is tessellated (more verts) before you apply the material and applying the appropriate smoothing settings that we discussed earlier. You could make sure that your verts are working correctly by doubling up and not welding your seams, assuming of course that you are going to use a full displacement so Z Fighting does not occur.

Thank You

Eric Ketchum

Hey Hoeloe -

Here is what you need to do to completely eliminate the vertex jaggedness from the tessellation. Open your cube (model) in Maya and select all the Vertices on the top face including the outer edge and set the vertex normals to (0,0,1) then on the bottom face and outer edge set the vertex normals to (0,0,-1). Add an Edge Loop underneath the top outer edge and above the bottom outer edge and move it as close as you can in both directions without, obviously, overlapping the actual box outer edge. Assuming you use the Dot Product calculation to only displace based on the 0,0,1 vector, you should know how enough normal and vertex information to avoid the jaggedness created when the VertexNormalWS node is givben a vertex with more than one normal (which the outer edges have before we reset them.).

Here is an image of the cube set up in Maya:

Thank You

Eric Ketchum

Yes that does fix it, but I did already try that. The problem is that this messes up the UVs, causing a very ugly and visible seam. I have posted a picture in an earlier comment.

Having said that, the image you posted appears to have two normals at each vertex on the border edge. I thought the point was to avoid this?