Sky Sphere Mesh being outlined by post process from Stylized Render demo

Running in to an issue with post process effects. I wanted to use the ‘toon outline’ effect from the Stylized Render demo and did get it working, but it also applies the outline effect to the sky mesh in a weird circular pattern. I’ll attach a screenshot of the effect.

Now I know that this outline material was set to not affect elements with Render Custom Depth enabled, but I also use Render Custom Depth to apply a blue outline to actors that the user can interact with when looking at them. If I enable RCD on the sky sphere mesh, it gets a permanent blue outline.

How can I address this? Is it something stencils could fix? I’m new to rendering and UE4 materials so it’s difficult working with post process effects, but I’m very close to achieving a look that I like.

I would turn off RCD on your skysphere. Is there a reason you need it?

Alternatively you could use stencils to ignore it but at that point definitely no reason to have RCD on.

RCD is off on the skysphere. That outline is an unbound postprocess effect that applies to everything that doesn’t have RCD enabled. I use RCD right now to outline objects in blue. Can I set a stencil value to just do nothing?

ahh, makes more sense. Two options. For any object you could check if the stencil index is some value and if it is then return the Scene Color.

Alternatively you could handle the sky sphere by making anything black in the custom depth be the scene color. This way only things with depth info would be affected.

Cool! I’m still very new to materials so I’m not sure exactly how either of those would look in the material blueprint, but that gives me an idea of how it should be set up. I’ll try playing around with it and see if I can figure it out.

Try this out:

Anything beyond the DistanceToBlendOut will use the SceneTexture:PostProcessInput0 (which is the scene color with post processing). This will apply to everything not added to custom depth because anything that isn’t set to render in it will be treated like it is the max distance away.

If you want things not RCD to render normally in front of things set to RCD then you need one more check to compare the actual Scene Depth versus the Custom Depth:

So here is a comparison of each step:

91199-capture3.jpg

This looks promising. I need to apply this to my current post-process effect which is responsible for the black outline, correct? I’m not entirely sure how to add this logic to the graph(which is already huge), so I tried this:

But I can’t seem to get the distance right. At 110,000,000 it applies an outline to everything, and at 100,000,000 it applies an outline to nothing. Neither of those numbers make sense to me. Any idea what I’m doing wrong?

With a complicated line shader like that it probably is already doing a bunch of logic to handle some of this. The links i had are to more understand what’s happening.

If you can send me the shader I could take a look at it. I went to download the Stylized Rendering project but for some reason the launcher is freezing when reaching the PreviewOrderState.

Sure, here is the material file I’m using. It’s the same one from the Stylized Rendering project but with the section I added above.

You’re right about there being other logic in the diagram, that may be why the distance check doesn’t work as expected.

Ahh thanks.

So the effect you are using uses mostly the default depth channel and not custom depth. Its intent seems to be highlighting everything except for things in the custom depth channel when PostProcessBlendWeight is 0. When it is set to 1 everything is highlighted that is appropriate by distance.

The sphere problem happens when the PostProcessBlendWeight is Less than 1 as it does not take the same depth into mind at those blend values. It clamps the values from .25 - 1 which means it will always show up to some extent and ruins the skysphere.

Changing the Clamp can fix it but your easiest solution to avoid this is to ignore what i posted before and just set your sky sphere to render in the custom depth. If you are using Epics sphere BP then just click on it, then in the component area of the details menu select the “Sky sphere mesh” component. In there you can set RCD to true.

The problem with that is I’m using RCD to draw a different outline effect when the player looks at something that can be used.

So if I set RCD on the sky mesh, this happens:

Do you think I could apply that distance check to the blu outline instead? That material BP is a little less complicated than the other post-process. This is also why I was thinking Stencils might be a solution? What if I made a stencil that was blue and one that was just translucent? And apply the blue stencil value to objects that are looked at? I still don’t really know how stencils work, just making a guess.

Using multiple post processes is probably expensive since they are so similar it would be great to combine them but may not be super manageable for you.

I’ve added some extra support to the shader you sent to show how you might handle this in a single post process.

Now I am altering the line thickness and color based on the stencil index. Also removed the constraint that was causing the striping.

https://drive.google.com/file/d/0B1mYraWgpJ4NMVVxS2djY0pVSUE/view?usp=sharing

You will need to have your Project Settings → Rendering → Custom Depth Stencil Pass = Enabled with Stencil.