I am testing on Mac OS X 10.12.6 (16G29) with Unreal Version: 4.18.2-3794801+++UE4+Release-4.18. The editor seems to run with Metal when it runs. I am using a Retina macbook. I have not tested on Windows.
SETUP
I am trying to follow this third party tutorial Multi-color Outline Post Process - Tom Looman . It at one point shows an attractive image of a “Buffer visualization” for “custom stencil” mode: https://i2.wp.com/www.tomlooman.com/wp-content/uploads/2015/10/ue4_coloredoutlines03.jpg?w=900
I have a project (I can provide a sample project) where I have set Project Settings → Postprocessing → Custom Depth Stencil Pass to “Enabled with Stencil”. I have created two cubes and checked “Custom RenderDepth” in both. “Custom Depth Stencil Write Mask” is set to “Default” and “CustomDepth Stencil Value” is set to 1 for one cube and 2 for the other cube.
Here are the cubes:
OBSERVED BEHAVIOR
If I go to the “Lit” menu , and choose “Buffer Visualization”, then “Custom Stencil”, I see only blackness.
If I then go to the menu, choose “Buffer Visualization”, then “Custom Depth”, I get some gray cubes that look plausibly like a visualization of the depth buffer of the two cubes. I can in this visualization use normal actions to move the camera such as click-dragging or option-click-dragging, if I select items in the World Outliner they are highlighted in the viewport, and if I delete an item in the World Outliner it disappears from the viewport.
If I go back to the menu, choose “Buffer Visualization”, then “Custom Stencil”, I get this:
First off, notice that this text is illegible. Whereas the screenshot from the tutorial above has nice legible block text, I get a slightly garbled but readable “255” on the left cube and on the right cube I cannot guess what number this is meant to be. Maybe this is a mac retina thing?
Second off, if I at this point perform camera movement such as click-dragging or option-click-dragging, nothing happens. I can select items in the world outliner or even delete them and nothing changes on screen. The viewport is simply not animating. If I switch back to “Lit” and move the camera around, then go back to “Custom Stencil”, I see the same frozen view of the two cubes above I last saw. The only way to get the “Custom Stencil” view to update is to switch first to Custom Depth and then switch to Custom Stencil. Custom Stencil will always show the scene with the content and camera angle of the most recently rendered Custom Depth frame.
EXPECTED BEHAVIOR
The text in the “Custom Stencil” buffer visualization should be readable. This is most important.
The “Custom Stencil” mode should animate and update and allow selection of objects the same way that the “Custom Depth” mode does.
If I have misconfigured my project somehow and this is why “Custom Stencil” is misbehaving, the “Custom Stencil” mode should show an error rather than illegible numbers.
The text in Mac Retina mode is also (in addition to being pixel junk) very small; if this is possible, it would be nice if it could be the same size as it is in the screenshots in the Tom Looman tutorial.