It seems that when generating the navmesh, the Agent Radius parameter is used not only to stop the nav mesh short of walls, but also results in a margin of the same radius anywhere there is a dropoff, such as in the image below.
Can this be prevented or configured somehow? I’d like my AI to be able to walk right to the edge of a ledge. Furthermore, I want them to be able to navigate along ledges which are narrower than their capsule.
Navmesh gets generated AgentRadius-away from any unwalkable area, may it be a collision, a lack of “ground” or “null” navigation area. The whole point of it being able to guarantee that every point on navmesh is actually reachable by navigation agent using it. If boundaries around ledges were less then agent’s capsule then the agent could slip down, which is undesirable.
You can try increasing resolution of your navmesh by decreasing CellSize, but that will impact navmesh generation time.
All in all we do plan to add a way to manually place navmesh pieces (like on an otherwise un-navigable places), but that’s low in our backlog.
Cheers,
–mieszko
Hey Mieszko, thanks for the reply.
If boundaries around ledges were less then agent’s capsule then the agent could slip down, which is undesirable.
I don’t really understand this part. In the attached image (I can’t get the image formatted inline for some reason), while I certainly wouldn’t expect the navmesh to allow the situation on the left, I would think a lot of people would want the one in the middle to be considered navigable and wouldn’t have their character set up to slip down from that position. As I understand it, it’s currently impossible to have navmesh generated for the situation on the right, no matter how lowCellSizeis set.
Anyway, is this something that would be relatively easy for me to modify in the engine source, or is it an assumption made deep in the detour code?
This is not doable without deep and dark knowledge of Recast Navigation library’s code. There be dragons 
Also, if you achieve that, regardless of the way do did that, you’d end up with images like this:
(or worse, depending on your skelmeshes and animations)
It’s true it would require some significant logic to control foot placement in order to look right for the general humanoid character case. Well anyway, it’s clearly a big job. Plus I looked into the recast code recently to see how point projection worked, I’d rather not dig too much deeper. I thought float* vectors were a thing of the past!
Any ideas for a workaround to allow AI navigation along very narrow ledges which won’t have navmesh? I’m guessing generating nav link proxy between start and end of ledge, but allowing navigation to an arbitrary point along the ledge might be awkward.