Clients net cull distance doesn't seem to take origin rebasing into account

It seems that the actor relevancy using the culling distance ignores world origin rebasing for clients.
When I teleport 2 characters very far away in order to trigger a world origin rebasing, the server is working fine (he sees the client) but the client doesn’t see the server.
If I check “Always relevant” or increase the cull distance to cover the distance from the location before the teleportation then everything works fine.

I’m using world composition with “enable world origin rebasing” and “enable multiplayer world origin rebasing” with a listen server.

Is it a bug or do I need some extra steps to make it work for clients?