Client destroy actor?

Is it not possible for clients to destroy actors?
I have some randomization in my levels where I want to destroy some actors, this works for the server but when the client is trying to destroy an actor nothing happens.
Is there something I’m missing/is there a workaround? I don’t want to have to send a ton network traffic just to set up the level.

Currently I’m doing an infinite world game where the players can be quite far - so I’m trying to Not replicate actors but instead have dummy actors on the client that act as the other player when they are close but otherwise everyone has their own objects and actors.
Sadly UE4 appears to not like me too much so far as Clients cannot destroy actors themselves and when I add a pawn, I cannot Not add it to every client too, as only the server is allowed to issue possession order, thus I have to add them to the server first. I wanted to do the possession and then remove their originals from the clients/server but obviously if such an order is issued on clients, nothing happens, while server deletes it from everyone not just itself.

In short I would be interested as well.

EDIT:

I realized that I was wrong. Clients can destroy actors. The reason why I couldn’t destroy said actors was because it was the server creating them. You can only destroy your own stuff. Which is sort of logical. Now if only I could make a client possess a pawn it created…