If you add meshes to the foliage painter in the landscape and then remove someone added in the map from the foliage list, in the editor looks like are gone, but at press play the meshes spawn.
I was able to reproduce this on my end and have entered a bug report, UE-32010, to be assessed by the development staff. Additionally, I tried using build to force the map to remove the actors with no success, so at present your workaround (restarting the project) is the only known workaround.