Light Linking? Highlights?

Anyone know how to get your lights to not effect the specularity in the scene? I have lights in my scene that are being used as a fill light, but you can see bright highlights in the scene. I want to turn that off on those lights. Anyone know how?

Set their Min Roughness value to 1. Details Panel > Light(extend by clicking the little arrow)