Trying to use Light channels, but have issues

I am trying to use light channels to light only certain static meshes.
It seems to work fine, except for when I turn off shadow casting on the lights. Is this a limitation I missed?

When I have a movable light on channel 0, with shadow casting ON
Next to a cube receiving light from only channel 1, the cube is black, all is fine

When I have a movable light on channel 0, with shadow casting OFF
Next to a cube receiving light from only channel 1, the cube is black at some angles, but will flicker and show the lights effect at other. Can someone explain to me what is causing this strange behaviour?