Collision detection doesn't always work

Hi I am working on a Pinball game. The problem i am having is for some reason my ball likes to randomly pass through objects (the walls, the flippers, etc.). In my physics setting I have CCD and sub-stepping turned on (been playing with sub-stepping’s settings) and that didn’t fix my issue entirely. The only thing is that might be the issue is that I am using a physics material to get the ball to act properly (seen in image below). I have no idea if the ball and or the flippers are moving to fast for the detection to work or if the material is making this happen?

Any and all advice is welcome. Thank you for your time.