Ugly border around objects in post-process shaders

Hello all,
I am currently trying to put together a cel-shader using the usual post process technique. Instead of just picking dark or bright though, I’d like to map different light levels to tints on a texture. I am met with an issue when I get to sampling the texture for the proper tint level.
Here is the result of the part of the material that gives me issues.

Notice the white pixels around the edge of the sphere on the lower left, and the gray pixels around the edge of the sphere at the top. Those pixels result is ugly artifacts in the final post process. Here is the node setup to replicate this:

And also, here is the texture I am using as the color ramp.

This seems like a simple thing, but I guess I must be missing something. Any ideas?