ENHANCED DOT PITCH
Keep in mind that the mask is a single size - you shouldn't scale it up with the game image. Targeting a specific TV, say 27" with a specific dot pitch, will let you calculate what the mask should look like. Create it at some obscene resolution (4000x3000?) so it will look smooth. Scale up the game image to that size, apply the mask, add some bloom, then scale the whole thing down to your output resolution. This should give you a fairly accurate image and will probably be very tolerant of scaling at non-integer increments. Using 2x,3x,etc isn't exactly the right aspect ratio for a lot of these games, you typically require some extra horizontal stretching.