Maya 4.5:
RenderGlobals->RenderOptions->Composite (Composite Threshold)
Controls the amount of edge anti-aliasing in a matte if Composite is on. If you are rendering images for use in a video game, and you are using 8 or 16 bit color, set Composite Threshold to 1 for smooth matte edges, with no jagged edges. The default value is 0.
Maya 5.0:
RenderGlobals->RenderOptions->Color/Compositing
Premultiply
If this option is on (default), premultiplication takes place. If this option is off, the premultiply threshold option is enabled.
If on, Maya renders objects so that they are not anti-aliased against the background. For example, a pixel on the edge of an object is not mixed with the background color. (In TIFF terms, Maya generates unassociated alpha.) If off, Maya anti-aliases objects against the background. Premultiply is off by default.