пару часов поисков привели к ответу, напишу здесь, чтобы не забыть самому и помочь кому-то еще )
It's a known problem with VRay & Max 2012.
Possible solutions:
- Disabling the "Improve Quality Progressively" checkbox (Display Performance Panel for Nitrous viewports).
- Switch back from Nitrous to DX.
- Some folks have this problem when multiple windows are open or running multiple programs. Close everything but Max.