Описание из хэлпа.
To create a mask with Auto-trace:
In the Timeline window, adjust the work area to the range of frames for which you want to create mask shape keyframes.
Select one or more layers in the Composition window.
Choose Layer > Auto-trace.
Select the appropriate options in the Auto-trace dialog box, and then click OK. One or more masks appear in the Composition window and in the Timeline window for that layer.