Solution. Click in the project monitor and start drawing your rotoscope matte. Then the rotoscope keyframe options become enabled.