You can draw any 2D shapes using the Paint functionality.
Please check the
[Tutorials]:Paint And Touch\Paint_Elements.xcomp
[Tutorials]:Paint And Touch\Paint_Path.xcomp
examples.

The texture coming out from Paint View can be used as mask.

I'm not sure what do you mean by doing masks in 3D scene. Could you elaborate?

What about polygon type of masks? are there nay easy way to do such masks like for example AE?

About 3D masks i mean using 3D shapes as masks for other shapes?

If you check [Tutorials]:Paint And Touch\Paint_Path.xcomp it show how to build a path.  If you only use Lines in the path you get a polygon. It can be automatically closed.

If you render the 3D objects with pure white material (for e.g. using Flat_Solid.xshad) onto a black background you get a mask.
But I'm still not sure this was your question.  If not, please describe an actual example.

I have checked these examples but I think they are not user friendly for doing masks.

What I am asking for an easy way to drew masks using preview window like other softwares like After effects,vizrt,etc

also are there  any tools to merge these masks (subtract,merge,etc)

Unfortunately at the moment there isn't any more user friendly method.
Alternatively you can draw the mask in Photoshop, and use it as a texture.

For merge operations use the Blender module with Blending Modes like Add, Multiply, Subtract.