How to determine whether a given object is a mask - actionscript-3

Question

Apparently, in Adobe's wisdom, both the object being mask, and the masking object contain a "mask" property. This leads to a cyclical reference that prevents determining which is the actual mask and which is the masked.
For example...
var clip:MovieClip = new MovieClip();
clip.name = "clip";
addChild(clip);
var boundary:Shape = new Shape();
boundary.name = "boundary";
clip.addChild(boundary);
clip.mask = boundary;
trace(clip.mask.name); // outputs "boundary"
trace(clip.mask.mask.name); // outputs "clip"
I've iterated through the properties of both clip and boundary, and there doesn't seem to be anything unique that sets them apart. My first thought was to force a removal of the superfluous "mask" reference in boundary, however, that also sets the mask property in clip to null, thereby removing the mask.
My second thought was to check the parent relationship of a mask. If the parent is the same as the object's mask, then the object in question is itself the mask.
var a:Array = [clip, boundary];
for each (var item in a) {
if (item.mask == item.parent) {
trace(item.name + " is a mask");
}
}
// outputs "boundary is a mask"
Seems to work, and after checking the API reference on masks, it's clear that when caching, a mask will need to be a child of the masked, however... it's also valid to have a mask at the same depth as the masked (I do this from time to time when a mask needs to not travel with the masked content).
For example...
MainTimeline ¬
0: clip ¬
0: boundary
... can also be laid out as ...
MainTimeline ¬
0: clip ¬
1: boundary
So, there's the conundrum. Any ideas on how to resolve this?

Answer 1

The "best" hack I've found so far is to run hitTestPoint on the objects (after making sure they have something to hit under the target). Masks do not appear to ever return true for a full pixel hit test. This seems to work in most basic situations that I've tested:
public function isMask(displayObject:DisplayObject):Boolean {
// Make sure the display object is a Class which has Graphics available,
// and is part of a mask / maskee pair.
if ((displayObject is Shape || displayObject is Sprite) && displayObject.mask) {
// Add a circle at the target object's origin.
displayObject['graphics'].beginFill(0);
displayObject['graphics'].drawCircle(0, 0, 10);
var origin:Point = displayObject.localToGlobal(new Point());
var maskLocal:Point = displayObject.mask.globalToLocal(origin);
// Add a circle at the same relative position on the "mask".
displayObject.mask['graphics'].beginFill(0);
displayObject.mask['graphics'].drawCircle(maskLocal.x, maskLocal.y, 10);
// No matter which is the actual mask, one circle will reveal the other,
// so hit testing the origin point should return true.
// However, it seems to return false if the object is actually a mask.
var hit:Boolean = displayObject.hitTestPoint(origin.x, origin.y, true);
displayObject['graphics'].clear();
displayObject.mask['graphics'].clear();
// Return true if the hit test failed.
return !hit;
} else {
return false;
}
}
Obviously you'd want to cache the graphics in case the objects already have some, and it could do with something more elegant than casting as Sprite so that it can handle Shapes, but it's a start.
Edit: Accessing ['graphics'] lets this accept Shapes, but obviously isn't super efficient. I'm not sure what the best method would be, short of adding an interface.

Answer 2

Great question, haven't run into this before. I wasn't aware of the cyclical reference.
If your masks are exclusively masks, I would suggest just incorporating that into your naming convention. For example calling it clipMask as opposed to boundary.
As noted in the comments, in the situation where the mask is on the same display list, you could use getChildIndex() to compare their position on the display list of the parent.
Typically in that situation I'll have the mask layered over the other display object. This is not enforced obviously, and I don't believe that it has any effect on the result of the mask visually. But it's easier to maintain for a large group than a naming convention.
Still not ideal obviously.