pyautogui / pyscreeze - locate works w/o region parameter, but doesn't work w/ region parameter - pyautogui

Using pyautogui, I'm trying to locate an image on screen.
This finds the image (on primary monitor w/ 2560x1440 resolution), but takes nearly 5 sec:
icon1 = pyautogui.locateOnScreen('[filepath]\\icon1.png', grayscale=True)
To reduce search time, I provide a region parameter, but this doesn't find the image. It executes in ~0.7 sec
icon1 = pyautogui.locateOnScreen('[filepath]\\icon1.png', region=(1500,100,950,1100), grayscale=True)
Eliminating the grayscale parameter doesn't change the result - still ~0.7 execution time and image not found:
icon1 = pyautogui.locateOnScreen('[filepath]\\icon1.png', region=(1500,100,950,1100))
Then I tried setting the region parameter to the entire screen, but image not found and execution time is ~1.5sec, so it's searching (judging by longer execution time w/ larger region), just not finding.
icon1 = pyautogui.locateOnScreen('[filepath]\\icon1.png', region=(0,0,2559,1439))
Any suggestions on what to try next? Thanks.

I figured out what's happening. As a preface, I am working with multiple monitors, and have made modifications to pyautogui and pyscreeze as described in these 2 pyautogui github comments:
https://github.com/asweigart/pyautogui/issues/9#issuecomment-527236475
https://github.com/asweigart/pyautogui/issues/321#issuecomment-629819495
With these modifications, pyautogui works with multiple monitors, and pyautogui.position() and pyautogui.moveTo() treat the upper left corner of the primary monitor as coordinate (0,0) - this means monitors above or to the left of primary monitor will have at least one negative coordinate.
But pyscreeze is returning image search results based on treatment of the upper left corner of the upper left monitor as coordinate (0,0) - so all coordinates on extended desktop are positive.
The region parameter I was passing to locateOnScreen() was based on using pyautogui.position() reported coordinates, so the search was being executed on an incorrect area of the screen.
Once I figured that out, I was able to pass the correct region specification.
To illustrate, I inserted a print statement in pyscreeze to show where the image is found. And a print of the image search return. And in my code, I printed pyautogui.position().
retVal = Box(left=8008, top=3171, width=29, height=31)
Point(x=4182, y=1026)
mouse position:
Point(x=4182, y=1026)
So you can see that coordinates are quite different. Pyscreeze is showing coordinates based on 0,0 being upper left corner of upper left monitor, while the image search return variable are coordinates based on 0,0 being upper left corner of primary monitor, so I'm able to take the image search result variable and pass that to pyautogui.moveTo() and move the mouse there (moveTo needs coordinates based on 0,0 being upper left corner of primary monitor).
The issue stems from adopting changes from two different people to pyautogui and pyscreeze. I suppose I could adjust the changes to one of these packages to offset the coordinates so that both are referencing the same monitor's upper left corner as coordinate (0,0).

Your region is set wrong. Use (0,0,2559,1439).
And if this doesn't work make a new screenshot specifying the region for it, save it and look if that just captured screenshot would be found on the screen.
The code below should always work as expected:
from time import perf_counter as T
import pyautogui
# NOTICE that in pyautogui: region=(left, top, width, height)
pyautogui.screenshot('pyautogui_region_screenshot.png', region=(300, 500, 300, 200))
sT = T()
img = pyautogui.locateOnScreen('pyautogui_region_screenshot.png', region=(300, 500, 300, 200))
eT = T()
print(f'img = pyautogui.locateOnScreen took: {eT-sT:9.5f} seconds')
print(img)
By the way: check out https://stackoverflow.com/questions/72410958/taking-a-screenshot-of-a-specific-portion-of-the-screen-in-python to switch to another Python module ( mss ) for doing screenshots of small regions of the screen.
https://pyautogui.readthedocs.io/en/latest/screenshot.html

Related

Is it possible to limit the device/camera movement in Facebook AR Studio?

For example there is an image using canvas with a rectangle in World Space. In doing so the camera/device can look around freely with the image placed into the "real" world. I wonder if there is a way to limit that "movement", spanning left to right, top to bottom but the device/camera view is limited at a certain point. Even if users turn the device/camera 360degrees, the view is stuck at a certain point. Say if the user pans left the camera/device stops at rotationY: 9, If right then stops at rotationY :-15, rotationX is stuck at 0.
I saw there's a BoundBox in the documentation but not sure what that is. There's a DeviceMotionModule but no idea how to use it. I don't know what the script example given is suppose to do.
Look into using DeviceMotion. https://sparkar.com/ar-studio/learn/documentation/reference/classes/devicemotionmodule
The script example rotates the 3d plane according to the rotation of the phone.
You will have to do some maths to position your objects according to rules and signal you get from DeviceMotion.
Using the reactive module you can access the "Clamp" method which is actually made to restrict values between two bounds. I recently found this out because I had a similar problem. From this page
clamp(x: ScalarSignal, min: ScalarSignal, max: ScalarSignal): ScalarSignal
Returns a signal with the value that is the value of the given x signal constrained to lie between the values of the given min and max signals.
Note: The behavior is undefined if min is greater than max.

Jittering lines when scrolling a TMX map in cocos2d-x

I feel like I'm missing some fundamental concept as to why I am getting flickering when moving a tile map around.
I create a layer. In it, I add a TMXTiledMap.
_tileMap = TMXTiledMap::create("TMX/32Map.tmx");
_tileMap->setScale(1.f);
_floorLayer = _tileMap->getLayer("Floor");
this->addChild(_tileMap);
for(const auto& l : _tileMap->getChildren()) {
static_cast<SpriteBatchNode*>(l)->getTexture()->setAliasTexParameters();
}
this->scheduleUpdate();
In the update I move the layer.
Vec2 newPos = this->getPosition();
newPos.x = (newPos.x - 1);
newPos.y = (newPos.y - 1);
this->setPosition(newPos);
I realize I'm not moving it by dt. If I move it by dt I get an overall jumpiness to the whole layer. I understand this is due to how it renders partial pixels. But if I move it by one pixel like above, I get this # looking set of lines on the screen about 64 pixels or so on top and bottom and about 224 pixels from the left and right
That is when the window is 1024x768. If I make a 320x240 window, I don't see the lines and if I make it 640x480 I only see them on the left and right sides right near the edge of the screen.
Ultimately I'd just like to smoothly scroll a tile map around. Any help would be super appreciated, because I just can't seem to get started on this project.
For me working solution was to change CC_FIX_ARTIFACTS_BY_STRECHING_TEXEL in ccConfig.h from 1 to 0. Find ccConfig.h in cocos/base/.

What's the purpose of Canvas.Context Save and Restore in this example?

This page shows some animations in HTML5 canvas. If you look at the source of the scroller, there's a statement to save the context after clearing the rectangle and restoring it after the animation. If I substitute the restore statement with another ctx.clearRect(0, 0, can.width, can.height statement, nothing works. I thought the restore is restoring the cleared rectangle but it seems its restoring more info. What's that extra info that's needed for the next frame?
I am not looking for HTML5 textbook definitions of Save and Restore but I want to understand why they are needed in this specific example.
UPDATE
It's frustrating to get an answer where I specifically already mentioned in the question I don't want to get the definitions of save() and restore(). I already know Save() saves the state of the context and Restor()e restores it. My question is very specific. Why is restore() used in the manner in the example when all the Save did is saved an empty canvas. Why is restoring an empty canvas not the same as clearing it?
Canvas state isn't what's drawn on it. It's a stack of properties which define the current state of the tools which are used to draw the next thing.
Canvas is an immediate-mode bitmap.
Like MS Paint. Once it's there, it's there, so there's no point "saving" the current image data, because that would be like saving the whole JPEG, every time you make a change, every frame...
...no, the state you save is the state which will dictate what coordinate-orientation, dimension-scale, colour, etc, you use to draw the NEXT thing (and all things thereafter, until you change those values by hand).
var canvas = document.createElement("canvas"),
easel = canvas.getContext("2d");
easel.fillStyle = "rgb(80, 80, 120)";
easel.strokeStyle = "rgb(120, 120, 200)";
easel.fillRect(x, y, width, height);
easel.strokeRect(x, y, width, height);
easel.save(); // stores ALL current status properties in the stack
easel.rotate(degrees * Math.PI / 180); // radians
easel.scale(scale_X, scale_Y); // any new coordinates/dimensions will now be multiplied by these
easel.translate(new_X, new_Y); // new origin coordinates, based on rotated orientation, multiplied by the scale-factor
easel.fillStyle = "gold";
easel.fillRect(x, y, width, height); // completely new rectangle
// origin is different, and the rotation is different, because you're in a new coordinate space
easel.clearRect(0, 0, width, height); // not even guaranteed to clear the actual canvas, anymore
easel.strokeRect(width/2, height/2, width, height); // still in the new coordinate space, still with the new colour
easel.restore(); // reassign all of the previous status properties
easel.clearRect(0, 0, width, height);
Assuming that you were only one state-change deep on the stack, that last line, now that your canvas' previous state was restored, should have successfully cleared itself (subpixel shenanigans notwithstanding).
So as you can see, it has very, VERY little to do with erasing the canvas.
In fact, it has nothing to do with erasing it, at all.
It has to do with wanting to draw something, and doing the basic outlining and sweeping colours/styles, and then manually writing in the colours for the smaller details on top, and then manually writing all of the styles back the way they were before, to go back to sweeping strokes for the next object, and on and on...
Instead, save general states that will be reused, create a new state for smaller details, and return to the general state, without having to hard-code it, every time, or write setter functions to set frequently-used values on the canvas over and over (resetting scale/rotation/affine-transforms/colours/fonts/line-widths/baseline-alignment/etc).
In your exact example, then, if you're paying attention, you'll see that the only thing that's changing is the value of step.
They've set the state of a bunch of values for the canvas (colour/font/etc).
And then they save. Well, what did they save?
You're not looking deep enough. They actually saved the default translation (ie: origin=0,0 in original world-space).
But you didn't see them define it?
That's because it's defined as default.
They then increase the step 1 pixel (actually, they do this first, but it doesn't matter after the first loop -- stay with me here).
Then they set a new origin point for 0,0 (ie: from now on, when they type 0,0 that new origin will point to a completely different place on the canvas).
That origin point is equal to x being the exact middle of the canvas, and y being equal to the current step (ie: pixel 1 or pixel 2, etc... and why the difference between starting at 0 and starting at 1 really doesn't matter).
Then what do they do?
They restore.
Well, what have they restored?
...well, what have they changed?
They're restoring the point of origin to 0,0
Why?
Well, what would happen if they didn't?
If the canvas is 500px x 200px, and it starts at 0,0 in our current screen space... ...that's great...
Then they translate it to width/2, 1
Okay, so now when they ask to draw text at 0,0 they'll actually be drawing at 250, 1
Wonderful. But what happens next time?
Now they're translating by width/2, 2
You think, well, that's fine... ...the draw call for 0,0 is going to happen at 250, 2, because they've set it to clear numbers: canvas.width/2, 2
Nope. Because current 0,0 is actually 250,1 according to our screen. And one translation is relative to its previous translation...
...so now you're telling the canvas to start at it's current-coordinates' 0,0 and go left 250, and down 2.
According to the screen (which is like a window, looking at the map, and not the map, itself) we're now 500px to the right, and 3 pixels down from where we started... And only one frame has gone by.
So they restore the map's coordinates to be the same origin as the screen's coordinates (and the rotation to be the same, and the scale, and the skew, etc...), before setting the new one.
And as you might guess, by looking at it, now, you can see that the text should actually move top to bottom. Not right to left, like the page says...
Why do this?
Why go to the trouble of changing the coordinate-system of the drawing-context, when the draw commands give you an x and y right there in the function?
If you want to draw a picture on the canvas, and you know how high and wide it is, and where you'd like the top-left corner to be, why can't you just do this:
easel.drawImage(myImg, x, y, myImg.width, myImg.height);
Well, you can.
You can totally do that. There's nothing stopping you.
In fact, if you wanted to make it zoom around the screen, you could just update the x and y on a timer, and call it a day.
But what about if you were drawing a game character? What if the character had a hat, and had gloved hands, and big boots, and all of those things were drawn separate from the character?
So first you'd say "well, he's standing at x and y in the world, so x plus where his hand is in relation to his body would be x + body.x - hand.x...or was that plus..."
...and now you have draw calls for all of his parts that are all looking like a notebook full of Grade 5 math homework.
Instead, you can say: "He's here. Set my coordinates so that 0,0 is right in the middle of my guy". Now your draw calls are as simple as "My right hand is 6 pixels to the right of the body, my left hand is 3 pixels to the left".
And when you're done drawing your character, you can set your origin back to 0,0 and then the next character can be drawn. Or, if you want to attempt it, you can then translate from there to the origin of the next character, based on the delta from one to the other (this will save you a function call per translation). And then, if you only saved state once the whole time (the original state), at the end, you can return to 0,0 by calling .restore.
The context save() saves stuff like transformation color among other stuff. Then you can change the context and restore it to have the same as when you saved it. It works like a stack so you can push multiple canvas states onto the stack and recover them.
http://html5.litten.com/understanding-save-and-restore-for-the-canvas-context/

scaling object to match field of view

I am overlaying some clickable hotspots on top of a proprietary panorama viewer application in flash (as3), and I need to make sure that the hotspots scale according to the changing field of view as the user zooms in / zooms out, but I'm not sure what formula to use.
I set a maximum and minimum field of view of 90 and 25, respectively. I've been given some suggestions of how to calculate the scale of the icons:
from the maker of the panorama software:
Scale => 1/tan(FoV)
This doesn't seem to work for me. And:
scalar += (ZOOM_SCALE_UPPER - ZOOM_SCALE_LOWER) * ( ZOOM_LIMIT_OUT - tempFOV )/( ZOOM_LIMIT_OUT-ZOOM_LIMIT_IN) ;
hotspot.scaleX = hotspot.scaleY = scalar;
Gets me close, but at some point the hotspot stops scaling even though the panorama continues to scale. I thought I could just do something like:
diffFOV = previousFOV - currentFOV.
hotspot.scale = currentScale*(1-diffFov)
But that's not quite right either. Everything gets way too big or too small.
Any ideas?
You may be over thinking it.
//assume we change the scale
var NEW_SCALE:Number = currentScale*(1-(previousFOV-currentFOV));
//1. change the scale of the parent containing both the view and the hotspots
viewSprite.scale = NEW_SCALE;
//this way the hotspot and the panorama will scale together
//2. if they are not in the same parent... then set them both to the same view
hotspot.scale = panorama.scale;
Only thing you may have to do after is reposition if they are not registered on their center point.

Finding a free area in the stage

I'm drawing rectangles at random positions on the stage, and I don't want them to overlap.
So for each rectangle, I need to find a blank area to place it.
I've thought about trying a random position, verify if it is free with
private function containsRect(r:Rectangle):Boolean {
var free:Boolean = true;
for (var i:int = 0; i < numChildren; i++)
free &&= getChildAt(i).getBounds(this).containsRect(r);
return free;
}
and in case it returns false, to try with another random position.
The problem is that if there is no free space, I'll be stuck trying random positions forever.
There is an elegant solution to this?
Let S be the area of the stage. Let A be the area of the smallest rectangle we want to draw. Let N = S/A
One possible deterministic approach:
When you draw a rectangle on an empty stage, this divides the stage into at most 4 regions that can fit your next rectangle. When you draw your next rectangle, one or two regions are split into at most 4 sub-regions (each) that can fit a rectangle, etc. You will never create more than N regions, where S is the area of your stage, and A is the area of your smallest rectangle. Keep a list of regions (unsorted is fine), each represented by its four corner points, and each labeled with its area, and use weighted-by-area reservoir sampling with a reservoir size of 1 to select a region with probability proportional to its area in at most one pass through the list. Then place a rectangle at a random location in that region. (Select a random point from bottom left portion of the region that allows you to draw a rectangle with that point as its bottom left corner without hitting the top or right wall.)
If you are not starting from a blank stage then just build your list of available regions in O(N) (by re-drawing all the existing rectangles on a blank stage in any order, for example) before searching for your first point to draw a new rectangle.
Note: You can change your reservoir size to k to select the next k rectangles all in one step.
Note 2: You could alternatively store available regions in a tree with each edge weight equaling the sum of areas of the regions in the sub-tree over the area of the stage. Then to select a region in O(logN) we recursively select the root with probability area of root region / S, or each subtree with probability edge weight / S. Updating weights when re-balancing the tree will be annoying, though.
Runtime: O(N)
Space: O(N)
One possible randomized approach:
Select a point at random on the stage. If you can draw one or more rectangles that contain the point (not just one that has the point as its bottom left corner), then return a randomly positioned rectangle that contains the point. It is possible to position the rectangle without bias with some subtleties, but I will leave this to you.
At worst there is one space exactly big enough for our rectangle and the rest of the stage is filled. So this approach succeeds with probability > 1/N, or fails with probability < 1-1/N. Repeat N times. We now fail with probability < (1-1/N)^N < 1/e. By fail we mean that there is a space for our rectangle, but we did not find it. By succeed we mean we found a space if one existed. To achieve a reasonable probability of success we repeat either Nlog(N) times for 1/N probability of failure, or N² times for 1/e^N probability of failure.
Summary: Try random points until we find a space, stopping after NlogN (or N²) tries, in which case we can be confident that no space exists.
Runtime: O(NlogN) for high probability of success, O(N²) for very high probability of success
Space: O(1)
You can simplify things with a transformation. If you're looking for a valid place to put your LxH rectangle, you can instead grow all of the previous rectangles L units to the right, and H units down, and then search for a single point that doesn't intersect any of those. This point will be the lower-right corner of a valid place to put your new rectangle.
Next apply a scan-line sweep algorithm to find areas not covered by any rectangle. If you want a uniform distribution, you should choose a random y-coordinate (assuming you sweep down) weighted by free area distribution. Then choose a random x-coordinate uniformly from the open segments in the scan line you've selected.
I'm not sure how elegant this would be, but you could set up a maximum number of attempts. Maybe 100?
Sure you might still have some space available, but you could trigger the "finish" event anyway. It would be like when tween libraries snap an object to the destination point just because it's "close enough".
HTH
One possible check you could make to determine if there was enough space, would be to check how much area the current set of rectangels are taking up. If the amount of area left over is less than the area of the new rectangle then you can immediately give up and bail out. I don't know what information you have available to you, or whether the rectangles are being laid down in a regular pattern but if so you may be able to vary the check to see if there is obviously not enough space available.
This may not be the most appropriate method for you, but it was the first thing that popped into my head!
Assuming you define the dimensions of the rectangle before trying to draw it, I think something like this might work:
Establish a grid of possible centre points across the stage for the candidate rectangle. So for a 6x4 rectangle your first point would be at (3, 2), then (3 + 6 * x, 2 + 4 * y). If you can draw a rectangle between the four adjacent points then a possible space exists.
for (x = 0, x < stage.size / rect.width - 1, x++)
for (y = 0, y < stage.size / rect.height - 1, y++)
if can_draw_rectangle_at([x,y], [x+rect.width, y+rect.height])
return true;
This doesn't tell you where you can draw it (although it should be possible to build a list of the possible drawing areas), just that you can.
I think that the only efficient way to do this with what you have is to maintain a 2D boolean array of open locations. Have the array of sufficient size such that the drawing positions still appear random.
When you draw a new rectangle, zero out the corresponding rectangular piece of the array. Then checking for a free area is constant^H^H^H^H^H^H^H time. Oops, that means a lookup is O(nm) time, where n is the length, m is the width. There must be a range based solution, argh.
Edit2: Apparently the answer is here but in my opinion this might be a bit much to implement on Actionscript, especially if you are not keen on the geometry.
Here's the algorithm I'd use
Put down N number of random points, where N is the number of rectangles you want
iteratively increase the dimensions of rectangles created at each point N until they touch another rectangle.
You can constrain the way that the initial points are put down if you want to have a minimum allowable rectangle size.
If you want all the space covered with rectangles, you can then incrementally add random points to the remaining "free" space until there is no area left uncovered.