Hw Mouse/keyboard emulator for intelligent agents

Posted by Giuseppe Marullo on May 29, 2009, 5:20 am

Hi all,
I am investigating the use/creation of a macro recorder / scripting tool
that should be able to act as a human to manage a set of computers for
daily tasks.

So far, there are different sw tools able to record and playback both
keyboard and mouse actions, so nothing new.

I need to use something that will execute mouse and possibly keyboard
actions while doing some processing before, passing then such actions to
the OS.

There are tools able to recognize bitmaps within the screen, so I could
detect if and where a particular button is, get its centroid and use the
X, Y coordinates to accurately use the mouse to click on it.

I would like to automate the part when the program records the macro, so
I would like that when I need to click on a button, I would press a
"This click is supposed to be done over a button, please do a image
capture of it" button on the mouse itself.

The recording software, would pass the click action to the OS, after
having done a screen capture.
After the click "execution", another screen capture would identify the
area that is different and record it, each time I need it.

After that, I could create/refine the session having all the button
bitmaps already taken with the right size, so I could search each bitmap
and correctly position the mouse over it on each step.

I need a mouse emulator that will talk with my program, and my program
will pass the mouse events to the OS but the mouse itself should not be
detected as such to the OS.

The mouse cannot be detected as a real mouse to the OS, or I won't be
able to intercept the clicks before the application receive them, so I
could not perform actions like screen capture.

Important: I need to use the computers through RDP sessions, so pure
graphics handling, 99% of the time, not local windows.

Just to make an example, if I just needed: left and right click, I could
use the middle button for triggering the action needed using a PopUpMenu
with the appropriate "commands", like LeftClickOnAButton vs
SimpleLeftClick, so the program could take appropriate action before
passing the leftclick to the OS.

Before starting reinventing the wheel, I was wondering if such device
already exists, every emulator I found acts like a "normal" mouse or
allows a pc to boot without keyboard.

TIA,

Giuseppe

Appl-to-Be-Automated <- mouse events <-My_macro/script <-NotSoMouse(hw)

Posted by studleylee on May 31, 2009, 6:20 pm

Of interest would be http://www.autohotkey.com/
I use it daily. For quick utility creation it's indispensable!
-Lee

Similar Threads	Posted
CFP: International Conference on Intelligent Agents, Web Technologies and Internet Commerce	March 15, 2005, 12:25 am
CFP: 2006 International Conference on Intelligent Agents, Web Technologies and Internet Commerce	July 13, 2006, 9:21 pm
CFP: International Conference on Intelligent Agents, Web Technologies and Internet Commerce - IAWTIC08	June 3, 2008, 8:47 pm
CFP: IEEE in cooperated International Conference on Intelligent Agents, Web Technologiesand Internet Commerce	June 29, 2005, 10:58 pm
CFP: IEEE in cooperated International Conference on Intelligent Agents, Web Technologies and Internet Commerce	August 11, 2005, 8:20 pm
CFP: IEEE in cooperated International Conference on Intelligent Agents, Web Technologies and Internet Commerce	August 16, 2005, 2:21 am
CFP: IEEE in cooperated International Conference on Intelligent Agents, Web Technologies and Internet Commerce	August 26, 2005, 2:31 am
CFP: IEEE in cooperated International Conference on Intelligent Agents, Web Technologies and Internet Commerce	May 5, 2006, 2:52 am
CFP: IEEE in cooperated International Conference on Intelligent Agents, Web Technologies and Internet Commerce	May 18, 2006, 1:37 am
CFP: IEEE in cooperated International Conference on Intelligent Agents, Web Technologies and Internet Commerce	May 24, 2006, 2:23 am