Monday, 27 January 2014

Simulation of Injecting Multi Touch Event in Android

Sample Code which explains how a Multi Touch Event could be dispatched if the Device has root permissions.

the PointerCoords[], we have to set the pressure and size to 1, the default values are 0.
For
public static MotionEvent obtain (long downTime, long eventTime, int action,
         float x, float y, int metaState)

     Create a new MotionEvent, filling in a subset of the basic motion values. 
    Those not specified here are: device id (always 0), pressure and size (always 1), 
x and y precision (always 1), and edgeFlags (always 0).

since the default pressure and size are 1, so we don't need to set them.
Tips for creating gestures:
1. following the real gesture sequence, since we want to simulate the real gestures
  • override the onTouchEvent() to check the real events received by application. These events can also be used for comparison of real user touch events and generated touch events Take browser for example:
    a) @Override public boolean onTouchEvent(MotionEvent event) { Log.i("WebView", event.toString() + event.getAction()); boolean rt = super.onTouchEvent(event); return rt; }
  • manually touch screen to get the real gesture sequence from onTouchEvent() in a). We can follow the event sequence when generating events. -- If we don’t follow the gesture event sequence, the instrumented events may be rejected.
  • Here is a valid event sequence of zoom gesture, (the downTime is the same for all the events)
    i. ACTION_DOWN of one start point
    ii. ACTION_POINTER_2_DOWN of two start points
    iii.ACTION_MOVE of two middle points
    iv. ACTION_POINTER_2_UP of two end points
    v. ACTION_UP of one end point
2. use the API MotionEvent.obtain correctly
  • There are two most used obtain() API.
public static MotionEvent obtain (long downTime, long eventTime, int action, float x, float y, int metaState)
AND
public static MotionEvent obtain(long, long, int, int, android.view.MotionEvent.PointerProperties[], android.view.MotionEvent.PointerCoords[], int, int, float, float, int, int, int, int)
The first one is usually used for single point gestures, like fling, scroll, click etc. The parameters (pressure, size, xPresion, yPresion) for this function are all set to 1.
And the second one is a more general one, and can be used for multi-touch events generation. While for the second one, we have to set the pressure, size in pointerCoords of each touch point to 1.
Here is the example to generate the zoom gesture:
public static void generateZoomGesture(Instrumentation inst,
        long startTime, boolean ifMove, GestureInfo.Point startPoint1,
        GestureInfo.Point startPoint2, GestureInfo.Point endPoint1,
        GestureInfo.Point endPoint2, int duration) {

    if (inst == null || startPoint1 == null
            || (ifMove && endPoint1 == null)) {
        return;
    }

    long eventTime = startTime;
    long downTime = startTime;
    MotionEvent event;
    float eventX1, eventY1, eventX2, eventY2;

    eventX1 = startPoint1.x;
    eventY1 = startPoint1.y;
    eventX2 = startPoint2.x;
    eventY2 = startPoint2.y;

    // specify the property for the two touch points
    PointerProperties[] properties = new PointerProperties[2];
    PointerProperties pp1 = new PointerProperties();
    pp1.id = 0;
    pp1.toolType = MotionEvent.TOOL_TYPE_FINGER;
    PointerProperties pp2 = new PointerProperties();
    pp2.id = 1;
    pp2.toolType = MotionEvent.TOOL_TYPE_FINGER;

    properties[0] = pp1;
    properties[1] = pp2;

    //specify the coordinations of the two touch points
    //NOTE: you MUST set the pressure and size value, or it doesn't work
    PointerCoords[] pointerCoords = new PointerCoords[2];
    PointerCoords pc1 = new PointerCoords();
    pc1.x = eventX1;
    pc1.y = eventY1;
    pc1.pressure = 1;
    pc1.size = 1;
    PointerCoords pc2 = new PointerCoords();
    pc2.x = eventX2;
    pc2.y = eventY2;
    pc2.pressure = 1;
    pc2.size = 1;
    pointerCoords[0] = pc1;
    pointerCoords[1] = pc2;

    //////////////////////////////////////////////////////////////
    // events sequence of zoom gesture
    // 1. send ACTION_DOWN event of one start point
    // 2. send ACTION_POINTER_2_DOWN of two start points
    // 3. send ACTION_MOVE of two middle points
    // 4. repeat step 3 with updated middle points (x,y),
    //      until reach the end points
    // 5. send ACTION_POINTER_2_UP of two end points
    // 6. send ACTION_UP of one end point
    //////////////////////////////////////////////////////////////

    // step 1
    event = MotionEvent.obtain(downTime, eventTime, 
                MotionEvent.ACTION_DOWN, 1, properties, 
                pointerCoords, 0,  0, 1, 1, 0, 0, 0, 0 );

    inst.sendPointerSync(event);

    //step 2
    event = MotionEvent.obtain(downTime, eventTime, 
                MotionEvent.ACTION_POINTER_2_DOWN, 2, 
                properties, pointerCoords, 0, 0, 1, 1, 0, 0, 0, 0);

    inst.sendPointerSync(event);

    //step 3, 4
    if (ifMove) {
        int moveEventNumber = 1;
        moveEventNumber = duration / EVENT_MIN_INTERVAL;

        float stepX1, stepY1, stepX2, stepY2;

        stepX1 = (endPoint1.x - startPoint1.x) / moveEventNumber;
        stepY1 = (endPoint1.y - startPoint1.y) / moveEventNumber;
        stepX2 = (endPoint2.x - startPoint2.x) / moveEventNumber;
        stepY2 = (endPoint2.y - startPoint2.y) / moveEventNumber;

        for (int i = 0; i < moveEventNumber; i++) {
            // update the move events
            eventTime += EVENT_MIN_INTERVAL;
            eventX1 += stepX1;
            eventY1 += stepY1;
            eventX2 += stepX2;
            eventY2 += stepY2;

            pc1.x = eventX1;
            pc1.y = eventY1;
            pc2.x = eventX2;
            pc2.y = eventY2;

            pointerCoords[0] = pc1;
            pointerCoords[1] = pc2;

            event = MotionEvent.obtain(downTime, eventTime,
                        MotionEvent.ACTION_MOVE, 2, properties, 
                        pointerCoords, 0, 0, 1, 1, 0, 0, 0, 0);

            inst.sendPointerSync(event);
        }
    }

    //step 5
    pc1.x = endPoint1.x;
    pc1.y = endPoint1.y;
    pc2.x = endPoint2.x;
    pc2.y = endPoint2.y;
    pointerCoords[0] = pc1;
    pointerCoords[1] = pc2;

    eventTime += EVENT_MIN_INTERVAL;
    event = MotionEvent.obtain(downTime, eventTime,
                MotionEvent.ACTION_POINTER_2_UP, 2, properties, 
                pointerCoords, 0, 0, 1, 1, 0, 0, 0, 0);
    inst.sendPointerSync(event);

    // step 6
    eventTime += EVENT_MIN_INTERVAL;
    event = MotionEvent.obtain(downTime, eventTime, 
                MotionEvent.ACTION_UP, 1, properties, 
                pointerCoords, 0, 0, 1, 1, 0, 0, 0, 0 );
    inst.sendPointerSync(event);
}

8 comments:

  1. Hi thanks a lot for sharing this article. I just wanted to know what is the value of duration and event_min_interval here ?

    ReplyDelete
  2. The above two variables are solely used for simulation purpose.
    1) EVENT_MIN_INTERVAL is used to set the next motion event's down time.
    you can see below that everytime a MotionEvent is deployed a variable named "eventTime is incremented". So that variable is used as the next MotionEvent's down time.

    2) duration - Since the above code was written for simulation purpose it is used as a factor to decide if there is ACTION_MOVE event available.

    if you go through the code again you will be able to understand much more better.

    Anyways the sole purpose of the above code was to make readers understand how 2 finger events are dispatched.

    If you still feel the explanation is not enough do bug me.

    ReplyDelete
  3. I have such requirement in a project on which i am working currently, can you please share a working demo of this which will be really helpful for me and other people on web and which can save days for people like us. any help would be greatly appreciated.

    ReplyDelete
  4. When I worked for VNC Client and Server code for automobile industry I just tried the above code to test if the MotionEvent is getting deployed appropriately.
    I am not supposed to share the working code here except the sample code above.

    ReplyDelete
  5. Hi Nandhan, thanks for posting this article. Can this simulation inject the events outside the context of the app, i.e., in any other application if we use a background service? You stated that it needs a rooted device. My device is rooted, and I have used Motionevent + instrumentation as you did, but I used the standard motionevent for single touch, not the one you used. When I try to inject outside the app I get security exception, can you please tell me how you solved this problem? Thanks.

    ReplyDelete
  6. Well, it was 2 years before, I don't remember what was done.
    Am Sorry.

    ReplyDelete
  7. Thanks for sharing this! I work on an automation tool called Appium (Selenium for mobile apps) and this article really helped me understand how to automate multi pointer actions.

    ReplyDelete