Drag

JavaScript

import GboxSDK from "gbox-sdk";

const gboxSDK = new GboxSDK({
  apiKey: process.env["GBOX_API_KEY"] // This is the default and can be omitted
});

async function main() {
  const box = await gboxSDK.create({ type: "android" });

  await box.action.drag({
    start: { x: 100, y: 150 },
    end: { x: 200, y: 200 },
    duration: "50ms"
  });

  // Natural language start/end 
  await box.action.drag({
    start: "Chrome App",
    end: "Trash",
    duration: "800ms"
  });

  await box.action.drag({
    path: [
      { x: 100, y: 150 },
      { x: 200, y: 200 },
      { x: 300, y: 250 }
    ],
    duration: "50ms"
  });
}

main();

{
  "message": "Action executed successfully",
  "actionId": "c9bdc193-b54b-4ddb-a035-5ac0c598d32d",
  "actual": {
    "start": {
      "x": 100,
      "y": 150
    },
    "end": {
      "x": 400,
      "y": 300
    },
    "duration": "500ms"
  },
  "screenshot": {
    "trace": {
      "uri": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUA..."
    },
    "before": {
      "uri": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUA..."
    },
    "after": {
      "uri": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUA..."
    }
  }
}

POST

boxes

{boxId}

actions

drag

JavaScript

import GboxSDK from "gbox-sdk";

const gboxSDK = new GboxSDK({
  apiKey: process.env["GBOX_API_KEY"] // This is the default and can be omitted
});

async function main() {
  const box = await gboxSDK.create({ type: "android" });

  await box.action.drag({
    start: { x: 100, y: 150 },
    end: { x: 200, y: 200 },
    duration: "50ms"
  });

  // Natural language start/end 
  await box.action.drag({
    start: "Chrome App",
    end: "Trash",
    duration: "800ms"
  });

  await box.action.drag({
    path: [
      { x: 100, y: 150 },
      { x: 200, y: 200 },
      { x: 300, y: 250 }
    ],
    duration: "50ms"
  });
}

main();

{
  "message": "Action executed successfully",
  "actionId": "c9bdc193-b54b-4ddb-a035-5ac0c598d32d",
  "actual": {
    "start": {
      "x": 100,
      "y": 150
    },
    "end": {
      "x": 400,
      "y": 300
    },
    "duration": "500ms"
  },
  "screenshot": {
    "trace": {
      "uri": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUA..."
    },
    "before": {
      "uri": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUA..."
    },
    "after": {
      "uri": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUA..."
    }
  }
}

Authorizations

Authorization

string

header

required

Enter your API Key in the format: Bearer . Get it from https://gbox.ai

Path Parameters

boxId

string

required

Box ID

Example:

"c9bdc193-b54b-4ddb-a035-5ac0c598d32d"

Body

application/json

Drag Simple
Drag Advanced

Drag action configuration with start and end points.

Operation flow:

Touch finger at "start" coordinates
Move to "end" coordinates within the "duration" time and lift finger

start

required

Start point of the drag path (coordinates or natural language)

Show child attributes

start.x

number

required

X coordinate of a point in the drag path

Example:

200

start.y

number

required

Y coordinate of a point in the drag path

Example:

300

Example:

{ "x": 100, "y": 150 }

end

required

End point of the drag path (coordinates or natural language)

Show child attributes

end.x

number

required

X coordinate of a point in the drag path

Example:

200

end.y

number

required

Y coordinate of a point in the drag path

Example:

300

Example:

{ "x": 400, "y": 300 }

duration

string

default:500ms

Duration to complete the movement from start to end coordinates

Supported time units: ms (milliseconds), s (seconds), m (minutes), h (hours) Example formats: "500ms", "30s", "5m", "1h" Default: 500ms

Example:

"500ms"

options

Action Common Options · object

Action options. When options.screenshot is provided, ALL deprecated screenshot fields (outputFormat, presignedExpiresIn, screenshotDelay, screenshotRange, includeScreenshot) will be completely ignored.

Show child attributes

options.screenshot

Screenshot options. Can be a boolean to enable/disable screenshots, or an object to configure screenshot options.

Show child attributes

options.screenshot.outputFormat

enum<string>

default:base64

Type of the URI. default is base64.

Available options:

base64,

storageKey

Example:

"base64"

options.screenshot.presignedExpiresIn

string

default:30m

Presigned url expires in. Only takes effect when outputFormat is storageKey.

Supported time units: ms (milliseconds), s (seconds), m (minutes), h (hours) Example formats: "500ms", "30s", "5m", "1h" Default: 30m

Example:

"30m"

options.screenshot.delay

string

default:500ms

Delay after performing the action, before taking the final screenshot.

Execution flow:

Take screenshot before action
Perform the action
Wait for screenshotDelay (this parameter)
Take screenshot after action

Example: '500ms' means wait 500ms after the action before capturing the final screenshot.

Supported time units: ms (milliseconds), s (seconds), m (minutes), h (hours) Example formats: "500ms", "30s", "5m", "1h" Default: 500ms Maximum allowed: 30s

Example:

"500ms"

options.screenshot.phases

enum<string>[]

Specify which screenshot phases to capture.

Available options:

before: Screenshot before the action
after: Screenshot after the action
trace: Screenshot with operation trace

Default captures all three phases. Can specify one or multiple in an array. If empty array is provided, no screenshots will be taken.

Available options:

before,

after,

trace

Example:

["before", "after"]

Example:

{
  "outputFormat": "base64",
  "presignedExpiresIn": "30m",
  "delay": "500ms",
  "phases": ["before", "after"]
}

options.model

enum<string>

default:gelato

Model to use for natural-language target resolution. Defaults to 'uitars'.

Available options:

gpt-5,

gpt-4o,

gelato,

ui-tars,

openai-computer-use

Example:

{
  "screenshot": {
    "outputFormat": "base64",
    "presignedExpiresIn": "30m",
    "delay": "500ms",
    "phases": ["before", "after"]
  }
}

Response

200 - application/json

Drag action result with actual parameters used

Result of drag action execution with actual parameters used

message

string

required

message

Example:

"Action executed successfully"

actionId

string

required

Unique identifier for each action. Use this ID to locate the action and report issues.

Example:

"c9bdc193-b54b-4ddb-a035-5ac0c598d32d"

actual

Drag Action Actual Parameters · object

required

Actual parameters used when executing the drag action

Show child attributes

actual.start

Drag Path Point · object

required

Start point of the drag

Show child attributes

actual.start.x

number

required

X coordinate of a point in the drag path

Example:

200

actual.start.y

number

required

Y coordinate of a point in the drag path

Example:

300

Example:

{ "x": 100, "y": 150 }

actual.end

Drag Path Point · object

required

End point of the drag

Show child attributes

actual.end.x

number

required

X coordinate of a point in the drag path

Example:

200

actual.end.y

number

required

Y coordinate of a point in the drag path

Example:

300

Example:

{ "x": 400, "y": 300 }

actual.duration

string

default:500ms

required

Duration of the drag

Supported time units: ms (milliseconds), s (seconds), m (minutes), h (hours) Example formats: "500ms", "30s", "5m", "1h" Default: 500ms

Example:

"500ms"

Example:

{
  "start": { "x": 100, "y": 150 },
  "end": { "x": 400, "y": 300 },
  "duration": "500ms"
}

screenshot

Action Result Screenshot · object

Optional screenshot data. Only present when screenshots are requested via options.screenshot.phases or deprecated fields

Show child attributes

screenshot.trace

Action Result Screenshot Operation Trace · object

URI of the screenshot before the action with operation trace

Show child attributes

screenshot.trace.uri

string

required

URI of the screenshot with operation trace

Example:

"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUA..."

Example:

{
  "uri": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUA..."
}

screenshot.before

Action Result Screenshot Before · object

URI of the screenshot before the action

Show child attributes

screenshot.before.uri

string

required

URI of the screenshot before the action

Example:

"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUA..."

screenshot.before.presignedUrl

string

Presigned url of the screenshot before the action

Example:

"https://example.com/xxxxx/xxxxx/xxxxx"

Example:

{
  "uri": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUA..."
}

screenshot.after

Action Result Screenshot After · object

URI of the screenshot after the action

Show child attributes

screenshot.after.uri

string

required

URI of the screenshot after the action

Example:

"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUA..."

screenshot.after.presignedUrl

string

Presigned url of the screenshot before the action

Example:

"https://example.com/xxxxx/xxxxx/xxxxx"

Example:

{
  "uri": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUA..."
}

Example:

{
  "trace": {
    "uri": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUA..."
  },
  "before": {
    "uri": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUA..."
  },
  "after": {
    "uri": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUA..."
  }
}

Touch Scroll

⌘I

Model

Box

UI Action

Run Code

Command

File System

Browser

Android

Media

Authorizations

Path Parameters

Body

Response