64 lines
3.9 KiB
Plaintext
64 lines
3.9 KiB
Plaintext
|
name: get_window_state
|
||
|
|
|
||
|
|
description:
|
||
|
|
Walk a running app's AT-SPI tree and return BOTH a structured `elements` array (preferred) AND a Markdown rendering of the same tree (back-compat). Every actionable element is tagged with [element_index N] in the markdown and as `element_index` in the structured array.
|
||
|
|
|
||
|
|
PREFERRED CONSUMERS read `structuredContent.elements` (one entry per indexed row with `element_index`, `role`, `label`, `value`, `enabled`, `selected`, `frame: {x,y,w,h}` when AT-SPI reports usable bounds, `parent_index`, `depth`). The markdown `tree_markdown` stays available and unchanged in shape for existing text-parsing callers — but new fields will only be added to the structured side. Set `query` to project BOTH representations to matching rows plus their ancestor chain while preserving original indices. `total_element_count` reports the complete snapshot and `returned_element_count` reports the projection.
|
||
|
|
|
||
|
|
Always returns BOTH the element tree AND a screenshot — ground on both and cross-check (the tree lies on some surfaces). Choose the modality at ACTION time: an element ax action (element_index/element_token → accessibility rung) or an element px action (x,y → pixel rung off this screenshot). capture_mode is deprecated and ignored. On Wayland, where output capture cannot prove the requested surface's identity, the truthful tree is returned without a screenshot and `screenshot_error.code` is `surface_identity_unproven`.
|
||
|
|
|
||
|
|
Optional `max_elements` / `max_depth` bound the AT-SPI walk to mitigate context-window blow-up on Electron / large web apps that produce 10k+ element trees. When applied, BOTH the markdown and the structured elements are truncated identically. Omit both for current default behaviour.
|
||
|
|
|
||
|
|
input_schema:
|
||
|
|
{
|
||
|
|
"additionalProperties": false,
|
||
|
|
"properties": {
|
||
|
|
"capture_mode": {
|
||
|
|
"description": "DEPRECATED and ignored. get_window_state always returns BOTH the element tree and a screenshot — ground on both. The modality is chosen at action time by how you address the target: an element ax action (element_index/element_token) or an element px action (x,y). Any value (including the old \"som\"/\"screenshot\" aliases) is accepted but has no effect.",
|
||
|
|
"enum": [
|
||
|
|
"ax",
|
||
|
|
"vision"
|
||
|
|
],
|
||
|
|
"type": "string"
|
||
|
|
},
|
||
|
|
"include_screenshot": {
|
||
|
|
"description": "Default true — returns a grounding screenshot alongside the tree. Set false to skip the grab and return tree only (the cheap path for re-indexing before an element ax action).",
|
||
|
|
"type": "boolean"
|
||
|
|
},
|
||
|
|
"max_depth": {
|
||
|
|
"description": "Cap on the AT-SPI tree walk depth. Omit for the default (uncapped). Lower for deeply nested apps.",
|
||
|
|
"minimum": 1,
|
||
|
|
"type": "integer"
|
||
|
|
},
|
||
|
|
"max_elements": {
|
||
|
|
"description": "Cap on total AT-SPI nodes walked. Omit for the default (5 000). Lower for huge web/Electron trees.",
|
||
|
|
"minimum": 1,
|
||
|
|
"type": "integer"
|
||
|
|
},
|
||
|
|
"pid": {
|
||
|
|
"type": "integer"
|
||
|
|
},
|
||
|
|
"query": {
|
||
|
|
"description": "Optional case-insensitive substring. Projects both tree_markdown and structured elements to matches plus ancestors while preserving original indices. Compare total_element_count with returned_element_count.",
|
||
|
|
"type": "string"
|
||
|
|
},
|
||
|
|
"screenshot_out_file": {
|
||
|
|
"description": "When set, write the PNG to this file path (~ expanded) instead of embedding base64 in the response. The structured output carries screenshot_file_path instead.",
|
||
|
|
"type": "string"
|
||
|
|
},
|
||
|
|
"session": {
|
||
|
|
"description": "For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session.",
|
||
|
|
"type": "string"
|
||
|
|
},
|
||
|
|
"window_id": {
|
||
|
|
"description": "Native window identifier from list_windows.",
|
||
|
|
"type": "integer"
|
||
|
|
}
|
||
|
|
},
|
||
|
|
"required": [
|
||
|
|
"pid",
|
||
|
|
"window_id"
|
||
|
|
],
|
||
|
|
"type": "object"
|
||
|
|
}
|