Scrcpy Gotchas

scrcpy-mcp Gotchas

Coordinate systems: always use native coordinates

scrcpy_tap, scrcpy_swipe, scrcpy_long_press, and scrcpy_scroll take device-native coordinates — the same space that scrcpy_ui_dump / scrcpy_ui_find_element report in their bounds and tapX/tapY, and the same space adb shell input tap uses. So the reliable workflow needs no scaling math:

// Find the element (returns native tapX/tapY), then tap them straight through
const el = scrcpy_ui_find_element({ text: "<label>" })  // → { tapX, tapY }
scrcpy_tap({ x: el.tapX, y: el.tapY })

The input tools scale native → video-frame coordinates internally before sending the touch to the scrcpy server, so you never compute a scale factor yourself.

Don't tap screenshot pixels

scrcpy_screenshot returns a downscaled image (e.g. 576×1024 at maxSize=1024), not the native frame. Tapping a pixel coordinate read off the screenshot lands in the wrong place. Always source coordinates from ui_dump / ui_find_element (native), never from the screenshot.

Why internal scaling is required: the scrcpy server's touch protocol is not resolution-independent — the touch message's screen size must exactly match the encoder's video-frame size, or the server silently drops the event. The tools handle this conversion; you just supply native coords.

ui_find_element matching rules

The four search criteria have different matching behaviors:

All criteria are combined with AND logic — an element must satisfy every supplied criterion.

// text: case-insensitive substring — "bater" matches "Batería"
scrcpy_ui_find_element({ text: "bater" })  // ✓

// resourceId: exact match only — must be full string
scrcpy_ui_find_element({ resourceId: "android:id/title" })  // must match exactly

// className: full qualified name — "Button" won't match "android.widget.Button"
scrcpy_ui_find_element({ className: "android.widget.Button" })  // full name required

Accent caveat

text and contentDesc matching lowercases both sides and uses String.includes(). However, Unicode normalization differences between the XML dump (from uiautomator) and your query string can cause mismatches. Searching for an accented word (e.g. "Batería") may return 0 even though the element displays that exact text, while a partial ASCII-only substring (e.g. "bater") will find it.

Rule of thumb: Search with lowercase ASCII-only substrings when possible. If that fails, use ui_dump and check the raw XML for the actual byte sequence.

Search list/feed items by contentDesc, not text

For tappable cards — search results, feed rows, media thumbnails — the visible label is often rendered by a child view that the clickable wrapper ViewGroup does not expose as text. A text: query then returns 0 matches (or only matches an unrelated element like the search bar).

The label instead lives in the wrapper's contentDesc, which is usually richer (e.g. it bundles secondary metadata into one string) and easier to disambiguate between similar items:

// May return 0 — the result card has no `text`
scrcpy_ui_find_element({ text: "<title>" })

// Matches the clickable wrapper's contentDesc, which carries the label
scrcpy_ui_find_element({ contentDesc: "<title>" })
// → { contentDesc: "...", tapX, tapY, clickable: true }

Rule of thumb: if text: finds nothing for something you can clearly see on screen, retry with contentDesc: (or fall back to scrcpy_ui_dump and grep the XML) before assuming it isn't there.

contentDesc quoting can break parsing

uiautomator dump wraps attribute values in double quotes (") by default. But when a value itself contains double quotes (common in X/Twitter content-desc fields), it switches to single quotes ('). The parseUiNodes() parser only handles double-quote delimiters:

HTML attr regex: content-desc="([^"]*)"
                            ↑ only matches "..."

So content-desc='tweet with "quoted" text' → contentDesc: "" (empty string — parser misses it entirely).

Impact: ui_find_element won't find tweets/items whose contentDesc contains double quotes. The element is still in the dump but its contentDesc is silently empty.

Workaround: for text that reliably exists, use text: queries instead. For X timeline tweets, search by partial lowercase ASCII snippets that avoid quoted substrings.

// This tweet: "a company just fired 90% of their dev team because claude code..."
// Won't match — contentDesc parsing fails due to embedded double quotes
scrcpy_ui_find_element({ contentDesc: "claude" })  // → 0 results

// But this promoted tweet has no embedded quotes → contentDesc parses fine
scrcpy_ui_find_element({ contentDesc: "runpod" })  // → found

Non-ASCII text input (tildes, ¡, ¿, ñ...)

ADB's input text (used by scrcpy_input_text) crashes on non-ASCII characters like ¡, ¿, ñ, accented vowels (é, á, etc.), and !. The command throws a NullPointerException and silently fails.

Workaround: Use scrcpy_clipboard_set with paste: true instead — it handles any Unicode text:

// BAD — crashes with special characters
scrcpy_input_text({ text: "¡Hola, mamá!" })

// GOOD — clipboard handles all characters
scrcpy_clipboard_set({ text: "¡Hola, mamá!", paste: true })

Critical: paste: true requires an active scrcpy session. Without one, the clipboard is set but the paste action is silently skipped (message: "Paste action not performed"). Always scrcpy_start_session before sending non-ASCII text.

Exclamation mark (!) causes partial text loss

Even though scrcpy_input_text returns success, using ! in the string can cause partial text loss — leading characters may be dropped. For example, "Hello world!" may type as "llo world!" (the ! causes the ADB input text command to partially fail). The clipboard workaround above avoids this completely.

Start the scrcpy session early

Start a scrcpy session proactively when doing multi-step interactions. It unlocks:

• Fast clipboard paste (scrcpy_clipboard_set with paste: true)
• Fast screenshots (10-50× faster than ADB)
• Fast input (tap, swipe, etc.)

Without a session, clipboard paste silently degrades and every screenshot is a slow ADB call.

Session persists across screen on/off

Input (tap, swipe) and screenshots continue to work even when the screen is turned off via scrcpy_screen_off. Screenshots taken while screen is off will show a black/dark frame. Use scrcpy_screen_on to wake the device when needed.

clipboard_get may be unreliable

scrcpy_clipboard_get can fail even with an active scrcpy session, especially on Android 10 devices. If it returns an error, the clipboard content is still likely not critical — rely on direct state checks via ui_dump instead.

Scroll vs Swipe for scrolling content

scrcpy_scroll does NOT work reliably on RecyclerView or ScrollView containers. Many Android apps (Settings, Chrome, launchers, messaging apps) use RecyclerView for scrollable lists. scrcpy_scroll may return success but produce no visible movement.

Use scrcpy_swipe instead for scrolling:

// BAD — may have no effect on RecyclerView
scrcpy_scroll({ x: 720, y: 1500, dx: 0, dy: 1 })

// GOOD — swipe gestures reliably scroll all scrollable views
scrcpy_swipe({ x1: 720, y1: 1800, x2: 720, y2: 800, duration: 500 })

For a swipe-down (content scrolls up): y1 > y2. For a swipe-up (content scrolls down): y1 < y2. Duration 300-500ms is a good default for natural scrolling.

Watch for momentum side effects: Long swipes can carry momentum that collapses toolbars, scrolls beyond intended position, or triggers overscroll actions. For precise scrolling, use shorter swipe distances (e.g., 200-400px delta).

Send button tap vs. Enter key

When sending messages in WhatsApp (and likely other messaging apps), scrcpy_tap on the send button can silently miss — the hit target is small and easy to land just outside. Prefer scrcpy_key_event with ENTER instead, which is more reliable.

Verified key events

These keycodes have been confirmed working via scrcpy_key_event:

Use scrcpy_key_event with the string name (e.g., "HOME") or numeric keycode.

app_start reuses existing tasks

scrcpy_app_start brings an app to the foreground but reuses its existing task, preserving scroll position, UI state, and back stack. This is efficient for quick app switching but can be surprising:

// Opens Settings but preserves scroll position from previous session
scrcpy_app_start({ packageName: "com.android.settings" })

To force a clean launch, prefix the package name with +:

scrcpy_app_start({ packageName: "+com.android.settings" })

This force-stops the app before launching, clearing its state.

Tool context

Use only scrcpy_* tools when controlling an Android device. Do not reach for browser tools (browser_*, chrome-devtools_*) in an Android context. Common mistake: using browser_wait instead of just proceeding to the next action.

Locale-independent targeting

Visible text labels (in text, contentDesc, hint) are locale-dependent. What reads "Chats" in English is "Chats" in Spanish, "Discussions" in French, etc. Hardcoding text strings makes the skill fragile across devices.

Always prefer locale-independent selectors first:

Workflow for text-based targeting:

1. Use scrcpy_ui_dump to discover the actual label on the device
2. Search with partial, lowercase substrings — avoids case/accent/unicode issues
3. Fall back to resourceId + className when possible

In this document, example labels are shown as generic patterns like <app-name> or with English/Spanish examples marked (locale-dependent). Treat them as illustrations of the type of text to expect, not exact strings to copy.

X (Twitter) specifics

• Package name: com.twitter.android
• Timeline is a RecyclerView (android:id/list) inside nested ViewPager + ScrollView. Scroll with scrcpy_swipe.
• Timeline tweets store all content in contentDesc — the row's text field is always empty. The contentDesc bundles author, handle, tweet body, engagement stats, and timestamp into one long string.
• Ad tweets with external links open in Chrome Custom Tabs, not in X's own browser. Use BACK to return to X.
• Bottom navigation tabs (5 tabs: home, search, grok, notifications, messages) are android.widget.FrameLayout elements with contentDesc labels (locale-dependent). Active tab has selected="true". Discover actual labels via ui_dump.
• Top timeline tabs (e.g., "For You" / "Following") use contentDesc on the parent LinearLayout; active tab has selected="true".
• Long swipes collapse the toolbar — toolbar and tab bar scroll away with the ViewPager's nested scroll. BACK restores them.
• The new-posts banner appears when fresh content loads while scrolled. It has resourceId="com.twitter.android:id/banner_text". Tapping it scrolls to top.
• Search/Explore tab has a HorizontalScrollView sub-tab bar (tab_layout). Tab labels are locale-dependent; discover via ui_dump. Trending topic titles use text (not contentDesc).
• Ad badge has resourceId="com.twitter.android:id/tweet_ad_badge_top_right" with a short text label (locale-dependent, e.g. "Ad" / "Anuncio").

Chrome/web browsing

• Package name: com.android.chrome
• Web content IS accessible via ui_dump when pages have proper accessibility markup (semantic HTML, ARIA). Plain pages like google.com may not expose content.
• The WebView (android.webkit.WebView) is scrollable — use scrcpy_swipe on the web content area (below the toolbar, y > 280).
• URL bar is an EditText at com.android.chrome:id/url_bar. Tap it, paste a URL via clipboard_set, and press ENTER to navigate.
• Off-screen web elements show as bounds="[0,0][0,0]" or zero-dimension bounds. Elements scrolled past the top show bounds="[x,top][x,top]" (height=0). Filter these out: only tap elements with positive height.
• Chrome Custom Tabs (opened from other apps' links) have a simpler toolbar with close, share, and minimize buttons. Use BACK to dismiss them.
• Web pages with complex a11y trees produce enormous XML dumps (30KB+). ui_find_element with targeted criteria is far more efficient than parsing the full ui_dump response.