Image Matcher Integration

Integrate Matcher into image-matching-webui

This skill automates the integration of a new local feature matching method into the image-matching-webui project. Given a GitHub repository URL, it follows the project's established patterns to add the matcher as a fully functional option in the WebUI.

Prerequisites

• The target repository must be a local feature matching method (sparse or standalone)
• You must be working inside the image-matching-webui project root

Step-by-Step Integration Guide

Step 1: Analyze the Target Repository

1. Clone the repo to /tmp/<repo-name> for analysis (do NOT add as submodule yet)
2. Identify the matcher type:
   • Dense/Standalone matcher: Takes raw images as input, performs detect+describe+match internally (e.g., LoMa, RoMa, LoFTR). required_inputs = ["image0", "image1"]
   • Sparse matcher: Takes keypoints+descriptors as input, performs matching only (e.g., LightGlue, SuperGlue). required_inputs includes keypoints0, descriptors0, etc.
3. Find the core model class and its API:
   • How to initialize the model (constructor args, config options)
   • How to run inference (forward method signature)
   • What the model outputs (keypoints, matches, scores, etc.)
   • Model weight download URLs
4. Check dependencies in pyproject.toml or requirements.txt
5. Identify model variants (e.g., different sizes: B/L/G/R)
6. Check for device/amp issues:
   • Does the model use torch.autocast or mixed precision (mp, amp)?
   • Does it manage its own device placement (like LoMa's loma.device)?
   • On MPS/CPU, does it produce dtype mismatches?

Step 2: Add Git Submodule

git submodule add <repo-url> imcui/third_party/<RepoName>

• Use the original repo name (PascalCase) as the submodule directory name
• If the repo has a fork in the project's org (e.g., Vincentqyw/xxx or agipro/xxx), prefer the fork

⚠️ CRITICAL: Never modify third_party code directly

imcui/third_party/ contains pinned third-party submodules — you are NOT the owner of this code. If a dependency needs a compatibility fix (e.g., API changes in PyTorch/kornia, import path changes):

1. Fork the original repo to the agipro GitHub account
2. Apply the fix in the fork and push
3. Replace the submodule URL in the main repo to point to the fork:

   git submodule deinit -f imcui/third_party/<RepoName>
   git rm -f imcui/third_party/<RepoName>
   rm -rf .git/modules/imcui/third_party/<RepoName>
   git submodule add https://github.com/agipro/<RepoName>.git imcui/third_party/<RepoName>

4. Update .gitmodules — the URL must point to the fork

Example: EfficientLoFTR was forked to agipro/EfficientLoFTR to fix a kornia.utils.grid → kornia.utils import change required by kornia 0.8+.

Step 3: Create Matcher Implementation

Create imcui/hloc/matchers/<matcher-name>.py following these rules:

3.1 Import Pattern

import sys
from pathlib import Path

from .. import logger
from ..utils.base_model import BaseModel

# Add third_party to sys.path for import
<matcher>_path = Path(__file__).parent / "../../third_party/<RepoName>/src"
sys.path.append(str(<matcher>_path))

# Import the model class
from <module> import <ModelClass>

• If the third_party repo has src/ as the package root, append <path>/src
• Some repos put the package directly at root level — adjust accordingly

3.2 Class Definition

The class MUST:

• Inherit from BaseModel
• Be the only BaseModel subclass in the file (the dynamic_load function asserts this)
• Define default_conf dict with all configurable parameters
• Define required_inputs list

class MatcherName(BaseModel):
    default_conf = {
        "name": "two_view_pipeline",
        "model_name": "<default_variant>",
        "max_keypoints": 2048,
        # ... other config
    }
    required_inputs = [
        "image0",
        "image1",
        # For sparse matchers, also include:
        # "keypoints0", "scores0", "descriptors0",
        # "keypoints1", "scores1", "descriptors1",
    ]

3.3 _init Method

• Use self.conf (not the conf parameter) to access merged config
• Log model loading with logger.info()
• Handle model weight downloading:
  • If weights are on HuggingFace: use self._download_model(repo_id=MODEL_REPO_ID, filename=...)
  • If weights are auto-downloaded by the model (e.g., torch.hub), let it handle
  • If weights need manual download, document the URL
• Platform compatibility: On non-CUDA devices, disable mixed precision:

  if not torch.cuda.is_available():
      # Disable mp/amp for CPU/MPS compatibility

3.4 _forward Method

For standalone matchers (input: raw images):

The data dict contains preprocessed tensors:

• data["image0"]: shape (1, C, H, W), float32, range [0, 1]
• data["image1"]: shape (1, C, H, W)`, float32, range [0, 1]

If the model expects PIL images or file paths, convert:

img0 = data["image0"].cpu().numpy().squeeze() * 255
img0 = img0.transpose(1, 2, 0)  # CHW -> HWC
img0 = Image.fromarray(img0.astype("uint8"))

For sparse matchers (input: keypoints + descriptors):

# Repackage data for the model's expected format
input = {
    "image0": {"image": data["image0"], "keypoints": data["keypoints0"], ...},
    "image1": {"image": data["image1"], "keypoints": data["keypoints1"], ...},
}
return self.net(input)

3.5 Output Format

The _forward method MUST return a dict with these keys:

Dense matchers that output matched keypoints only:

pred = {
    "keypoints0": kpts0,      # torch.Tensor, shape (N, 2), pixel coords in resized image
    "keypoints1": kpts1,      # torch.Tensor, shape (N, 2)
    "mconf": confidence,      # torch.Tensor, shape (N,), match confidence scores
}

Dense matchers that can separate detected vs matched keypoints (PREFERRED):

pred = {
    "keypoints0": all_kpts0,     # All detected keypoints (for UI "Keypoints" display)
    "keypoints1": all_kpts1,     # All detected keypoints
    "mkeypoints0": matched_kpts0, # Matched keypoints (for UI match lines)
    "mkeypoints1": matched_kpts1, # Matched keypoints
    "mconf": confidence,          # Match confidence scores
}

Sparse matchers (LightGlue-style):

# Return the model's raw output — match_dense.py handles the rest
return self.net(input)

Key output rules:

• keypoints0/1: ALL detected keypoints (shown in UI "Open for More: Keypoints")
• mkeypoints0/1: Only MATCHED keypoints (shown as match lines in UI)
• If mkeypoints0/1 is missing, match_dense.py falls back to keypoints0/1
• Coordinates must be in pixel space of the resized image (not normalized [-1,1])
• mconf should be real confidence scores (not all-ones)

Step 4: Add Matcher Configuration

4.1 imcui/hloc/configs/matchers.py

Add a configuration entry for each model variant:

"<matcher-name>": {
    "output": "matches-<matcher-name>",
    "model": {
        "name": "<matcher-module-name>",  # Must match the .py filename in matchers/
        "model_name": "<variant>",         # Passed as conf["model_name"]
        "max_keypoints": 2048,
        # ... other model-specific config
    },
    "preprocessing": {
        "grayscale": False,     # True for LoFTR-style; False for most modern matchers
        "force_resize": True,
        "resize_max": 1024,
        "width": 640,
        "height": 480,
        "dfactor": 8,          # Image dimensions must be divisible by this
    },
},

Key rules:

• "name" in model must match the Python filename (e.g., "loma" → loma.py)
• Each model variant gets its own top-level entry (e.g., loma-b, loma-l, loma-g)
• preprocessing.grayscale: Set True only for models that expect 1-channel input (e.g., LoFTR)
• preprocessing.dfactor: Ensure resized dimensions are divisible by this value

4.2 config/app.yaml (local dev) AND imcui/config/app.yaml (package default)

BOTH files must be updated identically. Add an entry under matcher_zoo:

<DisplayName>:
  matcher: <matcher-config-name>    # Must match key in matchers.py
  standalone: true                   # true = takes two images directly (no separate extractor needed)
  skip_ci: false                     # DEFAULT: false. Only set to true for heavy matchers (see rule below)
  info:
    name: <DisplayName>              # Display name in WebUI dropdown
    source: "Venue Year"             # e.g., "ECCV 2026", "ICCV 2023"
    paper: <paper-url>               # arXiv or published paper URL
    github: <github-url>             # Official GitHub repo
    display: true                    # Whether to show in WebUI
    efficiency: medium               # low (heavy), medium, high (fast)

Key rules:

• standalone: true means the matcher takes two raw images directly (no separate feature extractor needed)
• standalone: false means the matcher requires a feature extractor (e.g., SuperPoint+LightGlue)
• For feature+matcher combos, use <extractor>+<matcher> format (e.g., superpoint+lightglue)
• Set enable: false for very heavy models that most users won't use by default
• CRITICAL: Both config/app.yaml and imcui/config/app.yaml must be kept in sync

skip_ci rule

skip_ci defaults to false — you can omit the field entirely for most matchers. Only set skip_ci: true when the model is too heavy to run in CI (GitHub Actions CPU-only Ubuntu runner with 7GB RAM):

Rule of thumb: if the model has variants like B/L/G/R, the "B" (base) variant usually passes CI, larger ones may not. When unsure, omit skip_ci (defaults to false) and let CI tell you — if it OOMs, set it to true in a follow-up commit.

Step 5: Handle Platform Compatibility

Common issues and fixes:

5.1 Mixed Precision (MP/AMP) Issues

On MPS/CPU, torch.autocast with float16/bfloat16 causes:

• RuntimeError: Input type (c10::Half) and bias type (float) should be the same
• RuntimeError: Input type (MPSFloatType) and weight type (torch.FloatTensor) should be the same

Fix pattern (before importing third-party code):

# Patch module-level amp_dtype before import
import <module>.device as _device
if not torch.cuda.is_available():
    _device.amp_dtype = torch.float32

# Also patch submodule local bindings
import <module>.submod as _submod
if not torch.cuda.is_available():
    _submod.amp_dtype = torch.float32

After model construction:

if not torch.cuda.is_available():
    cfg = dataclasses.replace(cfg, mp=False)
    for module in self.net.modules():
        if hasattr(module, "amp"):
            module.amp = False

5.2 Device Mismatch Issues

If the model manages its own device (like LoMa's loma.device):

# Override .to() to keep model on its expected device
def to(self, device=None, **kwargs):
    return super().to(<model_expected_device>, **kwargs)

5.3 Inference Mode vs No Grad

If a model uses @torch.inference_mode() but you need to pass its outputs to another module that requires grad tracking:

with torch.no_grad():
    output = model.detect_and_describe(...)
    output = output.clone()  # Detach from inference mode graph