scene_service.scene_graph.image_relations

Image-grounded VLM relation inference (VLM-primary scene graph).

Replaces the text-only, coordinates-as-JSON relation pass with one that lets the model see the scene: tracked objects are projected from the map frame into the current RGB keyframe, numbered boxes are drawn, and a single multimodal VLM call enumerates the relations among the numbered objects. Box numbers map back to object_id to build edges.

Why this layer is image-grounded and geometry is not: contact/containment (“monitor on the desk”) is exactly where coordinate-only reasoning and AABB tests fail (full-volume desk box, perspective), and a VLM reads it holistically; metric robot-actionable facts (reachable_by, near) stay deterministic in the geometric layer and are intentionally excluded from this vocabulary.

Functions

annotate_frame(rgb_bgr, boxes, *[, max_dim])	Draw numbered rectangles on a copy of rgb_bgr and return it as a base64 JPEG string (no data-url prefix), or None if encoding fails.
build_image_relation_user_text(legend)	User-turn text accompanying the annotated image: the box-number → label legend plus the instruction.
parse_image_relations(raw, box_to_oid)	Turn the VLM's {"edges": [...]} into validated graph edges.
project_box(T_cam_map, K, center, extent, ...)	Project a 3D box to a clipped 2D pixel rect (u0, v0, u1, v1), or None when the object is not usefully visible.
project_point(T_cam_map, K, p_world)	Project a map-frame point to (u, v, depth) pixels, or None if it is behind the camera.

Classes

ImageRelationInferer(llm_client, *, max_dim, ...)

Whole-scene image-grounded relation pass: project → annotate → one VLM call → parse.

class scene_service.scene_graph.image_relations.ImageRelationInferer(llm_client, *, max_dim: int | None = None, frame_change_threshold: float | None = None, failure_backoff_base_s: float | None = None, failure_backoff_max_s: float | None = None, clock: Callable[[], float]=<built-in function monotonic>)

Bases: object

Whole-scene image-grounded relation pass: project → annotate → one VLM call → parse. Successful results are reused while the visible scene input remains perceptually and geometrically unchanged.

async infer(nodes: list[SceneGraphNode], bundle: tuple[ndarray, Any, ndarray]) → ImageRelationResult | None

Return this round’s typed relation result or None when it cannot run.

Failure and backoff remain distinct so the builder advances hysteresis only after a real failed call. A successful empty edge list is cached.

property inference_counts: dict[str, int]

class scene_service.scene_graph.image_relations.ImageRelationResult(edges: tuple[SceneGraphEdge, ...], outcome: Literal['processed', 'cached', 'failed', 'backoff'])

Bases: object

Atomic relation result with the inference decision that produced it.

edges: tuple[SceneGraphEdge, ...]

outcome: Literal['processed', 'cached', 'failed', 'backoff']

scene_service.scene_graph.image_relations.annotate_frame(rgb_bgr: ndarray, boxes: list[tuple[int, tuple[int, int, int, int]]], *, max_dim: int = 960) → str | None

Draw numbered rectangles on a copy of rgb_bgr and return it as a base64 JPEG string (no data-url prefix), or None if encoding fails.

boxes is [(box_id, (u0, v0, u1, v1)), ...]. The frame is downscaled so its longest side is ≤ max_dim (after drawing, so coordinates need no rescale) to bound VLM token cost. cv2 is imported lazily — the projection helpers above stay importable without it.

scene_service.scene_graph.image_relations.build_image_relation_user_text(legend: list[tuple[int, str]]) → str

User-turn text accompanying the annotated image: the box-number → label legend plus the instruction. The image itself is attached as a separate multimodal part by the caller (chat_json(images=...)).

scene_service.scene_graph.image_relations.parse_image_relations(raw: dict, box_to_oid: dict[int, str]) → list[SceneGraphEdge]

Turn the VLM’s {"edges": [...]} into validated graph edges.

Drops edges whose box numbers are unknown, self-edges, duplicates of an already-seen (source, target, relation), and relations outside IMAGE_RELATION_VOCAB (after normalization). Confidence is bounded to [0, 1]. Never raises on a malformed entry — it is skipped.

scene_service.scene_graph.image_relations.project_box(T_cam_map: ndarray, K: Any, center: tuple[float, float, float], extent: tuple[float, float, float], yaw: float, img_w: int, img_h: int) → tuple[int, int, int, int] | None

Project a 3D box to a clipped 2D pixel rect (u0, v0, u1, v1), or None when the object is not usefully visible.

Projects the 8 corners, keeps those in front of the camera, takes their pixel min/max and clips to the image. Returns None if fewer than two corners are in front (object mostly behind the camera) or the clipped rect has no area (entirely out of frame).

scene_service.scene_graph.image_relations.project_point(T_cam_map: ndarray, K: Any, p_world: tuple[float, float, float]) → tuple[float, float, float] | None

Project a map-frame point to (u, v, depth) pixels, or None if it is behind the camera.

T_cam_map is the 4×4 camera-optical→map transform (as built by the perception detector); its inverse takes a map point into the camera-optical frame, then the pinhole model with intrinsics K (fx/fy/cx/cy) gives the pixel. depth (camera-frame z) is returned so callers can z-sort or gate on it. Points at/behind the image plane (z ≤ 0) return None.