> ## Documentation Index
> Fetch the complete documentation index at: https://docs.summerengine.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Lumera single image scene reconstruction skill for AI agents

> Lumera-style single image to editable 3D scene — VLM-parsed object boxes and parametric lights, per-object meshes, HDR probe, .tscn assembly, refinement loop. A Summer Engine skill for AI agents.

`lumera-single-image-scene-reconstruction` is a skill from the [Summer skill library](/mcp/skills): a guide your AI agent reads when the task matches, so it builds the good version on the first try. Lumera-style single image to editable 3D scene — VLM-parsed object boxes and parametric lights, per-object meshes, HDR probe, .tscn assembly, refinement loop.

| | |
| - | - |
| Status | Preview: not yet exercised in-engine by the Summer team |
| Domains | Asset pipeline, Pipeline, 3D models, Scenes and project |
| Version | `0.1.1` |
| Engine version | `>=4.6` |
| Recommended set | No, installed by `--all` or by name |

## When your agent uses it

* Turn this concept art or screenshot into an editable 3D scene with separate objects and lights
* Block out a level from a single reference image
* Reconstruct a room from one photo as movable meshes and tunable lights, not a point cloud or splat
* Build a bounded agent loop that refines a generated scene against a reference image

## Install

`summer setup` installs every skill for your agent. To add only this one:

```bash theme={null}
npx -y summer-engine@latest skills install lumera-single-image-scene-reconstruction --agent claude-code
```

Swap `claude-code` for your agent, such as `codex` or `cursor`. See [Skills](/mcp/skills#quick-start) for every option.

## The skill

## Outcome

Turn one reference image into an **editable, engine-native scene** in Summer Engine: individual `MeshInstance3D` objects with transforms, countable parametric light nodes, and an HDR environment probe — not a fused point cloud, Gaussian splat, or baked texture. This adapts the Lumera pipeline (arXiv:2607.20889, Tsinghua AIRSUN lab), which was built on another engine, to Summer Engine.

**Caveat:** Lumera has no public code, and the paper lists engine-portable extraction as future work. This skill documents the architecture and the parts an agent can build in Summer Engine today (assembly, scene representation, refinement loop, evaluation), and flags the parts that require external model components (VLM box/light parsers, per-object mesh generator, HDR estimator).

## When to Use

* Bootstrapping a level/blockout from concept art or a screenshot, where artists need movable objects and re-tunable lights afterward.
* Producing 3D scene conditions for 3D-conditioned video generation (a use case the paper calls out).
* You specifically need **editable entities** (instances, lights, probes) rather than a render-able blob.

## When NOT to Use

* You only need a visual backdrop: a Gaussian-splat or depth-mesh bake is cheaper and looks as good or better.
* Large unbounded outdoor scenes: the paper reports \~17 m Chamfer-L2 drift outdoors — geometry is unreliable at that scale.
* Exact light counts/positions are critical: individual-light localization F1 is 0.209 at 0.5 m; intensity error is \~2.7× (log-MAE 0.431). Expect hand-tuning.

## Core Principle

Treat image-to-3D as **game-engine structured parsing**: engines already store exactly the entities downstream tools need — object components, transforms, cameras, materials, environment probes, and countable parametric lights. The pipeline therefore parses those entities directly instead of reconstructing geometry first and deriving entities later.

Lumera's stages (from paper abstract + appendices):

1. **Lumera-Box** — VLM parses object-level 3D boxes (position, size, yaw) from the image.
2. **Lumera-Light** — VLM parses parametric light tuples `(x, y, z, r, g, b, I)`.
3. **Per-object mesh reconstruction** — a mesh per parsed box.
4. **HDR environment estimation** — environment probe (IntrinsicHDR partially compensates for SkyLight/unseen near-camera lights).
5. **Bounded agentic refinement loop** — dual-agent (geometry stage, then light stage) edits + verifies against the reference image.
6. **Assembly** — emit an engine-native scene (`.tscn` in Summer Engine).

Dataset backing it: Lumera-2K, 2,513 game-engine projects → 3.73M components, 63M object instances, 102.6K parametric lights, 95.1K camera views.

## Scene / Node Shape (Summer Engine target)

The assembly stage should emit this node tree so output stays inspectable/editable:

```
ReconstructedScene (Node3D)
├── Camera3D                    # recovered viewpoint (intrinsics from parsing stage)
├── WorldEnvironment
│   └── Environment
│       └── Sky (HDR panorama from stage 4) + ambient light settings
├── Objects (Node3D)
│   └── MeshInstance3D × N      # one per parsed box; transform = box center/yaw
│       └── unique material per object (so meshes are replaceable/refinable)
└── Lights (Node3D)
    ├── OmniLight3D / SpotLight3D / DirectionalLight3D × M
    │     # position/rotation from (x,y,z); color from (r,g,b);
    │     # light_energy from I (mind log-scale calibration, see Tunables)
    └── (the sky-light equivalent is carried by WorldEnvironment)
```

Keep objects and lights as **separate siblings under named containers**. That separation is what makes the scene editable rather than just looking 3D.

## Implementation Steps

### 1. Define the intermediate scene schema

Before touching models, define the serializable tuple set the pipeline passes between stages (this is the engine-native contract):

```gdscript theme={null}
# object record parsed by the box stage
class_name ParsedObject
var box_center: Vector3
var box_size: Vector3      # must be > 0; validate (paper strips non-positive sizes)
var yaw: float             # watch 0 vs ±π/2 symmetry for near-square objects
var label: String
var id: int                # unique; duplicates are a known model failure mode

# light record parsed by the light stage
class_name ParsedLight
var position: Vector3
var color: Color           # r,g,b
var intensity: float       # I, log-scale calibration needed (~2.7x error typical)
var kind: String           # point / spot / directional mapping
```

Add a **repair layer** on ingest: drop/fix invalid labels, non-positive sizes, duplicate IDs. The paper notes sanitized evaluation silently removed these, and recommends constrained decoding or a repair layer in the inference path.

### 2. Box + light parsing (external VLM components)

Lumera-Box/Lumera-Light are VLMs SFT'd on Lumera-2K; there is no public checkpoint. Options: use a hosted VLM prompted to emit the JSON tuples above, or fine-tune an open VLM on your own engine-exported scenes (`.tscn` scenes are text and easy to export; this is the "build Lumera-2K for your engine" step the paper identifies as missing for other engines).

### 3. Per-object mesh generation + HDR environment estimation

Also external model components in Lumera (per-object mesh reconstruction; IntrinsicHDR-style environment estimation). For a Summer Engine blockout you can substitute: box mesh (`BoxMesh` sized to the parsed box) per object and a generated/estimated panorama `Sky`. Swap in real reconstructed meshes later without changing the schema.

### 4. Assembly: write a `.tscn`

Deterministic GDScript/tool script: instantiate the node shape above from the repaired records. A `.tscn` is text, so generating it directly keeps the output diffable and versionable. Save with `PackedScene.pack()` / `ResourceSaver.save()` from an editor tool.

### 5. Bounded dual-agent refinement loop

The paper's Algorithm 1 (Appendix H.7), adapted:

* **Two stages in order:** geometry (objects/boxes), then lights. Switch the agent scope and prompt head per stage, but **inherit scene state `s` and conversation history `M`** across the switch — this avoids a cold start and lets the light-stage verifier refer to object identities already confirmed in the geometry stage.
* **Each round:** agent proposes an action `a_t` (move/resize/add/remove an entity) given the history, reference image, and current scene state; the executor applies it to the scene; a verifier renders the result and reports `r_t` against the reference image.
* **Sliding-window history:** keep only the system prompt + most recent `L` action/report pairs (`M_{t+1} = Tail_L(M_t ∪ {a_t, r_t})`, per VIGA). Skip records adjacent to rolled-back steps. This keeps 30+ rounds inside the VLM context window.
* **Bounded:** hard round limits `T_g` (geometry) and `T_l` (light), plus early termination when the verifier report triggers it. Never run unbounded — that is what keeps this agentic loop "bounded" and cheap.

### 6. Evaluation harness

Reproduce the paper's metrics on held-out scenes so improvements are measurable:

| Target | Metric |
| - | - |
| Boxes | 3D mAP, IoU-B, F-score (Lumera-Box reference: mAP 0.1141, IoU-B 0.2472, F 0.2762) |
| Lights (scene-level) | non-empty-scene recall (reference: 0.998) |
| Lights (per-light) | F1 @ 0.5 m (reference: 0.209 — weak; expect hand-tuning) |
| Matched lights | median position error (0.261 m), median ΔE2000 color error (4.59), intensity Pearson r (0.628), log-intensity MAE (0.431) |
| Geometry (scene) | Chamfer-L2 (outdoor reference ≈ 17 m — large unbounded scenes drift) |
| Relations | SRF / anchor recall (known weak point; WildDet3D beats Lumera-Box here) |

## Tunables

| Parameter | Meaning | Notes |
| - | - | - |
| `T_g`, `T_l` | max refinement rounds per stage | paper keeps 30+ rounds total inside context; bound them |
| `L` | history window size (action/report pairs) | trade context use vs. verifier memory of confirmed objects |
| Light intensity calibration | map predicted `I` to `light_energy` | \~2.7× brightness error typical; log-scale loss recommended by authors |
| Yaw symmetry handling | treat 0 and ±π/2 as equivalent for near-square footprints | explicit paper limitation; add symmetry classes or snap checks |
| Supervision scope (if fine-tuning) | lights whose influence intersects the view frustum, not just visible sources | authors' recommended fix for missing SkyLight/carriers |

## Failure Modes & Gotchas

* **Merged/baked outputs defeat the point.** If assembly emits one combined mesh or a splat, you've built an image-to-3D renderer, not Lumera. Keep per-object nodes + parametric lights.
* **Invalid model outputs are common:** non-positive box sizes, duplicate IDs, ID fallbacks. Validate and repair before assembly; the paper's benchmark silently sanitized these.
* **Yaw ambiguity** on near-square objects (0 vs ±π/2 look identical) — single-token yaw serialization in prior work does not handle symmetry.
* **Relation structure is unsolved** (object-to-object spatial relations): anchor recall/SRF are Lumera's weakest scores. Don't promise correct relative layouts.
* **Per-light localization is weak** (F1 0.209 @ 0.5 m) while scene-level light *count* recall is strong (0.998). Frame the output as "right number of lights, roughly right places."
* **SkyLight / near-camera lights whose source is off-screen** are not supervised; the environment probe (IntrinsicHDR) only partially compensates.
* **Large outdoor scenes drift geometrically** (\~17 m Chamfer-L2); consider joint camera+geometry calibration if you need metric outdoor consistency.
* **Engine conventions don't transfer for free.** The dataset's light conventions come from another engine; Summer Engine's light units and probes differ, so recalibrate intensity and validate the representation on Summer-native scenes.

## Verification

Not yet verified in Summer Engine. To validate an implementation:

1. Assemble a scene from a test image; confirm the `.tscn` contains separate, selectable `MeshInstance3D`s and light nodes (editability check).
2. Move/replace one mesh and re-tune one light in the editor without breaking the scene.
3. Run the evaluation harness (§6) against a held-out set of known scenes and compare to the reference numbers above.
4. Confirm the refinement loop terminates within `T_g`/`T_l` and the history window never exceeds the VLM context budget.

## Confidence

`extracted` — Architecture, metrics, refinement-loop design, and limitations are drawn directly from the arXiv paper (abstract, intro, appendices H.7–I as retrieved; middle method sections were truncated and are not documented here) and the author's announcement post. The Summer Engine node mapping and GDScript schema are adaptations, not from the sources; no public Lumera code exists to reference.

## Related skills

* [`environment-kit`](/skills/environment-kit): Generate a modular environment kit: walls, floor tiles, pillars, doors, arches in one visual style.
* [`3d-lighting`](/skills/3d-lighting): Light a 3D scene: sun, omni and spot lights, WorldEnvironment, sky, shadows and ambient.
* [`scene-composition`](/skills/scene-composition): Scene structure: node hierarchy, when to extract sub-scenes, reusable prefabs, instance vs add node.

***

Source: [`library/skills/lumera-single-image-scene-reconstruction/SKILL.md`](https://github.com/SummerEngine/summer/blob/main/library/skills/lumera-single-image-scene-reconstruction/SKILL.md), rendered from `summer-engine@4.3.0` on npm.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.