Lumera single image scene reconstruction skill for AI agents

lumera-single-image-scene-reconstruction is a skill from the Summer skill library: a guide your AI agent reads when the task matches, so it builds the good version on the first try. Lumera-style single image to editable 3D scene — VLM-parsed object boxes and parametric lights, per-object meshes, HDR probe, .tscn assembly, refinement loop.


Status	Preview: not yet exercised in-engine by the Summer team
Domains	Asset pipeline, Pipeline, 3D models, Scenes and project
Version	`0.1.1`
Engine version	`>=4.6`
Recommended set	No, installed by `--all` or by name

When your agent uses it

Turn this concept art or screenshot into an editable 3D scene with separate objects and lights
Block out a level from a single reference image
Reconstruct a room from one photo as movable meshes and tunable lights, not a point cloud or splat
Build a bounded agent loop that refines a generated scene against a reference image

Install

summer setup installs every skill for your agent. To add only this one:

npx -y summer-engine@latest skills install lumera-single-image-scene-reconstruction --agent claude-code

Swap claude-code for your agent, such as codex or cursor. See Skills for every option.

The skill

Outcome

Turn one reference image into an editable, engine-native scene in Summer Engine: individual MeshInstance3D objects with transforms, countable parametric light nodes, and an HDR environment probe — not a fused point cloud, Gaussian splat, or baked texture. This adapts the Lumera pipeline (arXiv:2607.20889, Tsinghua AIRSUN lab), which was built on another engine, to Summer Engine. Caveat: Lumera has no public code, and the paper lists engine-portable extraction as future work. This skill documents the architecture and the parts an agent can build in Summer Engine today (assembly, scene representation, refinement loop, evaluation), and flags the parts that require external model components (VLM box/light parsers, per-object mesh generator, HDR estimator).

When to Use

Bootstrapping a level/blockout from concept art or a screenshot, where artists need movable objects and re-tunable lights afterward.
Producing 3D scene conditions for 3D-conditioned video generation (a use case the paper calls out).
You specifically need editable entities (instances, lights, probes) rather than a render-able blob.

When NOT to Use

You only need a visual backdrop: a Gaussian-splat or depth-mesh bake is cheaper and looks as good or better.
Large unbounded outdoor scenes: the paper reports ~17 m Chamfer-L2 drift outdoors — geometry is unreliable at that scale.
Exact light counts/positions are critical: individual-light localization F1 is 0.209 at 0.5 m; intensity error is ~2.7× (log-MAE 0.431). Expect hand-tuning.

Core Principle

Treat image-to-3D as game-engine structured parsing: engines already store exactly the entities downstream tools need — object components, transforms, cameras, materials, environment probes, and countable parametric lights. The pipeline therefore parses those entities directly instead of reconstructing geometry first and deriving entities later. Lumera’s stages (from paper abstract + appendices):

Lumera-Box — VLM parses object-level 3D boxes (position, size, yaw) from the image.
Lumera-Light — VLM parses parametric light tuples (x, y, z, r, g, b, I).
Per-object mesh reconstruction — a mesh per parsed box.
HDR environment estimation — environment probe (IntrinsicHDR partially compensates for SkyLight/unseen near-camera lights).
Bounded agentic refinement loop — dual-agent (geometry stage, then light stage) edits + verifies against the reference image.
Assembly — emit an engine-native scene (.tscn in Summer Engine).

Dataset backing it: Lumera-2K, 2,513 game-engine projects → 3.73M components, 63M object instances, 102.6K parametric lights, 95.1K camera views.

Scene / Node Shape (Summer Engine target)

The assembly stage should emit this node tree so output stays inspectable/editable:

ReconstructedScene (Node3D)
├── Camera3D                    # recovered viewpoint (intrinsics from parsing stage)
├── WorldEnvironment
│   └── Environment
│       └── Sky (HDR panorama from stage 4) + ambient light settings
├── Objects (Node3D)
│   └── MeshInstance3D × N      # one per parsed box; transform = box center/yaw
│       └── unique material per object (so meshes are replaceable/refinable)
└── Lights (Node3D)
    ├── OmniLight3D / SpotLight3D / DirectionalLight3D × M
    │     # position/rotation from (x,y,z); color from (r,g,b);
    │     # light_energy from I (mind log-scale calibration, see Tunables)
    └── (the sky-light equivalent is carried by WorldEnvironment)

Keep objects and lights as separate siblings under named containers. That separation is what makes the scene editable rather than just looking 3D.

Implementation Steps

1. Define the intermediate scene schema

Before touching models, define the serializable tuple set the pipeline passes between stages (this is the engine-native contract):

# object record parsed by the box stage
class_name ParsedObject
var box_center: Vector3
var box_size: Vector3      # must be > 0; validate (paper strips non-positive sizes)
var yaw: float             # watch 0 vs ±π/2 symmetry for near-square objects
var label: String
var id: int                # unique; duplicates are a known model failure mode

# light record parsed by the light stage
class_name ParsedLight
var position: Vector3
var color: Color           # r,g,b
var intensity: float       # I, log-scale calibration needed (~2.7x error typical)
var kind: String           # point / spot / directional mapping

Add a repair layer on ingest: drop/fix invalid labels, non-positive sizes, duplicate IDs. The paper notes sanitized evaluation silently removed these, and recommends constrained decoding or a repair layer in the inference path.

2. Box + light parsing (external VLM components)

Lumera-Box/Lumera-Light are VLMs SFT’d on Lumera-2K; there is no public checkpoint. Options: use a hosted VLM prompted to emit the JSON tuples above, or fine-tune an open VLM on your own engine-exported scenes (.tscn scenes are text and easy to export; this is the “build Lumera-2K for your engine” step the paper identifies as missing for other engines).

3. Per-object mesh generation + HDR environment estimation

Also external model components in Lumera (per-object mesh reconstruction; IntrinsicHDR-style environment estimation). For a Summer Engine blockout you can substitute: box mesh (BoxMesh sized to the parsed box) per object and a generated/estimated panorama Sky. Swap in real reconstructed meshes later without changing the schema.

4. Assembly: write a `.tscn`

Deterministic GDScript/tool script: instantiate the node shape above from the repaired records. A .tscn is text, so generating it directly keeps the output diffable and versionable. Save with PackedScene.pack() / ResourceSaver.save() from an editor tool. The paper’s Algorithm 1 (Appendix H.7), adapted:

Two stages in order: geometry (objects/boxes), then lights. Switch the agent scope and prompt head per stage, but inherit scene state s and conversation history M across the switch — this avoids a cold start and lets the light-stage verifier refer to object identities already confirmed in the geometry stage.
Each round: agent proposes an action a_t (move/resize/add/remove an entity) given the history, reference image, and current scene state; the executor applies it to the scene; a verifier renders the result and reports r_t against the reference image.
Sliding-window history: keep only the system prompt + most recent L action/report pairs (M_{t+1} = Tail_L(M_t ∪ {a_t, r_t}), per VIGA). Skip records adjacent to rolled-back steps. This keeps 30+ rounds inside the VLM context window.
Bounded: hard round limits T_g (geometry) and T_l (light), plus early termination when the verifier report triggers it. Never run unbounded — that is what keeps this agentic loop “bounded” and cheap.

6. Evaluation harness

Reproduce the paper’s metrics on held-out scenes so improvements are measurable:

Target	Metric
Boxes	3D mAP, IoU-B, F-score (Lumera-Box reference: mAP 0.1141, IoU-B 0.2472, F 0.2762)
Lights (scene-level)	non-empty-scene recall (reference: 0.998)
Lights (per-light)	F1 @ 0.5 m (reference: 0.209 — weak; expect hand-tuning)
Matched lights	median position error (0.261 m), median ΔE2000 color error (4.59), intensity Pearson r (0.628), log-intensity MAE (0.431)
Geometry (scene)	Chamfer-L2 (outdoor reference ≈ 17 m — large unbounded scenes drift)
Relations	SRF / anchor recall (known weak point; WildDet3D beats Lumera-Box here)

Tunables

Parameter	Meaning	Notes
`T_g`, `T_l`	max refinement rounds per stage	paper keeps 30+ rounds total inside context; bound them
`L`	history window size (action/report pairs)	trade context use vs. verifier memory of confirmed objects
Light intensity calibration	map predicted `I` to `light_energy`	~2.7× brightness error typical; log-scale loss recommended by authors
Yaw symmetry handling	treat 0 and ±π/2 as equivalent for near-square footprints	explicit paper limitation; add symmetry classes or snap checks
Supervision scope (if fine-tuning)	lights whose influence intersects the view frustum, not just visible sources	authors’ recommended fix for missing SkyLight/carriers

Failure Modes & Gotchas

Merged/baked outputs defeat the point. If assembly emits one combined mesh or a splat, you’ve built an image-to-3D renderer, not Lumera. Keep per-object nodes + parametric lights.
Invalid model outputs are common: non-positive box sizes, duplicate IDs, ID fallbacks. Validate and repair before assembly; the paper’s benchmark silently sanitized these.
Yaw ambiguity on near-square objects (0 vs ±π/2 look identical) — single-token yaw serialization in prior work does not handle symmetry.
Relation structure is unsolved (object-to-object spatial relations): anchor recall/SRF are Lumera’s weakest scores. Don’t promise correct relative layouts.
Per-light localization is weak (F1 0.209 @ 0.5 m) while scene-level light count recall is strong (0.998). Frame the output as “right number of lights, roughly right places.”
SkyLight / near-camera lights whose source is off-screen are not supervised; the environment probe (IntrinsicHDR) only partially compensates.
Large outdoor scenes drift geometrically (~17 m Chamfer-L2); consider joint camera+geometry calibration if you need metric outdoor consistency.
Engine conventions don’t transfer for free. The dataset’s light conventions come from another engine; Summer Engine’s light units and probes differ, so recalibrate intensity and validate the representation on Summer-native scenes.

Verification

Not yet verified in Summer Engine. To validate an implementation:

Assemble a scene from a test image; confirm the .tscn contains separate, selectable MeshInstance3Ds and light nodes (editability check).
Move/replace one mesh and re-tune one light in the editor without breaking the scene.
Run the evaluation harness (§6) against a held-out set of known scenes and compare to the reference numbers above.
Confirm the refinement loop terminates within T_g/T_l and the history window never exceeds the VLM context budget.

Confidence

extracted — Architecture, metrics, refinement-loop design, and limitations are drawn directly from the arXiv paper (abstract, intro, appendices H.7–I as retrieved; middle method sections were truncated and are not documented here) and the author’s announcement post. The Summer Engine node mapping and GDScript schema are adaptations, not from the sources; no public Lumera code exists to reference.

environment-kit: Generate a modular environment kit: walls, floor tiles, pillars, doors, arches in one visual style.
3d-lighting: Light a 3D scene: sun, omni and spot lights, WorldEnvironment, sky, shadows and ambient.
scene-composition: Scene structure: node hierarchy, when to extract sub-scenes, reusable prefabs, instance vs add node.

Source: library/skills/lumera-single-image-scene-reconstruction/SKILL.md, rendered from summer-engine@4.3.0 on npm.

​When your agent uses it

​Install

​The skill

​Outcome

​When to Use

​When NOT to Use

​Core Principle

​Scene / Node Shape (Summer Engine target)

​Implementation Steps

​1. Define the intermediate scene schema

​2. Box + light parsing (external VLM components)

​3. Per-object mesh generation + HDR environment estimation

​4. Assembly: write a .tscn

​5. Bounded dual-agent refinement loop

​6. Evaluation harness

​Tunables

​Failure Modes & Gotchas

​Verification

​Confidence

​Related skills