3D-PAQA Details

Preference-aligned 3D quality, step by step

The project builds criterion-wise supervision for real 3D assets, validates its alignment with human judgment, and distills it into an efficient 3D evaluator.

01 / Motivation

Real 3D assets need a different quality target.

Most existing 3D-QA benchmarks start from a small set of clean objects and inject synthetic degradations such as noise, compression, or downsampling. That setup measures distortion severity, but it does not fully capture naturally occurring artifacts in diverse asset repositories.

  • Asset quality is shaped by geometry, texture, material, plausibility, and visible artifacts.
  • Human preference matters for evaluation, curation, and downstream generation pipelines.
Task definition slide for 3D-PAQA
3D-PAQA maps an arbitrary 3D asset to a criterion-wise quality vector.
02 / Task

Direct per-asset scoring replaces reference dependence.

The task estimates a quality vector for an arbitrary 3D asset. It does not require a pristine reference, an artificial distortion operator, or pairwise A/B aggregation for every target asset.

The six axes cover holistic preference, plausibility, artifacts, and component-level geometry, texture, and material quality.

Holistic

  • Preference
  • Plausibility
  • Artifacts

Components

  • Geometry
  • Texture
  • Material

Output

A quality vector that can diagnose where an asset is strong or weak.

03 / Annotation

Exemplar anchors ground subjective judgments.

The large-scale annotation pipeline injects human preference through stratified exemplar anchors. Each target is shown with multi-view evidence and quality-spanning references so the MLLM judgment is grounded by examples.

  • Visual prompting supplies criterion-specific exemplar anchors.
  • Relative Ranking positions the target among anchored quality levels.
  • Repeated anchor sampling and ensembling reduce anchor-specific bias.
3D-PAQA annotation process with exemplar images and Qwen2-VL
The annotation process combines visual prompts, criteria instructions, and relative ranking outputs.
04 / Dataset

The supervision scales across assets and criteria.

3D-PAQA gathers preference-aligned annotations for over 260K Objaverse-based assets across ten semantic domains. The criteria separate overall perception from component-level weaknesses.

  • Natural artifacts instead of hand-injected distortions.
  • Six quality signals for richer diagnosis than one global score.
  • MLLM-aligned labels designed for training and analysis.
Dataset overview slide for 3D-PAQA
Dataset overview from the project presentation.
05 / Evaluator

A compact 3D model distills the quality signal.

The project trains a lightweight evaluator on 3D-PAQA annotations. A Point Transformer-v3 backbone predicts criterion-wise quality directly from sampled 3D features.

  • Inputs include point cloud signals such as RGB, normals, and PBR attributes.
  • The evaluator has 33M parameters, roughly 0.046% of the 72B teacher scale.
  • Preference-aligned supervision remains useful after distillation.
Efficient 3D evaluator slide for 3D-PAQA
The distilled evaluator predicts the same criterion-wise quality structure from 3D features.
06 / Evidence

Criterion-wise scores become diagnostic signals.

The analysis shows that 3D-PAQA annotations align with human preferences and capture perceptual quality beyond simple geometry complexity. Component-level axes also reveal localized failures.

  • 72B relative ranking is adopted for preference-aligned supervision.
  • The compact evaluator achieves strong human alignment on overall preference.
  • High geometry quality does not guarantee strong texture or material quality.
3D-PAQA diagnostic signals slide showing geometry and texture scores
Lower-scoring criteria point to the asset component that needs attention.
07 / Refinement

The evaluator closes the loop as a generation reward.

Beyond offline benchmarking, the 3D-Evaluator can act directly as a verifier inside a generation loop. An LLM agent writes a Blender (bpy) script that builds and exports a mesh; the 3D-Evaluator scores the asset on the six criteria and returns the criterion-wise scores together with the current weakest criterion; the agent then revises its script to raise that criterion and re-exports.

  • Guided only by this per-criterion feedback, each seed asset matures into a coherent, detailed one over successive iterations.
  • For the castle example, texture rises by +1.03, material by +1.02, and overall preference by +1.03 (3.28 → 4.32), with the largest gains on the initially weakest criteria.
  • Reported as a feasibility study: the visible, human-verifiable transformation is the primary evidence, with the rising scores as corroboration.
Each evaluation is a single local forward pass of the 33M model: ≈$0 and <1s, versus ≈$0.064 and ~15s for an equivalent MLLM judge — roughly 16 refinement steps per $1 for the MLLM judge against unbounded local steps for the 3D-Evaluator, which is what makes closed-loop quality-guided generation practical.
Evaluator-guided asset refinement across five objects, seed to final, with per-criterion score gains
An LLM agent generates and iteratively revises each asset in Blender using the 3D-Evaluator as the verifier; the criterion weakest at the seed (highlighted) improves the most.

Applications and current limits

The quality signal supports evaluation, repository curation, and quality-guided asset refinement: Section 07 above demonstrates a first feasibility study where the evaluator steers an LLM agent's generation loop. Tighter integration into generative models themselves (e.g. differentiable rendering or score-distillation objectives), a systematic study of refinement gains at scale, downstream validation as a dataset curator, and transfer to AI-generated assets remain future work.