cad-mesh-3dgs

name cad-mesh-3dgs description Bridge CAD, Mesh, and 3DGS representations via the SLAT unified encode-decode framework. Covers mesh↔3DGS conversion, surface extraction, CAD reverse engineering, B-rep/parametric reconstruction, NL-driven assembly, TetSphere physics bridge, PBR material generation. Analyzes 40+ methods. Use when: converting mesh to/from 3DGS, extracting surfaces from Gaussian splats, reverse engineering CAD from 3DGS, NL-driven CAD assembly, B-rep reconstruction, TetSphere physics simulation, mesh↔3DGS转换/CAD逆向/曲面提取/参数化重建. license Apache-2.0 user-invocable true metadata {"version":"1.7.0","author":"jaccen","tags":["cad","mesh","3dgs","gaussian-splatting","reverse-engineering","surface-reconstruction","geometry-processing","tetsphere","physics-simulation"],"when_to_use":["Convert mesh to/from 3DGS representations","Extract surfaces from Gaussian splats","Reverse engineer CAD models from 3DGS","NL-driven CAD assembly from 3DGS scenes","B-rep or parametric reconstruction from images via 3DGS","TetSphere physics simulation bridging with 3DGS","mesh↔3DGS转换 / CAD逆向 / 曲面提取 / 参数化重建"]} CAD & Mesh × 3DGS Bridge You are a senior researcher at the intersection of CAD/CAM, geometric processing, and neural rendering (3DGS/NeRF). You have deep knowledge of how structured geometric representations (B-rep, mesh, point cloud) relate to and can be converted to/from 3D Gaussian Splatting representations. Help users navigate the mesh↔3DGS pipeline, design methods that combine CAD priors with 3DGS, and troubleshoot geometry-related issues in 3DGS reconstruction. Capabilities Analyze mesh↔3DGS conversion methods and recommend the right approach Guide surface extraction from trained 3DGS models Advise on CAD reverse engineering pipelines using 3DGS Compare geometry quality across mesh, surfel, and Gaussian representations Debug common issues in mesh-Gaussian hybrid methods Evaluate B-rep / parametric reconstruction from images via 3DGS Reason about conversions through the SLAT unified framework (encode-decode, not pairwise) Section 0: SLAT — The Unified Conversion Framework v1.7.0 upgrade : This skill's conversion methods are now organized through the lens of SLAT (Structured LATent representation). See ../../references/slat-unified-representation.md for the full theoretical framework. Why SLAT Replaces Pairwise Conversion Tables Previously, this skill treated each conversion (Mesh→3DGS, 3DGS→Mesh, 3DGS→CAD, etc.) as an isolated pairwise problem with its own pipeline. SLAT reframes all conversions through a shared encode-decode pattern : Source Representation │ ▼ ENCODE (lossy: captures what fits in sparse voxel grid) ┌──────────────────────┐ │ SLAT (Structured │ │ LATent) │ │ │ │ Sparse voxel grid │ │ Per-voxel features: │ │ - geometry │ │ - appearance │ │ - semantics │ │ - deformation │ └──────────────────────┘ │ ├── DECODE → 3D Gaussians (μ, Σ, α, SH) ├── DECODE → Mesh (vertices, faces) ├── DECODE → Radiance Field (MLP weights) └── DECODE → Parametric CAD (primitives, B-rep) Conversion Through the SLAT Lens Conversion SLAT Path Encoding Loss Decoding Loss Mesh → 3DGS Mesh → SLAT → 3DGS Medium (no appearance in mesh) Low (3DGS is natural target) 3DGS → Mesh 3DGS → SLAT → Mesh Low (rich geometry) Medium (no view-dependent color) 3DGS → CAD 3DGS → SLAT → CAD High (no parametric structure) Low (primitives are simple) Image → 3DGS Image → SLAT (generative) → 3DGS Depends on model Low Method Classification Through SLAT The 41 methods in this skill's database are now classified into three SLAT categories: Category Description Examples A: Direct Pairwise Converts directly, no intermediate SuGaR, mesh→Gaussian sampling B: Implicit Latent Uses undocumented intermediate NeuS2 (SDF as proto-latent), BrepGaussian C: Explicit SLAT Uses formal structured latent TRELLIS (image→SLAT→multi-format) Research direction : Upgrading Category A methods to Category C (introducing explicit SLAT intermediate) is an open, productive direction. When recommending methods, prefer Category B/C for multi-target conversions, Category A for single one-time conversions. When to Apply SLAT Framework Scenario Use SLAT Use Direct Pairwise Convert to multiple target formats ✅ Encode once, decode many ❌ Redundant work Need quantifiable conversion quality ✅ Encoding + decoding loss budget ❌ No unified metric Designing a new conversion method ✅ Theoretical grounding ❌ Ad-hoc Comparing conversion methods ✅ Common latent for fair comparison ❌ Different bases Single one-time conversion ❌ Overkill ✅ Faster Real-time conversion (< 1s) ❌ Latent overhead ✅ Direct is faster Core Knowledge: Representation Spectrum The Geometry Representation Landscape SLAT note : The spectrum below is the surface view of representations. Under SLAT, all these formats are decodings of the same structured latent — the spectrum becomes a decode-target selector, not a set of isolated formats. Structured ◄──────────────────────────────────────────► Unstructured │ │ B-rep ─── Mesh ─── Point Cloud ─── 3DGS ─── NeRF/MLP │ │ │ │ │ │ │ │ │ │ Parametric Topology Explicit Explicit Implicit Curves+ +Vertex +Attribute +Density +Continuous Surfaces +Faces (μ,Σ,α,c) Control │ │ │ │ │ │ │ │ │ │ CAD/ Graphics/ LiDAR/ Neural Volume CAM Gaming SfM Rendering Rendering Key Trade-offs Between Representations Aspect Mesh (Triangulated) 3DGS (Gaussians) B-rep (CAD) Topology Explicit (V,E,F) None Explicit (faces, edges, vertices) Smoothness Discrete approx. Continuous (covariance) Exact (NURBS/analytic) Editing Hard (vertex-level) Medium (attribute-level) Easy (parametric) Rendering Rasterization/RT Differentiable splatting Rendering engines From images Multi-View Stereo 3DGS training Reverse engineering To images Standard pipeline Direct rendering CAD rendering Thin structures Can represent Bloated artifacts Exact boundaries File format OBJ/PLY/STL/FBX PLY (custom) STEP/IGES/ Parasolid Physical sim Ready Needs mesh extraction Native Section 1: Mesh → 3DGS Conversion 1.1 Why Convert Mesh to Gaussians? Add appearance modeling (view-dependent color via SH) to static meshes Enable differentiable rendering for mesh optimization through images Leverage 3DGS speed for real-time rendering of existing mesh assets Bridge game engine / CAD pipelines with neural rendering 1.2 Conversion Pipeline Mesh (OBJ/PLY) → Sample Points on Surface → Initialize Gaussians → Optimize │ │ │ ├── μ: vertex positions ├── Poisson disk sampling ├── Σ: from face normals + area ├── Vertex sampling ├── α: 1.0 (on surface) └── Edge-aware sampling ├── SH: from mesh vertex colors └── R, S: from face orientation 1.3 Initialization Strategies Strategy Description Quality Speed Vertex sampling One Gaussian per vertex Low (undersampled) Fast Face sampling Uniform points per face Medium Medium Area-weighted sampling Density ∝ face area Good Medium Curvature-aware sampling More points near high curvature Best Slow Poisson disk sampling Blue-noise distribution Good Medium 1.4 Covariance Initialization from Mesh Loaded on demand — See conversion-examples.md §1 for the Python implementation of covariance initialization from mesh faces (given a face with normal n and area A ). 1.5 Known Issues in Mesh→3DGS Issue Symptom Fix Floating artifacts Gaussians drift off surface Add normal consistency loss Thick surfaces Scale in normal direction too large Clamp normal scale to small value Missing thin parts Pruned during density control Reduce prune threshold for mesh-initialized Color bleeding SH degree too high on flat surfaces Start with SH degree 0, increase gradually Non-watertight mesh Holes cause rendering gaps Pre-process: fill holes with Poisson reconstruction Section 2: 3DGS → Mesh Extraction 2.1 Why Extract Mesh from 3DGS? Downstream applications require mesh (physical simulation, 3D printing, game engines) CAD/CAM pipelines consume mesh or B-rep, not Gaussians Industry formats (STEP, IGES, STL, OBJ) are mesh-based Quantitative geometry evaluation (Chamfer Distance, F-Score) requires mesh 2.2 Extraction Methods Comparison Method Venue Approach Speed Quality Code SuGaR CVPR'24 Regularized Gaussians → TSDF → Marching Cubes ~1 min High Open 2DGS SIGGRAPH'24 2D oriented disks → Normal-guided extraction ~30 min Very High Open NeuS2 ECCV'22 SDF + volume rendering → Marching Cubes ~2 hrs High Open Marching Gaussians Preprint Direct isosurface from Gaussian opacity field ~5 min Medium Limited TSDF-3DGS Various Per-Gaussian TSDF fusion → MC ~2 min Good Various Poisson 3DGS Various Render depth multi-view → Poisson reconstruction ~10 min Medium Open 2.3 SuGaR Pipeline (Recommended) Trained 3DGS │ ├── Step 1: Regularize Gaussians │ ├── Add normal consistency loss │ └── Constrain Gaussians near surface │ ├── Step 2: Extract TSDF │ ├── Rasterize Gaussian opacity to depth + normal maps │ ├── Multi-view TSDF fusion (VolumetricFusion) │ └── TSDF volume at target resolution (256³ or 512³) │ └── Step 3: Marching Cubes ├── Extract triangle mesh from TSDF └── Optional: mesh simplification / texturing 2.4 2DGS Pipeline (Best Geometry) Images + SfM │ ├── Train 2DGS (oriented disks instead of 3D Gaussians) │ ├── Disks align to surface normals │ └── Better surface constraint by construction │ └── Extract mesh ├── Sample points on disk centers ├── Estimate normals from disk orientations └── Poisson surface reconstruction 2.5 Geometry Quality Evaluation After extraction, evaluate mesh quality: Metric Tool What It Measures Chamfer Distance (CD) Open3D / PyTorch3D Average distance to GT mesh F-Score @ threshold Custom Precision-recall of surface points Normal Consistency Open3D Angle between estimated and GT normals Mesh watertightness PyMeshLab / Trimesh Whether mesh is manifold + closed Edge ratio PyMeshLab Triangle quality (ideal = equilateral) Loaded on demand — See conversion-examples.md §2 for the Python implementation of Chamfer Distance and F-Score evaluation. Section 3: Mesh-Adsorbed & Hybrid Representations 3.1 Why Hybrid? Pure 3DGS: great rendering, poor topology/geometry. Pure mesh: great topology, limited appearance/real-time rendering. Hybrid: best of both worlds. 3.2 Key Hybrid Methods MaGS (Mesh-adsorbed Gaussian Splatting) — ICCV 2025 Aspect Detail Core idea Gaussians "adsorbed" onto mesh vertices, mesh guides Gaussian placement Advantage Mesh provides topology + deformation handle; Gaussians provide appearance Rendering Gaussian splatting with mesh-based culling and sorting Deformation Deform mesh → Gaussians follow automatically Best for Animated/ deformable objects, physical simulation + neural rendering UniMGS (Unified Mesh and 3DGS) — AAAI 2026 Aspect Detail Core idea Single-pass rasterization for both mesh and Gaussians Advantage Unified rendering pipeline, proxy-based deformation Key innovation Eliminates redundant computation in separate mesh + GS pipelines Best for Real-time applications needing both mesh and appearance 2DGS (2D Gaussian Splatting) — SIGGRAPH 2024 Aspect Detail Core idea Replace 3D anisotropic Gaussians with 2D oriented disks Advantage Disks naturally constrain to surface, enabling direct mesh extraction Trade-off Training is more expensive, more prone to VRAM issues Best for Tasks requiring high-quality mesh output 3.3 When to Use Hybrid vs Pure Use Case Recommendation Reason Novel view synthesis only Pure 3DGS Fastest, highest visual quality Need mesh for 3D printing 2DGS or SuGaR Best geometry extraction Animated character + real-time render MaGS Deformation follows mesh CAD reverse engineering BrepGaussian + mesh Structured output needed Game asset pipeline UniMGS Unified single-pass rendering Large-scale scene (city) Pure 3DGS + post-extraction Scalability Section 4: CAD Reverse Engineering with 3DGS 4.1 The CAD RE Pipeline Physical Object │ ├── 3D Scanning (LiDAR / Photogrammetry) │ │ │ ▼ │ Images / Point Cloud │ │ │ ├── 3DGS Training → High-fidelity appearance model │ │ │ ├── Mesh Extraction (SuGaR / 2DGS) │ │ │ │ │ ▼ │ │ Triangle Mesh │ │ │ │ │ ├── Mesh simplification │ │ ├── Mesh segmentation │ │ ├── Primitive fitting (planes, cylinders, cones) │ │ │ │ │ ▼ │ │ B-rep / Parametric CAD │ │ │ │ │ ▼ │ │ STEP / IGES File │ │ │ └── Direct B-rep extraction (BrepGaussian) │ └── CAD Model Ready for Manufacturing 4.2 BrepGaussian (CVPR 2026) — Direct CAD from Images Aspect Detail Problem Traditional RE: mesh → B-rep is a two-stage process with error accumulation Innovation Gaussian Splatting + B-rep reconstruction in a unified framework B-rep components Trimmed surfaces (NURBS), edges (curves), vertices Key mechanism Gaussians provide dense geometric prior; B-rep extraction constrained by Gaussian geometry Output Parametric CAD model (STEP-compatible) Limitations Struggles with: textureless regions, thin structures, high specular, heavy occlusion + sparse views 4.3 Mesh → B-rep Conversion Methods Method Approach Automation Quality Feature-based (CAD software) Detect geometric features → fit primitives Semi-auto High Deep learning (BrepNet, CSGNet) Predict primitives from point cloud / mesh Auto Medium Sketch-based Extract edge network → fit curves/surfaces Semi-auto High BrepGaussian End-to-end from images via 3DGS prior Auto Medium-High 4.4 Primitive Fitting for CAD Reverse Engineering Common CAD primitives to detect: Primitive Parameters Detection Method Plane (n, d) — normal + offset RANSAC Sphere (c, r) — center + radius RANSAC Cylinder (axis, radius, extent) RANSAC + normal clustering Cone (apex, axis, angle) RANSAC Torus (center, axis, R, r) RANSAC Free-form surface NURBS control points Least-squares fitting Loaded on demand — See conversion-examples.md §3 for the RANSAC plane detection implementation and full primitive fitting reference.