HeteroArch-GS: Aerial-Ground Mesh-guided Gaussian Splatting for Heterogeneous Architectural Landmarks with a Real-World Dataset

Abstract

Supplementing aerial photogrammetric models with ground-level imagery to recover fine detail has become standard practice for large-scale landmark reconstruction. Yet aerial-ground fusion under 3D Gaussian Splatting (3DGS) remains fundamentally challenged. Extreme viewpoint and scale disparities, exacerbated by the irregular topologies of heterogeneous architectures, provoke severe optimization conflicts that naive joint training cannot reconcile, yielding geometric collapse and pervasive blur.

In this paper, we propose HeteroArch-GS, a mesh-guided framework that uses oblique photogrammetric meshes as geometric priors to guide 3DGS optimization for aerial-ground reconstruction of heterogeneous landmarks. Specifically, we convert the aerial mesh into continuous differentiable geometric fields that jointly regularize the position, orientation, and anisotropic shape of Gaussian primitives throughout optimization. Together with mesh-guided anchor initialization and multi-directional pseudo-view supervision, these priors drive the Gaussian distribution toward surface-aligned representations, ensuring high-fidelity rendering across both aerial and ground viewpoints.

To support this, we construct AGC Landmarks, a novel real-world RGB dataset capturing diverse heterogeneous landmarks with aerial, ground, and object-centric perspectives. Extensive experiments on AGC Landmarks demonstrate that HeteroArch-GS matches the prior state-of-the-art on in-distribution rendering while clearly surpassing it on out-of-distribution object-centric views. Geometrically, our method achieves the highest F1-scores across all scenes. These results confirm that mesh-guided priors produce more compact and geometrically faithful reconstructions.

Challenges

• Naive aerial-ground 3DGS optimization is unstable: aerial-only and ground-only settings may work within their own view domains, but joint optimization can degrade both domains. Heterogeneous landmarks amplify this conflict because overhangs, hollow spaces, thin components, and non-Lambertian materials generate ambiguous or inconsistent photometric gradients.

• Real-world aerial-ground benchmarks remain insufficient: existing evaluations still rely heavily on synthetic cities, where aerial and street-level views share simplified geometry and controlled radiometric states. Practical landmark captures instead involve temporal illumination changes, local occlusions, irregular structures, complex materials, noise, and dynamic disturbances, leaving a substantial gap between benchmark performance and deployable aerial-ground reconstruction.

Contributions

• We construct AGC Landmarks, a real-world optical image dataset for heterogeneous architectural landmarks, explicitly covering aerial, ground, and object-centric perspectives. Beyond benchmarking neural rendering on complex topologies, it exposes real illumination changes across unconstrained outdoor captures and provides a concrete basis for studying lighting models in the wild.

• We propose HeteroArch-GS, a mesh-guided framework for aerial-ground 3DGS. Building on an existing 3DGS backbone, it turns the existing oblique photogrammetric mesh into a strong geometric and textural prior through anchor initialization, surface-aware regularization, and pseudo-view supervision. This keeps Gaussian primitives tied to plausible physical surfaces and substantially improves rendering from out-of-distribution object-centric views.

• We introduce an Efficient Lazy Loading Strategy that alleviates the memory bottleneck of large aerial and ground image collections, enabling joint aerial-ground 3DGS optimization under limited computational resources.

AGC Landmarks

We collected 10 building-scale landmarks, with an emphasis on architectural diversity and structural heterogeneity. These targets cover six representative types: hollow buildings, large venues, irregular envelopes, ancient buildings, castle, and sculpture. Together, these scenes stress reconstruction in multiple ways. Their hollow layouts and dense decorative details create severe self-occlusion, while reflective glass, curved roofs, lakeside backgrounds, and cliff-side terrain further challenge robust geometry and appearance modeling. For each landmark, we acquire three complementary image subsets:

• Aerial imagery : captured along oblique photogrammetry flight paths to cover the entire survey area;

• Ground imagery : captured horizontally and at slight upward angles from a specific height above the ground to simulate pedestrian viewpoints;

• Object-centric imagery : captured via dense, close-range photogrammetry orbiting the target to ensure complete capture of intricate details from multiple perspectives.

HeteroArch-GS

HeteroArch-GS starts from an existing photogrammetric mesh and oriented oblique aerial images. To complement the aerial observations, we additionally capture ground images with RTK positioning. These ground images are further refined through PPK correction in DJI Terra and integrated with the aerial images, yielding a unified aerial-ground triangulation result. The mesh then serves as an explicit proxy for scene geometry and appearance, guiding three core components: Mesh-Guided Anchor Initialization, Mesh-Guided Geometric Regularization, and Mesh-Guided Pseudo-View Supervision. During optimization, color supervision and geometric regularization jointly guide the Gaussian primitives to recover faithful appearance while staying close to the underlying scene structure. To train under limited hardware memory, we introduce an on-demand data fetching mechanism that keeps only a fixed-capacity CPU cache and loads images as needed for fast retrieval.

Rendering Results

We evaluate our method against several state-of-the-art baselines including 3DGS, 2DGS, PGSR, Scaffold-GS, Octree-GS, and Horizon-GS. Our method consistently produces sharper details, fewer artifacts, and better geometric fidelity.

Runtime and Memory Efficiency

To break capacity-throughput trade-off and efficiently handle large-scale dataset, we introduce a streaming architecture termed efficient lazy loading. Our method significantly reduces runtime and decreases both GPU and CPU memory consumption.