Semi-Supervised Adversarial Recognition of Refined Window Structures for Inverse Procedural Facade Modeling

Abstract

Deep learning methods are notoriously data hungry, which requires a large amount of labeled samples. Unfortunately, the large amount of interactive sample labeling efforts has greatly hindered the application of deep learning methods, especially for 3D modeling tasks, which requires heterogeneous samples. To alleviate the work of data annotation for learned 3D modeling of facade, this paper proposed a semi-supervised adversarial recognition methods for the inverse procedural modeling. Beginning with textured LOD-2 (level of details) models, we use the classical constitutional neural networks to recognize the types and estimate the parameters of windows, from the image patches. The window types and parameters are assembled into a procedural grammar. A simple procedural engine is built inside an existing 3D modeling software, which produces fine-grained window geometries. In order to obtain a useful model from few labeled samples, we leverage the generate adversarial network to train the feature extractor semi-supervisedly. The adversarial training strategy can also exploit the large amount of unlabeled data to make the training phase more stable. Experiments using publicly available facade image dataset reveals that the proposed training strategy can obtain about 10% improvement in classification accuracy and 50% improvement in parameter estimation under the same network structure. In addition, performance gains are more pronounced when testing against unseen data featuring different facade styles.

Pipeline

Results

Reconstruction of Single Buildings

*without group processing* *with group processing* *without group processing* *with group processing* *without group processing* *with group processing* *without group processing* *with group processing* *without group processing* *with group processing* *without group processing* *with group processing*

Reconstruction of Blocks

*without group processing* *with group processing* *without group processing* *with group processing*

Rendering Results

Modeling Processes

Batch Editing

The video demonstrations are also available in Bilibili.

LOD Creation

The LOD creation video demonstrations are also available in Bilibili.

Window Generation