Academic Positions

  • now 2019

    Professor

    Faculty of Geosciences and Environmental Engineering
    Southwest Jiaotong University

  • 2019 2015

    Postdoctoral Fellow

    Land Surveying and Geo-Informatics
    The Hong Kong Polytechnique University

  • 2013 2011

    Research Assistant

    Land Surveying and Geo-Informatics
    The Hong Kong Polytechnique University

Education & Training

  • 2015 2010

    PhD. student

    State Key Laboratory of Surveying Mapping and Remote Sensing
    Wuhan University

  • 2010 2006

    Undergraduate student

    School of Remote Sensing and Information Enginnering
    Wuhan University

Honors, Awards and Grants

  • 2019
    Award of Advances of Geo-Information in China, First Place (地理信息科技进步一等奖)

    Awarded to our project "Aerial-Ground Integration for the Efficient Detection of Illegle Building Constructions", rank 3/10

  • 2018
    Award of Advances of Geo-Information in China, First Place (地理信息科技进步一等奖)

    Awarded to our project "Photorealistic 3D GIS and its Applications in the Mountainous City", rank 2/10

  • 2018
    The Emerging Star Award of GIS by the 7th College GIS Forum (高校GIS新锐奖)

    This award is granted to the top 10 youth college staffs in the GIS field.

  • 2016
    Special Merit Award (R. Alekseev Award) and Gold Medal of the 44th International Exhibition of Inventions of Geneva

    The International Exhibition of Inventions of Geneva is one of the most important specialized and large-scale invention exhibitions in the world. It is supported by the Swiss Federal Government, the State and the City of Geneva, as well as the World Intellectual Property Organization. The exhibition this year attracted more than 700 producers and inventors from 45 countries and over 60000 visitors during the five days exhibition from April 13 to 17, 2016.

    The award is presented to our project, "Precise Topographic Mapping Model", in which we developped a integrated combined adjustment model for the lunar topographic mapping. This method has been used in China's previous lunar exploration missions. Please see our papers:

    Wu, B., Hu, H.*, Guo, J., 2014. Integration of Chang'E-2 imagery and LRO laser altimeter data with a combined block adjustment for precision lunar topographic modeling. Earth and Planetary Science Letters 391, 1-15.

  • 2016
    The Second Honorable Mention for the 2016 Talbert Abrams Award

    The purpose of the Award is to encourage the authorship and recording of current, historical, engineering, and scientific developments in photogrammetry. Three papers from 12 issues of the 2015 PE&RS journal are selected. The paper being awarded is:

    Hu, H.*, Zhu, Q., Du, Z., Zhang, Y., Ding, Y., 2015. Reliable spatial relationship constrained feature point matching of oblique aerial images. Photogrammetric Engineering and Remote Sensing 81 (1), 49-58.

  • Sept. 2014
    National Scholarship
    Rank 2 out of more than 50 PhD. students in the State Key Laboratory of Surveying Mapping and Remote Sensing for the National Scholarship award.
  • Mar. 2014
    John I. Davidson President’s Award
    This Award is presented by ASPRS with funding provided by the ASPRS Foundation to to encourage and commend those who publish papers of practical or applied value in PE&RS, the official journal of ASPRS.

    This award is granted to our paper: Wu, B., Hu, H., Zhu, Q., Zhang, Y., 2013. A flexible method for zoom lens calibration and modeling using a planar checkerboard. Photogrammetric engineering and remote sensing 79 (6), 555-571.
  • Sept. 2010
    The Freshman Scholarship
    Rank 1st in the graduate entrance examination for the State Key Laboratory of Surveying Mapping and Remote Sensing. And this shcolarship is granted to six out of more than 100 graduate students.
  • Sept. 2008
    Kwang-Hua Scholarship
    Top 30% of the undergraduate students in School of Remote Sensing and Information Enginnering.

Current Courses

  • 2020 spring

    Basis of Photogrammetry

    41 Undergraduate Student, 17 weeks, SWJTU

Research Projects

  • image

    Research project

    • 2024 2021

      National Natural Science Foundation of China (自然科学基金面上项目), PI

      Binary Integer Programming for Building Reconstruction in Complex Urban Environment

    • 2022 2019

      National Key R&D Program of China (国家重点研发计划子课题), PI

      Precision DEM Reconstruction using InSAR for Rapid Emergence Response

    • 2021 2017

      National Natural Science Foundation of China (自然科学基金青年项目), PI

      A supervised metric learning approach for efficient aerial oblique image matching using neighborhood information,

    • 2021 2017

      National Natural Science Foundation of China (自然科学基金重点项目), Co-I

      Theory and methods of the oblique photogrammetry for precision building reconstruction,

    • 2018 2016

      Open Fund of Key Laboratory of Urban Land Resources Monitoring and Simulation, Ministry of Land and Resources, PI

      LOD reconstruction of buildings from oblique images,

Filter by year:

Sort by year:

Structure-Aware Completion of Photogrammetric Meshes in Urban Road Environment

Qing Zhu, Qishen Shang, Han Hu*, Haojia Yu, Ruofei Zhong, 2020.
Journal Paper (preprint)ISPRS Journal of Photogrammetry and Remote Sensing

Abstract

Photogrammetric mesh models obtained from aerial oblique images have been widely used for urban reconstruction. However, the photogrammetric meshes also suffer from severe texture problems, especially on the road areas due to occlusion. This paper proposes a structure-aware completion approach to improve the quality of meshes by removing undesired vehicles on the road seamlessly. Specifically, the discontinuous texture atlas is first integrated to a continuous screen space through rendering by the graphics pipeline; the rendering also records necessary mapping for deintegration to the original texture atlas after editing. Vehicle regions are masked by a standard object detection approach, e.g. Faster RCNN. Then, the masked regions are completed guided by the linear structures and regularities in the road region, which is implemented based on Patch Match. Finally, the completed rendered image is deintegrated to the original texture atlas and the triangles for the vehicles are also flattened for improved meshes. Experimental evaluations and analyses are conducted against three datasets, which are captured with different sensors and ground sample distances. The results reveal that the proposed method can quite realistic meshes after removing the vehicles. The structure-aware completion approach for road regions outperforms popular image completion methods and ablation study further confirms the effectiveness of the linear guidance. It should be noted that the proposed method is also capable to handle tiled mesh models for large-scale scenes. Dataset and code are available at this URL.

Multientity Registration of Point Clouds for Dynamic Objects on Complex Floating Platform Using Object Silhouettes

Wang, F., Hu, H.*, Ge, X., Xu, B., Zhong, R., Ding, Y., Xie, X., Zhu, Q.*, 2020.
Journal PaperIEEE Transactions on Geoscience and Remote Sensing.

Abstract

This article is focused on a challenging topic emerging from the registration of point clouds, specifically the registration of dynamic objects with low overlapping ratio. This problem is especially difficult when the static scanner is installed on a floating platform, and the objects it scans are also floating. These issues make most of the automatic registration methods and software solutions invalid. To solve this problem, explicit exploration of the static region is necessary for both the coarse and fine registration steps. Fortunately, determining the corresponding regions can be eased by the intuitive realization that in urban environments, natural objects neither present straight boundaries nor stack vertically. This intuition has guided the authors to develop a robust approach for the detection of static regions using planar structures. Then, silhouettes of the objects are extracted from the planar structures, which assist in the determination of an SE(2) transformation in the horizontal direction by a novel line matching method. The silhouettes also enable identification of the correspondences of planes in the step of fine registration using a variant of the iterative closest point method. Experimental evaluations using point clouds of cargo ships with different sizes and shapes reveal the robustness and efficiency of the proposed method, which gives 100% success and reasonable accuracy in rapid time, suitable for an online system. In addition, the proposed method is evaluated systematically with regard to several practical situations caused by the floating platform, and it demonstrates good robustness to limited scanning time and noise.

Topographic and Geomorphological Mapping and Analysis of the Chang’E-4 Landing Site on the Far Side of the Moon

Wu, Bo and Li, Fei and Hu, Han and Zhao, Yang and Wang, Yiran and Xiao, Peipei and Li, Yuan and Liu, Wai Chung and Chen, Long and Ge, Xuming and others, 2020
Journal PaperPhotogrammetric Engineering and Remote Sensing

Abstract

The Chinese lunar probe Chang’E-4 successfully landed in the Von Kármán crater on the far side of the Moon. This paper presents the topographic and geomorphological mapping and their joint analysis for selecting the Chang’E-4 landing site in the Von Kármán crater. A digital topographic model (DTM) of the Von Kármán crater, with a spatial resolution of 30 m, was generated through the integrated processing of Chang’E-2 images (7 m/pixel) and Lunar Reconnaissance Orbiter (LRO) Laser Altimeter (LOLA) data. Slope maps were derived from the DTM. Terrain occlusions to both the Sun and the relay satellite were studied. Craters with diameters ≥ 70 m were detected to generate a crater density map. Rocks with diam- eters ≥ 2 m were also extracted to generate a rock abundance map using an LRO narrow angle camera (NAC) image mosaic. The joint topographic and geomorphological analysis identi- fied three subregions for landing. One of them, recommended as the highest-priority landing site, was the one in which Chang’E-4 eventually landed. After the successful landing of Chang’E-4, we immediately determined the precise location of the lander by the integrated processing of orbiter, descent and ground images. We also conducted a detailed analysis around the landing location. The results revealed that the Chang’E-4 lander has excellent visibility to the Sun and relay satellite; the lander is on a slope of about 4.5° towards the southwest, and the rock abundance around the landing location is al- most 0. The developed methods and results can benefit future soft-landing missions to the Moon and other celestial bodies.

A semantic enhancement method for photorealistic mesh model based on local parameterization (局部表面参数化的实景三角网模型语义增强方法)

WANG Libin, HU Han*, ZHU Qing, DING Yulin, CHEN Min, 2020
Journal PaperActa Geodaetica et Cartographica Sinica (测绘学报)

Abstract

With the advances in structure-from-motion and multi-view stereo, state-of-the-art oblique photogrammetric solutions can obtain city-scale photorealistic mesh models automatically. However, the mesh models are lack of fine geometric structure and semantic free. Aiming at solving this issue, it is proposed that a semantic enhancement method for photorealistic mesh models based on local surface parametrization. The basic idea behind the proposed method is that, through the representation of surface tree, it is converted that the seamless fusion of semantic components and photogrammetric mesh models to a replacing operation in a local area. The two 3D models are parametrized to 2D space in the local region and seamless merged and replaced in the UV space by 2D constrained delaunay triangulation (CDT). The replaced semantic components are then reversed transform to 3D space, which finalize the semantic enhancement automatically. Experiments on oblique images in Shenzhen reveal that the proposed method can effectively realize the automatic seamless fusion of semantic components with an open boundary and photorealistic mesh models. Compared with the commercial software Maya, based on the method of insertion and fusion, the proposed method has practical value for improving modeling efficiency.

Leveraging Photogrammetric Mesh Models for Aerial-Ground Feature Point Matching Toward Integrated 3D Reconstruction

Qing Zhu, Zhendong Wang, Han Hu*, Linfu Xie, Xuming Ge, Yeting Zhang
Journal PaperISPRS Journal of Photogrammetry and Remote Sensing

Abstract

Integration of aerial and ground images has been proved as an efficient approach to enhance the surface reconstruction in urban environments. However, as the first step, the feature point matching between aerial and ground images is remarkably difficult, due to the large differences in viewpoint and illumination conditions. Previous studies based on geometry-aware image rectification have alleviated this problem, but the performance and convenience of this strategy are still limited by several flaws, e.g. quadratic image pairs, segregated extraction of descriptors and occlusions. To address these problems, we propose a novel approach: leveraging photogrammetric mesh models for aerial-ground image matching. The methods have linear time complexity with regard to the number of images. It explicitly handles low overlap using multi-view images. The proposed methods can be directly injected into off-the-shelf structure-from-motion (SFM) and multi-view stereo (MVS) solutions. First, aerial and ground images are reconstructed separately and initially co-registered through weak georeferencing data. Second, aerial models are rendered to the initial ground views, in which color, depth and normal images are obtained. Then, feature matching between synthesized and ground images are conducted through descriptor searching and geometry-constrained outlier removal. Finally, oriented 3D patches are formulated using the synthesized depth and normal images and the correspondences are propagated to the aerial views through patch-based matching. Experimental evaluations using five datasets reveal satisfactory performance of the proposed methods in aerial-ground image matching, which succeeds in all of the ten challenging pairs compared to only three for the second best. In addition, incorporation of existing SFM and MVS solutions enables more complete reconstruction results, with better internal stability.

Deep Fusion of Local and Non-Local Features for Precision Landslide Recognition

Qing Zhu, Lin Chen*, Han Hu*, Binzhi Xu, Yeting Zhang, Haifeng Li*
Journal Paper PreprintIEEE GRSL

Abstract

Precision mapping of landslide inventory is crucial for hazard mitigation. Most landslides generally co-exist with other confusing geological features, and the presence of such areas can only be inferred unambiguously at a large scale. In addition, local information is also important for the preservation of object boundaries. Aiming to solve this problem, this paper proposes an effective approach to fuse both local and non-local features to surmount the contextual problem. Built upon the U-Net architecture that is widely adopted in the remote sensing community, we utilize two additional modules. The first one uses dilated convolution and the corresponding atrous spatial pyramid pooling, which enlarged the receptive field without sacrificing spatial resolution or increasing memory usage. The second uses a scale attention mechanism to guide the up-sampling of features from the coarse level by a learned weight map. In implementation, the computational overhead against the original U-Net was only a few convolutional layers. Experimental evaluations revealed that the proposed method outperformed state-of-the-art general-purpose semantic segmentation approaches. Furthermore, ablation studies have shown that the two models afforded extensive enhancements in landslide-recognition performance.

Fast and Regularized Reconstruction of Building Façades from Street-View Images using Binary Integer Programming

Han Hu*, Libin Wang, Yulin Ding, Qing Zhu
Conference PaperISPRS Congress 2020

Abstract

Regularized arrangement of primitives on building façades to aligned locations and consistent sizes is important towards structured reconstruction of urban environment. Mixed integer linear programing was used to solve the problem, however, it is extreamly time consuming even for state-of-the-art commercial solvers. Aiming to alleviate this issue, we cast the problem into binary integer programming, which omits the requirements for real value parameters and is more efficient to be solved . Firstly, the bounding boxes of the primitives are detected using the YOLOv3 architecture in real-time. Secondly, the coordinates of the upper left corners and the sizes of the bounding boxes are automatically clustered in a binary integer programming optimization, which jointly considers the geometric fitness, regularity and additional constraints; this step does not require \emph{a priori} knowledge, such as the number of clusters or pre-defined grammars. Finally, the regularized bounding boxes can be directly used to guide the façade reconstruction in an interactive envinronment. Experimental evaluations have revealed that the accuracies for the extraction of primitives are above 0.85, which is sufficient for the following 3D reconstruction. The proposed approach only takes about 10% to 20% of the runtime than previous approach and reduces the diversity of the bounding boxes to about 20% to 50%.

Object-based incremental registration of terrestrial point clouds in an urban environment

X. Ge and H. Hu*, 2020
Journal PapersISPRS Journal of Photogrammetry and Remote Sensing

Abstract

Registration of terrestrial point clouds is essential for large-scale urban applications. The robustness, accuracy, and runtime are generally given the highest priority in the design of appropriate algorithms. Most approaches that target general scenarios can only fulfill some of these factors, that is, robustness and accuracy come at the cost of increased runtime and vice versa. This paper proposes an object-based incremental registration strategy that accomplishes all of these objectives without the need for artificial targets, aiming at a specific scenario, the urban environment. The key is to decompose the degrees of freedom for the SE(3) transformation to three separate but closely related steps, considering that scanners are generally leveled in urban scenes: (1) 2D transformation with matches from line primitives, (2) vertical offset compensation by robust least-squares optimization, and (3) full SE(3) least-squares refinement using uniformly selected local patches. The robustness is prioritized in the whole pipeline, as structured first by a primitive-based registration and two least-squares optimizations with robust estimations that do not require specific keypoints. An object-based strategy for terrestrial point clouds is used to increase the reliability of the first step by the line primitives, which significantly reduces the search space without affecting the recall ratio. The least-squares optimization contributes to achieve a global optimum for the accurate registration. The three coupling steps are also more efficient than segregated coarse-to-fine registration. Experimental evaluations for point clouds acquired in both a metropolis and in old-style cities reveal that the proposed methods are superior to or on par with the state-of-the-art in robustness, accuracy, and runtime. In addition, the methods are also agnostic to the primitives adopted.

Interactive Correction of a Distorted Street-View Panorama for Efficient 3D Façade Modeling

Zhu, Q., Zhang, M., Hu, H.*, Wang, F., 2020
Journal PapersIEEE Geoscience and Remote Sensing Letters

Abstract

Facade features are important in large-scale LoD-3 reconstruction in urban environments, and street-view panoramas are arguably the best option for detailed 3D fac ¸ade modeling. However, despite the plethora of street-view panoramas available, few studies have explored the metric capabilities of panoramas. This is due in part to the complexities of system integration and in part to problems associated with projection (e.g., distortion at the tops of buildings) and deformation (e.g., the bending of straight structures). In an effort to solve these problems, this study introduces a flexible and practical solution using only a single panorama. The key is to efficiently rectify panoramas using image-space line constrained deformation inspired by the as-rigid-as-possible deformation of surface meshes. The image is then re-projected using gnomonic projection on a properly selected tangent plane. The proposed approach requires a reason- able amount of user interaction to select and position the vertical line segments. The tangent point is also chosen empirically for each panorama. The rectified images can then be imported into off-the-shelf 3D modeling solutions as reference images for interactive sketching. Experimental evaluations reveal the effectiveness of the image-space rectification: after proper scaling, the semantic-aware 3D fac ¸ade models achieve decimeter-level accuracy with respect to the reference surface mesh.

A Multi-Primitive-Based Hierarchical Optimal Approach for Semantic Labeling of ALS Point Clouds

Ge, X., Wu, B., Li, Y., Hu, H., 2019.
Journal PapersRemote Sensing 11 (10), 1243.

Abstract

There are normally three main steps to carrying out the labeling of airborne laser scanning (ALS) point clouds. The first step is to use appropriate primitives to represent the scanning scenes, the second is to calculate the discriminative features of each primitive, and the third is to introduce a classifier to label the point clouds. This paper investigates multiple primitives to effectively represent scenes and exploit their geometric relationships. Relationships are graded according to the properties of related primitives. Then, based on initial labeling results, a novel, hierarchical, and optimal strategy is developed to optimize semantic labeling results. The proposed approach was tested using two sets of representative ALS point clouds, namely the Vaihingen datasets and Hong Kong’s Central District dataset. The results were compared with those generated by other typical methods in previous work. Quantitative assessments for the two experimental datasets showed that the performance of the proposed approach was superior to reference methods in both datasets. The scores for correctness attained over 98% in all cases of the Vaihingen datasets and up to 96% in the Hong Kong dataset. The results reveal that our approach of labeling different classes in terms of ALS point clouds is robust and bears significance for future applications, such as 3D modeling and change detection from point clouds.

Impact cratering in and around the Orientale Basin: Results from recent high-resolution remote sensing datasets

Wu, B., Wang, Y., Lin, T., Hu, H., Werner, S.C., 2019
Journal PapersIcarus 333, 343-355.

Abstract

Impact craters are the predominant geological features on the lunar surface. Data about the craters are crucial for inferring information about surface age, the generation processes of the geological units, and the sequences of geological events. The higher-resolution remote sensing datasets collected by recent lunar missions enable the investigation of impact craters of smaller size and provide more accurate information. This paper presents an investigation of the distribution and population characteristics of impact craters in and around the Orientale Basin based on high-resolution datasets. First, an update to the crater catalogue for the Orientale Basin and its surrounding area is provided, including craters as small as 1 km in diameter. Based on the updated crater catalogue, the crater densities and depth-to-diameter ratios in and around the Orientale Basin are investigated. The inclusion of small craters enables a crater density map with higher resolution, revealing a significantly higher crater density of the study area. Also, the surface age of Orientale Basin is estimated from the size–frequency distribution (SFD) using the new crater catalogue, showing an age of 3.75 Ga using the production function of Neukum et al. (2001), which is in good agreement with previous studies. Finally, distribution patterns of secondary craters around the Orientale Basin are investigated, indicating the Orientale Basin may be caused by an oblique impact with a downrange direction of about 235°–260° and an offset strength towards the direction of about 305°–350°.

Color balancing and geometrical registration of high-resolution planetary imagery for improved orthographic image mosaicking

Hu, H., Wu, B.*, Chen, L., 2019
Journal PapersPlanetary and Space Science https://doi.org/10.1016/j.pss.2019.104719

Abstract

High-resolution orthographic image mosaics of planetary surfaces are an important prerequisite for a range of scientific research and applications. Because images are collected under different conditions, the individual images will inevitably present color differences and geometrical misalignments. Current color balancing methods generally adopt a linear model, including gain and offset values, based on a reference image. However, global color consistency may be severely influenced by the reference image, and in complex scenarios the linear model may not be able to remedy the color differences. This paper presents a novel color balancing approach that does not require a reference image. It incorporates a novel regularization term in the derivative of the color transferring model, which guaranties minimal contrast variation of the color model and removes the ambiguity of the nullspace of the optimization. In addition, to adapt to more complex color differences, a spline model in the color space is proposed in place of the linear model, and a special parametrization of the spline is exploited for least-squares optimization. Based on the two competing goals of minimizing color differences and preserving transfer regularities, the spline model is solved in a single global optimization for all images. Furthermore, an effective geometrical registration method is also used to reduce the misalignments in the object space. Experimental evaluations using two typical image datasets for Mars and the Moon reveal that the proposed approach achieves favorable geometrical consistency. Compared with two off-the-shelf solutions, it both achieves better color consistency and preserves better color contrast.

Planetary3D: a photogrammetric tool for 3d topographic mapping of planetary bodies

H. Hu and B. Wu*, 2019
Conference PapersISPRS Ann. Photogramm. Remote Sens. Spatial Inf. Sci

Abstract

Planetary remote sensing images are the primary datasets for high-resolution topographic mapping and modeling of the planetary surfaces. However, unlike the mapping satellites for Earth observations, cameras onboard the planetary satellites generally present special imaging geometries and configurations, which makes the stereo photogrammetric process difficult and requires a large number of manual interactions. At the Hong Kong Polytechnic University, we developed a unified photogrammetric software system, namely Planetary3D, for 3D topographic mapping modeling of various planetary bodies using images collected by various sensors. Planetary3D consists of three modules, including: (1) the pre-processing module to deliver standardized image products, (2) the bundle adjustment module to alleviate the inconsistencies between the images and possibly the reference frame, and (3) the dense image matching module to create pixel-wise image matches and produce high quality topographic models. Examples of using three changeling datasets, including the MRO CTX, MRO HiRISE and Chang’E-2 images, have revealed that the automatic pipeline of Planetary3D can produce high-quality digital elevation models (DEMs) with favorable performances. Notably, the notorious jitter effects visible on HiRISE images can be effectively removed and good consistencies with the reference DEMs are found for the test datasets by the Planetary3D pipeline.

Image-Guided Registration of Unordered Terrestrial Laser Scanning Point Clouds for Urban Scenes

Ge, X., Hu, H.*, Wu, B., 2019
Journal PapersIEEE Transactions on Geoscience and Remote Sensing https://doi.org/10.1109/TGRS.2019.2925805

Abstract

This paper presents an image-guided end-to-end registration approach for globally consistent 3-D registration of unordered terrestrial laser scanning (TLS) point clouds. The proposed method can handle arbitrary point clouds with reasonable pairwise overlap without knowledge about their initial position and orientation, without requiring artificial targets, and without needing to record the order of the scanning. One of the novel contributions of the proposed approach lies in the optimization of a scanning network. We retrieve the similarities of all scans based on a vocabulary tree using both the geometrically rectified panorama images and the corresponding 3-D point clouds. The approach also highlights the integral optimization in both the coarse and fine registration. A pose graph is introduced to realize global optimization at the end of the coarse step without primitives. After that, the results act as the inputs to start the pairwise fine registration, which is then followed by the minimum loop expansion (MLE) refinement. Comprehensive experiments demonstrated network optimization rates of over 60% using the image-guided strategy. Using the pose-graph optimization method, successful registration rates (SRRs) increased to 100% for all tested cases. The MLE not only accelerates the speed of the convergence but also improves registration accuracy, which reached 0.1 m and 0.1° in the translation and rotation angles, respectively.

Precision global DEM generation based on adaptive surface filter and Poisson terrain editing (in Chinese)

HU Han, DING Yulin*, ZHU Qing, JIANG Jie, WEN Xuehu, ZHANG Li, TANG Wei, YANG Jun, ZHONG Ruofei
Journal PapersActa Geodaetica et Cartographica Sinica, 2019, 48(3): 374-383

Abstract

Aerial laser scanning or photogrammetric point clouds are often noisy at building boundaries. In order to produce regularized polygons from such noisy point clouds, this study proposes a hierarchical regularization method for the boundary points. Beginning with detected planar structures from raw point clouds, two stages of regularization are employed. In the first stage, the boundary points of an individual plane are consolidated locally by shifting them along their refined normal vector to resist noise, and then grouped into piecewise smooth segments. In the second stage, global regularities among different segments from different planes are softly enforced through a labeling process, in which the same label represents parallel or orthogonal segments. This is formulated as a Markov random field and solved efficiently via graph cut. The performance of the proposed method is evaluated for extracting 2D footprints and 3D polygons of buildings in metropolitan area. The results reveal that the proposed method is superior to the state-of-art methods both qualitatively and quantitatively in compactness. The simplified polygons could fit the original boundary points with an average residuals of 0.2 m, and in the meantime reduce up to 90% complexities of the edges. The satisfactory performances of the proposed method show a promising potential for 3D reconstruction of polygonal models from noisy point clouds.

Hierarchical Regularization of Building Boundaries in Noisy Aerial Laser Scanning and Photogrammetric Point Clouds

Xie, L., Zhu, Q.*, Hu, H.*, Wu, B., Li, Y., Zhang, Y., Zhong, R., 2018.
Journal Papers Remote Sensing 10 (12), 1996.

Abstract

Aerial laser scanning or photogrammetric point clouds are often noisy at building boundaries. In order to produce regularized polygons from such noisy point clouds, this study proposes a hierarchical regularization method for the boundary points. Beginning with detected planar structures from raw point clouds, two stages of regularization are employed. In the first stage, the boundary points of an individual plane are consolidated locally by shifting them along their refined normal vector to resist noise, and then grouped into piecewise smooth segments. In the second stage, global regularities among different segments from different planes are softly enforced through a labeling process, in which the same label represents parallel or orthogonal segments. This is formulated as a Markov random field and solved efficiently via graph cut. The performance of the proposed method is evaluated for extracting 2D footprints and 3D polygons of buildings in metropolitan area. The results reveal that the proposed method is superior to the state-of-art methods both qualitatively and quantitatively in compactness. The simplified polygons could fit the original boundary points with an average residuals of 0.2 m, and in the meantime reduce up to 90% complexities of the edges. The satisfactory performances of the proposed method show a promising potential for 3D reconstruction of polygonal models from noisy point clouds.

Intact Planar Abstraction of Buildings via Global Normal Refinement from Noisy Oblique Photogrammetric Point Clouds

Zhu, Q., Wang, F.*, Hu, H.*, Ding, Y., Xie, J., Wang, W., Zhong, R., 2018
Journal Papers ISPRS International Journal of Geo-Information 7 (11), 431-452

Abstract

The second column is the normal vector estimated through the proposed method, see the sharp feature preserved on the building edge.

Oblique photogrammetric point clouds are currently one of the major data sources for the three-dimensional level-of-detail reconstruction of buildings. However, they are severely noise-laden and pose serious problems for the effective and automatic surface extraction of buildings. In addition, conventional methods generally use normal vectors estimated in a local neighborhood, which are liable to be affected by noise, leading to inferior results in successive building reconstruction. In this paper, we propose an intact planar abstraction method for buildings, which explicitly handles noise by integrating information in a larger context through global optimization. The information propagates hierarchically from a local to global scale through the following steps: first, based on voxel cloud connectivity segmentation, single points are clustered into supervoxels that are enforced to not cross the surface boundary; second, each supervoxel is expanded to nearby supervoxels through the maximal support region, which strictly enforces planarity; third, the relationships established by the maximal support regions are injected into a global optimization, which reorients the local normal vectors to be more consistent in a larger context; finally, the intact planar surfaces are obtained by region growing using robust normal and point connectivity in the established spatial relations. Experiments on the photogrammetric point clouds obtained from oblique images showed that the proposed method is effective in reducing the influence of noise and retrieving almost all of the major planar structures of the examined buildings.

Block adjustment and coupled epipolar rectification of LROC NAC images for precision lunar topographic mapping

Hu, H. and B. Wu.*, 2018
Journal Papers Planetary and Space Science 160, 26-38

Abstract

Anaglyph on the border area between NAC-L and NAC-R
Example of NAC DEMs
Example of NAC DEMs

The narrow-angle camera (NAC) of the lunar reconnaissance orbiter camera (LROC) uses a pair of closely attached pushbroom sensors to obtain a large swath of coverage while providing high-resolution imaging. However, the two image sensors do not share the same lenses and cannot be modelled geometrically with a single physical model. The irregular image network leads to difficulties in conducting the block adjustment to remove inconsistencies between the NAC images in both intra- and inter-track cases. In addition, the special image network requires two to four stereo models, each with an irregular overlapping region of varying size. The stereo configuration of NAC images also creates severe problems for the state-of-the-art image matching methods. With the aim of using NAC stereo pairs for precision 3D topographic mapping, this paper presents a novel approach to the block adjustment and coupled epipolar rectification of NAC stereo images. This approach removes the internal inconsistencies in a single block adjustment and merges the image pair in the disparity space, thus requiring estimation of only one stereo model. Semi-global matching is used to generate a disparity map for the stereo pair; each correspondence is transformed back to the source image, and 3D points are derived via photogrammetric space intersection. The experimental results reveal that the proposed approach is able to reduce the gaps and inconsistencies caused by the inaccurate boresight offsets between the two NAC cameras and the irregular overlapping regions and to finally generate precise and consistent 3D topographic models from the NAC stereo images.

Illumination invariant feature point matching for high-resolution planetary remote sensing images

Wu, B.*, Zeng, H., Hu, H.
Journal Papers Planetary and Space Science 152, 45-54.

Abstract

Despite its success with regular close-range and remote-sensing images, the scale-invariant feature transform (SIFT) algorithm is essentially not invariant to illumination differences due to the use of gradients for feature description. In planetary remote sensing imagery, which normally lacks sufficient textural information, salient regions are generally triggered by the shadow effects of keypoints, reducing the matching performance of classical SIFT. Based on the observation of dual peaks in a histogram of the dominant orientations of SIFT keypoints, this paper proposes an illumination-invariant SIFT matching method for high-resolution planetary remote sensing images. First, as the peaks in the orientation histogram are generally aligned closely with the sub-solar azimuth angle at the time of image collection, an adaptive suppression Gaussian function is tuned to level the histogram and thereby alleviate the differences in illumination caused by a changing solar angle. Next, the suppression function is incorporated into the original SIFT procedure for obtaining feature descriptors, which are used for initial image matching. Finally, as the distribution of feature descriptors changes after anisotropic suppression, and the ratio check used for matching and outlier removal in classical SIFT may produce inferior results, this paper proposes an improved matching procedure based on cross-checking and template image matching. The experimental results for several high-resolution remote sensing images from both the Moon and Mars, with illumination differences of 20°–180°, reveal that the proposed method retrieves about 40%–60% more matches than the classical SIFT method. The proposed method is of significance for matching or co-registration of planetary remote sensing images for their synergistic use in various applications. It also has the potential to be useful for flyby and rover images by integrating with the affine invariant feature detectors.

Integration of aerial oblique imagery and terrestrial imagery for optimized 3D modeling in urban areas

Wu, B., Xie, L., Hu, H., Zhu, Q., Yau, E., 2018
Journal Papers ISPRS Journal of Photogrammetry and Remote Sensing 139, 119-132.

Abstract

Photorealistic three-dimensional (3D) models are fundamental to the spatial data infrastructure of a digital city, and have numerous potential applications in areas such as urban planning, urban management, urban monitoring, and urban environmental studies. Recent developments in aerial oblique photogrammetry based on aircraft or unmanned aerial vehicles (UAVs) offer promising techniques for 3D modeling. However, 3D models generated from aerial oblique imagery in urban areas with densely distributed high-rise buildings may show geometric defects and blurred textures, especially on building façades, due to problems such as occlusion and large camera tilt angles. Meanwhile, mobile mapping systems (MMSs) can capture terrestrial images of close-range objects from a complementary view on the ground at a high level of detail, but do not offer full coverage. The integration of aerial oblique imagery with terrestrial imagery offers promising opportunities to optimize 3D modeling in urban areas. This paper presents a novel method of integrating these two image types through automatic feature matching and combined bundle adjustment between them, and based on the integrated results to optimize the geometry and texture of the 3D models generated from aerial oblique imagery. Experimental analyses were conducted on two datasets of aerial and terrestrial images collected in Dortmund, Germany and in Hong Kong. The results indicate that the proposed approach effectively integrates images from the two platforms and thereby improves 3D modeling in urban areas.

Geometric Integration of Hybrid Correspondences for RGB-D Unidirectional Tracking

Tang, S., Chen, W., Wang, W., Li, X., Darwish, W., Li, W., Huang, Z., Hu, H., Guo, R., 2018
Journal Papers Sensors 18 (5): 1385.

Abstract

Traditionally, visual-based RGB-D SLAM systems only use correspondences with valid depth values for camera tracking, thus ignoring the regions without 3D information. Due to the strict limitation on measurement distance and view angle, such systems adopt only short-range constraints which may introduce larger drift errors during long-distance unidirectional tracking. In this paper, we propose a novel geometric integration method that makes use of both 2D and 3D correspondences for RGB-D tracking. Our method handles the problem by exploring visual features both when depth information is available and when it is unknown. The system comprises two parts: coarse pose tracking with 3D correspondences, and geometric integration with hybrid correspondences. First, the coarse pose tracking generates the initial camera pose using 3D correspondences with frame-by-frame registration. The initial camera poses are then used as inputs for the geometric integration model, along with 3D correspondences, 2D-3D correspondences and 2D correspondences identified from frame pairs. The initial 3D location of the correspondence is determined in two ways, from depth image and by using the initial poses to triangulate. The model improves the camera poses and decreases drift error during long-distance RGB-D tracking iteratively. Experiments were conducted using data sequences collected by commercial Structure Sensors. The results verify that the geometric integration of hybrid correspondences effectively decreases the drift error and improves mapping accuracy. Furthermore, the model enables a comparative and synergistic use of datasets, including both 2D and 3D features

Road Extraction from VHR Remote-Sensing Imagery via Object Segmentation Constrained by Gabor Features

Chen, L., Zhu, Q., Xie, X., Hu, H., Zeng, H., 2018.
Journal Papers ISPRS International Journal of Geo-Information 7 (9), 362.

Abstract

Automatic road extraction from remote-sensing imagery plays an important role in many applications. However, accurate and efficient extraction from very high-resolution (VHR) images remains difficult because of, for example, increased data size and superfluous details, the spatial and spectral diversity of road targets, disturbances (e.g., vehicles, shadows of trees, and buildings), the necessity of finding weak road edges while avoiding noise, and the fast-acquisition requirement of road information for crisis response. To solve these difficulties, a two-stage method combining edge information and region characteristics is presented. In the first stage, convolutions are executed by applying Gabor wavelets in the best scale to detect Gabor features with location and orientation information. The features are then merged into one response map for connection analysis. In the second stage, highly complete, connected Gabor features are used as edge constraints to facilitate stable object segmentation and limit region growing. Finally, segmented objects are evaluated by some fundamental shape features to eliminate nonroad objects. The results indicate the validity and superiority of the proposed method to efficiently extract accurate road targets from VHR remote-sensing images

Multi View Image Precise Texture Mapping Method Based on Frame Buffer (in Chinese)

ZHU Qing, WENG Qiqiang, HU Han*, WANG Feng, WANG Weixi, YANG Weijun, ZHANG Pengcheng
Journal PapersJournal of Southwest Jiaotong University, 2018

Multiple Point Clouds Data Fusion Method for 3D City Modeling (in Chinese)

ZHU Qing, LI Shiming, HU Han*, ZHONG Ruofei, WU Bo, XIE Linfu
Journal PapersGeomatics and Information Science of Wuhan University, 2018

Precision 3D surface reconstruction from LRO NAC images using semi-global matching with coupled epipolar rectification

Hu, H. and B. Wu.*, 2017
Conference Papers ISPRS Archives, the International Symposium on Planetary Remote Sensing and Mapping, 13–16 August 2017, Hong Kong

Abstract

The Narrow-Angle Camera (NAC) on board the Lunar Reconnaissance Orbiter (LRO) comprises of a pair of closely attached high-resolution push-broom sensors, in order to improve the swath coverage. However, the two image sensors do not share the same lenses and cannot be modelled geometrically using a single physical model. Thus, previous works on dense matching of stereo pairs of NAC images would generally create two to four stereo models, each with an irregular and overlapping region of varying size. Semi-Global Matching (SGM) is a well-known dense matching method and has been widely used for image-based 3D surface reconstruction. SGM is a global matching algorithm relying on global inference in a larger context rather than individual pixels to establish stable correspondences. The stereo configuration of LRO NAC images causes severe problem for image matching methods such as SGM, which emphasizes global matching strategy. Aiming at using SGM for image matching of LRO NAC stereo pairs for precision 3D surface reconstruction, this paper presents a coupled epipolar rectification methods for LRO NAC stereo images, which merges the image pair in the disparity space and in this way, only one stereo model will be estimated. For a stereo pair (four) of NAC images, the method starts with the boresight calibration by finding correspondence in the small overlapping stripe between each pair of NAC images and bundle adjustment of the stereo pair, in order to clean the vertical disparities. Then, the dominate direction of the images are estimated by project the center of the coverage area to the reference image and back-projected to the bounding box plane determined by the image orientation parameters iteratively. The dominate direction will determine an affine model, by which the pair of NAC images are warped onto the object space with a given ground resolution and in the meantime, a mask is produced indicating the owner of each pixel. SGM is then used to generate a disparity map for the stereo pair and each correspondence is transformed back to the owner and 3D points are derived through photogrammetric space intersection. Experimental results reveal that the proposed method is able to reduce gaps and inconsistencies caused by the inaccurate boresight offsets between the two NAC cameras and the irregular overlapping regions, and finally generate precise and consistent 3D surface models from the NAC stereo images automatically.

Hierarchical Regularization of Polygons for Photogrammetric Point Clouds of Oblique Images

Xie, L.*, Hu, H., Zhu, Q., Wu, B., Zhang, Y., 2017
Conference Papers ISPRS Hannover Workshop: HRIGI 17 – CMRT 17 – ISA 17 – EuroCOW 17, 6–9 June 2017, Hannover, Germany

Abstract

Comparison of regularization results: proposed (top), Douglas-Peuker (middle), Dyken (bottom)

Despite the success of multi-view stereo (MVS) reconstruction from massive oblique images in city scale, only point clouds and triangulated meshes are available from existing MVS pipelines, which are topologically defect laden, free of semantical information and hard to edit and manipulate interactively in further applications. On the other hand, 2D polygons and polygonal models are still the industrial standard. However, extraction of the 2D polygons from MVS point clouds is still a non-trivial task, given the fact that the boundaries of the detected planes are zigzagged and regularities, such as parallel and orthogonal, cannot preserve. Aiming to solve these issues, this paper proposes a hierarchical polygon regularization method for the photogrammetric point clouds from existing MVS pipelines, which comprises of local and global levels. After boundary points extraction, e.g. using alpha shapes, the local level is used to consolidate the original points, by refining the orientation and position of the points using linear priors. The points are then grouped into local segments by forward searching. In the global level, regularities are enforced through a labeling process, which encourage the segments share the same label and the same label represents segments are parallel or orthogonal. This is formulated as Markov Random Field and solved efficiently. Preliminary results are made with point clouds from aerial oblique images and compared with two classical regularization methods, which have revealed that the proposed method are more powerful in abstracting a single building and is promising for further 3D polygonal model reconstruction and GIS applications.

Robust point cloud classification based on multi-level semantic relationships for urban scenes

Zhu, Q., Li, Y., Hu, H.*, Wu, B., 2017
Journal Papers ISPRS Journal of Photogrammetry and Remote Sensing 129, 86-102

Abstract

Classification results of 15 dense-matched point cloud tiles

The semantic classification of point clouds is a fundamental part of three-dimensional urban reconstruction. For datasets with high spatial resolution but significantly more noises, a general trend is to exploit more contexture information to surmount the decrease of discrimination of features for classification. However, previous works on adoption of contexture information are either too restrictive or only in a small region and in this paper, we propose a point cloud classification method based on multi-level semantic relationships, including point–homogeneity, supervoxel–adjacency and class–knowledge constraints, which is more versatile and incrementally propagate the classification cues from individual points to the object level and formulate them as a graphical model. The point–homogeneity constraint clusters points with similar geometric and radiometric properties into regular-shaped supervoxels that correspond to the vertices in the graphical model. The supervoxel–adjacency constraint contributes to the pairwise interactions by providing explicit adjacent relationships between supervoxels. The class–knowledge constraint operates at the object level based on semantic rules, guaranteeing the classification correctness of supervoxel clusters at that level. International Society of Photogrammetry and Remote Sensing (ISPRS) benchmark tests have shown that the proposed method achieves state-of-the-art performance with an average per-area completeness and correctness of 93.88% and 95.78%, respectively. The evaluation of classification of photogrammetric point clouds and DSM generated from aerial imagery confirms the method’s reliability in several challenging urban scenes.

An Adaptive Dense Matching Methodfor AirborneImages Using Texture Information

Zhu, Q., Chen, C., Hu, H.*, Ding, Y., 2017
Journal Papers Acta Geodaetica et Cartographica Sinica 46(1), 62-72

Abstract

Semi-global matching (SGM) is essentially a discrete optimization for the disparity value of each pixel, under the assumption of disparity continuities. SGM overcomes the influence of the disparity discontinuities by a set of parameters. Using smaller parameters, the continuity constraint is weakened, which will cause significant noises in planar and textureless areas, reflected as the fluctuations on the final surface reconstruction. On the other hands, larger parameters will impose too much constraints on continuities, which may lead to losses of sharp features. To address this problem, this paper proposes an adaptive dense stereo matching methods for airborne images using with texture information. Firstly, the texture is quantified, and under the assumption that disparity variation is directly proportional to the texture information, the adaptive parameters are gauged accordingly. Second, SGM is adopted to optimize the discrete disparities using the adaptively tuned parameters. Experimental evaluations using the ISPRS benchmark dataset and images obtained by the SWDC-5 have revealed that the proposed method will significantly improve the visual qualities of the point clouds.

Enhanced RGB-D Mapping Method for Detailed 3D Indoor and Outdoor Modeling

Tang, S., Zhu, Q., Chen, W., Darwish, W., Wu, B., Hu, H., Chen, M., 2016
Journal PapersSensors 16 (10), 1589.

Abstract

RGB-D sensors (sensors with RGB camera and Depth camera) are novel sensing systems that capture RGB images along with pixel-wise depth information. Although they are widely used in various applications, RGB-D sensors have significant drawbacks including limited measurement ranges (e.g., within 3 m) and errors in depth measurement increase with distance from the sensor with respect to 3D dense mapping. In this paper, we present a novel approach to geometrically integrate the depth scene and RGB scene to enlarge the measurement distance of RGB-D sensors and enrich the details of model generated from depth images. First, precise calibration for RGB-D Sensors is introduced. In addition to the calibration of internal and external parameters for both, IR camera and RGB camera, the relative pose between RGB camera and IR camera is also calibrated. Second, to ensure poses accuracy of RGB images, a refined false features matches rejection method is introduced by combining the depth information and initial camera poses between frames of the RGB-D sensor. Then, a global optimization model is used to improve the accuracy of the camera pose, decreasing the inconsistencies between the depth frames in advance. In order to eliminate the geometric inconsistencies between RGB scene and depth scene, the scale ambiguity problem encountered during the pose estimation with RGB image sequences can be resolved by integrating the depth and visual information and a robust rigid-transformation recovery method is developed to register RGB scene to depth scene. The benefit of the proposed joint optimization method is firstly evaluated with the publicly available benchmark datasets collected with Kinect. Then, the proposed method is examined by tests with two sets of datasets collected in both outside and inside environments. The experimental results demonstrate the feasibility and robustness of the proposed method.

Texture-Aware Dense Image Matching using Ternary Census Transform

Hu, H.*, Chen, C., Wu, B., Yang, X., Zhu, Q., Ding, Y., 2016
Conference PapersISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences III-3, 59-66, Prague Czech, 12-19 July.

Abstract

Textureless and geometric discontinuities are major problems in state-of-the-art dense image matching methods, as they can cause visually significant noise and the loss of sharp features. Binary census transform is one of the best matching cost methods but in textureless areas, where the intensity values are similar, it suffers from small random noises. Global optimization for disparity computation is inherently sensitive to parameter tuning in complex urban scenes, and must compromise between smoothness and discontinuities. The aim of this study is to provide a method to overcome these issues in dense image matching, by extending the industry proven Semi-Global Matching through 1) developing a ternary census transform, which takes three outputs in a single order comparison and encodes the results in two bits rather than one, and also 2) by using texture-information to self-tune the parameters, which both preserves sharp edges and enforces smoothness when necessary. Experimental results using various datasets from different platforms have shown that the visual qualities of the triangulated point clouds in urban areas can be largely improved by these proposed methods.

Enhanced RGB-D Mapping Method for Detailed 3D Modeling of Large Indoor Environments

Tang, S., Zhu, Q., Chen, W., Darwish, W., Wu, B., Hu, H., Chen, M., 2016
Conference PapersISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences III-3, 19-26. Prague Czech, 12-19 July.

Abstract

RGB-D sensors are novel sensing systems that capture RGB images along with pixel-wise depth information. Although they are widely used in various applications, RGB-D sensors have significant drawbacks with respect to 3D dense mapping of indoor environments. First, they only allow a measurement range with a limited distance (e.g., within 3 m) and a limited field of view. Second, the error of the depth measurement increases with increasing distance to the sensor. In this paper, we propose an enhanced RGB-D mapping method for detailed 3D modeling of large indoor environments by combining RGB image-based modeling and depth-based modeling. The scale ambiguity problem during the pose estimation with RGB image sequences can be resolved by integrating the information from the depth and visual information provided by the proposed system. A robust rigid-transformation recovery method is developed to register the RGB image-based and depth-based 3D models together. The proposed method is examined with two datasets collected in indoor environments for which the experimental results demonstrate the feasibility and robustness of the proposed method

Robust Low-Altitude Image Matching Based on Local Region Constraint and Feature Similarity Confidence

Chen, M., Zhu, Q., Huang, S., Hu, H., Wang, J., 2016
Conference PapersISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences III-3, 19-26. Prague Czech, 12-19 July.

Abstract

Improving the matching reliability of low-altitude images is one of the most challenging issues in recent years, particularly for images with large viewpoint variation. In this study, an approach for low-altitude remote sensing image matching that is robust to the geometric transformation caused by viewpoint change is proposed. First, multiresolution local regions are extracted from the images and each local region is normalized to a circular area based on a transformation. Second, interest points are detected and clustered into local regions. The feature area of each interest point is determined under the constraint of the local region which the point belongs to. Then, a descriptor is computed for each interest point by using the classical scale invariant feature transform (SIFT). Finally, a feature matching strategy is proposed on the basis of feature similarity confidence to obtain reliable matches. Experimental results show that the proposed method provides significant improvements in the number of correct matches compared with other traditional methods.

Orientation of Oblique Airborne Image Sets - Experiences from the ISPRS/EuroSDR Benchmark on Multi-Platform Photogrammetry

Gerke, M., Nex, F., Remondino, F., Jacobsen, K., Kremer, J., Karel, W., Hu, H., Ostrowski, W., 2016
Conference PapersThe International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences XLI-B1, 185-191. Prague Czech, 12-19 July.

Abstract

During the last decade the use of airborne multi camera systems increased significantly. The development in digital camera technology allows mounting several mid- or small-format cameras efficiently onto one platform and thus enables image capture under different angles. Those oblique images turn out to be interesting for a number of applications since lateral parts of elevated objects, like buildings or trees, are visible. However, occlusion or illumination differences might challenge image processing. From an image orientation point of view those multi-camera systems bring the advantage of a better ray intersection geometry compared to nadir-only image blocks. On the other hand, varying scale, occlusion and atmospheric influences which are difficult to model impose problems to the image matching and bundle adjustment tasks. In order to understand current limitations of image orientation approaches and the influence of different parameters such as image overlap or GCP distribution, a commonly available dataset was released. The originally captured data comprises of a state-of-the-art image block with very high overlap, but in the first stage of the so-called ISPRS/EUROSDR benchmark on multi-platform photogrammetry only a reduced set of images was released. In this paper some first results obtained with this dataset are presented. They refer to different aspects like tie point matching across the viewing directions, influence of the oblique images onto the bundle adjustment, the role of image overlap and GCP distribution. As far as the tie point matching is concerned we observed that matching of overlapping images pointing to the same cardinal direction, or between nadir and oblique views in general is quite successful. Due to the quite different perspective between images of different viewing directions the standard tie point matching, for instance based on interest points does not work well. How to address occlusion and ambiguities due to different views onto objects is clearly a non-solved research problem so far. In our experiments we also confirm that the obtainable height accuracy is better when all images are used in bundle block adjustment. This was also shown in other research before and is confirmed here. Not surprisingly, the large overlap of 80/80% provides much better object space accuracy – random errors seem to be about 2-3fold smaller compared to the 60/60% overlap. A comparison of different software approaches shows that newly emerged commercial packages, initially intended to work with small frame image blocks, do perform very well.

Stable least-squares matching for oblique images using bound constrained optimization and a robust loss function

Hu, H., Ding, Y.*, Zhu, Q., Wu, B., Xie, L., Chen, M., 2016.
Journal PapersISPRS Journal of Photogrammetry and Remote Sensing 118, 53-67.

Abstract

Least-squares matching is a standard procedure in photogrammetric applications for obtaining sub-pixel accuracies of image correspondences. However, least-squares matching has also been criticized for its instability, which is primarily reflected by the requests for the initial correspondence and favorable image quality. In image matching between oblique images, due to the blur, illumination differences and other effects, the image attributes of different views are notably different, which results in a more severe convergence problem. Aiming at improving the convergence rate and robustness of least-squares matching of oblique images, we incorporated prior geometric knowledge in the optimization process, which is reflected as the bounded constraints on the optimizing parameters that constrain the search for a solution to a reasonable region. Furthermore, to be resilient to outliers, we substituted the square loss with a robust loss function. To solve the composite problem, we reformulated the least-squares matching problem as a bound constrained optimization problem, which can be solved with bounds constrained Levenberg–Marquardt solver. Experimental results consisting of images from two different penta-view oblique camera systems confirmed that the proposed method shows guaranteed final convergences in various scenarios compared to the approximately 20–50% convergence rate of classical least-squares matching.

An asymmetric re-weighting method for the precision combined bundle adjustment of aerial oblique images

Xie, L., Hu, H.*, Wang, J., Zhu, Q., Chen, M., 2016
Journal PapersISPRS Journal of Photogrammetry and Remote Sensing 117, 92-107.

Abstract

Combined bundle adjustment is a fundamental step in the processing of massive oblique images. Traditional bundle adjustment designed for nadir images gives identical weights to different parts of image point observations made from different directions, due to the assumption that the errors in the observations follow the same Gaussian distribution. However, because of their large tilt angles, aerial oblique images have trapezoidal footprints on the ground, and their areas correspond to conspicuously different ground sample distances. The errors in different observations no longer conform to the above assumption, which leads to suboptimal bundle adjustment accuracy and restricts subsequent 3D applications. To model the distribution of the errors correctly for the combined bundle adjustment of oblique images, this paper proposes an asymmetric re-weighting method. The scale of each pixel is used to determine a re-weighting factor, and each pixel is subsequently projected onto the ground to identify another anisotropic re-weighting factor using the shape of its quadrangle. Next, these two factors are integrated into the combined bundle adjustment using asymmetric weights for the image point observations; greater weights are assigned to observations with fine resolutions, and those with coarse resolutions are penalized. This paper analyzes urban and rural images captured by three different five-angle camera systems, from both proprietary datasets and the ISPRS/EuroSDR benchmark. The results reveal that the proposed method outperforms the traditional method in both back-projected and triangulated precision by approximately 5–10% in most cases. Furthermore, the misalignments of point clouds generated by the different cameras are significantly alleviated after combined bundle adjustment.

Geometric integration of high-resolution satellite imagery and airborne LiDAR data for improved geopositioning accuracy in metropolitan areas

Wu, B., Tang, S., Zhu, Q., Tong, K., Hu, H., Li, G., 2015.
Journal PapersISPRS Journal of Photogrammetry and Remote Sensing 109, 139-151

Abstract

High-resolution satellite imagery (HRSI) and airborne light detection and ranging (LiDAR) data are widely used for deriving 3D spatial information. However, the 3D spatial information derived from them in the same area can be inconsistent. Considering HRSI and LiDAR datasets taken from metropolitan areas as a case study, this paper presents a novel approach to the geometric integration of HRSI and LiDAR data to reduce their inconsistencies and improve their geopositioning accuracy. First, the influences of HRSI’s individual rational polynomial coefficients (RPCs) on geopositioning accuracies are analyzed and the RPCs that dominate those accuracies are identified. The RPCs are then used as inputs in the geometric integration model together with the tie points identified in stereo images and LiDAR ground points. A local vertical constraint and a local horizontal constraint are also incorporated in the model to ensure vertical and horizontal consistency between the two datasets. The model improves the dominating RPCs and the ground coordinates of the LiDAR points, decreasing the inconsistencies between the two datasets and improving their geopositioning accuracy. Experiments were conducted using ZY-3 and Pleiades-1 imagery and the corresponding airborne LiDAR data in Hong Kong. The results verify that the geometric integration model effectively improves the geopositioning accuracies of both types of imagery and the LiDAR points. Furthermore, the model enables the full comparative and synergistic use of remote sensing imagery and laser scanning data collected from different platforms and sensors.

Reliable spatial relationship constrained feature point matching of oblique aerial images ( Talbert Abrams Award, Second Honorable Mention)

Hu, H., Zhu, Q., Du, Z., Zhang, Y., Ding, Y., 2015
Journal Papers Photogrammetric Engineering and Remote Sensing 81 (1), 49-58

Abstract

This paper proposes a reliable feature point matching method for oblique images using various spatial relationships and geometrical information for the problems resulted by the large view point changes, the image deformations, blurring, and other factors. Three spatial constraints are incorporated to filter possible outliers, including a cyclic angular ordering constraint, a local position constraint, and a neighborhood conserving constraint. Other ancillary geometric information, which includes the initial exterior orientation parameters that are obtained from the platform parameters and a rough DEM, are used to transform the oblique images geometrically and reduce the perspective deformations. Experiment results revealed that the proposed method is superior to the standard SIFT regarding both precision and correct matches using images obtained by the SWDC-5 system.

A Quick and Affine Invariance Matching Method for Oblique Images

Xiao, X., Guo, B., Li, D., Zhao, X., Jiang, W., Hu, H., Zhang, C., 2015
Journal Papers Acta Geodaetica et Cartographica Sinica 44 (4), 414-421

Abstract

This paper proposed a quick, affine invariance matching method for oblique images. It calculated the initial affine matrix by making full use of the two estimated camera axis orientation parameters of an oblique image, then recovered the oblique image to a rectified image by doing the inverse affine transform, and left over by the SIFT method. We used the nearest neighbor distance ratio(NNDR), normalized cross correlation(NCC) measure constraints and consistency check to get the coarse matches, then used RANSAC method to calculate the fundamental matrix and the homography matrix. And we got the matches that they were interior points when calculating the homography matrix, then calculated the average value of the matches' principal direction differences. During the matching process, we got the initial matching features by the nearest neighbor(NN) matching strategy, then used the epipolar constrains, homography constrains, NCC measure constrains and consistency check of the initial matches' principal direction differences with the calculated average value of the interior matches' principal direction differences to eliminate false matches. Experiments conducted on three pairs of typical oblique images demonstrate that our method takes about the same time as SIFT to match a pair of oblique images with a plenty of corresponding points distributed evenly and an extremely low mismatching rate.

Phase Grouping Line Extraction Algorithm Using Overlapped Partition

Wang, J., Zhu, Q., Zhang, Y., Hu, H., 2015
Journal Papers Acta Geodaetica et Cartographica Sinica 44 (7), 768-774

Abstract

Aiming at solving the problem of fracture at the discontinuities area and the challenges of line fitting in each partition, an innovative line extraction algorithm is proposed based on phase grouping using overlapped partition. The proposed algorithm adopted dual partition steps, which will generate overlapped eight partitions. Between the two steps, the middle axis in the first step coincides with the border lines in the other step. Firstly, the connected edge points that share the same phase gradients are merged into the line candidates, and fitted into line segments. Then to remedy the break lines at the border areas, the break segments in the second partition steps are refitted. The proposed algorithm is robust and does not need any parameter tuning. Experiments with various datasets have confirmed that the method is not only capable of handling the linear features, but also powerful enough in handling the curve features.

Contour Cluster Shape Analysis for Building Damage Detection from Post-earthquake Airborne LiDAR

He, M., Zhu, Q., Du, Z., Zhang, Y., Hu, H., Lin, Y., Qi, H., 2015
Journal Papers Acta Geodaetica et Cartographica Sinica 44 (4), 407-413

Abstract

Detection of the damaged building is the obligatory step prior to evaluate earthquake casualty and economic losses. It's very difficult to detect damaged buildings accurately based on the assumption that intact roofs appear in laser data as large planar segments whereas collapsed roofs are characterized by many small segments. This paper presents a contour cluster shape similarity analysis algorithm for reliable building damage detection from the post-earthquake airborne LiDAR point cloud. First we evaluate the entropies of shape similarities between all the combinations of two contour lines within a building cluster, which quantitatively describe the shape diversity. Then the maximum entropy model is employed to divide all the clusters into intact and damaged classes. The tests on the LiDAR data at El Mayor-Cucapah earthquake rupture prove the accuracy and reliability of the proposed method.

An adaptive surface filter for airborne laser scanning point clouds by means of regularization and bending energy

Hu, H., Ding, Y., Zhu, Q., Wu, B., Lin, H., Du, Z., Zhang, Y., Zhang, Y., 2014
Journal Papers ISPRS Journal of Photogrammetry and Remote Sensing 92, 98-111

Abstract

Descriptions of the bending energy and the transformed compensation value generated during interpolation of the corresponding raster surface. (a) The interpolated raster surface, (b) the generated bending energy as a by-product of the TPS interpolation and (c) the transformed bend_gain by piece-wise linear interpolation from the bending energy raster given a upper bound of 0.3 m.

The filtering of point clouds is a ubiquitous task in the processing of airborne laser scanning (ALS) data; however, such filtering processes are difficult because of the complex configuration of the terrain features. The classical filtering algorithms rely on the cautious tuning of parameters to handle various landforms. To address the challenge posed by the bundling of different terrain features into a single dataset and to surmount the sensitivity of the parameters, in this study, we propose an adaptive surface filter (ASF) for the classification of ALS point clouds. Based on the principle that the threshold should vary in accordance to the terrain smoothness, the ASF embeds bending energy, which quantitatively depicts the local terrain structure to self-adapt the filter threshold automatically. The ASF employs a step factor to control the data pyramid scheme in which the processing window sizes are reduced progressively, and the ASF gradually interpolates thin plate spline surfaces toward the ground with regularization to handle noise. Using the progressive densification strategy, regularization and self-adaption, both performance improvement and resilience to parameter tuning are achieved. When tested against the benchmark datasets provided by ISPRS, the ASF performs the best in comparison with all other filtering methods, yielding an average total error of 2.85% when optimized and 3.67% when using the same parameter set.

Integration of Chang'E-2 imagery and LRO laser altimeter data with a combined block adjustment for precision lunar topographic modeling

Wu, B., Hu, H.*, Guo, J., 2014
Journal Papers Earth and Planetary Science Letters 391, 1-15

Highlights

  • Cross-mission and cross-sensor data integration for precision lunar topographic modeling.
  • Image residuals reduced from tens of pixels before the adjustment to sub-pixel level after adjustment.
  • After block adjustment the Chang'E-2 DEM showed good consistency with the LOLA DEM.

Abstract

Lunar topographic information is essential for lunar scientific investigations and exploration missions. Lunar orbiter imagery and laser altimeter data are two major data sources for lunar topographic modeling. Most previous studies have processed the imagery and laser altimeter data separately for lunar topographic modeling, and there are usually inconsistencies between the derived lunar topographic models. This paper presents a novel combined block adjustment approach to integrate multiple strips of the Chinese Chang'E-2 imagery and NASA's Lunar Reconnaissance Orbiter (LRO) Laser Altimeter (LOLA) data for precision lunar topographic modeling. The participants of the combined block adjustment include the orientation parameters of the Chang'E-2 images, the intra-strip tie points derived from the Chang'E-2 stereo images of the same orbit, the inter-strip tie points derived from the overlapping area of two neighbor Chang'E-2 image strips, and the LOLA points. Two constraints are incorporated into the combined block adjustment including a local surface constraint and an orbit height constraint, which are specifically designed to remedy the large inconsistencies between the Chang'E-2 and LOLA data sets. The output of the combined block adjustment is the improved orientation parameters of the Chang'E-2 images and ground coordinates of the LOLA points, from which precision lunar topographic models can be generated. The performance of the developed approach was evaluated using the Chang'E-2 imagery and LOLA data in the Sinus Iridum area and the Apollo 15 landing area. The experimental results revealed that the mean absolute image residuals between the Chang'E-2 image strips were drastically reduced from tens of pixels before the adjustment to sub-pixel level after adjustment. Digital elevation models (DEMs) with 20 m resolution were generated using the Chang'E-2 imagery after the combined block adjustment. Comparison of the Chang'E-2 DEM with the LOLA DEM showed a good level of consistency. The developed combined block adjustment approach is of significance for the full comparative and synergistic use of lunar topographic data sets from different sensors and different missions.

Integration and Coregistration of Multisource Lunar Topographic Data Sets for Synergistic Use

Wu, B., Guo, J., Hu, H., 2014.
Book Chapters In: Jin, S. (Eds.), Planetary Geodesy and Remote Sensing. Taylor & Francis Group/CRC Press, Boca Raton, FL, USA, pp. 99-120

3D Building Facades and Roofs Objects Extraction from Pixel Height Map

Huang, M., Du, Z., Zhu, Q., Zhang, Y., Hu, H., 2014
Journal Papers Geomatics and Information Science of Wuhan University 39 (10), 1221-1224

Abstract

Aiming at the problem of extracting 3D building facades and roofs automatically, a 3D building extraction algorithm from pixel height map, which is a product of oblique airborne imagery, is therefore designed and implemented preliminarily. Compared with no or gradual change on the ground or roofs, vertical facade planes show large height gradients. So, this paper take advantage of the height gradient feature to extract facades, at the same time use the maximum entropy threshold feature to extract roof. Oblique images used in this research were collected by Beijing Geo-Vision Tech.Co.,Ltd. with the SWDC-5 Digital aerial photographic system. The results show that it is an effective method to combine the height gradient feature with maximum entropy threshold feature for extracting buildings from pixel height map.

A flexible method for zoom lens calibration and modeling using a planar checkerboard

Wu, B., Hu, H., Zhu, Q., Zhang, Y., 2013.
Journal PapersPhotogrammetric Engineering and Remote Sensing 79 (6), 555-571

Abstract

This paper presents a fl exible method for zoom lens calibration and modeling using a planar checkerboard. The method includes the following four steps. First, the principal point of the zoom-lens camera is determined by a focus-of-expansion approach. Second, the infl uences of focus changes on the principal distance are modeled by a scale parameter. Third, checkerboard images taken at varying object distances with convergent image geometry are used for camera calibration. Finally, the variations of the calibration parameters with respect to the various zoom and focus settings are modeled using polynomials. Three different types of lens are examined in this study. Experimental analyses show that high precision calibration results can be expected from the developed approach. The relative measurement accuracy (accuracy normalized with object distance) using the calibrated zoom-lens camera model ranges from 1:5 000 to 1:25 000. The developed method is of signifi cance to facilitate the use of zoom-lens camera systems in various applications such as robotic exploration, hazard monitoring, traffi c monitoring, and security surveillance.

Co-registration of lunar topographic models derived from Chang’E-1, SELENE, and LRO laser altimeter data based on a novel surface matching method

Wu, B., Guo, J., Hu, H., Li, Z., Chen, Y., 2013
Journal Papers Earth and Planetary Science Letters 364, 68-84.

Abstract

Various lunar digital topographic models (DTMs) have been generated from the data collected from earlier and recent lunar missions. There are usually inconsistencies among them due to differences in sensor configurations, data acquisition periods, and production techniques. To obtain maximum value for science and exploration, the multi-source lunar topographic datasets must be co-registered in a common reference frame. Only such an effort will ensure the proper calibration, registration, and analysis of the datasets, which in turn will permit the full comparative and synergistic use of them. This study presents a multi-feature-based surface matching method for the co-registration of multiple lunar DTMs that incorporates feature points, lines, and surface patches in surface matching to guarantee robust surface correspondence. A combined adjustment model is developed for the determination of seven transformation parameters (one scale factor, three rotations, and three translations), from which the multiple DTMs could be co-registered. The lunar DTMs derived from the Chang’E-1, SELENE, and LRO laser altimeter data in the Apollo 15 landing area and the Sinus Iridum area are examined in this study. Small offsets were found among the Chang’E-1, SELENE, and LRO DTMs. Through experimental analysis, the developed multi-feature-based method was proven able to effectively co-register multiple lunar DTMs. The performances of the multi-feature-based surface matching method were compared with the point-based method, and the former was proven to be superior to the latter.

A Multi-Level Cache Approach for Realtime Visualization of Massive 3D GIS Data

Li, X., Xu, W., Zhu, Q., Hu, J., Hu, H., Zhang, Y., 2012
Journal Papers International Journal of 3-D Information Modeling 1 (3), 37-48.