UAV-Based Crown Size Estimation and 3D Training Simulator

Abstract

Crowd estimation using Unmanned Aerial Vehicles (UAVs) is an emerging research area with the potential to map the crowd spread over a large geographical area. However, estimating the crowd is a challenging estimation problem especially when the camera is moving with the possibility to count the same individual multiple times. Most of the existing crowd estimation methods have focused on using individual frames to predict from a single static camera. UAV-based approaches pose the problems of (1) a moving video camera, so the same person may appear in several frames and may be counted more than once, (2) multiple counts as the crowd may move during the capture time and (3) more varied different viewpoints, which could require extensive additional training and testing data. This thesis introduces the existing challenges in the domain and attempts to overcome some of the issues by proposing a novel 3D crowd estimation method for use with a UAV. To develop the crowd estimation method in 3D, the work has been divided into three phases: (a) in the first phase a 3D crowd simulator was developed and synthetic crowd data was captured by flying a virtual UAV within the crowd simulator, (b) in the second phase, 2D crowd estimation methods were trained and tested on the synthetic data to validate the simulator, and (c) finally, in the third phase, a 3D crowd estimation method for use with a virtual UAV was developed and evaluated. For the development of a 3D crowd estimation method using a UAV, a large quantity of annotated data was required to train and test the system. However, gathering the crowd data is a challenging, expensive and time-consuming task due to legal restrictions in the UK which place limits on flying a UAV in the real-world, particularly over crowds. Considering the challenges of gathering and annotating real-world data, a 3D crowd simulation system was developed using the Unreal Engine as a platform, along with the use of MakeHuman to generate varied synthetic crowd data. The system is demonstrated using three different scenes with a number of crowd sizes and virtual UAV flight paths. In the second phase of our research work, synthetic data captured from the crowd simulator has been tested using models pre-trained on two existing state-of-the-art deep learning 2D crowd estimation methods (MCNN and CMTL). Our findings demonstrate that the simulated data is close to the real-world captured data and could be used to train the crowd estimation models. In addition, the model was trained using the existing CMTL and MCNN architectures using synthetic data from the simulator and tested against the existing real-world images from the ShanghaiTech dataset. The findings again demonstrated that the simulated data is close to the real-world data and could be useful for training future crowd models. In the third and final phase of our research, we have proposed a novel 3D crowd estimation system using a virtual UAV intending to solve the existing problems of (a) a moving camera, (b) multiple counts and (c) different viewpoints. The method involves first reconstructing a 3D model of the scene using the video data from the virtual UAV, then applying a 2D crowd detection method to each frame, mapping the 2D detections onto the 3D model, and finally merging nearby/overlapping detections into single detections. The method is evaluated by aligning the ground-truth 3D locations with the automated detections and comparing the correct and incorrect detections. The method has been tested on three different scenarios that were captured by flying a virtual UAV following the two different flight paths: circular and straight over the synthetic static crowd of size 50, 100 and 200, respectively. Testing the captured data across all three scenarios evidenced that the proposed method performs better using a circular path in all scenarios. The final result using a circular path across the 4 different crowd sizes and 3 scenes was an accuracy of 67.33%, demonstrating the effectiveness of the technique. Future work is needed to extend the methods detailed here to handle dynamic crowds. The output of the proposed method shows consistency on a circular path however, the method did not perform as expected using the data captured following a straight path due to the involvement of a photogrammetry tool for 3D reconstruction that requires image overlap to achieve the model completeness and this subsequently makes the circular path a better choice for accurate estimation of the crowd when compared with the straight path.

Date of Award	2022
Original language	English
Awarding Institution	Aberystwyth University
Supervisor	Bernie Tiddeman (Supervisor)