Imaging and Sensing for Unmanned Aircraft Systems. Volume 1: Control and Performance
2: ECE Department, Karunya University, Coimbatore, India
3: Instituto Tecnologico de Aeronautica, Sao Jose dos Campos, Brazil
4: Department of Computer Science, Lulea University of Technology, Lulea, Sweden
5: Autonomous and Intelligent Systems Laboratory, RMIT University, Melbourne, VIC, Australia
This two volume book set explores how sensors and computer vision technologies are used for the navigation, control, stability, reliability, guidance, fault detection, self-maintenance, strategic re-planning and reconfiguration of unmanned aircraft systems (UAS). Volume 1 concentrates on UAS control and performance methodologies including Computer Vision and Data Storage, Integrated Optical Flow for Detection and Avoidance Systems, Navigation and Intelligence, Modeling and Simulation, Multisensor Data Fusion, Vision in Micro-Aerial Vehicles (MAVs), Computer Vision in UAV using ROS, Security Aspects of UAV and Robot Operating System, Vision in Indoor and Outdoor Drones, Sensors and Computer Vision, and Small UAVP for Persistent Surveillance. Volume 2 focuses on UAS deployment and applications including UAV-CPSs as a Testbed for New Technologies and a Primer to Industry 5.0, Human-Machine Interface Design, Open Source Software (OSS) and Hardware (OSH), Image Transmission in MIMO-OSTBC System, Image Database, Communications Requirements, Video Streaming, and Communications Links, Multispectral vs Hyperspectral Imaging, Aerial Imaging and Reconstruction of Infrastructures, Deep Learning as an Alternative to Super Resolution Imaging, and Quality of Experience (QoE) and Quality of Service (QoS).
Inspec keywords: image sensors; autonomous aerial vehicles
Other keywords: sensor and vision integration; UAV avionics; imaging capabilities
Subjects: Image sensors; Mobile robots; Image sensors
- Book DOI: 10.1049/PBCE120F
- Chapter DOI: 10.1049/PBCE120F
- ISBN: 9781785616426
- e-ISBN: 9781785616433
- Page count: 362
- Format: PDF
-
Front Matter
- + Show details - Hide details
-
p.
(1)
-
1 Introduction to advances in UAV avionics for imaging and sensing
- + Show details - Hide details
-
p.
1
–21
(21)
An unmanned aerial vehicle (UAV) - aka drone, unmanned aircraft system or remotely piloted aircraft system - is an aircraft without a human pilot on board. Its flight can be controlled autonomously by computers in the vehicle or by remote control. UAVs can uniquely penetrate areas, which may be too dangerous or too difficult to reach for piloted craft. The UAV cyber-physical system comprises all the subsystems and interfaces for processing and communication functions performed by the embedded electronic system (avionics) and the ground control station. To accomplish the desired real-time autonomy, the avionics is highly tied with aerodynamics sensing and actuation. An entirely autonomous UAV can (i) obtain evidence about the environment, (ii) work for an extended period of time without human interference, (iii) move either all or part of itself all over its operating location devoid of human help and (iv) stay away from risky situations for people and their assets. This chapter intends to introduce the material addressed in further chapters of this book. The next sections go through some concepts that are recurrent in the book.
-
2 Computer vision and data storage in UAVs
- + Show details - Hide details
-
p.
23
–45
(23)
Unmanned aerial vehicles (UAVs) handle operations such as inspection, mapping, monitoring, and surveying, to cite a few, that assist image processing, aerial imaging, and computer vision (CV) algorithms. The UAVs with cameras collect a massive amount of images and videos for various research and commercial applications. Furthermore, UAVs have various sensors, e.g., thermal, magnetic, sound, light, and speed, to collect environment details for specific research as well as commercial usages. Thus, this chapter focuses on acquiring, storing, processing, and compressing images and videos. This chapter describes how CV software impacts tasks such as processing, communications, storage, and compression besides other applications specific to a UAV cyber-physical system (CPS). Additionally, Section 2.2 explains the general architecture of the cloud-based UAV-CPS, the challenges, and design goals. Section 2.3 discusses memory usage in UAV, specific requirements, limitations of onboard storage, and general solutions. Section 2.4 briefs the UAV data logging (DL), primary benefits, and protocol standardisation with examples. Section 2.5 grants a view of the different types of DL, requirements, and proposes solutions. Section 2.6 discusses future trends of data storage, data processing, control, the impact of big data, complexity, privacy barriers, infrastructure, and other challenges.
-
3 Integrated optical flow for situation awareness, detection and avoidance systems in UAV systems
- + Show details - Hide details
-
p.
47
–74
(28)
Optical flow (OF) plays a decisive role in visual situation awareness, detection and obstacle avoidance systems for unmanned aerial vehicles (UAVs), which are cyberphysical systems (CPSs) that interact with the environment through sensors and actuators. The use of cameras allows the integration of computer vision (CV) algorithms with the inertial navigation systems (INS). The movement of characteristics of the image fused with the dynamic of the UAVs allows us to improve the process of remoting sense, avoid obstacles or estimate the position and velocity of the UAV. In the literature, there are various algorithms to locate characteristics points between two consecutive images. However, the computation time and consumption of physical resources such as memory features are due to embedded systems. This chapter shows (i) how to integrate the movement of the pixel textures (OF) in the image with INS data, (ii) compares different algorithms to match points between consecutive images, (iii) implements a process to encounter points between consecutive images and (iv) implements a computationally less expensive and with less memory consumption algorithm. A case study about using the field-programmable gate array (FPGA) as part of the visual servoing is discussed showing how to integrate results into the CV hardware system of a UAV and addressing the need to handle issues such as multi-resolution.
-
4 Introduction to navigation and intelligence for UAVs relying on computer vision
- + Show details - Hide details
-
p.
75
–99
(25)
Vision-based sensors (VBSs) provide several advantages to unmanned aircraft systems (UASs) primarily due to a large amount of data they are able to capture, and their reduced size, weight, power, and cost compared to other state-of-the-art sensors. A number of vision-based navigation (VBN) methods have emerged recently, which aim to maximise state-estimation performance and reduce reliance on the global navigation satellite system. This chapter identifies and describes some of the most popular visual navigation strategies for the UAS to acquaint the reader with this important field of study. VBN methods presented here include visual servoing, optical flow-based state estimation, visual odometry and terrain referenced visual navigation. Reference system architectures and relevant mathematical models for these methods are presented to facilitate a more in-depth understanding. A review of these methods and their applications to various UAS use-cases is conducted, focussing primarily on seminal work in this domain. The limitations of this sensing modality are also presented, along with a discussion of future trends including multi-spectral imaging and biomimetic systems to inform the reader of key gaps and research avenues in this field.
-
5 Modelling and simulation of UAV systems
- + Show details - Hide details
-
p.
101
–121
(21)
The modelling of a system's dynamics and performing simulation are commonly used techniques and almost critical components in the modern development of manned and unmanned systems alike. This chapter addresses the need for modelling and simulation of unmanned air vehicle systems and reviews historical developments in the field, current techniques used and future developments.
-
6 Multisensor data fusion for vision-based UAV navigation and guidance
- + Show details - Hide details
-
p.
123
–144
(22)
Significant advances in sensor technology, along with economies of scale due to large production volumes, have supported the miniaturisation of navigation sensors, allowing widespread low-cost integration on unmanned aircraft systems (UAS). In small-size UAS applications, standalone sensors are not a viable option since the reduction in navigation sensor form-factor, weight and cost typically results in lowered accuracy and precision. The fusion of multiple sensor measurements in UAS navigation systems can support greater accuracy, integrity and update rates than is achievable employing individual sensors. This chapter introduces the fundamentals of state-estimation methods employed on UAS and presents different sensor integration architectures, along with an assessment of their advantages and trade-offs. Attention is devoted primarily to recursive optimal estimation algorithms such as the Kalman filter and its variants owing to its prolific employment in various classes of UAS. The need to support robust navigation performance in the global navigation satellite system denied environments, and the proliferation of visual sensors has led to the development of numerous methods for integrating visual sensor measurements (primarily) with inertial sensors. Therefore, the reader is introduced to the most popular system architectures for visual-inertial sensor integration in order to provide an understanding of the current state-of-the-art and to support the identification of future research pathways.
-
7 Vision-based UAV pose estimation
- + Show details - Hide details
-
p.
145
–172
(28)
As the use of unmanned aerial vehicles increased, studies regarding their autonomous flight became an academic field of great interest for researchers. Until recently, most studies based their developments using an inertial measurement unit (IMU) and a global navigation satellite system (GNSS) as the main sensors to calculate and estimate the UAVs pose. These sensors, on the other hand, have several limitations, which can affect the navigation, therefore, the fully autonomous aspect of the system. Images captured during flight, computer vision algorithms, and photogrammetry concepts have become a core source of data to estimate the UAVs pose in real-time, therefore, composing new alternative or redundant navigation systems. Several algorithms have been proposed in the scientific community, each one working better in specific situations and using different kinds of imaging sensors (active and passive sensors). This chapter describes the main visual-based pose estimation algorithms and discusses where they best apply and when each fails. Fresh results depict the development of new strategies that will overcome the remaining challenges of this research field.
-
8 Vision in micro-aerial vehicles
- + Show details - Hide details
-
p.
173
–216
(44)
A micro-aerial vehicle (MAV) is a kind of minor version of unmanned aerial vehicles (UAVs), which has a size restriction. MAVs have a wide range of applications in commercial, research, government, and military purposes. UAVs are valuable tools due to some of their useful advantages such as safety, fl exibility, relatively low cost of ownership, and ease of operation. Bio-inspired robots have been presented that can both fly and move on land, either by jumping or walking. One of the most critical parts of such vehicles (MAVs) is the vision (visual perception) system. In this chapter, different technologies for designing the vision systems for bio-inspired MAVs will be reviewed.
-
9 Computer vision in UAV using ROS
- + Show details - Hide details
-
p.
217
–241
(25)
This chapter presents the concepts, advantages, and practical examples of computer vision on robot operating system (ROS), applied explicitly on unmanned aerial vehicle (UAV) domain. ROS was built with the concept of abstraction (using messages interfaces) to allow re-use of software. This concept allows the integration with already built source code for computer vision tasks with the ROS environment. In conjunction with the available simulators for ROS, it is possible to design UAV solutions in a model-based concept, anticipating many operational and technical constraints prior to operational release.
-
10 Security aspects of UAV and robot operating system
- + Show details - Hide details
-
p.
243
–259
(17)
Unmanned aerial vehicles (UAVs) are an enabling technology being used to improve typical tasks such as delivery, agriculture, inspection, rescue, catastrophe response, among others. Initially used on military applications, nowadays they are increasing its presence as the commercial platform and consumer electronics. Despite its benefits and being an evolving technology, it embraces aspects such as complex architecture and mission-critical applications that require a rigorous development process to guarantee operational success during its whole life cycle. News about security-related events is becoming commonplace, compromising systems aspects such as confidentiality, integrity, and availability (i.e., security pillars), resulting in operational disruption, financial losses and safety incidents. This chapter preliminary presents typical, published and ongoing researches on security flaws on UAV domain and possible scenarios that can arise from them. The following security strategies to the resilient operation is presented to support new UAV designs based on typical security approaches (e.g., authentication, cryptography), current researches, and insights derived from aircraft design guidelines, which relates to intentional unauthorized interaction (cyber-threat) and its effects to safety. Finally, deployment of the robot operating system on consumer UAV (parrot AR.Drone 2) is performed, and a cybersecurity assessment is presented containing its findings, mitigations, and proposals to strengthen its operation (resiliency).
-
11 Vision in indoor and outdoor drones
- + Show details - Hide details
-
p.
261
–280
(20)
This chapter explores how the type of environmental impacts the CV techniques, algorithms and specific hardware to be used. Indoor environments, also known as controlled environments, generally rely on solutions based on beacons, proximity sensors and image processing for data acquisition. In this case, as the environment is controlled, the illuminance of the scene is adjusted and sensors are previously positioned, which facilitates the development and execution of these systems. In outdoor environments, generally known for uncontrolled environmental variables, frequently require solutions based on image processing techniques to provide the data acquisition. In this environment, the non-constant variation of the illuminance of the scene and the great variation of the background of images are important complicating factors for the operation of the image processing algorithms. In addition, constructions and buildings block the signal of sensors and global positioning systems making it even harder to treat the exceptions caused by these factors. Each exception being treated in a CV system has a computational cost that can be high. If this is considered in applications using embedded hardware, some projects simply become infeasible. Researchers put great effort attempting to optimise the software for high performance and better use of the hardware resources, so that less processing power is demanded as well as positively impacting the energy savings. This chapter presents a review of the main CV techniques currently used in the development of mission control software for the use in indoor and outdoor environments, providing autonomy navigation and interaction for these aerial robots.
-
12 Sensors and computer vision as a means to monitor and maintain a UAV structural health
- + Show details - Hide details
-
p.
281
–307
(27)
This chapter discusses new approaches to sense and acquire vibration data and to pre-process these data on aeroelastic certification test flights. These new approaches aim to reduce the time to identify the aeroelastic phenomenon and to reduce the size of hardware that must be boarded in the aircraft, thus minimising the risks and costs of the vibration tests. The presented experiments construct a way to develop a non -contact measurement system for flight vibration tests in the aircraft certification process. These experiments have shown that the techniques used today for in-flight trials will be obsolete in the near future, as the aeronautical structures are becoming lighter every day, thus not admitting any additional mass for instrumentation in-flight trials.
-
13 Small UAV: persistent surveillance made possible
- + Show details - Hide details
-
p.
309
–331
(23)
In typical intelligence surveillance and reconnaissance (ISR) missions, persistent surveillance is commonly defined as the exercise of automatic intelligence discovery by monitoring a wide area coverage for hours of operation at a high altitude leveraging aerial platforms (manned or unmanned). The platform can be large enough to carry a matrix of high-resolution sensors and a rack of high-performance computing equipment to process in real-time all sensors' feeds. With the current ISR growing in capability, engineering and optics-based aerial surveillance to find a suitable design solution became a design challenge. More onboard processing is desired for an increasing fidelity/resolution sensors' feed, while matching a constraining SWaP (size, weight, and power) budget requirements in a bandwidthconstrained operating theatre. The advent in small unmanned aerial vehicle (sUAV) technology, able to carry sophisticated optics payloads and to take aerial images from strategic viewpoints has become unavoidable in nowadays battlespace contributing in moving forward the ISR capabilities. The constrained on-board processing power in addition to the strict limit in the flying time of sUAV are amongst the serious challenges to overcome to enable cost-effective persistent surveillance based on sUAV platforms. All previous examples show that tailoring the sensors to match the platforms' environment is a challenging endeavour and therefore architects have shifted their design methodology to be based on hardware and software open architectures as a centrepiece of their approach in building cost-effective surveillance solution design. This chapter is a brief introduction to hardware and software building blocks for developing persistent surveillance systems. In our context, the focus is in particular on Electro-Optic (EO, visual spectrum) and Infrared (IR) integrated solutions leveraging computer vision techniques for surveillance missions.
-
14 Conclusions
- + Show details - Hide details
-
p.
333
–335
(3)
The current awareness in UAVs has prompted not only military applications but also civilian uses. Aerial vehicles' requirements aspire to guarantee a higher level of safety comparable to see-and-avoid conditions for piloted aeroplanes. The process of probing obstacles in the path of a vehicle, and to determine if they pose a threat, alongside measures to avoid problems, is known as see-and-avoid or sense and-avoid involves a great deal of decision-making. Other types of decisionmaking tasks can be accomplished using computer vision and sensor integration since they have great potential to improve the performance of UAVs. Macroscopically, Unmanned Aerial Systems (UASs) are cyber-physical systems (CPSs) that can benefit from all types of sensing frameworks, despite severe design constraints such as precision, reliable communication, distributed processing capabilities, and data management.
-
Back Matter
- + Show details - Hide details
-
p.
(1)