CNN-based Image Processing algorithm for autonomous optical navigation of Hera mission to the binary asteroid Didymos

Hera mission is the European Space Agency’s contribution to the international collaboration with NASA for the planetary defence, i


Introduction
The Asteroid Impact Deflection Assessment (AIDA) is an international collaboration between NASA and the European Space Agency (ESA), with the primary objective of planetary defence by deflecting a binary asteroid system using kinetic impact.The NASA contribution to this mission is the Double Asteroid Redirection Test (DART), a planetary defence-driven test of a kinetic impactor launched on the 24th of November 2021 that performed the deflection in September 2022 [1].ESA's segment of AIDA is Hera mission, whose objectives are to investigate the properties of the binary asteroid system, to observe the results of DART's impact and to provide information for asteroid impact threat mitigation, mining and science purposes [2].

Parameter Didymos Dimorphos
Gravitational parameter [km The proximity operations of Hera mission consist of different phases that depend on the mission objectives.In this work the Early Characterization Phase (ECP), 30 to 20 km distant from the target, and the Detailed Characterization Phase (DCP), 23 to 9 km distant from the target, are considered as case scenario with the objective of conducting physical and dynamical characterizations of Didymos [5].
A vision based navigation system is designed to guarantee autonomy for the observation of the binary asteroid from short distance.This system includes an on-board camera taking images of the asteroid, an Image Processing (IP) algorithm that extracts information from these images, and a navigation filter that processes the visual data to estimate the spacecraft position, velocity and attitude with respect to the binary system.The camera used is the Asteroid Framing Camera (AFC).For the ECP and the DCP, the navigation strategy is centroid-based, meaning that the IP algorithm is designed to extract the position of the Centre of Mass (COM) of the primary body, and then to estimate the Line Of Sight of the spacecraft, with the purpose of enabling autonomous attitude navigation [6].
Nevertheless, standard IP algorithms' performances are highly dependent on the intrinsic properties of the captured images.Factors such as the Signal-to-Noise ratio, illumination conditions, the presence of other bodies in the image and the irregular shape of the asteroid can affect the accuracy of the extracted visual information.Standard IP algorithms introduce correction terms that depend on the Sun phase angle (Sun-asteroid-spacecraft angle) to reduce the error caused by the illumination conditions.
Recent years have seen an increase of the implementation of Convolutional Neural Networks (CNNs) in space image processing.One of its main advantages over the standard IP algorithms is the robustness against disturbances and adverse characteristics of the images.Most of the CNNs process the input image with a network typically consisting of a series of high-to-low resolution subnetworks.This process reduces the input's resolution, which is then recovered through a low-to-high process.With this procedure, the extracted visual data have low spatial precision and accuracy that could not meet the requirement of an autonomous attitude navigation system.Therefore, this work adopts the High-Resolution Network (HRNet) architecture that has the main characteristic of maintaining a high-resolution representation through the whole network while exchanging information across the parallel multiresolution subnetworks.This process leads to a keypoints regression with higher accuracy given images with high resolution [7].The HRNet has been developed for 2D human pose estimation problems but has already found its spaceborn application to the monocular pose estimation of satellites problems, where the HRNet is used to estimate a set of landmarks from the input images of the target satellite [8,9].
To estimate the relative position of the spacecraft, a range measurement from the asteroid is required.The on-board instrument to measure the range with Didymos is the Planet ALTimeter (PALT), a lidar experiment that determines the distance to the asteroids with an accuracy of 0.5 m operational at a distance ranging from 500 m to 14 km.Therefore, it cannot be used during the ECP for range measurements, but it can be used during part of the DCP.
This work addresses the IP challenges related to the irregular shape of the asteroid, the disturbance caused by the presence of the secondary body in the images and the adverse illumination conditions.We develop an HRNet-based IP algorithm that takes as input synthetic images generated during the ECP and the DCP trajectories and outputs the elements summarized in Table 2, reducing and confining the errors introduced by the mentioned factors.Since it is not a direct measurement but it is derived from the images, the estimated range is a pseudorange measurement.With these measurements the navigation filter is able to estimate the relative state of the spacecraft with respect to Didymos, increasing the robustness of the navigation strategy and of the proximity operations.The estimation of the position of the centroid of the secondary body represents a unique contribution of this work.Another contribution of this work is the regression of the subsolar (SS) point on the surface of Didymos, which is leveraged with a novel methodology to estimate the Sun phase angle.
This paper is structured as follows.Section 2 reviews the state of the art of methodologies applied for image processing algorithms to solve the centroiding, range and phase angle problems.Section 3 describes more in detail the developed HRNet-based IP algorithm.Section 4 performs the numerical simulations and analyses the results.Finally Section 5 concludes this research and recommends future research directions.

Related methods
In this section, current IP algorithms that estimate the quantities shown in Table 2 are reviewed.To the authors' knowledge, the methodologies to estimate the range and the Sun phase angle from images are limited, as these parameters are usually obtained from lidars, altimeters and Sun sensors.These methodologies have been tested with the shapes of Didymos prior to the updates provided by DART mission, hence they will not work with the shape known currently.

Centroiding algorithms
Methods that leverage the position of the Centre of Brightness (COB), the centroid of the image when each pixel is weighted by its intensity, on the image have been proposed.To locate the COB, the IP algorithm needs to perform several steps: windowing, to prevent Dimorphos to be included in the computation and detect a window that includes the primary, thresholding and binarization, to reduce the noise and locate the brightest area of the image, and finally the computation of the COB, which represent the centre of the located area [10].Gil-Fernández and Ortega-Hernando [10] applies the offset   between the COB and the COM to determine the position of the latter.The offset   depends on the Sun phase angle, the spacecraft latitude with respect to the primary Equatorial plane and the shape of the asteroid.Nevertheless, when the lighting circumstances are inadequate this method becomes inaccurate.Pugliatti et al. [11] applies data-driven scattering laws to determine with higher accuracy the relationship between the position of the COB and the COM.
Centroid Apparent Diameter, ellipse and limb fitting techniques rely on the a priori knowledge of the apparent size and shape of the asteroid.Moreover, they require that the shape of the model is regular [12,13].
The current IP algorithm implemented by Hera for the centroiding problem resolution is the Maximum Correlation with a Lambertian Sphere (MCLS).This algorithm estimates the size of the sphere with Lambertian reflectance that maximizes the normalized correlation with the binarized image of the asteroid.Hence, the bright pixels of the image play a major role in this IP technique, thus making it highly depending on the position of the Sun [6].The dependency on the illumination conditions is minimized in this research by applying CNNs.
While Hera is focusing on identifying the COM of both bodies, DART performed the impact while aiming for the COB of Dimorphos.In the final 4 h prior to the impact with Dimorphos, DART used the Small-body Maneuvering Autonomous Real-Time Navigation (SMART Nav) algorithm with the images captured by the on-board Didymos Reconnaissance and Asteroid Camera for Optical navigation (DRACO) to perform autonomous terminal navigation.Prior to relying fully on SMART Nav, the onboard ephemeris have been updated with the ground-processed Optical Navigation data collected by DRACO.The capability of the algorithm to be updated real-time with groundprocessed data allowed to target the COB of Dimorphos with high accuracy without needing to identify the position of its COM [14,15].

Pseudorange
Monocular Depth Estimation (MDE) represents the task of measuring the distance of each pixel of an input image relative to the camera by leveraging depth maps.MDE requires as input RGB images that contain a larger amount of information per pixel, and a strong knowledge of the target shape in order to match relevant feature with the corresponding depth map [16].Nevertheless, the synthetic images used in this work are greyscale, and the precise models of Didymos and Dimorphos can only be reconstructed during the close approach of the Hera's spacecraft [3].Therefore, in this work we propose a method that depends only on the shape of the target, with no constrain on its surface appearance.
Methods that rely on the calculation of the apparent radius have been proposed [17].By comparing the radius of the asteroid with the major length () of the projection of the asteroid on the image plane, the range is calculated.Nevertheless, the estimated range is noisy and not accurate, as the irregular shape of the asteroid generates different values of .In order to meet the accuracy requirements for the navigation strategy of Hera mission, in this research we develop a method of calculating multiple values of  for each image in order to average the irregularity, which reduces the error introduced by the irregular shape of Didymos.

Sun phase angle
Pugliatti et al. [11] compared different methodologies to measure the Sun phase angle obtained by analysing a database of images of Didymos generated with the 3D computer graphics software Blender.These methods rely on the characterization of the blob of pixels representing the asteroid in the binarized image generated with Blender.The eccentricity  of this blob of pixels is found to be correlated with the phase angle.This relationship is described with the following second-order polynomial: The coefficients  0 ,  1 and  2 are evaluated fitting the data in the least-square sense.The estimation of the Sun phase angle is accurate for larger phase angle values.Their second method involved the application of Neural Networks and CNNs to spot relationships between different geometrical features of the blob of pixels representing the asteroid.It has been found that the CNNs outperform all the other methods with their capability of extracting spatial information from images.In this work the CNNs are used to estimate the position of the SS point that is used to estimate the Sun phase angle.

Methodology
In this section, the methodology is described in detail.Fig. 1 shows the main steps of the overall pipeline.With the given reference trajectories from ESA and Planet and Asteroid Natural scene Generation Utility (PANGU) software, two sets of images are generated, one without the presence of Dimorphos and one with both bodies.Both sets are processed to retrieve the Ground Truth (GT) keypoints which are used to supervise the training and validation of the HRNet.The former set is used to determine the GT position of the   , 24 points on the visible border of the primary and the SS point in the image.The GT keypoint   is retrieved with the second set, which is also used for the training and validation of the HRNet.
The trained HRNet is then applied to estimate the position of the keypoints of the testing dataset with both bodies.Finally, the pinhole camera model is used with the regressed keypoints to estimate the pseudorange and the Sun phase angle.Details of the main steps are described in the rest of this section.

Reference trajectories
The adopted reference frame is the Target Body Equatorial Inertial (TBEqI), which has the origin located on Didymos, the -axis pointing towards the vernal equinox, and the XY plane coplanar to the equatorial plane of Didymos.The relative motion of the Sun around Didymos is retrograde as the binary system's orbit obliquity with respect to the ecliptic plane is larger than 90 • , as shown in Table 1.
The ECP trajectory is provided by ESA.Fig. 2 illustrates the trajectory of the spacecraft, together with the position of the Sun (scaled down in the illustration) and the orbit of the secondary.The position of the Sun is calculated using the Jet Propulsion Laboratory Small Body Database [18].The trajectory consists of 4 hyperbolic arcs, with an initial epoch of   = 9012 days and a final epoch of    = 9026 days, calculated in the Modified Julian Date 2000.The only forces considered for each arc are the point mass gravitational attractions of both the primary and the secondary asteroids.Orbital manoeuvres are performed at the joint of two arcs.The durations of the 1st and 3rd arcs are both 4 days while the durations of the 2nd and 4th arcs are both 3 days.The range from the primary varies between a minimum of 20 km and a maximum of 30 km as shown in Fig. 3.It can be seen from Fig. 2 that the ECP trajectory is located in between the Sun and Didymos, in order to provide the AFC camera with bright images of both bodies for Line of Sight navigation [5].Fig. 3 shows that the phase angle  for the ECP trajectory is lower than 90 • , meaning that the spacecraft is always seeing the day side of the asteroid.
The DCP trajectory is provided by GMV Aerospace and Defence, who is in charge of the development of the Guidance, Navigation and Control system of Hera mission.Fig. 4 illustrates the trajectory of the spacecraft, together with the position of the Sun (scaled down in the illustration) and the orbit of the secondary.The trajectory consists of 8 hyperbolic arcs with a total duration of 28 days.The first arc is the transition between ECP and DCP, followed by repetitive trajectories with a range from the primary varying between approximately 9 km and 23 km, as shown in Fig. 5.The minimum distance of 9 km is designed to ensure that the full shape of Didymos is within the Field Of View (FOV) of the AFC even in the presence of 100 m navigation error [5].Fig. 5 shows that the DCP trajectory is also designed to face the camera to the binary system during the day side.

Image generation
The software PANGU is used to generate the database of images for this research.PANGU is a simulation tool that models planet and asteroids surfaces and provides a high-fidelity visualization of images while operating at near real-time speeds.The software has been developed by the STAR-Dundee engineering company [19].The models of Didymos, Dimorphos and the camera are provided by GMV, and are based on ground observations.Didymos' model is near-spherical and is a spinning top with an elevated ridge along the equator.The shape of    Dimorphos is known to be near-ellipsoidal and its is approximated by scaling down the shape model of Itokawa that was the target asteroid of the Hayabusa mission.DART' s images confirmed the near-ellipsoidal shape of Dimorphos, while providing different results for Didymos, which appeared to be ellipsoidal as well, with an extent along its equatorial axes of 849 m and 851 m, larger than the 620 m extent along its rotation axis [20].Nevertheless, in this work we refer to the pre-impact models of Didymos and Dimorphos, with final remarks in Section 5 about the effects that the change of shape could have on the proposed pipeline and in the results obtained.
The software generates greyscale images detected by the camera and shows them on the PANGU viewer, which is a plane with the size of the image (shown in Table 3) and the origin of the coordinated frame set at the top left corner.The horizontal and the vertical axes of the plane are referred as -direction and -direction respectively.The flight file system of PANGU is operated in order to visualize the   binary asteroid system during the trajectories.Flight files are the input to PANGU and they control the viewer to generate images taken at selected epochs of the reference trajectory in the TBEqI reference system, considering the position of the Sun (range, Azimuth and Elevation) and the positions and the orientations (quaternions) of both the binary asteroid system and the AFC camera (joined with the spacecraft) [19].
For asteroid imaging, the AFC has its boresight pointing towards the primary and the vertical axis of the camera is perpendicular to the direction of the Sun with respect to the spacecraft [5].PANGU adopts the boresight, the vertical and the horizontal axes of the camera respectively as the Z-the Y-and the -axis of the camera reference frame [19].Therefore, the position vector of the Sun with respect to the spacecraft lies on the XZ plane of the camera frame.As a result, the images shown in the PANGU viewer always represent the binary system illuminated from the right side.Fig. 6 shows two sample images generated at different epochs of the ECP trajectory, together with the -and the -directions of the PANGU viewer.

Centroids of Didymos and Dimorphos
When the camera is pointing perfectly towards the primary or the secondary, the latter are displayed in the middle of the PANGU viewer.With the conditions that the camera is pointing directly to either the primary or the secondary, the Geometrical Centre (GC) of the selected body that is the arithmetic mean position of all the points belonging to the body, is located at the central pixel with the coordinates (, ) = (512, 512) pixels in the PANGU viewer.The COMs of the primary and the secondary almost coincide with their GCs because of their respectively near-spherical and ellipsoidal shapes.Since the images used in this work are all generated with PANGU, it is assumed that the GCs of Didymos and Dimorphos are their centroids.
Training the CNN algorithm with a set of images with perfect pointing conditions will result in an issue of lacking label variability.To overcome this issue, a pointing error represented by spherical coordinates and defined by two angles  and  is introduced at each epoch of the trajectory in the boresight direction of the camera reference system, as shown in Fig. 7.As a result, the generated images are shifted from the central position of the PANGU viewer.In order to make sure that both bodies lie within the FOV of the AFC camera, random values within an interval of [−0.5, 0.5] • are considered for both  and  for the ECP.With these values, the primary and secondary locations are shifted around in the PANGU viewer.By calculating the shift in pixels of the primary and the secondary from their central position, the GT pixel coordinates of   and   are calculated for each value of  and , as shown in Fig. 8. Since Dimorphos is not included in the FOV of the AFC during the DCP, its GT position is not considered during this phase.To ensure that the asteroid is fully visible in the images even at the closest approaches, the values of  and  are limited to an interval of [−0.25, 0.25] • for the DCP.

Pinhole camera model, pseudorange and sun phase angle
The pinhole camera model is implemented in PANGU using the properties of the AFC camera, shown in Table 3.In the pinhole camera model, also called perspective camera model, the camera aperture is considered as a point rather than a lens.Therefore, this model is used to achieve a first-order approximation of the relationship between the coordinates of a point in the 3D space and its projection onto the 2D image plane of the camera [24].
Fig. 9 shows the geometry of a pinhole camera model.In this model the centre of projection C is the origin of the camera reference frame and the image plane is located at the focal length  = 10.6 cm ( Table 3).It can be seen from Fig. 9 that a point P with coordinates (,  , ) in space is mapped to a point with the coordinates (    ,    , ) on the image plane [22].The units conversion from meters to pixels for an object of length  on the image plane is given by  ⋅  , where  is the number of pixels representing the object and  is the pixel size (Table 3).Therefore, an object of length  in meters on a plane of the 3D space at distance  from the camera and parallel to the image plane is projected onto the latter with the dimensions in pixels defined by Eq. (2).
In order to apply the pinhole camera model to estimate the pseudorange of the spacecraft from Didymos, the shape of Didymos is approximated as a sphere of radius  = 390 m (Table 1).Given that on the image plane the length of the asteroid radius has a value of   ,  can be solved from Eq. ( 2), which is the distance from the centre of projection C and the COM of Didymos, i.e. the range.Therefore, once   of an image captured by the camera is obtained, the pseudorange can be calculated with Eq. (3).
Fig. 10(a) shows how   is calculated for a generic synthetic image of the asteroid, which is measured by the number of pixels from   to the point P on the asteroid's border along the positive direction, considering that the asteroid is always illuminated from the right side.Dimorphos is hidden from the images as its presence in front of Didymos or near its border would disturb the evaluation of   .Nevertheless, the shape of the asteroid is irregular and the accuracy of its approximation to a sphere depends on the relative attitude of the asteroid with respect to the spacecraft.To reduce the error introduced by the irregularity of the shape in the calculation of   , multiple points   on the border within the angular aperture 2 are considered, as shown in Fig. 10(b).
The distance    is evaluated for each point   and the average value n is calculated and used in Eq. ( 3) to calculate the pseudorange.To find the optimal value for  and for the number of points   that minimize the error, the following steps are taken: 1. Given an image of the asteroid, an angular aperture 2 with  ∈ [0, 110] • is considered; the maximum value of  is determined considering that the asteroid is illuminated from the right side of the image plane, as explained in Section 3.2.2. A number of points   at an equal angular distance from each other are taken on the asteroid's border; an upper limit of 50 points is selected considering the computational complexity (memory storage of the intrinsic parameters of the architecture and computational time) of the keypoints regression of the HRNet; 3.For each point   ,    and the average n are calculated, and the pseudorange with the asteroid is evaluated with Eq. (3). 4. Finally, the pseudorange is compared with the GT range (i.e. the range obtained from the reference trajectory shown in Section 3.1) for each image and the Mean Absolute Error (MAE) is determined with Eq. ( 4).
Following this procedure, Fig. 11 is obtained with the -axis representing the angle  ∈ [0, 110] • , the -axis representing the number of points   ∈ [1, 50] and the colourbar representing the MAE.It is illustrated by the red dots that the MAE has its local minima ( < 350 m) in three different regions.The minima have a similar value of  around 90 • , and different values of number of points   .Hence, considering the whole illuminated side of Didymos instead of only one point on the -direction justifies the approximation of the shape of the asteroid with a sphere and reduces the error in the pseudorange calculation.Relying on one singular point, as in the left region of Fig. 11, poses a major risk of failure in the case that the HRNet is not capable of regressing that point accurately.On the other hand, the error obtained with 50 points is not low enough to justify the computational complexity implicated in the regression of such a high number of keypoints.Therefore, in this study we select 24 points   on the border within an angular aperture 2 with  = 87 • .The Sun phase angle estimation relies on the SS point on Didymos, as shown in Fig. 12.Given an image captured by the camera, the COM and the SS projections are detected on the image plane.Dimorphos is hidden from the images as it can cause disturbance in the estimation of the position of the SS.The distance in pixels between these two points, defined here as   , together with the average radius of the asteroid n , are then used to calculate the phase angle  with Eq. ( 5), as shown in Fig. 12(a).Eq. ( 5) applies twice the approximation of the shape of the asteroid as a sphere: one for using n and one for using the inverse of the sine function to calculate .

HRNet
The HRNet architecture is shown in Fig. 13.The network maintains the high resolution representations of the input images by connecting multiple subnetworks in parallel.The first stage is a high-resolution subnetwork.New stages are formed from the gradual introduction of high-to-low subnetworks.To maintain the high-resolution representation, repeated multiscale fusions are performed using low-resolution representation of the same depth and level.The last high-resolution representation is then used for the regression of the selected visual data [7].
The keypoints to regress for each image generated during the ECP are 27 points, which are the   and the   , 24 points on the visible border and the SS.For the DCP images the keypoints are 26, since the   is not visible.Each input image of the HRNet is coupled with the corresponding keypoints that are used to supervise the training to regress the keypoints locations on the testing dataset.For this work, the CNN architecture of the pose-hrnet-w32 that was previously implemented in [25] is used, where 32 represent the widths of the high-resolution subnetwork in the last three stages.During the training, the validation dataset is used beside the training one to compute the validation losses and avoid overfitting.The Adam optimizer is used with a cosine decaying learning rate with initial value of 10 −3 and decaying factor of 0.1.The total parameters involved in the training process are 28, 536, 443.
Dimorphos is not hidden in the images as done in Section 3.3.2,so that the HRNet is trained to regress the location of the keypoints despite the disturbance introduced by the presence of Dimorphos.The input database for the ECP consists of 10 083 (59.91%) images for training, 1266 (7.52%) images for validation and 5031 (29.89%) images for testing, obtained by sampling the trajectory respectively every 100, 800 and 200 s and discarding the images where Dimorphos is outside of the camera frame or behind the primary.The images generated with the DCP are 448 (2.68%) i.e. sampling every 90 min the trajectory and discarding the ones showing the border of Didymos partially outside of the FOV.The dataset of the DCP is used only for testing, in order to assess the ability of the HRNet to generalize the keypoint regression to images showing the binary system in different configurations.The network is trained for 210 epochs.The training of the network is run on the virtual machine provided by Google Colab with the NVIDIA V100 Tensor Core GPU, and it takes around 48 h for the completion of the 210 epochs.The whole HRnet architecture together with its trained weights and biases has a total weight of 109 MB.

Results
In this section, the results of the HRNet-based IP algorithm for the estimation of the centroids of Didymos and Dimorphos, the pseudorange from the primary and the Sun phase angle are presented.Firstly, the accuracy of the HRNet on the estimation of the positions of the keypoints for the ECP is evaluated with the metric defined as follows: where   represents the mth keypoint; the index  refers to the th image of the  = 5031 images of the ECP testing dataset;   is the Root Mean Squared Error (RMSE) between the GT position of the mth keypoint    and the estimation    using the HRNet.The  value of 7.746 pixels obtained by the MCLS IP algorithm developed by GMV to estimate the position of the centroid of Didymos during the ECP of Hera mission is given as a reference, since their estimation meets the pointing accuracy required by the mission [6].This result is obtained by applying the algorithm over a set of 243 images of the 1st arc of the ECP generated with PANGU by GMV.
Further, the accuracy of the developed algorithm on the estimation of the position of the centroids during the ECP is evaluated by the metric defined in Eq. ( 7), which represents the error in pixels between the position of   and   with respect to their GT value.This metric is evaluated for both the − and the − directions of the PANGU viewer.
=   −   (7) The MCLS IP algorithm estimated the position of the COM of Didymos with an average error   of 4.83 pixels on the −direction and 3.95 pixels on the −direction.The accuracy of the pseudorange and the Sun phase angle during the ECP is assessed through the error between the GT values and their estimations.The distribution of the error is also analysed for both parameters by comparing it with the Gaussian distribution that is the ideal one for navigation filters [26].
The performance of the HRNet-based IP algorithm on the estimation of the position of the centroid of Didymos, the pseudorange and Sun phase angle are also evaluated for the DCP, in order to assess the possibility and robustness of generalization to completely new images.
A final note concerns the computational effort required to compute the regression of the keypoints when the trained network is used on the testing dataset.On the NVIDIA V100 Tensor Core GPU provided by Google colab, the average computational time required for regressing the keypoints on a single image is 0.02 s, in contrast with the 30 s required by the MCLS IP algorithm [6].This proves another main advantage of using Machine Learning against standard IP algorithms.

Accuracy of the keypoints regression
Fig. 14 is a bar chart of the  values obtained for the 27 keypoints regressed by the HRNet during the ECP.The first two and the last bars represent the  of the two COMs and of the SS position estimations respectively.The other 24 keypoints are the ones defined in Section 3.3 on the border of the asteroid.It is observed that the worst performance of the HRNet is obtained for the first keypoint ( = 87 • ) on the border of the asteroid and the best performance for the SS point, with the  of 13.161 pixels and 1.973 pixels, respectively.
The difference of the estimation accuracy among the keypoints depends on the different lighting conditions and on the shape of the asteroid.Nevertheless, the values of the  differ from the one obtained using the MCLS IP algorithm by a maximum of around 5 pixels.Therefore, the HRNet is able to estimate the position of the keypoints with the same accuracy of the MCLS IP algorithm, thus meeting Hera mission's pointing accuracy requirements.Fig. 15 illustrates two sample images generated by PANGU during the first and the third arc, together with the keypoints estimated by the HRNet.

Accuracy
Fig. 16 represents the error   for the centroiding of Didymos in both  and -directions of the PANGU viewer reference frame, for the ECP testing dataset of 5031 images.It can be seen that the error oscillates around 3.35 pixels in the -direction with a maximum and a minimum values of 18.78 and −7.9 pixels and around 1.41 pixels in the -direction with a maximum and a minimum values of 17.46 and −15.27 pixels, respectively.The standard deviations of the error   are   = 3.74 pixels and   = 5.65 pixels which means that 68.27% of the Didymos centroiding estimation error is contained between −6.7 pixels and 0.78 pixels for the -direction and between −5.8 pixels and 5.5 pixels for the -direction.The reduced values of the mean and the standard deviations of   allow the utilization of Didymos centroiding estimations in a navigation filter.Comparing with the results on the   obtained by GMV, the HRNet-based IP algorithm is more accurate to solve the centroiding of Didymos.

Illumination conditions
As mentioned in Section 3.2, the binary system is always illuminated from the right side in the images.Hence, the influence of the illumination can be inferred by the error on the -direction.The systematic error of 3.35 pixels is negligible considering that it is lower than the peak-to-peak amplitude (26.68 pixels) of the error itself.Therefore, the developed centroiding algorithm is not affected by the illumination conditions of the asteroid.

Dimorphos' disturbance
The subset of images with the projection of Dimorphos on the surface of Didymos are 193 out of the 5031 of the testing dataset.The RMSE value for these images is 7.4 pixels.The difference of this value from the reference value is 0.346 pixels, which is negligible compared to the size of the image plane.
When Dimorphos is located between the Sun and Didymos, its shadow is projected on the surface of the primary.537 out of the 5031 images of the testing dataset present this condition.For this subset, the value of the RMSE is 7.27 pixels, and the difference from the reference value is 0.476 pixels, which is negligible compared to the size of the image plane.Therefore, the proposed centroiding algorithm is robust to Dimorphos' disturbance.Fig. 17 shows four examples of these two disturbances analysed in this section.

Accuracy
Fig. 18 illustrates the centroiding results of Dimorphos in the and -directions of the PANGU viewer for the 5031 images of the ECP testing dataset.The average value of the error is 3.05 pixels for the direction and 0.107 pixels for the -direction.The standard deviations of the error   are   = 10.26 pixels and   = 5.52 pixels which means that 68.27% of the Dimorphos centroiding estimation error is contained between −6.71 pixels and 13.81 pixels for the -direction and between −5.1 pixels and 5.94 pixels for the -direction.The reduced values of the mean and the standard deviations of   allow the utilization of Dimorphos centroiding estimations in a navigation filter.The peaks of the error shown in Fig. 18 correspond to two epochs where Dimorphos is in eclipse because of the shadow of Didymos, as it will be explained in Section 4.3.2.

Dimorphos in eclipse
There are 593 images out of the ECP testing dataset that include Dimorphos in partial or total eclipse because of the primary's shadow.The centroiding algorithm is still capable of estimating the centroid as in this subset the secondary's boundary is visible even in the worst case scenario of a total eclipse conditions, as shown in Fig. 19 (the boundary illumination has been enhanced in the figure for illustration purposes).The RMSE value for this subset of images is 30.24pixels.Fig. 20 shows the results of two images representing the partial and the total eclipse of Dimorphos.
It can be seen from Fig. 18 that the IP algorithm fails for image 2038 and 3747 of the ECP testing dataset, as the boundary of Dimorphos  is not illuminated enough for the HRNet to determine the centroid position.

ECP pseudorange measurement
Fig. 21 shows the results obtained of the pseudorange estimation with the HRNet-based IP algorithm during ECP.The upper plot shows that the estimation is similar to the ground truth illustrated in Fig. 3 in Section 3.1.The lower plot in Fig. 21 shows that the absolute percentage error is smaller than 10% and oscillates around a mean value of 2.1385%.The proposed algorithm meets the Hera mission requirements which requires the absolute percentage error on the measurement of the pseudorange to be smaller than 10%.When the range reaches its local minima in the 1st and 3rd arcs, the absolute percentage error is higher, due to the fact that the images of the asteroid are appearing larger in the image plane.As a consequence, the non-spherical shape becomes more predominant and the approximation as a sphere is less accurate.In particular, the error near the local minima of the range in the third arc is larger because the phase angle is the lowest (Fig. 3) hence the asteroid is the brightest and its irregular shape is the most visible.
Fig. 22 illustrates the distribution of the percentage error of the pseudorange estimation during ECP, which is similar to a white noise with the mean value of  = 0.1298% near 0. The value of the error that appears the most, which is represented by the peak of the histogram, is 0.4695%.The standard deviation of the error is  = 2.6888% which means that 68.27% of the pseudorange measurements have a percentage error ranging between −2.559% and 2.8186% with respect to the ground truth.The distribution of the error is symmetrical ( = 0.0176) and light-tailed ( = 3.0067) which indicates that the percentage error distribution is similar to a Gaussian distribution.Therefore, the algorithm to estimate the pseudorange from images of Didymos can be implemented in the navigation filter of the spacecraft.

ECP Sun phase angle estimation
Fig. 23 shows the Sun phase angle estimation obtained by the HRNet-based IP algorithm during the ECP.The absolute error is moderate and reaches the peak value of 20 • , although the SS position is estimated with a higher accuracy, as shown in Fig. 14.The error is larger because the sphere approximation of the shape of the asteroid for the estimation of the phase angle is introduced twice in Eq. ( 5).Nevertheless, the mean error is small (6 • ) and the estimated phase angle is similar to the GT value shown in Fig. 3. Fig. 24 illustrates the distribution of the error of the phase angle estimation during ECP.Compared with the pseudorange estimation, the white noise approximation is less accurate considering that the mean value of the error is high ( = 5.976 • ).The peak of the distribution is reached at 6.192 • and the standard deviation is  = 3.5639 • , which means that 68.27% of the estimations have an error ranging between 9.5399 • and 2.4121 • .The distribution of the error is right skewed ( = 0.5654) and light-tailed ( = 3.0946).In conclusion, the algorithm to estimate the Sun phase angle is affected by a limited systematic error that is due to the approximation of the shape of the asteroid as a sphere.Ultimately, the Sun phase angle measurement is not ideal for a navigation filter, but it can still be implemented for redundancy.

Estimation of 𝐶𝑂𝑀 𝐷𝑖𝑑 during DCP
Fig. 25 illustrates the centroiding results of Didymos in the -and -directions of the PANGU viewer for the 448 images of the testing dataset of the DCP.The average value of the error is 22.04 pixels for the -direction and 26.31 pixels for the -direction and the respective RMSE values are 62.03 pixels and 79.22 pixels.Since during the DCP the range from the asteroid is smaller, the apparent size of the asteroid is bigger in the images and the estimation accuracy is lower than that of the ECP.This is also illustrated by comparing Fig. 25 with Fig. 5.The epochs when the spacecraft is getting closer to Didymos coincide with the increase of the error of the estimation of the centroid.Fig. 26 shows the norm of the DCP error   in both directions for each image with respect to the range.When the range value is larger than about 12.4 km, the asteroid is far enough for the HRNet-based IP algorithm to provide an accurate estimation of the position of the centroid.The number of images where the range of the asteroid is lower than 12.4 km is 102.Therefore, the HRNet-based IP algorithm provides accurate results for 77.33% of the DCP images.Considering that the HRNet has been trained using the ECP trajectory that has a minimum range with respect to the asteroid of 20 km, the algorithm is proved to be robust by generalizing the keypoint regression for further 7.6 km.Similarly to the error obtained with the   , the accuracy of the estimations is lower when the spacecraft is closer than 12.4 km to the asteroid.When the range is higher than this value, the algorithm provides an accurate estimation of the range, as it can be seen by comparing Figs.27 and 28 with Fig. 5.The errors in the pseudorange and the Sun phase angle estimations can be reduced if the HRNetbased IP algorithm results are combined with the ones of the PALT instrument, which is in operation only when the range is smaller than 14 km.

Conclusion
This paper develops a CNN-based IP algorithm addressing the problem of centroiding (Geometrical Center), range and the phase angle estimations for autonomous optical navigation around a binary asteroid system.The ECP and DCP proximity operations of Hera mission to Didymos system are studied as case scenario.The challenges tackled by the developed methodology include adverse illumination conditions of the target, irregular shape of the asteroid and the disturbances caused by the presence of the secondary.
The results show that the HRNet-based IP algorithm is able to estimate the position of the centroids of the primary, the pseudorange and the Sun phase angle with high accuracy for the ECP.In particular the centroiding method is robust to the presence of Dimorphos and its shadow projected on the surface of the primary.Besides, it exhibits no dependency on the illumination condition.If higher accuracy is required, the training database can be augmented using additional images generated from different trajectory segments around the target body.Moreover, the algorithm estimates with high accuracy the position of the centroid of the secondary body, even in partial and total eclipse conditions, which is another unique contribution of this work.The methodology to estimate the pseudorange is robust to the disturbances caused by the irregular shape of the asteroid and the presence of Dimorphos.In particular, the percentage error in the pseudorange estimation has a near-Gaussian distribution, which is ideal for navigation filters.The algorithm can also provide an accurate estimation of the position of the SS point on the surface of the asteroid during the ECP, which is used to estimate the Sun phase angle.Nevertheless, the estimation of the latter is moderately affected by the irregular shape of the asteroid and the error does not present a Gaussian distribution.
The trained net from the ECP is able to generalize its solution to the DCP if the range is larger than a threshold value of 12.4 km, in terms of the centroid of Didymos, the pseudorange and the Sun phase angle.If higher accuracy is required for lower ranges, the training database  can be augmented using additional images generated with the DCP or closer trajectories.
Nevertheless, the proposed pipeline is applied to the pre-impact models of Didymos and Dimorphos.Assuming that the same methodology is applied to images representing the actual shapes of Didymos and Dimorphos with the post impact orbital parameters, the obtained results will be different.The regression of the 27 keypoints will not be affected, as the non-spherical shapes of the target will be learned by the HRNet, as it can be seen from the regression of the COM of Dimorphos represented in this work by the irregular shape of Itokawa.Hence, the estimation of the position of the COMs from images representing the updated shapes of Didymos and Dimorphos will not be affected.On the other hand, the estimation of the range from the primary and the Sun phase angle are both based on the approximation of the shape of the primary to a sphere.Therefore, in order to apply the same pipeline, it should be taken into account the ellipsoidal shape of Didymos.It is important to address that the potential unstable motion of Dimorphos caused by the impact can cause its continuous reshaping, that might not cease at Hera's arrival [3].In that case, a fine-tuning of the HRNetbased IP algorithm with a dataset of images taken during the first days of the ECP is necessary in order to learn the position of the COM of Dimorphos.
Our developed methodology contributes to the SSA by improving the robustness and the autonomy of the navigation strategy of the first mission ever testing asteroid deflection.Specifically, the unique contribution represented by the estimation of the SS point can be applied to any object in space.The pseudorange and Sun phase angle estimation methodologies can be employed for any space object with a near-spherical shape, hence it can also support the tracking of space debris and other uncooperative objects.
Future work would go into the direction of other applications for the HRNet-based IP algorithm.For instance, an additional output useful for the navigation during the proximity operations is the pose estimation.Subsequently, the algorithm will be incorporated with a navigation filter to estimate the state of the spacecraft and to quantitatively evaluate the improvement of navigation performance.

A
.Kaluthantrige et al.

Fig. 3 .
Fig. 3. Range and phase angle of the ECP trajectory.

Fig. 5 .
Fig. 5. Range and phase angle of the DCP trajectory.

Fig. 6 .
Fig. 6.PANGU viewer with two sample images captured at different points of the ECP trajectory.

Fig. 8 .
Fig. 8. Sample images generated during the 2nd, the 3rd and 4th arc of the ECP with Ground Truth position of the centroids of Didymos and Dimorphos.

A
.Kaluthantrige et al.

Fig. 15 .
Fig. 15.Keypoints regression results for two images of the 1st and the 3rd arc.

A
.Kaluthantrige et al.

Fig. 17 .
Fig. 17.Centroiding results for images with disturbances due to the presence of Dimorphos.

Figs. 27
Figs. 27 and 28 shows the pseudorange and the Sun phase angle estimation obtained by the HRNet-based IP algorithm during the DCP.Similarly to the error obtained with the   , the accuracy of the estimations is lower when the spacecraft is closer than 12.4 km to the asteroid.When the range is higher than this value, the algorithm provides an accurate estimation of the range, as it can be seen by comparing Figs.27 and 28 with Fig.5.The errors in the pseudorange and the Sun phase angle estimations can be reduced if the HRNetbased IP algorithm results are combined with the ones of the PALT instrument, which is in operation only when the range is smaller than 14 km.

Table 2
Outputs of the CNN-based IP algorithm.