Jitter reduction using native fiducials in rotating mirror ultra-fast microphotography

Rotating mirror cameras represent a workhorse technology for high speed imaging in the MHz framing regime. The technique requires that the target image be swept across a series of juxtaposed CCD sensors, via reflection from a rapidly rotating mirror. Employing multiple sensors in this fashion can lead to spatial jitter in the resultant video file, due to component misalignments along the individual optical paths to each CCD. Here, we highlight that static and dynamic fiducials can be exploited as an effective software-borne countermeasure to jitter, suppressing the standard deviation of the corrected file relative to the raw data by up to 88.5% maximally, and 66.5% on average over the available range of framing rates. Direct comparison with industry-standard algorithms demonstrated that our fiducial-based strategy is as effective at jitter reduction, but typically also leads to an aesthetically superior final form in the post-processed video files.


Introduction
Rotating-mirror framing cameras readily facilitate high-speed imaging in the MHz framing regime: a capability that has been exploited for defence, aerospace and medically-related studies. In this last category, the recent realisation that ultrasound-stimulated microbubble cavitation may be utilised for therapeutic purposes [1] has led to an intense research effort to ascertain the details of cavitation processes at [sub-]microsecond temporal resolutions [2][3][4][5][6][7].
The benefits of rotary mirror systems, relative to their non-optomechanical counterparts, include the superior image resolution, high intrinsic dynamic range, and the relatively large sequence record depth, all of which are preserved even at maximum framing rate [7]. Annoyingly, spatial 'jitter' is present in the recompiled video files, so that countermeasures are required to restore the inter-frame alignment to its optimal aesthetic presentation. To address this, industry standard 2D image stabilization algorithms can be implemented, including: optimized cross-correlation of discrete Fourier transforms [8]; and global motion estimation methods [9,10]. However, in circumstances where target scenes exhibit nonuniform exposures from frame to frame; or during periods of rapid in-scene change, these orthodox curative strategies can be found wanting. Moreover, other technical issues regarding the spatio-temporal accuracy of rotary cameras can also present problems [11][12][13][14]. The purpose for the present study was to address such issues by making innovative use of native fiducials.
We evaluated images captured on a commercial rotary mirror framing camera over its full range of framing rates. Camera operation served to expose each of the 62 CCD imaging sensors in a sequential manner, whereupon their data are compiled to form a video file. Inspection of resultant videos showed two extraneous characteristics: (i) relatively small, static features, superimposed on the image and present on all individual frames; (ii) dynamic background features, usually elongated in nature, which were seen to sweep in the ydirection due to the intrinsic movement of the rotating mirror. The static features are presumed to be dust particles that have settled on a conjugate optical component [5,15], whereas the dynamic features appeared to be microscopic imperfections on the facets of the rotating mirror. We chose to term these naturally occurring features 'fiducials', given that they can be exploited as true points of reference within the resultant image files, thus providing a convenient alternative to orthodox system calibration approaches [5,[13][14][13][14][15][16][17][18]].

Methodology & Apparatus
The hardware comprises a rotating mirror high speed camera (Cordin 550-62) ( Fig. 1) fitted with an objective lens (magnification: 63x, numerical aperture: 0.9) and Leica Monozoom 7. A pentagonal beryllium mirror driven by a gas turbine is designed to rotate at a maximum of 12,500 revolutions per second (enabling framing rates of up to 4 MHz) [12,19]. Internally, the camera is constructed around 62 CCD image sensors, each comprised of a 1000 × 1000 pixel array, and arranged in two radial and symmetric 72° arcs either side of the mirror. The CCDs are housed in consecutive image frames labeled F0 to F63. CCDs are not present at locations F15 and F47: rather, these regions serve as apertures to allow direct image beam entry to the rotary mirror chamber -the image from the target having been divided by an upstream beam-splitter so that light is directed into the housing from two separate locations (blue arrows in Fig. 1) and towards two different facets (horizontal red rays) on the rotating mirror (labelled M). At any instant, two images of the target are thus circulating within the mirror chamber, one of which will lie on an active CCD bank, the other being formed somewhere within the two 108° arcs of 'dead-space' between the arrays of CCD banks. This situation is illustrated in Fig.1 where the left hand image is reflected onto the first CCD at frame F0 just at the instant when the right hand image has finished being exposed at frame F63. The geometry of the mirror and nature of the recording process is such that a continuous 64 frame movie can be taken without any temporal interruption other than the presence of 'black-out' frames at the entrance aperture locations (F15 & F47). Imaging was undertaken with frame rates, Ω, ranging from 0.1 to 2.0 million frames per second (Mfps)

Results
Sequences thus recorded were found to exhibit 'jitter', i.e., the image presented to each CCD appears somewhat displaced from frame to frame, which is aesthetically non-optimal. We determined to ascertain the nature of this resultant jitter, as well as developing a countermeasure to its occurrence. Static fiducials, as described earlier, were exploited towards this end. The process involved capture of multiple sequences across a range of frame rates (125kfps, 400kfps, 700kfps, 1.5Mfps and 2Mfps). As sequence recording starts randomly at any one of the 8 available CCD banks (Fig. 1), we first reformatted all sequences in terms of the absolute CCD position used to acquire specific frames within each video: this in a bid to quantify whether the inter-frame misalignments arising were directly linked to misaligned optical components. Rather than show the results for all 62 channels, only a subset of individual images, from banks 2 through to 4 (i.e. 24 frames in all, from F16 through F40 (viz Fig. 1)), are used for illustrative purposes for the rest of this paper. In each sequence file, a random static fiducial was selected, and its pixel coordinates were tracked across the following 24 frames using commercial image processing software (ImageJ v.1.46r using the MTrackJ plug-in [20]). Subsequently, in any particular frame i (where i = 0-23), the horizontal and vertical pixel positions of that pre-selected fiducial were defined as x i and y i respectively (the pixel position of the reference frame F16 (or frame i=0) is thus designated (x 0 , y 0 ).
Similarly, the relative displacements for the static fiducial (in pixels) for successive frames with reference to frame i = 0 are: (1) and (2) The inter-frame displacements for static fiducials obtained at different frame rates are displayed in Fig. 2. Evidently, measurements on dx i and dy i usually exhibited small variances as a function of frame rate. This suggested that misalignment of optical components along the beam path to each CCD sensor was the dominant factor contributing to image jitter, rather than internal vibrations. Here, vibration related deviations (relative to an arbitrary origin: dx i=0 & dy i=0 ) appear minimal (i.e. displacements were <10 pixels, over all framing rates) compared to the few more severely misaligned frames (at i = 2 or 8 for example (Fig. 2)) This resulted in |dx i | and |dy i | having maxima of 57 and 16 pixels respectively (which corresponds to 5.7% and 1.6% of the associated CCD lateral dimension), again underscoring the need for an alignment compensation algorithm.
Consistent dependence of these image displacements on the absolute frame number (F) became obvious, facilitating direct implementation of a correction algorithm. An arbitrary reference frame (i=0, corresponding to F16) was selected, and all subsequent frame displacements were assigned relative to this. Specifically, we measured the x-& ydisplacements for a static fiducial over all subsequent frames, then mathematically compensated for these in the recompiled sequence. This is equivalent to a standard tracking algorithm where the displacement of some specific feature between each image frame is monitored relative to the Cartesian position of a nominated static fiducial -after which we then calculate dx i and dy i over all frames using Eq. (1) & Eq. (2). The x-y coordinates of the target are designated as, x i,o and y i,o for the i th frame, and similarly, the uncorrected relative displacements (dx i,o , dy i,o ) for the target fiducial are: (3) and (4) where i is restricted to specific frames of interest and x 0,o and y 0,o represents target coordinate position of the arbitrary reference frame. Jitter compensated relative displacements (dx i* , dy i* ) are then represented thus: (5) and (6) The presence of dynamic fiducials (Fig. 3) was also exploited by selecting both a static fiducial and a dynamic fiducial for each individual sequence set, and continually monitoring their respective pixel coordinates. Using the jitter correction algorithm (Eqs. (3)-(6)), we obtained the corrected coordinate displacements of the dynamic fiducial, and in turn, its Cartesian velocity components V x and V y respectively (note that a hardware-borne diagnostic read-out on frame rate is used to calculate this and the units are expressed as [m.s −1 ]). The dynamic fiducial shown in Fig. 3 is seen to displace along the y-axis (Fig. 3(ah)). Its relative displacement can be jitter-corrected by subtracting the relative displacement of a nominated static fiducial. Subsequent calculations return values of |V x | and |V y | at 0.01 m.s −1 & 1.00 m.s −1 respectively. As |V x | values in all cases were less than 2% of |V y |, they are presumed negligible. A graph of |V y | against Ω for both raw uncorrected data, and its jitter corrected counterpart are shown in Fig. 4. Here, it is evident that uncorrected data can have a large deviation from the mean. The fiducial-based jitter-correction procedure reduced such deviations significantly. Moreover, Fig. 4 supports the view that the average dynamic fiducial velocity is indeed correlated with mirror rotational speed. The superior aesthetic enhancement arising via our approach, is underscored in Media 2, where sequence excerpts from: (a) a raw data file; (b) the fiducial-based corrected version of that same raw data file; and (c), an industry-standard algorithm [8] corrected version of the raw data in (a), are displayed together for ease of comparison.

Conclusions
Having observed image jitter within image sequences acquired on a rotating mirror camera (Media 2 inset (a)), we showed that this apparent frame-to-frame displacement arose mainly due to small geometric misalignments in the optical hardware. Judicious choice of static and dynamic fiducials allowed us to execute a correction algorithm that compensated for camera jitter (Media 2 inset (b)) which performed comparably with an industry standard processing routine [8] (Media 2, inset (c)) but also produced a superior aesthetic result. Moreover, closer inspection over many imaging sequences showed that the five mirror facets each exhibit characteristic features (scratches/minute imperfections): i.e., each facet has a signature fiducial pattern, a realisation that facilitates analytical discrimination of sequence records on a facet-by-facet basis: useful information for undertaking downstream data diagnostics, and especially with identifying sources of aberration. This effectively adds an additional level of inspection.
We also demonstrated that jitter-corrected velocities of dynamic fiducials are correlated to frame rates recorded by the camera (Fig. 4), and moreover, that implementation of a jitter correction algorithm suppressed the standard deviation, relative to uncorrected data, by 88.5% maximally, and 66.5% on average over all framing rates. Analysis suggests that native fiducials are exploitable as accurate embedded scales for micro-imaging scenarios and moreover, that users should be able to accurately (and independently from the hardware) predict instantaneous framing rates. Whilst we have used the serendipitous appearance of native fiducials for our present analyses, we of course recognise that deliberately introduced fiducials, such as optical graticules or embossed mirrors could perform the same duties.

Supplementary Material
Refer to Web version on PubMed Central for supplementary material. (inset) Representative image of two 10-micrometer diameter glass calibration beads, and indication of the coordinate system used in the methodology.