NanoJ: a high-performance open-source super-resolution microscopy toolbox

Fluorescence microscopy has been ubiquitously used in biological studies since its invention in the 20th century. It can reveal subcellular structures and interactions between specifically labelled molecules and allows the quantification of their dynamic behaviour in living cells [1]. Extraction of this biologically relevant quantitative information from fluorescence microscopy data typically requires digital image processing and analysis [2]. In recent years, super-resolution microscopy (SRM) techniques have extended the spatial resolving power of fluorescence microscopy beyond the diffraction limit [3–5]. Most SRM techniques use large quantities of raw data, often reaching several gigabytes to generate a single super-resolution image, thus requiring specialised high-performance image analysis tools. Several SRM image processing packages are available, such as ThunderSTORM [6], LAMA [7] and SIMcheck [8] but each of these is focused on a specific type of SRM modality. Here, we present NanoJ, a highly versatile set of image acquisition and analysis methods developed to improve the reliability and quantifiability of microscopy experiments, with a particular focus on the demands of live-cell SRM. NanoJ is available as a series of ImageJ-based plugins which can be used independently or concomitantly. NanoJ (figure 1) is comprised of the following modules: NanoJ-Core—general image correction tools including drift correction and channel registration, both based on cross-correlation analysis; NanoJ-SRRF—an analytical approach capable of extracting super-resolution data from a short sequence of diffraction-limited images, which can be acquired using most microscopes [9, 10]; NanoJ-SQUIRREL—an algorithm to evaluate resolution and the presence of artefacts in super-resolution images [11]; NanoJ-VirusMapper—a single particle analysis method to generate nanoscale models of biological structures such as viruses [12–14]; NanoJ-Fluidics—a hardware and software framework to control fluidics devices, enabling automation of multiplexed experiments [15]. Thus, the NanoJ framework is capable of solving common imaging problems with broad biological applications and is compatible with a multitude of fluorescence microscope setups and experimental protocols.

Zoom In Zoom Out Reset image size

NanoJ has been designed to integrate with the popular ImageJ or Fiji image analysis software [16, 17] and is easily installed as a standard set of plugins. NanoJ is also fully open-source and user-friendly. The graphical user interfaces (GUIs) are straightforward to use and its routines can be easily integrated within larger image analysis pipelines through the ImageJ macro language.

NanoJ is designed to be an accessible tool for both non-expert users and developers. Each of its modules possesses its own separate manual and documentation. NanoJ is implemented in both Java (https://java.com/) and OpenCL (https://khronos.org/opencl), the latter language being used for high-performance analysis of image data through the use of graphical processing units (GPUs). To date, it encompasses four Java ARchive (JAR) packages (NanoJ-SRRF, NanoJ-SQUIRREL, NanoJ-VirusMapper, NanoJ-Fluidics) that all depend on a central package (NanoJ-Core). The core package hosts the libraries that enable high-performance GPU-based computing analysis and a set of basic image analysis helper methods. The modular nature of NanoJ means that its components can be updated independently and the framework can be easily extended by appending new analytic packages.

Sample drift commonly occurs during the acquisition of SRM data, often as a result of gradual changes in the temperature of microscope components. Drift introduces motion blur artefacts and thereby a loss of resolution. While most modern microscopes have an active focus-lock device that stabilises the motion of the sample in the axial direction (minimising focal drift), the sample will still be prone to lateral movement (figure 2(a)). However, in the case where the raw data is made up of a sequence of consecutive frames acquired rapidly, as is common in SRM methods such as single molecule localization microscopy (SMLM) [3, 4] or fluctuation-based approaches [9, 18, 19], this lateral drift can be estimated (figure 2(b)) and analytically corrected for each time frame via post-processing (figures 2(c) and (d)). NanoJ breaks the task of drift correction into two distinct parts: estimation, followed by translation. As a first step, NanoJ-Core estimates the linear drift between two images by calculating their cross-correlation matrix (CCM) (figure 2(b)). The location of the peak intensity in the CCM determines the linear shift between the two images, and precise sub-pixel accuracy is achieved by up-scaling the CCM using a bicubic spline interpolation. Depending on the type of acquisition, the reference frame can either be the first frame of the raw data or the immediately preceding frame. Figure 2(d) shows the drift in a 100-frame dataset as measured with respect to the first frame. Once drift is estimated, the dataset can be directly corrected by analytically translating each individual frame using a bicubic spline interpolation (figure 2(c)). We estimated the performance of NanoJ-Drift correction (see supplementary note 3 and figure S1a (stacks.iop.org/JPhysD/52/163001/mmedia)) at varying levels of signal-to-noise ratio (SNR) and established that the errors remain within 30 nm even at very low SNR and drop quickly below 10 nm for SNR  >  6, appropriate for SRM.

Zoom In Zoom Out Reset image size

The interpolation process will, however, change the noise properties of the resulting dataset [20], which can have an influence on further analysis requiring specific assumptions in noise properties such as maximum likelihood estimation (MLE) single-molecule localization analyses.

For the specific case of SMLM datasets with sparse blinking, there will only be a weak correlation across frames, as there is little observable structure conserved between consecutive time points. One common strategy to alleviate this low correlation is to add fiduciary landmarks to the sample, such as static fluorescent beads. As an option NanoJ-Core can temporally bin images within the dataset, thus increasing the correlation between frames and allowing their shift to be more accurately estimated [21].

Drift estimation in NanoJ differs from strategies applied by other SRM algorithms, such as ThunderSTORM [6], by analysing unprocessed raw data instead of post-processed super-resolution reconstructions. This allows the estimation to be decoupled from the super-resolution image reconstruction algorithm and hence drift-corrected raw data can be analysed using a wider range of methods including SRRF [9] and SOFI [18]. Furthermore, NanoJ-SRRF (as described in a following section) can import the drift curve (figure 2(d)) created by NanoJ-Core Drift Correction and use this information directly during analysis without the need to pre-translate each frame in the raw dataset.

In multicolour fluorescence microscopy, images acquired in different spectral channels are often misaligned as a result of chromatic aberrations in the optical path and the use of different filter sets for each colour. This misalignment is frequently ignored in conventional microscopy as it typically occurs on a scale smaller than the diffraction limit. However, this effect becomes non-negligible in the context of SRM [22]. Channel registration is therefore essential for multicolour SRM studies quantifying colocalization or interactions between different structures [23–25]. Advanced imaging research groups tend to develop their own analysis scripts for their particular applications, and thus no user-friendly tools are readily available. Therefore, NanoJ-Core channel registration offers a unique tool to perform this registration easily and robustly on a wide variety of datasets.

The shift between different spectral channels is usually inhomogeneous across a field of view, which prevents the use of typical CCM-based approaches such as the one described for NanoJ-Core drift correction. In order to characterise the spectral misalignment across a field of view, it is necessary to image a sample where the same structure can be observed across all spectral channels of interest. Furthermore, the sample should contain structures occupying the whole field of view. A typical sample for this characterisation is a coverslip coated with a large number of beads labelled with multiple fluorescent dyes [26]. Following multicolour imaging of this sample, NanoJ-Core can be used to calculate a nonlinear 2D spatial transform describing the misalignment for each channel relative to a reference channel. Given that chromatic misalignment is a fixed property of an optical system, NanoJ-Core can then apply these transforms to realign other multicolour datasets acquired using the same optical path (figure 3) [27, 28].

Zoom In Zoom Out Reset image size

To generate these non-linear channel registration fields NanoJ-Core first calculates local, linear misalignments between channels (figure 3(a)). This is achieved by dividing the image into small areas ('blocks'). For each block, the shift is calculated by finding the cross-correlation peak position as shown in figures 2(a) and (b). These local shift values are then interpolated across the whole field-of-view using an inverse distance weighting interpolation [29]. This generates two smooth shift maps describing the misalignments in the horizontal and vertical directions (figure 3(b)). For a given channel, each pixel value within the horizontal/vertical shift map indicates the horizontal/vertical displacement that needs to be applied to that pixel to align it with the reference channel. NanoJ-Core can then be used to apply these maps to align channels in the dataset of interest (figure 3), provided they have been acquired using the same optical configuration. Here, we measured the target registration error [28, 30] (TRE, see supplementary note 3 and figure S1b) in typical multi-colour configurations and showed that the TRE remains below 50 nm for all combinations of channels.

NanoJ-Core performs the channel registration by creating a new image representing each channel, where the intensity value for each pixel coordinate corresponds to the intensity value from the original image at the equivalent coordinate corrected for local shift. For cases in which these coordinates are not discrete (sub-pixel shift), a bicubic spline interpolation is used to recover pixel values in continuous space. Because the shift map can be extrapolated to continuous space, the registration procedure obtained from diffraction-limited images can also easily be used to correct super-resolved images obtained using the same optical configuration.

As part of the NanoJ framework, we include our recently developed SRM reconstruction algorithm: super-resolution radial fluctuations (SRRF), which is able to extract sub-diffraction information from a short burst of images acquired at high speed with modern fluorescence microscopes [9, 10]. SRRF is a purely analytical approach. It alleviates the need to use toxic photoswitching-inducing buffers [31], specialised fluorophores [32, 33], damaging high-intensity illumination [34] or specialised equipment [5, 35] when compared to other SRM methods [3–5, 35].

SRRF is based on similar principles to SMLM, with the key difference that it does not rely on the detection of spatio-temporally isolated fluorophores. Instead, SRRF generates a magnified pixel grid where each pixel value relates to the probability of fluorophores existing in that corresponding region of space. To do this, SRRF calculates the local radial symmetry ('radiality') in each pixel of the magnified image using local intensity gradient information. The obtained radiality value will be high when a point-spread-function (PSF) profile transiently becomes dominant, highlighting the presence of a fluorescent molecule at that location. Furthermore, the fluctuations of radiality values follow the underlying natural intensity fluctuations of fluorophores, which have a distinct temporal signature to that of noise [18].

Therefore, a temporal correlation of the radiality at each pixel can be projected into a final image, where the structures of interest will be better resolved.

Thanks to its low-illumination requirement, SRRF is highly suited to enable live-cell SRM and allows for long SRM timelapse acquisitions [10]. Figure 4 shows a typical live-cell SRRF acquisition of a COS-7 cell expressing UtrCH-GFP (a probe for actin filaments) imaged at 33.3 Hz. The dataset shown is part of a longer  >  30 min time-course dataset (see supplementary movie). SRRF allows the observation of protein dynamics (e.g. actin) during long periods with no perceivable phototoxicity.

Zoom In Zoom Out Reset image size

NanoJ-SRRF, the software implementation of the SRRF algorithm, uses GPU computing whenever possible to accelerate radial symmetry calculations. This is achieved by implementing part of its analytic engine in OpenCL, with a fallback of execution to the CPU if a compatible graphics card is not detected.

SRM techniques are more complex than conventional diffraction-limited microscopy. This complexity arises from the sample preparation requirements (SMLM), the microscope hardware (stimulated emission depletion microscopy or STED, structure illumination microscopy or SIM) and the post-acquisition image processing (SMLM, SIM). Unsuitable choices in any of these underlying variables can lead to artefacts within the final image and the potential for false conclusions to be drawn. NanoJ-SQUIRREL (super-resolution quantitative image rating and reporting of error locations) is an algorithm that highlights the presence of artefacts in super-resolution images. It does so by calculating quantitative maps showing both local SRM image quality and resolution [11] and can thus be used to optimise acquisition and analysis workflows [10].

The central concept of SQUIRREL is that a diffraction-limited image and the corresponding SRM image of the same region should contain the same underlying structure, just at different resolutions. Thus, the diffraction-limited image can be treated as a high-confidence benchmark against which the SRM image can be compared. The SQUIRREL image quality assessment algorithm formalises this comparison analytically.

NanoJ-SQUIRREL quality assessment requires the user to provide an acquired diffraction-limited image and the SRM image of the same structure (figure 5(a)). As both images represent the underlying fluorophore distribution but at different resolutions, there exists a blurring function that can convert the SRM image into its diffraction-limited equivalent. SQUIRREL calculates this blurring function and applies it to the SRM image. The blurred image is then compared against the diffraction-limited reference. This generates three quality indicators. An error map is produced that maps the discrepancy between the blurred SRM image and the reference image at every pixel (figure 5(b)). This highlights regions where the super-resolution image is inconsistent with the reference image. For example, the three insets in figure 5(b) show typical SMLM artefacts including incomplete structures and interstructure mislocalization. Two global quality metrics are also generated for each image: the RSP (resolution scaled Pearson's correlation coefficient) and the RSE (resolution scaled error). The RSP can take a value in the interval [−1, 1] and describes the structural agreement between the blurred super-resolution and reference images. Here, higher values indicate better agreement, with an RSP of 1 indicating a perfect structural match. The RSE describes the mean intensity mismatch between the blurred super-resolution and reference images; in this case lower values represent better agreement with a value of 0 indicating a perfect intensity match.

Zoom In Zoom Out Reset image size

The purpose of SRM is to resolve finer structural detail than is achievable with conventional diffraction-limited microscopy. It is therefore useful to have an objective measurement of resolution within a super-resolution image, for example to enable meaningful analysis and structural hypotheses that stay in line with the actual precision of the data. The current standard for measuring image resolution in SMLM images is Fourier ring correlation (FRC) [36]. This method involves comparing two independently acquired super-resolution images of the same field-of-view so that they only differ by their noise components. For SMLM datasets the two SRM images are usually obtained by splitting localizations from odd and even frames. The correlation between these two SRM images is measured at different frequencies in Fourier space; the frequency at which this correlation drops below a set threshold indicates the resolution of the image. FRC has been previously implemented in ImageJ [36], but only gives a single resolution measurement for the entire field of view. However, resolution is not necessarily homogeneous across the SRM image (figure 5(c)). This is particularly true for SMLM methods as localization accuracy depends strongly on labelling density and laser illumination intensity, which can both vary considerably within a single field of view.

Furthermore, FRC can generate biased measurements for certain fluorophore distributions such as point-like patterns. Therefore, an additional feature of the NanoJ-SQUIRREL plugin is local mapping of FRC resolution across an image. To do this, the user provides an image stack comprising two independent renderings of the same dataset (e.g. through the odd/even frames splitting as described above). The images are then spatially split into equal-sized blocks and FRC analysis is run locally on each block. For blocks where there is insufficient correlation to generate an FRC resolution value, a resolution value is interpolated from neighbouring blocks. Figure 5(c) shows the FRC map obtained from the SRM image shown in (a). This map highlights that the resolution in this image varies between 85 and 36 nm.

It is important to note that high resolution (that is, a low FRC value) does not imply that the super-resolution image has depicted structures correctly; it only means that there is low variation in the locations of the fluorophores between the two rendered images. Therefore, the error mapping functionality within NanoJ-SQUIRREL can complement FRC-mapping in order to obtain a more complete perspective on super-resolution image quality.

As part of the NanoJ framework, we include a unique single-particle analysis (SPA) tool called NanoJ-VirusMapper. It is the first open-source, freely available algorithm for unbiased, high-throughput SPA of fluorescence imaging and allows the structural modelling of viruses and other macromolecular complexes [12–14]. The principle of SPA is to image many identical copies of a structure, independently of its orientation, and align and combine them to build an averaged structural map of the underlying structure with high SNR [37–41].

The SPA implementation of VirusMapper facilitates automatic processing of multiple images to detect, segment, align, classify and average thousands of individual structures. It is entirely general, assuming no underlying symmetry or other properties of the imaged structure. Here, we illustrate this with models of three distinct vaccinia virus substructures [42]: core, lateral bodies (LBs) and membrane. All three substructures were labelled on viral particles and imaged with SIM (figure 6(a), top). VirusMapper can explicitly incorporate information from multiple fluorescence channels to enable multi-component modelling. Here, the images of the LBs and the core were used to find the orientations of each particle in the dataset. This allowed us to create simultaneous models of the three components of the virus (figure 6(a), bottom).

Zoom In Zoom Out Reset image size

VirusMapper can also be used to separately model different 3D orientations of a structure. Here, we highlight this capability on a study of the division ring in the thermoacidophilic archaeon Sulfolobus acidocaldarius, strain DSM639 (figure 6(b)). Sulfolobus cells were labelled for both their outer S-layer and the division ring, as marked by the archaeal ESCRT-III homolog CdvB. The images of the division ring were used to identify different orientations of the cells (figure 6(b)) and separate templates were created using VirusMapper. Parallel models were then generated of the division ring and the S-layer using these templates for each orientation. The models obtained from the two channels can then be overlayed (figure 6(b), bottom). VirusMapper is compatible with any fluorescence microscopy method and has been demonstrated with SIM, STED microscopy [12] and SMLM (figure S1). The general approach of the method enables it to even be applied to correlative combinations of methods (figure S1).

NanoJ-Fluidics is a hardware and software framework for precise and accurate automated liquids exchange [15]. It was developed to enable automation of sample treatment and labelling of live or fixed specimens directly on the microscope stage [15, 43]. The NanoJ-Fluidics hardware component is composed of customisable, low-cost and robust LEGO^® syringe pumps and a liquid removal peristaltic pump, all controlled by simple Arduino^® electronics. It is compatible with off-the-shelf imaging chambers, without the need for any microfabrication. Its control software (the NanoJ-Fluidics software) is ImageJ-based and can be fully integrated with microscopy acquisition software. We have demonstrated the applicability of NanoJ-Fluidics in multiple experimental contexts, including in-situ correlative live-to-fixed super-resolution imaging, multimodal super-resolution imaging and event-driven fixation [15]. The approach can also be easily extended to protocol optimisation (e.g. titrating antibody concentrations or adjusting imaging buffer composition) or liquid exchange protocols integrated with the imaging (e.g. drug delivery or automated event-driven fixation).

Here, we demonstrate NanoJ-Fluidics by acquiring a high quality multicolour SMLM dataset, where STORM and DNA-PAINT [44] imaging strategies are combined into a single workflow (figure 7(a)). This approach is particularly suited to multi-target imaging [32], which is difficult to achieve with standard sample preparation techniques due to the low number of suitable fluorophores available for SMLM microscopy. With NanoJ-Fluidics, we can seamlessly perform all labelling steps in an automated and reliable manner directly on the microscope stage. We showcase this using a four-channel acquisition of actin with STORM, and mitochondria, vimentin and clathrin with DNA-PAINT (figure 7(b)). NanoJ-Fluidics' highly customisable nature has already spawned several alternative designs from the community (https://github.com/HenriquesLab/NanoJ-Fluidics/wiki), which are in constant development. NanoJ-Fluidics makes imaging protocol automation readily available to researchers, hence improving not only the reliability and repeatability of the protocols, but also the range of protocols that are achievable.

Zoom In Zoom Out Reset image size

The NanoJ framework provides a unique and comprehensive set of tools to support fluorescence imaging, from data acquisition and protocol optimisation to structural quantification. It offers powerful and user-friendly solutions to common pitfalls of image analysis such as drift correction and channel registration (NanoJ-Core). It extends the tools available for SRM to expert and novice users alike and aligns with the general aim of the community to extend SRM to live-cell applications, offering the first open-source analytical approach for live cell SRM imaging (NanoJ-SRRF). NanoJ offers a way to correlate dynamic live-cell data with high-resolution fixed cell data in an automated and highly reproducible manner (NanoJ-Fluidics). It incorporates an analytical method for the generation of accurate, high-content, molecular specific models (NanoJ-VirusMapper). Finally, the NanoJ framework provides tools to quantitatively assess SRM image quality by spatially mapping unbiased quality metrics, artefacts (errors) and FRC resolution (NanoJ-SQUIRREL), with the aim to improve standards in assessing and reporting microscopy data.

The NanoJ framework gives access to user-friendly yet robust analytical methods in tandem with new approaches to perform live-cell SRM experiments. A typical SRM live-cell imaging experiment using the NanoJ framework would entail acquiring a SRRF-compatible raw dataset in any modern fluorescence microscope; performing drift correction and channel registration; followed by SRRF analysis for SRM image reconstruction; and finally, quality check using SQUIRREL. Used concomitantly in this manner, these tools can be used to address a large variety of biological questions and can be seamlessly repeated for imaging protocol optimisation and complemented using NanoJ-VirusMapper and/or NanoJ-Fluidics. Thus, the NanoJ-enabled scientific pipeline allows any user, regardless of the experience level, to seamlessly obtain SRM quantitative data of the highest scientific quality. The NanoJ framework was developed to facilitate access and expand the options of research groups using fluorescence microscopy. Hence, speed, reliability, performance and cross-compatibility are of paramount importance. In this context, NanoJ has been designed for high-performance image analysis, using GPU computing, ensuring the quick processing of large data volumes [45–49]. Further, NanoJ's modular and open-source nature and its integration within ImageJ allow it to be used in conjunction with other analysis software packages [50–53]. We are continuously supporting, adapting and expanding the framework to include new approaches, such as 3D imaging.

We expect NanoJ to set the standard for useful, open-source, high performance methods for the whole microscopy community.

NanoJ follows opensource software and hardware standards. Each of its modules can be installed by enabling the corresponding code repository in Fiji or by following the instructions on the corresponding websites:

• https://github.com/HenriquesLab/NanoJ-Core
• https://github.com/HenriquesLab/NanoJ-SRRF
• https://bitbucket.org/rhenriqueslab/NanoJ-SQUIRREL
• https://bitbucket.org/rhenriqueslab/NanoJ-VirusMapper
• https://github.com/HenriquesLab/NanoJ-Fluidics

First and foremost, we would like to thank the ImageJ and Fiji open-source development community. Their astonishing work has been the inspiration to the creation of NanoJ. There has been a large number of beta-testers and NanoJ users whose feedback helped us to continually update and improve its code, to the best of our capacity we keep an updated list of significant contributors in the website corresponding to each NanoJ module. We thank Professor Ralf Jungmann at Max Planck Institute of Biochemistry Munich for reagents and advice. This work was funded by grants from the UK Biotechnology and Biological Sciences Research Council (BB/M022374/1; BB/P027431/1; BB/R000697/1; BB/S507532/1) (R H, P M P and R F L), the UK Medical Research Council (MR/K015826/1) (R H), the Wellcome Trust (203276/Z/16/Z) (S C, R H and B B), Core funding to the MRC Laboratory for Molecular Cell Biology, University College London (MC_UU12018/7) (J M), the European Research Council (649101—UbiProPox) (J M) and the Centre National de la Recherche Scientifique (CNRS ATIP-AVENIR program AO2016) (C L). N G and R D M G were funded by the Engineering and Physical Sciences Research Council (EP/L504889/1). P A was supported by a PhD fellowship from the UK's Biotechnology and Biological Sciences Research Council. Research by B B was supported by UCL, Cancer Research UK (C1529/A17343), BBSRC (BB/P001440/1) and MRC (MC_CF12266). K L T and G T R are supported by a 4 year MRC Research Studentship. D A is presently a Marie Curie fellow (Marie Sklodowska-Curie 750673). A C L and F H were supported by the Swedish Research Council, grant 621-20134685.

These contributions follow the Contributor Roles Taxonomy guidelines: https://casrai.org/credit/. Conceptualization: K L T, R F L, N G, R D M G, P A, D A, G T R, B B, J M, C L, P M P, S C, R H; data curation: K L T, R F L, R D M G, G T R, J M, C L, P M P, S C, R H; formal analysis: K L T, R F L, R D M G, C L, S C; funding acquisition: A C L, B B, C L, R H; investigation: K L T, R F L, R D M G, G T R, C L, P M P, S C; methodology: R F L, N G, R D M G, P A, J M, C L, P M P, S C, R H; project administration: R F L, B B, J M, C L, P M P, S C, R H; supervision: R F L, A L, B B, J M, C L, P M P, S C, R H; validation: K L T, R F L, N G, R D M G, P A, D A, C L, P M P, S C, R H; visualization: K L T, R F L, R D M G, D A, C L, P M P, S C, R H; writing (original draft): K L T, R F L, R D M G, D A, C L, P M P, S C, R H; writing (review and editing): all authors.

The authors declare no competing financial interests.