Microscope-Cockpit: Python-based bespoke microscopy for bio-medical science

We have developed “Microscope-Cockpit” (Cockpit), a highly adaptable open source user-friendly Python-based Graphical User Interface (GUI) environment for precision control of both simple and elaborate bespoke microscope systems. The user environment allows next-generation near instantaneous navigation of the entire slide landscape for efficient selection of specimens of interest and automated acquisition without the use of eyepieces. Cockpit uses “Python-Microscope” (Microscope) for high-performance coordinated control of a wide range of hardware devices using open source software. Microscope also controls complex hardware devices such as deformable mirrors for aberration correction and spatial light modulators for structured illumination via abstracted device models. We demonstrate the advantages of the Cockpit platform using several bespoke microscopes, including a simple widefield system and a complex system with adaptive optics and structured illumination. A key strength of Cockpit is its use of Python, which means that any microscope built with Cockpit is ready for future customisation by simply adding new libraries, for example machine learning algorithms to enable automated microscopy decision making while imaging.


Introduction
Why we need Cockpit Biomedical research has benefited from significant engineering advances that have increased the speed and sensitivity of many types of scientific instrumentation. Microscope technology, in particular, is evolving rapidly through advances in optical techniques and image processing, enabling major scientific discovery. However, the rapid adoption and application of advanced microscopies by biomedical scientists is constrained, especially when it relies on commercialisation to make the necessary technology accessible to the end-users.
It can often take several years to develop innovations from research instruments into user-friendly, off-the-shelf systems [1][2][3][4] . We report a collaboration between the University of Oxford and the University of California San Francisco (UCSF) that grew out of the OMX microscope project, dating from the early 1990s, which is extensively documented in the supplement of 1.
To accelerate technology adoption, many labs are building custom microscopes based on newly developed imaging methods, such as light sheet 5 , lattice light sheet 3 , 3DSIM 1,6 , STED 7 , and MINFLUX 8 , or to exploit new analysis techniques, such as STORM analysis 9,10 , 3B 11 , and SIMFLUX 12 . These instruments require flexible, computerised control of a wide range of individual components (lasers, mirror actuators, filters, objectives, detectors, etc.) allowing more rapid deployment of novel techniques, and often providing faster and more sensitive operation than commercial systems 3,13,14 . However, integration of these components into a single user-friendly platform that is focused on users' scientific application, rather than dictated by the control infrastructure, can be a major challenge. There have been three general solutions adopted by the community for software control of bespoke microscope hardware.
1. Using individual manufacturer-provided packages for each piece of hardware. This can provide a straightforward solution for simpler systems, but fundamental incompatibilities between each vendor's control models can prevent integration of multiple components.
2. Using LabVIEW, a commercial visual programming tool from a leading hardware manufacturer, National Instruments (Austin, Texas, United States).
3. Use of a dedicated microscope control package, of which several are available, with the most widely used being Micro-manager (µManager): an open source Java and C++ based software platform for controlling a range of microscope hardware such as stands, cameras, and stages. A less widely-adopted solution is a bespoke platform written in Python to control the OMX microscope 1,15,16 , and this forms the starting point of the Microscope-Cockpit project described in this manuscript.
An important feature for custom microscopes is the freedom to design the system around a specific set of experiments, as defined by the user. With a specific application in mind, an instrument can be optimised to make the best possible compromises for that application. For example, imaging deep into biological tissue can involve significant optical aberrations leading to degraded image quality. This can be compensated for by the application of adaptive optics techniques, at a cost of increased complexity and marginally decreased light efficiency. However, it is desirable that this capability is easily enabled and exploited by microscope users, typically biologists with limited experience of the underlying engineering, and be reproducible over months or years of use. Many such custom microscope designs have been published [17][18][19][20][21][22] . In all of these cases, control of the many hardware components in a complex and timed manner is challenging, particularly for time sensitive applications involving living specimens. Correction settings must be measured and applied in coordination while simultaneously minimising additional exposure of the sample.
Finally, the democratisation of artificial intelligence (AI)-based image analysis approaches means that a wealth of tools are potentially available to users. However, current systems, with some specific exceptions, are not well suited for the integration of bespoke novel AI-approaches with the imaging process. This limitation can be significantly reduced by an appropriately flexible package for control of bespoke systems.

Philosophy and implementation of Cockpit
We set out to create a package that is suitable for biologists to carry out a range of experiments in a time efficient manner, at scale. At the same time, the software has to control a wide range of electronic and optical hardware devices and microscope types. Many bespoke microscope systems that lack eyepieces, such as systems built by physical scientists in collaboration with biologists, make sample navigation particularly challenging. Improving navigation on this type of system is a major feature of Cockpit. mosaic tool, allowing large areas of a microscope specimen to be imaged and then navigated in real time. This is similar to Micro-Magellan 23 , but with the additional feature of keeping even thousands of images in Graphic Processing Unit (GPU) memory for instant access at various levels of detail. It utilises Python-Microscope 24 to communicate with a wide range of microscope hardware and is able to exploit a master timing control device to provide hardware triggering for digital devices and synchronised analogue voltages. This enables highly reproducible timing over a range of time scales even in extremely complex experiments with multiple devices.
The simple interface design ( Figure 1) enables users to focus on collecting the data they require without having to devise complex control infrastructure for systems of any level of complexity. It also enables easy navigation of even large sample regions. This can be especially important on bespoke microscopes which often lack eyepieces. For these types of system, Cockpit includes a touchscreen-optimised display window, providing large buttons and simple control over most functions ( Figure 3).
The principal design goals for the Cockpit interface were to: • Provide an easy-to-use, simple "Cockpit" control program for a wide range of microscopes.
• Hide from the user the complexity required to control an advanced microscope.  • Minimise the effort required to change hardware or integrate new hardware.
• Enable high-precision hardware timing signals to optimise experimental control, facilitate complex systems and maximise speed.
Cockpit provides an integrated microscope control package that enables all these features while being user friendly and portable across many different microscopes and common operating systems.
The first version of the Cockpit software was developed in UCSF to drive the OMX microscope 1,15,16 , which was later commercialised. This code was re-written at UCSF and later released as open source under a BSD license. From this concept we have extensively developed the implementation of Cockpit to realise its full potential.

Implementation
Cockpit is implemented in the Python programming language 25 . The code consists of four main components: devices, handlers, interfaces, and the Graphical User Interface (GUI). Devices represent the individual physical devices, handlers represent control components of the devices, handlers from single or multiple devices are aggregated into interfaces, and the GUI component allows user interaction with the different interfaces.

Devices and handlers
The hardware control side of Cockpit has two parts: devices and handlers. A Cockpit device maps to a single physical device while handlers abstract the control components of the device to be used. For example, an XY stage device provides two handlers, one per axis. A laser device also provides two handlers, a light source handler and a light power handler: the light source handler controls the light source state -on or off -while the light power handler controls its intensity.
Most of the Cockpit code does not interact with Cockpit devices directly. Rather, when the Cockpit program starts, it constructs a "depot" object whose role is to obtain and keep track of the different device handlers. This architecture enables rapid changes in the choices of hardware devices, so they can be very easily added or removed from a complex system as it is being developed. This approach prevents potential conflicts or confusions such as addressing devices that do not exist or adding devices that lack drivers.

Interfaces and GUI
Cockpit defines interfaces that are high-level abstractions for the system, with each controlling one or more handlers. For example, the stageMover interface is a unified translation manager that coordinates all translation stages, hiding the complexity of handling different stages for different axes and nested stages on the same axis. For instance, a system may include an XY stage, a coarse Z stage, and a high precision fast piezo Z stage. The GUI component provides different views onto the multiple interfaces and handlers, each exposing different levels of complexity.
Cockpit has a multi-window GUI, with each window serving a different purpose ( Figure 1). This provides the flexibility to distribute the windows over multiple monitors in whatever form best fits the system being used. The main window provides an at-a-glance view of the state of the various devices; the mosaic window enables navigation of the sample relative to an overview formed of tiled images; the macro stage window displays the position of the different stages; and the camera view the current images from all active cameras. In addition, Cockpit also has a simpler single-window touchscreen interface ( Figure 3) which duplicates a subset of the functionality provided on the other windows. This window is optimised for touchscreen interaction with large buttons for the most useful subset of functionality.
Cockpit uses wxPython, a Python wrapper to wxWidgets, to provide a user interface that has a native look and feel on the different platforms ( Figure 2). Within wxPython, we make extensive use of OpenGL to enable high performance display and interaction with large images or image mosaics up to gigapixel sizes.

Data acquisition experiments
In Cockpit, the acquisition of an image series, such as a Z-stack or a time lapse sequence, is defined by an experiment. Cockpit experiments are based around the concept of an action table which must be run with hardware triggers. The experiment settings are converted into a list of device actions, which is parsed into a list of pre-computed analogue values and digital levels for each time point on a timing device. The action table is uploaded to the timing device and then started, running the experiment utilising hardware triggers and analogue voltages, where appropriate, to control all active devices. This approach, adapted from Carlton et al. 1 , is able to produce both digital triggers and analogue signals with precise timing.
Effective execution of the experiment requires accurate timing. In order to ensure that experiment timing is independent of the host computer's performance or the Python implementation, the timing functionality is performed using dedicated hardware. This enables time-critical operation of the microscope imaging tasks. The timing system has been implemented on several different hardware platforms, a digital signal processing board (Innovative Integration M67-160 with an A4D4 daughter board), an NI FPGA board (National Instruments cRIO-9068), and a Red Pitaya single board computer (STEMlab 125-14). In this way, we have created a universal and adaptable microscope platform with outstanding timing precision and accuracy.
Output image files Experimental images are saved into files utilising the '.dv' file format, typically multiple wavelengths, Z-slices, and time points into a single file. Images from the mosaic window and single snaps can also be saved into this format, with the mosaic saved files having an associated text file defining XYZ positions of each collected image. The '.dv' format is an extension of the mrc file format, defined in detail in the MRC/CCP4 2014 (https://www.ccpem.ac.uk/mrc_format/mrc2014.php) file format specification 26 . The CCP4 consortium of the EM community continue to support and extend this file format. This support includes file validators and a detailed specification, which is compatible with the files used here but not identical.
The optical microscopy specific metadata are covered in the documentation. Although relatively uncommon, the file format is supported by the Bio-Formats (https://www.openmicroscopy. org/bio-formats/) project allowing import of '.dv' files, along with the associated metadata into software using this library including ImageJ, OMERO, and Matlab. Additionally, the Chromagnon image alignment tool will read and write '.dv' files and it is the native format for DeltaVision microscopes utilising the commercial package SoftWoRx. With no configuration to specific hardware, Cockpit will automatically start in simulation mode, providing simulated cameras, light sources, and stages. This allows testing and demonstration of the software without any hardware or configuration. This can be further extended by utilising the test devices within the associated Python-Microscope package. In this mode, with some configuration, the software can link simulated XYZ stage, camera, and filter-wheel to return a channel defined by the filter-wheel and a subregion defined by the stage position from a large multi-channel image. The simulation is further improved by blurring the image based upon the Z position of the stage. This mode was used for Figure 1 and Figure 3 in this manuscript.

Use cases
Cockpit is optimised to provide an effective user experience running a fully automated microscope on systems with no eyepieces. A significant component of this is powerful mapping and overview tools that provide a guide-map to the specimen landscape, which is interactive and highly responsive. The system can collect arbitrary mosaic images, which are acquired in an expanding spiral pattern. Acquired mosaic images are immediately uploaded to the graphics card as a texture map. This allows smooth and rapid display of these images from low zoom overview all the way to single pixel views. The graphics card can then render these images interactively in real time.
The typical texture memory on modern graphics cards mean that the system can navigate in real time around mosaics made from hundreds or thousands of individual images, with resulting mosaics up to gigapixels in size. Starting a new mosaic does not remove previous images, so multiple, possibly overlapping, mosaic areas can be collected. Keeping previous mosaic images means it is easy to navigate multiple regions of the sample without having to reacquire overview images. This greatly eases the navigation of complex samples.
There are two obvious use cases for this functionality. In the first case, one could utilise a low magnification lens to map a large section, or even the whole of the sample area, say a coverslip as shown in Figure 4(a). This produces a ∼250 megapixel image, via which the user can visit any site of choice and re-image points in fluorescence mode with higher magnification objectives (Figure 4(b)). The new detailed images are then "painted" onto the guide-map in real-time. As the second case, in samples such as tissue culture cells, where many similar regions are available, smaller regions can be imaged at higher resolution to select interesting targets for further imaging. The preservation of previous mosaics when a new one is initiated is a critical component of this process, as it allows easy navigation of multiple sample regions. It is then practical to return to the regions of most interest without requiring reimaging, which would waste time and incur unnecessary light exposure to the sample.
To improve the user-centred microscope experience, we divided the use of the microscope into two stages. The user starts an imaging session by rapidly exploring large regions or even the entire slide to create a guide map, while marking regions of interest for later inspection. In the first stage, the touchscreen can be used to build the guide map. The user then  establishes the conditions required for the desired images, such as choice of laser lines and their intensities, exposures, number of channels and cameras, Z sectioning range, and points to visit on the guide map. The use of a touchscreen enables rapid setting up of the conditions for the experiment, focusing on the most important features of the microscope hardware without providing full functionality.
In a second phase of acquisition, a more detailed software control window is used with a conventional keyboard to define any more advanced experimental settings, such as numeric values for time lapse interval which are difficult to set from a touch interface. Finally the experiment is run.
In this way, we have created a unique, near-instantaneous control environment that allows the user to start an imaging session by exploring the entire slide quickly and marking regions to visit and image in order to achieve their scientific goals. Our software provides a novel way for the user to plan their entire work flow rapidly and with efficient specimen usage.

Previously published systems and microscopes in development using Cockpit
To exemplify the flexibility of Cockpit, we present a number of systems that currently run using the software as the user interface. Along with previously published systems, we have a number of microscopes at earlier stages of development where Cockpit provides the user interface.
Cockpit has already been deployed in three systems reported in previous publications. The first is a cryo-super resolution microscope for correlative imaging, CryoSIM 30,31 ( Figure 5(a)).
The second demonstration was a laser-free spinning disk confocal microscope using adaptive optics (AO) for aberration correction 32 ( Figure 5(b)). Additionally, Cockpit was the basis of the control software in the implementation of a novel technique, IsoSense, for improved aberration correction in widefield microscopes and structured illumination 33 .

A simple widefield microscope
We present a simple, compact, fully automated, portable inverted microscope based around the Zaber MVR system. This provides a bare-bones inverted stand with motorised X, Y, and Z axes, a multi-position motorised filter turret, and LED-based fluorescence illumination. The system has a Ximea camera with a small physical footprint (approximately 30 mm cube) and a Red Pitaya single board computer running as a hardware timing device ( Figure 6).
This system is able to take fluorescence images in three colours, using the three illumination LEDs and a quad-bandpass dichroic filter set. The system has 4× and 63× air objectives. Although changing the objective involves physically removing the objective and its mount, this is easily achieved due to the spring-loaded kinematic mounting system. The Cockpit mosaic function allows large areas of a sample to be mapped quickly (Figure 4(a)), and then regions of interest can be selected and marked for return later or the mosaic stopped and z-stacks in one or more channels collected. If a live sample is used the system can be used for time lapse imaging.
This system provides a demonstration of the power and portability of Cockpit and it functions as a test bed for new software developments on practical hardware. The system's small size and portability enable easy transport of the system for demonstrations and collaborative projects in other laboratories.
A complex structured illumination microscope with adaptive optics In a further system, we are using Cockpit to maximise the performance of complex upright widefield structured illumination microscope, referred to as DeepSIM (Figure 7(a)). This incorporates not only translation stages, light sources, and cameras, but also a spatial light modulator (SLM) and a deformable mirror (DM) for AO. This set-up is required to perform super-resolution live imaging experiments deep in tissue specimens, such as on the Drosophila larval neuro-muscular junction preparation, a powerful model system for understanding synapse biology on a molecular level. In order to achieve this imaging the system has to synchronise the illumination lasers, the SLM, a polarisation rotator, the DM, the Z stage position, and the cameras. The lasers, SLM, DM, and cameras receive digital triggers from the Red Pitaya, while the Z-position and polarisation rotator state are controlled via analogue voltages, all synchronised at the µs level. Cockpit provides a simple user interface allowing selection of different experimental parameters, such as exposure time, laser power, Z step, and stack size (Figure 7(b)). The experiment module creates the relevant signals and timing information, transfers this to the Red Pitaya which then controls the hardware during the experiment.
Cockpit also provides a user-friendly interface to Microscope-AOTools 34 , facilitating use of a wavefront sensor for calibration of the DM then adaptively correcting sample induced aberrations via image-based metrics, using so-called sensorless AO. This interface also implements IsoSense 33 to improve aberration detection over a wide range of complex samples.
Although this system is extremely complex with a multitude of devices the user interface is clean and easy to use. The system has sensible defaults, thus minimising the expertise and interaction required to collect experimental data.

Example simple extension in Python
We include an example script which demonstrates how the functionality of Cockpit can easily be extended in Python. The script finds DAPI stained cell nuclei in images, utilising the mosaic functionality to scan large areas and the point marking features to record the centroids of the detected nuclei. The script utilises the OpenCV 35 framework to detect large roughly circular objects in images. Images taken for the mosaic are also trapped by the code which Gaussian blurs to reduce noise, binarise, and finally applies a Hough transform to find circular objects. The locations of these objects in the image are then transformed into stage coordinates and added to the marked point list. The Python code, a test image, and detailed instructions on how to setup a simulated microscope to run this without any microscope hardware is detailed at https://doi.org/10.5281/zenodo.5745648.
For this simple example the parameters are fixed and tuned to the data set used, and select a subset of nuclei within a size range and with defined circularity. The current code also doesn't reliably detect nuclei at the edges between images in the mosaic scan. However, it can easily detect a large number of cell positions to create a point list which could then be run through a multi-site experiment to collect data such as a 3D, multi-channel stack, or time lapse on a large number of cells semi-automatically.

Discussion
We have developed a Python-based GUI for controlling bespoke microscopes which can map the entire slide or dish, allowing a real time exploration of the entire specimen landscape. It also connects to separate hardware timing devices to enable high precision timing of even complex devices and experiments. A key property of our software is that it is focused around the needs of the user and their experimental design and workflow. Finally, using Python means that the microscope control can easily be modified to make use of the extensive machine learning algorithms available in easy to integrate libraries. This means that a microscope system that is controlled by Cockpit can readily be adapted to make decisions using machine learning algorithms during the acquisition process in order to modify the imaging conditions.

Existing control options
Cockpit has been introduced as an alternative to a number of existing microscope control options that are already available. Most published systems take one of three approaches: 1) utilise individual software packages provided by the device manufactures, 2) use LabVIEW to integrate LabVIEW-based drivers (VIs) provided by the manufacturers into a single custom GUI, or 3) use the open source microscope control package µManager 36 . We will discuss the relative merits of each of these options in turn.
Using individual software packages to control each device is simple and direct. However this approach has several severe drawbacks. Each hardware device needs its own software, which takes up screen space and computer resources. This also removes the ability of different hardware parts to automatically interact with each other, for instance changing a stage's position once a camera has finished collecting images. Producing Z-stacks, or multi-channel images on such a setup is awkward, slow and open to human error. These factors can lead to corruption of the results.
LabVIEW is a visual programming tool that allows the construction of "programs" by connecting modules with wires and building graphical interfaces. Many manufacturers provide software to allow their hardware to be used in LabVIEW, through so-called virtual instruments (VIs). Building clean and simple visual framework to control highly complex advanced optical systems in LabVIEW requires advanced expertise in the program's use. Because such skills are usually beyond the ability of most scientists in an academic setting, many one-off complex bespoke systems, built to demonstrate a new principle, are very hard to use routinely by experimental scientists. Despite their obvious importance, such systems can rarely be reproduced and adopted by others for routine use.
µManager is an open source generalised microscope control interface which works on top of ImageJ, a popular image analysis program. ImageJ is written in Java, and µManager is written in a combination of Java and C++. µManager's approach is most directly comparable to Cockpit. Many hardware device manufacturers provide µManager compatible libraries and instructions on how to connect and control their hardware. The package comes with mechanisms to run basic experiments, and the ability sequence commands. It should be noted that Pycro-Manager 37 has recently been developed and released to allow much closer integration between Python and µManager. Pycro-Manager seems likely to fill a similar position to Microscope-Cockpit, allowing direct Python based control of imaging experiments, integration with online analysis, and a range of other functionality. As this is a recent development, we have not explored this package in any detail.
Being based on ImageJ, µManager, by design, includes a large application with multiple windows. This also means that the majority of the interface must be written in Java; however the system also needs a C/C++ layer to interface the Java code to C/C++ based system libraries. Cockpit is written in pure Python, relying on the Python-Microscope package for hardware interfacing. Producing a strict separation of the user interface from the hardware control components. This has the additional benefit that connected hardware may be physically located on another computer, increasing scalability and allowing devices requiring incompatible hardware or software to still be seamlessly integrated.
The mosaic window in Cockpit is similar in concept to Micro-Magellan 23 , a µManager plugin. Being able to record a large area view, possibly from multiple areas, and have that available for instant navigation. The mosaic functionality dramatically speeds up experiment setup and finding the correct regions of interest, especially on bespoke systems with no eyepieces. Our mosaic interface utilises modern fast GPUs to enable instant access to even very large mosaic maps. Additionally, the touch screen interface allows easy and intuitive access to most of the functionality using a visual grammar that is very familiar to everyone.
The dedicated timing device interface allows fast and repeatable timing for both simple and complex experiments. It is useful in general but absolutely indispensable for experiments like the live cell adaptive optics SIM experiment (Figure 7(c)).
The standardised device interfaces from Python-Microscope mean that replacing one device with another of the same type is simply a matter of changing the address of the device in a configuration file. Table 1 shows a comparison between various features of the previously described microscope control software options. We have ranked features of the control software options with a traffic light colour scheme with green being best and red least good. The table includes the approaches mentioned above along with Matlab, a common alternative to LabVIEW, but a more conventional programming package with a range of extensions and with a large support base, and MetaMorph (https://www. moleculardevices.com/products/cellular-imaging-systems/acquisition-and-analysis-software/metamorph-microscopy) / SlideBook (https://www.intelligent-imaging.com/slidebook) included as representatives of commercial generalised microscope control packages.
Cockpit is still under active development, both within our labs and at several other laboratories across multiple countries. We are working on adding online image analysis to further enhance the mosaic functionality with machine learning, enabling the system to capture a large area of the sample quickly at low-resolution and then automatically identify features of interest for 3D-multi-channel acquisition.

Conclusions
We have developed Cockpit, a new paradigm for user-based software control of complex bespoke microscopes. The software is highly adaptable by a user or engineer with experience in Python. The key advantage of our approach is that a user is presented with a clean and simple interface that hides the complexity of the hardware, so they can focus on their experimental design and obtain data from the instrument with a high precision and reproducible workflow. We hope that the community will adopt this package and help us to continue to develop it.

CryoSIM imaging
The CryoSIM system has previously been published in detail 30,31 . It was used to image HeLa cells grown on carbon coated gold EM grids under standard tissue culture conditions. Cells were labelled with MitoTracker Green (Thermo Fisher) at 100 nM and LysoTracker Red DND-99 (Thermo Fisher) at 50 nM after 30 minutes of incubation, to give green mitochondria and red lysosomes. The grids with live cells on them were then blotted and plunge frozen in liquid nitrogen cooled liquid ethane (Leica EM GP2) before being transferred to liquid nitrogen storage. Once frozen, grids were preserved in liquid nitrogen at cryogenic temperatures on the imaging systems to prevent thawing and detrimental ice crystal formation. Structured Illumination Microscopy (SIM) images were collected with a 100x 0.9NA objective, 488 nm and 561 nm laser excitation, and emission collected at 525/50 and 605/70 on two Andor iXon EMCCD cameras. Images were reconstructed using SoftWoRx (GE Healthcare) and image quality was assessed with SIMcheck 38 .

Aurox AO imaging
The Aurox Clarity AO system is as previously published, the system utilised an Olympus IX70 microscope with a 60×

Metamorph/Slidebook
References illumination for SIM imaging. The system also includes a deformable mirror (Alpao DM-69) for aberration correction. The system has a 60x 1.1NA water dipping objective and samples were imaged in aqueous buffer. The AO components were calibrated and controlled using Microscope-AOTools 34 to reduce aberrations and produce good SIM imaging at depth in biological samples.
The samples were prepared by following the protocol 39 . 3rd instar Drosophila melanogaster larvae (Bruchpilot (Brp)-GFP strain) were dissected in HL3 buffer with 0.3mM Ca 2+ to prepare a so-called larval fillet, and the larval brains were removed. After this, larvae were stained for 15 minutes with 1:50 Horseradish Peroxidase (HRP) conjugated to Cy3 fluorophore to visualise the neurons, washed with HL3 buffer with 0.3mM Ca 2+ and imaged in HL3 buffer with 0mM Ca 2+ to prevent the larvae from moving.

Data availability
No data are associated with this article.

Software availability
Software

Open Peer Review
Is the rationale for developing the new software tool clearly explained? Partly

Is the description of the software tool technically sound? Yes
Are sufficient details of the code, methods and analysis (if applicable) provided to allow replication of the software development and its use by others? Partly

Is sufficient information provided to allow interpretation of the expected output datasets and any results generated using the tool? Yes
Are the conclusions about the tool and its performance adequately supported by the findings presented in the article? Yes also appears to require a motorised x,y,z stage which again is understandable for mosaic imaging but not a requirement for lots of other types of microscope. Also, could the authors clarify, does this statement, 'In addition, experiments require a device that can be programmed as source of hardware triggers, supplying digital signals and, optionally, analogue voltages.' mean Cockpit will also only work with a control board for hardware triggers? Or will it work to some degree without?
Otherwise Cockpit will be an extremely useful tool for the microscopy community going forward.

Is the rationale for developing the new software tool clearly explained? Yes
Is the description of the software tool technically sound? Yes

Is sufficient information provided to allow interpretation of the expected output datasets and any results generated using the tool? Yes
Are the conclusions about the tool and its performance adequately supported by the findings presented in the article? Yes Python-based Graphical User Interface (GUI). For hardware control it is using "Python-Microscope" (Microscope), and it includes coordinated control of hardware devices. It was verified using several bespoke microscopes, including a simple widefield system and a complex system with adaptive optics and structured illumination. It is written in the text that this system allows for using several cameras, but I did not see any actual demonstration of simultaneous operation with several cameras.
This Cockpit GUI allows for control of bespoke microscope systems, navigation of the entire slide landscape, and automated acquisition without the use of eyepieces. One of the functions that I miss a lot in this work is any sample auto-focusing pipeline. Sample's Z drift correction using IR laser back reflection or similar approaches became very popular and important for many experiments.
A key strength of Cockpit is its use of Python, which means that any microscope built with Cockpit is ready for future customisation by simply adding new libraries, for example machine learning algorithms to enable automated microscopy decision making while imaging. However, I would like to make a note that making a universal software platform for any bespoke microscopy system is a challenging task, and probably each new case will require readjustments of the Cockpit.
In this paper several software packages are compared to the Cockpit (such as LabVIEW, Matlab, µManager). It is important to mention that there are many more open-source alternatives that µManager that are able to do either hardware control or data analysis or both. Most of them are listed here.
Recently newly developed open-source Python-based systems, such as the Sync-Scope,1 and miEye2 appeared. They both allow for similar functionality as the Cockpit.
Other specific comments. It is written on page 3 "Many bespoke microscope systems that lack eyepieces, such as systems built by physical scientists in collaboration with biologists, make sample navigation particularly challenging.". To be precise, it all depends on what is the sample. If one is looking into diffraction limited spots that are dim, looking through eyepieces would not help to navigate around the sample.
involves physically removing the objective and its mount, this is easily achieved due to the spring-loaded kinematic mounting system.". Doesn't that mean that one has to remove the sample when changing objectives? In such a case, returning back to the original positions would be challenging.
On page 10: "If a live sample is used the system can be used for time lapse imaging.". Probably any dynamic sample can be used for time lapse imaging regardless it is living cells or just molecules in vitro.

7.
On page 11: "We have developed a Python-based GUI for controlling bespoke microscopes which can map the entire slide or dish, allowing a real time exploration of the entire specimen landscape.". I suggest using "interactive exploration".

8.
On page 12: "This has the additional benefit that connected hardware may be physically located on another computer, increasing scalability and allowing devices requiring incompatible hardware or software to still be seamlessly integrated.". In such case latency might become an issue. Please, provide explanation on this matter.

9.
On page 12: "We have ranked features of the control software options with a traffic light colour scheme with green being best and red least good. The table includes the approaches mentioned above along with Matlab, a common alternative to LabVIEW, but a more conventional programming package with a range of extensions and with a large support base, and MetaMorph (https://www. moleculardevices.com/products/cellular-imagingsystems/acquisition-and-analysis-software/metamorph-microscopy) / SlideBook (https://www.intelligent-imaging.com/slidebook) included as representatives of commercial generalised microscope control packages.". I think, Igor Pro is also used relatively widely in the community to control microscope's hardware and perform data analysis. 10.

Are sufficient details of the code, methods and analysis (if applicable) provided to allow replication of the software development and its use by others? Yes
Is sufficient information provided to allow interpretation of the expected output datasets and any results generated using the tool?

Joe Knapper
Physics and Astronomy, University of Glasgow, Glasgow, Scotland, UK I'd like to thank the authors for sharing this work, based around making the control of microscopy systems more intuitive for the user based on a GUI for Python-Microscope. I agree with the suggestions of the first round of reviews, and agree with the authors that these changes improved the article. While I recommend the article and tool are ready for publication, I would like to raise the following questions.
The inclusion of Table 1 gives more context for Microscope-Cockpit, but I hope the authors can share how it was assessed? As a subjective ranking, is it based on the feedback of multiple users and developers, public forums, etc?
Can the authors expand on the ease of use of exporting the '.dv' format? I note they list it is supported in ImageJ, OMERO, Matlab and Chromagnon, but as this work is Python-based, can the authors suggest a (preferably native) Python-compatible solution? omero-py, pyimagej, or ideally something only requiring a pypi installation? The first sentence of "Output image files" is also hard to follow -does a single '.dv' file contain multiple images, wavelengths, z-slices and time points?
The GPU-based mosaic for multiple large area scans appears to be an important feature of this work, and very useful for potential users. Can the authors clarify if this is positioning and blending images together based on areas of overlap, or simply placing images directly based on the stage position?
Finally, a note to the authors that currently it appears that Python >=3.10 is incompatible, based on wxPython depending on attrdict. Whether this is addressed in the paper or the program is up to the authors.
Is the rationale for developing the new software tool clearly explained? Yes

Is the description of the software tool technically sound? Yes
Are sufficient details of the code, methods and analysis (if applicable) provided to allow replication of the software development and its use by others? Yes

Is sufficient information provided to allow interpretation of the expected output datasets and any results generated using the tool? Yes
Are the conclusions about the tool and its performance adequately supported by the findings presented in the article? Yes

Emil Rozbicki
Glencoe Software, Inc, Seattle, WA, USA William Moore OME, University of Dundee, Dundee, UK

Sébastien Besson
Centre for Gene Regulation and Expression, School of Life Sciences, University of Dundee, Dundee, UK This paper introduces Microscope-Cockpit, a graphical user interface software that has been built on top of the historical software used to control the OMX microscope. It aims to provide a clean and user friendly interface allowing to create complex experimental set-up. Several use cases are included in the article to support the capacity of the software to adapt to different experimental set-ups. Overall the tool described in this article is scientifically valid but we would like to suggest some revisions detailed below. The Implementation section details the Python requirements of Microscope-Cockpit. Amongst those packages, the Python-microscope library appears as one of the most critical dependencies since it contains all the logic and support for handling and controlling microscopy devices. As part of this review, it came to our attention that this library has been recently published ( https://doi.org/10.1242/jcs.258955) 1  We suggest a paragraph should be added to the Implementation section that discusses the Data Generation/Output including a description of the file format as well as the tooling available for end-users of the acquisition system to read and analyze imaging data generated by Cockpit.
In the section discussing Cockpit in relationship Micro-Manager, a proposed key strength of Cockpit is its Python implementation allowing the "strict separation of the user interface from the hardware control components" while "the majority of [Micro-Manager] interface must be written in Java; however the system also needs a C/C++ layer to interface the Java code to C/C++ based system libraries". This argument omits Pycro-Manager, a Python bridge for Micro-Manager, published earlier this year (https://doi.org/10.1038/s41592-021-01087-6) 2 and mentioned in the aforementioned Python-microscope publication. The section mentioned above should be amended to cite this paper and include Pycro-Manager in the comparison with Cockpit. Much is made of the ability to use Python for custom modifications and the integration of thirdparty libraries. The text mentions that "the microscope control can easily be modified to make use of the extensive machine learning algorithms available in easy to integrate libraries" but this sentence rather suggests the extensibility is a property of the underlying python-microscope library rather than the Cockpit interface. The Cockpit reference documentation ( https://www.micron.ox.ac.uk/software/cockpit/) primarily describes the installation and the configuration of the software. To support the extensibility claim, we would suggest either to introduce a public example of Python extension to the default Cockpit UI and/or an amendment to the online documentation that would demonstrate this extension which could be linked from the article. Together with python-microscope paper, this paper introduces a new open-source solution for microscope control and imaging acquisition. Both in the introduction and in the discussion, Cockpit is compared to several equivalent commercial and open-source solutions, including the well-established Micro-Manager. This comparison is important and particularly useful for end users and imaging facility managers who need to make informed decisions e.g. when investing in new equipment or technologies. We suggest to make the outcome of this discussion more explicit and prominent e.g. under a form of a summary table that would include the different available ecosystems (individual hardware solutions, LabView, python-microscope/Microscope-Cockpit and Micro-Manager/Pycro-Manager) and compare their respective advantages and drawbacks.

Ian Dobbie
Response to review from authors.
Authors responses are in italics New text in the manuscript are in bold The Implementation section details the Python requirements of Microscope-Cockpit. Amongst those packages, the Python-microscope library appears as one of the most critical dependencies since it contains all the logic and support for handling and controlling microscopy devices. As part of this review, it came to our attention that this library has been recently published (https://doi.org/10.1242/jcs.258955) 1  We thank the reviewers for pointing this out. The Python-Microscope publication has been accepted at JCS since we submitted the article. We have revised the manuscript to reference the published article rather than the biorXiv preprint.
The article makes no mention of the file format written by Microscope-Cockpit. The Pythonmicroscope paper does not include any detail about the choice of data output for end-users either. From our minimal testing of the application, the only option available to the user is to write data as "DV files (.dv)", a proprietary file format. Is this the only choice available to the user? If so, this feels like a limitation of the current implementation which contradicts the claim for the open and universal mission of Cockpit. We suggest a paragraph should be added to the Implementation section that discusses the Data Generation/Output including a description of the file format as well as the tooling available for end-users of the acquisition system to read and analyze imaging data generated by Cockpit.

Output image files
Experimental images are saved into files utilising the '.dv' file format, typically multiple wavelengths, Z-slices and time points into a single file. Images from the mosaic window and single snaps can also be saved into this format, with the mosaic saved files having an associated text file defining XYZ positions of each collected image. The '.dv' format is an extension of the mrc file format, defined in detail in the MRC/CCP4 2014 file format specification. The CCP4 consortium of the EM community continue to support and extend this file format. This support includes file validators and a detailed specification, which is compatible with the files used here but not identical.
The optical microscopy specific metadata are covered in the documentation. Although relatively uncommon, the file format is supported by the Bio-Formats project allowing import of '.dv' files, along with the associated metadata into software using this library including ImageJ, OMERO and Matlab. Additionally, the Chromagnon image alignment tool will read and write '.dv' files and it is the native format for DeltaVision microscopes utilising the commercial package SoftWoRx.
In the section discussing Cockpit in relationship Micro-Manager, a proposed key strength of Cockpit is its Python implementation allowing the "strict separation of the user interface from the hardware control components" while "the majority of [Micro-Manager] interface must be written in Java; however the system also needs a C/C++ layer to interface the Java code to C/C++ based system libraries". This argument omits Pycro-Manager, a Python bridge for Micro-Manager, published earlier this year (https://doi.org/10.1038/s41592-021-01087-6) 2 and mentioned in the aforementioned Python-microscope publication. The section mentioned above should be amended to cite this paper and include Pycro-Manager in the comparison with Cockpit.
We thank the reviewers for pointing out this paper which was again published since our original submission and we have now added discussion about Pycro-Manager to the paper. We have revised the manuscript by adding the following text to the discussion section: It should be noted that Pycro-Manager [pinkard2021] has recently been developed and released to allow much closer integration between Python and uManager. Pycro-Manager seems likely to fill a similar position to Microscope-Cockpit, allowing direct Python based control of imaging experiments, integration with online analysis and a range of other functionality. As this is a recent development, we have not explored this package in any detail.
Much is made of the ability to use Python for custom modifications and the integration of third-party libraries. The text mentions that "the microscope control can easily be modified to make use of the extensive machine learning algorithms available in easy to integrate libraries" but this sentence rather suggests the extensibility is a property of the underlying python-microscope library rather than the Cockpit interface. The Cockpit reference documentation (https://www.micron.ox.ac.uk/software/cockpit/) primarily describes the installation and the configuration of the software. To support the extensibility claim, we would suggest either to introduce a public example of Python extension to the default Cockpit UI and/or an amendment to the online documentation that would demonstrate this extension which could be linked from the article.
The reviewers make a good point that would significantly strengthen the paper. We have written a short script that utilises some of the embedded functionality to find and mark cell nuclei in a typical DAPI stained sample. The code, along with a test image set, config files and detailed instructions on how to set it up are collected into a zenodo repository and available via a doi (10.5281/zenodo.5745648) link. In fact, this simulated microscope approach is a very powerful approach to setting up and testing such scripts. We have also added the following text to the manuscript.

Example simple extension in Python
We include an example script which demonstrates how the functionality of cockpit can easily be extended in python. The script finds DAPI stained cell nuclei in images, utilising the mosaic functionality to scan large areas and the point marking features to record the centroids of the detected nuclei. The script utilises the OpenCV framework to detect large roughly circular objects in images. Images taken for the mosaic are also trapped by the code which Gaussian blurs to reduce noise, binarise and finally applies a Hough transform to find circular objects. The locations of these objects in the image are then transformed into stage coordinates and added to the marked point list. The Python code, a test image and detailed instructions on how to setup a simulated microscope to run this without any microscope hardware is detailed at https://doi.org/10.5281/zenodo.5745648.
For this simple example the parameters are fixed and tuned to the data set used, and select a subset of nuclei within a size range and with defined circularity. The current code also doesn't reliably detect nuclei at the edges between images in the mosaic scan. However, it can easily detect a large number of cell positions to create a point list which could then be run through a multi-site experiment to collect data such as a 3D, multi-channel stack, or time lapse on a large number of cells semi-automatically.
Together with python-microscope paper, this paper introduces a new open-source solution for microscope control and imaging acquisition. Both in the introduction and in the discussion, Cockpit is compared to several equivalent commercial and open-source solutions, including the well-established Micro-Manager. This comparison is important and particularly useful for end users and imaging facility managers who need to make informed decisions e.g. when investing in new equipment or technologies. We suggest to make the outcome of this discussion more explicit and prominent e.g. under a form of a summary table that would include the different available ecosystems (individual hardware solutions, LabView, python-microscope/Microscope-Cockpit and Micro-Manager/Pycro-Manager) and compare their respective advantages and drawbacks.
We thank the reviewers for this suggestion and we have added a table of feature comparison between a range of microscope control options as a traffic light colour coded table and the following text to the manuscript. Table 1 shows a comparison between various features of the previously described microscope control software options. We have ranked features of the control software options with a traffic light colour scheme with green being best and red least good. The table includes the approaches mentioned above along with Matlab, a common alternative to LabVIEW, a more conventional programming package with a range of extensions and with a large support base, and MetaMorph/ SlideBook included as representatives of commercial generalised microscope control packages.
[New table included in revised version] quite competitive with micromanager yet (which is fine, that will come with time). Integration with a tool like napari would be a great choice for broadening the usability of Cockpit. It is nice to see that Cockpit is being tested on multiple systems, although as mentioned it would be good to see a fair comparison between Cockpit's current (and future planned) drivers relative to a mature project like micromanager.