1. Introduction
High-speed imaging has shown an exceptional potential in capturing ultrafast transient phenomena in a variety of applications such as screening the physiological processes in biological tissues [
1,
2,
3,
4] high-throughput blood cells screening [
5,
6], fluorescence confocal and lifetime microscopy [
7,
8] which require cameras with capture rates between Kfps to Mfps. In the fields of biomedical research and clinical applications for example, high-speed imaging allows the detection and tracking of cells, plasma, and other molecules of interest in a specimen individually or as a group with high sensitivity and precision. Bulky design, high cost and complex operation of the conventional high-speed imaging systems make them almost inaccessible in technical environments especially in the resource-limited areas. As a result, the development of portable, low cost and low maintenance high-speed imaging systems is particularly important for such environments.
At the early development stages of high-speed cameras, the rotating mirror camera [
9] was one of the first commercially available instruments to achieve 25 Mfps. The principle of this scheme relied on the rotation of a single mirror that directed the incident light (frames) towards a film strip and was replaced by the array of Charged-Coupled Devices (CCD) in the later development stages (e.g., Brandaris) [
10]. Alongside the aforementioned rotation mirror high-speed cameras, there are several other commercially available devices (such as IX and Phantom cameras) that can operate at several hundred Kfps frame rate which can be suitable for a range of applications such as combustion studies, that is the investigation of the chemical reaction between the substances, schlieren imaging (a non-invasive method to visualize density gradients within otherwise invisible flows), microfluidics, particle image velocimetry, material tensile testing and many more [
11,
12,
13,
14]. The operation of these instruments rely on the high-speed electronics such as the custom-made charge-coupled devices (CCD) and complementary metal oxide semiconductor (CMOS) detectors that can achieve such high frame rates that are extremely costly to build and demand for high storage capacities and high maintenance costs. The necessity of observing phenomena at high frame rates have driven the imaging society to push the boundaries of the imaging technologies to achieve superior technologies compared to their previous generations and achieve Gfps and up to Pfps frame rates where they have proven their significance in various applications such as observing the dynamics of laser-induced breakdowns [
15], observing the 3D shapes of the hidden objects using ultrafast time-of-flight imaging techniques [
16], non-scanning tomographic multispectral imaging methods that enables single-shot cross-sectional image acquisition at femtosecond time scale [
17], ultrafast spectroscopic videography of dynamic non-repetitive events [
18] and three-dimensional high-speed imaging [
19]. Several novel ultrafast time-stretched imaging systems have also been developed that are capable of achieving Mfps—Tfps frame rates [
20,
21,
22,
23]. However, these imaging technologies are also associated with series of drawbacks such as the requirement for expensive short-pulse laser (operation only in active mode), the imposed limitations on the amount of pulse dispersion to avoid overlapping of the adjacent pulses that is also known as data-mixing, faster frame rates at the expense of imaging contrast, spatial resolution and detection sensitivity as well as their requirements for high data transmission, processing and storage capacities to enable capturing of the ultrafast phenomenon at high capture rates and longer durations.
In the recent years, Compressive Sensing (CS) has been introduced to the high-speed imaging systems that operates by identifying the sparse representation of a signal in a known basis that immensely reduce the number of the required samples well-below the Nyquist rate. After the acquisition step, the data can be recovered via the series of recovery methods (optimization algorithms, deep learning based methods etc.) [
24]. One method of CS based high-speed image acquisition is by introducing optical encoders in the optical layer of the imaging system to allow overlapping of the adjacent frames. These optical encoders are implemented in various forms such as translating coded aperture to compress the data in spatial and temporal domains that allows for the full overlap of the adjacent frames [
25] as well as the use of digital mirror device (DMD) in their optical systems and applying controlled shifts to the adjacent frames that enable partial overlaps in the recorded data [
26,
27,
28] hence achieving frame rates from 1 Kfps up to 70 Tfps. Furthermore, by using CS based acquisition techniques in the time-stretched ultrafast imaging systems considerable improvements have been made on the temporal resolution and data storage and transmission capacities [
29,
30]. The range of achieved advancements from these CS enabled imaging systems is a strong indication of the versatility of the CS framework in high-speed imaging applications and a robust proof that CS based systems can improve the overall performance of the system by a substantial amount. Current CS based imaging techniques however are associated with their unique disadvantages such as the monochrome scaled captures, sacrificed spatial resolution and requirements for the controlled amounts of shift between the adjacent overlapping that constraints the sequence depth (short duration of recording) in these imaging modalities. Furthermore, these imaging systems have extremely high built and maintenance costs as well as oversized dimensions and highly complex operations. As a result, overcoming these shortcomings while achieving high acquisition rates in the system becomes an essential requirement for majority of the affected applications.
To address these matters, we propose a novel coded compressive rotating mirror (CCRM) camera that enables high-speed imaging of a transient phenomenon at a substantially lower build cost and operation complexity using the off-the-shelf components in a compact instrument design. Our approach implements amplitude encoding (grey scale) in a dramatically simplified rotating mirror camera setup where the continuous exposure of the dynamic scene on the rotating mirror and the constant frame sweep across the detector surface enables the CCRM camera to reach the lowest possible distance between the sweeping overlapped frames that is a single pixel on the detector. This principle avoids the controlled positioning of the frames on the detector and allows the encoded frames to be uniformly overlapped within a single pixel of the adjacent ones and therefore reach the highest compression ratio and the highest number of recorded frames (sequence depth) in this type of imaging schemes [
25,
26,
27,
28].
CCRM camera differs from other CS based 2D imaging systems such as the proposed scheme at [
25] that have demonstrated a reconstruction of 147 fully overlapped frames from a single acquisition where the binary encoding and the fully overlapped frames are replaced with amplitude encoding and pixel-shifted overlapping frames that enables the camera to capture much higher number of frames in both monochrome and colored formats and achieve higher reconstruction quality. CCRM also differs from the proposed scheme at [
27] with a demonstrated 32 reconstructed frames during the experiments and a proposed potential of 500 frames from a single acquisition where the proposed CCRM camera eliminates the pulsed illumination from the target (scene) and the controlled amounts of shift between the adjacent frames hence allowing the scene to be continuously captured from a natural self-luminant target or using a continuous external light source without any frame drops during the full exposure period of the detector and achieving a sequence depth of 1400 frames from a single capture. Furthermore, the recovery of the data using an amplitude encoder instead of a the binary format that is used in the previous studies, eliminates the binning process on the data hence significantly increasing the spatial resolution and the total number of overlapped and recovered frames compared to the aforementioned CS based imaging modalities. Additionally, for the first time, the implementation of the calibration tracer blocks (discussed in the Principles section of this paper) provides a precise spatial location of the individual frames on the detector that enables the data reconstruction with higher fidelity. The heavily compressed data from the CCRM camera that are encoded by the optical encoder can only be recovered using that pattern which is considered as the “key” to the reconstruction algorithm. This property of the proposed scheme enables a highly secure and efficient data storage that utilizes the use of the CCRM imaging technology in the fields such as medical imaging and military based applications that require highly secured data handling and processing capabilities.
2. Materials and Methods
Depicted in
Figure 1 is the configuration and operation principle of the proposed coded compressive rotating mirror (CCRM) imaging system. In the capturing stage, the image of the dynamic scene at time t is focused on a spatial encoder (with M rows and N columns), and then focused on the rotation mirror driven by a high speed motor that further reflects the image onto a 2 dimensional (2D) photo detector (such as CMOS or CCD). The detector module should have at least M rows and (N + F − 1) columns in size with moderate readout speed. The rotation of this mirror continuously sweeps the individual frames across the surface of the detector module and overlaps them during a single exposure that creates a single pixel shift in-between the adjacent frames. Therefore, M × (N + F − 1) pixels on the detector will record F of the coded frames at different times that are positioned within a single pixel distance from each other throughout the continuous illumination from the dynamic scene. Capturing a scene in a single exposure of the detector, eliminates the limitation of digitization and readout time of the camera from the proposed scheme.
The mathematical representation of data acquisition process of the proposed CCRM imaging system can be formulated as
where
is the captured data by the detector in a vectorized format,
is the linear operator of frame shifting and overlapping that is built upon F identity matrices,
is the obtained motion profile of the sweep from the calibration points on the encoder in the form of a diagonal matrix,
is the matrix that holds F encoding pattern of
in a diagonal form,
represents the original frames in a vectorized format, and
n is the additive zero mean Gaussian noise.
Estimating x from y in Equation (
1) is known as an ill-posed linear inverse problem (LIP) [
24], i.e., there is more than one feasible solution to this problem. The formulated sensing matrix referred to as TCA in Equation (
1) enables an extremely high compression to be achieved on the observed temporally shifted and overlapped data however it is to be noted that this type of compression does not satisfy the Restricted Isometry Property (RIP) [
24] used in the general compressive sensing framework therefore, the data reconstruction is associated with the inevitable artefacts that is known as a lossy recovery. Many reconstruction methods such as dictionary learning based algorithms, Bayesian, Gaussian mixture models and maximum likelihoods have demonstrated their capabilities in solving such equations. Of these, we implement the Alternating Direction Method of Multipliers (ADMM) [
31] method that applies variable splitting to the cost function and solves the shaped Lagrange equations accordingly. This approach transforms the obtained equation into a minimization problem and solves the equation by minimizing the energy function via the repetitive calculations. Here, we select the total variation (TV) [
32,
33] as the regularizer function where it has demonstrated high performance on various CS based algorithms [
34,
35]. One of the advantages of using TV over the other regularizers is the edge preservation property that prevents hard smoothing of the edge features. This key characteristic averts the spatial information from merging with the background features and therefore preventing the loss of the critical information such as the boundaries and intensity amplitudes per pixels that are essential is applications such as high throughput cell screening where the cell counting the exact shape of the individual cells are the defining factors in the analysis.
Image reconstruction of the original scene can be achieved by solving the minimization problem that is
where
are the variable regularization and denoiser threshold parameters that are adjusted based on the calculated error at each iteration and
is the regularizer weight for each horizontal, vertical and temporal domains and
D is a regularization function that promotes sparsity in the dynamic scene. The linear operation of frame shifting and overlapping (represented by operator T) results in the compression of the data in CCRM camera. This data compression ratio can be expressed as:
and therefore, higher numbers of captured frames
F (longer durations of captures) result in higher compression ratio.
The obtained compression from the CCRM camera has the highest ratio compared to the other CS based imaging systems [
25,
26,
27,
28] that is attribute to the single pixel shift modality in the operation principle of the proposed imaging system that allows the overlap of higher number of consecutive frames in a single capture with high data reconstruction fidelity. Rotation of the motor is often associated with various types of flaws such as the backlash error, vibration at high speeds, design imperfections in the mirror holder attached to the motor etc. Hence, due to these inevitable mechanical errors, position of individual frames on the vertical axis of the detector may differ to the adjacent frames. On the other hand, Having a precise indication on the position of frames on the captured data is a necessity for the data reconstruction. Tracing these vertical movements are achieved by implementing the calibration blocks that are positioned on the encoder pattern in a designated location that leaves a distinct movement trace on the detector and is unique for each capture. Depicted in
Figure 2a,b, are the two designated calibration blocks at the top left corner of the encoding pattern with the dimensions of 2 × 2 and 4 × 4 pixels where they leave a trace of the movement on the detected image. These tracer lines are then extracted from the scanned data and used to define the exact position of each frame in the compressed scan. The 2 × 2 block is used as a primary calibration block which is also used in the forward model and the reconstruction algorithm, however for the scenes with the lower amounts of illumination intensities on the CMOS, detecting the full calibration line with all the pixels present in the line can be a challenging task therefore, a larger 4 × 4 block with higher light throughput assists the primary block in defining the position of the frames.
Figure 2c illustrates the scan of a static scene and marking of the boundaries in the selected area where the vertical shift in each frame is compared against the first (reference) frame. The primary calibration line color is marked as red in this picture for better visibility. The extracted motion profile is noted as C in Equation (
1) that provides the vertical shift parameters to the reconstruction algorithm. Furthermore, as the rotation of the mirror generates the sequence of the overlapped frames on a cylindrically curved surface, the scene will encounter the so called “Petzval Field Curvature” aberration in which the paraxial sections in the image plane will appear slightly out of focus. The amount of blur on the frames will depend on the degree of rotation on the flat surface of the detector that is 2 degrees in the CCRM camera and therefore the error produced by the filed curvature aberration is 0.037 mm which is negligible. This aberration however can be eliminated by selecting the central section of the field of view that effectively crops the blurred sections on the image plane.
In the optical setup of the CCRM camera, the dynamic scene is first collected by a 10× infinity-corrected microscopic objective (Thorlabs, RMS10×) and directed towards the lens tube. This lens is dependant on the requirements of the dynamic scene where different types of lenses with suitable properties (magnification, NA etc.,) can be adapted to observe the high-speed phenomenon. The 2D image is then focused onto a static encoder built in-house, with diverse pattern varieties and different spatial resolutions. The encoder patterns are printed on a soda-lime glass and have a 1:1 ratio between the blocked and transparent pixels. An example of this pattern is depicted in
Figure 2b. These patterns are randomly generated where data reconstruction using different patterns yields similar imaging performance. Although the printed encoder has a high extinction ratio, however as the light passes through the transparent pixels of the optical encoder, the intensity values of the adjacent pixels interfere with the neighbouring pixels due to the light diffraction and changes the encoder pattern from binary into grey scale pattern as depicted in
Figure 2a.
Previous studies [
26,
27] have attempted to reverse the encoder pattern back into the binary format by implementing the commonly used binning process (e.g., 2 × 2 or 3 × 3) on the detector however this comes at a cost of the reduced spatial resolution of the detected image. To reduce the amount of interference between the adjacent pixels, encoding pixels with larger dimensions can be printed on the mask, however, this results in a lower encoding resolution on the scene, therefore having a negative effect on the recovery of the data. Our proposed new approach is to exploit this feature of the optical encoder in the CCRM camera where by acquiring a reference image of the encoder pattern on the detector prior to recording of the dynamic scene, the requirements for the binning process is eliminated hence achieving higher spatial resolution compared to the aforementioned methods. Structural differences between the observed and designed patterns have been depicted in
Figure 2a,b where the acquired pattern from the detector has a grey scale rather than the binary format which is then used in the reconstruction algorithm.
The achieved frame rates (
) from the CCRM camera setup is calculated by the following expression
where
R is the rotation speed of the mirror (rounds per second),
L is the orthogonal distance between the mirror and the detector surface and P is the width of each pixel in the detector (the distance between adjacent frames). In the conventional rotating mirror camera systems [
9,
10], the frame rate could be calculated using similar formulas to Equation (
4) which is inversely proportional to the distance between adjacent frames that is the width of the detector with the dimensions of at least N × P. In this regard, the proposed CCRM camera operates at N times faster frame rate comparing to the conventional rotation mirror camera systems with the same mirror rotation speed R and arm length L. Assuming N = 500 and the gap in between the adjacent detectors to accommodate the series of detector modules in the alignment of the conventional rotation mirror camera system, the CCRM operates in at least 700 times faster frame rate than the conventional rotation mirror camera systems with the same R and L parameters. In the conventional imaging systems, the frame rate of a camera is determined by the exposure time of the detector [
9,
10,
11,
12,
13,
14] and therefore limited by the speed of the detector, while in the CCRM camera, the entire dynamic scene is captured during single exposure of the detector that makes the frame rate of the CCRM camera independent of the detector’s exposure time and enables the use of low speed detectors to achieve high frame rate. In the experiments, CCRM camera uses an ordinary off-the-shelf CMOS module with dimensions of 1216 × 1936 pixels (5.86
m per pixel) and a maximum frame rate of 47 fps to capture dynamic scenes at up to 120 Kfps frame rate. The electrical motor used for the rotation of the mirror is a low-cost unit with the maximum rotation speed of 0.5 rotations-per-second (rps). This setup with the aforementioned components can record 1400 frames with spatial dimensions of 350 × 350 and 200 × 500 pixels with the data compression ratios of 280 and 368, respectively. For types of applications with requirements for larger spatial dimensions, the aforementioned CMOS module can capture data sequence with dimensions of
or can be replaced with another CMOS module with higher active area to accommodate for larger capture area or longer capture durations. These frames could be reconstructed with high fidelity using the formulated minimization function stated at Equation (
2).
3. Results
High-speed imaging is an imperative tool in medical and bio-chemistry fields with extensive application in high throughput cell screening [
20,
26,
30,
36] microparticle flow cytometry [
12,
13,
23,
37] and many others. In microfluidics and lab-on-chip studies, the diverse range of imaging applications with distinct properties of the samples require the imaging setup to be adapted to that particular experiment with minimum effort. The adaptability of the CCRM camera setup becomes evident when the frame rate of the system can be easily controlled by adjusting the rotation speed of the motor without altering the optical system or using the CCRM camera for both types of active (light source required) or passive (self-luminant) applications without any modifications in the system.
This is considered as one of the fundamental advantages of CCRM camera where the versatility and adaptability in imaging various types of scenes for variety of applications with different requirements in size, imaging speed, illumination properties etc. enables the proposed imaging technology to capture the dynamic scenes without any compromises in the operation principle of the system.In this regard, we configure the CCRM camera to observe flowing droplet samples in the microfluidic chip from injecting two types of immiscible liquids, transparent oil and chemical dye through a motorised dispenser into the chip channels. In this experiment, a continuous white light source is used to illuminate the microfluidic chip where droplets have the flow rate of 0.9 m/s in the channels. The dynamic scene is collected from the object (using a 10×, 0.25-NA, 10.6 mm WD objective lens) and focused onto the encoder. The secondary lens system then collects the encoded image and directs it towards the rotating mirror. Three separate captures are recorded at the rates of 3, 50 and 120 Kfps and reconstructed through the proposed algorithm.
Depicted in
Figure 3a is a section of the encrypted and compressed captured data with dimensions of 200 × 1900 pixels and the compression ratio of 368 (obtained from Equation (
3)) for the droplets flow experiment at the frame rate of 120 Kfps.
Figure 3b,d illustrates some of the reconstructed frames obtained from solving Equation (
2) by using the encoding key pattern (see
Supplementary Movie 1 for the full reconstructed data captured at 120 Kfps). Achieved results demonstrate a high reconstruction quality with clear and distinguishable droplets flowing in the microfluidic chip with well-preserved edge features in the frames. Noticeable examples of this are the conserved bent edges for the first droplet at frames 30, 40 and 50 of the 120 Kfps capture dataset. Imaging of the droplets flow in the microfluidic chip at various frame rates demonstrates the capabilities of the CCRM imaging system in microfluidics and lab-on-chip studies where it can be easily adapted to record other bio-chemical phenomenon with their distinct properties and capture rate requirements.
In addition to the medical and bio-chemical applications, high-speed imaging has also been utilized in observing ultra-fast events in other fields such as laser ablation and plasma dynamics [
22], fluorescence imaging [
26], explosions and chemical reactions [
38], surface inspection [
39] and many others. Particular example of this type of events that are continuously investigated in applied thermal engineering fields are so called “micro-explosions”. This phenomena can be observed in various scenarios such as electrically triggered atomic emission spectroscopy (AES) [
40], atomization process resulting in droplet disintegrations [
41] and laser-induced sample ignitions [
42]. Recording and observing these micro-explosions can help the investigators in gaining deeper understanding of the observed phenomenon. In general, these non-repetitive events that trigger a deformation process of a substance (gas, liquid etc.) in micro-resolution that are associated with the so called “burst” process can be regarded as micro-explosions. An example of this type of event is the ignition of the gas lighter which uses an alloy of iron and cerium that emanates sparks during the rapid oxidization process where the dynamics of the scene share similarities with the aforementioned micro-explosion based applications. Here, in our next demonstration, we configure our experiment on the micro-explosion event in which the observed dynamic scene is the generated sparks from a handheld lighter module.
Depicted in
Figure 4a,b are the captured and the reconstructed data from movement of the generated sparks, respectively (see
Supplementary Movie 2 for the reconstructed data captured at 120 Kfps). As the dynamic of the scene has a 3 dimensional property, the random movements of the sparks occur in all spatial dimensions however the captured and reconstructed data are considered to be in a 2D format. As the light intensity varies in different sections of the device, higher amounts are more noticeable at the center of the spark groups.
The proposed reconstruction algorithm has efficiently conserved the features of the event by preserving the sharp boundaries of the generated spark particles that are well distinguishable in the scene. Travel direction of the particles are random in the 3-dimensional spatial domain therefore majority of the particles are not positioned on the focus plane of the lens hence different intensity levels are observed per particle. To reconstruct this scene, Equation (
2) has been extended into color domain where by separating the RGB channels, applying the algorithm on each individual channel and blending them back into single images, colored reconstruction of the frames can be achieved. The process of reconstructing the individual channels are decoupled form each other therefore they can be determined in a parallel fashion, hence the accelerated reconstruction can be achieved.
Distinct properties in the conducted experiments demonstrate the flexibility of the CCRM camera in capturing the high-speed dynamics of the scenes under diverse conditions with unique characteristics, such as
Illumination conditions, where in the first experiment (microfluidics), an external light source was used to provide the sufficient light for the scene to be observed whereas the second scene is a self-luminant transient phenomena that does not require the external source.
Dynamics of the scenes, in which the first demonstration can be regarded as a controlled experiment where the scene is a repetitive event whereas the second scene has a random and non-repetitive nature.
Imaging depth, where the first scene occurs in a 2 dimensional space where only the front surface (primary focus plane) of the chip is observed whereas for the latter experiment, the movement of the sparks happen in a 3D space.
Colour properties, in which the first experiment was observed and reconstructed in monochrome whereas the second experiment was conducted in RGB format which can provide extra information about the compounds in the observed event.