Combining machine learning with interferometric structured illumination microscopy for the imaging of dynamic process in three-dimensions. 

Session

Poster Session Two

Authors

Dr Edward Ward (3), Ms Rebecca McClelland (2), Mr Jacob Lamb (1), Dr Meng Lu (3), Professor Clemens Kaminski (3)

Affiliations

1. Cambridge University
2. university of Cambridge
3. University of Cambridge

Abstract text

Background

Structured illumination microscopy (SIM) is one of the most popular techniques for super-resolution imaging of live biological samples due to its speed and low illumination intensities. In a typical SIM setup, the sample is illuminated with sinusoidal patterns of excitation light, which produce interference patterns with the fine structures of the sample. By shifting and rotating the structured excitation, super-resolution information can be extracted from these changing interference patterns. The resolution achievable in SIM is proportional to the periodicity of the excitation patterns and the optimum periodicity is different for each excitation wavelength. With traditional SIM techniques based on diffraction gratings or spatial light modulators, the requirement for wavelength-dependent pattern periodicity prohibits simultaneous imaging in multiple colours. This means that different structures must be imaged sequentially, introducing temporal offset between channels which makes observations of dynamic interactions unreliable.

We address this issue with Machine-learning Assisted, Interferometric Structured Illumination Microscopy, MAI-SIM. [1] Using interferometry to produce the excitation patterns enables simultaneous imaging of different fluorescent labels with the pattern periodicity optimised for all wavelengths. Furthermore, we supplement the instrument with a reconstruction method based on machine learning. This not only enables multi-colour imaging, but also provides real-time reconstructions that are less prone to errors from noise and uneven pattern phase shifts. 

Results

The instrument is based on a Michelson interferometer where wedge fringes are projected onto the sample to generate the sinusoidal illumination. This interferometric approach has the advantage that the periodicity of the pattern varies with excitation wavelength, a property that is vital for maintaining the resolution increase with multiple colours. The rotation and phase-stepping of the patterns is achieved using a single galvanometric mirror, which greatly reduces the cost and complexity of the instrument.  

A limitation of classical reconstruction algorithms is that they require near-perfect phase steps to achieve reliable reconstructions; our novel image reconstruction method is capable of handling uneven phase stepping without a loss in performance. The real-time reconstructions are performed using a convolutional neural network specifically trained for interferometric SIM data, wherein the patterns needed for reconstruction are different for each wavelength and it is not possible to achieve the ideal steps for all wavelengths simultaneously. Using this method, we can achieve de-noising, pattern estimation and reconstruction of multiple colour channels simultaneously in 10s of milliseconds on a consumer-grade graphics card. [2] We also make use of video transformer networks for post-acquisition reconstruction, where the temporal information is used to increase reconstruction performance on rapidly moving objects. [3]

In this presentation we will show how MAI-SIM can be implemented on existing widefield systems at minimal cost and demonstrate the advantages of interferometric SIM by highlighting the importance of simultaneous multi-colour acquisition in biological imaging. We will also show how the technique can be expanded to the 3D regime and how it can provide volumetric sample information and video-rate reconstructions, viewed from multiple angles.

References

[1] Ward, E.N., Hecker, L., Christensen, C.N. et al. Machine learning assisted interferometric structured illumination microscopy for dynamic biological imaging. Nature Communications 13, 7836 (2022)

[2] Charles N. Christensen, Edward N. Ward, Meng Lu, Pietro Lio, and Clemens F. Kaminski, ML-SIM: universal reconstruction of structured illumination microscopy images using transfer learning. Biomed. Opt. Express 12, 2720-2733 (2021)

[3] Christensen, C.N., Lu, M., Ward, E.N., Lio, P. and Kaminski, C.F. Spatio-temporal Vision Transformer for Super-resolution Microscopy. arXiv preprint arXiv:2203.00030 (2022)