Deep learning advances super-resolution imaging

Deep learning advances super-resolution imaging research highlights MICROSCOPY Two approaches apply deep learning to improve single-molecule localization microscopy. n single-molecule localization microscopy Approaches that generate complete methods such as photoactivated images from relatively sparse input data can Ilocalization microscopy (PALM) and yield artifacts. Zimmer’s team addressed this stochastic optical reconstruction microscopy issue by developing an algorithm that can (STORM), samples are imaged over multiple identify and reduce artifacts by comparing rounds; in each round a random subset of the generated image with the wide-field fluorophores is activated and imaged at image. This approach was inspired by diffraction-limited resolution. The precise NanoJ-SQUIRREL, developed by Ricardo positions of these individual emitters are Henriques’s lab. A visual representation of the Deep-STORM determined, and after multiple rounds Zimmer notes that a major focus was network architecture. Reproduced with permission a composite super-resolution image is identifying the best way to train the neural from Nehme et al. (2018), The Optical Society. generated from the localized fluorophores. network. For this, they developed a data- Localization microscopy tends to augmentation strategy that allowed them to require thousands of rounds of imaging to effectively increase the number of training generate a high-resolution image, because or blurred images,” explains Zimmer. The images without more experimental data. in each round, sparse emission is preferred. researchers used their approach to generate Still, he recalls, “it was somewhat surprising Sparse emission minimizes the likelihood high-quality images of microtubules, nuclear to see that ANNA-PALM only needs to of simultaneous emission from closely pores and mitochondria, and found that be a trained on a few super-resolution positioned fluorophores, which confounds they were able to obtain super-resolution images—in some cases just one.” He explains their precise localization. The end result images of more than a thousand cells in that ANNA-PALM will improve with time is long acquisition times, which hinder around three hours—an astonishing feat for if trained on more data. Shechtman was throughput and most live-cell applications. the field. also surprised by the training of the neural Many existing methods address these Another method, Deep-STORM, was network; he notes that small numbers of challenges, including some that allow developed by Yoav Shechtman, Tomer experimental images could train the network accurate localization in images where Michaeli and their joint student Elias successfully, and that “while training fluorophore emission is dense. Although Nehme at the Technion – Israel Institute of the net on experimental measurements these work well in some cases, they can Technology. In Deep-STORM, no a priori produced the best results, training the net limit image quality and resolution. For this knowledge regarding the underlying object on simulated data, of which we could easily reason, two groups independently developed is used. Instead, the artificial neural network generate huge amounts, already yielded approaches to improve the acquisition speed ‘learns’ to extract information directly from excellent images.” of PALM/STORM while maintaining image images of dense blinking emitters, after Although these methods represent early resolution. In both cases, the researchers being trained on correct emitter positions. days in the application of deep learning used deep learning to generate super- This allows the trained model to infer to super-resolution microscopy, they are resolution images from a relatively small correct emitter positions in images where poised to have an important impact on the number of frames of localization microscopy emission is dense and output a super- field and herald a bright future for this area. data. Deep learning is a type of machine resolution image of a structure rapidly. Shechtman says that user-friendly versions learning that uses artificial neural networks Here, the ability to image densely labeled of these tools are an important future goal, to learn a mapping between input and samples corresponds to a reduced total and notes that his group is developing a output data. Once trained, these models can acquisition time. Using their approach, the stand-alone version of Deep-STORM. predict outputs from supplied input data. researchers were able to outperform existing Zimmer says that his team is currently One of the two methods, artificial neural algorithms for image generation from developing tools to facilitate training for use network accelerated PALM (ANNA-PALM), densely labeled frames of synthetic data with ANNA-PALM. ❐ was developed by Christophe Zimmer and images of microtubules. and his student Wei Ouyang at the Institut Although the two approaches differ Rita Strack Pasteur. In ANNA-PALM, an artificial in their neural networks and training neural network is trained on localizations approach, they both can be used to generate Published online: 31 May 2018 from a small number of frames matched super-resolution images that are appropriate https://doi.org/10.1038/s41592-018-0028-9 with dense localization data obtained from for quantitative analysis. One important long-duration acquisitions of the same distinction between the output from both Research papers Nehme, E. et al. Deep-STORM: super-resolution structures. The neural network can then ANNA-PALM and Deep-STORM and that single-molecule microscopy by deep learning. Optic a produce accurate super-resolution images of traditional PALM/STORM is that the 18, 2334–2536 (2018). from images generated from a smaller neural networks produce super-resolution Ouyang, W. et al. Deep learning massively accelerates number of frames. “This strategy resembles images that are not composed of super-resolution localization microscopy. Nat. Biotechnol. 36, 460–468 (2018). how humans recognize objects in noisy a compilation of emitter positions. Nature Methods | VOL 15 | JUNE 2018 | 403–409 | www.nature.com/naturemethods © 2018 Nature America Inc., part of Springer Nature. All rights reserved. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Nature Methods Springer Journals

Deep learning advances super-resolution imaging

Free
1 page

Loading next page...
1 Page
 
/lp/springer_journal/deep-learning-advances-super-resolution-imaging-W9KLOQ1WI1
Publisher
Springer Journals
Copyright
Copyright © 2018 by The Author(s)
Subject
Life Sciences; Life Sciences, general; Biological Techniques; Biological Microscopy; Biomedical Engineering/Biotechnology; Bioinformatics; Proteomics
ISSN
1548-7091
eISSN
1548-7105
D.O.I.
10.1038/s41592-018-0028-9
Publisher site
See Article on Publisher Site

Abstract

research highlights MICROSCOPY Two approaches apply deep learning to improve single-molecule localization microscopy. n single-molecule localization microscopy Approaches that generate complete methods such as photoactivated images from relatively sparse input data can Ilocalization microscopy (PALM) and yield artifacts. Zimmer’s team addressed this stochastic optical reconstruction microscopy issue by developing an algorithm that can (STORM), samples are imaged over multiple identify and reduce artifacts by comparing rounds; in each round a random subset of the generated image with the wide-field fluorophores is activated and imaged at image. This approach was inspired by diffraction-limited resolution. The precise NanoJ-SQUIRREL, developed by Ricardo positions of these individual emitters are Henriques’s lab. A visual representation of the Deep-STORM determined, and after multiple rounds Zimmer notes that a major focus was network architecture. Reproduced with permission a composite super-resolution image is identifying the best way to train the neural from Nehme et al. (2018), The Optical Society. generated from the localized fluorophores. network. For this, they developed a data- Localization microscopy tends to augmentation strategy that allowed them to require thousands of rounds of imaging to effectively increase the number of training generate a high-resolution image, because or blurred images,” explains Zimmer. The images without more experimental data. in each round, sparse emission is preferred. researchers used their approach to generate Still, he recalls, “it was somewhat surprising Sparse emission minimizes the likelihood high-quality images of microtubules, nuclear to see that ANNA-PALM only needs to of simultaneous emission from closely pores and mitochondria, and found that be a trained on a few super-resolution positioned fluorophores, which confounds they were able to obtain super-resolution images—in some cases just one.” He explains their precise localization. The end result images of more than a thousand cells in that ANNA-PALM will improve with time is long acquisition times, which hinder around three hours—an astonishing feat for if trained on more data. Shechtman was throughput and most live-cell applications. the field. also surprised by the training of the neural Many existing methods address these Another method, Deep-STORM, was network; he notes that small numbers of challenges, including some that allow developed by Yoav Shechtman, Tomer experimental images could train the network accurate localization in images where Michaeli and their joint student Elias successfully, and that “while training fluorophore emission is dense. Although Nehme at the Technion – Israel Institute of the net on experimental measurements these work well in some cases, they can Technology. In Deep-STORM, no a priori produced the best results, training the net limit image quality and resolution. For this knowledge regarding the underlying object on simulated data, of which we could easily reason, two groups independently developed is used. Instead, the artificial neural network generate huge amounts, already yielded approaches to improve the acquisition speed ‘learns’ to extract information directly from excellent images.” of PALM/STORM while maintaining image images of dense blinking emitters, after Although these methods represent early resolution. In both cases, the researchers being trained on correct emitter positions. days in the application of deep learning used deep learning to generate super- This allows the trained model to infer to super-resolution microscopy, they are resolution images from a relatively small correct emitter positions in images where poised to have an important impact on the number of frames of localization microscopy emission is dense and output a super- field and herald a bright future for this area. data. Deep learning is a type of machine resolution image of a structure rapidly. Shechtman says that user-friendly versions learning that uses artificial neural networks Here, the ability to image densely labeled of these tools are an important future goal, to learn a mapping between input and samples corresponds to a reduced total and notes that his group is developing a output data. Once trained, these models can acquisition time. Using their approach, the stand-alone version of Deep-STORM. predict outputs from supplied input data. researchers were able to outperform existing Zimmer says that his team is currently One of the two methods, artificial neural algorithms for image generation from developing tools to facilitate training for use network accelerated PALM (ANNA-PALM), densely labeled frames of synthetic data with ANNA-PALM. ❐ was developed by Christophe Zimmer and images of microtubules. and his student Wei Ouyang at the Institut Although the two approaches differ Rita Strack Pasteur. In ANNA-PALM, an artificial in their neural networks and training neural network is trained on localizations approach, they both can be used to generate Published online: 31 May 2018 from a small number of frames matched super-resolution images that are appropriate https://doi.org/10.1038/s41592-018-0028-9 with dense localization data obtained from for quantitative analysis. One important long-duration acquisitions of the same distinction between the output from both Research papers Nehme, E. et al. Deep-STORM: super-resolution structures. The neural network can then ANNA-PALM and Deep-STORM and that single-molecule microscopy by deep learning. Optic a produce accurate super-resolution images of traditional PALM/STORM is that the 18, 2334–2536 (2018). from images generated from a smaller neural networks produce super-resolution Ouyang, W. et al. Deep learning massively accelerates number of frames. “This strategy resembles images that are not composed of super-resolution localization microscopy. Nat. Biotechnol. 36, 460–468 (2018). how humans recognize objects in noisy a compilation of emitter positions. Nature Methods | VOL 15 | JUNE 2018 | 403–409 | www.nature.com/naturemethods © 2018 Nature America Inc., part of Springer Nature. All rights reserved.

Journal

Nature MethodsSpringer Journals

Published: May 31, 2018

References

You’re reading a free preview. Subscribe to read the entire article.


DeepDyve is your
personal research library

It’s your single place to instantly
discover and read the research
that matters to you.

Enjoy affordable access to
over 18 million articles from more than
15,000 peer-reviewed journals.

All for just $49/month

Explore the DeepDyve Library

Search

Query the DeepDyve database, plus search all of PubMed and Google Scholar seamlessly

Organize

Save any article or search result from DeepDyve, PubMed, and Google Scholar... all in one place.

Access

Get unlimited, online access to over 18 million full-text articles from more than 15,000 scientific journals.

Your journals are on DeepDyve

Read from thousands of the leading scholarly journals from SpringerNature, Elsevier, Wiley-Blackwell, Oxford University Press and more.

All the latest content is available, no embargo periods.

See the journals in your area

DeepDyve

Freelancer

DeepDyve

Pro

Price

FREE

$49/month
$360/year

Save searches from
Google Scholar,
PubMed

Create lists to
organize your research

Export lists, citations

Read DeepDyve articles

Abstract access only

Unlimited access to over
18 million full-text articles

Print

20 pages / month

PDF Discount

20% off