Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You or Your Team.

Learn More →

Equivalent-accuracy accelerated neural-network training using analogue memory

Equivalent-accuracy accelerated neural-network training using analogue memory Neural-network training can be slow and energy intensive, owing to the need to transfer the weight data for the network between conventional digital memory chips and processor chips. Analogue non-volatile memory can accelerate the neural-network training algorithm known as backpropagation by performing parallelized multiply–accumulate operations in the analogue domain at the location of the weight data. However, the classification accuracies of such in situ training using non-volatile-memory hardware have generally been less than those of software-based training, owing to insufficient dynamic range and excessive weight-update asymmetry. Here we demonstrate mixed hardware–software neural-network implementations that involve up to 204,900 synapses and that combine long-term storage in phase-change memory, near-linear updates of volatile capacitors and weight-data transfer with ‘polarity inversion’ to cancel out inherent device-to-device variations. We achieve generalization accuracies (on previously unseen data) equivalent to those of software-based training on various commonly used machine-learning test datasets (MNIST, MNIST-backrand, CIFAR-10 and CIFAR-100). The computational energy efficiency of 28,065 billion operations per second per watt and throughput per area of 3.6 trillion operations per second per square millimetre that we calculate for our implementation exceed those of today’s graphical processing units by two orders of magnitude. This work provides a path towards hardware accelerators that are both fast and energy efficient, particularly on fully connected neural-network layers. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Nature Springer Journals

Equivalent-accuracy accelerated neural-network training using analogue memory

Download PDF
22 pages

Loading next page...
 
/lp/springer_journal/equivalent-accuracy-accelerated-neural-network-training-using-analogue-84dE5rmJWD
Publisher
Springer Journals
Copyright
Copyright © 2018 by Macmillan Publishers Ltd., part of Springer Nature
Subject
Science, Humanities and Social Sciences, multidisciplinary; Science, Humanities and Social Sciences, multidisciplinary; Science, multidisciplinary
ISSN
0028-0836
eISSN
1476-4687
DOI
10.1038/s41586-018-0180-5
pmid
29875487
Publisher site
See Article on Publisher Site

Abstract

Neural-network training can be slow and energy intensive, owing to the need to transfer the weight data for the network between conventional digital memory chips and processor chips. Analogue non-volatile memory can accelerate the neural-network training algorithm known as backpropagation by performing parallelized multiply–accumulate operations in the analogue domain at the location of the weight data. However, the classification accuracies of such in situ training using non-volatile-memory hardware have generally been less than those of software-based training, owing to insufficient dynamic range and excessive weight-update asymmetry. Here we demonstrate mixed hardware–software neural-network implementations that involve up to 204,900 synapses and that combine long-term storage in phase-change memory, near-linear updates of volatile capacitors and weight-data transfer with ‘polarity inversion’ to cancel out inherent device-to-device variations. We achieve generalization accuracies (on previously unseen data) equivalent to those of software-based training on various commonly used machine-learning test datasets (MNIST, MNIST-backrand, CIFAR-10 and CIFAR-100). The computational energy efficiency of 28,065 billion operations per second per watt and throughput per area of 3.6 trillion operations per second per square millimetre that we calculate for our implementation exceed those of today’s graphical processing units by two orders of magnitude. This work provides a path towards hardware accelerators that are both fast and energy efficient, particularly on fully connected neural-network layers.

Journal

Nature – Springer Journals

Published: Jun 6, 2018

References

  • Deep learning
    LeCun, Y; Bengio, Y; Hinton, G
  • A million spiking-neuron integrated circuit with a scalable communication network and interface
    Merolla, PA
  • Convolutional networks for fast, energy-efficient neuromorphic computing
    Esser, SK
  • An all-analog expandable neural network LSI with on-chip backpropagation learning
    Morie, T; Amemiya, Y
  • Experimental demonstration and tolerancing of a large-scale neural network (165,000 synapses), using phase-change memory as the synaptic weight element
    Burr, GW
  • Acceleration of deep neural network training with resistive cross-point devices: design considerations
    Gokmen, T; Vlasov, Y
  • Neuromorphic computing using non-volatile memory
    Burr, GW
  • Fully parallel write/read in resistive synaptic array for accelerating on-chip learning
    Gao, L
  • Training and operation of an integrated neuromorphic network based on metal-oxide memristors
    Prezioso, M
  • Optimization of conductance change in Pr1−x Ca x MnO3-based synaptic devices for neuromorphic systems
    Jang, J-W; Park, S; Burr, GW; Hwang, H; Jeong, Y-H
  • Utilizing multiple state variables to improve the dynamic range of analog switching in a memristor
    Jeong, YJ; Kim, S; Lu, WD
  • Ferroelectric artificial synapses for recognition of a multishaded image
    Kaneko, Y; Nishitani, Y; Ueda, M
  • A non-volatile organic electrochemical device as a low-voltage artificial synapse for neuromorphic computing
    Burgt, Y
  • Gradient-based learning applied to document recognition
    Lecun, Y; Bottou, L; Bengio, Y; Haffner, P
  • Towards on-chip acceleration of the backpropagation algorithm using non-volatile memory
    Narayanan, P
  • Learning representations by backpropagating errors
    Rumelhart, DE; Hinton, GE; Williams, RJ
  • Parallel programming of resistive cross-point array for synaptic plasticity
    Xu, Z
  • High-precision tuning of state for memristive devices by adaptable variation-tolerant algorithm
    Alibart, F; Gao, L; Hoskins, BD; Strukov, DB
  • Li-ion synaptic transistor for low power analog computing
    Fuller, EJ
  • Long short-term memory
    Hochreiter, S; Schmidhuber, J
  • Access devices for 3D crosspoint memory
    Burr, GW
  • Recovery and drift dynamics of resistance and threshold voltages in phase-change memories
    Ielmini, D; Lacaita, AL; Mantegazza, D
  • Matching properties of MOS transistors
    Pelgrom, MJM; Duinmaijer, ACJ; Welbers, APG
  • What is predictive technology model (PTM)?
    Cao, Y