Neural network robustness via binary activation
| DWPI Title: Computer-implement method for increasing neural network robustness, involves stopping training of artificial neural network when activation functions converge to threshold activation and network exhibits spiking behavior |
| Abstract: A method of increasing neural network robustness. The method comprises defining an artificial neural network comprising a number of bounded ramp activation functions. The network is trained iteratively in a layer-by-layer fashion. Each iteration increases the slope of the activation functions toward a discrete threshold activation and stops when the activation functions converge to the threshold activation and the network exhibits spiking behavior. Alternatively, weight agnostic neural networks are created, wherein nodes in the networks comprise fixed shared weights. A subset of networks is identified that comprise activation functions compatible with neuromorphic hardware and are tested with a specified number of shared weight values. A score is generated for each combination of network and weight value according to performance and mapping to neuromorphic hardware, and the networks are ranked. The networks are then combined according to ranking to create a new network that exhibits spiking behavior. |
| Use: Computer-implemented method for increasing neural network robustness. Can also be used for training binary-activation neural networks. |
| Advantage: The method enables utilizing a neuromorphic processor to potentially offer milliwatt scale computation while maintaining state-of-the-art-performance in an efficient manner. |
| Novelty: The method involves using a number of processors to perform the steps of defining an artificial neural network. The artificial neural network comprises a number of bounded ramp activation functions. The artificial neural network is trained in a layer-by-layer fashion. Each iteration of training increases the slope of the bounded ramp activation functions toward a discrete threshold activation. A training of the artificial neural network is stopped when the activation functions converge to the threshold activation and the artificial neural network exhibits spiking behavior. |
| Filed: 5/7/2021 |
| Application Number: US17314751A |
| Tech ID: SD 15133.0 |
| This invention was made with Government support under Contract No. DE-NA0003525 awarded by the United States Department of Energy/National Nuclear Security Administration. The Government has certain rights in the invention. |
| Data from Derwent World Patents Index, provided by Clarivate All rights reserved. Republication or redistribution of Clarivate content, including by framing or similar means, is prohibited without the prior written consent of Clarivate. Clarivate and its logo, as well as all other trademarks used herein are trademarks of their respective owners and used under license. |