Access the full text.
Sign up today, get DeepDyve free for 14 days.
Tom Bosc (2016)
Learning to Learn Neural NetworksArXiv, abs/1610.06072
Charles Audet, J. Dennis (2006)
Mesh Adaptive Direct Search Algorithms for Constrained OptimizationSIAM J. Optim., 17
Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, Ameet Talwalkar (2016)
Hyperband: A Novel Bandit-Based Approach to Hyperparameter OptimizationJ. Mach. Learn. Res., 18
(2012)
Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning. 26-31 pages
K. Simonyan, Andrew Zisserman (2014)
Very Deep Convolutional Networks for Large-Scale Image RecognitionCoRR, abs/1409.1556
Jianfeng Liu, N. Ploskas, N. Sahinidis (2019)
Tuning BARON using derivative-free optimization algorithmsJournal of Global Optimization
A. Conn, K. Scheinberg, L. Vicente (2009)
Global Convergence of General Derivative-Free Trust-Region Algorithms to First- and Second-Order Critical PointsSIAM J. Optim., 20
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, N. Gimelshein, L. Antiga, Alban Desmaison, Andreas Köpf, E. Yang, Zach DeVito, Martin Raison, A. Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, Soumith Chintala (2019)
PyTorch: An Imperative Style, High-Performance Deep Learning Library
(2014)
Metric Optimization Engine
Barret Zoph, Quoc Le (2016)
Neural Architecture Search with Reinforcement LearningArXiv, abs/1611.01578
Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, S. Guadarrama, Trevor Darrell (2014)
Caffe: Convolutional Architecture for Fast Feature EmbeddingProceedings of the 22nd ACM international conference on Multimedia
Charles Audet, Kien-Cong Dang, D. Orban (2012)
Optimization of algorithms with OPALMathematical Programming Computation, 6
Steven Young, Derek Rose, T. Karnowski, Seung-Hwan Lim, R. Patton (2015)
Optimizing deep learning hyper-parameters through an evolutionary algorithmProceedings of the Workshop on Machine Learning in High-Performance Computing Environments
M. Abramson (2004)
Mixed Variable Optimization of a Load-Bearing Thermal Insulation System Using a Filter Pattern Search AlgorithmOptimization and Engineering, 5
G. Litjens, Thijs Kooi, B. Bejnordi, A. Setio, F. Ciompi, Mohsen Ghafoorian, J. Laak, B. Ginneken, C. Sánchez (2017)
A survey on deep learning in medical image analysisMedical image analysis, 42
Jasper Snoek, H. Larochelle, Ryan Adams (2012)
Practical Bayesian Optimization of Machine Learning Algorithms
J. Nocedal, Stephen Wright (2000)
Numerical Optimization (Springer Series in Operations Research and Financial Engineering)
L. Bottou (2012)
Stochastic Gradient Descent Tricks
(2010)
MNIST handwritten digit database
A. Deshpande (2019)
A Beginner’s Guide to Understanding Convolutional Neural Networkshttps://adesh-pande3.github.io/adeshpande3.github.io/A-Beginner’s-Guide-To-Understanding-Convolutional-Neural-Networks. https://adeshpande3.github.io/adeshpande3.github.io/A-Beginner’s-Guide-To-Understanding-Convolutional-Neur-al-Networks
Esteban Real, A. Aggarwal, Yanping Huang, Quoc Le (2018)
Regularized Evolution for Image Classifier Architecture SearchArXiv, abs/1802.01548
J. Bergstra, R. Bardenet, Yoshua Bengio, B. Kégl (2011)
Algorithms for Hyper-Parameter Optimization
B. Zoph, V. Vasudevan, J. Shlens, Q. V. Le (2018)
Learning transferable architectures for scalable image recognitionProceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE
P. Lorenzo, J. Nalepa, M. Kawulok, L. Ramos, J. Ranilla (2017)
Particle swarm optimization for hyper-parameter selection in deep neural networksProceedings of the Genetic and Evolutionary Computation Conference
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, E. Yang, Zach DeVito, Zeming Lin, Alban Desmaison, L. Antiga, Adam Lerer (2017)
Automatic differentiation in PyTorch
M. Porcelli, P. Toint (2017)
BFO, A Trainable Derivative-free Brute Force Optimizer for Nonlinear Bound-constrained Optimization and Equilibrium Computations with Continuous and Discrete VariablesACM Transactions on Mathematical Software (TOMS), 44
M. Suganuma, S. Shirakawa, T. Nagao (2017)
A genetic programming approach to designing convolutional neural network architecturesProceedings of the Genetic and Evolutionary Computation Conference
T. Elsken, J. Metzen, F. Hutter (2018)
Efficient Multi-Objective Neural Architecture Search via Lamarckian EvolutionarXiv: Machine Learning
Charles Audet, J. Dennis (2000)
Pattern Search Algorithms for Mixed Variable ProgrammingSIAM J. Optim., 11
Prasanna Balaprakash, Michael Salim, T. Uram, V. Vishwanath, Stefan Wild (2018)
DeepHyper: Asynchronous Hyperparameter Search for Deep Neural Networks2018 IEEE 25th International Conference on High Performance Computing (HiPC)
Hiva Ghanbari, K. Scheinberg (2017)
Black-Box Optimization in Machine Learning with Trust Region Based Derivative Free AlgorithmArXiv, abs/1703.06925
A. Conn, K. Scheinberg, L. Vicente (2009)
Introduction to Derivative-Free Optimization, 8
M. Abramson, Charles Audet, J. Chrissis, J. Walston (2007)
Mesh adaptive direct search algorithms for mixed variable optimizationOptimization Letters, 3
K. O’Shea, Ryan Nash (2015)
An Introduction to Convolutional Neural NetworksArXiv, abs/1511.08458
T. Tieleman, G. Hinton (2012)
Lecture 6COURSERA: Neural networks for machine learning
Barret Zoph, Vijay Vasudevan, Jonathon Shlens, Quoc Le (2017)
Learning Transferable Architectures for Scalable Image Recognition2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition
Bowen Baker, O. Gupta, Nikhil Naik, R. Raskar (2016)
Designing Neural Network Architectures using Reinforcement LearningArXiv, abs/1611.02167
Charles Audet, C. Tribes (2017)
Mesh-based Nelder–Mead algorithm for inequality constrained optimizationComputational Optimization and Applications, 71
J. Bergstra, Daniel Yamins, David Cox (2013)
Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures
D. Golovin, Benjamin Solnik, Subhodeep Moitra, G. Kochanski, J. Karro, D. Sculley (2017)
Google Vizier: A Service for Black-Box OptimizationProceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining
Charles Audet, Sébastien Digabel, C. Tribes (2018)
The Mesh Adaptive Direct Search Algorithm for Granular and Discrete VariablesSIAM J. Optim., 29
S. Smithson, Guang Yang, W. Gross, B. Meyer (2016)
Neural networks designing neural networks: Multi-objective hyper-parameter optimization2016 IEEE/ACM International Conference on Computer-Aided Design (ICCAD)
Charles Audet, W. Hare (2017)
Derivative-Free and Blackbox Optimization
V. Torczon (1997)
On the Convergence of Pattern Search AlgorithmsSIAM J. Optim., 7
A Beginner's Guide To Understanding Convolutional Neural Networks. https: //adeshpande3.github.io/adeshpande3.github.io/A-Beginner's-Guide-To-Understanding-Convolutional
X. Bouthillier, C. Tsirigotis, F. Corneau-Tremblay, P. Delaunay, R. Askari, D. Suhubdy, M. Noukhovitch, D. Serdyuk, A. Bergeron, P. Henderson, P. Lamblin, M. Bronzi, C. Beckham (2019)
Oríon - Asynchronous Distributed Hyperparameter OptimizationRetrieved September 19, 2020, from https://github.com/Epistimio/orion. DOI:https://doi.org/10.5281/zenodo.3478592, 19
Charles Audet, Vincent Béchard, Sébastien Digabel (2006)
Nonsmooth optimization through Mesh Adaptive Direct Search and Variable Neighborhood SearchJournal of Global Optimization, 41
Alexandre Mello, Jonathan Matos, M. Stemmer, A. Britto, Alessandro Koerich (2019)
A Novel Orthogonal Direction Mesh Adaptive Direct Search Approach for SVM Hyperparameter TuningArXiv, abs/1904.11649
Charles Audet, D. Orban (2006)
Finding Optimal Algorithmic Parameters Using Derivative-Free OptimizationSIAM J. Optim., 17
T. Elsken, J. H. Metzen, F. Hutter (2018)
Neural Architecture Search: A SurveyTechnical Report. arXiv. http://arxiv.org/abs/1808.05377
J. Bergstra, Yoshua Bengio (2012)
Random Search for Hyper-Parameter OptimizationJ. Mach. Learn. Res., 13
S. Le Digabel, S. M. Wild (2015)
A Taxonomy of Constraints in Simulation-Based OptimizationTechnical Report G-2015-57. Les cahiers du GERAD. http://www.optimization-online.org/DB_HTML/2015/05/4931.html
Diederik Kingma, Jimmy Ba (2014)
Adam: A Method for Stochastic OptimizationCoRR, abs/1412.6980
R. Hooke, T. Jeeves (1961)
`` Direct Search'' Solution of Numerical and Statistical ProblemsJ. ACM, 8
I. Loshchilov, Frank Hutter (2016)
CMA-ES for Hyperparameter Optimization of Deep Neural NetworksArXiv, abs/1604.07269
M. Kokkolaras, Charles Audet, J. Dennis (2001)
Mixed Variable Optimization of the Number and Composition of Heat Intercepts in a Thermal Insulation SystemOptimization and Engineering, 2
S. Levine, P. Pastor, A. Krizhevsky, Deirdre Quillen (2016)
Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collectionThe International Journal of Robotics Research, 37
(2019)
VGG16: Convolutional Network for Classification and Detection. https://neurohive.io/en/popularnetworks/vgg16
Matthias Feurer, Aaron Klein, Katharina Eggensperger, J. Springenberg, Manuel Blum, F. Hutter (2015)
Efficient and Robust Automated Machine Learning
M. Powell (2009)
The BOBYQA algorithm for bound constrained optimization without derivatives
Fabian Pedregosa, G. Varoquaux, Alexandre Gramfort, V. Michel, B. Thirion, O. Grisel, Mathieu Blondel, Gilles Louppe, P. Prettenhofer, Ron Weiss, Ron Weiss, J. Vanderplas, Alexandre Passos, D. Cournapeau, M. Brucher, M. Perrot, E. Duchesnay (2011)
Scikit-learn: Machine Learning in PythonArXiv, abs/1201.0490
Sébastien Digabel (2009)
NOMAD: Nonlinear Optimization with the MADS Algorithm
(2019)
Introduction to Convolutional Neural Networks. https://www.vaetas.cz/posts/introconvolutional-neural-networks
Sébastien Digabel (2010)
Algorithm xxx : NOMAD : Nonlinear Optimization with the MADS algorithm
Yoshua Bengio (2012)
Practical Recommendations for Gradient-Based Training of Deep Architectures
O. Castro-Orgaz, W. Hager (2019)
and sShallow Water Hydraulics
G. Diaz, Achille Fokoue, G. Nannicini, H. Samulowitz (2017)
An effective algorithm for hyperparameter optimization of neural networksArXiv, abs/1705.08520
Martin Wistuba, Nicolas Schilling, L. Schmidt-Thieme (2017)
Scalable Gaussian process-based transfer surrogates for hyperparameter optimizationMachine Learning, 107
Han Xiao, Kashif Rasul, Roland Vollgraf (2017)
Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning AlgorithmsArXiv, abs/1708.07747
Sébastien Digabel, Stefan Wild (2015)
A Taxonomy of Constraints in Simulation-Based OptimizationarXiv: Optimization and Control
M. Abramson, Charles Audet, J. Dennis (2004)
Filter Pattern Search Algorithms for Mixed Variable Constrained Optimization Problems
F. Hutter, H. Hoos, Kevin Leyton-Brown (2011)
Sequential Model-Based Optimization for General Algorithm Configuration
Arber Zela, Aaron Klein, S. Falkner, F. Hutter (2018)
Towards Automated Deep Learning: Efficient Joint Neural Architecture and Hyperparameter SearchArXiv, abs/1807.06906
Yann LeCun, L. Bottou, G. Orr, K. Müller (2012)
Efficient BackProp
A. Krizhevsky (2009)
Learning Multiple Layers of Features from Tiny Images
M. Hassan (2019)
VGG16: Convolutional Network for Classification and Detectionhttps://neurohive.io/en/popular-networks/vgg16/.
John Duchi, Elad Hazan, Y. Singer (2011)
Adaptive Subgradient Methods for Online Learning and Stochastic OptimizationJ. Mach. Learn. Res., 12
The performance of deep neural networks is highly sensitive to the choice of the hyperparameters that define the structure of the network and the learning process. When facing a new application, tuning a deep neural network is a tedious and time-consuming process that is often described as a “dark art.” This explains the necessity of automating the calibration of these hyperparameters. Derivative-free optimization is a field that develops methods designed to optimize time-consuming functions without relying on derivatives. This work introduces the HyperNOMAD package, an extension of the NOMAD software that applies the MADS algorithm [7] to simultaneously tune the hyperparameters responsible for both the architecture and the learning process of a deep neural network (DNN). This generic approach allows for an important flexibility in the exploration of the search space by taking advantage of categorical variables. HyperNOMAD is tested on the MNIST, Fashion-MNIST, and CIFAR-10 datasets and achieves results comparable to the current state of the art.
ACM Transactions on Mathematical Software (TOMS) – Association for Computing Machinery
Published: Jun 26, 2021
Keywords: Deep neural networks
Read and print from thousands of top scholarly journals.
Already have an account? Log in
Bookmark this article. You can see your Bookmarks on your DeepDyve Library.
To save an article, log in first, or sign up for a DeepDyve account if you don’t already have one.
Copy and paste the desired citation format or use the link below to download a file formatted for EndNote
Access the full text.
Sign up today, get DeepDyve free for 14 days.
All DeepDyve websites use cookies to improve your online experience. They were placed on your computer when you launched this website. You can change your cookie settings through your browser.