How to do Experiments (Extended Advertisement) David S. Johnson AT&T Research d s j @ r e s e a r c h . a r t . com There has been much encouragement recently for the experimental analysis of algorithms. This encouragement has come both from those like myself who are already actively experimenting and from funding agencies who view experimentation as providing a pathway from theory into practice. Unfortunately, many who attempt to enter the field of experimental analysis discover that it is fraught with pitfalls. In many ways, the implementation of an algorithm is the easy part. The hard part is successfully using that implementation to produce meaningful and valuable (and publishable!) research results. I am currently drafting A Theoretician's Guide to the Experimental Analysis of Algorithms which I hope will assist would-be experimenters in producing good experimental papers that get accepted into journals and relevant conferences like SODA. My original plan was to publish the Guide in this issue of S I G A C T News, but it has become too lengthy and the publication deadline has drawn too near. Consequently, I shall simply use this page to announce the availability of an evolving draft on the Web, and to solicit feedback plus suggested examples of good and bad experimental work that I might cite. A postscript version of the current draft is available via anonymous FTP from the following sites: dimacs, rutgers, edu in the file p u b / d s j / t e m p / e x p e r , ps netlib, art. corn in the filenetlib/att/math/dsj/exper.ps The paper will be organized around a collection of general principles that I will illustrate by describing some of the more egregious violations that I have encountered as editor, referee, and survey paper author. The current draft list seven principles, whose shorthand descriptions are (1) Perform newsworthy experiments, (2) Tie your paper to the literature, (3) Use efficient implementations, (4) Use testbeds that can support general conclusions, (5) Provide explanations and measurements to back them up, (6) Ensure reproducibility, and (7) Ensure comparability and give the full picture. Several of these principles may be somewhat controversial, at least as I elaborate them. Indeed, although there is much common agreement on what makes good experimental analysis of algorithms, certain aspects have been the subject of much debate, such as the relevance of running time comparisons. Other points of view, as well good pointers into the literature on experimental methodology, can be found in two recent papers from the operations research literature: W. S. Barr, B. L. Golden, J. P. Kelly, M. G. C. Resende, and W. R. Stewart, "Designing and reporting on computational experiments with heuristic methods," J. Heuristics 1"1 (1995), 9-32, and C. C. McGeoch, "Toward an experimental method for algorithm simulation," INFORMS J. Computing 8:1 (1996), 1-15. 87

/lp/association-for-computing-machinery/how-to-do-experiments-extended-advertisement-63gM2Ff0bi