Access the full text.
Sign up today, get DeepDyve free for 14 days.
A relationship between test case length and test oracles. Testers can decide what type of oracle to use if they have long or short test cases
P. Gall, A. Arnould (1995)
Formal Specifications and Test: Correctness and Oracle
J. Hicinbothom, W. Zachary (1993)
A Tool for Automatically Generating Transcripts of Human-Computer InteractionProceedings of the Human Factors and Ergonomics Society Annual Meeting, 37
(1994)
The black art of GUI testing. Dr. Dobb's
L. White, Husain Almezen, Nasser Alzeidi (2001)
User-based testing of GUI sequences and their interactionsProceedings 12th International Symposium on Software Reliability Engineering
E. Weyuker (1982)
On Testing Non-Testable ProgramsComput. J., 25
have automated the above approach by developing a technique that we call execution extraction , a form of reference testing
M. Young (2001)
Test Oracles
D. Peters, D. Parnas (1994)
Generating a test oracle from program documentation: work in progress
(2002)
Bypassing the GUI
M. Soffa, A. Memon (2001)
A comprehensive framework for testing graphical user interfaces
(2003)
Software Research, Inc., capture-Replay Tool Available at http://soft.com
G. Rothermel, Sebastian Elbaum, Alexey Malishevsky, P. Kallakuri, Xuemei Qiu (2004)
On test suite composition and cost-effective regression testingACM Trans. Softw. Eng. Methodol., 13
A. Memon, Adithya Nagarajan, Qing Xie
Automating Regression Testing for Evolving Gui Software *
L. Burdy, Yoonsik Cheon, D. Cok, Michael Ernst, J. Kiniry, Gary Leavens, K. Leino, E. Poll (2003)
An overview of JML tools and applicationsInternational Journal on Software Tools for Technology Transfer, 7
Roong-Ko Doong, P. Frankl (1994)
The ASTOOT approach to testing object-oriented programsACM Trans. Softw. Eng. Methodol., 3
D. Peters, D. Parnas (1998)
Using Test Oracles Generated from Program DocumentationIEEE Trans. Software Eng., 24
Yoonsik Cheon, Gary Leavens (2002)
A Simple and Practical Approach to Unit Testing: The JML and JUnit Way
A. Memon, M. Pollack, M. Soffa (2000)
Automated test oracles for GUIs
A. Memon, Qing Xie (2005)
Studying the fault-detection effectiveness of GUI test cases for rapidly evolving softwareIEEE Transactions on Software Engineering, 31
B. Myers (1995)
User interface software toolsACM Trans. Comput. Hum. Interact., 2
Quantile-Quantile
(2000)
Object - Oriented specificationbased testing using UML statechart diagrams
G. Bernot, M. Gaudel, B. Marre (1991)
Software testing based on formal specifications: a theory and a toolSoftw. Eng. J., 6
D. Luckham, F. Henke (1985)
An Overview of Anna, a Specification Language for AdaIEEE Software, 2
S. Antoy, R. Hamlet (2000)
Automatically Checking an Implementation against Its Formal SpecificationIEEE Trans. Software Eng., 26
David Rosenblum (1991)
Specifying concurrent systems with TSLIEEE Software, 8
T. Ostrand, Aaron Anodide, Herbert Foster, Tarak Goradia (1998)
A visual test development environment for GUI systemsACM SIGSOFT Software Engineering Notes, 23
L. Jagadeesan, A. Porter, C. Puchol, J. Ramming, L. Votta (1997)
Specification-based Testing of Reactive Software: Tools and Experiments Experience ReportProceedings of the (19th) International Conference on Software Engineering
Malte Finsterwalder (2001)
Automating Acceptance Tests for GUI Applications in an Extreme Programming Environment
Peter Vogel (1993)
An integrated general purpose automated test environment
A. Memon, Adithya Nagarajan, Qing Xie (2005)
Automating regression testing for evolving GUI softwareJ. Softw. Maintenance Res. Pract., 17
C. Wohlin, P. Runeson, Martin Höst, Magnus Ohlsson, B. Regnell, Anders Wesslén (2000)
Experimentation in software engineering: an introduction
D. Richardson (1994)
TAOS: Testing with Analysis and Oracle Support
Brian Marick (1998)
When Should a Test Be Automated
A. Memon, M. Soffa (2003)
Regression testing of GUIs
(2003)
Available at http://junit.sourceforge.net
A new \number of faults detected per comparison" measure that provides a starting point for GUI oracle comparison
L. Dillon, Qing Yu (1994)
Oracles for checking temporal properties of concurrent systems
(2003)
Capture-Replay tool 2003
all windows (LOI3): the set of all triples for all widgets of all windows. Note that the constraint for this set is simply TRUE since it is the complete state of the GUI
K. Havelund, Grigore Roşu (2004)
An Overview of the Runtime Verification Tool Java PathExplorerFormal Methods in System Design, 24
E. Siepmann, A. Newton (1994)
TOBAC: a test case browser for testing object-oriented software
Moonjoo Kim, Mahesh Viswanathan, H. Ben-Abdallah, Sampath Kannan, Insup Lee, O. Sokolsky (1998)
MaC: A Framework for Run-Time Correctness Assurance of Real-Time Systems
(1995)
Intuitive Biostatistics
L. Dillon, Y. Ramakrishna (1996)
Generating oracles from your favorite temporal logic specifications
A. Memon (2003)
Advances in GUI testingAdv. Comput., 58
L. Bousquet, F. Ouabdesselam, J. Richier, N. Zuanon (1999)
Lutess: a specification-driven testing environment for synchronous softwareProceedings of the 1999 International Conference on Software Engineering (IEEE Cat. No.99CB37002)
Hong Zhu, P. Hall, J. May (1997)
Software unit test coverage and adequacyACM Computing Surveys (CSUR), 29
A. Memon, M. Pollack, M. Soffa (2001)
Hierarchical GUI Test Case Generation Using Automated PlanningIEEE Trans. Software Eng., 27
H. Barringer, A. Goldberg, K. Havelund, Koushik Sen (2004)
Rule-Based Runtime Verification
Guidelines for test designers on how to develop test oracles, their relative strengths and weaknesses
A. Memon, Ishan Banerjee, Adithya Nagarajan (2003)
GUI ripping: reverse engineering of graphical user interfaces for testing10th Working Conference on Reverse Engineering, 2003. WCRE 2003. Proceedings.
J. Paakki, P. Inverardi (2003)
Proceedings of the 9th European software engineering conference held jointly with 11th ACM SIGSOFT international symposium on Foundations of software engineering
J. Gannon, Paul McMullin, R. Hamlet (1981)
Data Abstraction, Implementation, Specification, and TestingACM Trans. Program. Lang. Syst., 3
active window (LOI2): the set of all triples for all widgets that are a part of the currently active window W . The constraint is written as ( inW indow (#1 ; W ))
(1999)
Using pre-oracled data in model-based testing
(2005)
Received May
A. Memon, Ishan Banerjee, Adithya Nagarajan (2003)
What test oracle should I use for effective GUI testing?18th IEEE International Conference on Automated Software Engineering, 2003. Proceedings.
D. Richardson, S. Aha, T. O'Malley (1992)
Specification-based test oracles for reactive systemsInternational Conference on Software Engineering
Sebastian Elbaum, Srikanth Karre, G. Rothermel (2003)
Improving web application testing with user session data25th International Conference on Software Engineering, 2003. Proceedings.
T. Graves, M. Harrold, Jung-Min Kim, A. Porter, G. Rothermel (2001)
An empirical study of regression test selection techniquesACM Trans. Softw. Eng. Methodol., 10
A. Memon, M. Soffa, M. Pollack (2001)
Coverage criteria for GUI testing
M. Blackburn, R. Busser, A. Nauman (2002)
Interface-driven Model-based Test Generation of Java Test Drivers
Article 4, Publication date
R. Bril, R. Krikhaar, A. Postma
Journal of Software Maintenance and Evolution: Research and Practice Architectural Support in Industry: a Reflection Using C-posh
B. Green (2000)
Software test automationACM SIGSOFT Softw. Eng. Notes, 25
David Rosenblum (1992)
Towards A Method Of Programming With AssertionsInternational Conference on Software Engineering
Jason Su, P. Ritter (1991)
Experience in testing the Motif interfaceIEEE Software, 8
R. Noonan, R. Prosl (2002)
Unit testing frameworks
(2007)
Designing and comparing automated test oracles for GUI - based software applications
Test designers widely believe that the overall effectiveness and cost of software testing depends largely on the type and number of test cases executed on the software. This article shows that the test oracle , a mechanism that determines whether a software is executed correctly for a test case, also significantly impacts the fault detection effectiveness and cost of a test case. Graphical user interfaces (GUIs), which have become ubiquitous for interacting with today's software, have created new challenges for test oracle development. Test designers manually “assert” the expected values of specific properties of certain GUI widgets in each test case; during test execution, these assertions are used as test oracles to determine whether the GUI executed correctly. Since a test case for a GUI is a sequence of events, a test designer must decide: (1) what to assert; and (2) how frequently to check an assertion, for example, after each event in the test case or after the entire test case has completed execution. Variations of these two factors significantly impact the fault-detection ability and cost of a GUI test case. A technique to declaratively specify different types of automated GUI test oracles is described. Six instances of test oracles are developed and compared in an experiment on four software systems. The results show that test oracles do affect the fault detection ability of test cases in different and interesting ways: (1) Test cases significantly lose their fault detection ability when using “weak” test oracles; (2) in many cases, invoking a “thorough” oracle at the end of test case execution yields the best cost-benefit ratio; (3) certain test cases detect faults only if the oracle is invoked during a small “window of opportunity” during test execution; and (4) using thorough and frequently-executing test oracles can compensate for not having long test cases.
ACM Transactions on Software Engineering and Methodology (TOSEM) – Association for Computing Machinery
Published: Feb 1, 2007
Read and print from thousands of top scholarly journals.
Already have an account? Log in
Bookmark this article. You can see your Bookmarks on your DeepDyve Library.
To save an article, log in first, or sign up for a DeepDyve account if you don’t already have one.
Copy and paste the desired citation format or use the link below to download a file formatted for EndNote
Access the full text.
Sign up today, get DeepDyve free for 14 days.
All DeepDyve websites use cookies to improve your online experience. They were placed on your computer when you launched this website. You can change your cookie settings through your browser.