Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 7-Day Trial for You or Your Team.

Learn More →

Summary Report of the First International Competition on Computational Models of Argumentation

Summary Report of the First International Competition on Computational Models of Argumentation Computational models of argumentation are an active research discipline within artificial intelligence that has grown since the beginning of the 1990s (Dung 1995). While still a young field when compared to areas such as SAT solving and logic programming, the argumentation community is very active, with a conference series (COMMA, which began in 2006) and a variety of workshops and special issues of journals. Argumentation has also worked its way into a variety of applications. For example, Williams et al. (2015) described how argumentation techniques are used for recommending cancer treatments, while Toniolo et al. (2015) detail how argumentation‐based techniques can support critical thinking and collaborative scientific inquiry or intelligence analysis.Many of the problems that argumentation deals with are computationally difficult, and applications utilizing argumentation therefore require efficient solvers. To encourage this line of research, we organised the First International Competition on Computational Models of Argumentation (ICCMA), with the intention of assessing and promoting state‐of‐the‐art solvers for abstract argumentation problems, and to identify families of challenging benchmarks for such solvers.The objective of ICCMA'15 is to allow researchers to compare the performance of different solvers systematically on common benchmarks and rules. Moreover, as witnessed by competitions in other AI disciplines such as planning and SAT solving, we see ICCMA as a new pillar of the community, which provides information and insights on the current state of the art and highlights future challenges and developments.This report summarizes the first ICCMA held in 2015 (ICCMA'15). In this competition, solvers were invited to address standard decision and enumeration problems of abstract argumentation frameworks (Dunne and Wooldridge 2009). Solvers' performance is evaluated based on their time taken to provide a correct solution for a problem; incorrect results were discarded. More information about the competition, including complete results and benchmarks, can be found on the ICCMA website.1TracksIn abstract argumentation (Dung 1995), a directed graph (A, R) is used as knowledge representation formalism, where the set of nodes A are identified with the arguments under consideration and R represents a conflict‐relation between arguments, that is, aRb for a, b ∈ A if a is a counterargument for b. The framework is abstract because the content of the arguments is left unspecified. They could, for example, consist of a chain of logical deductions from logic programming with defeasible rules (Simari 1992); a proof for a theorem in classical logic (Besnard and Hunter 2007); or an informal presumptive reason in favour of some conclusion (Walton, Reed, and Macagno 2008). The notion of conflict then depends on the chosen formalization. Irrespective of the precise formalization used, one can identify a subset of arguments that can be collectively accepted given inter‐argument conflicts. Such a subset is referred to as an extension, and (Dung 1995) defined four commonly used argumentation semantics — namely the complete (CO), preferred (PR), grounded (GR), and stable (ST) semantics — each of which defines an extension differently. More precisely, a complete extension is a set of arguments that do not attack each other,2 and in which arguments defend each other; a preferred extension is a maximal (with regard to set inclusion) complete extension; the grounded extension is the minimal (with regard to set inclusion) complete extension; and a stable extension is a complete extension such that each argument not in the extension is attacked by at least one argument within the extension.The competition was organized around four computational tasks of abstract argumentation: (1) Given an abstract argumentation framework, determine some extension (SE). (2) Given an abstract argumentation framework, determine all extensions (EE). (3) Given an abstract argumentation framework and some argument, decide whether the given argument is contained in some extension (DC). (4) Given an abstract argumentation framework and some argument, decide whether the given argument is contained in all extensions (DS).Combining these four different tasks with the four semantics discussed above yields a total of 16 tracks that constituted ICCMA'15. Each submitted solver was free to support any number of these tracks.ParticipantsThe competition received 18 solvers from research groups in Austria, China, Cyprus, Finland, France, Germany, Italy, Romania, and UK, of which 8 were submitted to all tracks. The solvers used a variety of approaches and programming languages to solve the competition tasks. In particular, 5 solvers were based on transformations of argumentation problems to SAT, 3 on transformations to ASP, 2 on CSP, and 8 were built on tailor‐made algorithms. Seven solvers were implemented in C/C++, 4 in Java, 2 used shell‐scripts for translations to other formalisms, and the remaining solvers were implemented in Haskell, Lisp, Prolog, Python, and Go.All participants were required to submit the source code of their solver, which was made freely available after the competition, to foster independent evaluation and exploitation in research or real‐world scenarios, and to allow for further refinements. Submitted solvers were required to support the probo (Cerutti et al. 2014)3 command‐line interface, which was specifically designed for running and comparing solvers within ICCMA.Performance EvaluationEach solver was evaluated over N different argumentation graph instances within each track (N = 192 for SE and EE, and 576 for DC and DS). Instances were generated with the intention of being challenging — one group of instances was generated so as to contain a large grounded extension and few extensions in the other semantics. This group's graphs were large (1224 to 9473 arguments), and challenged solvers that scaled poorly (that is, those that used combinatorial approaches for computing extensions). A second group of instances was smaller (141 to 400 arguments), but had a rich structure of stable, preferred, and complete extensions (up to 159 complete extensions for the largest graphs) and thus provided combinatorial challenges for solvers relying on simple search‐based algorithms. A final group contained medium‐sized graphs (185 to 996 arguments) and featured many strongly connected components with many extensions. This group was particularly challenging for solvers not able to decompose the graph into smaller components.Each solver was given 10 minutes to solve an instance. For each correctly and timely solved instance the solver received one point, and a ranking for each track was obtained based on points scored on all its instances. Ties were broken by considering total run time on all instances. Additionally, a global ranking of the solvers across all tracks was generated by computing the Borda count of all solvers in all tracks.Results and Concluding RemarksThe obtained rankings for all 16 tracks can be found on the competition website.4 The global ranking identified the following top three solvers: (1) CoQuiAAS, (2) ArgSemSAT, and (3) LabSATSolver. Another solver, Cegartix, participated in only three tracks (SE‐PR, EE‐PR, DS‐PR), but came top in all of these. It is interesting to note that these four solvers are based on SAT‐solving techniques. Additionally, an answer set programming‐based solver (ASPARTIX‐D) came first in the four tracks related to the stable semantics; there is a strong relationship between these semantics and the answer set semantics, which probably explains its strength in these tracks. Information on the solvers and their authors can also be found on the home page of the competition.Given the success of the competition, a second iteration will take place in 2017 with an extended number of tracks.Notes1. argumentationcompetition.org.2. S ⊆ A defends a if ∀bRA, ∃c ∈ S s.t. cRB, that is, all attackers of a are counterattacked by S.3. See also F. Cerutti, N. Oren, H. Strass, M. Thimm, and M. Vallati, M. 2015: The First International Competition on Computational Models of Argumentation (ICCMA15): Supplementary notes on probo (argumentationcompetition.org/2015/iccma15notes_v3.pdf)4. argumentationcompetition.org/2015/results.html.ReferencesBesnard, P., and Hunter, A. 2007. Elements of Argumentation. Cambridge, MA: The MIT Press.Cerutti, F., Oren, N., Strass, H., Thimm, M.; and Vallati, M. 2014. A Benchmark Framework for a Computational Argumentation Competition. In Proceedings of the 5th International Conference on Computational Models of Argument, 459–460. Amsterdam: IOS Press.Dung, P. M. 1995. On the Acceptability of Arguments and Its Fundamental Role in Nonmonotonic Reasoning, Logic Programming, and n‐Person Games. Artificial Intelligence 77(2): 321–357. dx.doi.org/10.1016/0004‐3702(94)00041‐XDunne, P. E., and Wooldridge, M. 2009. Complexity of Abstract Argumentation. In Argumentation in AI, ed. I. Rahwan and G. Simari, chapter 5, 85–104. Berlin: Springer‐Verlag. dx.doi.org/10.1007/978‐0‐387‐98197‐0_5Simari, G. 1992. A Mathematical Treatment of Defeasible Reasoning and Its implementation. Artificial Intelligence 53(2–3): 125–157. dx.doi.org/10.1016/0004‐3702(92)90069‐AToniolo, A., Norman, T. J., Etuk, A., Cerutti, F., Ouyang, R. W., Srivastava, M., Oren, N., Dropps, T., Allen, J. A.; and Sullivan, P. 2015. Agent Support to Reasoning with Different Types of Evidence in Intelligence Analysis. In Proceedings of the 14th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2015), 781–789. Richland, SC: International Foundation for Autonomous Agents and Multiagent Systems.Walton, D. N., Reed, C.; and Macagno, F. 2008. Argumentation Schemes. New York: Cambridge University Press. dx.doi.org/10.1017/CBO9780511802034Williams, M., Liu, Z. W., Hunter, A.; and Macbeth, F. 2015. An Updated Systematic Review of Lung Chemo‐Radiotherapy Using a New Evidence Aggregation Method. Lung Cancer (Amsterdam, Netherlands) 87(3): 290–5. dx.doi.org/10.1016/j.lungcan.2014.12.004 http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png AI Magazine Wiley

Summary Report of the First International Competition on Computational Models of Argumentation

Download PDF
3 pages

Loading next page...
 
/lp/wiley/summary-report-of-the-first-international-competition-on-computational-v0taTifTov

References (12)

Publisher
Wiley
Copyright
© Association for the Advancement of Artificial Intelligence
ISSN
0738-4602
eISSN
2371-9621
DOI
10.1609/aimag.v37i1.2640
Publisher site
See Article on Publisher Site

Abstract

Computational models of argumentation are an active research discipline within artificial intelligence that has grown since the beginning of the 1990s (Dung 1995). While still a young field when compared to areas such as SAT solving and logic programming, the argumentation community is very active, with a conference series (COMMA, which began in 2006) and a variety of workshops and special issues of journals. Argumentation has also worked its way into a variety of applications. For example, Williams et al. (2015) described how argumentation techniques are used for recommending cancer treatments, while Toniolo et al. (2015) detail how argumentation‐based techniques can support critical thinking and collaborative scientific inquiry or intelligence analysis.Many of the problems that argumentation deals with are computationally difficult, and applications utilizing argumentation therefore require efficient solvers. To encourage this line of research, we organised the First International Competition on Computational Models of Argumentation (ICCMA), with the intention of assessing and promoting state‐of‐the‐art solvers for abstract argumentation problems, and to identify families of challenging benchmarks for such solvers.The objective of ICCMA'15 is to allow researchers to compare the performance of different solvers systematically on common benchmarks and rules. Moreover, as witnessed by competitions in other AI disciplines such as planning and SAT solving, we see ICCMA as a new pillar of the community, which provides information and insights on the current state of the art and highlights future challenges and developments.This report summarizes the first ICCMA held in 2015 (ICCMA'15). In this competition, solvers were invited to address standard decision and enumeration problems of abstract argumentation frameworks (Dunne and Wooldridge 2009). Solvers' performance is evaluated based on their time taken to provide a correct solution for a problem; incorrect results were discarded. More information about the competition, including complete results and benchmarks, can be found on the ICCMA website.1TracksIn abstract argumentation (Dung 1995), a directed graph (A, R) is used as knowledge representation formalism, where the set of nodes A are identified with the arguments under consideration and R represents a conflict‐relation between arguments, that is, aRb for a, b ∈ A if a is a counterargument for b. The framework is abstract because the content of the arguments is left unspecified. They could, for example, consist of a chain of logical deductions from logic programming with defeasible rules (Simari 1992); a proof for a theorem in classical logic (Besnard and Hunter 2007); or an informal presumptive reason in favour of some conclusion (Walton, Reed, and Macagno 2008). The notion of conflict then depends on the chosen formalization. Irrespective of the precise formalization used, one can identify a subset of arguments that can be collectively accepted given inter‐argument conflicts. Such a subset is referred to as an extension, and (Dung 1995) defined four commonly used argumentation semantics — namely the complete (CO), preferred (PR), grounded (GR), and stable (ST) semantics — each of which defines an extension differently. More precisely, a complete extension is a set of arguments that do not attack each other,2 and in which arguments defend each other; a preferred extension is a maximal (with regard to set inclusion) complete extension; the grounded extension is the minimal (with regard to set inclusion) complete extension; and a stable extension is a complete extension such that each argument not in the extension is attacked by at least one argument within the extension.The competition was organized around four computational tasks of abstract argumentation: (1) Given an abstract argumentation framework, determine some extension (SE). (2) Given an abstract argumentation framework, determine all extensions (EE). (3) Given an abstract argumentation framework and some argument, decide whether the given argument is contained in some extension (DC). (4) Given an abstract argumentation framework and some argument, decide whether the given argument is contained in all extensions (DS).Combining these four different tasks with the four semantics discussed above yields a total of 16 tracks that constituted ICCMA'15. Each submitted solver was free to support any number of these tracks.ParticipantsThe competition received 18 solvers from research groups in Austria, China, Cyprus, Finland, France, Germany, Italy, Romania, and UK, of which 8 were submitted to all tracks. The solvers used a variety of approaches and programming languages to solve the competition tasks. In particular, 5 solvers were based on transformations of argumentation problems to SAT, 3 on transformations to ASP, 2 on CSP, and 8 were built on tailor‐made algorithms. Seven solvers were implemented in C/C++, 4 in Java, 2 used shell‐scripts for translations to other formalisms, and the remaining solvers were implemented in Haskell, Lisp, Prolog, Python, and Go.All participants were required to submit the source code of their solver, which was made freely available after the competition, to foster independent evaluation and exploitation in research or real‐world scenarios, and to allow for further refinements. Submitted solvers were required to support the probo (Cerutti et al. 2014)3 command‐line interface, which was specifically designed for running and comparing solvers within ICCMA.Performance EvaluationEach solver was evaluated over N different argumentation graph instances within each track (N = 192 for SE and EE, and 576 for DC and DS). Instances were generated with the intention of being challenging — one group of instances was generated so as to contain a large grounded extension and few extensions in the other semantics. This group's graphs were large (1224 to 9473 arguments), and challenged solvers that scaled poorly (that is, those that used combinatorial approaches for computing extensions). A second group of instances was smaller (141 to 400 arguments), but had a rich structure of stable, preferred, and complete extensions (up to 159 complete extensions for the largest graphs) and thus provided combinatorial challenges for solvers relying on simple search‐based algorithms. A final group contained medium‐sized graphs (185 to 996 arguments) and featured many strongly connected components with many extensions. This group was particularly challenging for solvers not able to decompose the graph into smaller components.Each solver was given 10 minutes to solve an instance. For each correctly and timely solved instance the solver received one point, and a ranking for each track was obtained based on points scored on all its instances. Ties were broken by considering total run time on all instances. Additionally, a global ranking of the solvers across all tracks was generated by computing the Borda count of all solvers in all tracks.Results and Concluding RemarksThe obtained rankings for all 16 tracks can be found on the competition website.4 The global ranking identified the following top three solvers: (1) CoQuiAAS, (2) ArgSemSAT, and (3) LabSATSolver. Another solver, Cegartix, participated in only three tracks (SE‐PR, EE‐PR, DS‐PR), but came top in all of these. It is interesting to note that these four solvers are based on SAT‐solving techniques. Additionally, an answer set programming‐based solver (ASPARTIX‐D) came first in the four tracks related to the stable semantics; there is a strong relationship between these semantics and the answer set semantics, which probably explains its strength in these tracks. Information on the solvers and their authors can also be found on the home page of the competition.Given the success of the competition, a second iteration will take place in 2017 with an extended number of tracks.Notes1. argumentationcompetition.org.2. S ⊆ A defends a if ∀bRA, ∃c ∈ S s.t. cRB, that is, all attackers of a are counterattacked by S.3. See also F. Cerutti, N. Oren, H. Strass, M. Thimm, and M. Vallati, M. 2015: The First International Competition on Computational Models of Argumentation (ICCMA15): Supplementary notes on probo (argumentationcompetition.org/2015/iccma15notes_v3.pdf)4. argumentationcompetition.org/2015/results.html.ReferencesBesnard, P., and Hunter, A. 2007. Elements of Argumentation. Cambridge, MA: The MIT Press.Cerutti, F., Oren, N., Strass, H., Thimm, M.; and Vallati, M. 2014. A Benchmark Framework for a Computational Argumentation Competition. In Proceedings of the 5th International Conference on Computational Models of Argument, 459–460. Amsterdam: IOS Press.Dung, P. M. 1995. On the Acceptability of Arguments and Its Fundamental Role in Nonmonotonic Reasoning, Logic Programming, and n‐Person Games. Artificial Intelligence 77(2): 321–357. dx.doi.org/10.1016/0004‐3702(94)00041‐XDunne, P. E., and Wooldridge, M. 2009. Complexity of Abstract Argumentation. In Argumentation in AI, ed. I. Rahwan and G. Simari, chapter 5, 85–104. Berlin: Springer‐Verlag. dx.doi.org/10.1007/978‐0‐387‐98197‐0_5Simari, G. 1992. A Mathematical Treatment of Defeasible Reasoning and Its implementation. Artificial Intelligence 53(2–3): 125–157. dx.doi.org/10.1016/0004‐3702(92)90069‐AToniolo, A., Norman, T. J., Etuk, A., Cerutti, F., Ouyang, R. W., Srivastava, M., Oren, N., Dropps, T., Allen, J. A.; and Sullivan, P. 2015. Agent Support to Reasoning with Different Types of Evidence in Intelligence Analysis. In Proceedings of the 14th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2015), 781–789. Richland, SC: International Foundation for Autonomous Agents and Multiagent Systems.Walton, D. N., Reed, C.; and Macagno, F. 2008. Argumentation Schemes. New York: Cambridge University Press. dx.doi.org/10.1017/CBO9780511802034Williams, M., Liu, Z. W., Hunter, A.; and Macbeth, F. 2015. An Updated Systematic Review of Lung Chemo‐Radiotherapy Using a New Evidence Aggregation Method. Lung Cancer (Amsterdam, Netherlands) 87(3): 290–5. dx.doi.org/10.1016/j.lungcan.2014.12.004

Journal

AI Magazine – Wiley

Published: Mar 1, 2016

There are no references for this article.