Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 14-Day Trial for You and Your Team.

Learn More →

Design preferences and cognitive styles: experimentation by automated website synthesis

Design preferences and cognitive styles: experimentation by automated website synthesis Background: This article aims to demonstrate computational synthesis of Web-based experiments in undertaking experimentation on relationships among the participants’ design preference, rationale, and cognitive test performance. The exemplified experiments were computationally synthesised, including the websites as materials, experiment protocols as methods, and cognitive tests as protocol modules. This work also exemplifies the use of a website synthesiser as an essential instrument enabling the participants to explore different possible designs, which were generated on the fly, before selection of preferred designs. Methods: The participants were given interactive tree and table generators so that they could explore some different ways of presenting causality information in tables and trees as the visualisation formats. The participants gave their preference ratings for the available designs, as well as their rationale (criteria) for their design decisions. The participants were also asked to take four cognitive tests, which focus on the aspects of visualisation and analogy-making. The relationships among preference ratings, rationale, and the results of cognitive tests were analysed by conservative non-parametric statistics including Wilcoxon test, Krustal-Wallis test, and Kendall correlation. Results: In the test, 41 of the total 64 participants preferred graphical (tree-form) to tabular presentation. Despite the popular preference for graphical presentation, the given tabular presentation was generally rated to be easier than graphical presentation to interpret, especially by those who were scored lower in the visualization and analogy-making tests. Conclusions: This piece of evidence helps generate a hypothesis that design preferences are related to specific cognitive abilities. Without the use of computational synthesis, the experiment setup and scientific results would be impractical to obtain. Background our websites. It would be useful to conduct experiments The manner of external representation (or presentation) quickly to compare different models of interpretation of could affect our way of working with the internal repre- information of a specific domain in particular cases. sentation (mentally) and our understanding of the infor- There may be certain styles of presentation or naviga- mation [1], e.g. in cockpit information displays for aviation tion that are generally demanded by users and can [2], but few results on graphical external representation either hinder or support users’ ability to interpret the can be generalised [3]. information. One problem affecting all websites is that there is no Tabular and graphical representations are common in reliable, general and abstract method for predicting the constructing visual arguments [4] and presenting rela- effect of presentation rhetorics and modality on under- tional data (especially quantitative data) [5]. Visualisation standing of the information. To improve knowledge of aviation accident events generally use causal trees to communication, we should investigate how sensitive represent the causal relations but there are few empirical people might be to differences in the way we construct studies on both preference and perception of causality visualisation. Specifically, we investigate users’ preferences * Correspondence: siu@inf.ed.ac.uk; dr@inf.ed.ac.uk for information visualisation styles and their perception of School of Informatics, University of Edinburgh, Edinburgh EH8 9AB, UK causality as required by aviation accident reporting. As the Full list of author information is available at the end of the article Leung et al. Automated Experimentation 2012, 4:2 Page 2 of 8 http://www.biomedcentral.com/1759-4499/4/2 Web is one of the main channels for publishing informa- the experiment participants were asked (unprompted) tion of aviation accidents, it is desirable to know about for their rationale before seeing these seven rationales how people would prefer the causal relations in accident and then they were asked (prompted) to identify if any events to be presented in a website and how they perceive of these rationales were similar to their own rationales. this causality. The user preference data are useful in the They were also asked if any of their rationales was not design and re-design of websites. To elicit preferences covered by these seven rationales. Software designs should reduce users’ cognitive load from people, we provide multiple designs for selection and [13]. We hypothesise that participants prefer one design study the rationale of their design decisions. Automated to other designs partially because the preferred design website synthesis saves time and effort in building websites for such designs. Few models and theories are available to suits their cognitive abilities. If this is true, the cognitive address computational website design. However, if we abilities of the participants should be related to their pre- view websites as a form of information visualisation, we ferences. The relationships among preferences of trees or can borrow some findings from automated diagram design tables, the cognitive test results of the participants, and [6] to serve as our experiment hypotheses. Some systems rationale for their preferences were studied in this experi- for automated diagram design have incorporated text to ment. As it is impossible to test numerous cognitive fac- enhance user understanding of graphical visualisation [7]. tors in a single experiment, the participants were only This kind of multimodal visualisation should be applicable tested on cognitive styles/abilities of visualisation and to website design. Expressiveness and effectiveness of gra- analogy-making, which we guessed were related to visual phical languages as proposed by Mackinlay [8] have been representations. influential to diagram visualisation models, including The main objectives of this experiment are as follows: source information characteristics [9], user-defined task specification [10], and user-defined layout preferences [7] � To see if there is any different preference for tables for automated diagram design. or trees in representing the given information; In general, we would like to see if people would prefer � To see if different preferences are based on differ- different representations to display the same information. ent priority in criteria/rationale; In particular, this study aims to elicit preferences of � To see if the preferences are related to the cogni- designers (and users) about visualisation patterns, particu- tive test results; and larly the preferences for tables and trees in visualising caus- � To see if the importance ratings of design criteria/ rationale are related to cognitive test results. ality information. In this case, we select trees and tables as the options for selection by users. Tree representations are commonly used to graphically represent causality in Methods printed documents. The causal relations are normally Participants represented by arrows or lines connecting causes and Sixty four students from the University of Edinburgh par- effects. ticipated in the experiment and received cash (GBP 10) If people do prefer a representation, it would be inter- as a reward. They were randomly assigned to one of the esting to see what rationale or criteria contribute to two groups according to a pre-generated random their preferences. We categorised common rationale/cri- sequence. Each group had 32 participants. The treat- teria mentioned in website design textbooks [11,12]: ments of these two groups differ in the order of using table and tree design generators (or simply called � easy to learn: the users do not need much time designers). All of the participants had the computer skills and effort to understand how it works; for browsing websites. The experiment took about � more visual: the users can understand through gra- 1.5 hours for each participant. No strict time limit was phical illustrations; enforced for tasks except cognitive tests, for which data � more informative: the users can know more details; were automatically collected. � more scalable: fewer changes are needed to handle more massive information; Web pages � more features represented: less characteristics Computational website synthesis provided basic facilities (important information) are left out; for generating websites and their functional components � more suggestive: the users can understand without such as menus and breadcrumbs. We mapped the infor- much guessing; and mation content items to appropriate components in spe- � more flexible: suitable for use in different situations. cifications. Presentation of accident event (content) information in visualisation formats (tabular cells or gra- As these seven rationales may not cover all possible phical trees) required mappings of attributes between rationales that are crucial to any particular preference, the content information and visualisation formats. One Leung et al. Automated Experimentation 2012, 4:2 Page 3 of 8 http://www.biomedcentral.com/1759-4499/4/2 of the major factors affecting the mapping decision was dropping the representational images (one at a time) in designers’ (and users’) preferences. a position relative to any specific tiny (1 × 1 pixels) Our approach to eliciting the designers’ preference is image dot (invisible). A drop of the representational to let the designers explore the available options and image was successful only if it was dropped within an then decide which option is the one they prefer. For area of a predefined radius to the anchor image. If a doing this, we gave the information of the customised representational image was dropped outside the speci- images (representing specific pieces of information) to a fied area, then it would return to its original position. home-made drag-and-drop web component as para- The repositioning would be activated whenever there was a change (e.g. change of window size which affects meters so that the users could design their preferred tables (Figure 1) and trees (Figure 2) by dragging and the relative positions of images). A piece of JavaScript Figure 1 Exploring options for tabular representation. Leung et al. Automated Experimentation 2012, 4:2 Page 4 of 8 http://www.biomedcentral.com/1759-4499/4/2 Figure 2 Exploring options for tree form representation. code was generated to feed these parameters to a Java- specifications for defining variables and variable types (e. Script component for image drag-and-drop manage- g. multi-answers or long text) and special webpage ele- ment. This approach was simpler (lightweight) than ments. Subsequent minor modifications to the generated many other approaches which used sophisticated (heavy- questionnaire forms were only for cosmetic purposes. weight) Java Applets or Flash objects to provide drag- and-drop functions. Tree and table generators The trees and tables were generated on-the-fly and Data collection viewable in a separate window of the web browser All input from the participants were collected by stan- together with other basic website navigation facilities, e. dard HTML forms and CGI (common gateway inter- g. menus and hyperlinks. The tables were generated as face) scripts, which were generated from simpler standard HTML tables while the trees were generated as Leung et al. Automated Experimentation 2012, 4:2 Page 5 of 8 http://www.biomedcentral.com/1759-4499/4/2 DOT diagram specifications for final image rendering by Results GraphViz on the server side before sending to the client Background of participants side. There was no significant difference in any background variable between two groups of participants as measured Tasks by the pre-experiment questionnaire and analysed by Each participant filled in a pre-experiment questionnaire Student’s t-test and Wilcoxon test. which collected background information such as their familiarity with the Web and aviation operations. Then Preferences the participants used table and tree designers to express Preferences were obtained from the preference ratings their preferences between tables and trees in represent- about participants’ preferences between tables and trees ing a given structure of causality information. The order as the representation of the given information. The par- of using table and tree designers were randomly ticipants rated the strength of their preferences as: assigned. Participants assigned to Group A used the tree designer first and then the table designer while those in � strong table preference, Group B used the table designer first and then the tree � moderate table preference, designer. In the designers, the participants used a custo- � marginal table preference, mised drag-and-drop facility to design their preferred � marginal tree preference, table and tree patterns. The participants submitted their � moderate tree preference, and preferred visualisation patterns astablesor trees.Their � strong tree preference. rationale for their preferred visualisation pattern were collected by a two-part post-experiment questionnaire. Based on the types of preferred representations, binary Part A of the questionnaire collected their rationale as classification gives two categories of participants: table the open-ended answers and their preference ratings of preferrers and tree preferrers. The number of participants table and tree visualisation. Part B of the questionnaire under these classifications of preferences were counted as collected their ratings of the importance against seven shown in Table 1. There were 23 participants preferred common preference criteria/rationale. The participants tables and 41 participants preferred trees. also took timed cognitive tests, including two (paper folding and surface development) visualisation tests and Rationales two (visual and verbal) analogy-making tests. The visua- Seven rationales/criteria were given to the participants to lisation tests were licensed from the Educational Testing rank using the numbers 1-7. Rank 1 is the most impor- Services of the USA, as provided in a kit of factor-refer- tant rationale or criteria in their preference decision. enced cognitive tests [14]. The visual analogy puzzles Rank 7 is the least important one. The summary statistics were the same as those selected by Thomas Evans [15] including median, mean, and standard deviation (SD) of in his study of visual analogy problems. The verbal ana- the overall ranking on the common design criteria/ratio- logy test was the sample questions of the Miller Analogy nale is shown in Table 2. This overall result indicates Test (MAT). that the participants found “more informative” and “easier to learn” as the most important two rationales for Data analysis their preference decisions. “More Visual” and “more sug- Variables included (1) preferences of tables or trees, (2) gestive” were moderately important. importance ratings of preference criteria/rationale, and (3) results of cognitive tests. Data were presented in Cognitive tests medians, means, and standard deviations (SD) for both The summary statistics of the test results are as shown parametric and non-parametric analyses, although only in Table 3. The test of surface development visualisation conservative non-parametric Wilcoxon test, Krustal- Wallis test, and Kendall rank correlation test were Table 1 Preference of table and tree representations reported in this article. Both parametric (e.g. t-test and Preferences Class No. of participants ANOVA) and non-parametric tests were run using R Strong table Table 3 statistical software [16] and its μStat package. In multi- Moderate table Table 14 ple comparison, P values was adjusted by Bonferroni Marginal table Table 6 correction. P values less than 0.05 were considered sta- Marginal tree Tree 6 tistically significant. In statistical tables, single asterisks Moderate tree Tree 15 (*) indicated P < 0.05 and double asterisks (**) indicated Strong tree Tree 20 P < 0.01. Leung et al. Automated Experimentation 2012, 4:2 Page 6 of 8 http://www.biomedcentral.com/1759-4499/4/2 Table 2 Rationales for preferences Table 4 Relationship between preferences and rationales Rationale Median Mean SD Rationale Original preferences Binary preferences 2 2 c P c P Easier to learn 3 3.09 2.01 More visual 4 4.06 1.78 Easier to learn 11.88 0.037 * 9.22 0.002 ** More informative 2 2.78 1.86 More visual 7.37 0.195 1.60 0.207 More scalable 5 4.90 1.42 More informative 5.10 0.404 1.25 0.264 More features 5 4.86 1.90 More scalable 5.07 0.408 1.48 0.225 More suggestive 4 3.98 2.00 More features 6.90 0.228 3.84 0.050 More flexible 5 4.86 1.78 More suggestive 7.37 0.194 5.69 0.017 * More flexible 8.57 0.127 0.23 0.633 seemed to be difficult to some participants. The median of the result was 0 and its standard deviation was high. statistically significant rank correlations coefficient (τ) ranged between around 0.201 - 0.263, which are only Preferences and rationales low to moderate in strength. As shown in Table 4 the differences in the rankings of the rationale “easier to learn” among different partici- Discussion pants with different strengths of preferences were found This study used website synthesis to construct an experi- statistically significant by using the Krustal-Wallis test. mental apparatus for Lab-on-the-Web. Using this The difference between table preferrers’ and tree prefer- approach we found a significant relationship among the rers’ rankings of the rationale “easier to learn” was highly participants’ preferences, rationale, and cognitive ability statistically significant as indicated by the Wilcoxon test. test results. Most (41 out of 64) participants claimed The significant difference related to the rationale “more themselves to be tree preferrers. The other participants suggestive” was only observed in binary preferences, not (23 out of 64) said they preferred tables. Both table pre- in the original classification of preferences and their ferrers and tree preferrers found their preferred represen- strengths. tations (tables or trees) more informative, without Table 5 showed the median rankings of rationale for significant difference in rankings of this rationale. As the preferences. Table preferrers found the criterion “easier task given to the participants is to represent information, to learn” to the most important rationale while the tree it is not surprising that the rationale “more informative” preferrers did not. was one of the most important rationale. In further stu- dies, it would be interesting to test whether the partici- Preferences and cognitive tests pants find the same rationale justifiable for their The relationship between the results of visual analogy preferences when they are given different goals or under test and preferences (and binary preferences) was highly different conditions. This might give us more insights significant (P < 0.01) according to the Kruskal-Wallis into how different preferrers perceive information. test (and Wilcoxon test), as shown in Table 6. It was To table preferrers, the rationale “easier to learn” was highly significant (P < 0.01) that the participants who more important than “more informative”.The rationale “easier to learn” was ranked as the most important by 13 performed better in the visual analogy test preferred trees (Table 7). out of 23 table preferrers, but not so important by tree preferrers. Tree preferrers ranked the rationale “more Rationale and cognitive tests suggestive” significantly higher thanthetablepreferrers The correlation between the importance rankings of did. To table preferrers, tables seemed to be easier to rationale ("easier to learn” and and the result rankings of cognitive tests are statistically significant (Table 8) Table 5 Ranking of rationales according to Kendall’s rank correlation test. The Rationale Table Preferers Tree Preferers Median Mean SD Median Mean SD Easier to learn 1 2.13 1.74 4 3.63 1.96 Table 3 Results of cognitive tests More visual 3 3.74 1.79 4 4.24 1.79 Test Median Mean SD More informative 3 3.04 1.99 2 2.63 1.80 Paper folding 5 5.33 2.24 More scalable 4 4.65 1.27 5 5.00 1.50 Surface development 0 8.59 11.14 More features 6 5.65 0.98 5 4.42 2.14 Visual analogy 17 15.89 3.88 More suggestive 5 4.78 1.86 3 3.54 1.94 Verbal analogy 8 9.97 7.49 More flexible 5 5.04 1.64 5 4.76 1.87 Leung et al. Automated Experimentation 2012, 4:2 Page 7 of 8 http://www.biomedcentral.com/1759-4499/4/2 Table 6 Relationship between cognitive test results and Table 8 Correlation between rationales and cognitive test preferences results Test Original preference Binary preference Tests Easier to learn More suggestive 2 2 c P c P τ P τ P Paper folding 9.16 0.103 5.10 0.024 * Paper folding 0.203 0.039 * -0.050 0.608 Surface development 6.09 0.298 0.15 0.697 Surface development 0.173 0.088 -0.070 0.484 Visual analogy 16.36 0.006 ** 13.50 0.000 ** Visual analogy 0.263 0.008 ** -0.201 0.040 * Verbal analogy 3.95 0.557 1.14 0.286 Verbal analogy 0.006 0.948 0.058 0.532 learn than trees. To tree preferrers, trees were more sug- low to moderate strengths of correlation in these results, it gestive than tables. These discrepancies in rationale rank- is probable that other factors (and other cognitive factors) ing indicate a perception difference between different may be also relevant to the participants’ preferences. preferrers in perceiving the given tables and trees. Further studies are required to delineate these Tree preferrers performed better than table preferrers in relationships. some tests of cognitive factors, particularly the paper fold- This experiment also demonstrated the technological ing visualisation test and visual analogy test. There was no significance of computational synthesis in enabling scienti- significant difference between table preferrers and tree fic experiments. This experiment re-used a website synthe- preferrers in their performance in other cognitive tests siser to generate websites on the fly so that the designers/ including the surface development visualisation test and users can explore the design space. Without using a web- the verbal analogy test. It is plausible that the interpreta- site synthesiser, this experiment would not be possible. tion of tree representations requires specific cognitive cap- Other than computational synthesis, the most relevant abilities such as visualisation and visual analogy-making; tool is Web content management system but it does not thus, those who do not feel comfortable with these tasks meet the requirement of this experiment. Apart from its would prefer tables and highly rank the rationale “easier to high cost, there is no Web content management system so learn” for their table preference. flexible as our our website synthesiser in accepting infor- Among all participants, there is a statistically significant mation mappings. This makes Web content management low-to-moderate rank correlation between the rankings of systems inapplicable to our experiment. Thus, computa- the rationale “easier to learn” and the results of visual ana- tional synthesis is the only available solution for this logy test and paper folding visualisation test. The low-to- experiment although computational synthesis following moderate rank correlation indicates that the participants definite patterns would limit the variability of graphical who performed better in such two cognitive tests ranked presentation. To enrich the variability and functionality of the rationale “easier to learn” to be less important. At graphical presentation, we would consider using web similar correlation strength, the participants who per- design patterns available in Web 2.0 and HTML5 to formed better in visual analogy test ranked the rationale enhance further studies. “more suggestive” to be more important. It seems that the visualisation and visual analogy-making abilities of the par- Conclusions ticipants might play a role in their preferences for tables The experiment reported in this article found significant and trees. Possibly (although we cannot prove this), table relationships between the participants’ design preference, preferrers lack sufficient cognitive capability to interpret rationale, and cognitive test performance. This work also graphical representations like trees; thus, they prefer tables exemplifies the use of a website synthesiser as an essen- as they are easier to learn. Tree preferrers would feel more tial instrument enabling the participants to explore dif- comfortable in making visual analogy and find graphical ferent possible designs, which were generated on the fly, representations like trees suggestive. As indicated by the before they selected their preferred designs. In the tested sample, more people prefer graphical to tabular presenta- tion. Despite the high preference for graphical presenta- Table 7 Comparison of cognitive test results between tion, the given tabular presentation was generally rated to different groups of preferrers be easier than graphical presentation to interpret, espe- Test Table Preferers Tree Preferers cially for those who score below average in the visualisa- Median Mean SD Median Mean SD tion and analogy-making tests. This piece of evidence Paper folding* 4 4.57 2.04 5 5.76 2.26 helps generate a hypothesis that design preferences are Surface development 0 7.61 11.16 0 9.15 11.22 related to specific cognitive abilities. Without the use of Visual analogy** 16 13.57 5.66 17 17.2 1.14 computational synthesis, the experiment setup and scien- Verbal analogy 9 9.96 6.47 7 9.98 8.08 tific results would be impractical to obtain. Leung et al. Automated Experimentation 2012, 4:2 Page 8 of 8 http://www.biomedcentral.com/1759-4499/4/2 Acknowledgements This study was partially supported by the UK EPSRC grant GR/M98302 for informatics research into communicating knowledge about accidents from synthesised websites. Competing interests The authors declare that they have no competing interests. Author details School of Informatics, University of Edinburgh, Edinburgh EH8 9AB, UK. State Key Lab QRCM and ICMS, University of Macau, Taipa, Macao, China. Department of Computing Science, University of Glasgow, Glasgow G12 8QQ, UK. Authors’ contributions DR and SL conceived this study. JL and CJ gave technical advice about web design and contribute knowledge in aviation accident domains. SL conducted the experiment and drafted the manuscript. DR and JL revised the manuscript. All authors read and approved the final manuscript. Published: 28 June 2012 References 1. Zhang J: The nature of external representations in problem solving. Cognitive Science 1997, 21(2):179-217. 2. Zhang J: Distributed representations as a principle for the analysis of cockpit information displays. The International Journal of Aviation Psychology 1997, 7(2):105-121. 3. Scaife M, Rogers Y: External cognition: how do graphical representations work? International Journal of Human-Computer Studies 1996, 45:185-213. 4. Oestermeier U, Hesse F: Verbal and visual causal arguments. Cognition 2000, 75:65-104. 5. Zhang J: A representational analysis of relational information displays. International Journal of Human-Computer Studies 1996, 45:59-74. 6. Kamps T: Diagram Design Springer-Verlag; 1999. 7. Mittal V, Roth S, Moore J, Mattis J, Carenini G: Generating explanatory captions for information graphics. Proceeedings of the International Joint Conference on Artificial Intelligence Montreal, Canada; 1996. 8. Mackinlay J: Automatic Design of Graphical Presentation. Phd thesis Computer Science Department, Stanford University; 1996. 9. Roth S, Mattis J: Data characterization for graphic presentation. Proceedings of the onference on Human Factors in Computing Systems (CHI) 10. Casner S: A task-analytic approach to the automated design of graphic presentations. CM Transactions on Graphics 1991, 10(2):111-151. 11. Krug S: Don’t Make Me Think: A Common Sense Approach to Web Usability New Riders; 2000. 12. Nielsen J: Designing Web Usability New Riders; 2000. 13. Detienne F: Software Design - Cognitive Aspects Springer-Verlag; 2002. 14. Ekstrom R, French J, Harmon H: Manual for kit of factor-referenced cognitive tests Education Testing Service; 1976. 15. Evans T: A heuristic program to solve geometric-analogy problems. Readings in computer vision Morgan Kaufmann; 1987. 16. R Development Core Team: R: A Language and Environment for Statistical Computing. Tech rep, R Foundation for Statistical Computing 2012 [http://www.r-project.org]. doi:10.1186/1759-4499-4-2 Submit your next manuscript to BioMed Central Cite this article as: Leung et al.: Design preferences and cognitive styles: experimentation by automated website synthesis. Automated and take full advantage of: Experimentation 2012 4:2. • Convenient online submission • Thorough peer review • No space constraints or color figure charges • Immediate publication on acceptance • Inclusion in PubMed, CAS, Scopus and Google Scholar • Research which is freely available for redistribution Submit your manuscript at www.biomedcentral.com/submit http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Automated Experimentation Springer Journals

Design preferences and cognitive styles: experimentation by automated website synthesis

Loading next page...
 
/lp/springer-journals/design-preferences-and-cognitive-styles-experimentation-by-automated-rEX8cJHjpg
Publisher
Springer Journals
Copyright
Copyright © 2012 by Leung et al; licensee BioMed Central Ltd.
Subject
Life Sciences; Computer Appl. in Life Sciences; Laboratory Medicine; Computational Biology/Bioinformatics
ISSN
1759-4499
eISSN
1759-4499
DOI
10.1186/1759-4499-4-2
pmid
22748000
Publisher site
See Article on Publisher Site

Abstract

Background: This article aims to demonstrate computational synthesis of Web-based experiments in undertaking experimentation on relationships among the participants’ design preference, rationale, and cognitive test performance. The exemplified experiments were computationally synthesised, including the websites as materials, experiment protocols as methods, and cognitive tests as protocol modules. This work also exemplifies the use of a website synthesiser as an essential instrument enabling the participants to explore different possible designs, which were generated on the fly, before selection of preferred designs. Methods: The participants were given interactive tree and table generators so that they could explore some different ways of presenting causality information in tables and trees as the visualisation formats. The participants gave their preference ratings for the available designs, as well as their rationale (criteria) for their design decisions. The participants were also asked to take four cognitive tests, which focus on the aspects of visualisation and analogy-making. The relationships among preference ratings, rationale, and the results of cognitive tests were analysed by conservative non-parametric statistics including Wilcoxon test, Krustal-Wallis test, and Kendall correlation. Results: In the test, 41 of the total 64 participants preferred graphical (tree-form) to tabular presentation. Despite the popular preference for graphical presentation, the given tabular presentation was generally rated to be easier than graphical presentation to interpret, especially by those who were scored lower in the visualization and analogy-making tests. Conclusions: This piece of evidence helps generate a hypothesis that design preferences are related to specific cognitive abilities. Without the use of computational synthesis, the experiment setup and scientific results would be impractical to obtain. Background our websites. It would be useful to conduct experiments The manner of external representation (or presentation) quickly to compare different models of interpretation of could affect our way of working with the internal repre- information of a specific domain in particular cases. sentation (mentally) and our understanding of the infor- There may be certain styles of presentation or naviga- mation [1], e.g. in cockpit information displays for aviation tion that are generally demanded by users and can [2], but few results on graphical external representation either hinder or support users’ ability to interpret the can be generalised [3]. information. One problem affecting all websites is that there is no Tabular and graphical representations are common in reliable, general and abstract method for predicting the constructing visual arguments [4] and presenting rela- effect of presentation rhetorics and modality on under- tional data (especially quantitative data) [5]. Visualisation standing of the information. To improve knowledge of aviation accident events generally use causal trees to communication, we should investigate how sensitive represent the causal relations but there are few empirical people might be to differences in the way we construct studies on both preference and perception of causality visualisation. Specifically, we investigate users’ preferences * Correspondence: siu@inf.ed.ac.uk; dr@inf.ed.ac.uk for information visualisation styles and their perception of School of Informatics, University of Edinburgh, Edinburgh EH8 9AB, UK causality as required by aviation accident reporting. As the Full list of author information is available at the end of the article Leung et al. Automated Experimentation 2012, 4:2 Page 2 of 8 http://www.biomedcentral.com/1759-4499/4/2 Web is one of the main channels for publishing informa- the experiment participants were asked (unprompted) tion of aviation accidents, it is desirable to know about for their rationale before seeing these seven rationales how people would prefer the causal relations in accident and then they were asked (prompted) to identify if any events to be presented in a website and how they perceive of these rationales were similar to their own rationales. this causality. The user preference data are useful in the They were also asked if any of their rationales was not design and re-design of websites. To elicit preferences covered by these seven rationales. Software designs should reduce users’ cognitive load from people, we provide multiple designs for selection and [13]. We hypothesise that participants prefer one design study the rationale of their design decisions. Automated to other designs partially because the preferred design website synthesis saves time and effort in building websites for such designs. Few models and theories are available to suits their cognitive abilities. If this is true, the cognitive address computational website design. However, if we abilities of the participants should be related to their pre- view websites as a form of information visualisation, we ferences. The relationships among preferences of trees or can borrow some findings from automated diagram design tables, the cognitive test results of the participants, and [6] to serve as our experiment hypotheses. Some systems rationale for their preferences were studied in this experi- for automated diagram design have incorporated text to ment. As it is impossible to test numerous cognitive fac- enhance user understanding of graphical visualisation [7]. tors in a single experiment, the participants were only This kind of multimodal visualisation should be applicable tested on cognitive styles/abilities of visualisation and to website design. Expressiveness and effectiveness of gra- analogy-making, which we guessed were related to visual phical languages as proposed by Mackinlay [8] have been representations. influential to diagram visualisation models, including The main objectives of this experiment are as follows: source information characteristics [9], user-defined task specification [10], and user-defined layout preferences [7] � To see if there is any different preference for tables for automated diagram design. or trees in representing the given information; In general, we would like to see if people would prefer � To see if different preferences are based on differ- different representations to display the same information. ent priority in criteria/rationale; In particular, this study aims to elicit preferences of � To see if the preferences are related to the cogni- designers (and users) about visualisation patterns, particu- tive test results; and larly the preferences for tables and trees in visualising caus- � To see if the importance ratings of design criteria/ rationale are related to cognitive test results. ality information. In this case, we select trees and tables as the options for selection by users. Tree representations are commonly used to graphically represent causality in Methods printed documents. The causal relations are normally Participants represented by arrows or lines connecting causes and Sixty four students from the University of Edinburgh par- effects. ticipated in the experiment and received cash (GBP 10) If people do prefer a representation, it would be inter- as a reward. They were randomly assigned to one of the esting to see what rationale or criteria contribute to two groups according to a pre-generated random their preferences. We categorised common rationale/cri- sequence. Each group had 32 participants. The treat- teria mentioned in website design textbooks [11,12]: ments of these two groups differ in the order of using table and tree design generators (or simply called � easy to learn: the users do not need much time designers). All of the participants had the computer skills and effort to understand how it works; for browsing websites. The experiment took about � more visual: the users can understand through gra- 1.5 hours for each participant. No strict time limit was phical illustrations; enforced for tasks except cognitive tests, for which data � more informative: the users can know more details; were automatically collected. � more scalable: fewer changes are needed to handle more massive information; Web pages � more features represented: less characteristics Computational website synthesis provided basic facilities (important information) are left out; for generating websites and their functional components � more suggestive: the users can understand without such as menus and breadcrumbs. We mapped the infor- much guessing; and mation content items to appropriate components in spe- � more flexible: suitable for use in different situations. cifications. Presentation of accident event (content) information in visualisation formats (tabular cells or gra- As these seven rationales may not cover all possible phical trees) required mappings of attributes between rationales that are crucial to any particular preference, the content information and visualisation formats. One Leung et al. Automated Experimentation 2012, 4:2 Page 3 of 8 http://www.biomedcentral.com/1759-4499/4/2 of the major factors affecting the mapping decision was dropping the representational images (one at a time) in designers’ (and users’) preferences. a position relative to any specific tiny (1 × 1 pixels) Our approach to eliciting the designers’ preference is image dot (invisible). A drop of the representational to let the designers explore the available options and image was successful only if it was dropped within an then decide which option is the one they prefer. For area of a predefined radius to the anchor image. If a doing this, we gave the information of the customised representational image was dropped outside the speci- images (representing specific pieces of information) to a fied area, then it would return to its original position. home-made drag-and-drop web component as para- The repositioning would be activated whenever there was a change (e.g. change of window size which affects meters so that the users could design their preferred tables (Figure 1) and trees (Figure 2) by dragging and the relative positions of images). A piece of JavaScript Figure 1 Exploring options for tabular representation. Leung et al. Automated Experimentation 2012, 4:2 Page 4 of 8 http://www.biomedcentral.com/1759-4499/4/2 Figure 2 Exploring options for tree form representation. code was generated to feed these parameters to a Java- specifications for defining variables and variable types (e. Script component for image drag-and-drop manage- g. multi-answers or long text) and special webpage ele- ment. This approach was simpler (lightweight) than ments. Subsequent minor modifications to the generated many other approaches which used sophisticated (heavy- questionnaire forms were only for cosmetic purposes. weight) Java Applets or Flash objects to provide drag- and-drop functions. Tree and table generators The trees and tables were generated on-the-fly and Data collection viewable in a separate window of the web browser All input from the participants were collected by stan- together with other basic website navigation facilities, e. dard HTML forms and CGI (common gateway inter- g. menus and hyperlinks. The tables were generated as face) scripts, which were generated from simpler standard HTML tables while the trees were generated as Leung et al. Automated Experimentation 2012, 4:2 Page 5 of 8 http://www.biomedcentral.com/1759-4499/4/2 DOT diagram specifications for final image rendering by Results GraphViz on the server side before sending to the client Background of participants side. There was no significant difference in any background variable between two groups of participants as measured Tasks by the pre-experiment questionnaire and analysed by Each participant filled in a pre-experiment questionnaire Student’s t-test and Wilcoxon test. which collected background information such as their familiarity with the Web and aviation operations. Then Preferences the participants used table and tree designers to express Preferences were obtained from the preference ratings their preferences between tables and trees in represent- about participants’ preferences between tables and trees ing a given structure of causality information. The order as the representation of the given information. The par- of using table and tree designers were randomly ticipants rated the strength of their preferences as: assigned. Participants assigned to Group A used the tree designer first and then the table designer while those in � strong table preference, Group B used the table designer first and then the tree � moderate table preference, designer. In the designers, the participants used a custo- � marginal table preference, mised drag-and-drop facility to design their preferred � marginal tree preference, table and tree patterns. The participants submitted their � moderate tree preference, and preferred visualisation patterns astablesor trees.Their � strong tree preference. rationale for their preferred visualisation pattern were collected by a two-part post-experiment questionnaire. Based on the types of preferred representations, binary Part A of the questionnaire collected their rationale as classification gives two categories of participants: table the open-ended answers and their preference ratings of preferrers and tree preferrers. The number of participants table and tree visualisation. Part B of the questionnaire under these classifications of preferences were counted as collected their ratings of the importance against seven shown in Table 1. There were 23 participants preferred common preference criteria/rationale. The participants tables and 41 participants preferred trees. also took timed cognitive tests, including two (paper folding and surface development) visualisation tests and Rationales two (visual and verbal) analogy-making tests. The visua- Seven rationales/criteria were given to the participants to lisation tests were licensed from the Educational Testing rank using the numbers 1-7. Rank 1 is the most impor- Services of the USA, as provided in a kit of factor-refer- tant rationale or criteria in their preference decision. enced cognitive tests [14]. The visual analogy puzzles Rank 7 is the least important one. The summary statistics were the same as those selected by Thomas Evans [15] including median, mean, and standard deviation (SD) of in his study of visual analogy problems. The verbal ana- the overall ranking on the common design criteria/ratio- logy test was the sample questions of the Miller Analogy nale is shown in Table 2. This overall result indicates Test (MAT). that the participants found “more informative” and “easier to learn” as the most important two rationales for Data analysis their preference decisions. “More Visual” and “more sug- Variables included (1) preferences of tables or trees, (2) gestive” were moderately important. importance ratings of preference criteria/rationale, and (3) results of cognitive tests. Data were presented in Cognitive tests medians, means, and standard deviations (SD) for both The summary statistics of the test results are as shown parametric and non-parametric analyses, although only in Table 3. The test of surface development visualisation conservative non-parametric Wilcoxon test, Krustal- Wallis test, and Kendall rank correlation test were Table 1 Preference of table and tree representations reported in this article. Both parametric (e.g. t-test and Preferences Class No. of participants ANOVA) and non-parametric tests were run using R Strong table Table 3 statistical software [16] and its μStat package. In multi- Moderate table Table 14 ple comparison, P values was adjusted by Bonferroni Marginal table Table 6 correction. P values less than 0.05 were considered sta- Marginal tree Tree 6 tistically significant. In statistical tables, single asterisks Moderate tree Tree 15 (*) indicated P < 0.05 and double asterisks (**) indicated Strong tree Tree 20 P < 0.01. Leung et al. Automated Experimentation 2012, 4:2 Page 6 of 8 http://www.biomedcentral.com/1759-4499/4/2 Table 2 Rationales for preferences Table 4 Relationship between preferences and rationales Rationale Median Mean SD Rationale Original preferences Binary preferences 2 2 c P c P Easier to learn 3 3.09 2.01 More visual 4 4.06 1.78 Easier to learn 11.88 0.037 * 9.22 0.002 ** More informative 2 2.78 1.86 More visual 7.37 0.195 1.60 0.207 More scalable 5 4.90 1.42 More informative 5.10 0.404 1.25 0.264 More features 5 4.86 1.90 More scalable 5.07 0.408 1.48 0.225 More suggestive 4 3.98 2.00 More features 6.90 0.228 3.84 0.050 More flexible 5 4.86 1.78 More suggestive 7.37 0.194 5.69 0.017 * More flexible 8.57 0.127 0.23 0.633 seemed to be difficult to some participants. The median of the result was 0 and its standard deviation was high. statistically significant rank correlations coefficient (τ) ranged between around 0.201 - 0.263, which are only Preferences and rationales low to moderate in strength. As shown in Table 4 the differences in the rankings of the rationale “easier to learn” among different partici- Discussion pants with different strengths of preferences were found This study used website synthesis to construct an experi- statistically significant by using the Krustal-Wallis test. mental apparatus for Lab-on-the-Web. Using this The difference between table preferrers’ and tree prefer- approach we found a significant relationship among the rers’ rankings of the rationale “easier to learn” was highly participants’ preferences, rationale, and cognitive ability statistically significant as indicated by the Wilcoxon test. test results. Most (41 out of 64) participants claimed The significant difference related to the rationale “more themselves to be tree preferrers. The other participants suggestive” was only observed in binary preferences, not (23 out of 64) said they preferred tables. Both table pre- in the original classification of preferences and their ferrers and tree preferrers found their preferred represen- strengths. tations (tables or trees) more informative, without Table 5 showed the median rankings of rationale for significant difference in rankings of this rationale. As the preferences. Table preferrers found the criterion “easier task given to the participants is to represent information, to learn” to the most important rationale while the tree it is not surprising that the rationale “more informative” preferrers did not. was one of the most important rationale. In further stu- dies, it would be interesting to test whether the partici- Preferences and cognitive tests pants find the same rationale justifiable for their The relationship between the results of visual analogy preferences when they are given different goals or under test and preferences (and binary preferences) was highly different conditions. This might give us more insights significant (P < 0.01) according to the Kruskal-Wallis into how different preferrers perceive information. test (and Wilcoxon test), as shown in Table 6. It was To table preferrers, the rationale “easier to learn” was highly significant (P < 0.01) that the participants who more important than “more informative”.The rationale “easier to learn” was ranked as the most important by 13 performed better in the visual analogy test preferred trees (Table 7). out of 23 table preferrers, but not so important by tree preferrers. Tree preferrers ranked the rationale “more Rationale and cognitive tests suggestive” significantly higher thanthetablepreferrers The correlation between the importance rankings of did. To table preferrers, tables seemed to be easier to rationale ("easier to learn” and and the result rankings of cognitive tests are statistically significant (Table 8) Table 5 Ranking of rationales according to Kendall’s rank correlation test. The Rationale Table Preferers Tree Preferers Median Mean SD Median Mean SD Easier to learn 1 2.13 1.74 4 3.63 1.96 Table 3 Results of cognitive tests More visual 3 3.74 1.79 4 4.24 1.79 Test Median Mean SD More informative 3 3.04 1.99 2 2.63 1.80 Paper folding 5 5.33 2.24 More scalable 4 4.65 1.27 5 5.00 1.50 Surface development 0 8.59 11.14 More features 6 5.65 0.98 5 4.42 2.14 Visual analogy 17 15.89 3.88 More suggestive 5 4.78 1.86 3 3.54 1.94 Verbal analogy 8 9.97 7.49 More flexible 5 5.04 1.64 5 4.76 1.87 Leung et al. Automated Experimentation 2012, 4:2 Page 7 of 8 http://www.biomedcentral.com/1759-4499/4/2 Table 6 Relationship between cognitive test results and Table 8 Correlation between rationales and cognitive test preferences results Test Original preference Binary preference Tests Easier to learn More suggestive 2 2 c P c P τ P τ P Paper folding 9.16 0.103 5.10 0.024 * Paper folding 0.203 0.039 * -0.050 0.608 Surface development 6.09 0.298 0.15 0.697 Surface development 0.173 0.088 -0.070 0.484 Visual analogy 16.36 0.006 ** 13.50 0.000 ** Visual analogy 0.263 0.008 ** -0.201 0.040 * Verbal analogy 3.95 0.557 1.14 0.286 Verbal analogy 0.006 0.948 0.058 0.532 learn than trees. To tree preferrers, trees were more sug- low to moderate strengths of correlation in these results, it gestive than tables. These discrepancies in rationale rank- is probable that other factors (and other cognitive factors) ing indicate a perception difference between different may be also relevant to the participants’ preferences. preferrers in perceiving the given tables and trees. Further studies are required to delineate these Tree preferrers performed better than table preferrers in relationships. some tests of cognitive factors, particularly the paper fold- This experiment also demonstrated the technological ing visualisation test and visual analogy test. There was no significance of computational synthesis in enabling scienti- significant difference between table preferrers and tree fic experiments. This experiment re-used a website synthe- preferrers in their performance in other cognitive tests siser to generate websites on the fly so that the designers/ including the surface development visualisation test and users can explore the design space. Without using a web- the verbal analogy test. It is plausible that the interpreta- site synthesiser, this experiment would not be possible. tion of tree representations requires specific cognitive cap- Other than computational synthesis, the most relevant abilities such as visualisation and visual analogy-making; tool is Web content management system but it does not thus, those who do not feel comfortable with these tasks meet the requirement of this experiment. Apart from its would prefer tables and highly rank the rationale “easier to high cost, there is no Web content management system so learn” for their table preference. flexible as our our website synthesiser in accepting infor- Among all participants, there is a statistically significant mation mappings. This makes Web content management low-to-moderate rank correlation between the rankings of systems inapplicable to our experiment. Thus, computa- the rationale “easier to learn” and the results of visual ana- tional synthesis is the only available solution for this logy test and paper folding visualisation test. The low-to- experiment although computational synthesis following moderate rank correlation indicates that the participants definite patterns would limit the variability of graphical who performed better in such two cognitive tests ranked presentation. To enrich the variability and functionality of the rationale “easier to learn” to be less important. At graphical presentation, we would consider using web similar correlation strength, the participants who per- design patterns available in Web 2.0 and HTML5 to formed better in visual analogy test ranked the rationale enhance further studies. “more suggestive” to be more important. It seems that the visualisation and visual analogy-making abilities of the par- Conclusions ticipants might play a role in their preferences for tables The experiment reported in this article found significant and trees. Possibly (although we cannot prove this), table relationships between the participants’ design preference, preferrers lack sufficient cognitive capability to interpret rationale, and cognitive test performance. This work also graphical representations like trees; thus, they prefer tables exemplifies the use of a website synthesiser as an essen- as they are easier to learn. Tree preferrers would feel more tial instrument enabling the participants to explore dif- comfortable in making visual analogy and find graphical ferent possible designs, which were generated on the fly, representations like trees suggestive. As indicated by the before they selected their preferred designs. In the tested sample, more people prefer graphical to tabular presenta- tion. Despite the high preference for graphical presenta- Table 7 Comparison of cognitive test results between tion, the given tabular presentation was generally rated to different groups of preferrers be easier than graphical presentation to interpret, espe- Test Table Preferers Tree Preferers cially for those who score below average in the visualisa- Median Mean SD Median Mean SD tion and analogy-making tests. This piece of evidence Paper folding* 4 4.57 2.04 5 5.76 2.26 helps generate a hypothesis that design preferences are Surface development 0 7.61 11.16 0 9.15 11.22 related to specific cognitive abilities. Without the use of Visual analogy** 16 13.57 5.66 17 17.2 1.14 computational synthesis, the experiment setup and scien- Verbal analogy 9 9.96 6.47 7 9.98 8.08 tific results would be impractical to obtain. Leung et al. Automated Experimentation 2012, 4:2 Page 8 of 8 http://www.biomedcentral.com/1759-4499/4/2 Acknowledgements This study was partially supported by the UK EPSRC grant GR/M98302 for informatics research into communicating knowledge about accidents from synthesised websites. Competing interests The authors declare that they have no competing interests. Author details School of Informatics, University of Edinburgh, Edinburgh EH8 9AB, UK. State Key Lab QRCM and ICMS, University of Macau, Taipa, Macao, China. Department of Computing Science, University of Glasgow, Glasgow G12 8QQ, UK. Authors’ contributions DR and SL conceived this study. JL and CJ gave technical advice about web design and contribute knowledge in aviation accident domains. SL conducted the experiment and drafted the manuscript. DR and JL revised the manuscript. All authors read and approved the final manuscript. Published: 28 June 2012 References 1. Zhang J: The nature of external representations in problem solving. Cognitive Science 1997, 21(2):179-217. 2. Zhang J: Distributed representations as a principle for the analysis of cockpit information displays. The International Journal of Aviation Psychology 1997, 7(2):105-121. 3. Scaife M, Rogers Y: External cognition: how do graphical representations work? International Journal of Human-Computer Studies 1996, 45:185-213. 4. Oestermeier U, Hesse F: Verbal and visual causal arguments. Cognition 2000, 75:65-104. 5. Zhang J: A representational analysis of relational information displays. International Journal of Human-Computer Studies 1996, 45:59-74. 6. Kamps T: Diagram Design Springer-Verlag; 1999. 7. Mittal V, Roth S, Moore J, Mattis J, Carenini G: Generating explanatory captions for information graphics. Proceeedings of the International Joint Conference on Artificial Intelligence Montreal, Canada; 1996. 8. Mackinlay J: Automatic Design of Graphical Presentation. Phd thesis Computer Science Department, Stanford University; 1996. 9. Roth S, Mattis J: Data characterization for graphic presentation. Proceedings of the onference on Human Factors in Computing Systems (CHI) 10. Casner S: A task-analytic approach to the automated design of graphic presentations. CM Transactions on Graphics 1991, 10(2):111-151. 11. Krug S: Don’t Make Me Think: A Common Sense Approach to Web Usability New Riders; 2000. 12. Nielsen J: Designing Web Usability New Riders; 2000. 13. Detienne F: Software Design - Cognitive Aspects Springer-Verlag; 2002. 14. Ekstrom R, French J, Harmon H: Manual for kit of factor-referenced cognitive tests Education Testing Service; 1976. 15. Evans T: A heuristic program to solve geometric-analogy problems. Readings in computer vision Morgan Kaufmann; 1987. 16. R Development Core Team: R: A Language and Environment for Statistical Computing. Tech rep, R Foundation for Statistical Computing 2012 [http://www.r-project.org]. doi:10.1186/1759-4499-4-2 Submit your next manuscript to BioMed Central Cite this article as: Leung et al.: Design preferences and cognitive styles: experimentation by automated website synthesis. Automated and take full advantage of: Experimentation 2012 4:2. • Convenient online submission • Thorough peer review • No space constraints or color figure charges • Immediate publication on acceptance • Inclusion in PubMed, CAS, Scopus and Google Scholar • Research which is freely available for redistribution Submit your manuscript at www.biomedcentral.com/submit

Journal

Automated ExperimentationSpringer Journals

Published: Jun 25, 2012

References