Individuals experience multi-attribute decision making in their everyday lives such as when they purchase a product, select a college, or decide on a house to rent. Multi-attribute decision making occurs when a consumer has several alternatives and attributes associated with a choice. When choosing a car to purchase, for example, one can consider price, engine, horsepower, fuel efficiency, and several other factors (Yoon and Hwang, 1995). With the advent of computing and information technology, consumers are now facing vast amounts of information for their choice making. Several studies also have shown that decision makers often experience information overload and become overwhelmed (Chen et al., 2009; Grant and Schwartz, 2011; Hahn et al., 1992).

In order to help decision makers, researchers have been applying information visualization techniques and have demonstrated that these techniques could improve decision performance (Carenini and Loyd, 2004; Williamson and Shneiderman, 1992). These visualizations enable people to explore large quantities of information and gain insights from them. Since multi-attribute decision making involves effortful cognitive processes, various forms of visual interfaces can help decision makers achieve higher decision quality and satisfaction (Bautista and Carenini, 2008; Savikhin et al., 2008). However, to show that the visualization was effective, researchers conducted controlled studies to context-free data to better understand the information-processing perspective or just measure subjective ratings such as satisfaction level. We need to investigate the impact of visualization techniques in a rather practical setting.

In this study, we use a visual aid "SimulSort" that is a tabular information visualization to help multi-attribute decision making. The practical data sets with context were about selecting the best apartments, laptops, and printers. We conducted an online study to show that visual aids help people to make a better decision.

2.1 Visual aids for multi-attribute decision making

To alleviate high cognitive activities that occurs during multi-attribute decision making, various information visualization techniques have been utilized by presenting insights more interpretable and by lowering the cognitive load for decision makers (Wittenburg et al., 2001). A parallel-coordinates is one of the common ways to project a large amount of multi-dimensional data on a 2-D display to analyze multi-attribute data (Wang et al., 2003). It provides an overview of the data, however, additional interaction is needed to read the raw data. TableLens is an example of a visual interaction tool with tabular form (Rao and Card, 1994). It expands the boundaries of a table that can be shown on a screen by presenting a large table in a symbolic style and allowing users to expand certain areas for detailed examination. Parallel bargrams is an example that exceeds a generic tabular form and sorts all the attributes in parallel rows at the same time (Wittenburg et al., 2001). It is designed to help consumers with multi-attribute mechanized purchasing decisions by providing simultaneously comparable attribute values for different alternatives. FOCUS (Spenke et al., 1996), EZChooser (Wittenburg et al., 2001), and InfoZoom (Spenke and Beilken, 2000) are some visualization tools that apply the idea of parallel bargrams. However, as these visual aids have several features implemented, the evaluation on the tools were mostly measuring the overall satisfaction level during a task, rather than on the decision-making performance.

SimulSort (SS) is another visualization technique that also aims to help with daily multi-attribute decision making (Hur and Yi, 2009). Compared to the visualizations explained above, the visual factor of the tool is more simple and it has a clear counterpart interface, Typical Sorting explained below, that provides an opportunity to conduct a controlled study. As shown in Figure 1(a), SS presents all columns sorted simultaneously so that one can see the relative values or utilities (i.e., pros and cons) of an alternative over multiple attributes. This visual representation is expected to avoid the constant shuffling of rows induced by sorting a column in the Typical Sorting (TS) interface (i.e., Figure 1(b)) and to offer insights to users by presenting the trend of the data at a glance. Typical Sorting can be considered as a traditional spreadsheet.

Figure 1. Example of the two interfaces, SimulSort and Typical Sorting, comparing two alternatives; item 15 is highlighted in green, and item 5 is highlighted in yellow. (a) SimulSort: The comparison of the two items can be done by comparing the vertical positions of the highlighted cells. (b) Typical Sorting: The comparison of the two items can be done by reading the face values of all of the cells

Modify Delete

2.2 Experiments using data with context

The main difficulty for conducting an experiment for decision making is that people have different preference structure, therefore it is hard to define what the best decision is. For example, when purchasing a car, some consumers might consider price as the most important factor, while others consider horsepower. Therefore, if we give a decision-making task with contextual data, the best option will be different for each participant. To overcome this difficulty, several studies were conducted with context-free data as they can guarantee what the best option has with calculating the utility in a mathematical way. For example, context-free data has attributes that are labeled with meaningless names such as 'm1' and 'm2' as shown in Figure 1(a). Therefore, people do not have any preference on the attributes and the attributes are considered to be equally important. Although context-free data helps to conduct a controlled experiment, the experiment setting is considered to be artificial.

For a more realistic experimental setting, contextual data should be used that has options with attributes and values considered from real life as shown in Table 1. One way to control different preference structure but use this kind of contextual data, Lurie (2004) suggested to inject the preference structure by giving explicit weights to attributes. For example, if there are attributes such as price, weight, distance to store, the experiment instruction explains that the weight of each attribute is 0.6, 0.2, and 0.2. However, it is hard to know whether the participants actually considered the different weight values. Lastly, Haubl and Trifts (2000) conducted an experiment by designing the dataset to have dominant and nondominant options. An option is dominated if at least one alternative is superior in at least one attribute while not being inferior in any other attribute. In contrast, a nondominated alternative is the case where no alternative is superior to another attribute without being inferior to at least one other attribute. If one selected a dominated option, it means that there is at least one option that is dominating the selected option, which makes the selection a poor choice. In other words, if a decision maker selects a nondominated option, it can be interpreted as a good decision because it dominates the other remaining options. The advantage of this measure is that it is immune to the weights of different attributes. Although a person weighs a certain attribute over another, the better decision is to select one of the nondominated options. Therefore, even with contextual data, the best choice is the same for everyone.

3.1 Dataset design

The author designed the three practical data sets with context (i.e., apartments, laptops, and printers), following the method by Haubl and Trifts (2000). Commonly, each data set had 30 alternatives for three brands with seven attributes. There are 10 alternatives for each brand and there is one nondominated alternative among the 10 alternatives. Thus, there are a total of three nondominated alternatives for the three brands. In the following description, the author explained the design method of the data sets based on an exemplar of the apartment data set.

Table 1 shows the attributes, how the value is preferred to be a better option (i.e., high or low), the number of levels for each attribute, and the values for each level. For each attribute, whether higher or lower value is preferred followed common sense as most people prefer a cheaper price to a higher price. The possible values in Table 1 for each attribute are sorted following the preference structure.

Table 2 shows the data set of apartment, which was used in this study. The data set has a total of 30 alternatives for three brands, each brand has 10 alternatives, and each alternative has seven attributes. Out of the 30 alternatives, three are mutually nondominated and one of each nondominated alternative is assigned to a certain brand (i.e., management in Table 2). Having one nondominated option for each brand guarantee that regardless of brand preference, there is a single most preferred alternative within that brand. Among the seven attributes, three have six levels each (i.e., rent, square feet, and distance as shown in Table 1) while the remaining four attributes have three levels each (i.e., maintenance, cleanness, office staff, and recommended rating as shown in Table 1). The three nondominated alternatives used the following rule: For the first three attributes with six levels, all three nondominated alternatives were assigned with the best level represented as one, the second best represented as other, and the third best as the remaining attribute. Because there are six combinations of value assignments, the author had selected three combinations for the nondominated options. For the other four attributes, the best level of the three-level values was assigned. At the end of Table 2, the error size column is the count of attributes that are dominated by the nondominated option. For the experiment, the order of the alternatives was randomized. To better understand the rationale of this dataset, if a decision maker selects Apt02, he or she is making a bad decision because Apt01 offers more square feet while its other attribute values remain the same.

3.2 Participants

A total of 220 crowdsourced workers were recruited. However, 93 participants were removed if they had different preference structure of what we assumed as in Table 1. Because this common sense might not be guaranteed and people may have different preferences for these values, the author collected their preference structure afterwards and removed the collected data if they were different from this assumption. Eventually, 127 legitimate participants (53 female) remained with 67 participants for SS and 60 participants for TS. The ages ranged from 19 to 64 years old (m = 33.0, s = 10.3).

3.3 Experiment design and procedure

We conducted the experiment through Amazon Mechanical Turk, a well-known crowdsourcing platform. The crowdsourcing approach has several advantages over conventional, controlled laboratory studies (Kittur et al., 2008), including recruiting a large number of participants with diverse backgrounds in a more natural environment.

The experiment was conducted as a between-subject study with two interfaces (SS vs. TS). Each participant had 10 practice trials with context-free data to learn how to use the interface 1. After the 10 practice trials, the participants had three trials with different data sets: apartment, laptop, and printer. For each trial, a scenario was given. The following scenario is for the apartment data set: "In this trial you are making a decision for yourself. Imagine that you have moved to a new city and you are finding an apartment with one bedroom. Attributes you can consider are Management, Rent ($), Square Feet, Maintenance Rating (Best: 10), Cleanness Rating (Best: 10), and Office Staff Service Rating (Best: 10), Distance to work place or school (minutes), and Recommended by others (%)".

3.4 Measurements

To determine the objective measurement of decision quality, I recorded whether final selection was the dominated or nondominated option. For the subjective measurement, the confidence level of the final decision was asked with a 7-point likert scale (7-point Likert scale question: from strongly disagree to strongly agree). The statement to evaluate confidence level was "I am confident of my decision by selecting the best option". The time spent making a decision for each trial was also recorded.

3.5 Reward design

The base payment for participating in the task was $1.80, which was rewarded if they participated. The maximum bonus was calculated based on their performance for each trial in which they could earn a maximum of $0.10. The payment for the 10 practice trials was proportional to participants' decision quality (Hur and Yi, 2009) and for the trials showing data with context, participants were paid $0.10 regardless of their selection. The trials showing data with context was meant to be realistic of making a decision for themselves, therefore, the author did not penalize for selecting a dominated option. If they were penalized by selecting a dominated option, it may have changed their behavior of making a choice. The average earning for the participants was $2.90 (s = 0.15).

4.1 Decision quality

The logistic regression model was used with PROC LOGISTIC in SAS. If one of the nondominated options was selected, it was graded as 1, and if not, it was graded as 0. This model was meant to predict the probability of selecting the nondominant option while using each interface. As shown in Figure 2, the interface showed that it has significant impact on improving the model fit (Wald Chi-Square = 18.4141, p < 0.0001). Using the SS interface, the odds of selecting a nondominated option, compared to not selecting it, increased by a factor of 2.516 (95% CI: 1.651, 3.834).

Figure 2. Distribution of the number of selections choosing the nondominated option

Modify Delete

4.2 Time spent

A repeated measures ANOVA test was employed using the PROC MIXED procedure in SAS. The interface (F(1,125) = 11.35, p < 0.001) had a significant main effect with the mean time spent for SS and TS as 137.4 seconds and 87.24 seconds, respectively.

To better understand this phenomenon, the author added the type of selected option, whether it was dominated to nondomiated option, to the model for analysis. The same repeated measures ANOVA test was employed as above. Both the interface (F(1,125) = 6.55, p = 0.0117) and type of selected option (F(1,77) = 0.033, p < 0.001) had significant main effects on the time spent. A post-hoc analysis for comparison with Bonferroni adjustment revealed that the type of selected option had significant effect for the participants who used SS (F(1,77) = 4.92, p = 0.0295) with the mean time spent for nondominated option and dominated option as 151.96 and 102.34 seconds, respectively. However, this effect was not present for the participants who used TS (F(1,77) = 0.68, p = 0.4111) with the mean time spent for nondominated option and dominated option as 96.33 and 78.53 seconds, respectively. A post-hoc analysis also revealed that with the case of not selecting a nondominated option, there was no significant difference between the two interfaces (F(1,77) = 0.98, p = 0.3260) with the mean time spent for SS and TS as 102.34 seconds and 78.53 seconds, respectively. However, in the case of selecting a nondominated option, there was a significant effect of the interface (F(1,77) = 8.06, p = 0.0058) with the mean time spent for SS and TS as 151.96 seconds and 96.33 seconds, respectively.

Figure 3. Interval plot of time spent for both SS and TS divided by groups selecting the dominated or nondominated option (95% CI for the mean)

Modify Delete

4.3 Confidence

The Kruskal-Wallis non-parametric test revealed that the type of interface had a significant impact on the perceived self-explanatory level (χ2 (1) = 8.0173, p = 0.0046). The mean rating was 5.82 for SS and 5.50 for TS, while 5 stands for 'Somewhat Agree' and 6 stands for 'Agree'. Therefore, although there is statistically significant difference, the participants were generally confident on their decisions with both SS and TS.

The purpose of this study was to examine whether SS could help the decision-making process showing data with context. The study of Hur et al. (2013) revealed that the decision quality was higher spending shorter time in SS compared to TS with context-free data. In this experiment with contextual data, the results of making a choice for an apartment, printer, and laptop, showed that SS compared to TS, led to an increased share of the cases that selected a nondominated option that was considered to be a good decision. On the other hand, the results for time spent making a decision had opposite results. With the data with context, participants spent a longer time with SS compared to TS. As shown in Figure 3, the longer time spent originates in cases particularly when participants selected the nondominated options.

People are known to make a cost-benefit tradeoff while establishing decision-making strategies (Payne et al., 1993; Shah and Oppenheimer, 2008). This means that although the decision makers can reach a better decision by applying more effort, if the effort to apply the search strategy exceeds their capacity, they tend to stop at the suboptimal choice. In this experiment, longer time spent may be interpreted as SS requiring more effort, however, the author believes that cost of additional effort with SS-in this case spending more time-might have been perceived worthwhile because participants could then expect the outcome to be good without making a trade-off between the two.

As a result of this experiment, the author could observe one optimal way of using SimulSort: to detect dominated options. Regardless of preference, selecting a dominated option is a poor decision. In a particular case, while comparing two options, SimulSort can help the user to easily find the bad option. This was also shown in the comments that one of the participants revealed for explaining the strategy they used: "I'd start by highlighting 1, making a series of binary judgments as I worked my way down. Many could be passed quickly [by] looking at the sweep of color; some I had to compare more closely by looking at relative positioning in the individual column". Another way to apply this strategy is to create a smaller set of options. The smaller set of options is often called considerations set (Hauser and Wernerfelt, 1990; Lapersonne et al., 1995). If one could easily rule out the bad choices, the remaining set of possibilities would become smaller, consisting of better options. However, further experiments should be conducted to understand different strategies while using these kinds of visual aids.

The use of context-free data in these previous studies had the advantage of understanding the information processing behavior by minimizing individual differences induced by the context. However, it is still important to know if SimulSort is beneficial in a rather realistic context. The effectiveness of decision quality while using visual aids was in line with studies that had context-free data. The results revealed that SimulSort could enhance the probability of making a better decision than compared to Typical Sorting. However, the time measurements had different results, which shows that considering the context of data has impact and it could change the decision-making behavior of the participants compared to context-free data. Moreover, the method of using dominated/nondominated options could provide a obejective measure for decision quality while not being biased by the individual's preference structure.

The author attempted to understand the impact with a realistic decision making context. However, there are several other ways to extend the study in terms of dataset size and context, different ways of measuring the decision outcome, as well as different visual interfaces. These techniques should be surveyed more comprehensively and examined for their effectiveness. If we can investigate thoroughly to deepen our understanding of "visualized decision making", this can lead to building guidelines for developing effective visual aids that will help consumers.