1.1 Background

Smart TV, equipped with Internet capability, is defined as a TV having better interaction ability with users than existing TVs. Smart TVs are also rapidly developed worldwide(Shin et al., 2013). Unlike existing TVs focusing on only media broadcasting, smart TVs have a system delivering various multimedia contents from network devices to users. Smart TVs also offer interactive Internet services including social networking, online games, shopping, and Web browsing (Lee et al., 2013). As such, smart TVs offer interactive multimedia experiences like smart devices, beyond TV, as an existing broadcasting device.

In a study of Cesar and Chorianopoulos (2009) on interactive TV, content control was regarded as a success factor together with content editing and content sharing. The use of a remote control can be a method to control contents. The existing remote control is a handheld input device without computing capability (Wang et al., 2011). However, as smart TVs emerged and remote controls became much more complicated than existing remote controls, a new interaction type, differentiated from the existing remote control type, was demanded. Therefore, many studies on smart TV-applicable, NUI-based interaction were carried out (Lee and Lee, 2013). Gesture interaction is the remote control-type next generation interface, and has an advantage for its natural and direct interaction using the hand (Shin and Choe, 2011; Baudel and Beaudouin-Lafon, 1993). Numerous manufacturers, such as Samsung, make efforts to implement direct and natural experiences to smart TV (Kim, 2014; Lee and Lee, 2013). Therefore, a natural interaction system including gesture remote control needs to be expanded to smart TVs, and researches on such a field should be steadily conducted.

Most previous studies on gestures were focused on recognition technology, and users do not worry about what gesture needs to be made for each operation command. Vatavu (2012) studied TV gesture control type that is defined by user him/herself. TV control type needs to be used so that any users can use by defining representative gestures for commands, rather than a type in which a user sets commands. As for gesture, a novice user should learn easily (ease of learning), and its use should be easy with a small recognition load of the user (ease of use) (Krum et al., 2002). This can be different from existing remote control by which a user can just press buttons corresponding to each control mode. For this reason, a proper number of commands need to be selected so that a user can easily learn, remember, and use the commands for gesture interaction. Each gesture command needs to be looked into about what style of gesture is suitable. There is a need to check what gesture style users prefer.

1.2 Gesture styles

Karam and Schraefel (2005) classified gesture styles in HCI based on gesture-associated previous studies as follows: (1) Deictic gesture, (2) Manipulative gesture, (3) Semaphoric gesture, (4) Gesticulation, and (5) Language gesture. The deictic gesture is an indicative gesture style pointing out spatial location. Such a gesture is used in virtual reality, as well as in the desktop environment. The manipulative gesture is based on actual existing manipulation. This is similar to making a gesture pushing upward like adjusting a control bar in the control command for volume up of a device. The semaphoric gesture is a sort of semaphore, and is defined as using a static or dynamic gestures' stylized dictionary (Quek et al., 2002). It is like making a static gesture connecting the thumb and index finger to express an "okay" symbol. The gesticulation means a gesture made generally during the talk, and it is expressed as speech and gesture interface, which has multimodal nature. This is also defined as "Coverbal gesture" (Bolt and Herranz, 1992; Kettebekov, 2004). Lastly, the language gesture is sign language recognition, and means the gesture expressed by drawing a letter's information. For example, it is like making a gesture of drawing "S" with the hand to express "stop" (Kim, 2014). This is different from the gesticulation style expressing a specific symbol which is stored in the recognition system and is based on language factors.

In conclusion, most previous studies are focused on recognition technology, and researches on which commands and gestures are required for users are inadequate, although gesture interaction is needed as a remote control type of smart TV. In this context, the purpose of this study is to find commands and gestures suitable for and preferred by users. An experiment is carried out targeting users, the control commands required for smart TV are extracted, and the corresponding gestures are selected. To this end, we looked at what gesture styles users use and confirmed what gestures they prefer among those styles. Through the results, this study was carried out with a process selecting smart TV control commands and gestures.

The experiment and analysis were conducted by referring to a study of Choi et al. (2012) and a gesture study method of Park and Han (2013). We carried out with a process of finding gestures based on the most suitable gesture factors and styles from control commands by checking gesture styles within the suitable gesture factors. Gesture types were constituted for grouping based on gesture types and in terms of preferred gestures in the analysis process before the experiment was carried out. Control commands used for remote control in a smart TV were also collected.

2.1 Gesture types

Detailed gesture types were constituted for smart TV by referring to a study on gestures of Park and Han (2013). The gesture types used in existing studies are touch screen-based types in the 2D environment, comprised of x axis and y axis. This study revised as the gestures applicable to 3D space by adding z axis. Table 1 shows the revised gesture types. Gesture types consist of "tap", "pose", and "path". These are divided into simple/pattern, static/dynamic, and staying/moving. Existing studies grouped gestures according to type. For example, users' tapping once with one finger and users' tapping once with two fingers were gathered in the same group, and reflected it to agreement level. This study also carried out grouping based on gesture types like existing studies.

2.2 Command selection for gestures

Commands by which smart TV's gesture interactions are possible were considered. The selection of control commands targeted LG smart TV (model: 47LM9600), and they were selected by referring to the commands selected from existing smart TV-related studies (Lee et al., 2011; Park and Han, 2013; Vatavu, 2012; Vatavu and Zaiti, 2014). As a result, 39 commands were collected within five applications (TV, photo, video, music, and Web browser) (Table 2).

Among the collected commands, the commands using duplicates for various applications were grouped. For channel which is used for home (basic application), a command for changing TV channel is employed, but grouping is possible with other commands having a similar nature in other applications. Concerning the photo application, it is a command to pass a photo, and is a command to play previous and succeeding contents in the case of video and music, and a command to open the previous and following pages in the case of Web browser. These were grouped with succeeding and previous commands. In this manner, we revised commands, and 18 commands were used in the experiment (Table 3).

3.1 Apparatus

A 47-inch (diagonal: 1920*1080 resolution) LG smart TV (Model: 47LM9600) was used in this study, and the experiment and analysis were carried out in the Web OS environment, which is LG smart TV's operating system, in order to look at commands required for smart TV. To videotape the process of participants' performing tasks, a Nikon D5300 model camera was used.

3.2 Metric

Concerned with commands, a study of Park and Han (2013) measured the consistency of gestures made by users through agreement level (Wobbrock et al., 2005). The study of Park and Han (2013) grouped the gesture factors (e.g., posture, location, touch, pose, path, and device) made by users per command and applied them. This study regarded the cases in which the numbers of gesture fingers were different even as one group. The agreement level equation is shown in Equation 1:

                          (1)

 means the agreement level on ith command, and  means the number of gestures made by users.  means the number of gestures within the jth group among the gestures made by users at the ith command. Park and Han (2013) reported a conclusion that the agreement levels of commands are suitable for mobile Web browser's gestures in the order of the highest agreement level. This is judged to be good for the selection of commands for which gestures are easy since the consistency level of gestures preferred by users in the commands can be checked. In the previous studies, the group with the highest  value was regarded as a top gesture. In other words, the most consistent gesture that users can come across with and express a command for is most suitable. In this way, the commands for smart TV and the most suitable gestures for the commands are found using the agreement level. The criteria for gesture grouping are needed in the process of producing agreement levels. In existing studies, grouping was conducted on the basis of gesture types suitable for smartphones. Likewise, the commands required for the current smart TVs are 2D-based. Previous studies reported that negative effects are brought about if 2D metaphor is used for control within the 3D environment because there is control difference between 2D and 3D (Liang and Green, 1994). They also reported that better performance can be shown in the case of a dimensionality match between task and input levels (Zhai and Milgram, 1998). Therefore, revision to 3D-based gestures suitable for smart TV reflecting z axis in the 2D-based (smartphone) gesture types in the existing studies is required.

3.3 Participants

This study targeted students in their 20s and 30s who most actively handle smart devices in which the use of gestures is possible, such as smartphone and tablet PC including smart TV. The participants were 20 undergraduate and postgraduate students at Hongik University. They consisted of 14 males and 6 females, and gender ratio was not considered. The mean age of the participants was 26.5.

3.4 Experimental design

The experiment was carried out in independent space within a laboratory, and was designed for all the participants to perform 18 command tasks with the within-subject design. The tasks were conducted by presenting commands in the order shown in Table 3, and the experiment was conducted in the same order for all the participants.

The questionnaire carried out prior to the experiment consisted of a 7-point Likert scale (1: not very so, 7: very so). Questions on personal information were asked first, and the participants answered such questions on smart TV use experience, whether gestures are used for smart devices, and their most favorite gesture.

3.5 Procedure

Before the experiment, questions on personal information and gestures were asked to the participants. The explanations on smart TV functions and applications were presented, as well as 18 gesture commands, 6 gesture types, and 5 gesture styles. Also, enough time to operate five applications for smart TV was offered to the participants. The participants were to select their preferred gesture type and style during the task performance by presenting the gesture type and style lists explained above prior to task performance. The participants could use one hand and both hands upon making gestures, and no time limit was offered. As with existing studies, they were instructed to pass, when they did not come across a gesture for the command concerned. The experiment was carried out in the mode of "think-aloud protocol". The reasons for the gesture concerned after making it according to each command, and opinions on commands were asked. All the task-performing processes of the participants were videotaped for an analysis.

4.1 Questionnaire

This study analyzed the preliminary questionnaire prior to the experiment. According to the personal information questionnaire survey result, 10 of the 20 participants did not have a smart TV or had no experience using it as half of the respondents had no experience using interactive functions except broadcasting functions on TV. All the participants answered with 4 or higher points on whether they use 2D gestures for smart TV (mean = 5.9, S.D. = 1.04). Most participants answered that they use a gesture moving to previous/next page in the application including Web browser in a smart phone, and that they use a rotation gesture upon editing a photo.

4.2 Gesture types

This study conducted grouping based on gesture type per command to draw agreement levels (Table 4). Figure 1 shows the calculated agreement level of each command.

Figure 1. Agreement levels of the gesture types about 18 commands sorted in descending order

Modify Delete

The participants preferred path-moving type gestures regarding 18 commands. The agreement level of pan and scroll commands was 1.00, showing the highest agreement level among 18 commands. The reason is that all the participants were grouped into one group by performing path-moving type gestures. This means that users prefer gestures moving using the hand with regard to the two commands. Meanwhile, the agreement level of stop command showed the lowest value at 0.42. The gestures made by the participants on the stop command were all in three groups, and 11, 6, and 3 gestures were made per group. Lower agreement level was shown since less difference between groups existed relatively, compared to other commands. This means that the gesture types reminded of or preferred by the participants were not consistent. Concerning volume up and volume down commands, their agreement levels were the same, and the constituted groups were the same as well. The same results were shown on such commands as next / previous and zoom in / zoom out. They are the commands having opposite nature of command, and it seems that the participants made gestures with different directivities, despite the gestures belonging to same type.

4.3 Gesture styles

Like gesture types, this study checked agreement levels to identify users' consistency on gesture styles (Table 5 and Figure 2).

Figure 2. Agreement levels of gesture styles about 18 commands sorted in descending order

Modify Delete

Upon looking at gesture styles, the participants preferred manipulative gestures in the following commands: next, previous, volume up, volume down, play, stop, zoom in, zoom out, pan, rotate, and scroll. The participants making such gestures said that those were based on experience using the gestures in a smartphone. They also replied that they came across the gestures directly looking at commands, in addition to smartphone. With regard to volume commands, the smart TV used for the experiment showed an effect of controlling clockwise and counterclockwise. However, only three participants made the corresponding gestures, and 11 participants came across the gestures manipulating up and down. On the other hand, control to left and right directions seems to be in place as their mental model, based on the experience passing to the previous or next content through another device. In conclusion, users seem to prefer manipulative gestures, based on the experience concerned with the commands associated with directivity.

As for mute command, a consistent gesture style was shown with semaphoric gestures. The participants used quiet gestures in line with the meaning of "mute" and the icon of "mute" or "x" sign meaning prohibition as semaphoric gesture. Concerning the remaining 7 commands, gesture styles were not consistent, and the participants preferred different gesture styles. Regarding "home" command, the participants preferred the language gesture, drawing the "H" of Home most (4 participants). Concerning source and history commands, the language gestures drawing "s" (5 participants) and "H" (5 participants) were relatively preferred much. The participants were not familiar with representing the commands with gestures, and did not directly come across them. Therefore, they said they preferred the language gesture style. However, the semaphoric gesture style was preferred most regarding the "refresh" and "bookmark" commands. As for "refresh", six participants came across an icon shaped like an object rotating clockwise, based on their experience of using smart TV's Web browser and PC's Web browser, making it as a gesture. Likewise, 7 participants answered that they came across Web browser's star-shaped icon and made it a gesture. Conclusively, users prefer the expression of language as gestures concerning commands if they do not have an experience using control commands with gestures. When the participants came across a familiar icon or symbol, they preferred semaphoric gestures. Lastly, concerned with the "toggle" command, they preferred manipulative and deictic gesture styles. Five participants preferred the manipulative style swiping palm to left and right directions, and four participants preferred deictic style, selecting with mouse pointing. However, the preferred two styles' difference is minimal, and it was unreasonable to define the gesture styles; thus, this study excluded them from the smart TV commands.

The experiment results imply the following: the gestures preferred by the users were based on the use experiences from other smart devices. The participants preferred using the gestures, used in smartphones, familiar to them in smart TV as well, and the gestures were generally path-moving type and manipulative style. When the participants came across icons or symbols symbolizing the commands concerned, they preferred semaphoric gestures. Concerning the commands awkward to manipulate with gestures, they showed a trend manipulating with language gestures. As for such commands, the need for making gestures is low to the users, and existing button control may be more positive.

This study selected the gestures of commands by arranging the gesture types and styles identified above. The arrangement process was carried out with numeric values considering the consistency of type and style by multiplying two agreement levels. Nine final commands were adopted on the basis of magical number presented by Miller (1956) as to final number of commands. Table 6 shows the gesture types and styles of smart TV suitable for gesture interactions. Also, Figure 3 shows the examples of gestures in line with preferred gesture types and styles.

Figure 3. Nine gesture examples and commands

Modify Delete

This study aims to draw commands for remote control of gestures in a smart TV and to define the gestures. We used gesture styles in addition to gesture types used for existing studies for gesture definition. With such a method, what gestures are desirable regarding smart TV's commands can be identified in reflection of the gestures preferred by users, beyond the agreement level method in previous studies. The commands and gestures required for smart TV control can be checked in this study, and each user's tendency by gesture type and style can be identified. Concerning gesture types, most users preferred gestures using the hand, and preferred manipulative gesture styles based on experience actually operating in the commands with directivity. Among other commands, the participants did not easily come across gestures in the case of unfamiliar commands for which they did not have the experience of using them, and they preferred the language gesture style. However, when the participants came across familiar symbols like icons in terms of commands, despite no experience of use, they preferred to define semaphoric signals with semaphoric gesture style. Based on the results, this study selected nine commands used for smart TV, and presented the preferred gesture types and styles. Through the findings in this study, the participants showed the most consistency in the path-moving type, moving the hand based on actual control that they experienced or knew and in the manipulative gesture style.

Some limitations exist in this study. Although the sample size of this study is 20 people, more samples will be needed for further study for better generalization of proposed results. More clear results can be revealed, if measurement indicators such as gesture making time and error rate are used by conducting a control experiment in which gestures are actually applied. Even though users' thought on commands was confirmed through interviews during the experiment, better results are expected in defining gesture styles if users' mental model-ascertaining procedure is applied to the study.

Many studies on gestures as natural UI/UX will be carried out, and many gestures can be applied to devices and services. In this regard, more research on gestures needs to be actively conducted. This study aims to confirm the gestures used for smart TV. However, such study method and results can be applied as a more developed mode to gesture study in the 3D environment, such as VR, as well as smart TV. In addition, the method used in this study should be utilized in various domains.