Human Computer Interaction & Touch Application Development Independent Study, Fall 2019

Abstract

This semester our team developed, tested, and launched and a fully functioning and robust web application which features several sub-applications designed to broadcast the message of two on-campus clubs and activities. Through this project-based learning environment, we learned extensively about touch-event driven software, web application development, human computer interaction (HCI), the user-centered design process, and collaborating with non-technical product clients.

Introduction

Interactive Media & Human Computer Interaction

As technology has improved, software applications have advanced significantly, not only enabling the user to accomplish increasingly complex tasks, but also enabling different relationships between technology and humans to be defined. Interactive media delivers content based off of the user’s interactions with the system (Augustyn). The field of Human Computer Interaction (HCI) focuses on optimizing the relationship between technology and its users ("What Is Human-Computer Interaction?"). Within the context of interactive media, HCI focuses on optimizing the design and layout of an application to create an intuitive and enjoyable application for the user.

Project Motivation

The Colorado School of Mines campus is filled with large displays which showcase campus events, announcements, and other information. Currently, these displays do not allow for user input and only consist of static content. Since interactivity generally leads to better engagement, the Mines campus is losing out on the opportunity to have more powerful and lasting impressions on students, staff, and community members alike.

Digital kiosks have become increasingly popular in the past five years, and are one of the most common forms of interactive media in public, high-traffic areas. Digital kiosks give the campus the ability to engage in real-time, custom communication with its increasing population. The remainder of this paper will discuss: (1) how the application for the new digital kiosks was implemented, (2) the goals of the application development process, (3) the current results from this project, and (4) how this work will be expanded upon in upcoming development cycles.

The final motivation for this project was to provide undergraduate computer science students with project-based learning opportunities. As part of this, one of the primary goals was to provide opportunities for students to collaborate with non-technical individuals throughout the development process. Accordingly, one of the key aspects included in this project was to develop the application content from external clients, thus providing student developers the opportunity to simulate real-world software development.

Project Overview & Implementation

In 2018, the Colorado School of Mines Computer Science Department purchased two 65" Large Format Touch Displays from NEC, each powered by an Intel NUC with Windows 10. This project focused on developing an interactive application for these interactive media displays (IMDs). The development of this application was broken down into four main sections:

1. The design, layout, and configuration of the base application/home page
2. The campus map sub-application
3. The educational brochure on food insecurity sub-application
4. The self-care tip wheel sub-application

The application for the project was developed with a Google API, Express, Vue.js, and Node.js stack. Vue.js is the component-based javascript frontend framework, and our server implements Express on top of Node.js. This connects the frontend to our database, the Google Sheets API.

The application for the project was developed with a GEVN stack--Google API, Express, Vue.js, Node.js. We used Vue.js, a component-based javascript framework, on the frontend. The backend is a Node.js server that used Express for its HTTP implementation. This connects the frontend to our database, the Google Sheets API.

Project Goals

The primary goals of this project can be broken down into three main categories: (1) web application development, (2) human computer interaction in a user-centered design process, and (3) content development in collaboration with external clients.

Web Application Development

1. Design and develop a touch-event driven application using HTML5, CSS3, and a web application framework
2. Learn proper web development practices and software engineering principles through a project-based learning environment
3. Create an interface that fits the Mines branding image and showcases the campus resources to users

HCI in a User-Centered Design Process

1. Discuss and apply proper design methodologies to create a visually appealing and accessible website
2. Engage in a user-centered design process to implement the most appropriate HCI methods/gestures for various interactions with the application

Content Development in Collaboration with External Clients

1. Create an avenue for non-technical individuals to convey various messages to the campus in an engaging way
2. Establish a general process for working with external groups for the IMDs

Project Results

Implementation & Programming Results

The goals of our project were three-fold, and overall we celebrated successes in each of these areas. Concerning the technical work, all four of the application sections were implemented to a final working project. While all of these still have opportunities for improvement, they are ready to launch when the IMDs are installed at the beginning of 2020. Figures 1 illustrates each of the four main sections as outlined in the project overview:

Figure 1) (a) The design, layout, and configuration of the base application’s home page, (b) the campus map sub-application, (c) the educational brochure on food insecurity sub-application, and (d) the self-care tip wheel sub-application.

Collaborative Development Results

The educational brochure on food insecurity and the self-care tip wheel sub-application were developed in collaboration with the Fighting Hunger at Mines and Every Oredigger clients respectively. In this process we achieved our goals for the content development and were able to continually improve the pages through product review sessions with the client. Two main reusable templates emerged from this process: (1) a request for proposals (RFP) document and (2) a project evaluation rubric. Initial feedback from the clients was positive for both the sub-applications themselves and the project development was a positive experience. Continual client feedback from an iterative design process was the largest asset in achieving these goals.

HCI Results

The most difficult part of the project was achieving our HCI goals. HCI is a relative and in-exact science that requires a multi-faceted approach, including an iterative design process and various levels of user testing. Largely, we had to consider HCI that centered around how the user navigates through the application (Jiuqiang et al.). This navigation brings the user to the other aspect of our HCI work, which we will call ‘information endpoints’. From researching HCI and navigation we learned the importance of leaving hints for how the user should interact with various components (Huang and Lai; Kim et al.). Largely we implemented this through color consistency. Colors were also used to indicate positive, negative, and neutral interactions. For example, we chose orange to indicate a ‘positive’ action. We deem an action positive if it moves you forward through the application and negative if it moves you backwards. Figure 2 illustrated how we used bright and consistent coloring throughout the various sub-applications.

Figure 2) Buttons throughout the application. Orange was chosen from the school color scheme to draw users towards these elements and naturally advance them through the application. The ‘Close this Window’ button is a negative action, and thus it has a more subtle coloring.

Another challenge in refining the HCIs of the application was determining how we would illicit user input. Because of security as well as the device capabilities of the IMDs, we constructed our app free of free-text input. Requiring free-text input from the user would require using an on-screen keyboard on a 65” monitor. This would be extremely frustrating to use for long inputs, and would require prolonged attention from the user. Because of the IMDs location in high traffic, public areas, we expect our users to only interact with the application for 1-3 minutes. Consequently we only implemented coarse input in our application for our information endpoints. Coarse input, as opposed to fine or free-text input, involves simple and short interactions from the user. Figure 3 illustrates a couple of the various ways we collected coarse inputs from the user to collect/deliver content.

Figure 3) Two different methods of coarse user input collection. The checkboxes change which map markers display and the orange buttons bring you to the next page of the modal, displaying the various items in the chosen category.

Beyond clicking, the gestures we implemented were swiping and pinching. Swiping was implemented in two main locations: in the brochure to advance left/right and in the wheel to start the spin animation. Swiping was chosen because it intuitively mirrors the actions a user would do on a physical implementation of the same format. Pinching indicated zooming, which the user can do on the map. On the opposite side of HCI considerations, preventing certain gestures was another challenge of this project. While we were developing the application as a browser-based one, we did not want the user to feel like they were simply ‘using a webpage’. This required preventing a majority of the default actions to various user-events, including but not limited to, text highlighting, pinching to zoom, right-clicking for the context-menu, and swiping to go back a page. The largest considerations we had to contemplate when removing default behavior was the implications it would have on accessibility. This was largely through user-testing, as the majority of accessibility standards apply to websites accessed through a variety of devices, and not controlled, known-device access like our application.

Overall through building this application we learned extensively about the web application development process, including industry standards and design principles. Through this project-based learning environment we were able to leverage this knowledge to create several powerful, visually appealing, and user friendly applications which will be displayed around the campus.

Future Work

While our application is ready to be launched live, there is still an abundance of work that can be done to improve and expand upon the project. The majority of improvements concern the HCIs of the system. One goal we wanted, but we were unable to implement, was real-time feedback gestures. For example, when the user swipes left/right to navigate through a brochure, we wanted to have the content change as they swiped, instead of after. This would require more complex animation and event-capturing, which was not available in the open-source library we implemented, vue2-touch-events. This also applies to the swiping of the wheel, which only spins after the user is done swiping. Another problem with this swipe implementation is it only catches swipes in the four cardinal directions, as opposed to diagonal swipes. This is particularly detrimental to the wheel, because initial testing has shown the user naturally swipes it diagonally.

Another improvement when considering the system’s HCIs is leaving the user more hints on how to interact with the system. Currently, the only hint we leave the user is the coloring and styling of buttons. However, if the user doesn’t understand this, they do not have any other suggestions on how to use the app. One of the shortcomings currently with the app is that is doesn’t indicate to the user explicitly that it is a touch screen. Implementing graphics as shown in Figure 4 would help suggest touching the screen to the user.

Figure 4) Graphics that could be displayed to the user after an extended period of time with no touch-events on the screen. Graphic from The Next Web, http://tnw.to/p3R4G.

Beyond the HCI of the IMDs, the other improvements to the system are largely minor changes to the sub-applications to make them more dynamic or to help them convey more information simply. We have documented these improvements in the code for future semesters and development teams. Additionally there are many opportunities with the IMDs to contribute to HCI or touch-driven applications. One possible study is expanding upon the relevance of location, color, and internal consistency of components within the application.

Acknowledgements

To Dr. Jeffrey Paone, our course adviser, for his expertise and dedication to our team.

References

Augustyn, Adam. "Interactive Media | Definition, History, Examples, & Facts". Encyclopedia Britannica, 2019, https://www.britannica.com/technology/interactive-media. Accessed 5 Dec 2019.

Huang, Hsinfu, and Hsin-His Lai. "Factors Influencing the Usability of Icons in the LCD Touchscreen". Displays, vol 29, no. 4, 2008, pp. 339-344. Elsevier BV, doi:10.1016/j.displa.2007.10.003.

Jiuqiang, Fu et al. Application Interface Structure Research Based on Touch Screen. School Of Design And Arts - Beijing Institute Of Technology, Beijing, China, 2016. Accessed 6 Dec 2019.

Kim, Jangwoon et al. Human Computer Interaction Using Multi-Touch Tabletop Display. Department of Computer Engineering Chonnam National University, Gwangju Korea, 2019.

"What Is Human-Computer Interaction?". The Interaction Design Foundation, https://www.interaction-design.org/literature/topics/human-computer-interaction.