Humans, space and interaction

Rapid technological progress is dramatically expanding our horizons where it comes to the interaction of humans with machines – also known as human-computer interaction, or HCI. Gone are the days when we had to swallow a 200-page manual in order to use a video recorder, on which five buttons had to be held down simultaneously, for three seconds, to enact a single function. HCI experts Dr. Robert Adelmann and Prof. Dr. Markus Stolze outline the latest developments.

Things have shifted up a gear in the field of HCI: from spatial communication, using technologies such as ­augmented reality, to the transformation of our digital tools; and passive gadgets to active assistants. Our Apple watches proactively alert us when our cardiac activity is irregular, our smartphones let us know in advance of traffic jams on the daily commute to work, and recommend we set off earlier than planned – all automated without any cues from us.

The evolution of the physical-digital world

As spatial beings, humans live in the physical world and interact, through the promptings of both evolution and environment, with other human beings. To these encounters we must now add interaction with products and the possibilities offered by the digital world. The great technological leaps of the last few years have made it easier for us to communicate with machines despite the technical complexity of the systems running in the background.

It is here that the spatial computing technologies of augmented reality (AR) and virtual reality (VR), combined with advances in machine learning and the ongoing networking of the “Internet of Things” are playing a significant role. So-called conversational interfaces and chatbots are allowing humans to conduct completely new kinds of exchanges with machines. Powerful perception of context in complex, real-world environments, based on image recognition, is also helping to drive rapid change – for example, in the context of self-driving vehicles or products such as Google Lens, which enable visual searches in the real world.

Seeing digital information in actual reality

A glance at the world through AR goggles is enough to identify which screws a worker will have to adjust ­without having to ask a technical expert for advice first, whilst looking at a building site through a tablet can reveal how a structure will eventually look. The concrete uses of AR have long-since moved from the drawing board to reality, providing genuine added value and innovation for the whole of society.

This direct interaction in the physical world promises a host of possible uses but will require us to rethink and redesign HCI; just bolting on previously well-established, classic 2D user-interfaces is not an option – where head-mounted displays (HMDs) no longer have traditional screens, as one example. Thanks to information and services, we can dispense with such screens and interact directly with things and places.

A lot of projects are, currently, still being implemented with smartphones or tablets, as these are widely used and companies can, therefore, avoid purchasing new or additional hardware.

Developments in HMDs are not standing still, however, as the presentation of Microsoft’s Hololens 2 at the Mobile World Congress at the end of February 2019 made clear – Microsoft is positioning its advanced AR solutions squarely in the field of industrial applications. The focus is on helping people who need both hands free to work and can be assisted in their undertakings with spatially located information. One example is the support provided in complex, custom assembly processes, such as those found in the auto industry and in machinery manufacturing, but also in the planning, construction and maintenance of buildings and infrastructure.

Interaction using all the senses

As HMDs become more widespread, the importance of multimodal interaction has also increased. This covers methods of interaction between humans and computers in which multiple sensory modalities are employed to input and output data. Examples include language/gesture recognition and eye tracking. Depending on the practical application, a combination of these may be preferable to make optimal use of the advantages of each technology.

Artificial intelligence for speech

Virtual assistants capable of voice interaction are becoming evermore important thanks to advances in machine learning and the availability of large volumes of data. In no time at all, voice-activated assistants and wireless headphones have established themselves as fixtures in households and vehicles, creating whole ecosystems for new applications and their distribution.

The Google Duplex project is another example of the big strides being made in AI and language, and involves the ongoing development of a digital assistant that has been trained to react in particular conversational situations, such as booking a restaurant table. Given sufficiently high volumes of training data, the assistant can deal with increasingly complex user-enquiries automatically.

Symbiosis versus replacement

It may not always be expedient to replace a human being with a voice-activated assistant. Companies are, therefore, increasingly focused on combining the strengths of each. In the finance industry, for example, an assistant used during appointments may be consulted by both the bank advisor and the customer to look up facts and data, however the principal exchange will still be carried out between the customer and the bank advisor.

Technology is adapting to humans

Developments such as spatial computing, AI, IoT and the ubiquitous networking of the “Internet of Things” enable us to leverage the practically limitless possibilities of digital technology in the real world – in other words, not just at our desks but in an increasing number of day-to-day situations. This will have wide-ranging and far-reaching consequences and open up many possibilities; not just for new products and innovations, but for society as a whole.

In HCI, such developments are helping us to enjoy the benefits and opportunities offered by technology whilst the technology itself is becoming less and less obtrusive – technology is increasingly becoming powerful enough to adapt to us humans, rather than the other way around.

Successful implementation of augmented reality projects with human computer interaction design

After desktop computing (both hands on the keyboard) and mobile computing (one hand free), the latest computing “revolution” to hit the headlines is augmented reality, or “spatial computing”. This innovation harbours potential for any field in which complex tasks are carried out with both hands and where quality and efficiency can be improved through access to context-specific and spatial data.

The challenge here is to embed this supporting technology into work processes in such a way that the potential for quality and efficiency gains can actually be realised. We have the knowledge required for user-centred design processes, but, unlike in desktop and mobile computing, where HCI design has long been accepted as a key success factor, we have some catching up to do when it comes to execution. Ensuring the success of AR projects in industrial applications will involve taking a leaf out of the book of the desktop and mobile computing world and choosing user-centred approaches rather than traditional, Tayloristic methods.

This article was written by Markus Stolze, Institute for Software Hochschule Rapperswil, and Robert Adelmann, HCI expert.