Blog Archive: 2020

Prototyping reality


One of our ongoing goals at the lab is to understand how best to take advantage Augmented Reality (AR) to annotate physical objects with digital media. Unfortunately, the objects we tend to focus on (such as mutli-function devices or printers) are often large and relatively immobile, making it difficult for us to visit remote sites to demonstrate our technologies.

To address this problem, we are experimenting with paper-based models of the physical objects we want to augment, which are much more lightweight and mobile while still approximating the embodied experience of a 3D device (see Figure 1). In order to register the paper-based models with AR tracking systems, we can either scan the entire paper-based object or, if the object corresponds to a cube or rectangular box, we can register each side as independent images (images may in fact correspond to registration images used in the actual scene). In either case, this paper-based object is mobile and easily reconfigurable, giving us much more flexibility in how, when, and where we present AR content (Figure 2).

Figure 1. Our printer paper prototype.
Figure 2. Viewing digital content affixed to the paper printer prototype with a mobile AR tool.

This approach represents somewhat of an inversion of typical paper-based prototyping methods in which user-interface elements are prototyped rather than the physical object against which they are registered (which do not exist for most 2D interfaces). Marc Rettig introduced lo-fi prototyping with paper UI elements in his influential paper Prototyping for Tiny Fingers , and this method was adopted rapidly throughout the user experience community. Recently, researchers have extended it to AR scenarios as well.

PapAR was one of the first to adapt paper prototyping techniques to AR for head-mounted displays. It is a straightforward design that involves a base layer with real-world elements drawn in paper similar to a typical paper prototype as well as a transparent overlay onto which are draw AR interactors. This is a simple and elegant “glass pane” approach familiar to user experience professionals.

Figure 3. In PapAR, authors move a transparent AR overlay over a sketched real-world scene.

Michael Nebeling’s work at the University of Michigan School of Information pushes this concept further. Inspired by issues with an earlier AR creation toolkit (the influential DART system), Nebeling et al. first built ProtoAR, which allows AR designers to integrate 2D paper sketches as well as 3D Play-Doh mockups into a prototype AR scene. The toolkit includes a desktop and a mobile app that creators can use to scan physical objects, integrate into an AR scene, and link to real-world markers.

The researchers later extended this toolkit to allow authors to adjust the representation of AR content live, facilitating Wizard-of-Oz style user testing (see their CHI presentation on this work).

Closer to our approach are tools that augment paper prototypes with digital resources to experiment with AR content. For example, the ARcadia system supports authoring AR-based tangible computing interfaces. In this system, content creators attach markers to paper prototypes then use a desktop tool to augment the prototypes with digital content.

We have a long tradition of using and extending lightweight prototyping methods at FXPAL. In light of recent events, we expect to focus future work on extending lightweight AR prototyping tools to support remote experimentation and design iteration.

Augmented Reality: Is this time different?


Ivan Sutherland’s Sword of Damocles, a head-mounted virtual and augmented reality system, was ungainly but remarkably forward-thinking. Developed over a half-century ago, the demonstration in the video below includes many of the components that we recognize today as critical to VR and AR displays, including the ability to display graphics via a headset, a positioning system, and an external computational mechanism.

Since then, AR and VR have experienced waves of hype that builds over a few years but reliably fades in disappointment. With the current excitement over consumer-level AR libraries (such as ARKit and ARCore), it is worth asking if anything is different this time.

The Augmented Connected Enterprise (ACE) team at FXPAL is betting that it is. We are currently building an AR-based remote assistance framework that combines several of our augmented reality, knowledge capture, and teleconferencing technologies. A future post will describe the engineering details of our work in more detail. Here we explore some of the problems that AR has faced in the past, and how we plan to address them.

In their paper “Drivers and Bottlenecks in the Adoption of Augmented Reality Applications” [1], Martinez et al. explored some typical pitfalls for AR technology, including No standard and little flexibility, Limited (mobile device) computational power, (Localization) inaccuracy, Social acceptance, and Amount of information (Distraction). We address each of these in turn below:

  • No standard and little flexibility
  • Limited (mobile device) computational power

Advances in contemporary technologies have largely addressed these two issues. As mentioned above, the market appears to be coalescing into two or three widely adopted libraries (specifically ARKit, ARCore, and Unity). Furthermore, limited computational power on mobile devices is a rapidly receding concern.

  • (Localization) inaccuracy

Caudell and Mizell echoed this issue in their paper introducing the term, “augmented reality” [2]. They wrote that, “position sensing technology is the ultimate limitation of AR, controlling the range and accuracy of possible applications.”

Addressing this concern involves scanning several real world objects in order to detect and track them in an AR scene. Our experiences so far reveal that, even if they aren’t yet ready for wide deployment, detection and tracking technologies have come a long way. The video below shows our procedure for scanning a 3D object with ARKit (adapted from this approach). We have found that ensuring a flat background is paramount to generating an object free of noisy background feature points. Other than that, the process is straightforward.

Scanning an object in this way generates a digital signature that our app can recognize quickly and accurately, allowing us to augment the physical object with interactive guides.

  • Social acceptance

The many issues associated with the launch of Google Glass made it clear that HMD devices are not yet acceptable to the consumer market. But our intuition is that focusing on the consumer market is inappropriate, at least initially, and that developers should instead target industrial settings (as Caudell and Mizell did at Boeing). A more appropriate metaphor for AR and VR devices (outside of their use in gaming) is a hard hat—something that you put on when you need to complete a task.

  • Amount of information (Distraction)

Martinez et al. are concerned that the “amount of information to be displayed in the augmented view may exceed the needs of the user.” This strikes us less as a bottleneck and more a design guideline—take care to make AR objects as unobtrusive as possible.

In addition to the issues above, we think there are at least two other problems standing in the way of widespread AR adoption:

  • Authoring

There are a variety of apps that can help AR content creators author scenes manually, including Amazon Sumerian, Apple Reality Composer, Adobe Aero, and ScopeAR WorkLink. However, with these tools designers still must create, import, place, and orient models, as well as organize scenes temporally. We think there are opportunities to simplify this process with automation.

  • Value

Finally, as with any technology, users will not adopt AR unless it provides value in return for their investments in time and money. Luckily, AR technologies, specifically those involving remote assistance, enjoy a clear value proposition: reduced costs and time wasted due to travel. This is why we believe the current wave of interest in AR technologies may be different. Previous advances in the quality of HMDs and tracking technologies were not met with similar increases in teleconfercing technologies and infrastructure. Now, however, robust, full media teleconferencing technologies are commonplace, making remote AR sessions more feasible.

Many tools already take advantage of a combination of AR and teleconferencing technologies. However, to truly stand in for an in-person visit, tele-work tools must facilitate a wide range of guided interaction. Experts feel they must travel to sites because they need to diagnose problems rapidly, change their point-of-view with ease to adapt to each particular situation, and experiment or interact with problems dynamically. This type of fluid action is difficult to achieve remotely when relaying commands through a local agent. In a future post, we will discuss methods we are developing to make this interaction as seamless as possible, as well as approaches for automated authoring. Stay tuned!

[1] T. P. Caudell and D. W. Mizell. “Augmented reality: An application of
heads-up display technology to manual manufacturing processes”. In
Proc. Hawaii Int’l Conf. on Systems Sciences, 659–669, 1992.

[2] Martínez, H. et al. “Drivers and Bottlenecks in the Adoption of Augmented Reality Applications”. Journal of Multimedia Theory and Application, Volume 1, 27-44, 2014.