Course Content

Prerequisites for a good AR experience

• Video integration has its own set of problems and limitations associated with it. Lighting conditions for feature recognition, data precision of extracted features and various other pre-requisites have to be understood, to start a well functioning AR experience. We will discuss the various ways to apply AR to a real world situation and show problematic scenarios that have to be anticipated in development.

World Recognition and ARKit background

• How does the ARKit sense the world and make it possible for us, to integrate our 3D scene into the live video feed? We will take a look at the underlying principle of visual-intertial odometry and the resulting abilities of the recognition algorithms for connecting our hierarchy to the displayed backdrop. Coordinate systems are a key element to combine objects seamlessly with the sensed 3D point cloud data of the ARKit.

Intersection hit-testing and dynamic objects

• Once we have a sense of our surroundings, it is crucial to be able to test for a valid point in the background space, so we can use the intersection to dynamically create a new object at this position. Instantiation is the key to this and together with Prefabs we will discuss a setup for cloning content into our runtime 3D scene. This enables us to „place“ things in the world displayed by the video picture.

Plane recognition and anchoring

• ARKit provides us with various high level information, that we can exploit for positioning purposes. Planes allow us to apply orientation on content according to the surfaces of tables, walls, etc. Dynamic anchoring then makes these content elements move and position themselves accordingly in conjunction with the background scene. How to set this up, so it is connected to the various desired GameObjects, is a critical step for placing stuff in a fixed manner in the scene.

Light estimation and occlusion

• For a seamless integration when the content is placed, an estimation of the displayed lighting situation in the video feed is necessary. Fortunately, ARKit provides the user with some data of the light scenario. We then discuss ambient light influences on the content in Unity and integrate the ARKit results. Occlusion is another topic needed for convincing positioning of the prefabs, since they need to „hide“ behind walls or occluding real world elements like tables and such. The ARKit interface to this ability is discussed in detail as well.

Debugging and visualizing the world around you

• Often it’s easier to understand a scenario if it is graphically described in a visual overlay. How to show what ARKit is actually doing provides us with the benefit of understanding faulty behaviour in placement and orientation of our objects. This is done by visualizing certain properties of the underlying point-cloud detection and using these informations for a better insight into a displayed scenario.

Performance constraints of mobile devices

• Mobile development presents us with special challenges in performance, since a lot of information has to be processed and displayed on a tiny handheld device in the application of Augmented Reality. Proper 3D geometry and performance constraints on logic and behaviours are paramount to allow for a smooth execution with high framerates on the devices. Since there is also a video feed involved, further processing overhead is present. We will discuss typical performance hogs and how to circumvent them.

More information

Target group

The target audience is developers intending to start on Augmented Reality development with Unity and Apple’s ARKit on iOS devices.

Prerequisites

Attendees must have experience in C# development in Unity and have an understanding of setting up Unity scenes for deployment. Theoretical background about 3D coordinate systems and Augmented Reality theory in general is needed, and understanding of 3D geometry and shaders is a plus. Prior iOS development knowledge is a plus.

Organizational matters