Hololens IxD
I have been part of the core Shell Design team at Microsoft Hololens from the early days on and have worked with various versions of hardware. Defining interaction design patterns, mental models and micro interactions, I have also been a point to person for Hardware Software Design interaction such as Buttons, Hardware Input and LED power states and transition. Overall my role has been to lead the core user interaction and to engage directly with scientists, engineers, pms and research.
The Shell
_____
The Shell is made up of the world around the user and holograms in the system. This is not the traditional playground for Shell Interaction and here is where the metaphors of desktop may break down or iterated on. We call this the Mixed World. Me and my colleague Josh Walton have coined the term and drive the design in order to gain traction for the product as a hole. The Shell also of Start Menu, where user can place an app in the world, Holograms 2d or 3d and various manipulations including object and content. I have been the lead designer for all Shell manipulation and interaction.
A glance at Shell in Computing
__
CLI (Command Line Interpreter) in Windows
MS DOS - No Graphical User Interface
Windows 1.0 Shell Design - 1985
Note here that windows are snapped against one another, and no close button.
Windows 3.0 Shell Design - 1990
Icons as Shortcut metaphors for running applications, Minimize, Bar to Move Windows, Overlapping Windows.
Windows 95 Shell Design - 1995
Iconic Close button, Start Menu, Full size and Minimize buttons, Permanent Desktop Shortcuts, Volume control, Notification Center with Time, File Manager replaced by "Windows Explorer" - Web browser.
IPhone OS 1 Shell Design - 2007
In pocket.
No Windows, no files.
Only Apps, No App bar.
Apps take up full screen.
iOS 10 Shell Design - 2016
Sensors that detect location, orientation, contextual search and information, voice input and voice Assistance (SIRI) for navigation - The interface is affected deeply by it's inputs and sensors.
From Desktop to Mobile to World
Hololens Shell for Windows Holographic
This meant that now the metaphors of desktop, file system, the recycle bin, close button, scroll bar had to all be thought about in a different light. The design had to consider the world, surfaces, rooms, objects and people as input and as essential part of the way we interact with the world. A large part of my time at Hololens has been rethinking these metaphors and considering what new forms we might look at holograms, question and define what they are, how they behave and in what way we interact with them.
"Infinite Screen"
Gesture as Input to Start/ Home
Apps Coexist within my Space, Home.
Placement
When a user chooses an app tile from the Start Menu, the Start Menu is dismissed and the app enters placement mode.
The app will be automatically adjusted (in size and position) to conform to the real world in the direction of the user's gaze. When an app is placed, it will start running.
Once the user has placed their holograms in their world they can go into adjustment mode by clicking the "Adjust" button on any app's app bar or saying, "Adjust". In adjustment mode, users can move, scale and remove holograms. 3D holograms can be rotated. Some 2D holograms support vertically resizing the app to show more content. When finished, user selects the 'Done' button on the app bar or says, "Done".
There is also an ideal placement "zone" that is defined by the comfort level of the focal plane:
App Bar + UI
The app bar appears above the 2D view of an app. By default, it allows you to adjust and remove the 2D view. Different apps may add additional options, to the app bar. In the picture below, the Edge app has added the Scroll, Drag, and Zoom options to the center.
The Back button will navigate back to previously viewed screens in the app. It will stop when you reach the beginning of the experiences that have been shown in the app, and will not navigate to other apps.
Cortana is the personal assistant on HoloLens just like on desktop and phone. It can be used to perform various system tasks, such as launching apps, rebooting your device, or to look up information for users. Users can gaze and say the command at the App UI.
Manipulation
Manipulation gestures can be used to move, resize or rotate a hologram when you want the hologram to react 1:1 to the user's hand movements. One use for such 1:1 movements is to let the user draw or paint in the world.
As with all gestures, the initial targeting for a Manipulation gesture should be done by Gaze. Once the press starts, any manipulation of the object is then handled by hand movements, freeing the user to look around while they manipulate.
Movement
Scaling and Rotation
Environment Sensing
Spatial Mapping
Spatial Mapping provides a detailed representation of real world surfaces in the environment around Hololens, allowing developers to create convincing mixed reality experience. By merging the real world with the virtual world, an application can make holograms seem real. A big part of my role in Surface Reconstruction has been to work directly with engineering team to define rules and behaviors for how holograms move and interact with the surfaces, define occlusion, collision, physics and navigation. In addition, part of my role has been to cue to the user how the machine senses and how the user perceives the errors of human-machine.
