How to develop a prototype for Microsoft HoloLens

If you’re a business or brand looking to develop a prototype for Microsoft Hololens, there’s never been a better time to get started. The next wave of transformative technologies are now rapidly beginning to emerge and augmented reality (AR) sits at the forefront of this transformation. Technology is moving from what we carry to what we wear. Today we’re exploring how to develop a prototype for the Microsoft Hololens.

Before we get into the guts of the content, it’s important to get a few things clear, especially if you’re thinking about how to extend your existing software offering onto the MS Hololens. As with any project, it’s essential to start with the end in mind and develop key success measures at the outset of your project. Generally speaking, this should be a numerical value and something you can measure. It also helps if you’re thinking about process efficiency as a starting point as there are obvious KPI’s you can use to get started. Hololens technology is incredible, but extending your software offering into AR is all about focusing on real value and delivering process efficiency gains rather than the technology itself. The first part of the process is to develop real-world use cases and to establish if you have a strong enough plan to support the development of a proof of concept or prototype.

Since its launch in 2015, The Microsoft Hololens has been generating significant interest throughout the global tech community. Big businesses are now starting to sit up and take notice of the Hololens and it’s potential capabilities when it comes to overcoming real-world commercial challenges. The Microsoft Hololens is the first device of its kind to alter the way in which we perceive solutions that alter reality and the world around us and the technology behind the hardware itself is both compelling and complex. For the uninitiated, the best way to describe the Hololens is that it enables businesses to merge together the real and virtual worlds, whereby physical objects in the real-world can coexist alongside digital objects in real-time. For many, the prevailing terms used to describe the technology are ‘Augmented Reality’ or ‘Mixed Reality’.

One of the associated complexities in terms of businesses and consumers adopting the Hololens is the cost of the device. The cost of the Microsoft Hololens is likely to clock in at nearly £3,000, however, there is some good news on this front if you’re looking to develop an app or prototype that can run on the device. The Universal Windows Platform enables developers to get started using an emulator, whereby once the app or prototype is developed, the technology will be capable of running on the Hololens too.

Hololens IDE’s and SDK’s

If you’re a business thinking about developing a prototype for the Hololens, it’s worth spending some time figuring out the development environment and how to setup your project. There are a bunch of options available from the outset. The Hololens is already being put to use across a wide variety of industry sectors, from financial services and healthcare to construction and space exploration. Within each industry sector, Hololens is already being used to enhance process efficiency, cut costs and drive new revenue streams. The Hololens has been designed by Microsoft to create process efficiency gains across all of these industry sectors, from big data visualisation in data centres to field worker operations and tools for surgeons. Given that the Hololens technology is relatively new, knowing where to start with your project can seem like a bit of a minefield. However, many of the technologies and development kits required to develop a prototype or application for Hololens have already been around for a while. In order to get started it’s worth considering the technical development environment and the toolkits that are currently available to use, these include: Windows 10 SDK, Unity, Vuforia, Microsoft Visual Studio and the Hololens Emulator (you’ll definitely need this unless you already own a Hololens device). When it comes to 3D modelling and design it’s worth exploring the following tools: 3DS Max, Autodesk Maya, Substance Painter and Adobe Photoshop. If your prototype concept utilises animation it’s worth checking out Biped for 3DS Max, Blender or Rigify. If you’re still unsure how to get started in the shortest possible timeframe, it’s worth checking out the Hololens section of the Microsoft website to learn about minimum system requirements and mandatory development tools. Once you’ve explored all of the above options and established which tools represent the best fit for your project requirements, you’re pretty much at first base and ready to get properly started designing and developing your Hololens app.

How to adapt holographic images for Hololens

Designing and adapting 3D objects for Hololens is much the same as developing 3D models for conventional use. The real differences only really become apparent when 3D models are being integrated via a holographic ecosystem such as the Hololens. Having spent considerable time using the Hololens, the experience itself is very impressive, however, if you’re serious about developing a usable prototype you’ll need to think about how to avoid what’s commonly known as ‘VR sickness’. If executed incorrectly, your Hololens app may cause the user to experience disorientation, nausea and general discomfort. Here at Mozenix, this is something we have experience in dealing with by deploying what’s referred to as a ‘near clip plane’. When developing a Hololens prototype using Unity, it’s absolutely essential that you adjust the near clipping plane to no less than a distance of 0.85m. Failure to factor this into your development will cause discomfort for the user as the 3D object will appear to be too close and literally cause their eyes to cross. There is also an optimal distance to consider when displaying 3D holographic objects in front of the user, generally speaking a distance of 2 metres represents the ideal solution (or as close to 2 metres as possible). This reduces what’s often referred to as ‘binocular rivalry’.  Without getting overly technical, the Hololens will project two pictures onto the left and right aspect of the device lens can be fully overlapped at a distance of approximately 2m from the users position. Sometimes, depending upon the type of application you’re developing for Hololens, sticking to these guidelines will not always be possible, in which case it’s recommended that objects should be positioned in what’s referred to as the ‘favourable zone’ which represents a distance of 1.25 to 5 metres from the user.

Another essential component of your development plans should be the frame rate of the application. In order to develop an optimal experience for your prototype, you should think about creating a frame rate of around 30 FPS (frames per second). If you manage to get this right you can create a slick and immersive holographic experience for the end user. However, if the frame rate is too small, you’ll cause AR sickness for the user. If the frame rate of your Hololens app is too high, the device wont be able to process the data fast enough as there are constraints when it comes to the processing power of the Hololens.

How to factor gaze recognition into your Hololens app

According to Microsoft, Gaze is: “the first form of input and is a primary form of targeting within mixed reality. Gaze tells you where the user is looking in the world and lets you determine their intent. In the real world, you’ll typically look at an object that you intend to interact with. This is the same with gaze”. The Microsoft Hololens calculates the orientation and overall position of the device users head (rather than their eyes) to establish their ‘gaze vector’. The best way to consider gaze vector is almost like a laser pointer directly ahead from right between the users eyes. As the Hololens user looks around the room, your prototype can intersect this beam, both with spatial mapping and it’s own holograms to establish which real-world or virtual object the user may be focusing on. When using the Microsoft Hololens, all interactions should derive their overall targeting from the gaze of the end user, as opposed to rendering or interacting with the users hand location. Once the interaction process has been initiated, hand movements can be used to control each gesture. With HUD’s (heads up display) you can create targeting sequences using either gaze or motion controllers.

Using gaze to interact with the Microsoft Hololens

When calculating the gaze of the end user, Microsoft Hololens records the position of the users head and processes over a gigabyte of data every single second. In order to gain a clear understanding of how this technology works, imagine a situation whereby a user is faced with multiple choices, in this instance a Hololens app that teaches users about the structure of our solar system and the position of different planets. Gaze technology works so that when the end user glances at a particular object, in this case a planet, the object itself is highlighted and relevant information is displayed beside the planet.

If you’re a business thinking about developing your first prototype for Hololens, you can use Unity to manipulate the gaze feature in a multitude of different ways. There is a technique you can deploy referred to as ‘billboarding’ that enables elements of the user-interface to always appear directly in the users field of view when they look in a specific direction. From a development perspective, this can be extremely useful when designing a prototype and an intuitive user interface that makes sense to the user. Developers can also deploy 3D cursors and gaze indicators that are capable of attracting the users attention and guiding them towards a specific object or 3D hologram.

How to create voice activated commands for Microsoft Hololens

Voice activated search and commands are a relatively new phenomenon. It’s estimated that voice activated toolkits such as Siri, Alexa and Cortana currently operate at roughly 95% accuracy. At present voice activated commands are considered to be under utilised by most technology users with the exception of Generation Z who tend to adopt the technology naturally. However, recent anlysis suggests that as the accuracy of voice commands increases from 95% to 99%, users will go from hardly using it at all, to using it all the time. Within the context of the Microsoft Hololens, using Cortana for voice control actually feels surprisingly intuitive. If you’re planning on developing a Hololens app or prototype, using voice functionality can be a great way to help users navigate your app. If you’re using Unity to program voice activated commands, there are a bunch of things you should consider. Firstly, voice commands should be short and sweet, this helps to simplify pronunciation recognition and is particularly helpful for detecting unusual accents or non-native speakers. Dont confuse the user with elaborate voice command instructions as this will have a damaging impact in terms of your user metrics. Secondly, when programming voice activated commands, consistency is absolutely crucial. You should focus on using one pre-defined command throughout the entire app for similar actions such as clicking or opening menus. You can also think about duplicating voice commands with alternatives that use a specific gesture instead. This helps to create a more rounded user experience and helps the user to navigate the app in a seamless manner, particularly when voice commands cant be recognised through unclear pronunciation of certain keywords or phrases.

How to use gestures when developing for Microsoft Hololens

Gestures enables users to interact with the holographic world and control specific functions when using the Hololens. They represent an innovative new approach when it comes to interacting with technologies that alter our surrounding reality. One of the key advantages of gesture based interactions are the fact that it enables the user to perform tasks with their hands. This is a particularly useful benefit across a variety of industry use cases, from surgeons performing operations, to field workers who need their hands to perform tasks. Gestures are used within the Hololens to perform actions such as clicking icons or opening menus. In fact, Hololens uses specific gestures to perform specific actions. For example, the gesture used to control the opening of the main menu is referred to as ‘bloom’. The bloom gesture can also be used to close down apps when you need to default back to the main menu. Other gesture types are referred to as continuous gestures. Continuous gestures always incorporate scrolling or rotating actions.

If you’re in the process of planning to develop an app for Hololens, or already in the process of developing one, it’s essential that you consider how best to utilise gestures to perform specific actions in order to create a seamless experience for the user. The Hololens should always respond promptly to each gesture, any delay will cause frustration for the user. You can achieve this by ensuring that gestures are promptly able to highlight icons, open apps and a variety of other useful functions. In the same way that you can program how to respond to a users gaze, you can also include code to warn the user that their gesture is out of range. Given the fact that Hololens, and mixed reality in general, is relatively new, it’s essential to understand the constraints of the device. Such limitations include the likes of object positioning or hand placement and sometimes the device will not always acknowledge these actions with the greatest degree of accuracy possible. Before you embark on a Hololens prototyping or app development project, it’s fundamentally important that you gain a thorough understanding of these constraints and limitations.

Commercial applications of Microsoft Hololens

So you’re thinking of developing an app or prototype for Microsoft Hololens? Chances are if you’re reading this blog, you’re one of the first to own a device (Microsoft has reportedly sold thousands rather than tens of thousands, let alone millions) or you’re early in the buying cycle and considering buying one at some stage in the future. If you’re seriously considering investing in a prototype, there are a few considerations to be absolutely clear about from the outset. Firstly, you must always start with the end in mind. You need to spend considerable time working out how you can measure the success of your project. This means developing a series of measurable KPI’s that will inform you whether or not the project has been a success. Secondly, your entire use case should stem backwards from your key success measures. Otherwise you’re just developing shiny technology with no real purpose or commercial benefit. If you’re running an operation with field workers, can Hololens be used to reduce call-out times or accelerate job completion? If you can develop a use case with a measurable outcome, you have the basis of a successful plan for a prototype and in these instances the technology can be applied across a number of industry sectors from education, entertainment and training to military, healthcare and financial services. There is a commonly misplaced assumption that Virtual and Augmented Reality are novelty technologies used solely for entertainment and gaming and that devices such as the Hololens lack commercial validity in terms of business use cases.

From an educational perspective, the Hololens has the potential to be hugely influential. In terms of creating an immersive environment for learning, few technologies can compete with AR. Within this specific context, the Hololens can be used to dramatically accelerate the learning process. Hololens can be used to deploy teaching materials remotely in real-time whilst the student can perform the required actions and receive instantaneous feedback. Perhaps more importantly, Hololens is a fantastic visualisation tool. This can be extremely useful in a training context whereby new employees can quickly get up to speed with complex processes and potentially dangerous machinery without actually coming into contact with physical machines or objects and placing them in risky situations. As well as demonstrating machinery and processes, the Hololens can be used an interactive tool that enables the trainee to experience how things function and operate.

Japan Airlines are now using the Hololens as a tool for training, enabling staff to handle enormous and sophisticated components for use in avionics systems and jet engines. Manufacturing production lines and construction environments are also a natural fit for the Hololens technology. The technology contains functionality that enables different Hololens users to experience the same reality in real-time, which is particularly useful for remote working scenarios. In essence, within these types of case studies, the ability to visualise images and 3D models in a new way enables decision makers to gain insight in a faster and more effective way. From car manufacturing (Volvo already utilises holographic computing for prototyping and construction purposes), to healthcare (microsurgery and operations) and financial services, Hololens is helping to solve real-world commercial problems and drive process efficiency, helping big companies to cut costs and drive new revenue streams.

If you’re a business or brand thinking about developing a prototype or app for Hololens, contact Mozenix today to kick-start the conversation.