One of the big selling points of Microsoft’s Hololens augmented reality headset is that it is untethered, in that it doesn’t need a power cable. Areas like optical design, simulation software, and display technology all have to be optimised to minimise power usage, while also making the system cost-effective, but projection technology is also part of the equation to make the most efficient use of light.
Micro-electromechanical systems (MEMS) have transformed projection devices to a certain extent – Texas Instruments’ digital light processing (DLP) projectors, for instance, are MEMS-based devices containing millions of mirrors that move individually to produce high definition images. Each mirror can be tilted precisely for fine gradation in light intensity.
MEMS scanning mirrors can vary in size – up to 10mm – but the MEMS structures that drive the mirror to move in two axes independently are 3-4µm in width. ‘[MEMS processing] is a way to make very small, very precise drive structures. Even though the mirrors are relatively large, we are able to make high-speed drive mechanisms that use much smaller micro-fabricated structures surrounding the mirror,’ explained Dr Andrew Miner from Mirrorcle Technologies, which produces high-speed micro-mirror technology.
Another benefit is that MEMS devices can be fabricated using silicon chip manufacturing methods, which allows hundreds, or thousands, of devices to be made on single wafer, reducing costs with production volume. This is especially important for consumer devices like augmented or virtual reality headsets, where the cost of components has to be kept down.
About a quarter of Mirrorcle Technologies’ business is in the AR/VR space, according to Miner. ‘Innovators in AR/VR are developing highly complex and advanced optical systems to provide high quality displays, which are very closely coupled and work together with human vision systems,’ Miner stated. ‘Here, projection, eye tracking, and other modalities of optical beam steering with MEMS mirrors become enabling technologies, especially as integrators target size and cost reduction.’
The complexity of headsets can vary, but Miner described a situation where a VR headset might have a MEMS mirror creating independent images for each eye. ‘More advanced headsets will use several mirrors per eye in a more complex optical train to deliver images in the most natural way to the human eye,’ he said.
Head-up displays
There are now many exciting uses for virtual and augmented reality, extending way beyond the gaming world to enhance fields like education and tourism. For example, this summer, visitors to The Shard – London’s tallest building – can experience its new virtual slide that speeds through the city’s skyline at 100mph.
There is a huge amount of competition in the virtual reality market to design headsets that achieve a balance between high-performance, but whose size, design, and price point are at an acceptable level to consumers. ‘I think the biggest challenge continues to be form factor, especially for the head-mounted type displays,’ commented Michael O’Keefe, CEO of Greenlight Optics. ‘Whether it’s in a helmet or it’s a pair of glasses, achieving a good performance while maintaining a form factor that people actually want to wear is really the challenge.’
Optical engineering firm Greenlight Optics works with VR firms to design custom optical systems and projection engines based on MEMS devices. It will also design the illumination optics as well as the projection optics or, in the case of a wearable display, the eye-piece optics.
Most of the time, LED-based illumination is used for VR and AR applications, noted O’Keefe, because of lower costs and fewer eye safety considerations. However, occasionally, lasers will be used, whereby Greenlight Optics will employ DLP MEMS devices from Texas Instruments.
DLP chips consist of an array of micro-mirrors known as the digital micro-mirror device (DMD), an electrical input MEMS that allows developers to perform spatial light modulation. Each DMD contains up to eight million micro-mirrors that can be controlled individually.
During operation, the DMD controller loads each underlying memory cell with a ‘1’ or a ‘0’. Next, a mirror reset pulse is applied, which causes each micro-mirror to be electrostatically deflected about a hinge to the associated plus or minus degree state. In a projection system, the plus degree state corresponds to an ‘on’ pixel, and the minus degree state corresponds to an ‘off’ pixel. Greyscale patterns are created by programming the on/off duty cycle of each mirror, and multiple light sources can be multiplexed to create full RGB colour images.
‘We have designed other systems using MEMS scanning mirrors, the types of MEMS device with either two mirrors or 2D MEMS. But the use of DLP is more typical,’ O’Keefe said. ‘DLP is very optically efficient as it does not require polarised light.’
For augmented reality – head-up displays (HUDs) in particular – DLP MEMS are used because these types of headsets have to compete with sunlight, meaning that the system has to be very bright. This is not the case for head-mounted VR devices typical of the gaming world, which are fully immersive and do not let any light into the headset. One application for an augmented reality HUD is in cars, where images are overlaid onto the road ahead from behind the wheel, providing the driver with information related to speed, navigation and position.
‘Because [HUDs] have to be bright in order to compete with sunlight, it’s important that we’re designing the most light-efficient system possible, so that the electrical power can be minimised, and we can achieve the brightness that’s needed within the etendue constraints of the system,’ explained O’Keefe.
‘It’s very important that the systems we design using DLPs are efficient, and that the illumination system is designed to capture as much light from the LEDs or lasers as possible and not waste that light,’ he added.
To do this, the most important thing is having a good etendue match between the light sources and the size of the DMD, and all the optics throughout the light engine, so that the light is not wasted, O’Keefe explained.
Generally, the company works at a low F number – which refers to the ratio of the system’s focal length to the diameter of the entrance pupil – to maximise the amount of light that’s captured from the LEDs and therefore maximise the brightness of the system. ‘The more recent DLP chips are an improvement over the previous version, because they allow us to work at a lower F number,’ said O’Keefe.
A DLP MEMS device contains millions of micro-mirrors that can tilt independently
Custom optics will then be designed depending on the headset’s packaging constraints and the objectives of the customer. In the case of a HUD, the custom projection optics will generally be off-axis, to allow the projection unit to be fitted in an appropriate place in the vehicle or helmet. A custom combiner will also be used to reflect the displayed image back to the eye.
In addition, the company will design a custom total internal reflection (TIR) prism, which is used to direct the light onto the DMD, so that when the micro-mirrors are in the on state, the light reflected off each mirror will reflect through the TIR prism and out through the projection lens or viewing optic. If the pixel mirror is tilted to the off state, the light is reflected not in the projection lens, but dumped into an area painted black so that it is not visible.
‘It’s basically a way to control every pixel to be on or off using tilted mirrors,’ O’Keefe said.
Although VR and AR devices are moving into new applications areas and are expected to change everyday experiences in the coming decade, they are still not at a point where they are commonly seen in homes, or education or industrial environments. Challenges associated with performance, size, and cost remain.
In terms of the DLP MEMS technology, smaller pixels will still be required before VR and AR become common, according to O’Keefe. ‘I think consumers and professionals are always going to push for the highest resolution possible at a smaller size, and a lot of that is dictated by the size of the pixel. They have become a lot smaller in the last few years, but I think there is probably still more shrinking that’s required on the pixel side,’ he noted.
In addition, he sees the emergence of affordable RBG lasers, or very efficient LEDs in the right form factor, as important for realising the virtual and augmented reality dream.
Tracking players in the room
Besides the headsets, MEMS devices are used in other AR/VR applications, such as inside a games console to track individuals as they play. Unlike a DLP that contains an array of many mirrors, these devices consist of one single mirror, normally between 1 and 10mm in diameter. According to Miner, what’s unique about these scanning mirrors is that they can move in two axes independently while having very fast response times. ‘This allows light to be directed in 3D space, or light to be captured from space… and directed to a sensor,’ he said.
The advantage of using a single, analogue control mirror, according to Miner, is that a wide scene can be scanned and then specific regions focused in on. ‘You can zoom in on one corner of the scene and image in very high resolution if needed, and zoom back out to image a much broader scene,’ he explained.
The company recently demonstrated a MEMS-based device that could be used to track a player, or multiple players, in a virtual reality or gaming environment.
‘In this demo, we use our mirror to scan a beam of light across the room, and we are searching for a specific highly-reflective material,’ explained Miner. ‘So if a player puts this reflective piece of tape on their t-shirt, for example, we are able to scan the room and locate this piece of tape.’
Once located, the mirror can then monitor and scan around the reflective tape on the player as they jump, duck, or turn in space. The MEMS device would work in conjunction with a camera that’s used to image the scene.
‘From this demo we have engaged with several VR players to explore it further,’ Miner said.
The concept of tracking players is not new, and there are various different approaches being looked at. For example, Microsoft’s Xbox games console images the room with infrared cameras, and then shape recognition software is used to recognise the form of a body in order to identify players, Miner noted. ‘That is a different way to do it and there are different advantages to different methods,’ he said.
