Reporting from the EMVA’s conference in Copenhagen, Greg Blackman discovers how angular illumination and computational imaging can create gigapixel resolution using low-cost optics
In 2016 at the Zeiss symposium Laura Waller from UC Berkeley gave a talk about computational imaging. She presented work that turned microscope images from a five-megapixel camera into an image equal to one gigapixel resolution.
The team at UC Berkeley used a cheap LED array to illuminate the scene under the microscope with a random pattern and then, with a mathematically complex technique called Fourier Ptychography, reconstructed the final, high-resolution image.
Waller’s presentation was a something of a revelation to Dr Christian Hörr, head of software development at Carl Zeiss Optotechnik, who told the story of seeing these images at the 2016 Zeiss symposium during a talk of his own at the European Machine Vision Association’s business conference, in Copenhagen on 17 May.
‘When I saw this picture for the first time I thought it was fake, but it’s not,’ he said. He went on to say that, thinking of the potential consequences of the work – turning five-megapixel images into a one-gigapixel image using illumination and computation – ‘this is something that really threatens the core business of Zeiss’.
‘Zeiss in the past was pushing [development] towards optics, towards hardware. If you can now get away with a smartphone and a simple LED array, which might cost less than $100, and you can achieve the same kind of quality as a high-end microscope, that’s really threatening for Zeiss,’ he stated.
‘I’m glad to see that Zeiss takes these things very seriously,’ he added, saying the cooperation between Zeiss and UC Berkeley is a close one.
Zeiss now builds angular illumination into some of its products, namely its Smartzoom 5 digital microscope which contains eight LEDs that can be activated and deactivated individually.
The microscope will capture eight different images that are illuminated in a known, specific way. By understanding which images are neighbours and calculating weighted averages, an image can be produced with all the shadows and glare removed, for instance. This has applications in PCB inspection, for removing glare from images of the boards.
Zeiss’s Comet Pro fringe projection system incorporates 15 LEDs, eight ambient LEDs and seven additional LEDs that illuminate the scene under known conditions. This system can combine fringe projection with photometric stereo, which Hörr said results in the effective resolution of the system increasing by ‘at least a factor of two, and we think we can go further’. The system uses a GPU to do the computation.
Using this device for sheet metal inspection in automotive production, for example, highlights surface defects, but it also means stamped holes can be inspected, which wouldn’t be possible with standard fringe projection. By illuminating from different directions and combining the images gives a continuous shadow around the hole, which produces a 2D image edge at high contrast. On top of that, glare can be removed. Hörr said the technique can capture the boundary points around a hole with sub-pixel accuracy.
There are also applications of angular illumination and computational imaging in the consumer space, for producing photorealistic renderings of items like shoes for web images that can be rotated 360°.
Zeiss's Smartzoom 5 digital microscope can remove glare from images by using angular illumination. Credit: Zeiss
Hörr concluded his presentation with a plea to all the camera suppliers at the EMVA conference to not build the next 100-megapixel camera, because it creates more data which is hard to process quickly. ‘Please think about adding some LEDs to your camera, some interfaces so that we can control the light sources,’ he said.
‘It takes more time to acquire these images and it requires experts to implement the algorithms, but we think it’s worth the effort,’ he added.
Measuring the invisible
Removing the glare from images is particularly useful when inspecting very shiny surfaces. Karri Niemelä, CTO of Finnish firm FocalSpec, presented the company’s line confocal imaging technology during the EMVA conference, which is able to capture 3D topography of very shiny surfaces, along with tomography data and 2D intensity images of transparent objects.
FocalSpec’s chromatic confocal imaging technique gives 70nm z-axis repeatability and can measure up to 10 million 3D data points per second. Its next system, set to launch in Q3 2019, will operate at 14,000 lines per second (2,048 points per line).
The measurement principle uses optics to split white light into a continuous spectrum at the focal plane, and then the reflected light is measured. A shift in the wavelength peak gives information about how far away the object is from the lens and therefore the height profile.
Niemelä said the technology could be used to inspect smartphone curved glass enclosures for defects and to view imperfections inside the display. Tomographic imaging of a camera module lens assembly can be achieved, as well as 3D vision tasks like PCB inspection.
At the conference, the EMVA also recognised Dr Johannes Meyer, a PhD graduate from the Karlsruhe Institute of Technology, with the Young Professional Award for his work on light field methods for inspecting transparent objects.
Top Image: Sample datasets for multiplexed illumination coding in Fourier Ptychography. The captured images correspond to each LED pattern.
Credit: Lei Tian, Xiao Li, Kannan Ramchandran, and Laura Waller, 'Multiplexed coded illumination for Fourier Ptychography with an LED array microscope,' Biomed. Opt. Express 5, 2376-2389 (2014)
