Augmented Reality: Enhancing Your Field of Vision
Augmented Reality: Enhancing Your Field of Vision
The Microsoft HoloLens allows users to video chat with experts remotely for step-by-step guidance. Image: Microsoft
When Tony Stark suits up as Iron Man, his helmet drops down and a heads-up display covers his face. While he can still see the outside world, the display overlays what he sees with readings on altitude, targets, power, and all the things a busy superhero needs to survive the workday.
Stark is looking at the world through the vision of augmented reality, or AR. Now, real-life engineers and technicians are starting to do the same.
Where virtual reality isolates users from the world around them, augmented reality offers the best of both natural vision and digital information. It can provide real-time assistance to workers in remote locations, help workers learn new skills, or walk technicians through a repair. It provides a mobile and intuitive way for engineers to interact with sensor data, digital twins, and their physical counterparts.
“AR is doing for humans what the concept of the “Internet of Things” has done for machines,” said Jim Heppelmann, CEO of PTC, a computer software and services company based out of Boston. It gives them a simple way to access the information the IoT collects just by looking.
Automation, manufacturing, and design companies have started to pay attention—and are exploring the value of this disruptive technology.
Several industries are diving into the world of augmented reality. Companies like Airbus, Vestas, and Howden have invested in building up their AR infrastructure for design, maintenance, and training purposes. Image: Microsoft
“We recognize that there's a lot of potential in AR for training and development. It's immersive, engaging, and creates excitement to those that participate in it,” said Walter Davis, head of talent and learning applications at Aggreko, a supplier of temporary power generation, heating, and cooling equipment in Houston. “There’s a big potential for this technology, not just for training but also as an enterprise solution.”
“We’ve been on this journey for the last five years,” said Mike Campbell, executive vice president of AR at PTC. “At first, the conversation with our industrial customers was about what AR is and how it is different than virtual reality. But now the conversation that we’re having is about the value AR can bring, how it can be used to reduce scrap and rework, improve worker efficiency, and perform your job more safely.”
Augmented reality certainly has the power to disrupt the way some industrial manufacturers operate, but exactly how are companies and workers adopting to this new tool?
ASME's Mechanical Engineering Special Report on manufacturing explores the world of augmented reality and features a deep dive into AR with industry videos, demos, our ASME TechCast podcast, a whitepaper on AR, and more.
Learn more here: Manufacturing Special Report on Augmented Reality
The Many Realities: Augmented vs. Virtual
Before diving into augmented reality, we need to define how it compares with its sibling visualization technology, virtual reality (VR).
“With augmented reality, you’re interacting with the physical and the digital worlds together, taking 2D digital images or data and overlaying it on the real world,” said Matt Fleckenstein, senior director of mixed reality at Microsoft. “And in virtual reality, you’re immersed in a purely digital world.”
AR and VR are at opposite ends of the visualization technology spectrum. VR removes the real world altogether and replaces it with a digital one. The immersive environment can be photorealistic or animated, depending on the programming. This is one reason why Oculus Rift and HTC Vive, two popular VR systems used for gaming and video, are often promoted as the entertainment systems of the future.
Top Story: Augmented Reality Not Quite Real for Engineers
VR enables users from remote locations to collaborate on CAD models and simulations. They can, for example, virtually walk through a 3D simulation environment rather than look at it on a 2D computer screen.
VR hardware tools disconnect users from their environment by swapping headsets for the field of vision and hand controllers for touch. All the while, cameras capture and simulate your movement and location in the digital landscape. This isolation limits VR’s use in the field.
On the other side of the spectrum lies augmented reality. It doesn’t replace the world; it enhances it. A famous example of AR is the smartphone game Pokémon Go. The game’s world is built by using a smartphone camera to capture the outside world and digitally adding Pokémon characters. This hybrid environment can only be seen on the phone.
Basic AR applications use any device with a camera. Several use the powerful computing capabilities of smartphones and tablets to deliver some level of AR to the masses.
In manufacturing, AR’s sweet spot lies in the middle of the visualization technology spectrum, where it combines AR’s field-of-view enhancements with VR’s sensor-driven visor headsets and hand gesture controls.
Microsoft’s HoloLens headset, for example, houses a powerful graphics processor that overlays its surroundings with virtual images. These range from 3D reference models and 2D schematics and visual instructions to simulations and teleconferences with experts who use the HoloLens cameras to view a problem.
Recommnded for You: When Robots Interact with Humans
Users manage the screens with hand gestures that are captured by built-in cameras. This lets users interact with the digital world while freeing their hands to manipulate physical objects.
Epson, another contender, designed its Moverio Assist for inspection, remote assistance, and real-time collaboration. The company’s Moverio BT-300 and BT-350 smart glasses consist of a bright, sharp Si-OLED display, built-in cameras, and Wi-Fi capability.
Like the HoloLens, Moverio can guide users through an assembly or procedure by displaying documents and schematics in the user’s field of view. Users can make the documents transparent to keep their field of vision clear or minimize them. They can also use Moverio’s platform cloud service to call remote experts, who see the user's point-of-view through built-in cameras.
Augmented reality helps connect the world of data and computer models to real-life design. Engineers can visualize their 2D drawings into a 3D environment. Image: Microsoft
The Magic Leap One excels at spatial computing. The headset’s sensing technology maps the real-world environment, then programs the digital content to interact with the physical world as well as the user. The result is a mixture of AR and VR.
Engineers can use this to, say, populate a real factory with digital models of machines and processes to ensure everything fits with no obstructions. Magic Leap also readjusts those virtual objects, so their location remains aligned with the user’s natural eye focus.
Many headsets for manufacturing fall into this middle ground between the AR and VR, letting users see their surroundings enhanced with digital information. And engineers are finding ways to put them to work.
Best Use Cases for AR
The most common applications for AR include training, assistance, and production.
Training: More than 2 million skilled manufacturing jobs could go unfilled by 2025, according to Deloitte, a top management consulting firm. This is because skilled baby boomers are reaching retirement age, and manufacturers are having trouble attracting younger workers to replace them. The result is a skills gap in everything from equipment setup and assembly to operation and maintenance.
AR could help. A study by International Data Corp. found that 66 percent of respondents offer remote AR training to workers.
Another 52 percent use AR to capture and preserve training sessions for playback purposes.
“Millennial engineers, they are digital natives, and they thrive on getting information this way,” PTC’s Campbell said. “We can articulate the value of AR by teaching engineers how to do the job faster.”
Instead of having to read a manual on how to install and run a hydraulic pump, for example, technical professions could use an AR headset to simulate a pump. Unlike doing this on a display screen, they get to look at the equipment in three dimensions. They could then practice how to install the hardware, operate, and repair the pump—and review that training on the job.
IDC also found that AR accelerates new hires’ learning curve. Seventy-five percent of survey participants found that workers learned faster and retained information better by using AR headsets.
For seasoned workers, AR visuals can help them learn the ins and outs of new hardware, instructing them on installation, status, and performance. This reduces errors associated with starting up new equipment.
Editors’ Pick: Engineering Safety with Smart Helmets
Assistance: AR brings experts into the field with a level of collaboration not previously possible. While tablets and smartphones support remote video chats, AR enables the remote expert to see exactly what the user sees. Since the users’ hands are free, they can implement repairs instantly as the remote expert guides them through the process. AR also lets both user and expert share documentation, making each intervention a teachable moment.
A shared view of the work environment provides a visual way to exchange information and solve technical issues. This is why 60 percent of IDC study respondents ranked transfer of knowledge between new and seasoned employees as their most important reason for using AR.
AR is also good for bringing together multiple experts for a live video conference. In fact, 42 percent of IDC respondents said video conferencing was an important reason to buy an AR system.
Best of all, AR does not disrupt existing infrastructure or require extra or complicated equipment. Users simply plug into the existing Wi-Fi and dial an expert.
Production: AR is making a place for itself in the factory. Its ability to reference CAD models, drawing schematics, and plant layouts in a real-world environment is useful for engineers looking to optimize a plant’s layout and efficiency.
According to the IDC, 82 percent of respondents were positive about using AR in manufacturing. Their number one reason was to boost manufacturing efficiency, followed by reducing the time gap between training and operations.
Moviero Assist platform helps users in the field with inspection and repair by highlighting parts that need to be maintained or replaced. Image: Moverio
One popular use of AR is to layout production lines. While engineers can do this using 3D CAD models, the representation on the display does not always reflect the as-built environment. An engineer might design a manufacturing or assembly line, only to find out that a support pillar or low pipe makes that configuration impossible. The ability of AR to superimpose virtual equipment over a real-world space ensures a better fit on the factory floor.
A plant engineer could simulate a production run to see how all the elements of the new line worked together, or how a new machine might interact with already installed hardware. Operationally, IoT connectivity lets users review the state of machines as they walk around the floor.
AR quality assurance enables engineers to validate whether a part was made correctly. They can, for example, compare a manufactured part to its 3D design model. While CAD models have played this role before, AR takes it one step further by enabling technicians to overlay 3D models with actual parts, making it easier to visualize any flaws.
At PTC’s LiveWorx event this year, Howden, which provides air and gas handling equipment for industrial facilities, explained how AR brings data to life.
The company uses PTC’s Vuforia Studio to create interactive AR service and training applications from existing 3D models, said Maria Wilson, Howden’s global leader for data-driven solutions.
“The way we’re using the AR experience is to bind it to IoT data in the background areas, using the visualization method,” Wilson said. “We’re using AR to create virtual constructions that are deployed in the factory for workers.
“Our design engineers are used to looking at a 2D drawing to understand how the chemical plant operates,” she continued. “Today, they can see a holographic projection of that 2D drawing. Our engineers feel they can contribute more to the design and are being more efficient by using AR, and when we do that, we’re bringing the IoT sensor data to life in real-time.”
The Future of the AR Workforce
How far along are we from everyone having an AR headset in their tool bag? According to IDC data, not that far, because AR’s ability to quickly boost workforce capabilities brings a rapid return on investment.
According to the IDC, 62 percent of respondents claimed a positive ROI from implementing AR. Also, 65 percent claimed better knowledge transfer among employees and 60 percent of them said it improved employee efficiency.
Even so, the main hurdle for most customers remains price. That problem will likely solve itself as vendors continue to introduce more features for less money. Many see manufacturing and product service as important markets, so they are also developing specialized systems for these areas.
The learning curve for new technology is often a problem. Makers of AR devices, however, claim the technology to be fairly intuitive. This is especially true for young engineers who grew up with smartphones and sophisticated gaming equipment. The high-tech allure of AR probably makes manufacturing more attractive by association.
“When you look at that ROI analysis, the productivity gains that you’re getting for each employee are obvious,” Microsoft’s Fleckenstein said. “And as you are helping them to do their job and to be more effective at their job, employee satisfaction and retention goes up dramatically. This gives companies the ability to start closing the skills gap.”
It’s a compelling argument. And the economics will only improve as hardware and software improve, and prices come down. One day, all manufacturing plants may have some level of AR integrated into their workflow.
Iron Man may have had it first, but one day soon, AR will be part of every engineer’s toolbox.
Carlos M. Gonzalez is special projects manager.
Stark is looking at the world through the vision of augmented reality, or AR. Now, real-life engineers and technicians are starting to do the same.
Where virtual reality isolates users from the world around them, augmented reality offers the best of both natural vision and digital information. It can provide real-time assistance to workers in remote locations, help workers learn new skills, or walk technicians through a repair. It provides a mobile and intuitive way for engineers to interact with sensor data, digital twins, and their physical counterparts.
“AR is doing for humans what the concept of the “Internet of Things” has done for machines,” said Jim Heppelmann, CEO of PTC, a computer software and services company based out of Boston. It gives them a simple way to access the information the IoT collects just by looking.
Automation, manufacturing, and design companies have started to pay attention—and are exploring the value of this disruptive technology.
Several industries are diving into the world of augmented reality. Companies like Airbus, Vestas, and Howden have invested in building up their AR infrastructure for design, maintenance, and training purposes. Image: Microsoft
“We’ve been on this journey for the last five years,” said Mike Campbell, executive vice president of AR at PTC. “At first, the conversation with our industrial customers was about what AR is and how it is different than virtual reality. But now the conversation that we’re having is about the value AR can bring, how it can be used to reduce scrap and rework, improve worker efficiency, and perform your job more safely.”
Augmented reality certainly has the power to disrupt the way some industrial manufacturers operate, but exactly how are companies and workers adopting to this new tool?
ASME's Mechanical Engineering Special Report on manufacturing explores the world of augmented reality and features a deep dive into AR with industry videos, demos, our ASME TechCast podcast, a whitepaper on AR, and more.
Learn more here: Manufacturing Special Report on Augmented Reality
The Many Realities: Augmented vs. Virtual
Before diving into augmented reality, we need to define how it compares with its sibling visualization technology, virtual reality (VR).
“With augmented reality, you’re interacting with the physical and the digital worlds together, taking 2D digital images or data and overlaying it on the real world,” said Matt Fleckenstein, senior director of mixed reality at Microsoft. “And in virtual reality, you’re immersed in a purely digital world.”
AR and VR are at opposite ends of the visualization technology spectrum. VR removes the real world altogether and replaces it with a digital one. The immersive environment can be photorealistic or animated, depending on the programming. This is one reason why Oculus Rift and HTC Vive, two popular VR systems used for gaming and video, are often promoted as the entertainment systems of the future.
Top Story: Augmented Reality Not Quite Real for Engineers
VR enables users from remote locations to collaborate on CAD models and simulations. They can, for example, virtually walk through a 3D simulation environment rather than look at it on a 2D computer screen.
VR hardware tools disconnect users from their environment by swapping headsets for the field of vision and hand controllers for touch. All the while, cameras capture and simulate your movement and location in the digital landscape. This isolation limits VR’s use in the field.
On the other side of the spectrum lies augmented reality. It doesn’t replace the world; it enhances it. A famous example of AR is the smartphone game Pokémon Go. The game’s world is built by using a smartphone camera to capture the outside world and digitally adding Pokémon characters. This hybrid environment can only be seen on the phone.
Basic AR applications use any device with a camera. Several use the powerful computing capabilities of smartphones and tablets to deliver some level of AR to the masses.
In manufacturing, AR’s sweet spot lies in the middle of the visualization technology spectrum, where it combines AR’s field-of-view enhancements with VR’s sensor-driven visor headsets and hand gesture controls.
Microsoft’s HoloLens headset, for example, houses a powerful graphics processor that overlays its surroundings with virtual images. These range from 3D reference models and 2D schematics and visual instructions to simulations and teleconferences with experts who use the HoloLens cameras to view a problem.
Recommnded for You: When Robots Interact with Humans
Users manage the screens with hand gestures that are captured by built-in cameras. This lets users interact with the digital world while freeing their hands to manipulate physical objects.
Epson, another contender, designed its Moverio Assist for inspection, remote assistance, and real-time collaboration. The company’s Moverio BT-300 and BT-350 smart glasses consist of a bright, sharp Si-OLED display, built-in cameras, and Wi-Fi capability.
Like the HoloLens, Moverio can guide users through an assembly or procedure by displaying documents and schematics in the user’s field of view. Users can make the documents transparent to keep their field of vision clear or minimize them. They can also use Moverio’s platform cloud service to call remote experts, who see the user's point-of-view through built-in cameras.
Augmented reality helps connect the world of data and computer models to real-life design. Engineers can visualize their 2D drawings into a 3D environment. Image: Microsoft
Engineers can use this to, say, populate a real factory with digital models of machines and processes to ensure everything fits with no obstructions. Magic Leap also readjusts those virtual objects, so their location remains aligned with the user’s natural eye focus.
Many headsets for manufacturing fall into this middle ground between the AR and VR, letting users see their surroundings enhanced with digital information. And engineers are finding ways to put them to work.
Best Use Cases for AR
The most common applications for AR include training, assistance, and production.
Training: More than 2 million skilled manufacturing jobs could go unfilled by 2025, according to Deloitte, a top management consulting firm. This is because skilled baby boomers are reaching retirement age, and manufacturers are having trouble attracting younger workers to replace them. The result is a skills gap in everything from equipment setup and assembly to operation and maintenance.
AR could help. A study by International Data Corp. found that 66 percent of respondents offer remote AR training to workers.
Another 52 percent use AR to capture and preserve training sessions for playback purposes.
“Millennial engineers, they are digital natives, and they thrive on getting information this way,” PTC’s Campbell said. “We can articulate the value of AR by teaching engineers how to do the job faster.”
Instead of having to read a manual on how to install and run a hydraulic pump, for example, technical professions could use an AR headset to simulate a pump. Unlike doing this on a display screen, they get to look at the equipment in three dimensions. They could then practice how to install the hardware, operate, and repair the pump—and review that training on the job.
IDC also found that AR accelerates new hires’ learning curve. Seventy-five percent of survey participants found that workers learned faster and retained information better by using AR headsets.
For seasoned workers, AR visuals can help them learn the ins and outs of new hardware, instructing them on installation, status, and performance. This reduces errors associated with starting up new equipment.
Editors’ Pick: Engineering Safety with Smart Helmets
Assistance: AR brings experts into the field with a level of collaboration not previously possible. While tablets and smartphones support remote video chats, AR enables the remote expert to see exactly what the user sees. Since the users’ hands are free, they can implement repairs instantly as the remote expert guides them through the process. AR also lets both user and expert share documentation, making each intervention a teachable moment.
A shared view of the work environment provides a visual way to exchange information and solve technical issues. This is why 60 percent of IDC study respondents ranked transfer of knowledge between new and seasoned employees as their most important reason for using AR.
AR is also good for bringing together multiple experts for a live video conference. In fact, 42 percent of IDC respondents said video conferencing was an important reason to buy an AR system.
Best of all, AR does not disrupt existing infrastructure or require extra or complicated equipment. Users simply plug into the existing Wi-Fi and dial an expert.
Production: AR is making a place for itself in the factory. Its ability to reference CAD models, drawing schematics, and plant layouts in a real-world environment is useful for engineers looking to optimize a plant’s layout and efficiency.
According to the IDC, 82 percent of respondents were positive about using AR in manufacturing. Their number one reason was to boost manufacturing efficiency, followed by reducing the time gap between training and operations.
Moviero Assist platform helps users in the field with inspection and repair by highlighting parts that need to be maintained or replaced. Image: Moverio
A plant engineer could simulate a production run to see how all the elements of the new line worked together, or how a new machine might interact with already installed hardware. Operationally, IoT connectivity lets users review the state of machines as they walk around the floor.
AR quality assurance enables engineers to validate whether a part was made correctly. They can, for example, compare a manufactured part to its 3D design model. While CAD models have played this role before, AR takes it one step further by enabling technicians to overlay 3D models with actual parts, making it easier to visualize any flaws.
At PTC’s LiveWorx event this year, Howden, which provides air and gas handling equipment for industrial facilities, explained how AR brings data to life.
The company uses PTC’s Vuforia Studio to create interactive AR service and training applications from existing 3D models, said Maria Wilson, Howden’s global leader for data-driven solutions.
“The way we’re using the AR experience is to bind it to IoT data in the background areas, using the visualization method,” Wilson said. “We’re using AR to create virtual constructions that are deployed in the factory for workers.
“Our design engineers are used to looking at a 2D drawing to understand how the chemical plant operates,” she continued. “Today, they can see a holographic projection of that 2D drawing. Our engineers feel they can contribute more to the design and are being more efficient by using AR, and when we do that, we’re bringing the IoT sensor data to life in real-time.”
The Future of the AR Workforce
How far along are we from everyone having an AR headset in their tool bag? According to IDC data, not that far, because AR’s ability to quickly boost workforce capabilities brings a rapid return on investment.
According to the IDC, 62 percent of respondents claimed a positive ROI from implementing AR. Also, 65 percent claimed better knowledge transfer among employees and 60 percent of them said it improved employee efficiency.
Even so, the main hurdle for most customers remains price. That problem will likely solve itself as vendors continue to introduce more features for less money. Many see manufacturing and product service as important markets, so they are also developing specialized systems for these areas.
The learning curve for new technology is often a problem. Makers of AR devices, however, claim the technology to be fairly intuitive. This is especially true for young engineers who grew up with smartphones and sophisticated gaming equipment. The high-tech allure of AR probably makes manufacturing more attractive by association.
“When you look at that ROI analysis, the productivity gains that you’re getting for each employee are obvious,” Microsoft’s Fleckenstein said. “And as you are helping them to do their job and to be more effective at their job, employee satisfaction and retention goes up dramatically. This gives companies the ability to start closing the skills gap.”
It’s a compelling argument. And the economics will only improve as hardware and software improve, and prices come down. One day, all manufacturing plants may have some level of AR integrated into their workflow.
Iron Man may have had it first, but one day soon, AR will be part of every engineer’s toolbox.
Carlos M. Gonzalez is special projects manager.
Related Content
Exoskeleton Control Improved Through Machine Learning
Oct 22, 2024
Special control software controls the electric motors so that the exoskeleton can match its movements to the patient’s intention. A machine learning algorithm may provide better control.
Microscale Hydraulic Gripper Features Soft Touch
Sep 18, 2024
The shape memory polymer that makes up the gripper allows it to be more efficient than a purely hydraulic system.
Engineer Challenges Aren't All Project-Related
Jul 31, 2024
An engineer’s job is to solve problems, but increasingly these professionals face unprecedented challenges that are not all coming from their projects.
Podcast: AI and the Industrial Internet of Things
Jul 29, 2024
Learn how machine learning systems can sift through torrents of data to uncover ways to optimize manufacturing operations.