Augmenting Air Combat Training

Almost from the birth of powered flight, pilots have trained with various types of flight simulators.  But there's a crucial difference between being realistic and being real, especially when it comes to aerial combat. What if it were possible to combine imitation and reality?

That's what a California-based startup called Red 6 has accomplished with its Airborne Tactical Augmented Reality System (ATARS), which puts advanced augmented reality technology into the cockpit of an airplane. With ATARS, a pilot can fight virtual enemies, practice aerial refueling and formation flying, and work through various combat scenarios, all while pulling Gs in a high-speed, dynamic real-world environment.

A symbiosis between Hollywood visual effects, augmented reality technology, and military aerial combat training, the ATARS concept can trace its origins back to what was essentially a sophisticated videogame created by a small startup comprised of restless visual effects designers and tinkerers, including Red 6 co-founder and Chief Product Officer Glenn Snyder. "We basically turned a Roush Mustang into a giant videogame controller and let people drive around a race course," he remembered.  Snyder and his colleagues later expanded the system into a VR game that allowed "drivers" in various locations to race each other virtually.

Then a producer friend called.  "He said, we kind of have a crazy idea,” said Snyder. “We might want to put that thing you guys made into a plane."  Snyder initially thought the idea ludicrous. "Who the heck would let us put this thing in a plane and fly it around?  It's just outrageous. No one's going to do that."

But that's precisely what former RAF fighter pilot Dan Robinson and filmmaker Nick Bicanic wanted to do. "Dan just said, I'm building a plane. We could put it in mine," Snyder said.  "He was trying to figure out how to solve a problem with synthetic training and how to make it feel more real. We all realized we had pieces of what we were trying to build and we needed to get together." 

Editor’s Pick: Podcast: Leveraging Digital Twin for Aerospace and Defense

Robinson, a former Tornado pilot and the first non-American to fly the F-22 Raptor, was concerned with a worsening crisis in Air Force pilot training. Developing and honing the skill set necessary to fly the world's hottest planes is both expensive and risky. Aside from the costs of fuel, maintenance, and operating the aircraft, there's the ever-present possibility of the catastrophic loss of both the plane and the pilot in case of accident. Added to this are the inadequate ability of current U.S. aircraft to simulate the capabilities of potential adversaries for training purposes, and an ongoing shortage of pilots to fly them. Robinson realized that an AR-based synthetic training system could be the answer. 

Developing 3D Capability for Aerial Combat 

 
The three founded Red 6 in early 2018.  But translating an existing VR system from the two-dimensional environment of race car driving to the three-dimensional world of aerial combat wasn't a simple proposition. The first issue is orientation: making sure the ATARS has an accurate picture of where it is and what it's doing. Red 6's testbed aircraft is a modified Berkut 360, a piston-powered plane derived from Burt Rutan's Long-EZ. "The aircraft that we're flying in particular is extremely maneuverable," Snyder said. That posed a big challenge: keeping up with the plane's ability to move and change direction on a dime.

"We can get a pretty clean signal coming out of the GPS, but how do you read that data quickly? And how do you use it in a way that's not introducing some new inaccuracies? We found a unit that kicks data out at about 200 Hertz, which is remarkably fast for what we're trying to do,” said Snyder. “Most of the aircraft that we're dealing with in the military, you're only working in the 60-Hertz range. So to find that unit from NovAtel was a godsend. And then we went into months of flying this over and over again, just to validate that our results were accurate."

Weight and size are crucial factors for anything that goes into an airplane, and for a computer-based system, memory and processing power also depend on those considerations. Red 6's Berkut was custom-designed and built with such concerns in mind, but that's not possible with the military craft for which ATARS is intended. Said Snyder: "When we move into our military aircraft, we're taking that normal form factor, which is about the size of a small computer or maybe a larger laptop, and going down to about a child's shoebox. We're essentially working through every piece of the software and every piece of the visuals we're creating, and reducing while trying to keep the fidelity that we have."

Reader’s Choice: 5 Questions on the Future of Additive Manufacturing in Aerospace

The visuals Snyder mentioned are those experienced by the pilot wearing the ATARS augmented reality headset integrated into their helmet, which creates remarkably detailed, full-color, realistic images of an enemy aircraft, a refueling tanker, a friendly plane flying in formation, in whatever particular scenario might be required. Those scenarios are controlled by a backseat instructor, using a specially designed app on a ruggedized iPad that plugs into the ATARS system through a USB connection in the cockpit. 

Snyder said that in current military flight training, "They essentially have one student pilot go up in the front seat with an instructor in the backseat, and then another aircraft with an instructor in it. And the two instructors talk to each other about what the scenario is going to be. So we started looking at that and said, how can we do the same for the backseaters and essentially have this perfect augmented reality wingman be controlled by the backseater without having to add in a whole bunch of new controls?"

Taking the Risk Out of Training

The iPad application, Snyder explained, "gives you full control of the aircraft that you're seeing, what it's doing, what kind of aircraft it's portraying. They hop in and they say, ‘Today we're going to do air to air refueling. So they hit the air refueling button."

The advantages are clear, Snyder noted, recalling a conversation with one of Red 6's fighter pilot staffers. "He told me that the first time he ever air-to-air refueled was over the Atlantic in an F-15. If he had messed it up, he was going into the ocean. The idea that pilots aren't trained properly because the cost and risk is way too high kind of blows my mind."

With the basic ATARS concept proven, the next major step is moving the system into actual military aircraft, beginning with the T-38 Talon. "Now that we've prototyped and been able to show our proof of concept, the main problem that we're looking into is a lot of the mechanical engineering issues," Snyder said. Part of that is ensuring that ATARS can function in an aircraft that does things that don't happen in the Berkut -- such as occasionally using an ejection seat, which the Berkut doesn't have. 
 
"Coming up with the methodology to allow us to eject safely was never in our forefront because we just haven't had to solve that,” he said.  “But now we're moving into the T-38, we're essentially having to redesign all of the mechanical engineering." 

More for You: Podcast: Additive Manufacturing Revolutionizes Aerospace Design

That includes things such as the ATARS cabling attachments to the helmet, and finding ways to make the system operate with existing Air Force equipment. "We're taking the overall prototype that we have now and breaking down all its mechanical engineering into modules, and figuring out what's currently out there and what we can borrow from, to get ourselves to the safest product for our guys to be actually be able to fly with. The mechanical engineering team has actually had to scale recently from just three people up to over eight people now to solve that."

Along with the Air Force, Red 6 is currently working with a range of other companies and agencies to further develop and expand the ATARS technology. For the military, that means integrating pilot training with ground forces for full-fledged battle scenarios. Civilian applications are another huge market. 

"What's beautiful about the solution is it works across all training, whether commercial or military," Snyder said. "We're working in the civilian sector to not only help training but also commercial landings and issues that arise when you're trying to do as many flights as some of these larger commercial entities do."

From a souped-up videogame to military training tool and beyond, the ATARS concept is bringing augmented reality out into the big, wide world of tantalizing possibility. "It really is just a catalyst of so many ideas," Snyder said. "It's something that hasn't been available until now. It's endless when you start thinking about how to use it."

Mark Wolverton is a technology writer based in Narberth, Pa.