The Army recently received the new IVAS 1.2 prototype that it will use for a series of tests starting later this month. (US Army)
WASHINGTON — On its evolving quest to improve soldier training, the Army is eyeing Microsoft’s Integrated Visual Augmentation System (IVAS) as the centerpiece for a mixed-reality trainer that injects virtual threats into real world scenarios.
However, over a half dozen current and former service officials and industry sources tell Breaking Defense that a technical challenge has already forced users to modify that training while wearing IVAS. And with no ready solution in sight, it is uncertain when — or if — soldiers will be able to effectively use IVAS to train like they fight outdoors, or even inside buildings with doors and windows.
What’s known as dynamic occlusion limitations — the lack of ability to hide a virtual object behind a real one — is an issue for many mixed reality devices that insiders agree has no ready solution. It also means that unless the Army cracks the problem, it has one less IVAS use case, and it’s a big one.
As a basic example during a mixed-reality experience, when someone wearing a device like the IVAS stands in front of a real, closed door, they are not able to open that door and then suddenly see the virtual objects, like enemies, that had been placed behind the real world objects.
“We are working on it. We’ve identified it as a real issue,” Lt. Col. Mindy Brown told Breaking Defense on July 24 when asked about the impact of occlusion on the Army’s ability to conduct indoor training.
Brown oversees the Squad Immersive Virtual Trainer (SiVT) program which combines IVAS — with its computer processing “puck” and HUD — and other Microsoft equipment with a rifle connected to a weapon tracker and the 3D global terrain capability called One World Terrain. In essence, IVAS serves as the brain of SiVT; if IVAS doesn’t work right, neither will SiVT.
RELATED: Last stand for IVAS? New challenges, delays as Army debates future of augmented reality goggles
It’s through that SiVT program that the Army wants to piggyback off its potential $22 billion IVAS production deal to essentially create one device that is suitable for both combat and mixed-reality training.
Getting the design and software right on that IVAS combat capability is “still [priority] number one,” Doug Bush, the head of Army acquisition, told reporters last week. But he also wants to keep it “on track together” with the device’s other intended use as a key enabler to the mixed-reality trainer.
“For this new form factor (1.2), we also have to try to make sure it still works with the ‘train’ SiVT environment as we try to build out the additional battle drills,” he said. “But, I think it’s important to get the form factor right first, that will allow us to pivot.”
It’s that occlusion problem that is contributing to challenges using the IVAS-based mixed-reality trainer outdoors, and altering how soldiers would use it to train indoors too. And Brown said she does not anticipate having a potential solution before 2026, the tentative date for SiVT fielding – a date well after the service is likely to decide if it will stick with its Microsoft deal or scrap it. Microsoft did not respond to a request for comment by publication.
Doors And Windows
By using IVAS as part of a mixed-reality trainer, the ultimate goal was to create a way for soldiers to train in all 11 battle drills — different maneuver practices, often by smaller infantry units. The idea is for soldiers to train in the physical world while encountering virtual combatants, civilians and even animals. Once finished, they would be able to review what they got right, wrong and where they need to improve.
At the onset, Army officials acknowledged that developing one device for both combat and mixed-reality training wouldn’t be easy, and things like occlusion would be a challenge. It soon became clear that using IVAS outdoors for mixed-reality training would be difficult due, in part, to ever-changing light conditions. But even as outdoor problems persisted, the service thought it had indoor training, like Battle Drill 6, an exercise where a unit enters a building and clears it, under control.
“Full disclosure, that’s the easiest of those mission sets to do for something like IVAS and SiVT because it’s indoors. You can control a lot more variables, you can control the ambient light, it’s an easier problem to solve,” Brig. Gen. William Glaser, the director of the Synthetic Training Environment Cross-Functional team, told Janes in October 2021. “We’re not going to spend all this money just for that indoor capability. We’ve got to get it outdoors, day and night, for battle drills 1-11.”
Two years later industry and service officials acknowledge that the occlusion conundrum forces soldiers to conduct that indoor Battle Drill 6 differently than they would during an all-physical world scenario or when conducting combat operations.
Soldiers from the 82nd Airborne Division used a prototype of the Integrated Visual Augmentation System (IVAS) during a trench clearing exercise in October at Fort Pickett, Va. . (Bridgett Siter/US Army)
Online Army videos of soldiers conducting the standard room clearing drill vary, but doors, windows, and, sometimes, stairs are typically included. They often show a unit gathering outside a structure and then one soldier kicks in a door, or stands behind it keeping it open, before unit members quickly file into a room and secure it. The process repeats throughout other rooms and floors inside the structure.
But when using the IVAS mixed-reality capability, the system has trouble projecting virtual objects and figures behind real obstacles. It “perceives a closed door as a wall and an open door as a mere hole in the wall,” one industry source familiar with the program explained. “Consequently, door practice is impossible …. This is also true of windows.”
“The door is important: [It’s] kind of hard to train like you fight with no doors,” the industry source added.
Multiple Army officials familiar with the program, including Brown, confirmed this is a problem, and in some photos of Battle Drill 6 with soldiers wearing IVAS headsets as part of the mixed-reality trainer, doors or windows are either missing or propped open. However, those same officials insist that even if the Army can’t crack the door and window occlusion challenge, they believe soldiers will still benefit from a modified Battle Drill 6 training.
To try and get at the problem, Brown said the Army and Microsoft are exploring workarounds with digital twins and improving georeferencing, also known as localization. And because the occlusion problem is industry wide, not just related to IVAS, there are plenty of potential solutions to consider.
Potential Solutions On The Table
One augmented reality expert, not with Microsoft, explained to Breaking Defense that there are different ways to dynamically digital twin the world with the changes to the position of doors and windows in real time.
“Your synthetic environment needs to understand that the doors closed and doors open are based upon a sensor or a camera understanding that the doors are open or shut or in between,” he explained. “This goes for all dynamic moveable objects. Having a real-time update of the twin is how you solve that problem.”
That mitigation technique might also be applicable outside where real moving objects like cars, birds or falling leaves can create problems.
But even if the occlusion issue can be solved, there remain other challenges for IVAS outdoors, including how to handle bright natural light, according to Brown and Col. Nick Kioutas, the project manager for the synthetic environment. Kioutas noted that improving georeferencing is a big push outside.
“We’re trying to make it a little better for outdoors, but the hardest part is really georeferencing,” he explained. “It’s making sure that when you set up the training, that the simulation knows where you’re at on the terrain, that’s a little bit of a hardest case… [but] there’s multiple techniques that will allow us to fix that.”
Within the mixed-reality space, georeferencing and occlusion are linked. The former refers to the process used in a geographic information systems (GIS) and cartography to associate digital data, like maps or satellite images, with real geographic locations — making the simulated environment on which virtual projections live more like the real one.
While Microsoft, the service and other tech companies eye different occlusion and georeferencing mitigation avenues, Brown said she is hopeful they will make progress by 2026 when the IVAS-centered immersive training system is scheduled for fielding.
When asked if she assesses that her team can find adequate occlusion workarounds, Brown said, “I think so. I think it’s going to take some time though.”
But if Brown’s team and Microsoft are unable to figure out how to use IVAS for mixed-reality training both inside and outside and SiVT development is shelved, it would present another shortcoming for a system already grappling with a troubled development.
That doesn’t mean all training would not be possible with IVAS, however. Soldiers would still be able to use the multi-billion-dollar IVAS for things like marksmanship training and preparing for a mission in theater, IVAS product manager Lt. Col. Denny Dresch told Breaking Defense.
“You could use the mission planning feature within IVAS, and you could train off for that” instead of using PowerPoint slides on a laptop, he added.