Human-Machine Teaming Transforms Underwater Operations with Advanced Diver-AUV Collaboration

The electricity to an island goes out. To find the break in the underwater power cable, a ship traditionally pulls up the entire line or deploys remotely operated vehicles (ROVs) to traverse the line. But what if an autonomous underwater vehicle (AUV) could map the line and pinpoint the location of the fault for a diver to fix? This vision of underwater human-robot teaming is the focus of an MIT Lincoln Laboratory project that seeks to combine the complementary strengths of humans and robots for maritime operations.

The project, led by principal investigator Madeline Miller from the Advanced Undersea Systems and Technology Group, addresses a fundamental limitation in current underwater operations: divers and AUVs generally don't team effectively underwater. "Underwater missions requiring humans typically do so because they involve some sort of manipulation a robot can't do, like repairing infrastructure or deactivating a mine," Miller explains. "Even ROVs are challenging to work with underwater in very skilled manipulation tasks because the manipulators themselves aren't agile enough."

Complementary Capabilities

Beyond their superior dexterity, humans excel at recognizing objects underwater. However, humans working underwater face significant limitations in performing complex computations or moving quickly, especially when carrying heavy equipment. Robots, conversely, have advantages in processing power, high-speed mobility, and endurance.

"The modern world runs on undersea telecommunication and power cables, which are vulnerable to attack by disruptive actors," Miller notes. "The undersea domain is becoming increasingly contested as more nations develop and advance the capabilities of autonomous maritime systems. Maintaining global economic security and U.S. strategic advantage in the undersea domain will require leveraging and combining the best of AI and human capabilities."

Technical Challenges

To enable effective human-machine teaming underwater, Miller's team must overcome several technical challenges:

1. Navigation: Divers typically rely only on a compass and fin-kick counts for navigation. With few landmarks and potentially murky conditions caused by limited light at depth or biological matter in the water column, they can easily become disoriented and lost.

2. Perception: In darkness or turbid water, optical sensors (cameras) cannot generate useful images, while acoustic sensors (sonar) produce images that lack color and only show shapes and shadows. The historical lack of large, labeled sonar image datasets has hindered training of underwater perception algorithms. Additionally, the dynamic ocean environment can obscure objects, confusing artificial intelligence. For instance, a downed aircraft broken into multiple pieces, or a tire covered in mussel overgrowth, may no longer resemble their original forms.

3. Communication: Underwater acoustic communications suffer from low bandwidth and high latency. State-of-the-art data rates would require tens of minutes to send an uncompressed image from an AUV to a diver.

Technical Approach

Miller's team is developing hardware and algorithms to address these challenges:

Navigation Algorithms

The team built upon work started by the MIT Marine Robotics Group, led by John Leonard, to develop diver-AUV teaming algorithms. Leonard's group had demonstrated these algorithms through simulations and field testing in calm waters using human-paddled kayaks as proxies for both divers and AUVs.

"We quickly learned that you need more sensing capabilities on the diver when you factor in ocean currents," Miller explains. "With the algorithms demonstrated by MIT, the vehicle only needed to calculate the distance, or range, to the diver at regular intervals to solve the optimization problem of estimating the positions of both the vehicle and diver over time. But with the real ocean forces pushing everything around, this optimization problem blows up quickly."

The team integrated these algorithms into a mission-relevant AUV and began testing them under more realistic ocean conditions, initially with a support boat acting as a diver surrogate, and then with actual divers.

Perception System

For perception, the team is developing an AI classifier that can process both optical and sonar data mid-mission and solicit human input for any objects classified with uncertainty.

"The idea is for the classifier to pass along some information — say, a bounding box around an image — to the diver and indicate, 'I think this is a tire, but I'm not sure. What do you think?'" Miller describes. "Then, the diver can respond, 'Yes, you've got it right, or no, look over here in the image to improve your classification.'"

This feedback loop requires an underwater acoustic modem to support diver-AUV communication. Given the constraints of underwater communications, the team is investigating how to compress information into the minimum amount necessary to be useful, working within the low bandwidth and high latency of underwater communications and the size, weight, and power limitations of commercial off-the-shelf (COTS) hardware.

Hardware Integration

For their prototype system, the team procured mostly COTS sensors and built a sensor payload that would easily integrate into an AUV routinely employed by the U.S. Navy. Beyond sonar and optical sensors, the payload features an acoustic modem for ranging to the diver and several data processing and compute boards.

"The goal is to facilitate technology transition," Miller emphasizes. "By using components that can integrate with existing Navy AUVs, we increase the likelihood that this technology will be adopted in real-world operations."

Testing and Validation

The team has conducted extensive testing in various environments:

1. Coastal New England: They tested the sensor-equipped AUV and algorithms near Portsmouth, New Hampshire, with the University of New Hampshire's (UNH) Gulf Surveyor and Gulf Challenger coastal research vessels as diver surrogates.

2. Charles River: They conducted tests with an MIT Sailing Pavilion skiff as the surrogate, allowing for slower movement and relative motion more similar to how a diver and AUV would navigate together.

3. Great Lakes Research Center: Last summer, the team started testing equipment with human divers at Michigan Technological University's Great Lakes Research Center. Each diver swam holding the team's tube-shaped prototype tablet, dubbed a "tube-let," equipped with a pressure and depth sensor, inertial measurement unit, and ranging modem.

"A challenge during testing was coordinating the motion of the diver and vehicle, because they don't yet collaborate," Miller explains. "Once the divers go underwater, there is no communication with the team on the surface. So, you have to plan where to put the diver and vehicle so they don't collide."

During the Great Lakes testing, Caroline Keenan, a Lincoln Scholars Program PhD student jointly working in the laboratory's Advanced Undersea Systems and Technology Group and Leonard's research group at MIT, advanced her work on knowledge transfer from optical sensors to sonar sensors. She is exploring whether optical classifiers can train sonar classifiers to recognize objects for which sonar data doesn't exist, with the goal of reducing the human operator load associated with labeling sonar data and training sonar classifiers.

Real-World Applications

The technology being developed has potential applications across multiple domains:

1. Critical Infrastructure Inspection and Repair: Mapping underwater power cables, pipelines, and communication lines to identify faults and guide repair operations.

2. Search and Rescue: Locating objects or persons in underwater environments where visibility is limited.

3. Harbor Entry: Assuring safe passage for naval vessels in potentially contested waters.

4. Countermine Operations: Detecting and neutralizing underwater threats.

5. Environmental Monitoring: Studying marine ecosystems and tracking changes over time.

Limitations and Future Work

Despite promising progress, several limitations remain:

1. Environmental Constraints: Performance may degrade in extremely turbid water or at significant depths where optical sensors are ineffective and acoustic signals are attenuated.

2. Communication Bottlenecks: The low bandwidth of underwater communications limits the amount of information that can be exchanged between the diver and AUV.

3. Power Constraints: The size, weight, and power limitations of COTS hardware restrict the capabilities that can be integrated into diver-worn devices and AUVs.

4. Human-Machine Interface: The team still needs to develop an effective interface for divers to provide feedback to the AUV during operations.

As the internally funded research program comes to an end, Miller's team is now seeking external sponsorship to refine and transition the technology to military or commercial partners. The success of this project could significantly enhance underwater operations by combining human intuition and dexterity with robotic endurance and computational power.

Ultimately, the vision is to create a seamless human-machine teaming capability that allows divers and AUVs to operate as an integrated unit, each leveraging their respective strengths to accomplish tasks that would be difficult or impossible for either to achieve alone. As underwater infrastructure becomes increasingly critical to global economic security and military operations, this technology could provide a significant advantage in an increasingly contested domain.

For more information about the MIT Marine Robotics Group's work on diver-AUV teaming, visit their official research page. Additional details about the Advanced Undersea Systems and Technology Group can be found on the Lincoln Laboratory website.