MIT's UQ-DAAAM lets robots spot and fix their own fuzzy memories

MIT's earlier DAAAM system uses a vision-language model to caption objects and store them in a 3D map the robot can query later[1]. But partial, distant or blurry views produce mistakes, such as describing a grey rooftop chimney as a squirrel, and with no way to self-correct that error sticks around[1]. MIT's example of the stakes: a robot that has mislabelled a defibrillator as a fire extinguisher may not find it in an emergency[1].

The system scores the model's confidence in its own caption, then checks consistency against captions of the same object from other viewpoints, fusing both into one uncertainty number[1]. When that number is too high, a heuristic picks extra frames likely to reduce the doubt, and the team proved it can choose those frames without rerunning the vision-language model, which keeps it fast enough for a moving robot[1]. An LLM then fuses the selected captions, weighting newer and less ambiguous ones more heavily[1].

MIT says the refinement step resolves ambiguities for about 92 percent of objects while needing only two additional captions per object, and that it increased the accuracy of object descriptions by 5 percent compared to the state of the art[1]. It also answered questions about what objects were, where they were seen and when better than baseline methods[1].

"Allowing a robot to know what it doesn't know is going to be essential for this technology to be deployed safely," says senior author Luca Carlone[1]. Next the team plans to apply it to mobile manipulators and use high-uncertainty areas to tell robots where to look again[1].