MIT's FLEX lets robots read the intent behind a plain-language correction

In one demo a robot arm carries a cup of coffee across a cluttered table and a person types "Don't spill on the laptop!" A typical system would simply add the laptop to its list of things to avoid. FLEX goes further: it reasons that liquids and devices do not mix, and reroutes around a nearby phone as well, without being told[1].

"The idea is that there is some higher-level intention behind the words that you say," said study author Nathan Dennler, a postdoc in MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL). "People naturally get this. But you have to carefully explain that to a robot, which takes time and effort."

FLEX runs in three steps[1]:

• Add puts the object a person mentions onto the robot's list of features that matter for the task. This is how earlier methods already work.
• Generalize is the new part. A large language model, GPT-4o in this study, looks across everything the person has flagged and infers a broader category, such as fragile items.
• Mask uses the robot's computer vision to remove objects that are no longer in the scene, so the list does not grow stale.

The full loop takes about five seconds.

The team tested FLEX in simulation with a robot arm serving coffee, moving a candle and carrying a bowling ball. Volunteers typed commands such as "Stay away from the stack of paper" and "Don't drop on the mug". The robot obeyed each one literally and also generalised it, avoiding electronics, flammable objects and fragile things[1]. The report describes simulated trials only, not a physical robot.

The work comes from the Collaborative Learning and Autonomy Research (CLEAR) Lab led by Andreea Bobu, assistant professor of aeronautics and astronautics at MIT, with co-authors Audrey Lee and Ilia Sucholutsky of New York University.

Household and assistive robots will not ship with every owner's preferences built in. "Everyone's homes are different, and everyone cares about different things, so no robot comes out of the box knowing what matters to you," Bobu said[1]. Using a language model's common sense to fill the gaps between a few typed corrections is a practical way to make home robots less tedious to teach, provided the generalisations stay right outside the lab.