AI allows new robot to manipulate soft and pliable objects

Say hello to the robot called Bifrost. With the help of AI technology, it uses its tactile capabilities to manipulate soft and pliable objects to order.

“In spite of the impression many of us have from the media, robots still have a long way to go before they can manipulate so-called compliant objects with proficiency,” says Norwegian researcher Ekrem Misimi.

Misimi believes that a lot more research is needed before this can be realized.

“This is why we’re proud to have developed a method that enables robots to learn to manipulate objects of this kind. This is ground-breaking research offering major innovation potential,” he says.

“The technology currently under development may have many applications across a wide range of industries—but first and foremost, in the food sector, where there is no shortage of compliant objects.”

An AI-based approach

The new technology has emerged as part of a project, also called BIFROST, which in Norse mythology refers to a “bridge.” However, the inspiration for the method started in the kitchen, and more specifically with the stages involved in preparing a humble cod filet for dinner.

“We’re all familiar with this process without really thinking too deeply about it,” says Misimi. “But if you give it some thought, you find that you’re carrying out a number of different operations on the filet. In the past, robots were unable to perform such tasks, because up until recently they had only been trained to grip objects that were rigid and non-compliant. Moreover, it is challenging to get a robot to carry out new tasks that it hasn’t been trained to do,” he says.

In the field of AI, robots struggle with what is known as generalization capability—something at which humans are very adept. In order to succeed, the researchers have developed a new approach based on artificial intelligence. They train the robot by getting it to perform tasks similar to those it would have to perform in the real world. Understandably, these tasks do not involve cod filets to begin with.

Time-consuming tasks are the most difficult

On the table lies an elongated cloth bag filled with rice, here simulating a cod filet. It lies in the shape of an “I,” but in response to a command from the researchers, the robot manipulates it into a “C.”

“We have thus succeeded in getting the robot to manipulate a compliant three-dimensional object to order,” says Misimi. “On its own, this may not seem very impressive. However, the robot has not previously been trained using this particular shape,” he says.

Misimi adds that it is easy for humans to perform tasks for long periods until they finish what they intend to do, but this is much more challenging for a robot.

“A combination of learning and perception is essential if a robot is to complete such manipulative tasks.

“The more complex a task, the longer a robot will take to learn how to complete it,” says Misimi. “And the more time it takes, the more demanding it becomes for the robot.”

New opportunities

The research team has recently published an article on the method as part of the 2024 IEEE International Conference on Robotics and Automation (ICRA) proceedings. The team explains that AI-based robotic manipulation offers new opportunities that have previously been beyond our imaginations.

“What is so cool about this method is that the robot is trained exclusively using simulation,” says Misimi.

The knowledge is then carried forward into the real world, without any additional training. This is completely new. The researchers’ main focus has been on training the robots in skills inspired by what humans can do.

“In a way, we’re making them better placed to perform tasks that require human dexterity—tasks that today can only be performed by humans, both in terms of learning and perception,” says Misimi.

The term perception is used here to describe an ability to be aware of what is going on in our surroundings, so that a task can be planned ahead and then carried out. A combination of learning and perception is essential if a robot is to complete dexterous manipulation tasks.

Successive manipulations are tricky

When a robot is tasked with manipulating a given object, it has to combine its visual capabilities with perception.

“In such situations, it may also have to work for some time to complete the task,” says Misimi. “To manipulate the object into a new shape, the robot has to complete many different actions and, up until now, it’s been challenging to use robots for such tasks,” he says.

He goes on to explain that a robot commonly requires between 20 and 60 seconds to manipulate an object into a new shape, depending on the complexity of the process.

“It’s a challenge to get a robot to concentrate on a task for an extended period, but we have succeeded,” says Misimi.

Ideal for the food industry

The technology currently under development may have many applications across a wide range of industries.

“But first and foremost in the food, and especially the seafood, sectors, where there is no shortage of compliant objects for manipulation,” says Misimi. “This is important because the use of robots can help towards enabling Norway to maintain its food production industry, and in so doing may also promote more sustainable and locally-sourced food of better quality,” he says, going on to add,

“At an even higher level, knowledge obtained from this project may help us to address some of the fundamental challenges we face in modern robotics. This in turn will lead to the development of new technologies that will benefit both the industrial sector and wider society.

The research team has also succeeded in further developing the Bifrost robot such that it is now able to perform pushing as well as gripping actions. This enables the even more gentle manipulation of compliant objects and structures.