Exoskeleton Control Improved Through Machine Learning

People with severe loss of motor control or motor disorders—from spinal cord injuries or neuromuscular disorders such as muscle weakness or paralysis—require a therapist to support and direct them as they perform their exercises. Exoskeletons promise to let patients recover independently at their own pace, either in the clinic or at home.

These devices, known as rehabilitation exoskeletons (RE), are designed to attach to patients’ bodies and run on electric motors with their own power source. Special control software controls the electric motors so that the exoskeleton can match its movements to the patient’s desire. 

New research published in Nature provides a significant improvement to this control software by using reinforcement learning, a kind of artificial intelligence. Starting with an accurate model of the human skeleton and muscles, a team of researchers have successfully built a neural network that learns to control an exoskeleton using a type of machine learning known as reinforcement learning. In doing so it bypasses extensive clinical or laboratory training of patients who might need such devices. “We train everything in the simulation stage so the end controller can basically be put in the device without any experiments and achieve great benefits,” said Xianlian Zhou, associate professor and director of the BioDynamics Lab at the New Jersey Institute of Technology in Newark.

Control software is the key enabler of lower limb rehabilitation exoskeletons (LLRE), allowing the device to act as a go-between, with the patient’s own muscles and actions on one side, and the ground and environment on the other. The software responds to the patient’s own forces against the LLRE and also to changing conditions such as uneven ground or a tilt of the patient’s body. The software tries to keep the LLRE working during all possible conditions and compensates for any changes in its environment by increasing or decreasing the current to various motors.

It is difficult to design such software and keep it safe for interacting with the patient and their environment. The patient’s disabilities determine how much force each limb can apply on the LLRE, so the human forces on the LLRE are unpredictable and could vary substantially from one patient to another. The developers tune the parameters of the software for particular patients, but this procedure still has limitations, and there will always be cases when the software is unable to control the LLRE.

More on Assistive Devices: Exoskeleton Boot Improves Walking Speed

The most effective techniques to write such software, called model-based predictive control, are also the most laborious to write and test, and the most sensitive to a particular patient and the particular task they are trying to achieve. 

Enter: reinforcement learning.
 

Neural network training using reinforcement learning

A neural network-based approach is a promising alternative to traditional robotic control software. The idea is not to write the control software manually, but instead train a neural network to learn how to control the LLRE’s motion under various conditions using reinforcement learning (RL). The RL-trained neural network can then be used as control software.

Such training requires millions of trials and failures and can only realistically be performed in simulation. Models run millions of times until the neural network learns to control the LLRE’s motors under various conditions. The control software then transfers from the digital realm into a computer that controls the physical LLRE. “We wanted to use reinforcement learning to build a controller because developing a controller is a very hard thing to do and needs very specific skills and expertise,” Zhou said.

For You: Personalizing Assistance for Exoskeletons Matches User Need

In a separate effort, researchers have demonstrated a comprehensive, detailed model consisting of hundreds of bones and muscles, so that individual variations in bone lengths, weights, and muscle strength can be simulated in a digital model. A RL-based neural network can train to control this model to show a realistic-looking human walk, turn, jump, and so on.

Using this detailed model, researchers can demonstrate how a person might walk with a stiff knee, a deformed bone, or other symptoms, by modifying the model to vary the bones, joint stiffness, and muscle strength. Similarly, they can outfit the model with a prosthetic leg or predict how a patient’s gait might be improved after surgery.

Zhou and fellow researchers found such a human musculoskeletal model useful for training the LLRE and have successfully built a neural network that learns to control the LLRE using reinforcement learning. To take into account various kinds of disabilities that LLRE users might have, the researchers varied the possible muscle strengths of the patient in the musculoskeletal model and created randomized variations during training. The resulting LLRE control software learns to handle these variations automatically, so it can work for healthy as well as disabled patients. In this way, they were able to demonstrate an LLRE walking in the case of different disability conditions including flaccid quadriplegia, muscle weakness, and hemiplegic conditions (where half the body is paralyzed).

The next step involves transferring this control software to a physical LLRE and demonstrating clinical use. “With this approach we get to develop the controller just purely from simulation and then transfer the controller into the hardware without doing any testing in advance,” Zhou says. It saves hours upon hours of lab testing and gives wearers an adaptable ready-to-go device.

Poornima Apte is a technology writer in Walpole, Mass.