Stevens Institute of Technology: Long Wang to Develop Robotic Hands Technology System for Supporting Remote Factory Workforce Productivity

When the World Health Organization declared COVID-19 a global pandemic in March 2020, office workers the world over turned into remote workers almost overnight.

With a computer, keyboard and mouse as their primary occupational tools, such employees kept businesses functioning by transforming personal living spaces into remote satellite offices. Needing only an internet connection and the appropriate software, much of such work can be performed on just about any computer from just about anywhere.

But for manufacturing industry workers whose jobs require direct, physical access to equipment located in specific locations, remote working is not currently an option. Unable to accommodate workers on location safely under pandemic conditions, factories around the world shut down production for months, resulting in furloughs and layoffs, worldwide supply chain disruptions, and economic ramifications that are still being felt globally two years later.

Because manual manufacturing jobs are so technology- and location-specific, the average person cannot simply network into operating an industrial drill press or screwdriver from their living room couch.

Or can they?

Long Wang with the robotic hand prototypeStevens Institute of Technology mechanical engineering assistant professor and principal investigator for the Advanced Robot Manipulators Lab Long Wang has been awarded a $199,995 grant from the National Science Foundation (NSF) to extend the capabilities of the industrial workforce to do just that.

Titled “Tool-Grasping Compliance and Stability of Underactuated Hands in Model-Mediated Telemanipulation,” the project will leverage artificial intelligence to develop a complete, cost-effective solution for operating remotely controlled robotic hands with high levels of control, dexterity, and situational awareness to perform complex manual manufacturing tasks.

“Imagine a workforce crew that can drive a robot just like playing a video game, so that their remote proxy can help to inspect and complete tasks, like picking up a tool, fastening a screw with a screwdriver, or using a hammer,” Wang said. “The factory would be still there. It’s just we would have robots in the factory that are remotely controlled by human crews who can work from home.”

Although inspired by pandemic shutdowns of automobile assembly plants in particular, such technology would open possibilities to any industry that requires the presence and insight of human crews performing manipulation tasks in hazardous conditions, including environmental waste cleanup, space exploration and search and rescue efforts during natural disasters. The project holds the potential to reduce productivity loss and service interruptions that can result in long-term individual and global economic ramifications.

The two-year award is granted as part of NSF’s Engineering Research Initiation and Foundational Research in Robotics programs.

Form versus function
The number one impediment to developing a remote manufacturing workforce, said Wang, is cost. Fully articulated five-fingered robotic hands — like the ones you might picture from The Terminator franchise — are exorbitantly expensive.

“Those multi-fingered hands easily go for about $50,000 to $90,000 each hand. So if one hand costs you that much, and you just want that hand to pick up and fasten screws, that might be a killer factor to stopping factory owners from using such technologies,” Wang said.

To bridge the gap between the desire to build remote workforce capabilities and the financial feasibility to do so, Wang is developing a cost-effective robotic manipulator system that leverages intelligent algorithms to allow simpler, cheaper robotic hands to function beyond their inherent capabilities.

To understand how requires a little arithmetic.

A robotic (non-thumb) finger designed to mimic a human’s contains three knuckle joints with one individual actuator, or motor, controlling the movement of each joint. (The human thumb contains only two knuckle joints.) Four fingers multiplied by three actuators per finger, plus one thumb multiplied by two actuators per finger, results in 14 total actuators per full robotic hand.

A photo of the robotic hand prototypeBut while humans may each need five digits and 14 knuckles to navigate the world and the tools they’ve developed, task-oriented factory robots do not. Such robots do not need as wide a range of motion or functional capabilities as humans. Nor do they necessarily even need as many independently functioning knuckle joints.

“If I instead have three fingers, and each finger has three joints, I have nine joints in total. So you would think that I need nine actuators, right? But an underactuated hand means that we’re going to use fewer motors than the number of joints in the hand. That’s a promising strategy for lowering costs,” Wang said.

Wang is quick to emphasize that he is not inventing the underactuated hand, which was first developed in the 1990s. Rather, by using artificial intelligence to fill in the gaps between (three-fingered) form and (five-fingered) function, Wang is “proposing that we adopt this simplistic mechanical design and combine it with a smart algorithm so we can make less expensively built robotic hands work almost as well as the expensive ones.”

Awareness by proxy
In addition to designing robotic hands that move, grasp and manipulate discrete tools with high force and motion control, stability, and dexterity, Wang will develop a model to train the robot to assist its remote operator with tasks and situations that humans tend to navigate by instinct.

“As human beings, we do many things without thinking or having to calculate all of the mathematical elements behind the scenes,” he explained.

Those unconscious calculations, however, can mean the difference between success and failure when performing a task.

For example, Wang explained, when using a screwdriver, your brain is actively focused on the primary task at hand: turning the screwdriver to sink the screw.

During this process, you are not consciously contemplating the phenomenon of gravity or repeating a mantra to remind yourself to keep ahold of the screw. Yet you are, nonetheless, unconsciously evaluating all the factors necessary to ensure you do not drop the screw in the process of turning it.

When working onsite, such secondary tasks, Wang said, “would be handled naturally in our situational awareness realm — like a library of information — without having to interrupt our main stream of the workflow.”

But when removed from a situation, such as through teleoperation, that instinctual awareness and ability to respond is greatly reduced. It becomes much harder to unconsciously function when the user cannot explicitly “feel” the situation in context.

By capturing and instilling the kinds of experiential knowledge that can develop such “instinctual” awareness into the robot, however, Wang said his system will be able to anticipate and navigate potential pitfalls, thereby achieving better results.

“Situational awareness is not normally captured,” he said. “We want the robot system to provide smart assistance and help take care of those tedious and surrounding small things while you, the teleoperator or driver staying at home, can focus on the main task.”

Wang’s prototype will also include a novel controlling device similar to a mouse that will prompt the remote user via physical feedback to respond to circumstances occurring on the factory floor.

“The algorithm could predict, for example, that you are pushing your screwdriver in one direction too much and that you’re going to lose grasp of the screw,” he explained. “So when you drive the robot to move too much in that direction, the haptic device will be programmed to give you a little resistance, like a heads up telling you to stop or slow down. We’re trying to give the user some assistance so that when they do these remote manipulation tasks, we can help ensure success.”

The model will also be trained using pre-existing data of the chosen task in context — such as the table on which the necessary tools will lay, how many screws will be available and where — to mitigate communication latency delays that may occur between robot and controller. By attending to such details, Wang hopes to make it possible for human controllers offsite to react and respond to the factory environment as quickly and efficiently as if they were present onsite.

Wang noted how people often wonder why robots aren’t yet capable of achieving all the things they are depicted doing in popular entertainment.

But accomplishing even simple tasks with machines, he said, “is actually a lot more challenging than we would imagine because we human beings are at least tens-of-billions-of-dollars engineering machines.”

By combining AI-enhanced robotic technology with situational awareness intelligence, haptic feedback and model mediation, Wang aims to make it possible through his telemanipulation framework to develop a realistic system of robotic proxies that will allow manufacturing industry workers to be able to perform their jobs remotely just as easily as their office worker colleagues.