University of Waterloo signs partnership with Rogers
Smart cities are moving towards the automation of many essential services. Quality of life for city residents relies on the safety of its buildings, roads, bridges and other infrastructures. Manual inspection of these infrastructures, besides being costly, is a tedious task that requires special skills and intense labour — especially when access to some sites, like bridges, is not easy. Due to the sheer number of bridges and inspection timelines, there is a critical backlog for inspecting bridges in Canada that requires an immediate solution.
As a result of a University of Waterloo partnership with Rogers that began in 2019, Chul Min Yeum and Sriram Narasimhan, Waterloo professors of civil and environmental engineering, received research funding for the creation of a 5G smart city infrastructure monitoring and alerting system using cutting edge artificial intelligence (AI) and machine learning (ML) algorithms to automate the inspection of civil structures.
“With the recent development in robot technology, sensors, augmented and virtual reality, supported by advances in 5G mobile connectivity, remote inspection of civil infrastructure is now a feasible option,” says Yeum.
Remote inspection platform
The standard inspection technique is to examine infrastructure manually, where engineers look for signs of damage such as leaks, cracks, corrosion or any other distinctive marks. The Remote Inspection Platform (RIP) developed by Yeum and Narasimhan, their team members Zaid Al-Sabbag and Max Midwinter, and the Rogers team is an alternative approach to the manual inspection process.
RIP front-end robot systemThe RIP includes a front-end robot system, extended reality (XR)-equipped human inspector, inspection algorithms, and 5G connectivity, linking all elements together. The robot is equipped with color and thermal cameras and light detection and ranging (LiDAR) to scan the inspected site.
The robot scans the bridge, transferring real-time data through cameras and LiDAR into the Multi-Access Edge Computing (MEC) via Rogers’ 5G network.
Once data is received at the MEC, the application creates a 3D map of the bridge and analyzes it using AI. Use of the MEC is essential to cut down on latency and reduce the time needed for communication between the robot and the RIP apps. At this stage, the human inspector can assess the inspection results using augmented reality (AR), enabling interaction for further refinement and documentation. Moreover, these results are shared with remote virtual reality (VR) users, actively involving them in the inspection process.
Automatic damage detection
The 3D map stores defect and asset information about the bridge. The higher the resolution of the data collected the more accurate the ML techniques will perform. Also, based on requirements, a threshold can be introduced to establish the desired level of inspection accuracy. If the threshold is set low, the system can detect nearly all cracks in the bridge, but this may also result in a higher rate of false-positive detections.
Building inspector wearing VR headsetThe AR-equipped human inspector who supervises the process from a safe area nearby can step in if they are needed. The inspector is localized to the 3D map and can show damaged locations and add annotations through the AR headset which will help, in certain cases, identify areas that may be missed by the robot. The process can support multiple AR-enabled inspectors.
The VR-enabled human inspector, who normally sits far away in an office, receives the 3D bridge map, and can interact with the on-site AR inspector to support the inspection process remotely. The process can accommodate multiple VR-enabled inspectors allowing for collaborative inspection.
The role of 5G network
In 2019, the University and Rogers signed a partnership agreement to build a 5G wireless network enabled smart campus and provide research and development collaboration opportunities anchored in next generation networks. Without support from Rogers, Yeum’s team would not have access to the 5G network.
The remote inspection platform relies on wireless connectivity to enable wireless communication between the robot and the RIP application, whether at the MEC or in the cloud. Wireless connectivity is also needed to enable communications for the AR users. The amount of data transferred between the robot and the RIP apps is huge, demanding both low latency and high throughput. The 5G connectivity is not only needed for robot communication with the app, but it is also essential for the AR users to communicate with the robot and with the VR users collaborating remotely.
With the project extended for a third year, Yeum aims to incorporate a drone into the RIP to enhance data collection capabilities at close range. The team will also initiate collaborations with the Ministry of Transportation Ontario for bridge inspections and indoor inspections at nuclear power plants. Furthermore, they plan to implement voice chat functionality and gesture sharing to ensure seamless communication between the AR and VR systems.