ECE Assistant Professor Ivan Ruchkin has received funding from the National Science Foundation (NSF) for his project “Neuro-Symbolic Bridge: From Perception to Estimation & Control.” The project aims to enhance the safety and reliability of cyber-physical systems (CPS) by developing methods to repair the mismatch between neural network outputs (the raw data) and the downstream symbolic systems (the decision-making processes). Dr. Ruchkin’s work will utilize neuro-symbolic calibration and training methods in an effort to design safer and more predictable CPS which utilize neural network perception such as those deployed in such transportation, agriculture, and medicine. As a research platform for the system, his team will deploy it on small-scale racing cars. The team includes Dr. Ruchkin’s collaborator at Rennsselaer Polytechnic, Dr. Radoslav Ivanov.

Cyber-Physical Systems

A typical modern CPS consists of a neural network, used to process raw data (such as images from a camera) and symbolic logic components which in turn estimate states and issues control commands for the system. ‘Symbolic’ because they are based on first-principles models (such as how a car drives or how sensors work) which are necessarily written by humans.

The algorithms/models in these symbolic tasks make certain convenient assumptions about the sensor data. When neural networks are used for ‘perception,’ which is really processing rich sensor data (e.g., camera images or lidar scans), inherent errors in the data do not necessarily satisfy those assumptions. Despite these inaccuracies, CPS continue to be designed using the standard symbolic components for downstream tasks because they have worked in the past. This situation leads to a mismatch between the neural perception and symbolic downstream tasks, which can lead to poor performance and even catastrophic failures if perception unpredictably misbehaves (e.g., sudden glare from the sun creates anomalous data from a camera).

A New Path Forward

Ruchkin’s project will address this mismatch by performing a special kind of neural network training and subsequent calibration (similar to fine-tuning on the last neural layer). The novel approach in Ruchkin’s work is that the goal of the training adjustments is not to make the neural network perform better (as is normally the case), but instead to satisfy the assumptions of these downstream symbolic tasks, even if sometimes at the expense of accuracy.

Since the perception is neural and the downstream estimation/control is symbolic, Ruchkin characterizes the whole framework as neuro-symbolic. The benefits of neuro-symbolic calibration and training are being demonstrated on a 1/10-scale autonomous racing platform—the F1/10 car. In addition, the team will conduct outreach activities within Rensselaer Center for Open Source and Course-Based Undergraduate Research Experience (CURE) with the Center for Undergraduate Research at the University of Florida.

---