Under the supervision of Dr. Chen in his General Robotics Laboratory, I worked with PHd and upperclassmen to design a prototype system to take in and transform the microphone data into a spectrogram, using a convolutional neural network to analyze the spectrograms, and make live inferencing/prediction on where the drone is as well as classification for how much weight the drone is carrying. This information can then be scaled to the US Army, informing them about the location, model, velocity, and payloads of aircrafts.
Our initial prototype uses a respeaker 2.0 microphone array and Jetson to record and process the audio data indoors. We would fly the drone around, as well as with different weights, to train a CNN to predict the drone location and weight payload. Our ground truth data came from a VICON system set up with many cameras around the room and trackers on the drone. The Robot Operating System (ROS) collects the VICON positional data and raw audio, syncs up times, and outputs in an easily usable format.
Drone with weighted tether
VICON System
Flying Drone with VICON
Using Foxglove, we display the live predictions on top of the ground truth location from the VICON system.
Foxglove Live Inferencing Result
Foxglove Live Inferencing Result