A camera system developed by researchers at Carnegie Mellon University can sense sound vibrations with such precision and detail that it can reconstruct the music of a single instrument in a band or orchestra.
Even the most powerful and focused microphones cannot eliminate nearby sounds, ambient noise and the effect of acoustics when capturing audio. The new system developed at the Robotics Institute (RI) of the School of Computer Science uses two cameras and a laser to detect high-speed, low-amplitude surface vibrations. These vibrations can be used to reconstruct sound and capture isolated audio without inference or a microphone.
“We’ve invented a new way of seeing sound,” said Mark Sheinin, a postdoctoral research associate at the Illumination and Imaging Laboratory (ILIM) at RI. “It’s a new type of camera system, a new imaging device, that can see something that’s invisible to the naked eye.”
The team completed several successful demos of their system’s effectiveness in sensing vibrations and the quality of sound reconstruction. They captured isolated audio from individual guitars playing at the same time and individual speakers playing different music at the same time. They analyzed the vibrations of a tuning fork and used the vibrations of a bag of Doritos near a speaker to capture the sound of a speaker. This demo pays tribute to previous work by MIT researchers who developed one of the first visual microphones in 2014.
The CMU system improves dramatically over previous attempts to capture sound using computer vision. The team’s work uses ordinary cameras that cost a fraction of the high-speed versions used in previous research, while producing a higher-quality recording. The dual-camera system can capture vibrations from moving objects, such as the movement of a guitar as a musician plays it, while sensing individual sounds from multiple points.
“We’ve made the optical microphone much more practical and usable,” said Srinivasa Narasimhan, a professor of RI and head of the ILIM. “We’ve improved quality while reducing costs.”
The system analyzes the differences in speckle patterns of images captured with a roller shutter and global shutter. An algorithm calculates the difference in the speckle patterns of the two video streams and converts those differences into vibrations to reconstruct the sound.
A speckled pattern refers to the way coherent light behaves in space after being reflected off a rough surface. The team creates the speckle pattern by aiming a laser at the surface of the object producing the vibrations, such as the body of a guitar. That speckle pattern changes as the surface vibrates. A shutter captures an image by scanning it quickly, usually from top to bottom, and produces the image by stacking one row of pixels on top of the other. A global shutter captures an image all at once.
Joining the study with Sheinin and Narasimhan were Dorian Chan, a Ph.D. computer science student, and Matthew O’Toole, an assistant professor in the Department of RI and Computer Science.
“This system pushes the boundaries of what is possible with computer vision,” says O’Toole. “This is a new mechanism to capture high speed and small vibrations, and presents a new area of research.”
Most of the work in computer vision has focused on training systems to recognize objects or track them through space — research important for advanced technologies such as autonomous vehicles. The fact that this work enables systems to better see imperceptible, high-frequency vibrations opens up new applications for computer vision.
The team’s dual-shutter optical vibration detection system allows sound engineers to monitor the music of individual instruments without interference from the rest of the ensemble to fine-tune the overall mix. Manufacturers could use the system to monitor the vibrations of individual machines on a factory floor to spot early signs of maintenance needed.
“If your car starts making a weird noise, you know it’s time to have it looked at,” Sheinin said. “Now imagine a factory floor full of machines. Our system allows you to monitor everyone’s health by measuring their vibrations with a single stationary camera.”
Sketch a shape based on the sound
Mark Sheinin et al, Dual Shutter Optical Vibration Detection(2022)
Newly developed optical microphone sees sound like never before (2022, June 21)
retrieved on June 21, 2022
This document is copyrighted. Apart from fair trade for the purpose of private study or research,
part may be reproduced without written permission. The content is provided for informational purposes only.
#Newly #developed #optical #microphone #sees #sound #Global #Circulate