A Camera That Sees Vibrations Can Record the Sounds of Multiple Instruments Separately

A camera that sees vibrations can record the sounds of multiple instruments separately

While the sensitivity and performance of microphones have improved considerably since Alexander Graham Bell first patented them, they still have one major drawback that researchers at Carnegie Mellon University maybe finally overcome by using a some average video cameras

Put a microphone in a room with a bunch of musicians, and as you capture every last note and nuance of their individual performances, you’re left with a single recording where everything is mixed together. But to make that performance sound even better, you ideally want each instrument and musician to be recorded individually so that each performance can be remixed by a sound engineer with a capable ear.

Software tools have been developed to extract individual sounds from an audio recording, but the results just aren’t as good as what you would get from capturing one sound source with one microphone. That’s why mixing consoles are often so gigantic and expansive: Countless microphones with limited pick-up patterns must be set up to properly capture every part of a musical performance, from vocals to instruments, which adds up to a lot of equipment to get things right.

There’s really no way to redesign microphones to distinguish the captured sound vibrations moving through the air, which is why the researchers at Carnegie Mellon University Robotics Institute of the School of Computer Science have turned to video cameras instead. Pluck the strings of a guitar, and it will not only produce sound waves that vibrate through the air, but it will also vibrate the guitar itself in the process. With the right equipment, those vibrations can be visualized and analyzed to mimic the sounds produced, even if no sounds are recorded.

Optical microphones, as these camera systems are called, are not a new idea, but what the CMU researchers came up with and shared in a recently published paper,’Dual Shutter Optical Vibration Detection,’ is a way to make them work using cheap and more affordable camera gear.

The new system shines a bright laser light source on a vibrating surface, such as the body of a guitar, and captures the movements of the resulting speckled light pattern. Since the range of human hearing can detect sounds that oscillate as much as 20,000 times per second, optical microphones typically relied on expensive high-speed cameras to capture physical vibrations that oscillate just as quickly. But the new CMU system uses cameras at just 63 frames per second, which would seemingly miss the rapid movements of a tremor that happens 20,000 times per second.

The clever breakthrough here is the use of two different types of cameras at once: one with a global shutter that captures entire video frames, resulting in distinct speckled patterns, and one with a rolling shutter that captures the frames line by line from top to bottom. of the sensor, resulting in distorted speckled patterns that actually contain more information about how they move back and forth over time.

Using a custom algorithm, the captured frames from each camera can be compared to more accurately determine the movements of the vibrating speckled laser patterns up to 63,000 times per second — or as fast as an expensive high-speed camera could.

This approach allows audio to be extracted separately from different sources in a single video, such as multiple musicians each playing their own guitar, or even multiple speakers all playing different music.

The extracted audio isn’t as clear or high-fidelity as what a traditional mic can capture, but the optical mic can provide mixing engineers with an easy-to-follow way to monitor individual instruments during a live performance, and there’s little doubt over time. about the quality of the extracted audio will continue to be improved. In addition to music, the system has other interesting applications. A video camera that monitors all machines on a factory floor, or focuses on the engine of a moving car, can determine when individual parts or components are making an abnormal noise, indicating that maintenance may be necessary before a problem actually becomes a problem .

#camera #sees #vibrations #record #sounds #multiple #instruments #separately

Leave a Comment

Your email address will not be published. Required fields are marked *