WTF is acoustic fingerprinting?

What is acoustic fingerprinting?
This is the same technology that Shazam uses, the app that can identify a song on the radio. Acoustic fingerprinting enables a device to pick up as little as two seconds of a sound; it then translates it into a code that can be matched against a code in a database to identify what that sound is. Using a code means that the device doesn’t share actual audio from the user, which could mitigate privacy concerns. Companies like China-based ACRCloud license such acoustic fingerprinting technology to major apps like Alibaba, which offer services to users who use the audio features while watching TV or while shopping.

How does acoustic fingerprinting work with TV?
The technology is being used more and more with television, becoming embedded in apps for hit shows like The Voice. “It only records a few seconds of the sound from the TV, and it instantly recognizes what kind of program is playing or what TV advertisement is playing, and then [consumers] can press a button to get what they want,” said ACRCloud’s Peng Dong. Apps like Alibaba and WeChat, which has its own fingerprinting tech, offer rewards to people who use the audio services when prompted during shows and commercials and in stores.

What other purposes does audio content recognition serve in marketing and media?
With all these listening feelers out in the world in the form of mobile devices, marketers are able to get better measurements and data surrounding when their ads run. TV programs can get a better idea of how many people watch their shows, and from device data, they can know the audience better. In Mongolia, for instance, Peng said ACRCloud licenses its tech to many local apps that listen for TV programming during set hours each night to measure audiences.

The same happens in the U.S. through Audible Magic’s technology. “We work with advertising measurement companies, that are the equivalent to Nielsen, where they pay panelists to essentially know what they’re watching on TV. Basically, they supply their panelists with smartphones that will listen to the audio all day long, and tell them what TV shows or TV commercials they actually watch,” said Jay Friedman, a spokesman for Audible Magic.

Are there any privacy concerns?
The apps that employ audio fingerprinting are all opt-in experiences, and iPhones and Android devices reliably notify users any time an app requests access to the mic, according to Friedman. Also, the iPhone has an indicator — a red light — that shows when the microphone is on to prevent surreptitious recording.

Then why is this so controversial?
Well, there are concerns over specific devices, like Samsung TVs, which the company recently warned users could pick up unintended audio from inside homes. Also, Facebook users have expressed fears that the app is listening since it launched the ability to identify TV shows and music. Of course, those concerns were rumors, but it shows the sensitivity around the idea that phones are always listening. Most recently, the Federal Trade Commission warned Android phone developers against a different kind of listening, called acoustic watermarking, where the device picks up an inaudible frequency to collect data.

What is acoustic watermarking, then?
Acoustic watermarking listens for a background signal that is inaudible and could be buried in some other sound. The watermark identifies whatever is playing on TV and can help measure if a user saw a certain commercial, for instance. Last week, the FTC warned app developers not to put this to use without due notice given to consumers. “For acoustic fingerprinting, it has to have sound first; it doesn’t matter what sound is playing. It could be English, it could be Chinese or some chicken sound, but it has to have a sound. That’s the basic thing,” Peng said.

Is this technology being used in the real world?
Yes, some stores employ acoustic fingerprinting or watermarking to detect who visits, and they can send sales offers. If the consumer opts into receiving such offers, then a watermark could be detected through apps on the person’s phone, apps like Shopkick. Azher Ahmed, director of digital at DDB Chicago, said acoustic technology can be more effective than beacons because they have a wider range and don’t require Internet connections for the app to pick up on it.

What else can acoustic fingerprinting be used for?
It’s put to use in content rights management and lets companies like Facebook listen for copyrighted material as it’s uploaded. Any content that matches content with a copyright can be flagged, according to Friedman of Audible Magic. The same principle can be applied to media planning, and the technology can be used to show that ads ran correctly. “From a licensing perspective, it can validate that something is playing exactly according to the media buy,” Ahmed said.