‘Retailers have to be ready for it’: Visual search sits on the brink of a breakthrough

In 2016, in an effort to aid customers with this exact problem, the retailer added visual search to its mobile app by adding a camera icon to the search bar. When a customer takes or uploads a photo of a part they may need, machine learning and computer vision technology scan every product in Home Depot’s 1-million product catalog to render the most accurate results matching the image and tell you where in the store it is, or send you to the online page it can be found.

Rather than build the visual search in-house, as it typically does (90 percent of Home Depot’s code is written internally), the company hired the computer vision platform Slyce to lower the risk in investment. And it launched in beta mode, rare for the company.

“We knew this was the right way to solve this problem, but we were a little worried about this when we launched. It’s a lot harder to nail visual search accuracy,” said Matt Jones, Home Depot’s senior director of online and mobile product. “I tempered my expectations on adoption rates, and we partnered up. Vendors act as force multipliers.”

He was right to be skeptical. Two years in, and with some in-app and on-site promotion, Home Depot has found that visual search represents a single-digit percentage of all searches on the mobile app. But Jones said the company is still investing in improving the accuracy of the technology, as well as the user experience. It’s critical, he said. The retailer has to be ready — visual search isn’t quite there yet, but it will be soon.

“This is where, if you’re a retailer looking three years, five years out, you should be investing in,” said Yury Wurmser, visual search, marketing and retail analyst at eMarketer. “It’s going to grow a lot, and retailers need to be prepared for it.”

What can’t be put into words
According to eMarketer’s 2018 Visual Search report, about 1 billion visual searches happen per month. That’s a tiny fraction of overall search, which renders hundreds of billions of queries per month, but it’s growing. On Pinterest, monthly visual searches went from 250 million per month in February 2017 to 600 million per month in February 2018.

Retailers, particularly in the apparel, home goods and home improvement categories, are in the early days of using the technology. Amazon, Target, Bed Bath & Beyond, West Elm, Wayfair, Neiman Marcus, Macy’s, Asos, H&M and Forever 21 have visual search capabilities built into either their e-commerce sites or mobile apps to answer the search queries that can’t easily be asked in a text field, like “What kind of chandelier am I looking at?”

Some companies, like Wayfair and Farfetch, have built visual search technology in-house. At Farfetch, a 15-person computer vision team is working on powering the capability, not just to render accurate search results but to build product recommendations off of the context of an image, to act as a styling guide around what jacket matches which dress. It also works with the platform Syte to boost the technology.

“There’s a huge intellectual challenge with computer vision, which is finding meaning. But that’s where we unlock a huge level of consumer engagement. So we need to nail it,” said Farfetch CMO John Veichmanis. “We can’t just be a luxury search engine. We need to be able to break down the context of an Instagram, or any inspirational image, and say if you want to achieve this style, here’s what to buy.”

Veichmanis said that customers who use visual search on the Farfetch app spend more dwell time in the app than other customers, and that since launching the tool three months ago, the company will continue to improve upon the results to get personalized recommendations down to the individual level. That takes a lot of data, and metadata.

“It’s a heavy technical cost to launch and upkeep this technology,” said Vic Drabicky, founder of the data agency January Digital. “We’re telling retailers to start now, whether you build it yourself or partner with someone. Because this will shake out the way text search did: You’ll have your big scale platforms where searches start, but everyone will need their own version of it, too.”

All eyes on Pinterest
The race is also on among platforms. Pinterest wants to be the first platform to crack visual search at scale.

Several iterations of shoppable pins have passed through Pinterest’s feed, but the most recent version uses computer vision to pull up a feed of “products like this” below the pin. These products are all in stock online. Since launching that feed last year, traffic that Pinterest sends to retailers increased by 40 percent.

“We feel like this is what we are uniquely able to do, be a bridge between products in stock and images,” said Tim Weingarten, head of shopping products at Pinterest. “More and more of Pinterest will have shopping mode. We’ll never force it. But when you have the desire to shop, we want it to be so easy to start shopping.”

Right now, products can appear in Pinterest visual searches one of three ways: Retailers hand Pinterest the keys to their e-commerce catalogs so Pinterest can build product pins, tag them, and track details like availability and price accordingly; retailers upload their own lifestyle imagery and then tag their own products, so if a customer comes across the pin on the site they can shop the exact items in the photo; or brands and retailers can buy an ad in the shoppable product feed to be featured in results.

Pinterest wouldn’t disclose what it costs to be featured in product search results, but as of now, it’s the only platform that’s begun to monetize the tool, according to eMarketer. Google’s similar Style Match tool, and Amazon’s Lens results, don’t have sponsored product slots, yet.

As growth rates climb, and as marketers look for alternatives to the duopoly, Pinterest could finally have its moment.

“Down the line, you can really see this wedging its way into marketing budgets,” said Drabicky. “Pinterest has been a hidden gem, but it hasn’t been able to break through and become a have-to-do-it type of platform. Visual search has the opportunity to fill that spot, and especially as Facebook continues to trip, people are looking for that. Google will be great at visual search as well, but its approach has covered a lot more ground than retail. Pinterest has a crack in the door to be a mainstream partner.”

Halo effect
As platforms make headway in bringing visual search to scale, retailers will have to be ready on the back-end. In general, visual search will push e-commerce sites to better catalog their imagery and improve checkout pages so that people can easily purchase when they land on a product, especially on mobile. Wurmser predicts that in the next two years, many more visual search ad products will surface.

There’s more motivation than the platform partnership. Wayfair’s head of product, Matt Zisow, sees value in owning the end-to-end digital experience as it navigates visual search, computer vision and machine learning technology. Wayfair has a catalog of 10 million products, so visual search plays a role in everything from augmented reality interior design tools, to product searches, to replacements for out-of-stock items, to personal recommendations. Overall, making sure that every single product image on the Wayfair said is coded with a computer vision algorithm has lifted the user experience of the site, by increasing the relevance of results.

Veichmanis said that visual search will have long-term effects on how Farfetch predicts trends and inventory buys because it will compile visual data around customer interests earlier on.

The only thing left is to get more people to use it.

“Visual search has a strong use case all the way up and down the funnel, which makes it extremely valuable,” said Drabicky. “Now it’s just a matter of hammering it in — putting it in front of people’s faces — and saying, ‘This is the easiest way to find what you’re looking for.’”

Subscribe to the Digiday Retail Briefing: A weekly email with news, analysis and research covering the modernization of retail and e-commerce.