AI Briefing: This startup is using Salesforce for its AI-powered semantic search for e-commerce

“How people handle that today is [by using] a very separate machine-learning stack for the retrieval layer,” co-founder Nigel Daley told Digiday. “And most of the retrieval layers are kind of fairly dumbed down. They’ll try to bring up as many products as they can back and they think will [be of interest], but they kind of cast a wide net. Problem is, the wider the net, the more it costs.”

Rather than just focusing on text or visuals, Daley and his co-founder Lance Riedel are applying integrating both — thanks in part to prior experience in search, shopping and personalization at Pinterest. They said building Pinterest’s shopping platform helped them understand not only how to build search tools but also how to build a platform that understands users’ tastes and preferences that can then be used for personalized recommendations.

Launched in early 2024, the startup’s already raised $20 million and has early investors such as Twitter co-founder Biz Stone and Pinterest co-founder Evan Sharp. When asked why Vantage Discovery is integrating with Salesforce, Daley said Commerce Cloud’s size, functionality, data and the central role in commerce all play a part. “Also just kind of recognizing that search product understanding is so fundamental to retailers, whether they realize it or not, understanding how things go together.“

Here’s how it works: By integrating with Commerce Cloud, Vantage Discovery integrates with e-commerce platforms to understand their product catalogs using LLMs and visual/semantic analysis. Rather than relying on keywords, a shopper can look for styles based on more nuanced queries like style, location or occasion that might get lost via traditional keyword-based search. For example, Vantage will let a website understand “kicks” and “pumps” refer to shoes, or understand what a shopper means if they’re looking for a dress for a beach wedding. Vantage Discovery also has a tool to give shoppers more personalized search results based on images they upload to retailer’s website.

“We can go and discover more like the things that we thumbs up and less like the things we thumbs down,” said Riedel. “This is a great way to let users, once they’ve done a text query, go down the funnel right and start using the visual to navigate through the product catalog — very much like you would if you’re on Pinterest.”

One early adopter is Cooklist, a platform that helps grocery shoppers create recipes based on the food they buy. According to Cooklist COO and co-founder Brandon Warman, the app had struggled with basic search functionality in the past without being able to always effectively handle complex user queries or understand context. For example, users searching for “Christmas cookies” would get results with just “Christmas” in the title but without relevant options like gingerbread cookies. 

That’s why using Vantage’s platform for semantic search was especially helpful for understanding context. Since Cooklist integrated Vantage Discovery a few months ago, Warman said Cooklist hasn’t necessarily seen higher search volume per user, but it has already seen results from improved search relevancy. Metrics that have increased include average saves per recipe search, average recipe cooks per user and average item counts per shopping list. Cooklist has also been finding other ways improved search will help users that users haven’t yet tried — like searching for recipes for a five-year-old’s birthday party and getting more relevant results.

“As we move forward, there’s an opportunity for us to allow our users to be a little bit more wild with their search,” Warman said. “That’s also on us to kind of show or teach users all the things that the search is now capable of doing. We’re just in the beginning stages of this integration.”

Cooklist already has partnerships with major grocers including Walmart, Amazon, Target, Albertsons, and Tom Thumb. And though the app doesn’t use Salesforce, Warman said Vantage’s integration with Commerce Cloud could bring a “halo factor” since many of its current and future grocery partners use Salesforce. 

When asked if Vantage helps with Cooklist’s ads business, Warman said it’s not something the app has explored yet but it could hypothetically help improve content relevancy in the future. He also noted that semantic search upgrades from Vantage have already helped improve users’ personalized results. 

“Now that we’ve integrated Vantage search, that translates to higher basket value, or higher order values that we’re sending to the retailer,” Warman said. “So they’re benefiting from this increase in order value and increase in shopping list size from us, which is what we think is directly correlated to the semantic search advantage.”

Dreamforce overall

Vantage Discovery is just one of many AI-related updates from Dreamforce 2024. Last week, Salesforce debuted a number of new products for marketers and sales teams including a new virtual “marketing assistant” powered by generative AI and predictive AI. The new Agentforce for Marketing tool can be used for generating campaign briefs, defining target audiences, drafting landing pages, and building customer journeys — some tasks that competitors have already added to their marketing platforms. 

One key update is being able to take action with LLMs, said Salesforce Marketing Cloud CMO Bobby Jania. He offered an example of Agentforce’s abilities: “I can type in a prompt of what I’m looking for. I want shoppers who have a high customer lifetime value or who haven’t purchased in the last six months but have been on the website in the last month.”

“That is a pain to create manually,” Jania told Digiday. “Sometimes you have to code it. You have to understand all of your data, which requires potentially understanding the metadata. I was talking to a customer earlier this week who said they had 600 attributes per profile. Imagine as a marketer trying to remember what all those are and create a segment. But AgentForce will remember that it knows the metadata.”

Salesforce also debuted new tools powered by its own Einstein AI platform, including new AI-powered analytics through Salesforce’s Tableau platform. The company also announced new customer personalization tools through Salesforce’s Data Cloud to help with suggesting relevant products and content. Finally, Salesforce announced an updated version of Marketing Cloud that has more automation features.

Updates from Salesforce and other giants like Microsoft show how tech giants’ visions for AI agents are reshaping retail and commerce, according to Elav Horwitz, global head of innovation at McCann. However, Horwitz said it’s key companies not rush adoption at the expense of creativity even if AI agents can help streamline processes.

“Without human ingenuity driving them, we risk creating bland, homogeneous experiences for consumers,” Horowitz said. “Companies must carefully map out where AI can take over and where it can’t. The human touch is irreplaceable – automation can enhance but not erase personal connection.”

Salesforce could secure an advantage having so much customer data at its disposal, said Steven Moy, the former CEO of Barbarian Group who now runs his own consultancy for agencies and brands. He also thinks new AI updates might help accelerate businesses use data to find and fulfill new customer needs and change how they approach R&D.

Moy also wondered if the new tools will still have a steep learning curve. Another question is whether companies should use a variety of companies’ AI tools based on which works best or companies will prefer to consolidate uses within single platform. 

“[Companies] have been dying to get more customer data for every brand,” said Moy, who has helped companies overhaul their ecommerce strategies. “One aspect the AI could help is [finding and] creating more inclusive customer data. I can talk about an indexing engine the way we taught our engine look for imagery … We need to make the data more enriched and diverse and inclusive.” 

Prompts and Products — AI News and announcements 

• Microsoft debuted CoPilot Wave 2, which includes a range of new generative AI tools for enterprise customers.

• The Associated Press and the AI news startup AppliedXL announced a new partnership to help local newsrooms use AI to cover local city news and analyze how federal government decisions could have an impact locally.
• OpenAI unveiled its new Strawberry chatbot, which the startup claims has the ability to reason. However, OpenAI reportedly has also threatened to ban users that dare to ask Strawberry probing questions about its reasoning skills.
• The generative AI video startup Runway announced a new partnership with studio LionsGate that could let the entertainment giant utilize its content library in new ways. However, it merits noting that Runway and other AI startups are still facing legal fights related to the content AI models were trained on. (Runway also announced new video-to-video AI capabilities.)
• LinkedIn faced criticism for starting to train its AI models on user data before giving people a chance to opt out. 
• The FTC released a new report about how social companies engage in surveillance of users, prompting new concerns about privacy and AI.
• Snap debuted its new Spectacles smart glasses equipped with AI and augmented reality.  Meanwhile, Meta announced an extended partnership with Luxottica for its AI-powered Meta Ray-Ban smart glasses.
• Google announced it’s adding C2PA image content credentials for generative AI content in search and ads. Meanwhile, it also announced updates for YouTube that include using AI for creating videos.
• Speaking of Google, read more about Digiday’s ongoing coverage of the Google adtech antitrust trial.