Amazon Issued Patent for Alexa Technology to Determine Demographics, Emotional Status, and Health Conditions from a User’s Voice (and Offer Targeted Ads)

Amazon has been issued a patent on an Alexa technology that has the ability to determine certain physical and emotional characteristics of users based on their voice input, and offer help – sometimes in the form of offers for items for sale. While the patent doesn’t mean Amazon plans to launch products with this technology, it does reflect Amazon’s thinking on Alexa’s potential.

The patent, titled “Voice-based determination of physical and emotional characteristics of users,” is number 10,096,319. The patent covers Alexa’s ability to infer certain traits about Alexa users from their voice when determining how to respond:

Traits may include physical characteristics of a user (e.g., gender, age, ethnic origin, etc.), a physical condition or state of a user (e.g., sore throat, sickness, etc.), an emotional condition or state of a user (e.g., happy, sad, tired, sleepy, excited, etc.), and other traits.

Amazon provides an illustration in the application showing what it intends to cover. In the illustration, a woman is shown to be coughing and sniffling and tells Alexa she’s hungry. The Alexa system is able to determine her “abnormal physical or emotional condition,” and asks if she would like a chicken soup recipe, which she declines. At that point, Alexa takes the initiative to offer her another remedy and says, “By the way, would you like to order cough drops with 1 hour delivery?” When the woman accepts Alexa’s offer, Alexa confirms and concludes by saying, “Feel better!”

Digging deeper into the patent reveals some interesting details. Here are some highlights based on our reading of the patent:

1. Amazon has patented the technology to determine demographic characteristics of users from their voice — including gender, age, and ethnic origin:

For example, voice features may include a gender of the user, an age or age range of the user, an ethnic origin or language accent of the user, an emotion of the user, a background noise of the environment in which the user is located, and other voice features. As a result, content presented at a device may be specific to the user that is using the device (e.g., providing a voice input, etc.), as opposed to a user associated with the device, such as an owner of the device.

2. Amazon has patented the technology to determine physical characteristics of users from their voice — including certain health conditions:

In another example, a second voice processing or signal processing algorithm may be used to process or analyze the voice data to determine a health condition or status of the user. Detectable or determinable health conditions may include, among others, default or normal, sore throat, cold, thyroid issues, sleepiness, and other health conditions. Example algorithms may analyze breath sounds of the user based at least in part on the voice data and may use a cepstral feature set using SVMs and/or neural networks.

3. Amazon has patented the technology to determine emotional status of users from their voice — including joy, fear, and stress:

The first voice processing algorithm may be used to determine an emotional state of the user. Detectable or determinable emotions may include, among others, default or normal, happiness, joy, anger, sorrow, sadness, fear, disgust, boredom, stress, and other emotional states. Emotional states or conditions may be determined based at least in part on an analysis of pitch, pulse, voicing, jittering, and/or harmonicity of a user’s voice, as determined from processing of the voice data…

If it is determined that the user has an abnormal emotional state, the device or a connected computer may select a real-time emotional state of the user. The real-time emotional state of the user may be, for example, at least one of the happiness, joy, anger, sorrow, sadness, fear, disgust, boredom, stress, or other emotional states.

4. Amazon has patented the technology to target ads to users based on what it determines is their current physical and/or emotional condition:

A current physical and/or emotional condition of the user may facilitate the ability to provide highly targeted audio content, such as audio advertisements or promotions, to the user… For example, certain content, such as content related to cough drops or flu medicine, may be targeted towards users who have sore throats… In the example of FIG. 1, the cough drops manufacturer may have targeted users with sore throats for the promotional offer that was presented to the user 130. The targeting criteria for the promotional offer, or the offer generally, may include users with sore throats or users likely to have sore throats…

Audio content targeted to sleepy and bored users may be determined based at least in part on a data tag that identifies the voice data as a sleepy and bored user. For example, a musician may want to target an audio ad for his new album to users with “boredom” and “sleepy” conditions. Audio content for presentation may be selected from the candidate content and presented to the user. For example, the voice interaction device may audibly present “here’s a joke [ . . . ] By the way, this singer just released his new album for just $1.99. Do you want to preview it?” The user may respond affirmatively or negatively as desired.

Again, technology companies file patents often and do not end up launching products with the patented technology. However, the fact that Amazon filed this patent reflects the potential use cases Amazon sees in Alexa’s future.

In our view, these patented technologies raise some important ethical and philosophical questions that Amazon will likely need to take a clear stance on if it intends to launch products including these features. For example, what bounds should be placed on advertising based on these factors? In which scenarios is taking these factors into account at all when determining a response unethical? Would Amazon ever make such data available to Skill developers through Alexa APIs? What are the privacy implications of these systems knowing what a user’s home life is like?

At the same time, these technologies reflect the potential for “smart assistants” like Alexa to become much more emotionally intelligent and empathetic, creating better user experiences. For example, if Alexa could determine a user was in a happy mood when asking Alexa to “play music,” Alexa could automatically infer the user’s emotional state when choosing which songs to play.

Voice interfaces create opportunities for ambient computing to become deeply integrated in our physical spaces, and this patent addresses some important aspects of what that could mean for the future of human-computer interaction and commerce.

Follow TJI as we continue to track the development of the Alexa ecosystem and what it means for hardware developers, software developers, and customers.

Amazon Launches Several Solimo Brand Private Label First Aid Products

Continuing its expansion into private label health care products, Amazon has launched several new Solimo brand first aid items.

The new items include:

Vine reviews for Solimo Hand Sanitizer are dated since September, though the rest of the items don’t have any Vine reviews. It’s possible they may in the coming weeks.

This is Amazon’s first private label hand sanitizer product. It also offers hand sanitizer from the Amazon Exclusive Mountain Falls brand, which it also labels as “Our Brand”.

Currently, hand sanitizer searches in Amazon’s US store are dominated by sponsored listings and headline search ads from Purell. Purell is the flagship brand of GOJO Industries, Inc., which is headquartered in Akron, Ohio.

Purell and Mountain Falls dominate organic search results for “hand sanitizer” on Amazon’s US store as well, with Germ-x, Babyganics, Cleanwell, and Bath and Body Works also present.

Amazon is in the process of ramping up its physical retail infrastructure as well. It’s possible that Amazon could stock those stores with first aid items, decreasing the number of trips made to traditional retail drug stores like Walgreens, Longs, Rite Aid, Walmart, Target, and many others.

Perrigo Launches Amazon Exclusive Infant Formula Brand, “Love & Care”

Perrigo, one of the largest manufacturers of private label OTC pharmaceuticals, has launched a new Amazon Exclusive infant formula brand called Love & Care.

It’s the first infant formula Amazon Exclusive brand we’ve seen. Currently, Earth’s Best (owned by Hain Celestial Group), Similac (owned by Abbott Laboratories), Enfamil (owned by Mead Johnson), and Happy Baby brand infant formulas rank the highest in Amazon US search results for infant formula. Plum Organics, Happy Baby, Love & Care, and GoodSense are the top brands advertising on those searches.

While “Love & Care” brand items are not yet showing up with the “Our Brand” label in search results, Amazon is promoting the new Love & Care brand in the baby healthcare section of its US website with ads like the following:

Perrigo has not made any announcements that we can find, and the company has not yet responded to our request for comment. We will update if they do. Vine reviews have started coming in in recent days, suggesting that Perrigo may be waiting to promote the new brand until after some initial reviews come in.

Perrigo filed for the US trademark for “Love & Care” on May 9, 2018. The company also produces the Amazon Exclusive Basic Care brand, which includes a variety of first aid health care products and OTC medicines.

Follow TJI for the latest on Amazon’s private label and exclusive brand efforts.

Amazon Launches Revly, a New Private Label Vitamin Brand

Amazon has just launched a new private label vitamins brand called Revly. As of today, there are 11 Revly brand products live on Amazon’s US store. Vine reviews are still coming in in support of the launch.

The US vitamin market was estimated to be about USD $13 billion in 2017.

This marks the 5th private label brand under which Amazon has launched vitamins. It has also created private label vitamin products under the Amazon Elements, Solimo, Mama Bear, and Whole Foods’ 365 Every Day Value brands.

Amazon also sells vitamins under several Amazon Exclusive brands, including Nature’s Wonder, Metatrition, and Pure By Nature, that it also labels “Our Brands”.

Revly vitamins are generally priced on the higher end of Amazon’s private label brands. For example, Revly Vitamin D3 2000 IU gummies sell for $0.17 each, while Solimo Vitamin D3 2000 IU gummies sell for $0.06 each.

Top third party brands in organic US search results for vitamins currently include Nature’s Bounty, NatureWise, Garden of Life, Viva Naturals, Jarrow, and Bronson. Top advertisers currently are NatureWise, Bronson, and Pure.

Follow TJI for the latest on Amazon’s private label brand efforts.

Amazon Launches New Private Label Sports Supplements Brand, OWN PWR

Coming on the heels of its launch of its first private label mattresses, Amazon has launched a new private label sports supplements brand.

Amazon’s brand is called OWN PWR. Currently it includes 11 products — 8 general sports nutrition products (like whey protein and creatine) and 3 “Elite” pre-workout powder formulations.

Amazon has been making its OWN PWR items available on an invite-only basis since late August — we do not know how many invites Amazon sent out, but interested parties could apply on the product pages directly. However, as of today, OWN PWR products now appear to be available for sale without an invitation. All customer reviews we’re seeing are from the last few days.

Amazon has a number of private label and exclusive brands in the sports and nutritional supplement category.

Earlier this year, Amazon launched the Amfit Nutrition private label brand in its UK and EU stores. There are 2 products currently offered under the Amfit brand: whey protein powder and chocolate protein bars.

In addition, Amazon has multiple Amazon Exclusive brands in the US that offer nutritional supplement products. Amazon labels these “Our Brands” as well. They include:

  1. Enraged Nutrition – offers creatine and glutamine powders
  2. Flexatarian – offers a variety of whey protein and pre-workout powders
  3. P2N Peak Performance Nutrition – offers a variety of protein powders

As always, stay tuned to TJI for the latest as we continue to track Amazon’s private label efforts.

Amazon Hiring National B2B Healthcare Sales Leader, to be Based in Boston

Amazon is hiring a National Healthcare Sales Leader to further develop its business of selling supplies to hospitals and clinics. In the newly posted position, Amazon says that the business is experiencing “rocket ship” growth. The position is based out of Boston/Cambridge.

According to Amazon, “This leader will define strategy and execute tactical plans to help enterprise healthcare organizations (acute and non-acute) reinvent the way they buy indirect supplies.” This position will also, “Hire and Develop customer advisors and sales leaders to support continued, ‘rocket ship’ growth.”

Amazon also says it is looking for candidates with the, “Ability to converse with senior healthcare customer stakeholders including CPOs, VP’s Supply Chain, CIOs, and others,” as well as a, “Proven ability to close complex deals and help reps negotiate large deals.”

Amazon’s growing interest in the healthcare industry has garnered increasing attention over the last year.

The company has formed a Boston-based health care “non-profit-seeking” joint venture with Berkshire Hathaway and JP Morgan and hired a CEO, surgeon and public health leader Atul Gawande. Gawande has yet to announce more details on the initiative’s direction or strategy, other than that he wants it to, “Take some of the middlemen out of the system.”

A month ago, Amazon hired cardiologist Maulik Majmudar for what he described as an “exciting & challenging role.” Amazon has also recently hired Martin Levine, a primary care expert who has experience running a network of clinics in Seattle, and Taha Kass-Hout, the former US FDA chief health informatics officer. Amazon is also reportedly building a “health and wellness” team within its Alexa division that is working through regulatory issues like HIPAA compliance.

This summer, Amazon announced it had reached an agreement to acquire the online pharmacy Pillpack. The deal is expected to close by the end of 2018.

Amazon also offers a variety of exclusive retail brands targeting OTC medicine and essential consumer healthcare supplies.

The full job listing is embedded below.

What if Amazon Put Pharmacies in 3,000 Go Stores Across America?

Yesterday’s rumor that Amazon is considering opening 3,000 Go stores in the next three years sent shockwaves through the grocery and retail markets. Certainly, were Amazon to invest the billions of dollars needed to build out such a physical footprint needed to reach that scale, it would likely pose significant challenges for incumbent retailers, quick-serve restaurants, convenience stores as well.

However, given that Amazon is scheduled to close on its acquisition of PillPack by the end of the year — a story which itself sent its own respective shockwaves through the healthcare markets — another interesting angle on the “thousands of new Amazon stores” story is that of pharmacies and healthcare. With PillPack, Amazon will get pharmaceutical licenses in almost every state.  What would that mean for CVS, Walgreens, Walmart, Rite-Aid, grocery pharmacies, and the rest?

While Amazon has made several high-profile healthcare-related hires lately, we haven’t seen specific evidence that Amazon intends to provide physical healthcare services, a la Minute Clinics. But over time we think this is an area to keep an eye on given the potential revenue growth opportunity for Amazon.

Ted Gibson Launches Professional Beauty Products Exclusively on Amazon

Via BeautyPackaging:

Celebrity hair stylist/salon owner Ted Gibson has launched a new luxury professional hair care brand, sold exclusively on Amazon — at least for now.

Starring by Ted Gibson is Amazon’s first-ever professional beauty brand exclusive. The brand has partnered with Amazon Luxury Beauty and its new Beauty Exclusives program. Gibson told WWD, “Salons cannot afford to put a $10,000 or $15,000 opening order on their shelf anymore…the way women want to shop and the way hairdressers think is not that way anymore.”

Amazon Reportedly Planning on Opening Up to 3,000 Amazon Go Stores in 3 Years

Amazon is reportedly considering rapidly increasing its pace of investment in Amazon Go stores by opening up to 3,000 of them over the next 3 years. Per Bloomberg:

Amazon.com Inc. is considering a plan to open as many as 3,000 new AmazonGo cashierless stores in the next few years, according to people familiar with matter, an aggressive and costly expansion that would threaten convenience chains like 7-Eleven Inc., quick-service sandwich shops like Subway and Panera Bread, and mom-and-pop pizzerias and taco trucks. Chief Executive Officer Jeff Bezos sees eliminating meal-time logjams in busy cities as the best way for Amazon to reinvent the brick-and-mortar shopping experience, where most spending still occurs…

Amazon is targeting dense urban areas with lots of young, busy, affluent residents willing to spend a little more than a typical fast-food experience for better quality food, the people said. The target locations make it less of a threat to suburban gas station-convenience store combinations and more of a threat to big cities’ quick-service eateries, such as Subway Restaurants, Panera Bread Co. and Pret a Manger.

Our take:

  • Convenience stores have existed for a long time, but Amazon has unique data on consumer behavior and shopping patterns, and could positions its Go stores in locations it believes will be most likely to change behavior.
  • Most convenience stores are currently located at or near gas stations, and typically primarily stock candy, soft drinks, snacks, beer, and tobacco products.
  • We would view this as more of a threat to quick-serve restaurants, grocery stores, and (potentially) pharmacies.

For context, Amazon just opened its fourth Go store yesterday.

Amazon Launches 3 Mama Bear Brand Kids Vitamin Products

Expanding its push into the private label vitamin and supplement market, Amazon has launched three new kids vitamins products under its Mama Bear brand. The new items are:

Amazon recently launched 10 new adult vitamin products under the Amazon Elements brand as well. Other brand owners in the kids vitamin market include Bayer (which owns the Flintstones and One a Day brands), SmartyPants, and Rainbow Light.

Stay tuned to TJI for the latest developments as Amazon continues to expand its private label efforts.