Amazon Issued Patent for Alexa Technology to Determine Demographics, Emotional Status, and Health Conditions from a User’s Voice (and Offer Targeted Ads)

Amazon has been issued a patent on an Alexa technology that has the ability to determine certain physical and emotional characteristics of users based on their voice input, and offer help – sometimes in the form of offers for items for sale. While the patent doesn’t mean Amazon plans to launch products with this technology, it does reflect Amazon’s thinking on Alexa’s potential.

The patent, titled “Voice-based determination of physical and emotional characteristics of users,” is number 10,096,319. The patent covers Alexa’s ability to infer certain traits about Alexa users from their voice when determining how to respond:

Traits may include physical characteristics of a user (e.g., gender, age, ethnic origin, etc.), a physical condition or state of a user (e.g., sore throat, sickness, etc.), an emotional condition or state of a user (e.g., happy, sad, tired, sleepy, excited, etc.), and other traits.

Amazon provides an illustration in the application showing what it intends to cover. In the illustration, a woman is shown to be coughing and sniffling and tells Alexa she’s hungry. The Alexa system is able to determine her “abnormal physical or emotional condition,” and asks if she would like a chicken soup recipe, which she declines. At that point, Alexa takes the initiative to offer her another remedy and says, “By the way, would you like to order cough drops with 1 hour delivery?” When the woman accepts Alexa’s offer, Alexa confirms and concludes by saying, “Feel better!”

Digging deeper into the patent reveals some interesting details. Here are some highlights based on our reading of the patent:

1. Amazon has patented the technology to determine demographic characteristics of users from their voice — including gender, age, and ethnic origin:

For example, voice features may include a gender of the user, an age or age range of the user, an ethnic origin or language accent of the user, an emotion of the user, a background noise of the environment in which the user is located, and other voice features. As a result, content presented at a device may be specific to the user that is using the device (e.g., providing a voice input, etc.), as opposed to a user associated with the device, such as an owner of the device.

2. Amazon has patented the technology to determine physical characteristics of users from their voice — including certain health conditions:

In another example, a second voice processing or signal processing algorithm may be used to process or analyze the voice data to determine a health condition or status of the user. Detectable or determinable health conditions may include, among others, default or normal, sore throat, cold, thyroid issues, sleepiness, and other health conditions. Example algorithms may analyze breath sounds of the user based at least in part on the voice data and may use a cepstral feature set using SVMs and/or neural networks.

3. Amazon has patented the technology to determine emotional status of users from their voice — including joy, fear, and stress:

The first voice processing algorithm may be used to determine an emotional state of the user. Detectable or determinable emotions may include, among others, default or normal, happiness, joy, anger, sorrow, sadness, fear, disgust, boredom, stress, and other emotional states. Emotional states or conditions may be determined based at least in part on an analysis of pitch, pulse, voicing, jittering, and/or harmonicity of a user’s voice, as determined from processing of the voice data…

If it is determined that the user has an abnormal emotional state, the device or a connected computer may select a real-time emotional state of the user. The real-time emotional state of the user may be, for example, at least one of the happiness, joy, anger, sorrow, sadness, fear, disgust, boredom, stress, or other emotional states.

4. Amazon has patented the technology to target ads to users based on what it determines is their current physical and/or emotional condition:

A current physical and/or emotional condition of the user may facilitate the ability to provide highly targeted audio content, such as audio advertisements or promotions, to the user… For example, certain content, such as content related to cough drops or flu medicine, may be targeted towards users who have sore throats… In the example of FIG. 1, the cough drops manufacturer may have targeted users with sore throats for the promotional offer that was presented to the user 130. The targeting criteria for the promotional offer, or the offer generally, may include users with sore throats or users likely to have sore throats…

Audio content targeted to sleepy and bored users may be determined based at least in part on a data tag that identifies the voice data as a sleepy and bored user. For example, a musician may want to target an audio ad for his new album to users with “boredom” and “sleepy” conditions. Audio content for presentation may be selected from the candidate content and presented to the user. For example, the voice interaction device may audibly present “here’s a joke [ . . . ] By the way, this singer just released his new album for just $1.99. Do you want to preview it?” The user may respond affirmatively or negatively as desired.

Again, technology companies file patents often and do not end up launching products with the patented technology. However, the fact that Amazon filed this patent reflects the potential use cases Amazon sees in Alexa’s future.

In our view, these patented technologies raise some important ethical and philosophical questions that Amazon will likely need to take a clear stance on if it intends to launch products including these features. For example, what bounds should be placed on advertising based on these factors? In which scenarios is taking these factors into account at all when determining a response unethical? Would Amazon ever make such data available to Skill developers through Alexa APIs? What are the privacy implications of these systems knowing what a user’s home life is like?

At the same time, these technologies reflect the potential for “smart assistants” like Alexa to become much more emotionally intelligent and empathetic, creating better user experiences. For example, if Alexa could determine a user was in a happy mood when asking Alexa to “play music,” Alexa could automatically infer the user’s emotional state when choosing which songs to play.

Voice interfaces create opportunities for ambient computing to become deeply integrated in our physical spaces, and this patent addresses some important aspects of what that could mean for the future of human-computer interaction and commerce.

Follow TJI as we continue to track the development of the Alexa ecosystem and what it means for hardware developers, software developers, and customers.

Amazon Luring Game Developers to Alexa with New Consumable Virtual Goods APIs

A strategic priority for Amazon is growing the Alexa ecosystem. An important component of that strategy is attracting high quality skill developers to create compelling content for Alexa customers. Key to attracting developers is providing attractive monetization opportunities for skills to not only be engaging but also generate revenue. The Alexa team recently added some new monetization opportunities to that end with the launch of consumables.

Consumable virtual goods are fundamental to game developers who want to monetize via the freemium model. Under the freemium model, it’s free for customers to play the game, but players can enhance and advance gameplay more quickly through the use of items that must be purchased, and then used/depleted, and then purchased again. This has worked well for mobile and social game developers since it allows customers to make small purchases incrementally. Some customers end up buying a lot and are known as whales.

Previously, Alexa In-Skill Purchasing APIs allowed developers to monetize through subscriptions (recurring purchases that provide ongoing access to content) and entitlements (one-time purchases that do not expire).

One important note for developers – consumable prices must be in the range of $0.99 to $9.99. Subscription and entitlement prices can range from $0.99 to $99.99. So the consumable max is 90% less than other types of purchases.

Game development on Alexa is nascent, with most games falling into the quiz category. However, we have seen some early strategy games crop up, in addition to several social games designed to be played by groups together.

Stay tuned to TJI for the latest on the Alexa Skill monetization market.

Amazon Invests in a Home Builder For the First Time

Plant Prefab, which designs and builds prefabricated homes using what it calls “sustainable” methods, said today it has raised a $6.7 million Series A funding round from investors including the Amazon Alexa Fund and Obvious Ventures, the fund co-founded by Evan Williams of Twitter/Blogger.

It’s the first time we’re aware of that Amazon has directly invested in a home builder. It has previously announced partnerships with Lennar Homes in the US and with Embassy Edge in India offering smart home designs built on Alexa.

“Plant Prefab is a leader in home design and an emerging, innovative player in home manufacturing. We’re thrilled to support them as they make sustainable, connected homes more accessible to customers and developers,” said Paul Bernard, director of the Alexa Fund.

Plant Prefab says it has “installed 26 units in California and Utah as well as a multifamily project in Berkeley” in recent months.

Our take:

Given the importance of the home to Amazon, it is not shocking to see Amazon get more directly involved in home building. Amazon sees opportunity in being involved in home design and construction (particularly in the context of new home building methods for urban areas). Would Amazon ever design or build homes itself?

Amazon Hiring for 67 Alexa Automotive Engineering and Design Roles

Yesterday’s Echo product announcements were primarily focused on the home, but Alexa Auto made it into the lineup as well. While Amazon didn’t spend a large portion of its press event talking about its automotive plans, it is currently hiring for dozens of engineering design roles as part of its Alexa Automotive efforts.

Amazon is working with a variety of car manufacturers on Alexa integrations and announced the Auto SDK back in August.

Currently, Amazon.jobs shows 67 Alexa Automotive jobs currently open. The vast majority are engineering and design roles. Senior architect, software development, language specialist, VUI design, and QA roles are largely based out of the Santa Clara office, while other research scientist and marketing roles are generally based in Seattle.

Automotive acoustics engineer and product management roles are based in Sunnyvale, and partner manager and design technologist roles are based out of Munich and Tokyo. Several engineering roles are also based in Toronto and Gdansk.

The Big Picture on Today’s Echo Product Announcements: Amazon’s Subscription Services Strategy

In the future, we may experience Amazon primarily as a home (and business) operations automation company. A good portion of what we think of as “e-commerce” today may come to largely feel like automated supply replenishment happening in the background.

Amazon and Home Automation Infrastructure

In a certain sense, Amazon’s hardware strategy for the home is “boring.” Today, the company announced a clock, a microwave, a microphone, and speakers.

None of these products is going to set the world on fire by itself. But rather, they’re each a piece of the overall puzzle of building a system of unified home infrastructure that connects the hardware we see and touch every day with intelligent services that hold the data we need for daily life.

Today’s announcements are a clear indication that Amazon’s strategy is to build the nervous system that powers the home, and to build at-least-demonstration-quality hardware (or much better in some instances) in a variety of categories to accelerate the pace of innovation in and growth of the systems it wants to power.

A Subscription Services Future

Over 100 million people around the world subscribe to Amazon Prime today.  Millions more likely subscribe to Amazon Music Unlimited. And more are trying FreeTime Unlimited, Prime Channels, Audible, Kindle Unlimited, Pantry/Fresh, and other less-well-known subscription services that Amazon offers every day.

In fact, Amazon has created a subscription service around millions of consumable products it sells. Almonds? Subscribe & Save. Diapers? Subscribe & Save. OTC medicine? Subscribe & Save.

Amazon’s Dash Replenishment Service already re-orders supplies for your home (or business) when it detects supplies are low. (Why order potato chips when your smart potato chip container can automatically keep supply levels above desired thresholds for you.)

As more Alexa-connected devices are deployed in the home (and car), Amazon can provide more varied and sophisticated home services. For example, home security and automated appliance maintenance, just to name a couple related to products announced today.

And with those services comes increasing (and likely higher-margin) recurring subscription revenue. In some sense, selling toilet paper was just step one. (Though we believe Amazon could have improved e-commerce margins in the future as well as private label grows, but will probably just keep prices low.)


Amazon’s product announcements today:

  1. Echo Dot (3rd generation)
  2. Echo Show (2nd generation)
  3. Echo Plus (2nd generation)
  4. Echo Input
  5. Echo Link
  6. Echo Link Amp
  7. Echo Sub
  8. Echo Wall Clock
  9. Echo Auto
  10. Amazon Smart Plug
  11. AmazonBasics Microwave
  12. Ring Stick Up Cam
  13. FireTV Recast
  14. Alexa Connect Kit
  15. Alexa Guard, Local Voice Control, and More Alexa Features
  16. Alexa Presentation Language
  17. Alexa Smart Screen Device SDK

Amazon Releases Alexa Skills Kit SDK for Python

Per popular developer request, Amazon has been working on building an officially-supported Python SDK for ASK. The Alexa tools development team released a beta version back in June. Now, the team has announced that the ASK SDK for Python is generally available:

As we shared when we launched the beta version of the SDK, our goal is to reduce the amount of code you need to write to process Alexa requests and responses and to handle other common skill tasks. You can use the following key features:

  • Request Handling. Request handling in the SDK makes it easy for you to invoke the right code when Alexa sends you a request. You can write a single handler for multiple Alexa intents, or invoke different handlers based on nearly any request attribute. The ASK SDK for Python also introduces flexible handler registration, allowing you to use either decorators or traditional class-based implementations of handler features.
  • Response Building. You can deliver responses to your customers that include text-to-speech, audio and video streams, and cards and other visual elements. Customers will receive one or more of these elements depending on what device they are using. Using the SDK, you can build responses that include all of these elements.
  • Attribute Management. You can store and retrieve information at different scopes using attributes in the SDK. Attributes allow you to keep track of what happened so far, and to use this information to determine what happens next. You can define attributes that persist for a single request, for a single customer session, or for the lifetime of your skill.
  • Alexa API Calls. You can call nearly any Alexa API from within your skill logic using service clients in the SDK. The service clients automatically inject relevant endpoint and authentication-token information on your behalf.

ASK SDKs are also available for Node.js and Java.

Upcoming Alexa Dev Days to be Held in Montreal, Tokyo, Mexico City, and 3 Stops in Europe

Amazon’s Alexa developer education/evangelism teams have been traveling the world this year holding “Alexa Dev Days” to explain the fundamental concepts of how Alexa development works, and to help developers get their feet wet making Alexa skills.

In Q4 2018, Alexa Dev Days will be held in 6 cities around the world:

  • October 4, 2018: Montreal, Canada
  • October 12-13, 2018: Tokyo, Japan
  • November 7-8, 2018: Mexico City, Mexico
  • November 20, 2018: Paris, France
  • December 5, 2018: Milan, Italy
  • December 11, 2018: Madrid, Spain

Alexa Dev Days are a good opportunity for developers to meet other developers in their area interested in building voice apps. The Alexa Skills Kit is a toolkit for starting with Alexa development. Educational resources from Amazon are also online here.

Amazon says that there are over 20,000 Alexa-enabled devices on the market today.

Estimate: Echo Devices Represented 30% of Global Smart Speaker Shipments in Q2 2018

New estimates published yesterday peg Amazon Echo smart speaker shipments at 30% of the overall global market in Q2 2018. Strategy Analytics proffers that the Echo Dot represented 18% of overall shipments, with the Echo representing 12%.

Meanwhile, estimates peg Google devices as taking 27% of the market, with Google Home Mini representing 20% (larger than the Echo Dot) and Google Home taking 7%. They pegged Alibaba’s Tmall Genie at another 7%, with the rest of the market taking 36%.

Our take:

This is a race for the future of controlling home automation systems. We view the deployment of smart speakers into homes and cars as highly strategic for both Amazon and Google. Smart speaker shipment volume is as much about distribution channels as it is services. Amazon and Google are both investing extremely heavily in the AI and ML to power more sophisticated services that these devices are a primary interface to.

Amazon Smart Plug and Echo Sub Leaked on Amazon.co.uk

A day after reports that Amazon is working on several new Alexa-enabled devices, new additions to the Amazon smart home lineup of products have been apparently inadvertently posted and removed from Amazon’s UK store.

The first, Amazon Smart Plug, “works with Alexa to add voice control to any electrical socket,” per Pocket-lint. However, somewhat oddly, the price is listed at £95.00, which is several times higher than that of other smart plugs on the market today. We’d be surprised if it launched at that price.

The second, Amazon Echo Sub, is designed to be paired with another Echo device to achieve “down-firing, 100W deep bass sound through a 6″ woofer,” or two Echo devices “for rich left/right stereo sound,”  per Pocket-lint. It is listed for sale at £75.00, which seems more reasonable.

Our take:

  • Both are holes in the existing lineup of devices Amazon offers. In the smart home category, one could see Amazon getting into additional home controls like switches. In the audio category, Amazon could certainly improve the fidelity of the sound its devices produce for people who are sensitive to sound quality. Both products will pose challenges to competitors in these spaces.

Amazon is Reportedly Building a Microwave, Amp, and More

CNBC’s Eugene Kim just posted a report that Amazon is reportedly working on new Alexa-enabled devices for the home and car, including a microwave. Per Kim:

Amazon is doubling down on its Alexa-powered devices, with plans to release at least 8 new voice-controlled hardware devices before the end of the year, CNBC has learned.

The devices include, among others, a microwave oven, an amplifier, a receiver, a subwoofer, and an in-car gadget, people familiar with the matter said.

Our take:

  • We believe home appliances will be a major area of emphasis for Amazon as it increasingly builds out its home automation ecosystem. Amazon already makes a variety of Alexa-enabled hardware devices for the home, and is expanding into home automation and security through its Ring and Blink acquisitions. Currently, GE sells an Alexa-enabled microwave on Amazon, but Amazon should be able to achieve more seamless integration by building it directly.
  • We also believe that car appliances will be a major area of emphasis for Amazon. Logistically, there is so much opportunity in the car for voice-powered services to thrive.
  • Improving its lineup of smart audio hardware offerings also makes sense for Amazon. Currently, that need is filled by firms like Sonos, which sells the Alexa-enabled Sonos One. Amazon has not exactly targeted audiophiles with its current lineup of Echo devices.
  • Amazon is also working with home building partners to create “Amazon Experience Centers” – model homes with Alexa integrations (currently in US with Lennar and India with Embassy Edge). There could be an opportunity to sell homes stacked with Amazon goods that would create higher barriers to switching for homebuyers and give Amazon large B2B sales channels.