An Automated Assistant Enabled Vehicle

Posted in: Mobile

May 10, 2022

Join thousands of marketers to get the best search news in under 5 minutes. Get resources, tips and more with The Splash newsletter:

Table of Contents

What will Google’s Automated Assistants be capable of doing tomorrow? Chances are they will be involved in running smart homes and internet of things devices and helping us drive vehicles. A patent was just granted to Google this week about using an automated assistant to control a vehicle. This won’t be implemented soon, but it might be something we are driving in in the not too distant future.

An Automated Assistant Controlling a Vehicle in the Future

Humans may engage in human-to-computer dialogs with interactive software applications referred to herein as an “automated assistant.”

I have written a few different posts about Google’s Automated Assistants which interact with humans in a variety of ways.

Here are some previous posts I have written about automated assistants:

I have a speaker device which is an automated assistant. I use it to perform some searches and listen to music, and send some search results to my phone. It doesn’t do as many things as helping me drive a vehicle, but this patent may be an illustration of what Google’s automated assistant will be able to do in the future.

Related Content:

Under this patent, humans may provide commands and requests to an automated assistant using spoken natural language input (such as utterances), which may in some cases get converted into text and then processed, and by providing textual (e.g., typed) natural language input.

An Automated assistant can get integrated into a variety of electronic devices, including vehicles. Unlike other computers such as mobile phones, vehicles are generally in motion over a large area, and thus are more susceptible to bandwidth restrictions during communications with an outside server.

This can in part result from the vehicle moving through areas that do not provide adequate network coverage. This can affect automated assistant operations, which may involve many round trips between a vehicle computer and a remote server.

Automated assistants may have access to publicly-available data as well as user-specific data, which can get associated with a personal user account served by the automated assistant. An automated assistant serving many users may have many accounts with different data available for each account.

Commanding Automated Assistant

Thus, if one user makes a request to an automated assistant, and responding to the request involves accessing a second user account, the automated assistant may not be able to complete the request without prompting the second user to log in to their account and repeat the request.

As a result, computational and communication resources, such as network bandwidth and channel usage time, can get consumed by increasing many interactions between the vehicle computer and the server.

Other Users Overriding Restrictions

Implementations described herein relate to limiting vehicle automated assistant responsiveness according to restrictions that get used to determine whether certain input commands and certain users get restricted in certain vehicle contexts. Furthermore, implementations described herein allow for other users to override certain restrictions by providing authorization via an input to the vehicle computer or another computer.

Allowing other users to override such restrictions can preserve computational resources, as less processing resources and network bandwidth would get consumed when a restricted user does not have to rephrase and resubmit certain inputs in a way that would make the inputs permissible.

As an example, a passenger that provides a spoken input to a vehicle automated assistant such as “Assistant, send a message to Karen,” may get denied because the passenger is not the owner of the vehicle or otherwise permitted to access contacts accessible to the vehicle automated assistant.

As a result, the vehicle automated assistant can provide a response such as “I’m sorry, you are not authorized for such commands,” and the passenger would have to rephrase and resubmit the spoken input as, for example, “Ok, Assistant, send a message to 971-555-3141.”

Such a dialog session between the passenger and the vehicle automated assistant can waste computational resources as the later spoken input would have to get converted to audio data, transmitted over a network, and processed.

In a situation where available bandwidth gets limited or variable, such as for example in a moving vehicle, this might be particularly undesirable since the channel over which data gets communicated from the assistant device, over the network, may need to get used for a longer than desirable.

The length of time such a channel gets used might impact not only the operations of the automated assistant but also other software applications which rely on the network to send and receive information.

Such software applications may, for example, be present in the same device as the automated assistant (e.g. other in-vehicle software applications). However, implementations provided herein can eliminate such wasting of computational and communication resources by at least allowing other users to authorize the execution of certain input commands from a user, without requesting the user to re-submit the commands.

Restriction Of Access To Commands

A vehicle computer and an automated assistant can operate according to different restrictions for restricting access to commands and data that would otherwise be accessible via the vehicle computer and the automated assistant. A restriction can characterize particular commands, data, types of data, and any other inputs and outputs that can get associated with an automated assistant, thereby defining certain information that is available to other users via the automated assistant and the vehicle computer.

When a user provides a spoken utterance corresponding to a particular command characterized by a restriction, the automated assistant can respond according to any restriction that gets associated with the user and the particular command. As an example, when a user provides a spoken utterance that corresponds to data that originated at a computer owned by another user, the spoken utterance can satisfy a criterion for restricting access to such data.

However, in response to receiving the spoken utterance, the automated assistant can determine that the criterion gets satisfied and await authorization from the other user. The authorization can get provided by the other user to the vehicle computer and a separate computer via another spoken utterance and any other input capable of getting received at a computer.

A vehicle that includes the vehicle computer can include an interface, such as a button (e.g., on the steering wheel of the vehicle), that the other user can interact with (e.g., depress the button) in order to indicate authorization to the automated assistant.

In response to the automated assistant receiving authorization from the other user, the automated assistant can proceed with executing the command provided by the user, without necessarily requesting further input from the user.

Automated Assisstant Limiting Acess To Passengers

Another user can limit a passenger from accessing certain data while the other user and the passenger are riding in the vehicle. The other user can limit access to certain data while the vehicle is navigating along a particular route and to a particular destination. Therefore, when the vehicle completes the route and arrives at the particular destination, a restriction on access to the particular data and for the passenger can get released, thereby allowing the passenger to subsequently access such data.

For instance, when the other user is driving the vehicle and the passenger is riding in the vehicle, the passenger can provide a spoken utterance to an automated assistant interface of the vehicle. The spoken utterance can be, “Assistant, call Aunt Lucy.”

Automated Assistant Awaiting Authorization From The User

In response, and because the spoken utterance includes a request that will result in accessing the contact information of the user, the automated assistant can await authorization from the user before fulfilling the request. However, in order to eliminate having to repeatedly authorize or not authorize requests originating from the passenger, the user can provide another spoken utterance such as, “Assistant, do not respond to the passenger for the remainder of this trip.”

In response, the automated assistant can cause restriction data to get generated for limiting access to services (e.g., making phone calls) that would otherwise be available via the automated assistant.

In this way, the user would not have to repeatedly authorize or not authorize the automated assistant to respond to requests from the passenger, thereby eliminating the waste of computational resources and network resources. Furthermore, because the access restrictions can be set to “reset” at the end of a trip, or upon reaching a destination, the user would not have to explicitly request a reset of restrictions, thereby further eliminating the waste of computational resources and network resources.

The user can limit access to certain data to a passenger indefinitely and for an operational lifetime of the vehicle.

For instance, subsequent to the passenger providing the spoken utterance, “Assistant, call Aunt Lucy,” and while the automated assistant is awaiting authorization from the user, the user can provide a separate spoken utterance such as, “Assistant, never respond to the user.”

Automated Assistant Causing Restriction Data To Get Generated

In response, the automated assistant can cause restriction data to get generated (or for an operational lifetime of the vehicle, the vehicle computer, and the automated assistant) limiting access to services that would otherwise be available to a particular user via the automated assistant.

Depending on the occupancy of the vehicle, the automated assistant and the vehicle computer can operate according to an operating model that limits access to the automated assistant and the vehicle computer for certain passengers. As an example, when a user is the only person occupying a vehicle, a vehicle computer and an automated assistant that is accessible via the vehicle computer, can operate according to a first operating mode.

Occupancy Of Vehicle Determined Based On Output Of Sensors Or Operating Modes

The occupancy can get determined based on an output of sensors of the vehicle, the vehicle computer, and any other device that can provide an output from which occupancy can get estimated. The first operating mode can get selected based on the occupancy and can provide the user access to the first set of services, data, and commands, associated with the automated assistant.

When the occupancy gets determined to include more than the user, such as when the user is driving with passengers (e.g., a parent driving with many children as passengers), a second operating mode can get selected. In accordance with the second operating mode, the user can still access the first set of services, data, and commands–however, the passengers would only be able to access the second set of services, data, and commands.

The second set can be different than the first set, and the second set can be a reduced subset relative to the first set. For example, pushing the “talk” button on the head unit, when only a driver (e.g., an unrestricted user) is in the vehicle, can respond with private data without any further authorization.

However, if the “talk” button on the head unit gets pushed when a passenger (e.g., a restricted user) is in the vehicle with the driver, the automated assistant request further authorization to respond to someone (e.g., the passenger) pressing the “talk” button on the head unit.

While the second operating mode (e.g., a shared operating mode) is active, a passenger can attempt to access a service, data, and a command that is exclusively provided in the first set, and not the second set. In order to permit such access, the user (e.g., the driver) can provide inputs to the automated assistant and the vehicle computer, in order to authorize such access.

The user can provide, for example, an input to an interface such as a button and touch display panel, which can get located approximately within reach of a driver of the vehicle (e.g., a button on a steering wheel, a touch display panel integral to a dashboard and console). The authorizing input can get provided in response to the automated assistant soliciting authorization from the user (e.g., “Sorry, I need the authorization to do that . . . [authorizing input received]”).

Alternatively, the automated assistant can bypass soliciting the user for authorization, and, rather, passively wait to respond to a request from a passenger until the user provides an authorizing input.

However, if the user elects to have their automated assistant and their vehicle computer operate according to a third operating mode.

In the third operating mode, in which no option to provide such authorization is available, the automated assistant and the vehicle computer can operate such that the availability of certain operations, data, and services get limited for some passengers (at least relative to a user that is a primary and “master” user with respect to the automated assistant and the vehicle computer).

Automated Assistant Routines

An automated assistant can perform automated assistant routines. An automated assistant routine can correspond to a set and sequence of actions performed and initialized by the automated assistant in response to a user providing a particular input. The user can provide a spoken utterance such as, “Assistant, let’s go to work,” when the user enters their vehicle, in order to cause the automated assistant to perform a “Going to Work” routine.

The “Going to Work” routine can involve the automated assistant causing the vehicle computer to render graphical data corresponding to a daily schedule of the user and render audio data corresponding to a podcast selected by the user. It can generate a message to a spouse of the user indicating that the user is headed to work (e.g., “Hi Billy, I’m headed to work.”). In some instances, however, a passenger of the vehicle can provide the spoken utterance, “Assistant, let’s go to work.”

Depending on the mode that the vehicle computer and the automated assistant is operating in, the automated assistant can request that the driver, or another authorized user, provide permission to perform actions of a requested routing.

The Automated Assistant “Going to Work” Routine

For example, in response to the passenger invoking the “Going to Work” routine, the automated assistant can initialize performance rendering audio data corresponding to a particular podcast, and also prompt the driver for authorization to initialize other actions of the routine.

Specifically, the vehicle computer and server device can identify actions of the routine that involve accessing restricted data. In this instance, the vehicle computer and the server device can determine that the schedule of the user and the contacts of the user (for sending the message) get restricted data.

As a result, during the performance of the routine, the driver can get prompted times to give permission to execute any actions involving accessing restricted data.

If the driver gives authorization (e.g., via an assistant invocation task), by speaking an invocation phrase (e.g., “Ok, Assistant.”) or interacting with an interface (e.g., pressing a button), the routine can get completed. For instance, the message can get sent to the spouse and the schedule of the driver can get rendered audibly.

However, if authorization is not provided by the driver (e.g., the driver does not perform an assistant invocation task), the automated assistant can bypass the performance of such actions. When the driver does not provide authorization to complete the actions, alternative actions can get provided as options to the passenger.

For instance, instead of audibly rendering the schedule of the driver, the automated assistant can render public information about events that are occurring in the nearby geographic region.

Sending A Message

Instead of sending a message to the spouse of the driver, the automated assistant can prompt the passenger regarding whether they would like to have a message transmitted via their own account (e.g., “Would you like to login, in order to send a message?”). Restrictions on the data of the driver would get enforced while simultaneously providing assistance to a passenger who may be in the vehicle due to, for example, participation in a ride-sharing activity.

The above description gets provided as an overview of some implementations of the present disclosure.

Other implementations may include a system of computers and robots that include processors operable to execute stored instructions to perform a method such as of the methods described above and elsewhere herein.

This Automated Assistant-Enabled Vehicle is Described in this patent:

Modalities for authorizing access when operating an automated assistant enabled vehicle
Inventors: Vikram Aggarwal and Moises Morgenstern Gali
Assignee: GOOGLE LLC
US Patent: 11,318,955
Granted: May 3, 2022
Filed: February 28, 2019

Abstract:

Implementations relate to enabling of authorization of certain automated assistant functions via one or more modalities available within a vehicle.

Implementations can eliminate wasting of computational and communication resources by at least allowing other users to authorize execution of certain input commands from a user, without requesting the user to re-submit the commands.

The vehicle can include a computing device that provides access to restricted data, which can be accessed in order for an action to be performed by the automated assistant.

However, when a restricted user requests that the automated assistant perform an action involving accessing the restricted data, the automated assistant can be authorized or unauthorized to proceed with fulfilling the request via a modality controlled by an unrestricted user.

The unrestricted user can also cause contextual restrictions to be established for limiting functionality of the automated assistant during a trip, for certain types of requests, and/or for certain passengers.

Automated Assistant Enhanced Vehicle Conclusion

I have only written about the summary of this patent in this post. If you want more details about how this automated assistant patent will work, click through to the patent itself for more details about how it could work. This summary provides some insight into how control over a vehicle would be established using an automated assistant.

At this time Automated Assistants tend to be smaller devices such as smart speakers. Chances are that they will grow to do things such as power vehicles, as shown in this patent. The interface is different than the one that Google devices tend to use. They are in a more conversational format than a desktop or laptop computer. I was reminded of Android Auto while reading this post. I can see Google wanting to have cars controlled by something like android auto or the Automated Assistant.

About Bill Slawski

With more than 26 years of SEO experience and a Juris Doctor Degree, Bill Slawski is the foremost expert on Google’s patents as related to SEO. Patent Exploration is one of the quickest and most detailed ways to find new information about SEO. Bill is the Editor of SEO by the Sea, a prominent search engine optimization blog, where he is the author of over 1,300 posts. Bill’s experience includes Fortune 500 brands and some of the largest websites in the world. Bill is a contributing author for Moz, Search Engine Land, and Search Engine Journal. In 2014-2021, he spoke at industry-leading international conferences about topics including search engine algorithms, universal and blended search, personalization in search, search and social, and duplicate content problems, structured data, and schema