Alexa Skill with Azure Functions – Messaging

In the previous Alexa post I talked about building a List skill to integrate with a third-party list provider. This gives you a mechanism to react to changes in Alexa’s lists and write them to your external provider, but what about implementing a two-way sync?

When you setup account linking for your skill the user goes through an OAuth flow to authorise your app and this returns a token and a refresh token to Amazon. The Alexa infrastructure manages this securely and handles the refresh process for you. Therefore only your skill function can continue to access your third-party service. Therefore when you have a callback mechanism from your third-party provider you need a way of passing the change information into your skill to be processed. Luckily there is a messaging service to do this.

As with the list functionality there is a library to handle the messaging requests – Alexa.NET.SkillMessaging. The code you use to send the message will have to have the client id and client secret of your skill – you can find these from the Alexa Developer Console on the web.

var client = new AccessTokenClient(AccessTokenClient.ApiDomainBaseAddress);
var accessToken = await client.Send(clientId, clientSecret);

This access token can then be used to send messages to your skill. Each message consists of a payload which is a Dictionary<string,string> and a timeout. You create an SkillMessageClient and send the message to a specified user id. The Amazon user id is given to you when your skill is first enabled and the account is linked. The id is specific to the skill and cannot be used to personally identify a user.

var payload = new Dictionary<string, string> { { "Key", "Some Value" } };
var messageClient = new Alexa.NET.SkillMessageClient(alexaEndpoint + "/v1/skillmessages/users/", accessToken.Token);
var messageToSend = new Alexa.NET.SkillMessaging.Message(payload, 3600);
var messageId = await messages.Send(messageToSend, userId);

An extra complication is that there are multiple API endpoints for the SkillMessageClient depending on the region. This means you’ll have to store the endpoint along with the alexa userid so you know which to use for a specific user. If successful a unique id for the message is returned. In your skill code you have to enable support to recognise the incoming message and handle the action. In the case of a list change event from a third-party provider this would be to load the specific changed item and then write the values to the Alexa list.

As with the list support we need to register the messaging library so that the skill request can be correctly deserialised into a MessageReceivedRequest.

RequestConverter.RequestConverters.Add(new MessageReceivedRequestTypeConverter());

Then when reading your incoming request you can check the request type and add code to process the message. The MessageReceivedRequest contains a Message property with the dictionary of values sent from your other function. The user id is already included with all incoming requests in the Context.System.User.UserId property.

Combining this with the list support already discussed you can see how to use the ListManagement API to write changes into the Alexa lists.

Alexa List Skills with Azure Functions

When  I was building my Microsoft To-do Alexa skill I found Matteo’s series of blog posts on Alexa + Azure very useful. However I needed to go beyond the functionality described there and knew I’d need to delve deeper into the Alexa Skills Kit documentation. The first item I thought I’d blog about is the Alexa List Management API. I’m not going to open source the skill but I thought I’d share some pointers which could be useful to other developers.

What is a List Skill?

Alexa has built in lists for To-do items and Shopping List. These are quite basic (there are no due dates or reminders for example) but a third-party can extend them to synchronise them with another back-end. When you create a list skill you aren’t providing Alexa with an API into your datastore, but rather are taking responsibility to maintain synchronisation between two copies of the list. Your list items could be updated either from Alexa via the web or app or, more commonly, via a user’s voice command. To support this your skill subscribes to a number of list events which fire whenever an item is added, modified or deleted. Likewise on your backend you’ll need to handle changes and communicate them back to your Alexa skill. I’ll discuss this messaging infrastructure in a future post.

In its simplest form a List skill doesn’t have to have any speech interface of its own. It just has to handle the standard list events and require read (and likely write) permission on the Alexa household lists.

Building a List Skill

If your skill isn’t a regular custom skill you can’t edit the manifest through the Alexa Skill Kit dashboard and so you would use the Alexa Skill Kit command line tools. However rather than suffer the misery of a command prompt you can use trusty Visual Studio Code and the official Alexa Skills Kit Toolkit extension. This allows you to edit your skill metadata in Visual Studio Code’s editor and use the command palette to perform common operations like deploying the skill. The metadata for a skill is expressed in json and the editor has intellisense for the schema. A manifest for a list skill must contain an events object containing a list of standard event types:-

"events":{
      "endpoint":{
        "uri":"https://YourAzureFunctionAppEndpoint/api/FunctionName",
        "sslCertificateType": "Wildcard"
      },
      "subscriptions":[
       {
         "eventName": "SKILL_ENABLED"
       },
       {
         "eventName": "SKILL_DISABLED"
       },
       {
         "eventName": "SKILL_PERMISSION_ACCEPTED"
       },
       {
        "eventName": "SKILL_PERMISSION_CHANGED"
       },
       {
        "eventName": "SKILL_ACCOUNT_LINKED"
       },
       {
        "eventName": "ITEMS_CREATED"
       },
       {
        "eventName": "ITEMS_UPDATED"
       },
       {
        "eventName": "ITEMS_DELETED"
       },
       {
        "eventName": "LIST_CREATED"
       },
       {
        "eventName": "LIST_UPDATED"
       },
       {
        "eventName": "LIST_DELETED"
       }
      ]
    },
Also in order to be able to query the list items and write changes you must also request permissions. You’ll need to be careful as the user can revoke these:-
    "permissions": [
      {
        "name": "alexa::household:lists:read"
      },
      {
        "name": "alexa::household:lists:write"
      }
    ]
The endpoint you defined for events will receive all the event types so you’ll need to write code to read the event type and react accordingly. Now this is where we step into the unknown – the list events require a different request type than is covered with the Alexa.NET library. The good news is there are already a range of companion NuGet packages for specific Alexa APIs and Alexa.Net.ListManagement has what we need here.
For each of these libraries we have to add a line of code at the beginning of our function to tell the main Alexa.NET library how to deserialise the request. For list management this is:-
RequestConverter.RequestConverters.Add(new ListSkillEventRequestTypeConverter());
            
Then from our deserialised SkillRequest we can get the type to determine the type of request which is a specific type depending on the list event e.g. ListSkillItemCreatedRequest. The body of this request will contain a list id which may represent the To-Do list, Shopping List or a custom list. It will also contain one or more list item ids for the newly created item(s).

Modifying List Items

The other part of the ListManagement library is the ListManagementClient. This provides access to read and write list items and wraps the REST API. The constructor takes an access token which is passed to your skill in the skill request’s Context.System.ApiAccessToken property. With this (assuming you are granted the required permissions) you can query all list metadata and create, modify and delete list items. However these are operations you’ll mainly do triggered by changes in your linked back-end so you can keep Alexa’s copy of the list in sync with your own data.
In the next post I’ll look at how to implement messaging so that your own system can send a message to your skill to update items…

Talking About Tasks

Back in 2010 Microsoft released Windows Phone 7. It was a huge change from the Pocket PC/Windows Mobile OS which had preceded it and while it brought a modern UI and app-store infrastructure it missed a number of pieces of core functionality from the older phones. One of these was support for Tasks. I set about writing an app which became “Tasks In The Hand” and became very popular in those early days. Even when Microsoft later added basic Tasks functionality in Windows Phone 7.5 the app still had a healthy following because it supported views and features absent from the in-box app.

Skipping forward to today Microsoft’s Task story is rather different. After purchasing Wunderlist they began writing a new app called Microsoft To-Do which is available across multiple platforms – iOS and Android for mobile and Windows for desktop. Crucially though, under the hood, it’s still based on Office 365 (or Outlook.com for personal Microsoft IDs) for storage and so works just as well with the traditional Tasks view in Outlook on the desktop.

Back in 2010 we did not have voice assistants but now we have Alexa, Google Home and Cortana. If you get used to using Microsoft To-Do everywhere, as I have, you miss having integration with a voice assistant and so that is where I decided Tasks In The Hand needed to go next. Today my Alexa Skill was released into the store for anyone to connect with their Echo or similar device.

tasks-alexa-snippet

Tasks In The Hand in the Alexa Store

The skill links with your Microsoft ID, which is either associated with an Office 365 account or an Outlook.com personal account. Once setup you can add tasks to your Alexa To-Do list or items to your Amazon Shopping List and they’ll synchronise with your Microsoft account. You can modify, complete or delete these items via Microsoft To-Do and those changes are synched back to Alexa. After you’ve linked accounts any items you add to your default Tasks folder or Amazon Shopping List folder will also be synchronised with Alexa.

The Skill was built using Azure Functions and I found Matteo Pagani‘s series of Blog posts very useful with getting started working with Alexa Skill Kit. It uses Tim Heuer‘s excellent Alexa.NET package to handle the interactions with Amazon.

The skill is completely free, I hope you find it as useful as I have and please get in touch if you have any feedback.

Family Calendar with Amazon Echo

Update 15th February 2017

Alexa can now be paired with an Outlook.com account which means you no longer have to use this workaround.

 

We’ve used the “Family Room” feature from Windows Phone and although it’s been discontinued the shared Outlook.com calendar is still vital. We’ve got an Amazon Echo but when setting it up found that only Google calendar integration is available. It turns out that it is possible to hook this up albeit without the ability to add new events via Alexa.

First you’ll need a Google account. If you don’t already have one you can set one up just to use with Alexa. We’ll come back to Google later.

Now open your browser of choice and log into your Outlook.com account. Go to the Calendar app. On the left you should see a list of “Your calendars” and your shared calendar will be listed e.g. “Family Room”. Click Settings (The cog near the top right of the screen next to your profile picture) and Options. You’ll see a link in the tree of settings called “Publish calendar”. Click this then select the name of your shared calendar e.g. Family Room. Under “Show availability, titles and locations” click create. There will be two links created – you only need the ICS one.

Log into your Google Calendar account. On the left side there is a section called “Other calendars” with a small downwards arrow to the right. Click this and select “Add by URL”. Paste the URL you created earlier and click Add Calendar.

Google Calendar will now add all the events from your shared calendar and will periodically update when the Outlook.com ICS feed updates.

Finally log into the Alexa web portal (https://alexa.amazon.com or https://alexa.amazon.co.uk etc). Go to Settings and in the “Account” section select Calendar. Link your Google calendar (if you haven’t already) and you’ll see a list of available calendars. Because I don’t use my Google calendar other than for the Family Room import I selected only the Family Room option.

Once that’s done you can query Alexa about things in your family calendar!