Alexa Skill with Azure Functions – Messaging

In the previous Alexa post I talked about building a List skill to integrate with a third-party list provider. This gives you a mechanism to react to changes in Alexa’s lists and write them to your external provider, but what about implementing a two-way sync?

When you setup account linking for your skill the user goes through an OAuth flow to authorise your app and this returns a token and a refresh token to Amazon. The Alexa infrastructure manages this securely and handles the refresh process for you. Therefore only your skill function can continue to access your third-party service. Therefore when you have a callback mechanism from your third-party provider you need a way of passing the change information into your skill to be processed. Luckily there is a messaging service to do this.

As with the list functionality there is a library to handle the messaging requests – Alexa.NET.SkillMessaging. The code you use to send the message will have to have the client id and client secret of your skill – you can find these from the Alexa Developer Console on the web.

var client = new AccessTokenClient(AccessTokenClient.ApiDomainBaseAddress);
var accessToken = await client.Send(clientId, clientSecret);

This access token can then be used to send messages to your skill. Each message consists of a payload which is a Dictionary<string,string> and a timeout. You create an SkillMessageClient and send the message to a specified user id. The Amazon user id is given to you when your skill is first enabled and the account is linked. The id is specific to the skill and cannot be used to personally identify a user.

var payload = new Dictionary<string, string> { { "Key", "Some Value" } };
var messageClient = new Alexa.NET.SkillMessageClient(alexaEndpoint + "/v1/skillmessages/users/", accessToken.Token);
var messageToSend = new Alexa.NET.SkillMessaging.Message(payload, 3600);
var messageId = await messages.Send(messageToSend, userId);

An extra complication is that there are multiple API endpoints for the SkillMessageClient depending on the region. This means you’ll have to store the endpoint along with the alexa userid so you know which to use for a specific user. If successful a unique id for the message is returned. In your skill code you have to enable support to recognise the incoming message and handle the action. In the case of a list change event from a third-party provider this would be to load the specific changed item and then write the values to the Alexa list.

As with the list support we need to register the messaging library so that the skill request can be correctly deserialised into a MessageReceivedRequest.

RequestConverter.RequestConverters.Add(new MessageReceivedRequestTypeConverter());

Then when reading your incoming request you can check the request type and add code to process the message. The MessageReceivedRequest contains a Message property with the dictionary of values sent from your other function. The user id is already included with all incoming requests in the Context.System.User.UserId property.

Combining this with the list support already discussed you can see how to use the ListManagement API to write changes into the Alexa lists.

Playing Media with Xamarin Forms

Xamarin Forms has quite a rich set of controls which work natively across platforms however a big gap in the functionality is the ability to play audio or video content.

You can create custom controls for Xamarin and from the platform-specific renderers you have full access to the APIs provided by that platform to create complex controls of your own. However on a couple of projects I’ve needed to display a video and it felt to me like a fairly standard control we take for granted when doing “native” development.

The good news is that you don’t have to do this yourself because I’ve put my control up onto GitHub and NuGet and it’s ready to roll for Android, iOS and of course Windows UWP. If you’ve used the Windows MediaElement you already know how to use it too.

Simply add the InTheHand.Forms NuGet package and add a namespace to your XAML and add the MediaElement control. You can do the basics straight from XAML, but you can add a Name to the control and interact with it from the code too. There is full documentation for the control online. If you have any issues/feature suggestions please post them to GitHub.

The player controls can be toggled using the AreTransportControlsEnabled property. The displayed controls will match the native appearance of the target platform. For example the sample above uses a link to a Channel9 video on Xamarin Auth by @HoussemDellai and you can see below how it is rendered on an Android app and a UWP desktop app:-

screenshot_20170301-220605mediaplayer-uwp

I’ve also been working on support for the ms-appx scheme for Source Uris allowing you use a common way of referring to files in the application package. This should be in the next NuGet release (after 1.0.2017.301) along with a legacy Windows Phone 8.1 renderer.

I hope you find this useful in adding media to your Xamarin Forms projects!

Latest Version of InTheHand.Core

I’ve just updated the NuGet package for InTheHand.Core. This adds a VibrationDevice which works across all Xamarin Platforms (hardware permitting) and fixes a bug with PowerManager.RemainingChargePercent on iOS. The library now has 18 namespaces full of UWP APIs for all Xamarin platforms (and this doesn’t count the sister library InTheHand.UI). Check it out here:-

https://www.nuget.org/packages/InTheHand.Core/

Interactive Toasts and Windows Phone 8.1

Windows 10 brings a whole range of new APIs and functionality for building apps. However in the phone space it’s important to be mindful that there are a lot more Windows Phone 8.1 devices in use than Windows 10. Many (but not all) will get an update to Windows 10. A lot of clients are keen on supporting 8.1 to cover the largest group (as an 8.1 app will just run on 10). Of course your users on Windows 10 may miss out on cool features in this way unless you are prepared to maintain both 8.1 and 10 versions of your app. There are a number of areas of the Charming Apps libraries which support “lighting-up” of functionality. That means exposing Windows 10 APIs to Windows 8.x projects. In some cases this is a remapping of existing functionality from an old-style API to one which is source compatible with UWP. In other cases there is functionality which is just not there on 8.x but you can make it available when your app is run on a Windows 10 device.

In order to support this a bit of Reflection is required but we’ve hidden that messy stuff for you – you just call the same APIs and get graceful failure or the “light-up” experience when running on Windows 10. For many of these APIs the Charming Apps library provides iOS and Android implementations while exposing the same UWP API surface. There are a number of possible ways of detecting whether the device at runtime is running Windows 10. One of these is to use:-

if (InTheHand.Foundation.Metadata.ApiInformation.IsApiContractPresent("Windows.Foundation.UniversalApiContract", 1))

Another is to check:-

InTheHand.Environment.OSVersion.Version

(caveat – On 8.x we can return only Major.Minor.0.0 versions)

One of the new features in Windows 10 is the interactive toast notification. There is no API change to set these locally – it just requires a different XML template. So once you’ve checked your OS version you can switch between a traditional dumb toast or an interactive one such as:-

string xml = "<toast launch=\"yourstring\"><visual><binding template=\"ToastGeneric\" ><text>Big text</text><text>smaller text</text></binding></visual>"
                    + "<actions><input id=\"time\" type=\"selection\" defaultInput=\"2\"><selection id=\"1\" content=\"Breakfast\"/><selection id=\"2\" content=\"Lunch\"/><selection id=\"3\" content=\"Dinner\"/></input>"
         	    + "<action activationType=\"foreground\" content=\"Primary Action\" arguments=\"primary\"/><action activationType=\"foreground\" content=\"Secondary Action\" arguments=\"secondary\"/></actions>"
                    + "</toast>";

Notice that I’ve specified foreground activation. In a UWP project you’d also have the option of background activation where a background task is launched to carry out an action. In the case of foreground activation the system calls your app’s OnActivated method. However this is where it gets a bit more fun. The ActivationKind is not specified in Windows Phone 8.1, nor is the ToastNotificationActivatedEventArgs which is passed through. However the value of ActivationKind.ToastNotification is documented as 1010. The event args contains two items of interest, the Argument – the action which was selected and UserInput which is a string keyed dictionary of input id and selection id or input id and text entered in the case of a text field. I’ve added the following code in my OnActivated method to build up a string containing these values which can be passed to the applications main page using rootFrame.Navigate:-

	    string navigationParameter = null;

            switch (args.Kind)
            {
                case (ActivationKind)1010: //ToastNotification:
                    StringBuilder sb = new StringBuilder();
                    sb.Append(args.GetType().GetRuntimeProperty("Argument").GetValue(args).ToString());
                    IDictionary<string,object> selections = (IDictionary<string,object>)args.GetType().GetRuntimeProperty("UserInput").GetValue(args);
                    if (selections.Count > 0)
                    {
                        sb.Append("?");
                        foreach (KeyValuePair<string, object> pair in selections)
                        {
                            sb.Append(pair.Key + "=" + pair.Value.ToString() + "&");
                        }

                        //remove trailing &
                        sb.Length -= 1;
                    }
                    navigationParameter = sb.ToString();

                    break;
            }
This is just another way of adding value to the user while still maintaining a Windows Phone 8.1 codebase. If you later update your app to Windows 10 it’s very easy to do without much alteration to your code. You don’t even have to go as far as adding interaction to your toasts but you could use this technique of detecting the OS version and simply use the new toast templates.
There are other areas of the Charming Apps libraries which “light-up” when running on Windows 10. For example there is a UWP style Clipboard API which just works on Windows 10 whereas the separate Clipboarder app is required for Windows Phone 8.1 (not Silverlight).

 

Charming Apps – Latest Version

It’s been a few months since I did a NuGet release but lots has been going on in the code in the mean time. The focus this year has been less about harmonising Windows APIs across versions (although that’s still there of course), but widening the functionality to the Xamarin platforms. The result is that the vast majority of the APIs are now available across Android, iOS, Windows (UWP and 8.1) and Windows Phone (8.0, 8.1 Silverlight and 8.1 WinRT). In many cases I’ve added “light-up” support allowing Windows 10 functionality to be called from Windows (and Phone) 8.1 projects. This means you can write for maximum compatibility but take advantage of new features where available. The two packages are:-

https://www.nuget.org/packages/InTheHand.Core/

and

https://www.nuget.org/packages/InTheHand.UI/

As the name suggests the UI library separates out stuff which uses UI and so you can just use the former in a background task without worrying about dragging in UI dependencies.

The APIs are all modelled on UWP, the only difference being the root namespace is InTheHand rather than Windows and async methods return Task<T> rather than IAsyncOperation<T>.

The Core library features many namespaces around ApplicationModel – Interrogate app package metadata, make calls, send SMS and Emails, interact with Clipboard and Sharing. Beyond this are classes which take away the need to know which APIs to call on which platform – interrogate display properties, check network connectivity, read and write files and settings. This version extends the support for settings by supporting roaming settings on iOS just like Windows does – except these settings are roamed across the iCloud infrastructure rather than OneDrive. It’s important to add here that these settings only roam for devices on that specific platform type – Windows and iOS roaming data exist in separate worlds so data roamed from your iPhone Xamarin app won’t roam to your Windows laptop.

AnalyticsInfo is available and can report the OS version on iOS/Android and Windows 10. This is supported in the 8.1 libraries if running on the newer OS version. DisplayRequest allows your app to ask the device to keep the screen on – for example for a reader or video player. PowerManager gives you battery information.

Over in the InTheHand.UI library there are Badge and Toast notification APIs (Badge is not supported on Android). BadgeNotificationCreator and ToastNotificationCreator helper classes make it easy to create these notifications without having to deal with XML templates on Windows. A MessageDialog and a StatusBar which allows you to toggle a standard busy animation. On Windows there is also a unified SettingsPane API which uses the best available appearance depending on platform.

Going forwards I’m looking at removing the Windows Phone 8.0 (and possibly 8.1 Silverlight) target to simplify things and would be interested in any feedback on that. As always all the code is on GitHub:-

https://github.com/inthehand/Charming

Tap, Tap, Tap, Tap, Bang!

There are limitations to the write-once run everywhere approach of Xamarin Forms. One of these is the subtle difference in Button behaviour across platforms. One of our testers pointed out that you could rapidly tap a Button on Android and the handler would be called multiple times. Since the Tap was initiating a page navigation this would lead to multiple copies of the page being added to the back-stack.

The workaround necessary is simply to add a timeout to ensure that the Tap event is not handled more frequently than a specified timeout. We chose 500ms but your use-case may vary. I created a renderer derived from the built-in ButtonRender and searched a way to intercept tap events. It turns out that a lot of the methods which can be overridden are never called but I found a solution through the excitingly named OnFilterTouchEventForSecurity. The resulting renderer code is included in the following Gist and can be added to your Android project to enforce this behaviour.