Efficient Apple Watch CMSensorRecorder transport to AWS

Well, I have come full circle. A long time ago, this experiment ran through Rube Goldberg system #1:

• Dequeue from CMSensorRecorder
• Pivot the data
• Send it via WCSession to the iPhone
• iPhone picked up the data and queued it locally for Kinesis
• Then the Kinesis client transported data to Kinesis
• Which had a Lambda configured to dequeue the data
• And write it to DynamoDB

Then, I flipped the data around in an attempt to have the Watch write directly to DynamoDB:

• Dequeue from CMSensorRecorder
• Pivot the data to a BatchPutItem request for DynamoDB
• Put the data to DynamoDB
• (along the way run the access key Rube Goldberg machine mentioned earlier)

The problems with both of these approaches are the cost to execute on the Watch and the Watch's lack of background processing. This meant it was virtually impossible to get data dequeued before the Watch app went to sleep.

I did a little benchmarking over the weekend and found that brute force dequeue from CMSensorRecorder is fairly quick. And the WCSession sendFile support can run in the background, more or less. So, I will now attempt an alternate approach:

• Dequeue from CMSensorRecorder
• Minimal pivot of data including perhaps raw binary to a local file
• WCSession:sendFile to send the file to the iPhone
• Then iPhone gets the file and sends it itself to AWS (perhaps a little pivot, perhaps S3 instead of DynamoDB, etc.)
• (along the way a much simpler access key machine will be needed)

The theory is that this'll get the data out of the Watch quickly during its limited active window.

We'll see...

The limits of AWS Cognito

Well, after a slight hiatus, I spent a little time understanding how to use AWS Cognito in an application. I've now got a more or less running Cognito-as-STS-token-generator for the Apple Watch. Features:

• Wired to any or all of Amazon, Google, Twitter or Facebook identity providers
• Cognito processing occurs on the iPhone (hands STS tokens to Watch as Watch can't yet run the AWS SDK)
• Leverage Cognito's ability to 'merge' identities producing a single CognitoID from multiple identity providers
• Automatic refresh of identity access tokens

Here's the iPhone display showing all the identity providers wired:

Ok, the good stuff. Here what the access key flow now looks like:

There are a lot of actors in this play. They key actor for this article is the IdP. Here, as a slight generalization across all the IdPs, we have a token exchange system. The iPhone maintains the long lived IdP session key from the user's last login.  Then, the iPhone performs has the IdP exchange the session key for a short-lived access key to present to Cognito. For IdP like Amazon and Google, the access key is only good for an hour and must be refreshed...

Let me say that again; today, we need to manually refresh this token for Cognito before asking Cognito for an updated STS token! Cognito can't do this! FYA read Amazon's description here: "Refreshing Credentials from Identity Service" 

Especially in our case, where our credentials provider (Cognito) is merely referenced by the other AWS resources, we need to intercept the Cognito call to make sure that on the other side of Cognito, the 'logins' are up to date.

So, I replumbed the code to do just this (the 'opt' section in the above diagram). Now, a user can log in once on the iPhone application and then each time the Watch needs a token, the whole flow tests whether or not an accessKey needs to be regenerated.

For reference, here's the known lifetimes of the various tokens and keys:

• The Watch knows its Cognito generated STS Token is good for an hour
• Amazon accessTokens are good for an hour (implied expire time)
• Google accessToken is good until an expire time (google actually returns a time!)
• Twitter doesn't have expire so its accessKey is unlimited
• Facebook's token is good for a long time (actually the timeout is 60 days)
• TODO: do any of the IdPs enforce idle timeouts? (e.g. a sessionKey has to be exchanged within a certain time or it is invalidated...)

So, with all these constants, and a little lead time, the Watch->iPhone->DynamoDB flow looks pretty robust. The current implementation is still limited to having the Watch ask the iPhone for the STS since I haven't figured out how to get the various SDKs working in the Watch. I don't want to rewrite all the IdP fetch codes, along with manual calls to Cognito.

Plus, I'm likely to move the AWS writes back to the iPhone as the Watch is pretty slow.

The code for this release is here. The operating code is also in TestFlight (let me know if you want to try)

Known bugs:

• Google Signin may not work when the app is launched from the Watch (app crashes)
• Facebook login/logout doesn't update the iPhone status section
• The getSTS in Watch is meant to be pure async -- I've turned this off until its logic is a bit more covering of various edge cases.
• The webapp should also support all 4 IdP (only Amazon at the moment)

Wow: Multiple Identity Providers and AWS Cognito

I've finally found time to experiment with multiple identity providers for Cognito. Mostly to understand how a CognitoId is formed, merged, invalidated. It turns out this is a significant finding, especially when this Id is used, say, as a primary key for data storage!

Recall, the original sensor and sensor2 projects were plumbed with Login With Amazon as the identity provider to Cognito. This new experiment adds GooglePlus as a second provider. Here you can see the test platform on the iPhone:

Keep in mind that for this sensor2 application, the returned CognitoId is used as the customer's key into the storage databases. Both for access control and as the DynamoDB hash key.

The flow on the iPhone goes roughly as follows:

• A user can login via one or both of the providers
• A user can logout
• A user can also login using same credentials on a different devices (e.g. another iPhone with the application loaded)

Now here's the interesting part. Depending on the login ordering, the CognitoId returned to the application (on the watch in this case) can change! Here's how it goes with my test application (which includes "Logins" merge)

• Starting from scratch on a device
• Login via Amazon where user's Amazon identity isn't known to this Cognito pool:
• User will get a new CognitoId allocated
• If user logs out and logs back in via Amazon, the same Id will be returned
• If the user now logs into a second device via Amazon, the same Id will be returned
• (so far this makes complete sense)
• Now, if the user logs out and logs in via Google, a new Id will be returned
• Again, if the user logs out and in again and same on second device, the new Id will continue to be returned
• (this all makes sense)
• At this point, the system thinks these are two users and those two CognitoIds will be used as different primary keys into the sensor database...
• Now, if the user logs in via Amazon and also logs in via Google, a CognitoId merge will occur
• One, or the other of those existing Ids from above will be returned
• And, the other Id will be marked via Cognito as disabled
• This is a merge of the identities
• And this new merge will be returned on other devices from now on, regardless of whether they log in solely via Amazon or Google
• (TODO: what happens if user is logged into Amazon, has a merged CognitoId and then they log in using a second Google credential?)

This is all interesting and sort of makes sense -- if a Cognito context has a map of logins that have been associated, then Cognito will do the right thing. This means that some key factors have to be considered when building an app like this:

• As with my application, if the sensor database is keyed by the CognitoId, then there will be issues of accessing the data indexed by the disabled CognitoId after a merge
• TODO: will this happen with multiple devices going through an anonymous -> identified flow?
• It may be that additional resolution is needed to help with the merge -- e.g. if there is a merge, then ask the user to force a join -- and then externally keep track of the merged Ids as a set of Ids -> primary keys for this user...

Anyway, I'm adding in a couple more providers to make this more of a ridiculous effort. After which I'll think about resolution strategies.

TestFlight version of sensor2 is available in Apple Store

The current version of sensor2 is available as a test flight application if you'd like to try it out. This will give you a quick try of the basic sensor capture and display in the minimal website.

Recall it uses Amazon Login as the sole identity provider right now.

Drop me a note, or comment here if you'd like to try the TestFlight version.

sensor2 code cleanup -- you can try it too

After a bit of field testing, I've re-organized the sensor2 code to be more robust. Release tag for this change is here. Major changes include:

• The Watch still sends CMSensorRecorder data directly to DynamoDB
• However, the Watch now asks the iPhone for refreshed AWS credentials (since the AWS SDK isn't yet working on Watch, this avoids having to re-implement Cognito and login-with-amazon). This means that with today's code, the Watch can be untethered from the iPhone for up to an hour and can still dequeue records to DynamoDB (assuming the Watch has Wi-Fi access itself)
• If the Watch's credentials are bad, empty or expired and Watch can't access the iPhone or the user is logged out of the iPhone part of the app, then Watch's dequeuer loop is stopped
• Dependent libraries (LoginWithAmazon) are now embedded in the code
• A 'logout' on the phone will invalidate the current credentials on the Watch

This code should now be a bit easier to use for reproducing my experiments. Less moving parts, simpler design. I'll work on the README.md a bit more to help list the steps to set up.

And finally, this demonstrates multi-tenant isolation of the data in DynamoDB. Here's the IAM policy for logged in users:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "mobileanalytics:PutEvents",
                "cognito-sync:*",
                "cognito-identity:*"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Sid": "Stmt1449552297000",
            "Effect": "Allow",
            "Action": [
                "dynamodb:BatchWriteItem",
                "dynamodb:UpdateItem",
                "dynamodb:Query"
            ],
            "Resource": [
                "arn:aws:dynamodb:us-east-1:499918285206:table/sensor2"
            ],
            "Condition": {
                "ForAllValues:StringEquals": {
                    "dynamodb:LeadingKeys": [
                        "${cognito-identity.amazonaws.com:sub}"
                    ]
                }
            }
        }
    ]
}

In the above example the important lines are the condition -- this condition entry enforces that only rows with HashKey the same as the logged in user's cognitoId will be returned. This is why we can build applications with direct access to a data storage engine like DynamoDB!

You can read the details of IAM+DynamoDB here.

Anyway, back to performance improvements of the dequeue process. Everything is running pretty good, but the Watch still takes a long time to get its data moved.

A note on sensor2 dequeue performance

I've examined sensor2 dequeue performance. Some interesting observations indeed!

• A single dequeue loop (1250 samples for 25 seconds) time takes a bit over 7 seconds
• A little under 1 second of this time is getting data from CMSensorRecorder
• Around 4 seconds is required to prepare this data
• The time to send the samples to DynamoDB depends on the network configuration:
• 3 - 6 seconds when the Watch is proxying the network through iPhone using LTE network (with a few bars of signal strength)
• 2 - 4 seconds when the Watch is proxying the network through iPhone (6s plus) and my home WiFi
• Around 1.5 seconds when the Watch is directly connecting to network using home WiFi

Speeding up the data preparation will help some.  I will set a goal of 1 second:

• Hard coded JSON serializer
• Improvements to the payload signer
• Reduce the HashMap operations (some clever pivoting of the data)

New "serverless" site to explore sensor data

I have updated the UI to parse and render the data from the new data model. You can try it out here.

Recall the data flow is:

• CMSensorRecorder is activated directly on the Watch
• When the application's dequeue is enabled, the dequeued events are:
• Parsed directly into our DynamoDB record format
• Directly sent to DynamoDB from the Watch

And this pure static website directly fetches those records and pivots the data in a vis.js and d3.js format for display.

Next up:

• Get AWS Cognito into the loop to get rid of the long lived AWS credentials
• Work on the iOS framework memory leaks
• Speed up the dequeue (or, resort to a Lambda raw data processor)