Amazon's iOS SDK KinesisRecorder: bug found!

Recall earlier posts discussing 50% extra Lambda->DynamoDB event storage. It turns out the problem is the AWS SDK KinesisRecorder running in the iPhone. Unlike the sample code provided, I actually have concurrent saveRecord() and submitAllRecords() flows -- sort of like real world. And this concurrency exposed a problem in the way KinesisRecorder selects data for submit to Kinesis.

Root Cause: rowid is not a stable handle for selecting and removing records.

Anyway, I made a few changes to KinesisRecorder:submitAllRecords(). These changes are mostly to index records by their partition_key. This seems to work ok for me. However, it may not scale for cases where the KinesisRecorder winds up managing a larger number of rows. This needs some benchmarking.

Pull request is here.  And here's updated iPhone code to do the right thing.

As they say "now we're cookin' with gas!"

Here we see the actual storage rate is around the expected 50 per second. The error and retry rates are minimal.

Sooo, back to now analyzing the data that is actually stored in DynamoDB!