For my first serverless project, I built a service that manages an advertising account, by analysing ads and optimising the spend spread for a configured goal. I was surprised to find that one of the most difficult aspects, was in throttling outgoing API requests. Basically, I needed to avoid abusing a 3rd party service by hammering it with parallel or rapid fire requests.
I discovered all kinds of weird solutions:
- Push a "please wait" message onto an SQS queue, and set a Cloud watch alarm to fire when the queue length was >= 1 for at least one second. Have a variety of lambdas respond to the alert by checking if the have any work to do, and then racing to pop the queue and do one operation. I think I needed SNS in there too, for some reason. Fanout?
- Build a town clock on an EC2 and have it post "admit one" tokens to SNS. Nice and simple, but I was evaluating serverless, and adding a server felt like defeat. Might as well just run the whole thing on that EC2.
- Have on lambda pop request descriptions from a queue, and sleep for one second after sending each one. The request responses would have been pushed onto another queue. All of the other lambdas could just handle the events they cared about as quickly as data became available. Sleeping in a lambda just felt wrong, but for one second, it probably would have been fine.
In the end, I was still experimenting and exploring when my estimate loomed near, so I just wrapped up all of the application code and piped it all together with delays in loops that made API requests. Deployed it to Heroku in a hurry with minimal fuss.
What I learned:
- There are bad use cases for serverless, and I had started on one (SLS doesn't wait on no one)
- Serverless does encourage a clean architecture, from a code perspective: I was able to quickly transform it into a completely different type of application.
- AWS's services have lots of little surprises in their feature set: SQS doesn't fire any kind of event. SNS can't schedule events in the future (maybe I should have expected that one). Cloud watch events can only trigger a single lambda.
- There is a need for a town clock / scheduler as a service in AWS'. How many people have built that themselves?
> "- Push a "please wait" message onto an SQS queue, and set a Cloud watch alarm to fire when the queue length was >= 1 for at least one second. Have a variety of lambdas respond to the alert by checking if the have any work to do, and then racing to pop the queue and do one operation. I think I needed SNS in there too, for some reason. Fanout?"
When you evaluate the AWS serverless platform, I recommend looking at everything that is an event source, sink, or processor: Lambda, SNS, SQS, SWF, SES, Cognito, IAM, Kinesis, IoT, API Gateway, S3, DynamoDB.
I did look at Kinesis too, but for my use case, it didn't really help much over SQS. There is still no way to put a Kinesis event "in the future" so that I won't see it for a second, and I would have had to set the batch size to 1, if I was using Kinesis as a means to throttle an activity.
In the spirit of treating this like Lego, I suppose that maybe I could have uses SES to email itself an HTTP request description, and counted on the email lag to keep requests spread out over time. But at this point, things are getting so weird that I think it's time to just admit that "work on something at a specific pace, then go to sleep for a few hours" isn't the sort of task that the serverless architecture is well suited for. Serverless is for tasks that you want done on demand, as quickly as possible, imo.
Ah yes, I forgot about that. What I really needed from the clock was for it to somehow only trigger lambdas that actually had work to do. I didn't like the idea of them polling once per second per function, when 99℅ of the time, they would have nothing to do. I guess "town clock" was a mistake on my part.
I discovered all kinds of weird solutions:
- Push a "please wait" message onto an SQS queue, and set a Cloud watch alarm to fire when the queue length was >= 1 for at least one second. Have a variety of lambdas respond to the alert by checking if the have any work to do, and then racing to pop the queue and do one operation. I think I needed SNS in there too, for some reason. Fanout?
- Build a town clock on an EC2 and have it post "admit one" tokens to SNS. Nice and simple, but I was evaluating serverless, and adding a server felt like defeat. Might as well just run the whole thing on that EC2.
- Have on lambda pop request descriptions from a queue, and sleep for one second after sending each one. The request responses would have been pushed onto another queue. All of the other lambdas could just handle the events they cared about as quickly as data became available. Sleeping in a lambda just felt wrong, but for one second, it probably would have been fine.
In the end, I was still experimenting and exploring when my estimate loomed near, so I just wrapped up all of the application code and piped it all together with delays in loops that made API requests. Deployed it to Heroku in a hurry with minimal fuss.
What I learned:
- There are bad use cases for serverless, and I had started on one (SLS doesn't wait on no one)
- Serverless does encourage a clean architecture, from a code perspective: I was able to quickly transform it into a completely different type of application.
- AWS's services have lots of little surprises in their feature set: SQS doesn't fire any kind of event. SNS can't schedule events in the future (maybe I should have expected that one). Cloud watch events can only trigger a single lambda.
- There is a need for a town clock / scheduler as a service in AWS'. How many people have built that themselves?