telexed ~ c / ba49eba0-220radar:50 · infra_saasLIVE
← back
NO.
#ba49eba0
Topic
INFRA & SAAS
Source
Hacker News · Bootstrapped
Published
2022-06-28 13:23:44
Importance
★ 5/10 — radar 50

`CloudFront` + `Lambda@Edge` recursion turned into a **$4.6k** surprise bill

A self-calling edge function ran for nearly a day, while billing alerts lagged badly enough to miss the blast radius. If you use serverless on AWS, request alarms and hard stopgaps matter more than budget emails.

[ KEY POINTS ]
  1. A recursive Lambda@Edge path with a 30s timeout kept firing through CloudFront, burning over 70M GB-seconds before it was stopped.
  2. AWS Budgets did not catch it in time because CloudFront charges arrive with delay across regions; next-day visibility is too late for runaway loops.
  3. Removing the function within 30 minutes still was not enough, because the bad deploy had already been running for roughly 24 hours.
  4. The practical lesson is blunt: budget alerts are not a kill switch. Pair serverless with CloudWatch request alarms, per-region monitoring, and tighter blast-radius limits.
Originalnews.ycombinator.com/item?id=31907374Read original →

// related