Scaling Observability in Healthcare: Leadership Lessons from AWS, Terraform & CDKTF
Acronyms used: SRE = Site Reliability Engineering, CI/CD = Continuous Integration/Continuous Delivery, OTel = OpenTelemetry.
The day we stopped flying blind
In one of my previous engagements, we began as a single B2B product (and 9 employees). Traction pulled us into healthcare's full stack—providers, patients, pharmacies, wholesalers, insurers, and we were introducing a new microservice almost every week (also increased our headcount from 9 to 80+ in a year, to 250+ engineers by 3rd year).
One of those weeks, a system automating pharmacy–insurance workflows failed quietly. A critical data stream stalled. Nobody noticed for weeks, until a product rep spotted it, manually.
That incident changed how we worked. We had CloudWatch logs and a few metrics, but we couldn't see dependencies, health, or bottlenecks across the estate. The platform team was small and focused on revenue-heavy integrations. Most developers hadn't been trained to think like SREs. We decided to change the defaults.
- Every new service would ship with dashboards, alarms, and traces.
- We automated onboarding with Terraform and CDK for Terraform (CDKTF) so teams could self‑serve safely.
- We instrumented the stack with OpenTelemetry and enabled AWS X‑Ray to visualise requests and dependencies.
- For the frontend, we added Sentry. Alerts flowed to Slack/email and escalated via Zenduty.
- And we standardised deployments with Serverless Framework templates (`serverless.yml`) so every repo inherited the same patterns.
I will skip adding the original dashboards, so in this post, I'll focus on principles, playbooks, and pitfalls —the parts that matter when leading change (and that you can apply tomorrow).
Context at our scale
- 2k+ microservices (~700 services in each env) across four environments (dev, test, UAT, prod). Each environment is in its own AWS account.
- Mix of APIs via API Gateway and asynchronous workers behind Kinesis and EventBridge.
- Serverless Framework in every repo to keep deployment, config, and observability conventions consistent.
Why it matters: the problem wasn’t one broken Lambda it was organisational visibility. We needed a way to see health across accounts and services without bespoke work for each team. If Observability Access Manager would have been there, then CloudWatch cross‑account observability (via Observability Access Manager) would have given us a managed path to a single monitoring account. However to help with the recent context, I will try to fit OAM (Observability Access Manager) in the following section.
The trade‑offs we owned (and why)
1) AWS‑native first vs. third‑party suites
We trialled popular third‑party stacks. They were polished. They also introduced procurement/compliance delays, as well as operational drift, for teams already learning new patterns. We picked AWS‑native first because it integrated with Lambda/API Gateway/Kinesis out of the box and kept us moving. When we needed vendor‑neutrality, we used OTel/ADOT to keep an exit ramp open.
Leadership lens: Start where your teams can ship now, but design for optionality later.
2) Terraform modules vs. CDKTF
Terraform gave us repeatable modules. CDKTF met product teams in TypeScript/Go, which reduced the time from new service to first deploy. The trade‑off was training and the occasional provider drift. We handled it with templates, docs, and office hours.
Leadership lens: abstractions buy adoption, so long as you budget time for enablement.
3) Central control vs. local ownership
We tried platform‑owned dashboards first. They went stale. What finally worked was enforcing a floor (no telemetry, no deploy) while coaching two critical teams to become internal champions. They proved value fast and then pulled the rest of the org along.
Leadership lens: set guardrails centrally. Grow habits locally.
The scars that made it stick
1. False positives killed trust
Our first alarms spammed channels. Engineers muted them. We tuned thresholds, shortened evaluation windows for critical paths, and routed P0/P1 to Zenduty, P2/P3 to tickets. Trust returned.
2. Slow adoption without coaching
Dashboards existed, but teams didn't know how to read them. We ran a focused enablement program, starting with two teams that were experiencing the most significant challenges. They became thought leaders and unblocked peer teams.
3. Frontend saved the day (once)
A 5xx error blanked a page. An unhandled exception started failing Lambda invocations. Sentry's lower threshold triggered a first response, which pinged UI engineers, who then escalated the issue to the backend team and averted a broader incident.
Our operating model (the part you can copy tomorrow)
Golden templates over heroic effort
- Serverless Framework templates (`serverless.yml`) in every repo aligned deploys, IAM, and logs. Teams started with the right shape rather than a blank file.
- Terraform modules + CDKTF wrappers bootstrapped a service with dashboards, alarms, and tracing toggles from day one.
Result: new services arrived production‑shaped instead of "we'll add observability later."
Cross‑account visibility early (to be in the current context)
We linked all workload accounts to a single monitoring account using CloudWatch cross‑account observability / Observability Access Manager (OAM). One place to search metrics, logs, and traces across accounts—no DIY log shipping.
Tracing by default, not as a last resort
We enabled AWS X‑Ray and OpenTelemetry (via ADOT) to draw service maps and follow latency across boundaries. That moved us from guessing to knowing during incidents.
Cost awareness without turning this into a cost post
We monitored CloudWatch Logs Insights queries because they are billed per GB scanned. We taught teams to scope time windows and log groups and to prefer saved queries for common hunts. (If you want a deep cost analysis, I'll cover it in a separate post)
What didn't work (so you don't have to repeat it)
- Generic dashboards owned by the platform team. They aged fast, and nobody felt responsible.
- "Observability Fridays." Nice idea. Didn't move the needle. Enablement landed when we paired with two teams on their critical path.
- Everything everywhere alerts. We had to decide what matters, for whom, and where it should wake people up.
Outcomes that mattered (beyond tools)
- Detection time dropped from "weeks if a human spots it" to minutes with alarms and Sentry → Slack.
- MTTR for major systems settled under 20 minutes because traces and dashboards showed where to look, not just that something was wrong.
- Consistency improved: with Serverless + Terraform/CDKTF templates, every new repo arrived with observability as a first‑class concern.
- Optionality stayed intact: by standardising on OTel/ADOT, we kept the door open for non‑AWS backends in the future.
Pragmatic guidance if you're starting now
- Decide the minimum bar. Pick 5–7 metrics that define "healthy" for your top services and make them non‑negotiable.
- Codify it once. Put those metrics, alarms, and tracing toggles into templates (Serverless/Terraform/CDKTF). New repos inherit, old repos migrate gradually.
- Link your accounts. Stand up a monitoring account and connect sources with OAM. You get one pane to search metrics, logs, and traces.
- Coach two lighthouse teams. Make adoption visible. Let them tell the story.
- Tame noise early. Route P0/P1 to humans; demote the rest to tickets.
- Keep an exit ramp. Use OTel/ADOT instrumentation to avoid lock‑in.
👉 If you're scaling AWS and want a short, pragmatic review of your observability setup, book a slot in my calendar or reach out at hi@iamkaustav.com.
👉 Don’t forget to subscribe to my newsletter for more insights on my security and product development journey. Stay tuned for Part 2!
💡 A quick self-promotion: I’m developing a simple freelance management platform for technical freelancers. If you are interested be the founding users and get an early access, please join our waitlist.
