RCA service

Challenges

1. 01

   Slow responses from sequential lookups

   Problem

   Each analysis needs data for several pieces of equipment, and the first version fetched them one by one. Response time grew with every item.

   Approach

   I moved the lookups out of the loop and ran them concurrently, with sensible limits so a large request can't flood the systems it depends on.

   Outcome

   Analysis time stopped scaling linearly with the number of items, and the service handles bigger requests comfortably.

2. 02

   One bad dependency breaking every request

   Problem

   The service depends on external data sources that occasionally fail or time out, and a single failure used to fail the whole response.

   Approach

   I made those calls fail soft: skip what can't be fetched, log it, and return everything that could be resolved, with tests for each failure path.

   Outcome

   Users get partial, useful results instead of an error page, and failures are visible in logs instead of to users.

3. 03

   Setting the bar for a new codebase

   Problem

   A new service sets conventions everyone after you follows, good or bad.

   Approach

   From the first commits I added controller and service tests, a clear README for setup and deployment, a Dockerfile and a CI pipeline with security scanning.

   Outcome

   Teammates can run, test and deploy the service without asking me, and it has stayed easy to change.

What I took away

• Concurrency is a fix for latency, but limits are what make it safe.
• Partial results beat no results when the user can still act on them.
• Docs and tests in week one are cheaper than in month six.