Incident management at Mercado Livre works because the company treats it as three stages: prevent the failure, detect and contain it fast, and learn from it in a blameless postmortem. This article walks through the practices an engineering expert there described, from cell-based architecture to automatic rollback windows.
How does incident management work at Mercado Livre?
Incident management at Mercado Livre is organized in three stages: prevent incidents through resilient architecture, detect and contain them quickly with observability and runbooks, and learn from every failure in a blameless postmortem. Thiago Gambarra, an engineering expert (the Mercado Livre equivalent of a staff engineer) for the user registration platform, described this system in an April 2025 live session on the Full Cycle channel.
His team owns registration across Latin America. By his account, roughly 10 core applications in his team talk to more than 200 microservices in the wider registration ecosystem, serving Mercado Livre, Mercado Pago, and partner companies in countries such as Argentina and Mexico. When the front door of registration opens, the team is responsible for the health of every service in that chain, not just its own code.
What architecture prevents incidents before they happen?
Prevention starts with a written map of every failure point: external services, databases, third-party libraries. Once the team knows what can break, it chooses a resilience pattern for each dependency and weighs the cost of that pattern against the real business damage of the failure. Software that tolerates faults is expensive in money, complexity, and team maturity, so the trade-off is always explicit.
Two patterns carry most of the load in this setup, according to Gambarra.
### Circuit breakers and idempotency
A circuit breaker watches a failing dependency, opens when the error rate or health checks cross a threshold, and redirects traffic to an alternative flow, like a railway switch sending the train down another track. It can trip automatically or manually through a feature toggle. Retries are applied carefully: on a write operation without idempotency, a retry can duplicate the effect.
Idempotency closes that gap. Each operation carries a unique key; if a second request arrives with the same key, the system rejects it. That makes retries safe and prevents race conditions from double-processing a payment or registration step.
### Cell-based architecture and multi-cloud
Mercado Livre runs a multi-cloud footprint on AWS and GCP across multiple regions and applies cell-based architecture, called 'chaving' internally by the speaker: splitting the operational load into small scoped pieces by country, platform, or even mobile OS. A deployment mistake scoped to Brazil, or an infrastructure failure in one region, leaves every other cell serving users normally.
The company built its own internal developer platform, called Fury, to standardize all of this. Per the speaker, a platform organization of more than 600 people maintains it so product teams get a ready stack, pre-wired observability, and signed container images instead of making their own infrastructure decisions. New backend code is encouraged in Go or Kotlin with Spring, and the frontend uses React on Node.js. An internal 'mesa de portabilidade' forum keeps technical work aligned with business priorities.
How do logs and metrics differ in observability practice?
Logs and metrics answer different questions, and confusing them is expensive. A log records the detail of one operation: the request, the timestamp, the error. A metric is a low-cardinality indicator selected from that detail, such as the operation type or country. Storing searchable logs is one of the costliest parts of high-availability operations because full-text search over logs demands heavy compute, so hot log windows are short and cold storage is cheap.
Gambarra's team keeps logs hot for about one week in Elasticsearch with Kibana, then archives them in cold storage such as BigQuery for rare rehydration needs, for example when legislation requires an old record. Vendor tools such as Splunk carry steep license costs, which is why the hot window stays short.
The trade-offs between the two signal types look like this:
| Dimension | Logs | Metrics |
|---|---|---|
| Cardinality | High, per-operation detail | Low, pre-aggregated |
| Storage cost | Expensive while hot, cheap when cold | Cheap to retain long-term |
| Question answered | What happened in this specific request? | How is the system trending right now? |
| Typical retention | About one week hot, then archive | Long retention for baselines |
Metrics split into two families: application metrics like CPU and response time, monitored with New Relic APM, and business metrics like registrations started, stored in Datadog. The team adopted OpenTelemetry to avoid vendor lock-in, since New Relic's APM pricing was high and Fury abstracts the instrumentation so switching vendors means updating a dependency, not rewriting thousands of services. Alerts watch anomaly types by volume, both spikes (possible attack) and drops (possible failing service), and the team is beginning to experiment with AI-assisted anomaly detection because fixed statistical baselines miss expected events like a sponsored TV broadcast.
Two dates frame this stack's evolution: the Elasticsearch project that anchors the hot log window was first released in 2010, and OpenTelemetry — the layer that keeps the team portable between APM vendors — was formed in 2019 from the merger of OpenTracing and OpenCensus.
What happens when an incident is detected?
Detection triggers a predefined response, not an improvisation. A rotating on-call team, called 'guardia' internally, owns the incident, and one designated person directs it so decisions do not dissolve into debate. Runbooks document which decision to take for each failure scenario, ideally executable in one click.
Fury ties rollback to alerts. If a business, application, or dependency metric degrades during a deploy, the platform recommends a rollback, and severity sets the decision window the speaker described: 15 minutes for low severity, 5 for high, and 2 for critical, after which rollback happens automatically. His advice during an incident: stop diagnosing, contain the bleeding first, and save the root-cause work for the postmortem.
How does the team measure and learn from incidents?
Mercado Livre measures incident impact with a single headline KPI — 'tempo de afetação' (impact minutes) — and learns from incidents through blameless postmortems that are shared across teams. The metric folds detection and recovery into one number, similar to combining MTTD and MTTR, while the postmortem process turns every significant failure into categorized findings and actionable backlog items.
The headline KPI: 'tempo de afetação'
The speaker explained the weighted figure with an example: if Venezuela represents 1 percent of traffic and a metric drops 50 percent there for 1 hour, the contribution is roughly 0.5 percent-times-60-minutes of impact. The quarterly sum across all incidents must stay under 5 minutes per metric. Team quality scores, bonuses, and promotion visibility reference these numbers, and underperforming teams enter a 'freezing' process with extra expert review and stricter rollout-plan checks.
Blameless postmortems and shared learning
Every postmortem document states what happened, the root cause, which metric was affected, how many users or flows were hit, and a clear action plan; identifying the author of a bad line of code is explicitly out of scope — a principle the blameless-postmortem movement has championed since at least 2012, when Etsy began publishing its postmortem culture. Failures are categorized as software, dependency, infrastructure, or external events, so a telecom outage does not penalize a team that could do nothing about it.
A weekly incident review presents one team's postmortem to all teams, and findings that are critical enough become mandatory backlog items everywhere. Postmortems are also written for significant bugs that cause friction without an outage. This aligns with the reliability-engineering practice popularized by Google's SRE book in 2016, which ties incident review directly to engineering priorities.
| Dimension | Mercado Livre approach | Typical alternative |
|---|---|---|
| Headline metric | Impact minutes, weighted by traffic share | Separate MTTD and MTTR |
| Quarterly target | Under 5 minutes per metric | Error budgets per SLO |
| Postmortem scope | Includes friction-causing bugs, not just outages | Sev-1 outages only |
| Cross-team learning | Weekly review; critical findings become mandatory backlog items | Postmortems stay within the owning team |
- Detect and measure the incident in impact minutes, weighted by traffic share.
- Write a blameless postmortem covering what happened, root cause, affected metric, user impact, and an action plan.
- Present the postmortem at the weekly cross-team review and promote critical findings to mandatory backlog items.
FAQ
- Does Mercado Livre aim for zero failures? No. The speaker states plainly that no system is failure-free; the goal is to fail as little as possible, detect fast, recover fast, and learn so the same failure does not repeat. He confirmed major outages have happened even at Mercado Livre, as they have at Apple, Microsoft, and Facebook.
- Why not use a monolith instead of 200 microservices? The speaker cites Stack Overflow's low-cost monolith as valid for its context and warns teams without mature observability and disciplined service boundaries end up with a 'distributed monolith'. Mercado Livre values independent deploys and rapid change, which pushes it toward services, but he stresses the choice depends on team maturity, budget, and how mutable the product must be.
- What error threshold triggers alerts? The registration team's error-rate alert fires at 0.01 percent, per the speaker. He cautions that the right number depends on the business, and that returning wrong status codes (for example, a 400 for a server-side database failure) can hide real errors from monitoring.
- How is CAP theory applied in practice? The team asks, per feature, what an incident may degrade: during registration, the identity-verification step (KYC) can be deferred and requested later, keeping the main flow available. Consistency, availability, and partition tolerance are weighed against what the business can actually tolerate.
- What role does AI play in incident response? Developers use GitHub Copilot, ChatGPT, and Gemini in daily work, and the team is beginning to experiment with AI for anomaly detection beyond static algorithms. The speaker's position is that AI will not replace developers soon, but developers will need analytical and business skills to stay relevant.
Turn your own engineering talks into articles
The practices above came from a two-hour live session, and most of that knowledge usually stays locked inside video recordings. If you give talks, run interviews, or explain architecture on YouTube, Skala Blog turns that video into a written article: paste the URL, transcribe it, and generate a publishable draft you can review and edit.
Fork this article
Start a new branch from the same video, shaped your way. You keep the credit; the original keeps the attribution.
A fork in another language is filed as a translation of this article, so the two pages point at each other. You can unlink it later from the editor.
0/240
You are creating
- Format
- For
- Language
- Source
- Your angle
You will be asked to sign in before it is generated.
Buy credits