The Hidden Cost of System Degradations

Your monitoring shows green. SLAs are met. Users aren’t complaining. Yet your costs are high. Some user experiences feel poor. Welcome to the hidden world of degraded experiences. These are system behaviors that “work” but hurt your costs and user experience.

What are Degraded Experiences?

Degraded experiences occur when systems function within acceptable parameters but operate below optimal performance. Instead of failing completely, systems fall back to alternative behaviors that maintain core functionality while accepting tradeoffs in cost, performance, or user experience.

Take payment providers. You have two options. Provider A costs 2% commission. Provider B costs 3.5%. You prefer Provider A. But when Provider A fails, you use Provider B. Your system still works. Your SLA holds. But costs go up 75%.

Consider a non-critical service outage. A degraded experience might show less information. For instance, we skip some information when offering journeys to drivers if we cannot collect it fast enough. This affects the driver’s experience. But the core functionality works.

In summary, degraded experiences from non-critical service downtimes increase resilience but can lead to increased costs or reduced user experience. Understanding their frequency and duration is key to developing a high-quality system.

Visibility is Key

Metrics and alerts often focus on meeting SLAs by setting service level objectives (SLOs). These metrics, focused on critical flows, may overlook degraded behaviors or fallbacks.

Without proper visibility, degradations can become permanent and invisible. Since they don’t break critical flows, they can be hard to detect unless a user complains or unusual high costs are reported. How can we prevent this?

Cabify’s Systematic Approach

At Cabify, we initiated an engineering-wide effort to mitigate these scenarios and their impact, enhancing system quality.

Step one: treat degradations as first-class citizens in your architecture.

• Everything will eventually fail. Consider degradations in the initial product design phase.
• Degrade smartly. Use cost-effective fallbacks to minimize customer impact.
• Degradations should have the same visibility as critical flows. We must have metrics and alerts for all of them.

At Cabify, these statements are just the beginning. This mindset impacts our processes and developments.

• Our product initiative process includes a section on degradations. Both product and engineering should consider fallbacks.
• We provide libraries in different languages. Engineers use them to add observability to degradations consistently.
• This approach gives us a dashboard with a common view of system degradations. We tag them by affected flows and their impact.
• SLO-based alerting for degradations ensures each team knows about unusual degradations.

This approach is transforming our workflow. We found many degradations that went unnoticed for some time. The number of measured degradations now exceeds hundreds.

Key Takeaways

Degraded experiences are inevitable in distributed systems. The question isn’t whether they’ll happen. It’s whether you’ll know about them when they do.

Treat degradations as first-class citizens in your monitoring and alerting strategy. This transforms hidden costs into visible, manageable technical debt.

Start small. Find one fallback mechanism in your system. Add metrics around it. Set up alerts. You might be surprised by what you discover.