by hushhh
October 7, 2025

Welcome to the first real issue. It’s been a week, and I’ve already broken things. Let’s get into it.

What I’ve Been Building (and Breaking)

Project: Setting up monitoring with Prometheus and Grafana
What broke: I got Prometheus scraping my service and figured everything was fine. Then I noticed weird gaps in the data. Some metrics wouldn’t show up at all, and the graphs in Grafana looked off. After digging, I realized my /metrics endpoint was getting blocked when the app was under load. On top of that, Grafana was still showing old data from jobs I had renamed but never cleaned up.

What I learned:

  • Just exposing metrics isn’t enough. The endpoint needs to stay responsive under pressure
  • Messy or inconsistent labels will come back to bite you
  • Dashboards can look fine even when the data underneath is broken

Takeaway:
Monitoring only works if the system is built to be observable. If your metrics fail when things go wrong, they’re not going to help much.

A Video I Found Valuable

Cloud Computing Explained: The Most Important Concepts To Know – Be A Better Dev
Watch it on YouTube

If you’re building systems in the cloud or just want to understand how things actually work, this video is a solid primer on the key cloud computing concepts. It’s clear, practical, and connects the tools we use to the principles behind them.

Here’s what it covers:

  • Scaling vertically vs. horizontally and why horizontal scaling usually wins
  • Load balancing and autoscaling
  • What “serverless” really means (and doesn’t mean)
  • Event-driven architecture and why it matters
  • Container orchestration
  • Object storage, block storage, databases, and caches
  • Availability and durability
  • Infrastructure as code
  • Cloud networking and isolation

If you’re piecing together systems in a real-world environment, this is a great place to reinforce the fundamentals.

Paper I Read

KAVA: Latent Reasoning via Compressed KV-Cache Distillation
By Qualcomm AI Research | Link

This paper explores how language models can reason internally without writing out their full thought process. Normally, models do better when they explain themselves step-by-step, but that makes them slower and more expensive.

KAVA’s idea is to keep the benefits of reasoning without the extra text. A smaller student model is trained using the compressed KV-cache of a larger teacher model. The KV-cache is the internal memory of attention. So basically, how the model “thinks” as it generates responses. The student doesn’t copy the words, it copies the behavior.

Even though the cache is compressed and doesn’t map directly to tokens, it still contains enough structure for the student to learn from. The result is a model that can reason silently, with near the same accuracy as models using full chain-of-thought, but much faster and lighter.

If you’re interested in how reasoning might evolve beyond text or how to make models more efficient without making them worse, this paper is worth your time.

Three Engineering Reads Worth Your Time

Resiliency in Distributed Systems
Read it
A practical overview of how to design systems that stay up when things go wrong. It covers timeouts, retries, circuit breakers, rate limiting, and load shedding. Also digs into upstream versus downstream failure modes and how to think about both sides of the system boundary.

How Discord Stores Trillions of Messages
Read it
Discord walks through their migration from Cassandra to ScyllaDB and the lessons learned from scaling message storage at massive scale. Includes topics like handling hot partitions, building a coalescing layer, and validating a safe cutover in production.

Designing Robust and Predictable APIs with Idempotency
Read it
A great piece from Stripe on why idempotency matters, especially for APIs that handle financial operations. Covers how to use idempotency keys, what pitfalls to avoid, and how to think about side effects in retry scenarios.

Final Thoughts

That’s it for this week. Setting up monitoring reminded me that observability isn’t something you add later. It has to be built in from the start. And even small bugs are worth paying attention to they usually lead somewhere deeper.

If you’re building, breaking, or just thinking through hard problems, I’d love to hear what you’re learning. Reply anytime.

See you next week.
— hushhh