by hushhh
Oct 18, 2025
What I’ve Been Building (and Breaking)
Last week I got Prometheus and Grafana up and running. This week I hit a new problem. Some alerts weren’t firing when they should. The dashboards looked fine, but the numbers were lying.
Prometheus was scraping fine, but the app stopped updating its metrics in certain cases. The targets showed as “up,” but the data was stale. Grafana kept plotting old values, and nothing looked broken until things actually broke.
What I learned:
- “Up” in Prometheus doesn’t mean your metrics are healthy.
- Stale metrics are quiet but dangerous.
- Alerting on freshness matters just as much as alerting on failures.
Takeaway:
Monitoring isn’t useful if it reports the past. Make sure your metrics are alive, not just available.
A Video I Found Valuable
Game Theory: Introduction – By Daniel Bonevac
Game Theory offers a powerful lens for understanding strategic interactions which is highly relevant for building complex systems. This video provides a clear practical introduction to its foundational concepts which helps you think critically about how different agents (or system components) behave.
Here’s what it covers:
- What Game Theory is about: Strategic thinking, decision-making, and understanding conflict and cooperation among rational players.
- Zero-Sum vs Mutually Beneficial Games: Distinguishing scenarios where gains are balanced by losses (like market share) from those where all can benefit or lose together (like economic exchange).
- Simultaneous vs Sequential Games: How the timing of decisions, whether made without knowledge of others’ moves (e.g., stock market) or in response to them (e.g., chess) fundamentally alters strategy.
For engineers designing distributed systems, understanding game theory can help anticipate behavior, create robust protocols, and optimize resource allocation in multi-agent environments. It is a key framework for deeper system intuition.
Paper I Read
INFLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptation
By Tsinghua University & OpenBMB | Link
Most long-context work tries to replace dense attention with fancy sparse attention schemes, but that usually breaks compatibility with pretrained weights or requires retraining from scratch. This paper takes a much simpler and smarter route.
Instead of creating a new attention mechanism, INFLLM-V2 keeps the original dense attention exactly as it is and adds a switch that activates sparse attention only when the sequence gets long. The same weight matrix powers both modes. No need to duplicate layers, no need to re-pretrain, and the model can flip between fast local focus and wide-range context on demand.
The coolest part is that the sparse path reuses the dense weights directly, which means you can retrofit existing short-context models into long-context models without touching their core parameters. It turns long-context support from an architecture problem into a routing problem.
If you like elegant hacks over heavyweight redesigns, this one is worth a look.
Three Engineering Reads Worth Your Time
Data Gateway — A Platform for Growing and Protecting the Data Tier
Read it
Netflix’s solution for letting hundreds of teams use different databases without chaos. Their Data Gateway abstracts away database complexity, stops common anti-patterns, and enforces security. A solid case study on building a “paved road” that makes the right architectural choice easy for developers.
CAP Theorem: The Simple Explanation
Read it
A clear and practical breakdown of the fundamental trade-off in distributed systems. It explains what the CAP Theorem (Consistency, Availability, Partition Tolerance) actually means when choosing a database. Essential if you want to understand why you sometimes have to pick availability over consistency during network failures.
Ably’s Platform Scalability
An inside look at how a high-scale realtime platform is built for horizontal scalability. It covers the core architectural principles, including how they use consistent hashing to distribute load and a multi-layered design for independent scaling. It’s a great read on the practical challenges and solutions for building a system that can handle unpredictable traffic spikes.
Final Thoughts
This week’s Prometheus hiccup reminded me that observability is more than just collecting data. It’s about trusting what you see. Metrics can be sneaky and dashboards can lie if you’re not careful. Building monitoring means thinking ahead about how things can silently fail, not just when they shout.
If you’re digging into tricky bugs or tuning systems, keep an eye on the quiet failures. They often hide the real story.
As always, I’d love to hear what you’re building or breaking. Reply anytime.
See you next Saturday.
— hushhh