Halyard

Halyard sits between product code and the models it depends on. Instead of every service calling an embedding or generation endpoint directly, they call Halyard, and Halyard makes those calls fast, observable, and affordable.

The problem

Direct model calls were fine until traffic got spiky. Costs were unpredictable, one noisy tenant could starve the others, and nobody could answer a simple question: which feature is spending the inference budget?

The design

Halyard is a thin gateway with three jobs: batch, cache, and account.

class Budget:
    def __init__(self, limit_per_min: int):
        self.limit = limit_per_min
 
    def allow(self, tenant: str) -> bool:
        used = redis.incr(f"q:{tenant}:{minute()}")
        redis.expire(f"q:{tenant}:{minute()}", 90)
        return used <= self.limit  # over budget requests degrade, not crash

• Micro-batching with a bounded wait window to keep the GPU busy without punishing the tail.
• A shared cache on normalized inputs to absorb the head of the query distribution.
• Per-tenant budgets so one caller cannot degrade everyone else.

The outcome

The tail got dramatically calmer. p99 latency dropped by roughly half under peak load, and for the first time the inference spend was attributable to a feature rather than a mystery.