Fallback and degradation in AI services

An AI service depends on things you do not control: the LLM provider, the GPU, a vector database, a budget. All of them will fail. The question is what the user sees when they do.

Degradation = the service does less but does not go down:

• The LLM is down → prewritten explanations, rule-based grading, «the tutor is temporarily switched off».
• Retrieval is down → answer without sources, but say so.
• The budget is spent → switch off the expensive features, keep the cheap ones.

A kill switch = a flag a human can turn on in seconds, without a deploy: LLM_OFF, SIGNUP_OFF, PANIC (a static page only). You need it for the incidents you did not foresee.

The rule: faults in dependencies degrade; faults in your own logic stop (better a 503 than a wrong answer). And the degradation has to be tested — a fallback that has never run does not work.

import time, httpx, logging
log = logging.getLogger("svc")

class LLMRouter:
    def __init__(self, flags, budget):
        self.flags, self.budget = flags, budget
        self.fail_until = 0.0                      # a simple circuit breaker

    async def complete(self, prompt, fallback_text):
        if self.flags.get("LLM_OFF"):
            return fallback_text, "degraded:flag"
        if self.budget.exhausted():
            return fallback_text, "degraded:budget"
        if time.time() < self.fail_until:
            return fallback_text, "degraded:breaker"
        try:
            async with httpx.AsyncClient(timeout=20) as c:
                r = await c.post(URL, json={"prompt": prompt}); r.raise_for_status()
            self.budget.charge(r.json()["usage"])
            return r.json()["text"], "ok"
        except (httpx.HTTPError, KeyError) as e:
            log.warning("llm_fail %s", type(e).__name__)
            self.fail_until = time.time() + 60        # open the breaker for a minute
            return fallback_text, "degraded:error"

# the test: patch httpx so that the call fails → expect fallback_text and a status starting with "degraded"

AI-grafen's own LLMRouter does exactly this: ops.flag, a daily budget and a fallback to the node's prewritten text.