ModelDispatcher

Overview

Every application needs the same things from an LLM backend — swap providers without rewriting callers, survive a rate limit, keep one tenant from spending another's budget. Writing that per app means writing it badly several times. ModelDispatcher is the shared layer: one ModelProvider interface every backend implements, a fallback chain that intercepts exhaustion and escalates to the next candidate model transparently, and pre-flight token reservation reconciled after the call. It is the hub of this portfolio rather than a project beside it — KanDOne, HighFive, StepByLearn and JobFlowTracker all consume its published browser client, and AppMyTrip vendors it as a submodule.

Highlights

• Providers are hot-swappable behind a single interface, so a backend change never reaches calling code
• Rate limits and exhaustion escalate through a Chain of Responsibility instead of surfacing as errors
• Token-aware quotas per tenant: reserved before the call, reconciled against actual usage after
• A tool-calling agent loop with explicit state, written without a heavyweight agent framework
• Tested against Python 3.11, 3.12 and 3.13 with a coverage gate at 80%

In the code

• Chain-of-Responsibility fallback
• The ModelProvider strategy interface
• Token-aware multi-tenant quota manager
• Dependency-free agent loop