LLMOps Console
Designed and built end to end by one engineer. Architecture, gateway, backend, web app, tests, CI and infra.
A model gateway and an operations console for LLM apps. Apps call one gateway by an alias like agent-default, and the gateway picks the provider, strips personal data, checks budgets and rate limits, and records the cost of every call. An eval gate plays real agent conversations on each pull request, with a simulated user, and grades each one by what changed in the database. A release that breaks a safety rule, or drops below the saved baseline, is blocked.
- 3 plays per caseEvery agent conversation runs three times. It passes only if the right change really happened in the database, not if the reply just sounded right.
- Safety scored apartA broken rule, like a change made without the user's yes, blocks the release on its own, whatever the other scores say.
- Silent fallbackWhen a provider fails or sends an empty answer, the call moves to the backup model. The app never sees the failure.
- 100% safety across 126 playsAcross every play of the last eval run, no safety rule was broken.
What you'll see in the demo
Use the helpdesk demo first, so there are fresh model calls to look at.
- Press Open the demo. It signs you in to a read-only view, with nothing to type.
- Overview shows today's requests, cost, speed, error rate and the last eval.
- Cost and Requests show spend per customer and model, what the gateway hid, cache hits and a trace for each call.
- Quality shows each eval run. You see safety, how often the agent finished the job, and which failures were the agent's fault and which were the test's.
The code, the design and how to run it locally are in the llmops-console repo on GitHub.
github.com/masudur-rahman-musa/llmops-console ↗Skills used
LLM platform
- LLM gateway with model routing and fallback
- LLM-as-judge evals with a CI eval gate
- PII redaction
- Semantic caching
- Token cost tracking and budgets
LLM APIs
- Anthropic Claude API
- OpenAI API
- Azure OpenAI
Observability
- OpenTelemetry
- Langfuse
- Prometheus
- Grafana
Backend
- Python (async)
- FastAPI
- PostgreSQL
- Redis
- Qdrant
Cloud and infra
- Terraform
- Kubernetes (Kustomize)
- Docker
CI/CD and testing
- GitHub Actions
- Trivy image scans
- k6 load tests
- pytest
- vitest
Frontend
- Next.js
- React
- TypeScript