Secure LLM Gateway for a 63,000-Person Enterprise
36%
Lower P95 latency40%
Fewer content errors15%
Lower token costs
Client Overview
Business Problem
Situation Before
Core Pain
Risks
- Sensitive data could leak through LLM interactions that were neither filtered nor governed in any central way
- Under peak load the system could slow down or break its service level agreements with the business
- An LLM version upgrade could quietly change behavior and degrade results without anyone noticing
- Without auditing or cost attribution, AI resource consumption could not be tracked or explained
Constraints
- Availability had to meet strict enterprise standards, so the platform had to run with minimal downtime
- Internal data was sensitive, so every request and response needed careful handling and filtering
- Load was high and hard to predict because the organization had more than 63,000 people
- The platform had to integrate with the client's private cloud infrastructure and their own LLM
Project Goals
- Strengthen security and control so there is less risk of data leaking through LLM interactions.
- Reduce latency and improve response times for end users across the whole organization.
- Add full billing and audit, with cost tracking down to the token level and clear accountability for usage.
- Give the client clear data on how employees actually use the LLMs.
- Build a scalable, SOA-based API gateway with input/output validation and token billing.
- Optimize database and caching layers (PostgreSQL + Redis) to reduce query overhead.
- Implement content filtering via Guardrails and a banword service.
- Introduce an async processing pipeline (asyncio + RabbitMQ) for parallel LLM request handling.
- Establish robust CI/CD, regression testing, and monitoring to support safe LLM stack upgrades.
- A clear and measurable reduction in P95 latency
- A measurable improvement in system availability
- Fewer content-related errors and policy violations
- Lower token costs, achieved mainly through caching
- Zero critical regressions across major LLM version upgrades
Solution
The design of the platform included independently-deployable service components: an API Gateway for routing requests, validating requests and billing tokens; a History and Cache Layer based on Redis and PostgreSQL; a Content Filtering Service using Guardrails and banword-based filtering rules; an asyncio based and RabbitMQ based pipeline for parallel and non-blocking execution of LLM requests; a pre/post-processing layer for data normalization around model invocations.
The team developed the Prisma schema design and PostgreSQL queries optimization; implemented an API gateway using FastAPI, input validation and token calculations; and set up the Redis cache with TTL. That included a content filtering solution using Guardrails and banword service implementation. The team designed an asynchronous RabbitMQ pipeline for parallel LLM requests, reducing waiting time to get results from the LLM. Other areas involved in the project covered pre/post-processing of LLM data, upgrading of the LLM stack with regression fixes and CI/CD in GitLab.
All services were deployed on the client's own servers inside a private internal environment, rather than on third party cloud providers. An application was packaged using Docker then deployed to Kubernetes, which takes care of scheduling and managing containers within the cloud. The team was able to monitor and receive alerts on the health of their systems in real time using Prometheus and Grafana. The build and deployment of applications were done automatically using GitLab CI/CD. The team used GitLab as a source control system and used Jira and Confluence for task tracking and documenting.
Security was applied at several points. Both input and output were subject to content filtering, via Guardrails combined with a banword service; unsafe text was prevented from reaching the model and also prevented from leaving it to return to a user. All API endpoints had defensive validation, whereby all requests to the API were examined before being processed. Spending at the token level was logged, providing an audit trail. All integration was held entirely within the client's own network and none of it was exposed outside the network to any third party cloud infrastructure.
Key Steps
The project started in July 2025. The team came on board and set up how they would work, using Scrum with two week sprints. Most of this phase went into designing the architecture. They chose a service-oriented architecture (SOA) and decided which separate services the platform would need.
The team spent Q3 2025 building out the foundations of the platform. During this time they built the API gateway in FastAPI with token usage logic, designed the database schema in Prisma and set up the Redis caching layer with a time to live configuration to stop repeated identical queries from hitting the database. The Prometheus and Grafana monitoring stack was also set up at this stage.
Across the third and fourth quarters of 2025 the team added content filtering. They combined Guardrails with a custom banword service to screen both the text users sent in and the text the models sent back. They also built the pre and post processing layer that normalizes and transforms data right before and right after each LLM call.
From Q4 2025, the team focused on hardening and improving the LLM stack in a secure way. Pytest-based regression tests were executed in the CI/CD pipeline, and release roll-back functionality was introduced. The team watched for unusual behavior after each upgrade and optimized performance so response times stayed low under load.
Results & Impact
Project Snapshot
See Other Works
Bond Settlement Cut from 2 Days to 30 Minutes
End-to-end capital markets platform automating bond issuance, trading, settlement, and investor servicing
API Response Time Down 80% for a Bloomberg Rival
Financial analytics platform delivering real-time market data, news processing, and scalable backend services for fintech
Power Station Analytics: Algorithm Suite Delivered
Real-time equipment analytics platform for a power station with automated calculations, monitoring, and reliable fault detection
