Services / Intelligence
AI and LLM engineering with cost per request on the dashboard
We build production AI systems: retrieval-augmented generation with hybrid search and reranking, agent workflows where they are justified, model tiering and routing, evaluation harnesses that gate releases, and guardrails with PII redaction and audit logging. Cost per request and answer quality are instrumented from the first prototype, because both are what determine whether the feature survives its second month.
- Model routing by task class, typically 55 to 75 percent cheaper than single-model
- Quota planning against TPM, RPM and daily token ceilings before launch
- Evaluation suite in the pipeline so quality regressions fail the build
- An honest list of where we would not use an LLM at all
At a glance
- Platforms
- Amazon Bedrock, Azure OpenAI, Vertex AI
- Retrieval
- Hybrid BM25 plus vector, with reranking
- Vector stores
- pgvector, OpenSearch, Pinecone, Qdrant
- Bedrock daily quota rule
- Per-minute token quota multiplied by 1,440
- Cross-Region inference
- Roughly doubles effective throughput
- Own product experience
- Vedant AI receptionist, Suraksha fraud protection
- Enquiries
- nitesh@redcubical.com
Architecture
How a production RAG system is actually built
Retrieval quality determines answer quality. Almost every disappointing RAG deployment we have been asked to fix had a retrieval problem being blamed on the model.
The five stages of RAG
A production RAG pipeline has five stages that each need tuning: chunking that respects document structure, hybrid retrieval combining BM25 keyword search with vector similarity, reranking with a cross-encoder to reorder the candidate set, context assembly within a token budget, and generation with citation. Evaluation runs across all five, because a change in chunking invalidates conclusions drawn about the prompt.
-
Chunking that respects the document
Fixed-size windows split tables and clauses down the middle. We chunk on structural boundaries — headings, clauses, list items — with a modest overlap, typically 400 to 800 tokens with 10 to 15 percent overlap. Each chunk carries the document title and heading path as a prefix so it remains interpretable in isolation. Tables are extracted separately, because a table fragment retrieved without its header is worse than no result.
-
Hybrid retrieval: BM25 plus vector
Vector search finds semantic matches but fails on exact identifiers, product codes and rare terminology. BM25 handles those precisely but misses paraphrase. We run both and fuse the result sets with reciprocal rank fusion. On our own internal benchmarks hybrid retrieval lifts recall at rank 10 by 12 to 25 percent over dense retrieval alone, and the gain is largest exactly where users notice: specific-identifier queries.
-
Reranking with a cross-encoder
Retrieve 40 to 60 candidates cheaply, then rerank with a cross-encoder that scores each candidate against the actual query and keep the top five to eight. Reranking is the highest-return cheap improvement available, and it adds 60 to 250 milliseconds. It also allows a smaller final context, which reduces token cost and improves the model's attention to what matters.
-
Context assembly under a token budget
A fixed token budget per request, allocated deliberately across system prompt, retrieved context, conversation history and the reserved output. Ordering matters: models attend more reliably to the beginning and end of a long context. History is summarised rather than truncated. The budget is enforced in code, so a long document cannot silently push cost or latency out of range.
-
Generation with citation and abstention
The prompt requires the model to cite the chunk identifiers it used and to say it does not know when retrieval returned nothing relevant. Citations are validated against the retrieved set before the response reaches the user, so a fabricated citation is caught rather than displayed. An abstention rate that is too low is a warning sign, not a success metric.
Decision
Agent workflows, and when not to use one
When an agent is the right shape
Use an agent when the sequence of steps genuinely depends on intermediate results and the tool space is too large to hard-code. Do not use one when the workflow is deterministic, when errors are costly and hard to reverse, when latency budgets are tight, or when cost per request must be predictable. An agent trades control for flexibility, and most business processes want control.
Use an agent when
- The path is genuinely dynamic. A support triage that may need to check an order, a shipment, a payment or a policy depending on what it finds.
- The tool set is large. Twenty or more tools where enumerating every valid sequence is impractical.
- Iteration improves the answer. Research or reconciliation tasks where a second retrieval pass, informed by the first, materially helps.
- Latency tolerance is generous. Background or asynchronous work where ten to sixty seconds is acceptable.
- Actions are reversible or gated. Anything irreversible sits behind a human approval step.
Do not use an agent when
- The steps are known. If you can draw the flowchart, build the flowchart. Step Functions plus two model calls beats an agent on cost, latency and debuggability.
- Cost must be predictable. Agent loops have unbounded token consumption without hard iteration caps, and a runaway loop is a real invoice.
- Latency budget is under two seconds. Multi-turn tool use will not fit.
- Failures are expensive. Financial postings, irreversible communications and anything with regulatory consequence.
- Reproducibility matters. Non-determinism makes support and audit substantially harder.
Our rule of thumb: start with a fixed pipeline. Move to an agent when the pipeline has accumulated enough conditional branches that it has become an agent anyway, only worse.
- Hard iteration cap on every agent loop, plus a token ceiling per invocation and a wall-clock timeout
- Tool schemas with strict validation so a malformed call fails fast rather than producing a plausible but wrong action
- Idempotency keys on every tool with a side effect, because agents retry
- Human approval gates before any irreversible action, with the reasoning shown to the approver
- Full trace of every step — thought, tool call, arguments, result — retained for debugging and audit
- Least-privilege credentials per tool, scoped to the calling user rather than to a shared service account
Cost control
Model tiering and routing
The single largest cost lever in applied LLM work. Sending every request to a frontier model is the equivalent of running every query against your largest database instance.
How model routing works
We classify each task, assign it the cheapest model tier that passes the evaluation suite, and route requests accordingly at runtime. Simple classification and extraction go to small models. Reasoning and drafting go to mid tier. Only genuinely hard synthesis reaches frontier models. Across our engagements this typically reduces total inference spend by 55 to 75 percent against a single-frontier-model baseline with no measurable quality loss on the routed classes.
| Task class | Model tier | Typical tokens in / out | Indicative cost per 1,000 requests | Routing rule |
|---|---|---|---|---|
| Intent and sentiment classification | Small (Nova Micro, Haiku-class, Gemini Flash-Lite) | 400 / 20 | USD 0.15 – 0.60 | Always small tier. Escalate only when confidence is below threshold |
| Structured extraction from short text | Small | 900 / 180 | USD 0.50 – 1.80 | Small tier with strict schema validation; retry once on mid tier if validation fails |
| Semantic search query rewriting | Small | 250 / 60 | USD 0.10 – 0.45 | Always small tier. Latency-sensitive, sits on the critical path |
| RAG answer over 5 – 8 retrieved chunks | Mid (Sonnet-class, GPT-4o-class, Gemini Pro) | 4,500 / 400 | USD 12 – 22 | Mid tier default. Prompt caching on the system prompt and stable context |
| Long document summarisation | Mid, with map-reduce chunking | 30,000 / 1,200 | USD 60 – 130 | Mid tier plus batch inference where the request is not interactive |
| Code generation and review | Mid to frontier depending on repository size | 9,000 / 1,500 | USD 45 – 190 | Mid tier first; escalate to frontier when the diff spans multiple modules |
| Multi-step agent with tool use | Mid for tool selection, frontier for final synthesis | 18,000 / 2,500 across turns | USD 90 – 400 | Split-tier within one workflow. Iteration cap enforced |
| High-stakes reasoning and legal or clinical drafting | Frontier (Opus-class, o-series, Gemini Ultra-class) | 12,000 / 2,000 | USD 220 – 650 | Frontier only, always with human review before the output is used |
| Embedding generation for indexing | Embedding model, not a chat model | 600 / n/a | USD 0.02 – 0.12 | Batch, cached by content hash so unchanged documents are never re-embedded |
Indicative figures based on published per-token pricing across Bedrock, Azure OpenAI and Vertex AI at the time of writing, rounded to reflect real-world token variance. Provider pricing changes frequently, so treat these as ratios between tiers rather than as a quotation. What matters is the shape: roughly two orders of magnitude between the cheapest and most expensive class.
Platforms & capacity
Amazon Bedrock, Azure OpenAI or Vertex AI, and how to plan throughput on them
| Dimension | Amazon Bedrock | Azure OpenAI Service | Google Vertex AI |
|---|---|---|---|
| Model choice | Widest: Anthropic, Meta, Mistral, Cohere, AI21, Amazon Nova, plus imported custom models | OpenAI family only, plus a small models-as-a-service catalogue | Gemini family plus Model Garden third-party and open models |
| Throughput model | On-demand with per-model RPM and TPM quotas, plus Provisioned Throughput for guaranteed capacity | Per-deployment TPM allocation, plus Provisioned Throughput Units | Per-project and per-region quotas, plus provisioned throughput |
| Cross-region capacity | Cross-Region inference profiles distribute calls across Regions, roughly doubling effective throughput | Global and data-zone deployments available for some models | Multi-region endpoints available on selected models |
| Batch inference | Yes, at roughly 50 percent of on-demand price on supported models | Batch API at a substantial discount | Batch prediction at a discount |
| Prompt caching | Supported on selected models, large reduction on cached input tokens | Supported on selected models, automatic on eligible prefixes | Context caching, billed on storage duration |
| Built-in guardrails | Bedrock Guardrails with content filters, denied topics, word policy and PII handling, applied independently of the model | Azure AI Content Safety, integrated with the service | Vertex AI safety filters plus configurable thresholds |
| Data residency | Region-scoped, though cross-Region inference profiles widen the processing footprint and that must be disclosed in your data mapping | Region and data-zone options, with EU data boundary commitments | Region-scoped with residency commitments on selected services |
| Quota increases | Requested through AWS Service Quotas with a usage justification | Requested through an Azure support or capacity request | Requested through Google Cloud quota console |
| We recommend it when | You want model optionality without changing integration, you are already on AWS, or you need Guardrails as a separate control layer | You are committed to OpenAI models and already Microsoft-centred, or an enterprise agreement makes the commercials compelling | You are on Google Cloud, or Gemini's long-context and multimodal behaviour fits the task |
We build behind a provider abstraction so a model or provider can be swapped without rewriting the application. That abstraction is deliberately thin: it covers invocation, streaming, token accounting and error handling, and it does not attempt to hide provider-specific features such as Bedrock Guardrails. Over-abstracting to the lowest common denominator costs you the features you are paying for.
Throughput planning: TPM, RPM and daily token quotas
The most common production incident in LLM features is throttling under a load nobody modelled. It is entirely preventable arithmetic.
What TPM, RPM and TPD mean
TPM is tokens per minute, counting input and output together. RPM is requests per minute. TPD is tokens per day. Providers throttle on whichever limit you reach first. On Amazon Bedrock the default daily token quota for a model equals its per-minute token quota multiplied by 1,440, the number of minutes in a day, which means sustained peak traffic will exhaust the daily ceiling well before the day ends.
How we size capacity
- Measure tokens per request, per task class. Not an estimate. Instrument the prototype and record the distribution, because p95 token count is often double the mean once real documents arrive.
- Model peak requests per minute from real traffic shape, not daily average divided by 1,440. Support workloads peak hard on Monday mornings.
- Compute required TPM as peak RPM multiplied by p95 tokens per request, input plus output.
- Check the daily ceiling. On Bedrock, per-minute quota times 1,440 gives the daily allowance. If your sustained load is anywhere near peak for several hours, the daily ceiling binds before the minute one does.
- Check RPM separately. High-volume small-token workloads such as classification hit request limits long before token limits.
- Add headroom. We plan for 40 percent above modelled peak. Retries consume quota, and a throttling event that triggers retries consumes more of the quota that caused it.
How we increase effective throughput
- Cross-Region inference profiles. On Bedrock, these distribute invocations across multiple AWS Regions. Because quota is applied per Region, this can roughly double effective throughput without any quota request. Note that it widens where data is processed, so it must be reflected in your data mapping and privacy documentation.
- Quota increases via AWS Service Quotas. The only supported route. Submit a justified projection: expected RPM, TPM, growth curve and business context. Allow time; these are not instant.
- Provisioned Throughput for predictable high-volume workloads. You buy guaranteed capacity at a committed rate, which stops being expensive once utilisation is consistently high.
- Reduce demand rather than raise supply. Prompt caching, tighter contexts after reranking, routing simple classes to small models, and moving non-interactive work to batch inference. Cutting context length is often faster than getting a quota raised.
- Queue and shed deliberately. Non-interactive work goes through a queue with a concurrency limit set below your quota, so bursts are smoothed instead of throttled. Client-side rate limiting prevents your own retries from making an incident worse.
- Fallback tier on throttle. When the primary model throttles, route to a secondary model or Region rather than returning an error. Log every fallback, because a rising fallback rate is a capacity signal.
Quality
Evaluation harnesses and regression suites
How we know quality has not regressed
Every LLM feature ships with an evaluation suite that runs in the pipeline: a held-out set of real inputs with expected outputs, graded on metrics appropriate to the task. Retrieval is scored separately from generation so you know which stage regressed. Prompt changes, model version changes and chunking changes all fail the build if quality drops below the agreed floor.
| Stage or task | Metric | What it catches | Typical acceptance floor |
|---|---|---|---|
| Retrieval | Recall at k, mean reciprocal rank | Chunking and embedding regressions, index staleness | Recall at 10 above 0.90 on the golden set |
| Reranking | nDCG at 5 | Reranker model changes, candidate set too small | nDCG at 5 above 0.80 |
| Structured extraction | Field-level exact match and F1 | Schema drift, prompt changes altering output format | Above 0.95 on required fields |
| Classification | Macro F1 plus per-class confusion | Class imbalance, degradation on rare classes | Macro F1 above 0.88 |
| Grounded generation | Faithfulness to retrieved context, citation validity | Hallucination, fabricated or mismatched citations | Faithfulness above 0.95, citation validity 1.00 |
| Open generation | Rubric-based model-graded score with a human-labelled calibration subset | Tone, completeness and instruction-following regressions | Above 4.0 on a 5-point rubric |
| Abstention behaviour | False-answer rate on deliberately unanswerable questions | A model that will not say it does not know | Below 5 percent false answers |
| Safety and injection resistance | Pass rate on an adversarial prompt set | Guardrail regressions, prompt injection via retrieved content | 100 percent on the blocking set |
| Cost and latency | p50 and p95 tokens and milliseconds per request | Silent cost growth from prompt or context changes | Within 15 percent of the agreed budget |
Model-graded evaluation is calibrated against a human-labelled subset, and we report the agreement rate. An automated grader that has not been calibrated against human judgement is measuring its own preferences, not your quality.
Controls
Guardrails, PII redaction and audit logging
Input controls
- Personal data detection and redaction before the request leaves your boundary, with tokenisation so values can be restored locally afterwards
- Prompt injection pattern detection, applied to retrieved content as well as to user input
- Retrieval scoped to the calling user's permissions, so the index cannot become a way around your access controls
- Input length and token caps enforced before invocation
- Per-user and per-tenant rate limits
Output controls
- Content filtering through Bedrock Guardrails or the platform equivalent, configured independently of the model so a model swap does not reset your policy
- Citation validation against the retrieved set, blocking fabricated references
- Schema validation on structured output, with one retry then a deterministic fallback
- Denied-topic enforcement for subjects the system must refuse
- Secondary personal-data scan on output, since a model can echo data present in retrieved context
Audit and traceability
- Every invocation logged: prompt hash, retrieved chunk identifiers, model and version, token counts, latency, cost, guardrail decisions
- Correlation identifier linking the interaction to the application request and the user session
- Retention set to your policy, with personal data tokenised so erasure requests can be honoured
- Model and prompt version recorded with every response, so any past answer can be reproduced
- Complete agent traces, including tool calls and their arguments
Honest section
Where we would not use an LLM
We have talked clients out of AI features. These are the cases where a language model is the wrong tool and something older is better.
| Situation | Why an LLM is wrong | What to use instead |
|---|---|---|
| Arithmetic, totals and financial calculation | Token prediction is not computation. Accuracy is high but not guaranteed, and silent errors in money are unacceptable | Deterministic code. Let the model choose the calculation, never perform it |
| Deterministic business rules | Encoding a rules table in a prompt makes it non-deterministic, unauditable and more expensive to run | A rules engine or plain conditional logic, version controlled and tested |
| Structured data lookup | A query with a known shape does not need natural language interpretation, and RAG over rows loses precision | SQL against your database, exposed as a tool the model may call |
| Regulated decisions requiring explainability | Credit, insurance underwriting and clinical decisions typically require reasoning that can be reproduced and defended | Interpretable models with documented features, and human decision-makers |
| High-volume classification with abundant labelled data | A fine-tuned small classifier is faster, cheaper by orders of magnitude, and usually more accurate on a narrow task | Gradient-boosted trees or a compact fine-tuned transformer |
| Sub-100-millisecond latency requirements | Even the fastest hosted models rarely fit inside that budget once network time is counted | Cached results, precomputed values, or a small local model |
| Anything requiring a guaranteed identical answer every time | Sampling is probabilistic. Temperature zero reduces variance but does not remove it, and model versions change behaviour | Deterministic code, or a model call whose result is cached and version-pinned |
| Content nobody was going to read anyway | Generating volume for its own sake creates a maintenance and trust liability, not value | Write less. This is a product decision, not an engineering one |
Answers
AI, ML and LLM engineering questions
What is retrieval-augmented generation and when do we need it?
RAG retrieves relevant passages from your own content and supplies them to a language model as context, so answers are grounded in your data rather than the model's training. Use it when answers must reflect information the model has never seen, when content changes frequently, or when you need to cite a source. If the task depends only on general reasoning over text the user already supplied, RAG adds latency and cost for nothing.
How much does an LLM feature cost to run?
It depends on model tier and token volume, and the honest answer is that we instrument it rather than estimate it. A simple classification at scale on a small model can run under USD 0.30 per thousand requests. A multi-step agent on a frontier model with large context can exceed USD 40 per thousand. Routing the same workload across tiers by task class typically cuts total spend 55 to 75 percent against a single-frontier-model baseline, and that is the work that pays for itself fastest.
What are TPM, RPM and TPD quotas?
TPM is tokens per minute, RPM is requests per minute, TPD is tokens per day. Providers throttle on whichever you hit first. TPM counts input and output tokens together, so a long RAG context consumes quota even when the answer is short. On Amazon Bedrock the default daily token quota for a model equals the per-minute token quota multiplied by 1,440, the number of minutes in a day. That means sustained peak usage will exhaust a daily ceiling, and capacity planning has to be done against both windows.
How do we increase throughput if we hit Bedrock quotas?
Three levers. First, use a cross-Region inference profile, which distributes calls across multiple AWS Regions and can roughly double effective throughput because quota applies per Region. Second, request a quota increase through AWS Service Quotas, which is the only supported route and needs a justified usage projection. Third, reduce demand: prompt caching, shorter contexts, routing simpler tasks to smaller models, and moving non-interactive work to batch inference.
When should we not use an agent?
When the workflow is deterministic. If you can express the steps as a state machine, write the state machine. Agents introduce non-determinism, unpredictable token consumption and failure modes that are hard to reproduce. We use agents where the sequence of steps genuinely depends on intermediate results and the tool set is large enough that hard-coding every path is impractical. Most tasks labelled as needing an agent are better served by a fixed pipeline with one or two model calls in it.
How do you evaluate whether an LLM feature is good enough?
A held-out set of real inputs with expected outputs, graded automatically on task-appropriate metrics: retrieval hit rate and mean reciprocal rank for RAG, exact match or F1 for extraction, and rubric-based model-graded scoring for open generation with a human-labelled calibration subset. The suite runs in the pipeline so a prompt change, a model version change or a chunking change cannot silently degrade quality. Without this you are shipping on impressions.
How do you stop the model leaking personal or confidential data?
Layered controls. Detect and redact personal data before it reaches the provider, using pattern and named-entity detection with tokenisation so results can be re-hydrated locally. Apply input and output guardrails for prohibited content and prompt injection patterns. Scope retrieval to the calling user's permissions so the index cannot become a bypass for your access controls. Log every prompt, retrieval set and response for audit, with retention aligned to your policy. Note that redaction is high-recall, not perfect, so it is one layer among several rather than a guarantee.
Can you fine-tune a model for us?
We can, and we usually advise against starting there. Prompt engineering, better retrieval and few-shot examples resolve most quality gaps at a fraction of the cost, and they survive model upgrades. Fine-tuning earns its place for consistent output format, a specialised domain vocabulary, or distilling a frontier model's behaviour into a cheaper one at high volume. It creates a maintenance obligation: every base model upgrade means retraining and re-evaluating.
Start with an AI cost and quality review
Two weeks, fixed fee. We measure your current cost per request, build an evaluation baseline, model a routing strategy with projected savings, and audit your quota headroom against real traffic. You keep the evaluation harness either way.