• A customer files an insurance claim.

• A borrower asks for loan hardship assistance.

• A patient needs medical prior authorization.

• A compliance team reviews an internal control.

• A fraud alert lands in an analyst’s queue.

• A contract needs corporate legal review.

At first glance, these sound like normal, repeatable back-office operations. But inside the workflow, things get complicated quickly.

The human reviewer must pull together fragmented data from legacy systems, read shifting policy documents, check rigid eligibility rules, interpret risk scores, identify missing documentation, follow strict communication guidelines, and leave behind a flawless audit trail. Because the final outcome is sensitive, highly regulated, and financially meaningful, the organization cannot simply say:

“Let the LLM decide.”

That is the wrong framing. The better question is:

“How can an AI agent help a human make a better, faster, more consistent, and more auditable decision?”

That is where agentic AI becomes revolutionary. Not as an autonomous decision-maker, but as a governed workflow copilot.

The Wrong Design: LLM as the Decision-Maker

The tempting design for teams rushing a proof-of-concept to production is linear:

This is an extraordinary risk. In regulated workflows, decisions must follow approved policies, immutable business rules, and legal constraints. If an LLM misses a subtle exception clause, fabricates a compliance policy, or gives inconsistent guidance across two identical cases, the issue isn’t just a “bad user experience.” It is a catastrophic compliance, legal, and financial liability.

The first principle of enterprise AI system design is absolute: The LLM must never be the system of record for a decision.

The LLM can summarize data. It can retrieve context. It can explain complexity. It can draft communication pipelines. But the governed decision must come from approved rules, structured engines, deterministic models, and accountable humans.

Subscribe now

The Better Design: Governed Decision Support

A production-grade architecture completely separates responsibilities:

• Structured Data tells us what happened.

• Policy Retrieval tells us what rules apply.

• Decision Engines tell us what is mathematically allowed.

• LLM Agents translate logic into language (summarizing, explaining, drafting).

• Humans review, validate, and sign off.

To make this architecture concrete, let’s look at the two distinct operational execution modes required to run an enterprise operations desk: Proactive Automation (The Daily Batch Trigger) and Reactive Exploration (The Analyst Chat Interface).

1. The Proactive Automation Pipeline (Daily Batch)

The batch workflow is designed to automatically handle massive queues of cases running on automated orchestrators like Apache Airflow. Instead of making the LLM read raw, unorganized databases at runtime, the data is progressively filtered through a deterministic rules engine before the generative layer ever touches it.

2. The Reactive Exploration Pipeline (Analyst Chat)

When an operations analyst needs to dynamically drill down into a specific case, audit a particular rule, or query a corporate policy manual, running a heavy, end-to-end processing pipeline is incredibly inefficient.

Instead, a Router Agent short-circuits the workflow based entirely on the analyst’s structural intent—completely bypassing unnecessary data layers to deliver instantaneous answers.

This router design is what makes the system truly “agentic” in a commercial sense. The system architecture dynamically restructures its underlying components based entirely on the task at hand. If the analyst asks: “What documents are missing for this account?”, the Router short-circuits straight to the data layer and document checker tools, completely ignoring the complex compliance or drafting models.

RAG, Rules, Models, and LLMs: Who Does What?

Decoupling logic from language requires establishing explicit boundaries for every asset in your infrastructure stack:

Because the knowledge base in regulated operations constantly shifts (e.g., state rules change, compliance definitions update, communication templates adjust), RAG is the mandatory choice over model fine-tuning for policy grounding. RAG allows the model to fetch and cite the absolute latest approved text section at runtime, leaving a clear paper trail.

A simple rule of thumb: If the knowledge changes and needs citations, use RAG. If the behavior is stable and repeated, consider fine-tuning.

Subscribe now

Production Design Choices: Deep-Dive Q&A

When moving this blueprint from a local sandbox to a production-scale enterprise data cluster, systems engineers encounter specific architectural edge cases. Here is how this design safely mitigates them.

Q1: Why use an “Stateful Event Aggregator” instead of allowing the Summary Agent to read raw data logs directly at runtime?

The Trade-off: Processing Complexity vs. Operational Overhead

In an active enterprise system, a single case file contains a massive data footprint: years of structured tabular transactions, thousands of lines of conversational notes input by human operators over time, and uploaded PDF document packages.

Dumping this raw data directly into an LLM context window at runtime causes immediate production failures: it blows out the token context window, inflates API bills exponentially, and causes massive latency spikes (often exceeding 20 seconds). Furthermore, LLMs suffer from the “Lost in the Middle” phenomenon—failing to capture crucial exceptions when buried under dense walls of text.

The Solution: The architecture introduces an asynchronous Stateful Event Aggregator. As new account events or notes occur, lightweight background scripts incrementally process the data and update a highly condensed, centralized State JSON profile document. When the batch orchestrator fires, the Summary Agent reads a compact, structured timeline rather than a disorganized data swamp, dropping token overhead, slashing latency, and guaranteeing no critical exception variables are missed.

Q2: Why isolate validation into a single, external “Dual-Pass Compliance Guardrail” block rather than handling checks natively inside individual agent prompts?

The Trade-off: Decoupled Independent Safety vs. Inline Instruction Reliance

Tasks like checking text alignment against structured inputs or blocking prohibited phrases should never be handled via inline prompt engineering. Tasking a model with both creating content and auditing its own output splits its attention layers, leaving it highly vulnerable to instruction drift, prompt injections, and subtle hallucinations.

The Solution: We completely decouple verification by placing a dedicated Dual-Pass Compliance Guardrail box entirely outside the generative layer:

• PASS 1 (The Independent Evaluator): A highly restricted model that consumes only the raw, deterministic outputs of the quantitative Decision Engine and the generated text draft. It acts as an independent judge, validating whether the textual claims exactly match the mathematical logic calculated by the underlying rules engine.

• PASS 2 (The Deterministic Phrase Blocker): A non-probabilistic software module that relies entirely on traditional, high-speed regex tables to scan for legally hazardous language or structural modifications of approved corporate templates.

Q3: What happens when a generated asset fails the Compliance Guardrail? How does the architecture prevent a human operational deadlock?

The Trade-off: Uncompromising Enterprise Safety vs. Flow Throughput

If an architectural security layer simply throws a generic hard stop error or a broken screen when a compliance check fails, it introduces an operational bottleneck. The human analyst is left hanging, forced to manually type out calculations from scratch—completely erasing the efficiency gains built by the automation layer.

The Solution: The pipeline utilizes a structured Correction Loop with a Fallback-to-Template trigger. If Pass 1 or Pass 2 flags an issue, the system automatically routes the error logs back through a correction loop, giving the agent up to two chances to self-correct the text draft.

If the generation still fails validation, the probabilistic engine terminates entirely. The pipeline instantly swaps out the generative model and falls back onto a completely static, deterministic, token-swapped Legal Template hardcoded inside the system repo. The failure is quietly logged for prompt optimization, but the analyst’s workflow never freezes—they receive a pristine, compliant asset, allowing the human-in-the-loop operation to proceed without a second of downtime.

Multi Industry Pattern: Separating Logic from Language

While a borrower hardship case serves as an excellent reference model, this identical data assembly line scales across every major corporate operation desk:

The domain changes, but the core engineering paradigm stays completely intact:

Complex Case ──► Relevant Policy ──► Governed Checks ──► LLM Explanation/Drafting ──► Human Approval ──► Audit Trail

Final Takeaway

The future of AI agents in enterprise-scale operations is not fully autonomous decisioning. It is governed, auditable, human-in-the-loop decision support. The most valuable enterprise systems built today will not ask: “Can we get the LLM to make the right decision?” They will ask:

“Can we construct an architecture where the LLM helps the human understand the case, applies the right deterministic policy, communicates flawlessly, and leaves behind an ironclad audit trail?”

That is where AI agents actually fit.

Found this post valuable? Show some love with a like ❤️

Connect with me on LinkedIn | Visit my website

In real businesses, that’s rarely the right question.

The question that actually matters is:

That distinction changes everything downstream. A model that predicts a price is useful. A system that quantifies its own uncertainty, converts that uncertainty into a decision, observes realized outcomes, and adapts future decisions — that’s valuable.

In this project, I built a mock risk-aware pricing system using public housing data. The use case mirrors an iBuyer-style real estate model, but every design decision applies equally to:

The core insight driving everything here:

The Business Problem

An iBuyer makes near-instant offers to buy homes directly from sellers. The business model is deceptively simple:

The business makes money when:

So pricing is load-bearing. Offer too low and the seller walks. Offer too high and you absorb the loss on resale.

That creates the fundamental tension every pricing team lives with:

This means the pricing system can’t only optimize for prediction accuracy. It has to manage expected value, model uncertainty, downside risk, market drift, and feedback from realized outcomes — simultaneously.

Project Objective and Architecture

How can a pricing model estimate resale value, quantify its uncertainty, make a risk-adjusted offer, and then learn from what actually happened in the market?

The system has six main components

The dataset is King County housing sales (May 2014 – May 2015, 21,613 homes) augmented with macroeconomic features: 30-year mortgage rates, Seattle Case-Shiller index, unemployment, housing supply, sentiment, housing starts, and CPI.

Click here: Github Code

Subscribe now

Why the Time Split Matters

A random train/test split would inflate performance. In real pricing systems, you never price today’s homes using tomorrow’s data — you train on history and deploy into the future.

So, I used a strict chronological split:

The market didn’t stand still across these windows. Price per square foot rose into the test period, the 30-year mortgage rate moved from 4.19% to 3.84%, and the Seattle Case-Shiller index shifted by 7.4%.

Price per square foot rises into the test period — exactly the drift a real pricing team would face.

Macro indicators shift meaningfully across the sample window.

This setup creates the exact problem pricing teams care about: a model trained on earlier sales risks becoming stale when the market moves. That tension motivates every subsequent design decision.

Step 1: From Point Estimate to Probabilistic Valuation

A standard valuation model predicts one number. For a buy-offer decision, one number isn’t enough.

Consider two homes with the same median estimate:

To capture that difference, I trained a quantile regression AVM producing Q10 (downside), Q50 (fair value), and Q90 (upside) estimates for each home. The normalized uncertainty score is:

The critical test: is this uncertainty score actually predictive of pricing error?

It is. Low-uncertainty homes had 7.71% average error. Medium-uncertainty homes: 11.66%. High-uncertainty homes: 16.55%

Wider prediction intervals reliably correspond to higher realized error. The uncertainty score is not decorative — it ranks pricing risk.

Uncertainty is now actionable. If the model is less confident, the offer should reflect that.

Subscribe now

Step 2: Calibrating Confidence (Making Uncertainty Honest)

The raw quantile model was useful but not trustworthy enough for pricing. A Q10–Q90 interval should cover roughly 80% of outcomes. The raw model covered only ~65% — it was overconfident.

Calibration addresses this by widening the interval without touching the Q50 estimate:

The factor (1.45 in this project) is selected on validation data. The key conceptual point:

Calibration doesn’t make the prediction more accurate. It makes the confidence estimate more honest.

Calibrated intervals achieve coverage much closer to the intended 80% target.

I also benchmarked width scaling against isotonic calibration. Width scaling won: it hit closer to 80% coverage, kept intervals narrower, preserved Q50, and is far easier to explain to a business audience. Isotonic calibration overshot the target and led to unnecessarily conservative offers.

Calibration methods trade off reliability, interval width, and downstream offer conservatism.

Step 3: Translating Uncertainty into a Buy Offer

With calibrated bands in hand, the buy offer formula is:

Parameters:

• base spread = 5%

• penalty strength = 0.15

• max uncertainty penalty = 10%

  This produces a natural pricing policy:

Base spread is constant; uncertainty penalty scales with model confidence. The system prices risk, not just value.

This is where the project crosses from “ML model” to “pricing system.” The model output is no longer just a prediction — it’s the direct driver of a customer-facing decision.

Step 4: Knowing When Not to Auto-Price

One of the most practically valuable parts of this project: built-in escalation logic.

A pricing system shouldn’t blindly trust the model for every asset. If the uncertainty penalty hits the 10% cap, the home is flagged as a manual-review candidate — a case where even the maximum automated discount isn’t sufficient protection.

Results from the test set:

• 567 homes hit the cap (13.1% of test set)

• Average uncertainty score for capped homes: 86.0% vs. 39.0% for below-cap homes

• Capped homes showed higher view and waterfront exposure — the model was correctly flagging unusual, harder-to-price cases

Homes hitting the cap have dramatically higher model uncertainty and are natural candidates for manual review or no-instant-offer workflows.

The pattern generalizes: in lending, this becomes manual underwriting. In insurance, specialist review. In fraud, escalation. In marketplaces, operations review.

A well-designed system doesn’t just make decisions. It knows when a decision is too uncertain to automate.

Step 5: Closing the Loop with a Sell-Side Feedback Multiplier

So far, the system manages property-level uncertainty. But there’s a second, harder problem: market-level bias.

Even a well-calibrated model can drift when the market moves. To address this, I added a lightweight feedback loop from observed resale outcomes back into future offers. The feedback rule:

Each week, the system uses only prior outcomes — no lookahead. This loop doesn’t retrain the model. It applies a fast, lightweight market-level correction on top of the existing property-level calibration. The architecture intentionally separates two concerns:

Width calibration = property-level uncertainty reliability
Market multiplier = market-level bias correction

Subscribe now

What Happened in the Test Period?

During the held-out test period (March–May 2015), the market outperformed model expectations. The rolling multiplier moved above 1.0, raising future offers. This improved valuation accuracy and offer competitiveness

This is the core business tradeoff in any adaptive pricing system.

The feedback loop doesn’t resolve that tradeoff; it makes it explicit and measurable. In production, the north-star objective would be:

The public dataset doesn’t include seller acceptance, so this project measures the risk and margin side — the conversion benefit is directionally argued but not directly measured.

Step 6: Stress Testing Downside Scenarios

The actual test period was favorable. What happens when the market weakens?

I applied downside shocks to observed test prices and reran the feedback logic using only prior-week outcomes:

At 15% downside, the feedback loop improved total net P&L proxy by +$235.5M relative to the static baseline.

The symmetry matters. That bidirectional response is what makes this a feedback system, not just a correction mechanism.

Subscribe now

How I Think About Metrics for a Pricing System

Prediction accuracy is necessary but not sufficient. I organize metrics into four layers:

Model accuracy (How wrong is the fair-value estimate?)
MAE · MAPE · RMSE · signed bias

Uncertainty reliability (Can I trust the model’s confidence?)
Q10–Q90 coverage · quantile calibration · error stratified by uncertainty bucket

Business value (Does the pricing policy create value?)
Average net P&L proxy · ROI proxy · acceptance-adjusted contribution

Guardrails (Is the system taking too much or too little risk?)
Overpay-rate proxy · negative P&L rate · manual-review rate · offer competitiveness

In this project I used average net P&L proxy as the primary business metric. In production, I’d optimize acceptance-adjusted contribution profit — the product of conversion probability and margin.

What Comes Next in a Production System

A conversion model:

The missing piece is P(seller accepts offer). Adding it would shift the objective from margin maximization to expected contribution — explicitly trading off competitiveness and risk.

Segment-level feedback:

The current multiplier is market wide. A production system should test segmented multipliers by geography, price band, property type, and liquidity — accepting the tradeoff between stability and specificity.

Granular cost modeling:

Repair cost, holding duration, financing cost, concessions, and resale expenses should each be modeled separately rather than folded into a single spread assumption.

Leading indicators:

Resale outcomes arrive late. A faster feedback signal would incorporate listing views, saves, tours, days on market, and inventory levels — all of which move before transactions close.

Continuous calibration monitoring

Coverage should be tracked by market, price band, and cohort over time — not just measured once at model deployment.

The Broader Pattern

This project started with home pricing. The underlying pattern is universal.

Any business making financial decisions from predictions — approving or rejecting, bidding or passing, discounting or holding price, buying or routing to review — faces the same problem:

A model answers what is the predicted value?

That’s the difference between a model and a system. The gap between them is where most of the business value lives.

View the code here: Github Repo Link

Found this post valuable? Show some love with a like ❤️

Connect with me on LinkedIn | Visit my website

Every RAG demo looks amazing.

You type a question → it retrieves context → gives a smart answer.

Feels magical

But the moment you try to scale it?

• Latency starts creeping in

• Costs quietly explode

• And users… ask the same thing again and again

  ---

  Subscribe now

  ---

What’s Actually Happening Under the Hood

Every single query is doing this:

Humans don’t work like this

If someone asks you:

“What’s the interest rate for a loan modification?”

…and then later:

“How does the rate change during modification?”

You don’t go:

“Hold on, let me recompute everything from scratch.”

You reuse what you already know.

That’s what Semantic Caching does!

Traditional vs Semantic Cache

Exact matching is simple — but also very limiting.

In real systems, users rarely ask the same question the same way.

So instead of matching text, we match meaning — and route the query through a semantic cache before doing any expensive work.

The “aha” moment

This layer sits before everything expensive.

Which means:

• No embedding

• No retrieval

• No LLM

If there’s a hit, you’re done in milliseconds.

But how does it know the two queries are similar?

Short answer: embeddings + similarity

Subscribe now

Is that it? Did we solve the issue? Not Yet!

This is where real system design kicks in.

1. The Threshold Problem

Too strict:

• fewer cache hits → accurate but low reuse

Too loose:

• wrong answers → fast but risky

This is not just a technical choice—it’s a product decision.

2. Context changes everything

Same query ≠ same answer

• analyst vs customer

• different products

• updated policies

  

3. Cache invalidation

What happens when:

• data changes?

• policies update?

Typical fixes:

• TTL (expiry)

• versioned embeddings

• doc-based invalidation

  ---

So, do we only cache answers?

Nope. This is where things get interesting.

Example: Natural Language to SQL

In my previous job, I built a Natural Language to SQL model which dealt with questions like:

User asks:

“What’s the delinquency rate for loans in Texas?”

Later someone asks:

“Show delinquency trends in TX”

Different wording. Same intent

Subscribe now

Multi-Level Caching

Advance systems don’t stop here. Once you add semantic caching, things evolve:

Once you add semantic caching, you stop optimizing the model — and start optimizing the system.

Query Rewriting

Users ask the same question in many different ways. Query rewriting standardizes these variations into a consistent form before hitting the cache. This reduces noise in embeddings and increases the chances of a cache hit, making semantic caching more effective.

Retrieval Caching

Instead of caching final answers, you can cache the documents retrieved from the vector database. Since retrieval is often expensive and repeated, this lets you skip that step while still allowing the LLM to generate fresh responses when needed.

Hybrid Systems

Real systems combine multiple optimizations instead of relying on just one. They use semantic caching for repeated queries, rules for known patterns, and session memory for short-term context—ensuring the system does the least amount of work possible for each request.

If you’re building RAG systems…

Ask yourself:

• How many queries are you recomputing?

• What can you reuse?

• Where are you wasting LLM calls?

You might already have the answers…

You’re just not caching them yet

Found this post valuable? Show some love with a like ❤️

Connect with me on LinkedIn | Visit my website