Razorpay AI Buildathon, COD risk decisions
Nishchay
Decides which cash-on-delivery orders to flag for return-to-origin risk, pricing each decision in the merchant's own rupees.
- ₹72.9LNet vs the merchant's existing rule on 44,288 held-out orders (~₹18.8 lakh/month)
- 0.677PR-AUC on the held-out orders, against a 0.304 no-skill baseline
- 0.986Recall on the same held-out orders
- 18,827Live agent calls used to measure the Address Agent's lift: none measurable
How it works
- SimulateAgent-based order simulator calibrated to published RTO statistics
- Features27 tabular and 9 graph features per order
- ScoreLightGBM, then isotonic calibration, then conformal intervals
- DecideFour-action cost argmin derives the threshold from merchant costs
- Enrich3 bounded agents run after scoring; they cannot delay or change decisions
- MonitorDrift detection with ADWIN, Page-Hinkley and PSI
Problem
When a cash-on-delivery order is refused, it goes back to the merchant: return to origin, or RTO. Flagging risky COD orders reduces that, but every good customer wrongly flagged costs margin too. Most rules ignore the second cost. I built Nishchay for the Razorpay AI Buildathon (Track 02, AI Risk Manager) to make the trade in the merchant’s own rupees.
Approach
The system includes an agent-based order simulator calibrated to published RTO statistics. Each order gets 27 tabular and 9 graph features. LightGBM scores it, isotonic calibration turns the score into a probability I can price, and conformal prediction adds an interval.
The decision is a four-action cost argmin. I never set a cut-off by hand: the threshold is derived from the merchant’s costs, including the margin lost by wrongly flagging good customers. CatBoost and TabPFN were run as benchmarks.
LLMs stay off the decision path. Three bounded agents enrich an order only after the score is returned. A failure-injection endpoint, ?simulate_failure=agent_timeout, shows live that a stalled agent cannot delay or change a decision.
The service runs on FastAPI and SQLAlchemy with a React + Vite + TypeScript front end. Drift monitoring uses ADWIN, Page-Hinkley and PSI, and the codebase is strict-typed (ruff + mypy strict).
Results
On 44,288 held-out orders, a 116-day window never touched during development, the policy nets ₹72.9 lakh (~₹18.8 lakh/month) against the merchant’s existing rule. PR-AUC is 0.677 against a 0.304 no-skill baseline, at 0.986 recall. The false-positive cost is reported as its own line instead of being folded into the total.
I measured the agents as well. 18,827 live agent calls put the Address Agent’s lift at +0.0038 PR-AUC, with a 95% CI of [−0.012, +0.019]. That is no measurable benefit, and the write-up says so.
To test transfer, I ran the system on 151,112 real e-commerce transactions and documented what didn’t carry over: conformal coverage was 75% against a nominal 90%.
What I learned
Pricing decisions in rupees changes the conversation. A threshold derived from costs can be explained to a merchant; a hand-picked cut-off can’t.
Putting LLMs after the score made them safe to ship and easy to evaluate. It also made it obvious when one wasn’t earning its place, and the Address Agent wasn’t.
Coverage guarantees are only as good as the data they were calibrated on. The 75% result on real transactions is the number I’d want a merchant to see before the ₹72.9 lakh one. Logging all 53 spec deviations openly, instead of rewriting the spec to match the build, kept that honesty consistent.
Limits, stated not hidden
- On 151,112 real e-commerce transactions, conformal coverage was 75% against a nominal 90%.
- The Address Agent's lift was +0.0038 PR-AUC (95% CI [−0.012, +0.019]), so no measurable benefit.
- The rupee figures are measured on held-out evaluation orders, not on live merchant traffic.
- 53 deviations from the original spec are logged in the repo.