Industries / Retail & e-commerce
Commerce systems that hold up on the busiest hour of the year
We engineer the transactional side of retail: order management and distributed orchestration, available-to-promise inventory across channels, marketplace and ERP integration, headless storefronts, payments and fraud, returns, search relevance and product data quality. Designed against peak load, because in retail a system is judged on one hour in November rather than on a Tuesday in March.
- Oversell prevented by reservation, not by hopeful stock reads
- Peak readiness measured against the highest observed minute, not the daily average
- A load-test programme with named failure thresholds before the event
- No replatform cutover inside a peak quarter, whatever the pressure
At a glance
- Architectures
- Composable and headless, or monolithic where it is the right answer
- Order orchestration
- Sourcing, split shipment, ship-from-store, drop-ship
- Inventory
- ATP service with reservations and safety stock
- Channels
- Marketplaces, retail media, ERP, WMS and 3PL
- Peak engineering
- k6 and JMeter profiles, game days, freeze windows
- Availability target
- 99.95% on supported architectures
- Enquiries
- nitesh@redcubical.com
Decision
Composable commerce or a monolithic platform?
How to choose a commerce architecture
Default to the platform. Move to composable when you can name the constraint: multiple distinct front ends, an ERP or OMS that must own order and inventory truth, catalogue or traffic beyond what the platform holds, or a release cadence the platform blocks. Composable buys control and costs you a front end, an integration layer and a platform team you did not previously need.
| Factor | Monolithic platform (Shopify, Magento, Salesforce Commerce) | Composable or headless (platform back end, custom front end, MACH services) | Fully custom commerce |
|---|---|---|---|
| Time to first revenue | Weeks. Themes, apps and hosted checkout are already there | Three to six months before parity with what the platform gave you free | Nine months and up. Rarely justified |
| Where cost sits | Licence and revenue share, plus app subscriptions that accumulate quietly | Engineering salaries and cloud, plus several vendor subscriptions across CMS, search and PIM | Engineering only, and all of it forever |
| Front-end freedom | Constrained by theme model. Core Web Vitals often fought rather than designed | Complete. Next.js or similar with rendering strategy chosen per page type | Complete |
| Checkout ownership | Platform-owned, which keeps PCI scope small and conversion well optimised | Usually still platform or provider-hosted. Building your own checkout is the decision to resist longest | Yours, along with PCI scope and every regional payment nuance |
| Integration depth | App marketplace covers common cases and hits limits on bespoke ERP logic | Strong. An integration layer you control, with an anti-corruption boundary per system | Total, and total maintenance |
| B2B complexity: contract pricing, approval flows, account hierarchies | Possible with extensions, often awkward beyond two levels | Good fit. This is a common genuine driver for going composable | Only for genuinely unusual models |
| Peak behaviour | Vendor absorbs infrastructure scaling. Your apps and third-party calls are still your problem | Yours to engineer, which is both the cost and the point | Entirely yours |
| Team required to operate it | One to three engineers plus agency support | Six to twelve engineers as a realistic floor, including platform and front end | Fifteen and up |
| Our recommendation | Right for most retailers under roughly 40 million in online revenue with a single brand | Right for multi-brand, multi-region, B2B-heavy, or where an existing ERP and OMS own the truth | Right when your commerce model is genuinely not commerce as sold by platforms |
A pattern worth naming: retailers frequently go composable to fix page speed, then discover the bottleneck was third-party tags and an unoptimised image pipeline. Both are fixable on the platform for a fraction of a replatform. We check that before recommending the larger move, even though the larger move is the bigger engagement for us.
Orders
Order management and distributed order orchestration
What an OMS does and when you need one
An order management system is a state machine that decides where each line is fulfilled from, holds the truth across channels, and survives partial failure. Distributed orchestration adds sourcing across warehouses, stores and drop-ship vendors with split shipments. It is worth building when you have multiple fulfilment nodes, and pure overhead when you have one.
The sourcing decision
For each line, the orchestrator picks a fulfilment node against weighted objectives that pull against each other:
- Promise date achievability given the node's cut-off time, carrier collection schedule and transit time to the delivery postcode.
- Cost to serve: pick and pack cost at that node plus carrier rate for the zone and weight break.
- Shipment consolidation, because two parcels usually cost more than one and customers dislike split deliveries more than a day of extra wait.
- Inventory health: draining a store that is about to trade rather than a distribution centre with depth is a false saving.
- Node capacity: units per hour a site can actually pick today, which is the constraint that matters during peak and which most sourcing rules ignore.
Sourcing rules must be configuration owned by operations, versioned, and simulatable against yesterday's orders. A rule change that nobody can test against real order shapes is a change nobody will dare make in November.
State, failure and the awkward cases
- Explicit state machine per order and per line. Orders are not a status column. Allocation, reservation, release, pick, pack, ship, deliver, return each have their own transitions and their own failure states.
- Idempotent inbound everything. Channels resend orders, WMS confirmations arrive twice, payment webhooks retry. Deduplicate on channel order reference plus line, always.
- Partial cancellation and post-purchase edits are where naive models break. Cancel one line of a three-line order after one has shipped, and pricing, promotions, tax and refund all need recalculating on the remainder.
- Promotion re-proration on partial refund. A basket-level discount spread across lines must unwind correctly, or your refunds leak margin quietly for a year.
- Compensating actions rather than rollbacks. Once a warehouse has picked, you cannot undo it. You unpick, restock and post an adjustment.
- Reconciliation against the WMS and the channel daily, reporting orders present in one and absent in the other as a monitored metric.
Inventory
Available-to-promise, safety stock and preventing oversell
How oversell actually happens, and the fix
Available to promise is on-hand minus reserved minus safety stock, plus inbound receipts arriving before the promise date. The reason retailers oversell is not a wrong formula, it is that channels read availability instead of reserving it. One availability service issuing short-lived reservations with expiry, decremented at basket or checkout rather than at fulfilment, removes the class of bug entirely.
| Component | Definition | Common failure | What we implement |
|---|---|---|---|
| On-hand | Physical units the WMS believes are at a node | Drift against physical reality between cycle counts, worst on high-velocity lines | Cycle-count cadence weighted by velocity and value, plus variance reporting per node rather than one blended accuracy figure |
| Reserved | Units committed to accepted orders but not yet picked | Reservations created only at fulfilment, leaving a window where two channels both sell the last unit | Reservation at basket or checkout with a short expiry, released automatically on abandonment or payment failure |
| Safety stock | A buffer withheld from sale to absorb count error and demand spikes | One global percentage applied to everything, which starves fast lines and hoards slow ones | Per-node, per-line buffers derived from count accuracy and demand variability, reviewed on a schedule rather than set once |
| Inbound and in-transit | Purchase orders and transfers arriving before the promise date | Counted as sellable at the supplier promise date, which suppliers miss routinely | Confidence-weighted by supplier on-time history, and excluded from same-day and next-day promises |
| Channel allocation | Stock ring-fenced for a marketplace or a specific channel | Static allocations left in place after demand moves, stranding sellable stock | Soft allocation with automatic release when a channel underperforms against forecast |
| Publish latency | Time from a stock change to the channel showing it | Batch feeds every fifteen or thirty minutes, which during a sale is an eternity | Event-driven publication with a throttle, plus a full reconciliation sweep to correct any missed delta |
Two operational rules. First, one service owns availability and every channel asks it; the moment two systems each calculate availability, they will disagree and the disagreement will be discovered by a customer. Second, publish a slightly conservative figure during peak. Losing a marginal sale costs far less than a cancellation email, a refund and a review.
- Single availability service as the only source of free-to-sell, queried by web, app, marketplace feeds and the contact centre alike
- Reservations with expiry created transactionally on basket or checkout, released on abandonment, payment failure or timeout
- Idempotent stock movement events keyed on movement identifier so a replayed WMS feed cannot double-decrement
- Oversell as a monitored metric with an alert threshold, not a fact discovered from customer service volume
- Reconciliation sweep against the WMS at least hourly during peak, reporting variance by node and by line
- Pre-order and backorder as distinct states with their own promise logic rather than negative stock
- Store stock treated as lower confidence than distribution centre stock, with a larger buffer, because shop-floor accuracy is genuinely worse
- Kit and bundle availability derived from components at read time, never maintained as an independent number
Peak
Peak-event capacity engineering for Black Friday and festive sales
Peak is a minute-shaped problem. An email send or a festive drop can push an hour of normal traffic through in ninety seconds, and daily averages conceal it completely.
How we size and validate for peak
We size against the highest observed minute of the last comparable peak, multiply by the growth forecast, add 50 percent, then validate at twice that number. Then we test the shared stateful components in the order they actually fail: database connections, hot inventory rows, cache stampede, third-party rate limits, search cluster. The web tier is rarely the problem.
| Order of failure | Component | Symptom you see | Mitigation we implement |
|---|---|---|---|
| 1 | Database connection pool | Latency climbs across every endpoint at once and errors look random. The application appears healthy in isolation | Connection pooling through PgBouncer or RDS Proxy, pool sizing tested rather than guessed, read replicas for catalogue and search, and hard statement timeouts so one slow query cannot hold a connection |
| 2 | Hot rows and lock contention on inventory | Checkout latency spikes on the exact lines in the promotion while everything else looks fine | Reservation records rather than an updated counter, sharded or queued decrement for extreme lines, and short transactions that never span an external call |
| 3 | Cache stampede after an eviction or deploy | A cliff-edge collapse minutes after a release, or a sudden origin surge with no traffic change | Request coalescing, jittered time-to-live, stale-while-revalidate, and a pre-warmed cache as a documented step in the release runbook |
| 4 | Third-party rate limits | Payments, tax, address validation or fraud scoring start failing while your own metrics look clean | Negotiated peak limits in writing beforehand, circuit breakers with defined degraded behaviour, queued retries with jitter, and a decision agreed in advance on whether to accept an order when tax or fraud is unavailable |
| 5 | Search and faceting cluster | Category and search pages slow while product pages stay fast | Query result caching, facet cardinality limits, replica scaling ahead of the event, and a simplified fallback ranking path |
| 6 | Asynchronous worker backlog | Orders accepted but confirmation emails, WMS drops and ERP posts fall hours behind | Queue depth alerting with per-queue thresholds, priority separation so order release outranks marketing events, and pre-scaled worker capacity rather than reactive autoscaling |
| 7 | Observability and logging pipeline | Dashboards go blind exactly when you need them, because log volume tripled | Sampling policy set before peak, cardinality limits on labels, and a minimal always-on dashboard that survives an ingestion backlog |
| 8 | Web and application tier | Genuinely last, because it scales horizontally | Pre-scaled baseline rather than reactive autoscaling, since autoscaling reacts in minutes and a drop arrives in seconds |
Autoscaling is not a peak strategy. It reacts on a minutes-long horizon while a drop lands in seconds, so we pre-scale to the tested figure ahead of the window and treat autoscaling as a safety net above it. The cost of running over-provisioned for four days is trivially less than the cost of a failed peak.
-
Baseline from last peak, at minute granularity
Pull the highest single minute of requests, orders, basket adds and inventory reads from the previous comparable event, plus the traffic shape around it. Averages are useless here. If last year is not instrumented at that granularity, fixing the instrumentation is the first task.
-
Model the target and agree it in writing
Peak minute multiplied by growth forecast plus 50 percent becomes the target. Twice the target becomes the break-point test. Both numbers are signed off by commercial and engineering so nobody renegotiates the target after a failed test.
-
Load test on production-shaped data
k6 or JMeter profiles for browse, search, add-to-basket, checkout and order release in realistic ratios, against a full-size catalogue and full-size order history. Testing against ten thousand products tells you nothing about a two-million-product catalogue.
-
Find the break point deliberately
Push past the target until something fails, record what failed and at what number, fix it, and repeat. A test that passes tells you very little. A test that identifies the next bottleneck and its threshold tells you where you stand.
-
Third-party limits confirmed in writing
Payment provider, tax engine, fraud service, address validation, carrier rate APIs and marketplace feeds. Each one asked for its peak ceiling and its behaviour at that ceiling. Then a degraded-mode decision per dependency, agreed with commercial.
-
Code freeze and a game day
Structural change freezes six to eight weeks out; content and pricing changes continue. Then a rehearsed failure exercise: kill the search cluster, throttle the payment provider, exhaust the connection pool, and watch the team respond against the runbook.
-
Peak-day operating model
Pre-scaled infrastructure, a named incident commander per shift, a war-room channel with commercial present, a minimal dashboard of order rate and checkout success, and a pre-agreed list of features that can be switched off to protect checkout.
Money and goods back
Payments, fraud, returns and reverse logistics
Payments and fraud
- Tokenised throughout. Card data never reaches your servers, keeping PCI scope minimal and the vault problem with the provider.
- Authorisation and capture as explicit states. Capture at dispatch rather than at order, with authorisation expiry handled before it silently voids a peak order.
- Local payment methods per market, because coverage moves conversion more than checkout styling does. UPI in India, iDEAL in the Netherlands, Klarna and similar where deferred payment is expected.
- SCA and 3-D Secure exemption logic applied deliberately, since blanket challenge costs conversion and blanket exemption costs liability.
- False declines measured alongside fraud. The cost of rejecting good customers usually exceeds fraud loss and is almost never on a dashboard. We report approval rate, chargeback rate and manual-review volume together.
- Chargeback workflow with evidence assembly, deadline tracking and a win-rate metric, rather than an inbox and goodwill.
Returns and reverse logistics
- Returns are a supply chain, not a refund form. Authorisation, label generation, carrier collection, receipt at a returns node, grading, disposition and restock all need modelling.
- Grading drives disposition: resell as new, resell as open-box, refurbish, liquidate, recycle. Each has a different value recovery and a different accounting treatment.
- Restock timing is a real availability input. Units in a returns queue are neither sold nor sellable, and pretending otherwise distorts availability during peak returns season.
- Refund and promotion unwinding handled correctly on partial returns, including re-proration of basket-level discounts and correct tax reversal per line.
- Serial return behaviour identified and handled by policy rather than by an ad hoc decision at the desk.
- Reconciliation between refunds issued and goods received, because refund-before-receipt policies leak measurably and quietly.
Search relevance, personalisation and product data quality
Search and personalisation both fail for the same underlying reason: the product data is not good enough to rank or match on. Attribute completeness is the ceiling on every downstream capability, which is why product information management is a commerce concern rather than a content one.
- Search relevance is an evaluation problem. We build a judgement set from real queries with rated results before touching ranking, so a change can be shown to be an improvement rather than asserted to be one. Zero-result and low-click queries are a weekly report, not an annual audit.
- Synonyms, spelling and query understanding derived from your own query logs rather than a generic dictionary, because retail vocabulary is category-specific and often regional.
- Faceting depends on attribute completeness. A colour facet across a catalogue where 40 percent of lines have no colour attribute is worse than no facet, because it hides valid products.
- Personalisation ranked by expected value. Recency and category affinity deliver most of the achievable lift; deep behavioural modelling adds a fraction more for considerably more engineering, and we say so before quoting for it.
- PIM earns its place through workflow. Attribute governance, completeness scoring, enrichment tasks by owner, approval before publication, and per-channel and per-locale variants. If one team edits attributes for one channel, you do not need it yet.
- Supplier data is the input problem. Onboarding suppliers to a required attribute schema, with validation at ingest, prevents the enrichment backlog that otherwise grows faster than the team clearing it.
Answers
Retail and e-commerce engineering questions
Should we go headless or stay on a monolithic commerce platform?
Stay monolithic unless you can name the constraint that forces you off it. Headless earns its cost when you need several distinct front ends, deep integration into an existing ERP or OMS, or catalogue and traffic volumes the platform cannot hold. It costs you a front end you now own, an integration layer you now operate, and features that used to arrive free in a platform upgrade.
Why do we oversell during sales even though stock looks correct?
Because availability was read rather than reserved. If two channels read the same free-to-sell figure milliseconds apart, both accept the order. The fix is a single availability service that issues short-lived reservations with expiry as part of the add-to-basket or checkout transaction, so the decrement happens before payment rather than after fulfilment.
What actually breaks first under peak load?
Almost never the web tier, because that scales horizontally. It is the shared stateful things: database connection pool exhaustion, a hot row in inventory, cache stampede after an eviction, third-party rate limits on payment or tax services, and search cluster saturation. We test in that order because that is the order they fail in.
How much peak headroom should we plan for?
We size to the highest observed minute of the last comparable peak multiplied by the growth forecast, then add 50 percent, and we validate at twice that figure. Peaks are minute-shaped rather than day-shaped: a festive drop or an email send can put a full hour of normal traffic through in ninety seconds, and averages hide it entirely.
Can you replatform us before Black Friday?
We will not. Our rule is a hard code freeze on anything structural six to eight weeks before a known peak, and no replatform cutover inside the peak quarter. If a September conversation is aiming at a November cutover, the honest answer is January, and we would rather lose that scope than take responsibility for a failed peak.
Do we need a PIM?
If more than one system needs the same product attributes and more than one team edits them, yes, and the payback is usually visible within two quarters. Below that threshold a well-governed catalogue in the commerce platform is enough. The value of a PIM is workflow and governance over attributes, not storage.
How do you approach fraud without rejecting good customers?
Measure both sides. False-decline cost is usually larger than fraud loss and almost never reported, so we instrument approval rate, chargeback rate and manual-review volume together. Rules are transparent and versioned, scores support decisions rather than making them silently, and every declined order is reviewable with the reason recorded.
Is a distributed order management system worth it for a single warehouse?
No. Distributed orchestration pays for itself once you have multiple fulfilment nodes, ship-from-store, drop-ship vendors or split shipments. With one warehouse and one channel, the sourcing logic is trivial and an OMS adds a hop, a state machine and an operational surface for no benefit.
Start with a peak-readiness and architecture review
Three weeks, fixed fee. You get a minute-granularity load model, a break-point test with named thresholds, a third-party dependency ceiling map, an inventory and oversell assessment and a costed remediation plan. Yours to keep either way.