Services / Integrate
Service architecture and integration, sized to the problem you have
We design service boundaries from your domain, connect systems with event-driven messaging that survives partial failure, and build APIs your consumers can depend on. That includes saying no to microservices when a modular monolith is the correct answer, which is more often than the industry admits.
- Modular monolith first unless you can name the constraint microservices solve
- Boundaries from event storming with your domain experts, not from the org chart
- At-least-once delivery with idempotent consumers and the outbox pattern
- Consumer-driven contract tests so a deploy cannot silently break a caller
At a glance
- Messaging
- Kafka, MSK, SQS, SNS, EventBridge, RabbitMQ, Kinesis
- API styles
- REST, GraphQL, gRPC, webhooks, async events
- Gateways
- Amazon API Gateway, Kong, Apigee, Azure API Management
- Contract testing
- Pact, OpenAPI and Protobuf schema checks
- Legacy protocols
- SOAP, WSDL, IBM MQ, SFTP flat file, ODBC
- Availability target
- 99.95% on supported architectures
- Engineers
- 40+ across 10 delivery markets
- Enquiries
- nitesh@redcubical.com
Start here
When not to do microservices
This is the most consequential decision on this page and the one most often made for the wrong reasons. Read it before the rest.
Should you do microservices
Do not adopt microservices unless you can name the specific constraint they remove. Microservices solve an organisational problem: too many engineers contending over one deployment unit. They do not make code cleaner, faster or more reliable. Below roughly thirty engineers, a modular monolith gives you the boundaries without the distributed-systems bill.
Reasons that are not good enough
- "Our codebase is a mess." Distribution does not impose design discipline. It converts a tangled codebase into a tangled network of services, where the same coupling now fails intermittently and is harder to trace.
- "We need to scale." Almost always you need to scale one component. Extract that one. A single well-tuned monolith on modern hardware handles far more traffic than most teams assume.
- "We want to use different languages." Real, but expensive. Every language adds a build chain, dependency policy, security scanning path and hiring requirement. Justify it per service, not as a principle.
- "It is the modern architecture." It is a fifteen-year-old trade-off with well-documented failure modes. The companies most associated with it also have platform teams larger than your entire engineering organisation.
Reasons that justify it
- Deployment contention you can measure. Release trains delayed by unrelated changes, and a change failure rate driven by coupling between teams rather than by defects.
- Genuinely divergent scaling profiles. A component needing GPUs, or one hundred times the throughput of the rest, or a completely different burst pattern.
- Independent availability requirements. A payment path that must remain up while a reporting subsystem is down, with that isolation contractually required.
- Regulatory or data-residency isolation. A bounded context that must run in a specific jurisdiction or under separate access control and audit.
- Organisational reality. Multiple autonomous teams with distinct roadmaps, product ownership and on-call rotations, each able to own a service properly.
| Dimension | Modular monolith | Microservices |
|---|---|---|
| Transactions | One database transaction across modules. Correctness is cheap | Distributed. Sagas, compensations and eventual consistency visible to users |
| Debugging | A single stack trace covers the whole request | Distributed tracing required. Without it, root cause analysis is guesswork |
| Local development | Clone, run, work | Many services or a maintained set of fakes. Onboarding cost rises sharply |
| Deployment | One artefact. Simple, and a bottleneck once many teams contend for it | Independent per service. The main genuine benefit |
| Failure modes | Mostly total. The process is up or down | Mostly partial, which is harder. Timeouts, retry storms, cascading saturation |
| Team size where it fits | Up to roughly thirty engineers on one product | Multiple autonomous teams, typically thirty engineers upward |
| Cost of getting it wrong | Modules blur into a tangle. Recoverable with refactoring | A distributed tangle. Recovery means re-merging services, which is far harder |
The pragmatic path we recommend most often: build a modular monolith with strictly enforced module boundaries, publish domain events internally from day one, and extract a service only when a specific pressure justifies it. Extraction from a well-modularised monolith is measured in weeks. Recombining prematurely split services is measured in quarters.
Domain-driven design
Finding service boundaries that hold
How boundaries are decided
Boundaries come from the domain. We run event storming with your domain experts to map business events chronologically, cluster them into aggregates, and identify where the same word changes meaning. A boundary is right when a service can be correct using only its own data and a local transaction.
-
Event storming with domain experts
A facilitated session where operations, finance, support and engineering place business events on a timeline. Domain experts, not just architects, because the vocabulary lives with the people doing the work.
-
Identify aggregates and invariants
Group events around the entities whose consistency must be enforced together. An aggregate is the unit that must be transactionally consistent. Aggregates never straddle service boundaries.
-
Map bounded contexts by language
Where the same noun means different things, you have found a context boundary. A "customer" in billing carries payment terms and credit status. A "customer" in support carries entitlement and contact history. Forcing one shared model is how you get a table with ninety columns.
-
Define context relationships
For each pair of contexts, name the relationship: shared kernel, customer and supplier, conformist, or anticorruption layer. This determines who absorbs the cost when one side changes, and it is a negotiation between teams as much as a technical choice.
-
Test each candidate boundary
Three questions. Can it be correct with a local transaction? Can one team own it end to end? Can it be deployed without coordinating with another team? Three yeses is a viable service. Any no and the boundary moves.
-
Sequence extraction by value and risk
Extract the context with the highest change frequency and the lowest data entanglement first. It delivers the earliest benefit and teaches the team the pattern on the easiest case rather than the hardest.
Communication
Synchronous or asynchronous, and which event pattern
Sync or async
Use synchronous calls when the caller cannot proceed without the answer and the user is waiting. Use asynchronous events when the caller only needs the work to happen eventually. Every synchronous call is a runtime dependency that couples availability: three services at 99.9 percent chained synchronously give you 99.7 percent.
| Factor | Synchronous (HTTP, gRPC) | Asynchronous (events, queues) |
|---|---|---|
| Coupling | Temporal. Both services must be up at the same moment | Only the broker must be up. Consumers can be down and catch up |
| Availability arithmetic | Multiplies. Chained dependencies compound failure | Isolates. A slow consumer creates a backlog, not an outage |
| Consistency | Immediate. The caller sees the result | Eventual. The interface must be designed for it, with honest pending states |
| Error handling | Immediate and returnable to the caller | Retries, dead-letter queues and an operational process to drain them |
| Debugging | Straightforward request and response with a trace | Requires correlation identifiers and message tracing to reconstruct a flow |
| User experience fit | Reads, validation, authorisation, anything where the user waits for an answer | Notifications, reporting, downstream side effects, cross-system propagation |
| Use when | The caller genuinely cannot continue without the response | The caller needs the work done, not the answer now |
The most common architectural mistake we correct is a synchronous chain three or four services deep on a user-facing path. Each hop adds latency and a failure mode, and the timeout budget usually has not been calculated. If the total budget is 2 seconds and there are four hops, no single hop can be allowed 2 seconds, yet that is frequently the configured default.
| Pattern | What the event contains | Benefit | Cost | Use when |
|---|---|---|---|---|
| Event notification | Minimal. An identifier and event type, such as OrderPlaced with an order id | Loose coupling, tiny payloads, no data duplication | Consumers must call back to fetch detail, which reintroduces a synchronous dependency and load on the producer | Few consumers, cheap lookups, and the producer can absorb the callback traffic |
| Event-carried state transfer | The full relevant state at the time of the event | Consumers are autonomous. They can process with no callback, even while the producer is down | Data duplication, larger payloads, and every schema change ripples to consumers. Stale local copies must be reasoned about | Many consumers, expensive or rate-limited lookups, or availability isolation is the point |
| Event sourcing | The event log is the system of record. State is derived by replaying events | Complete audit history, temporal queries, the ability to rebuild any projection and to answer questions you had not thought of yet | Substantial. Schema evolution of historical events, snapshotting, eventual consistency everywhere, and a mental model most teams have never used | Genuine audit and temporal requirements: ledgers, trading, clinical records, regulated workflow. Rarely justified elsewhere |
| CQRS | Separate write model and one or more read models kept in sync by events | Reads and writes scale and are modelled independently. Complex queries stop distorting the write model | Two models to maintain, replication lag visible to users, and read-your-own-writes needs deliberate handling | Read and write loads differ by an order of magnitude, or reporting queries are damaging transactional performance |
| Transactional outbox | Events written to an outbox table in the same transaction as the state change, then published by a relay | Removes the dual-write problem without distributed transactions | A relay process to run and monitor, plus outbox table growth and cleanup | Always, whenever a service both writes state and publishes events. This is a default, not an option |
| Saga (orchestrated or choreographed) | A sequence of local transactions with compensating actions on failure | Business processes spanning services without two-phase commit | Compensation logic is business logic and must be designed with the domain. Partial states are visible. Testing is genuinely hard | Long-running multi-service processes such as order fulfilment. Not a substitute for a correct boundary |
Event sourcing and CQRS are frequently adopted together and frequently regretted. They are powerful in the narrow set of domains that genuinely need history as a first-class concern. If your requirement is "we want an audit trail", an append-only audit log alongside a conventional model gives you that at a fraction of the cost.
Messaging
Choosing a broker, and getting delivery semantics right
Messaging technology selection
Choose the simplest broker that meets the requirement. SQS or EventBridge for most work on AWS. Kafka or MSK when you need log replay, ordered partitioned streams, or multiple independent consumer groups over retained history. Kafka is a platform commitment, not a queue.
| Technology | Model | Ordering and retention | Operating cost | When it wins |
|---|---|---|---|---|
| Apache Kafka (self-managed) | Partitioned, replayable commit log with independent consumer groups | Ordered per partition. Retention by time or size, replay from any offset | High. Brokers, storage, rebalancing, partition planning, upgrades. Realistically a platform team | High-throughput streams, event sourcing, many independent consumers over retained history, stream processing |
| Amazon MSK or Confluent Cloud | Managed Kafka | Same as Kafka | Moderate. Cluster and partition design still yours; patching and replication are not | You need Kafka semantics without owning broker operations. Our default when Kafka is genuinely required |
| Amazon SQS and SNS | Point-to-point queues with visibility timeouts, plus FIFO queues. SNS adds topic fan-out, usually to several SQS queues so each consumer gets its own buffer | Standard queues are unordered and at-least-once. FIFO gives ordering per message group at lower throughput. SNS offers no ordering and no retention beyond delivery retries | Lowest. No infrastructure at all | Work queues, background jobs, simple one-to-many fan-out, absorbing bursts. Where we start unless something rules it out |
| Amazon EventBridge | Event bus with content-based routing rules, schema registry and archive with replay | No ordering. Archive supports bounded replay | Low. Serverless with per-event pricing | Cross-domain and cross-account event routing, third-party SaaS events, routing rules you want to change without redeploying producers |
| RabbitMQ or Amazon MQ | Broker with rich routing: exchanges, bindings, priorities, delayed messages | Ordered per queue. Messages removed on acknowledgement, no replay | Moderate. Cluster and queue management, mirrored queue configuration | Complex routing topologies, priority queues, per-message TTL, or an existing AMQP or JMS estate to preserve |
| Amazon Kinesis Data Streams | Sharded stream with a retention window | Ordered per shard, retention up to 365 days | Low to moderate. Shard capacity planning is the main work | AWS-native streaming with Lambda or Firehose consumers, telemetry and clickstream ingestion, when Kafka would be over-provisioned |
| Database table as a queue | Polling with row locking | Whatever you implement | Very low initially, and it becomes your problem as volume grows | Genuinely low volume with existing transactional guarantees. Honest and often correct for a small system. It stops being correct somewhere around a few hundred messages per second |
We choose Kafka far less often than clients expect. The question that settles it: do you need to replay history to a new consumer, or have multiple consumer groups read the same stream at different positions? If not, SQS and EventBridge will do the job with a fraction of the operating cost.
Delivery semantics, honestly
- At-most-once. Fire and forget. Messages can be lost. Acceptable only for genuinely disposable telemetry.
- At-least-once. What every practical broker gives you. Duplicates will happen, caused by retries after ambiguous timeouts, consumer restarts before acknowledgement, and rebalances. Design for it.
- Exactly-once. Achievable only within a single system's boundary. Kafka transactions provide it for Kafka-to-Kafka processing. The instant you write to a database, call an API or send an email, the guarantee ends.
- Exactly-once effects. The achievable and commercially meaningful goal: at-least-once delivery plus idempotent consumers. Processing the same message twice produces the same state as processing it once.
Making consumers idempotent
- Idempotency keys. Every mutating request and message carries a stable client-generated key. The consumer records processed keys and short-circuits repeats. Retention on that record must exceed the maximum possible retry window.
- Natural idempotency. Prefer operations that are safe to repeat. Setting a status to shipped is idempotent. Incrementing a counter is not. Reshape the operation where you can.
- Conditional writes. Optimistic concurrency on a version number, or a unique constraint that makes a duplicate insert fail loudly rather than silently double-count.
- The outbox pattern. Write state and event in one local transaction, publish from the outbox with a relay or change-data-capture. Without it you get published events for rolled-back changes, or committed changes with no event.
- Dead-letter queues with an owner. Every DLQ needs an alert, a runbook and a named owner. An unmonitored DLQ is a silent data-loss mechanism, and we find them in most estates we review.
- Poison-message handling. A bounded retry count, then quarantine. Without it one malformed message blocks a partition indefinitely.
API design
REST, GraphQL, gRPC, versioning and the gateway
API style selection
Use REST at the edge for public and partner APIs, because HTTP caching, tooling and comprehensibility win where you do not control the consumer. Use GraphQL when many clients need different projections of the same graph. Use gRPC between internal services where latency, streaming and a strict schema matter.
| Factor | REST over HTTP | GraphQL | gRPC |
|---|---|---|---|
| Best for | Public and partner APIs, webhooks, anything a third party integrates without your help | Rich clients, aggregation across services, mobile clients minimising round trips | Internal service-to-service, high call volume, bidirectional streaming |
| Caching | Excellent. CDN, proxy and browser caching work as designed | Hard. POST by default, so you need persisted queries or an application-level cache | None at the HTTP layer. Application-level only |
| Over-fetching | Present unless you add sparse fieldsets or purpose-built endpoints | Solved. Clients request exactly what they need | Controlled by message definition, so a new shape means a new method or field mask |
| Schema and typing | OpenAPI, adopted with varying discipline | Strong, introspectable schema by default | Strongest. Protobuf with generated clients in every supported language |
| Browser support | Native | Native | Requires gRPC-Web and a proxy |
| Main risk | Endpoint proliferation and inconsistent conventions across teams | Expensive nested queries, the N+1 resolver problem, and authorisation that must be enforced per field | Tight coupling to generated stubs, and a harder debugging story without good tooling |
REST at the edge and gRPC internally is a sound default. Adding GraphQL is justified when you have several distinct client types with genuinely different data needs. Adding GraphQL because a single web client makes too many requests is usually solved more cheaply with a purpose-built aggregation endpoint.
Versioning and deprecation policy
- Additive changes are not versioned. New optional fields and new endpoints ship without a version bump. Consumers must tolerate unknown fields, and we state that in the contract.
- Breaking changes get a new major version. Path-based for REST, a new package for Protobuf. URI versioning over header negotiation because it is visible in logs, caches and support conversations.
- Two major versions supported concurrently, with a minimum twelve-month overlap for public APIs and ninety days for internal ones.
- Deprecation is announced with data. Sunset and Deprecation headers on responses, a changelog entry, and direct contact with the consumers your telemetry shows are still calling the old version.
- Never change semantics under a stable contract. Altering what a field means without renaming it is the most damaging kind of breaking change, because every test still passes.
What belongs in the API gateway
- Authentication. Token validation, JWT signature and claim checks, mTLS termination for partner traffic. Verify once at the edge, propagate a verified identity inward.
- Rate limiting and quotas. Per consumer and per plan, with clear 429 responses carrying Retry-After.
- Routing and edge composition. Path and header routing to services, canary weighting, and blue-green switching.
- Observability. Correlation identifier generation, structured access logs, and per-route latency and error metrics.
- What does not belong. Business logic, data transformation beyond trivial shaping, and authorisation decisions that depend on domain state. A gateway that grows business logic becomes a shared bottleneck every team must queue behind, and it will not be tested like application code.
Legacy integration: adapters, and what to avoid
The pattern is always the same. An adapter service speaks the legacy protocol on one side and a clean, owned contract on the other, with an anticorruption layer so legacy field names, sentinel values and quirks never leak into new code. What varies is the transport.
| Approach | Where you find it | How we handle it | Risk |
|---|---|---|---|
| SOAP and WSDL | Insurance, banking, telecoms, ERP and government systems | Generate a client from the WSDL, wrap it in an adapter exposing REST or gRPC, translate faults into typed errors, and cache aggressively where the data allows | Manageable. Watch for WS-Security configuration, envelope size limits, and sessions with server-side affinity |
| Flat or fixed-width files over SFTP | Payroll, banking settlement, EDI, logistics, retail supplier feeds | Idempotent ingestion keyed on file name and content hash, strict schema validation, a quarantine directory for rejects, and reconciliation counts published as metrics | Moderate. The real risks are silent partial files, duplicate delivery and a missing file nobody notices for a day. Alert on absence, not only on failure |
| IBM MQ, JMS and message-oriented middleware | Established enterprise estates | A bridge consumer translating onto your modern broker, preserving correlation identifiers and applying idempotency at the boundary | Moderate. Transaction semantics differ from cloud brokers and must be mapped explicitly rather than assumed |
| Direct database-level integration | Very common, and almost always a mistake | We treat it as debt to be retired. If unavoidable, read-only against a replica, through views owned by the source team, with a written agreement on schema change notice | Highest. It couples you to another system internal schema, so their refactor becomes your outage. There is no contract, no validation and no invariant enforcement. Writing into another system tables bypasses its business rules entirely |
| Screen scraping and robotic automation | When a system has no interface at all and the vendor will not build one | Only as an explicitly temporary bridge, with a documented exit plan, a date, and monitoring that alerts the moment the interface changes | Highest and most brittle. A cosmetic UI change breaks it, and credentials must be stored to drive it. We will build it if it is genuinely the only route, and we will say plainly that it is a stopgap |
Database-level integration deserves the emphasis. It is the fastest thing to build and the most expensive thing to own. The source team did not agree to a contract, so they are entitled to change the schema whenever they like, and your integration breaks without anyone having done anything wrong. If you inherit one, put retiring it on the roadmap.
Honest answer
The failure modes distribution introduces, and how we contain them
What distribution costs you
A distributed system fails partially, which is harder than failing completely. Timeouts without budgets cause cascades, retries without jitter cause synchronised storms, and a shared thread pool lets one slow dependency exhaust capacity for everything. These are not edge cases, they are the normal operating condition at scale.
- Timeout budgets calculated end to end. If the user-facing budget is 2 seconds across four hops, each hop gets a fraction and passes its remaining budget downstream. A downstream service that knows only 40 ms remain should fail immediately rather than start work
- Circuit breakers on every remote dependency. Open on an error-rate threshold, half-open probe after a cooldown, and a defined fallback: cached data, a degraded response, or an honest error. Failing fast beats queueing behind something already broken
- Bulkheads. Separate connection pools and thread pools per dependency, so one slow downstream cannot consume all capacity. This is the single most effective containment pattern and the most frequently missing
- Retry with exponential backoff and full jitter, only on idempotent operations, with a hard attempt cap. Retries without jitter synchronise into a thundering herd that keeps a recovering service down
- Load shedding and back pressure. Reject beyond known capacity with 429 or 503 rather than accepting work you cannot complete. A queue that grows without bound converts a slowdown into an outage
- Graceful degradation designed per feature. Decide in advance which features may be switched off to protect the revenue path, and put them behind flags so the decision can be executed in seconds
- Distributed tracing on every request, with a correlation identifier propagated across HTTP, gRPC and message headers. Without it, cross-service root cause analysis is guesswork
- Consumer-driven contract tests with Pact. Consumers publish expectations, providers verify them in their own pipeline, and a provider cannot deploy a change that breaks a known consumer. This replaces the fragile full-estate integration environment
- Chaos exercises in non-production. Inject latency, kill instances, sever a dependency, exhaust a connection pool. Every resilience pattern above is a hypothesis until it has been tested
- Per-service SLOs with burn-rate alerting, so you page on user-visible impact rather than on every individual threshold breach
Answers
API and microservices questions
Should we move to microservices?
Most likely not yet. Microservices solve an organisational scaling problem: too many engineers contending over one deployment unit. If you have fewer than about thirty engineers, or you cannot name the deployment contention that is slowing you down, a modular monolith with enforced internal boundaries gives you almost all of the design benefit and none of the distributed-systems cost. We have talked more clients out of microservices than into them.
What is a modular monolith, and why do you recommend it first?
One deployable artefact containing modules with explicit public interfaces, separate database schemas per module, no cross-module table access, and communication through in-process interfaces or a local event bus. You get domain boundaries, independent testability and clear ownership while keeping a single transaction, a single deployment and a stack trace that spans the whole request. When a module genuinely needs independent scaling or deployment, it extracts cleanly because the boundary already exists.
How do you decide where service boundaries go?
From the domain, not the org chart or the database. We run event storming with your domain experts to map business events, identify aggregates and consistency requirements, then place boundaries where the language changes meaning. If two candidate services need a distributed transaction to stay correct, the boundary is in the wrong place and they should be one service.
Is exactly-once delivery possible?
Not across a network in the general case. What is achievable is at-least-once delivery combined with idempotent consumers, which produces exactly-once effects. That is what matters commercially. Kafka transactions give exactly-once semantics within Kafka, and that guarantee ends the moment you write to an external system. Any vendor claiming end-to-end exactly-once is describing at-least-once plus deduplication.
What is the outbox pattern and why does it matter?
You cannot atomically write to your database and publish to a broker. The outbox pattern writes the event into an outbox table inside the same database transaction as the state change, and a separate process reads that table and publishes. It converts a distributed-transaction problem into a local one. Without it you eventually get state changes with no event published, or events published for changes that rolled back.
REST, GraphQL or gRPC?
REST for public and partner APIs where cacheability, tooling and comprehensibility matter most. GraphQL when many different clients need different shapes of the same data and over-fetching is a genuine cost, accepting the caching and query-cost complexity that follows. gRPC for internal service-to-service calls where latency, streaming and a strict schema matter. Most estates end up with REST at the edge and gRPC inside, which is a reasonable place to land.
Can you integrate with our mainframe or legacy SOAP systems?
Yes. The pattern is an adapter service that speaks the legacy protocol on one side and a clean modern contract on the other, so the legacy shape does not leak into new code. We work with SOAP and WSDL, fixed-width and flat files over SFTP, IBM MQ, ODBC and stored-procedure interfaces, and where nothing else exists, robotic or screen-level automation as an explicitly temporary measure with a documented exit.
What does microservices actually cost you that a monolith does not?
Network calls that used to be function calls and can now fail partially, distributed tracing to answer questions a stack trace used to answer, eventual consistency visible in the user interface, schema and contract versioning across independently deployed services, per-service pipelines and on-call, and local development that needs many services or good fakes. Budget an extra platform and observability workstream. That cost is worth paying when the organisational problem is real, and pure waste when it is not.
Start with a three-week architecture review
Fixed fee. You get a context map from event storming with your domain experts, a decomposition recommendation that may well be "stay with a modular monolith", a messaging and delivery-semantics design, an integration inventory with the risk on each, and a sequenced plan. Yours to keep either way.