How to Take a Banking Product Real-Time

Wednesday, 2:47 a.m. The dashboard is glowing red.

Spring 2024. We'd cleared functional and integration testing and were rolling out to prod — and that's when the real load test began, this time on live traffic. Tarantool starts dropping requests twenty minutes in under real load. Not all at once — first one timeout in a thousand, then two, then ten. A pretty exponential curve on the chart that will turn the whole system into a pumpkin within half an hour.

I'm sitting at my monitor on a late-night call with two developers, my third cup of coffee going cold on the desk. One thought keeps circling in my head: the demo for the business stakeholder is six weeks out. And we've just discovered that the foundation of our architecture is cracking.

But that comes later. First — why break something that already worked at all.

Why batch is yesterday's news

The bank's pre-approved offers system had run in batch mode for years. Models run overnight, lists get built in the morning, customers get their SMS during the day. Conversion — a fraction of a percent. By the time a person sees the offer, they've long forgotten why they opened the app in the first place.

The brief sounded simple: a customer takes an action — the bank reacts within a second. Not tomorrow, not in an hour. Now.

Here's what the difference looks like on paper:

Parameter	Batch	Real-time
Latency	12-24 hours	< 1 second
Data freshness	Yesterday's snapshot	The present moment
Infrastructure cost	Lower (nightly windows)	Higher (24/7 clusters)
Architecture complexity	Linear (ETL pipeline)	Exponential (event mesh)
Error handling	Rerun the batch	Dead Letter Queue, retries, alerts
Scaling	Vertical (beefier server)	Horizontal (more nodes)
Debugging	Logs from a single run	Distributed tracing

On paper, it's a table. In practice, it's a complete rebuild of the architecture, the data, the processes, and the team's way of thinking. People who spent five years writing batch jobs now have to think in terms of events, windows, and watermarks. That's not learning a technology — it's reflashing the brain.

Architecture decisions: why this exact stack

Flink, not Spark Streaming

Picking a streaming engine is a decision that will cost you sleep for the next three years. On the table were Apache Spark Streaming, Kafka Streams, and Apache Flink.

Spark Streaming was out within a minute — micro-batch with latency measured in seconds didn't fit the SLA. We gave Kafka Streams serious consideration: it's lighter, baked into the ecosystem, doesn't need a separate cluster. But we needed stateful processing with windowing functions and complex enrichment logic. At the time, Kafka Streams struggled with that.

Flink won on three criteria. First — native event time support. When events arrive late (and from banking systems they always arrive late), that's not just a nice-to-have — it's the difference between a right and a wrong decision about a customer. Second — exactly-once semantics out of the box. In a banking context, a lost event isn't a metric on a dashboard — it's a potential customer complaint and the regulator's attention. Third — mature checkpoint/savepoint for fault tolerance. The system has to survive a node crash without losing state.

The price: on the Russian market you can literally count Flink specialists on your fingers. Recruiting turns from a funnel into a hunt.

Tarantool: speed and betrayal

Every event has to be enriched: customer profile, transaction history, current limits, product parameters. The classic route is a relational database. But when you're doing thousands of events per second with an SLA of tens of milliseconds, PostgreSQL becomes the bottleneck.

Tarantool solved this beautifully: an in-memory store with sub-millisecond latency. Data from the core systems replicates with minimal lag, and Flink jobs reach out to it for enrichment.

And then came that night.

The bug only showed up under load close to production. In the test environment — flawless. At half the live traffic — flawless. At seventy percent — sporadic timeouts. At target volumes — a degradation that turned the system into a vegetable within half an hour.

Week one of debugging: false trails. We blamed the network, the JVM, the Flink configuration. Week two: isolating the problem down to a specific pattern of concurrent requests in Tarantool. The exact scenario: when parallel read and write requests hit the same region of data at a certain frequency, the in-memory engine started to degrade.

I remember the meeting where we decided what to do. Two options on the table: wait for a fix from the community (indefinitely long) or an emergency pivot of part of the architecture. There was no time. We documented the bug, reported it to the community, and over the weekend rewrote the critical path onto PostgreSQL plus caching.

Latency went from microseconds to milliseconds. The architecture got dirtier. But it was stable. In a bank, predictability beats elegance.

Kafka as the nervous system

Apache Kafka is the transport layer of the whole system. Every microservice publishes and reads events through topics. Three things Kafka solves: decoupling services (each one only knows about its own topics), replay (you can reread events for debugging), and scaling through partitioning.

Kafka, by the way, was the one technology in the stack that never once threw us a surprise. It runs like a Swiss watch, as long as you don't mess with the retention settings too often.

Four languages in one project — deliberate chaos

Java, Scala, Lua, SQL. Sounds like a bad joke. But behind every choice is pragmatism, not aesthetics.

Java — the backbone of the microservices. Spring Boot, a huge pool of developers, mature libraries for Kafka.

Scala — the Flink jobs. The Flink API on Scala is an order of magnitude more expressive. Writing windowed aggregations in Scala is a pleasure. In Java it's a punishment.

Lua — stored procedures in Tarantool. With LuaJIT as the embedded language, it's the only option for complex enrichment logic.

SQL — analytical queries and rule configuration in the decision engine. Business users modify the rules without developers.

The price: finding a developer who reads both Java and Scala fluently is already a quest. And Lua developers for Tarantool are a rare breed you won't find on the open market. We grew our own. There was no other option.

Event-driven: beautiful in theory, painful in practice

Event Sourcing for critical business events. Every change is an event. A full history, the ability to replay any state at any point in time. A must-have for the banking audit that shows up at the worst possible moment and asks exactly what happened to a customer on February 17th at 2:23 p.m.

CQRS to separate the flows. Writes go through the event pipeline, reads come from projections in Tarantool and PostgreSQL. Independent scaling of each load.

Dead Letter Queue — our best friend in the first weeks after launch. An event that fails to process isn't lost — it lands in a separate topic for analysis. In the first week in prod, the DLQ ballooned to sizes we hadn't planned for. It turned out legacy systems were sending events in a format that didn't match the spec. Not occasionally — in five percent of cases. Five percent of millions of events is a lot of garbage in the DLQ.

The main pitfall is eventual consistency. In an event-driven system there's no guarantee that all services see the same state at the same time. For banking products that's terrifying: you can't offer a loan if data about the customer's current debt hasn't reached the scoring service yet. The fix is explicit ordering control through timestamps and watermarks in Flink. It works, but it adds a layer of complexity you have to explain to every new developer.

The failures that taught us

Failure one: synthetic data lies. We tested on synthetic data for three months and were confident in the system. Day one on real data — a cascade of errors. Legacy systems sent dates in four different formats. Fields the spec called "mandatory" arrived empty. Identifiers that were supposed to be unique came back duplicated. The test environment is a map, but the territory looks nothing like it.

Failure two: we shipped monitoring too late. For the first two months we were building business logic. Monitoring was a "we'll bolt it on later." As a result, when the problems started, we were reading tea leaves. After that I set a rule: no service goes to review without metrics, tracing, and alerts. Monitoring isn't the cherry on top — it's the foundation.

Failure three: we discovered canary deployments too late. The first releases went out to all traffic at once. One of them tanked conversion by thirty percent — a bug in the scoring logic that only surfaced on a specific customer segment. Four hours to detect, one hour to roll back. After that — canary only: one percent of traffic first, then ten, then fifty.

Failure four: we underestimated backpressure. Monday morning, peak load. The upstream system dumps the events that piled up over the weekend in one batch. Kafka copes — it always copes. Flink starts to choke: checkpoints grow, latency creeps up, and within twenty minutes consumer lag balloons to millions of messages. We hadn't built in a backpressure mechanism that would properly slow consumption under overload. We had to manually stop the jobs, take a savepoint, and restart with a capped read rate. After that incident, a backpressure strategy became part of the architecture review for every new Flink job.

What was left after the project

Architecture decisions in enterprise are always a compromise between the ideal and the possible. The ideal system exists in conference talks. The real one is a balance between performance, reliability, maintenance cost, and time to market.

Technology is ten percent of the problem. Flink, Kafka, Tarantool work. The other ninety percent is coordinating across teams, aligning API contracts, integrating with legacy, and managing the expectations of a business that wants "like Google, but in three months."

Technical debt in a real-time system isn't an abstract metric. Every delay in processing an event hits business metrics right now, in real time. We allocated twenty percent of each sprint to tech debt. It wasn't enough.

The concrete result the whole thing was for: the time from a customer's action to a personalized offer dropped from 12-24 hours to 800 milliseconds at the 95th percentile. Conversion into the target action grew several times over compared to batch mode. I won't quote exact numbers — NDA — but the difference was big enough that the business stakeholder, who'd spent half a year skeptically asking "why do we even need real-time?", came back after the first month in prod with a request to expand to four more product scenarios. The best proof that an architecture works is when the business asks for more.

And one more thing — production smells different from staging. Anomalies, spikes, corrupt data from systems that were last updated back when Medvedev was president. All of it only surfaces in battle. And if you're not ready for it, the battle will surface in you.

---

Original article: How to Take a Banking Product Real-Time, on Habr

Based on my talk at Saint HighLoad++ 2024, St. Petersburg.

Yours, DPUPP