Saint HighLoad++ 2024: Moving a Banking Product to Real-Time

An 800-Seat Hall. Two Microphones. The First Slide.

June 2024, Saint Petersburg, Saint HighLoad++. I'm standing backstage, counting the people in the hall. Eight hundred is the point where you can't make out faces anymore — just a mass. Next to me is Vladimir Avramov, the project's lead developer, calm as a boa constrictor. He'll be showing the code. I'll be telling the story of how we nearly killed the project with our own architectural decisions.

Thirty minutes for two years of work. The choice is simple: present the pretty architecture from the docs, or the truth. We chose the truth. Slides full of little boxes get forgotten within the hour. But the story of you rolling back a release on a Friday night — that one sticks. Afterward, people in the hallways thanked us for it more than for any technical slide.

Why a Lead and a Developer Onstage Together

A joint talk was a deliberate choice. It's one thing to show slides with boxes and arrows. It's another when a developer opens the IDE and shows how a specific Flink operator behaves under load. The HighLoad++ audience is engineers. They want code, not marketing.

The format worked like this: I give the context — the business problem, the constraints, why we chose this path. Avramov shows what the solution looks like in code and what happens when it gets hammered in production. The audience gets the full picture: strategy and tactics at once.

Later, at other conferences, I saw talks where a manager describes the architecture while his developer sits in the audience quietly shaking his head. We didn't have that — we'd agreed upfront: we tell it how it was, not how we wish it had been.

Napkin Architecture

HighLoad++ values specifics. The diagram lived in our documentation in exactly this form: customer event → Kafka in → Flink processes the stream and enriches it with data from an in-memory store → Kafka out → delivery channel. Along the way, Flink pings Tarantool for reference data and profiles, runs the business rules, and returns a decision.

Beautiful. On the slide. In practice, every component had a surprise in store — except Kafka, the only one that behaved itself.

Technology	Role	Upside	Risk
Flink	Stream processing	Exactly-once, stateful	Talent shortage in Russia
Tarantool	In-memory store	Minimal latency	Young community
Kafka	Transport	Reliability, scale	Minimal

A detailed breakdown of each decision is in the full version. Here, it's about what happened when we took all of it onstage.

The Bug We Showed the Room

The riskiest part of the talk was an honest account of a failure. We showed how, during load testing, we found a bug in Tarantool: under a certain pattern of concurrent requests, the in-memory store started to degrade. And how, over a single weekend, we rewrote the critical path. Before: event → Flink → Tarantool → decision. After: event → Flink → PostgreSQL and Kafka → decision.

When I put that slide up, someone in the room whistled. Latency went up by an order of magnitude, the architecture got dirtier. But the system's behavior became predictable.

After the talk, an engineer from the Tarantool team came up to us. He wasn't offended — he thanked us for the bug report. He said they'd reproduced it and were working on a fix. That's the best thing that can happen at a conference: you talk about a problem, and the people who can solve it come find you.

The Questions We Didn't See Coming

Preparing for the talk, we expected questions about Flink, Kafka partitioning, maybe about Lua in Tarantool. Reality had other plans.

The first question from the floor: "How did you sell the business on pivoting away from Tarantool? That's a schedule slip." Not a technical question — a management one. I took it: I honestly explained that we'd shown the business two options — wait for a fix with an open-ended timeline, or lose some speed but ship to production on time. The business chose predictability. It always does.

Second: "Four languages in one project — how do you do code review?" Avramov took that one: whoever reviews Scala code has to understand how that code interacts with the Lua procedures in Tarantool. Cross-functionality turns from a buzzword into a daily necessity. We didn't find universal people — we grew them inside the team.

The third question stuck with me the most: "Would you choose Flink again today?" My answer — yes, with a caveat. Flink is the right tool for the job. The problem isn't the tool; it's that people who know how to work with it are almost nonexistent on the Russian market. If you're not ready to grow the expertise in-house — don't go near it.

What I Stole from Other Speakers

A conference isn't just your talk. It's dozens of other people's.

At one of the sessions, an engineer from another company described how they'd solved a similar problem on Kafka Streams — without Flink. Simpler, cheaper, fewer staffing risks. Higher latency, but they stayed within their SLA. I sat there thinking: could we have done that? Honestly — no. Our SLA demanded a different level. But here's what got me: their architecture was an order of magnitude simpler to operate. Kafka Streams runs as a library inside the application — no separate cluster, no JobManager, no checkpoint voodoo. Maintaining our Flink infrastructure ate up a full engineer's time. Theirs — zero, because there was no infrastructure as a separate entity. Right there I wrote in my notebook: for the next project with a less strict SLA — Kafka Streams, no debate.

Another talk was about observability in event-driven systems. The speaker showed a distributed-tracing approach that we later partly adopted. Before the conference, we traced events the old-fashioned way — a correlation ID passed through every service. After, we added spans and a structured metric for each processing stage. Concretely: previously, during an incident, an engineer would open Kibana, search by correlation ID, and piece the picture together from the logs of five services by hand. After we introduced spans — open the trace in Jaeger, see the whole chain with timings. Incident diagnosis time dropped threefold. Seems obvious, right? But until you see it working on another team's live example, you keep putting it off for "later."

This is what makes conferences worth going to. Not for the "speaker" badge, but to look at your own system through someone else's eyes.

Prep: Two People, Double the Difficulty

A joint talk sounds like "we split the work in half, so it got easier." In practice — exactly the opposite. A solo talk you rehearse in front of a mirror. A joint one you have to sync with a living human who has his own rhythm, his own habits, his own sense of timing.

We started prepping two months out. The first run-through was a disaster. I talk for three minutes, hand off to Avramov, he talks for seven. I try to pick the pace back up — he hasn't finished his thought yet. Total: forty-five minutes instead of thirty. HighLoad++ won't give you an extra fifteen minutes. They'll just pull you off the stage.

After the fourth run-through, we introduced a strict protocol: every block on a timer, transitions on cue phrases. I end on a sentence that logically leads into the code. Avramov starts by putting the screen up. No "oh, and I'd also like to add" — anything you want to add, you write into the slides before the talk. The sixth run came in at twenty-eight minutes. The seventh — thirty on the dot.

But onstage it paid off. When two people work in sync, the room feels it. No awkward pauses, no wrestling over the microphone. And the audience sees it: these two didn't just prepare a presentation — they actually work together. And trust in what you're saying grows.

What Changed After the Talk

Preparing for the talk forced us to systematize what we knew intuitively. When you draw the architecture for a slide, you suddenly see the places where the arrows point the wrong way. Where a component does too much. Where there's no fallback.

After the conference, we ran an internal architecture retro. Three decisions came out of it. First — we formalized a fallback strategy for every external component. If Tarantool goes down, the system switches to PostgreSQL automatically, not on a call to whoever's on duty. Second — we reworked monitoring: instead of an alert saying "Tarantool latency is up," an alert saying "Tarantool latency is up, fallback activated, business impact: conversion down by X%." Context instead of raw numbers. Third — we launched an internal tech talk for adjacent teams. It turned out that half the problems at the seams came from people simply not knowing how the neighboring service worked.

A conference talk isn't a full stop. It's a catalyst for change. You go onstage and say "here's what we did," then come back and think, "we could do better."

Scars Instead of Slides

In enterprise, you can't bet on a single technology without a plan B. No matter how great it looks on benchmarks. And you can't bet on one specialist who "knows everything." Knowledge has to live in the system, not in someone's head.

If you're preparing a talk — tell the truth. The room doesn't need your perfect architecture. They need your scars. Because they have their own, and they want to know they're not the only ones.

And one more thing: if you get the chance to present as a pair — try it. Yes, the prep is twice as hard. Yes, you have to sync up. But the room gets two views of one system — strategic and tactical. And you get a partner who won't let you gloss things over onstage. Because he was there when it all broke, and he knows how it really went.

---

Based on my talk at Saint HighLoad++ 2024, Saint Petersburg.

Yours, DPUPP