Event-Driven Architectures Done Right, Apache Kafka • Tim Berglund • Devoxx Poland 2021



Far from a controversial choice, Kafka is now a technology developers and architects are adopting with enthusiasm. And it’s often not just a good choice, but a technology enabling meaningful improvements in complex, evolvable systems that need to respond to the world in real time. But surely it’s possible to do wrong! In this talk, we’ll look at common mistakes in event-driven systems built on top of Kafka:

-Deploying Kafka when an event-driven architecture is not the best choice.
-Ignoring schema management. Events are the APIs of event-driven systems!
-Writing bespoke consumers when stream processing is a better fit.
-Using stream processing when you really need a database.
-Trivializing the task of elastic scaling in all parts of the system.

It’s highly likely for medium- and large-scale systems that an event-first perspective is the most helpful one to take, but it’s early days, and it’s still possible to get this wrong. Come to this talk for a survey of mistakes not to make.

Lecture took place on Wednesday 25th August 2021 at 13:30 in Room 1

Tim Berglund is a teacher, author, and technology leader with Confluent, where he serves as the Senior Director of Developer Advocacy. He can frequently be found at speaking at conferences in the United States and all over the world. He is the co-presenter of various training videos on topics ranging from Git to Distributed Systems to Apache Kafka. He tweets as @tlberglund, blogs very occasionally at http://timberglund.com, and lives in Littleton, CO, USA with his wife, their three children having grown up.

Topics covered:
-What is Event-Driven Architecture
-Data Mesh Principles
-Scaling
-State management

#DevoxxPoland 2021 took place in the ICE Krakow Congress Centre on 25th – 27th August. During 3 days,
2.700 Devoxxians from 20 different countries attended #DevoxxPoland including 100+ speakers and another
600K developers enjoyed the presentations online. Making #Devoxx the biggest #Java conference in Poland.

Twitter: https://twitter.com/DevoxxPL
Instagram: https://www.instagram.com/grzegorz.duda.official/

Join us also here:
Technology Radar Review: https://dworld.pl/radar
Developers World Academy: https://dworld.pl/akademia
Devflix: https://devflix.pl

#IT #Development #SoftwareDevelopment

source

20 thoughts on “Event-Driven Architectures Done Right, Apache Kafka • Tim Berglund • Devoxx Poland 2021”
  1. I've enjoyed the first 30 min, and I always jump the gun on these comments (am I the only one? )…but at a 29:50 you mention more evolutionary architecture. Well I've been developing a web3 protocol since before bitcoin that came from the event sourcing side of things, with the idea that you can apply transforms as data ("event" analog) that live on the same Merkle DAG timelines to produce more "living" data. Think git applying semantic diffs (but not text-centric) to create/recreate branches per use case, as opposed to applying events strictly to rehydrate aggregate state.

    I see that the speaker is a git guru, and I actually reimplemented much of their internal DAG storage mechanism (even completely coincidentally using the ^ for a delimiter). Anyway it's hard to express to computer people because most are already enamored with bitcoin or existing event sourcing paradigms, but both share the same difficulty when it comes to scale: that a "single source of truth" is a very funny thing, and that we can produce more biological (evolutionary) code which even unifies microservices with the future of ANNs/RNNs/Transformers/IoT, and allows for sovereign node implementations (like the speaker's continual reminder that RDbs are good). It's just really good at unifying/simplifying what kubernetes and other orchestration systems do, with the ability to have consensus structure/specs data "on chain" and able to be shared alongside the data itself.

    Anyway, it's hard to find people who know ES and git internals extensively, and if you're interested you can look at ibgib on Github (prototype) & GitLab (low level DAG graphing lib). Also I have a few (crappy!) YT videos. Perhaps I could explain the evolution from ES design more in depth if you'd care to connect.

    Thanks for the video! (And getting this far in this comment hah)

  2. I will forever be indebted to you you’ve changed my whole life I’ll continue to preach about your name for the world to hear you’ve saved me from a huge financial debt with just little investment thanks so much Mrs. Michael kayla

  3. Was I the only one that got the reference from 24601 from Les Misérables prisoner 24601 – Jean Valjean? Loved that!!

  4. How do you handle data security in this scenario where a one business unit should not be able to see / consume the data of another business unit in a company? Some users might be able to see qty/price in their plant, other users maybe qty only.. how does this work in these events?

  5. Another pitfall of event driven architecture is that one service derives the whole context from the message it consumes. But sometimes, there is a business need for a service to maintain its own context in addition to what it reads from the topics. I know this is not the way to go for purely event driven services, but sometimes you are forced into a situation where you have to do this.

  6. Thanks! I hope you do engage more with Data Mesh. Zhamak seems to de-prioritize operational data flows but there is a massive opportunity to converge operational and analytic pipelines when you use an event driven architecture combined with polyglot persistence.

Leave a Reply

Your email address will not be published.

Captcha loading...