Many developers have already wrapped their minds around the basic architecture and APIs of Kafka as a message queue and a streaming platform. But can they keep it running in production? This talk contains real-world troubleshooting and optimization scenarios culled from the logs of Confluent technical support.
We’ll talk about the trade-offs between optimizing for the always-desirable outcomes of throughput, latency, durability, and availability. How many partitions should you use for a given topic? How much message batching should you configure in the producer? How many replicas should be required to acknowledge a write? What do you do when you see a partition growing inexplicably? When should you rearchitect your application to use the streaming API? We’ll answer these questions and more int his overview of common Kafka production issues.
almost tolerable at 1.75x speed and volume cranked up
the font is giving me headaches.
This guy is full of himself. Hot 🔥 air. The style of presenting is similar to what a 3rd grade teacher does in primary school.
Now if only the Kafka community would actually put together some reasonable how-tos on setting up "clusters" (even single pods behind services) in Kubernetes with the correct listener configs. Pretty much all of the documentation is for Docker (Wurstmeister) or running an operator off of a helm chart (Bitnami). The former is useless now due to the vastly different networking model (and Docker being effectively dead), and the latter is ridiculous tooling overhead for a development environment like Minikube while also hiding away much of the low-level customization that is needed by most enterprises.
The only one that actually works out of the box (and leaves reasonable breadcrumbs for modifying it) uses Google's out of date kafka on kubernetes image.
Waste of time
70% funny talks, 30% real content, start from 30:24
anyone from confluent can add… what changed from this talk to 2020?.. if any..
live simple life is actually a solution ..👍
Here I am with a 15 gb json file looking for a versatile db system.
were you talking about Splunk 🙂 , the log processor 🙂
Thanks. I cracked the exam watching your all videos. I prepared some notes, hope this can help people https://codingnconcepts.com/post/apache-kafka-ccdak-exam-notes/
I watched the video to get some lessons learned in production, but more than 50% of it is Kafka basics terminology and marketing.
so in summary:
1. Don't hire administrators that they don't know how to operate Kafka and didn't remember that one of the brokers was upgraded to newer version than the rest of the cluster
2. Poorly designed health check, again – poor administrator or whoever build that and not thoroughly test it. (moral of the story: Hire someone who know how to work with 'sharp knives')
3. And again.. poor guys who runs this cluster didn't paying attention what they are doing.
Hearing this, I have also got the motivation that I too can be a speaker like him. 🙏
Great talk thanks!
Does cloudera official support confluent product?
real Stuff begins after 30minutes
Just to make it clear
Replication Factor of 3
and yes of-course 4 Brokers
Q1: can a broker manage multiple partitions of same topic? In other words, can no. of broker be less than max no. of paritions in a topic?
Q2: If answer to above is 'no' then can we say that in order to add new partition we need to add new broker first?
Q3: What is best way to reduce no. of partitions? Should one just delete that extra partition, also will the broker handles deletion of its replica as well? OR is there a way to deactivate a partition I.e. set that extra partition to read only mode?
17:07 What if I use Kafka with just a single partition and a single broker ?. I kind of like of Kafka not just for its distributed nature but for how process streams and Ksql, but also the environment.
such nice arms on this guy. if only he waved his biceps around more. or wore a tank-top. hair needs a bunch of work though.
One may directly jump to the discussion on production issues which starts at 30:24
To add, "Commodity Hardware" here apparently means machines with a minimum recommended RAM size of 64GB.
Would I call that commodity hardware? Probably no. But just wanted to point that out.
TBH … They should have more experienced person giving these talks … This guy does not seem to have any hands on real experience
Tell me slow…what is a consumer group? Are they applications?
I like this PPT!
This is one of the worst presentations i came across.
So how many brokers do you guys usually stand up at your companies? Are you doing this in Azure?
SIEM – I think he was the word he was looking for. But I might be wrong…
"Starts with an S I'm blanking on it" ?
I heard "starts with an S I am plunking on it" LOL, I thought that was just a very nice joke from Tim to suggest Splunk.
Nice demo from Gavin Belson!
This one is a better insight on particular issues in Kafka production.. https://www.youtube.com/watch?v=MhyW1FUdN8I
What's the name of tool he used to create those sketches?