Apache Kafka Crash Course



Apache Kafka is a distributed stream processing software developed by LinkedIn and written in Scala and Java.

Chapters
0:00 Intro
2:32 Kafka Broker
5:39 Kafka Producer
7:11 Kafka Consumer
8:34 Kafka Partitions
12:50 Queue vs Pub-Sub
16:47 Consumer Group
23:06 ZooKeeper
29:45 Coding Example
1:02:48 Kafka Pros-Cons
1:17:00 Summary

Download slides here
https://payhip.com/b/Y0Cq

In this video I want to focus on the basics of Kafka, talk about how it works give finally spin up a Kafka cluster and write a producer and a consumer.

Kafka Components
_ producer
_consumer
_topic
_partition
How Kafka works
Example ( Ride sharing )
Spin Kafka cluster
Write a producer
Write a consumer
Kafka Applications
Pros and Cons

Source Code
https://github.com/hnasr/javascript_playground/tree/master/kafka

Docker commands spin up kafka
//if you ever mess up or you want to start fresh just stop and remove the containers and rerun the command

docker stop zookeeper kafka
docker rm zookeeper kafka

docker run –name zookeeper -p 2181:2181 -d zookeeper

docker run -p 9092:9092 –name kafka -e KAFKA_ZOOKEEPER_CONNECT=husseinmac:2181 -e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://husseinmac:9092 -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 -d confluentinc/cp-kafka

Resources

KIP 500 (Remove Zookeeper)
https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum

Support my work on PayPal
https://bit.ly/33ENps4

Become a Member on YouTube
https://www.youtube.com/channel/UC_ML5xP23TOWKUcc-oAE_Eg/join

🧑‍🏫 Courses I Teach
https://husseinnasser.com/courses

🏭 Backend Engineering Videos in Order
https://backend.husseinnasser.com

💾 Database Engineering Videos

🎙️Listen to the Backend Engineering Podcast
https://husseinnasser.com/podcast

Gears and tools used on the Channel (affiliates)

🖼️ Slides and Thumbnail Design
Canva
https://partner.canva.com/c/2766475/647168/10068

🎙️ Mic Gear
Shure SM7B Cardioid Dynamic Microphone
https://amzn.to/3o1NiBi

Cloudlifter
https://amzn.to/2RAeyLo

XLR cables
https://amzn.to/3tvMJRu

Focusrite Audio Interface
https://amzn.to/3f2vjGY

📷 Camera Gear
Canon M50 Mark II
https://amzn.to/3o2ed0c

Micro HDMI to HDMI 
https://amzn.to/3uwCxK3

Video capture card
https://amzn.to/3f34pyD

AC Wall for constant power
https://amzn.to/3eueoxP

Stay Awesome,
Hussein

source

25 thoughts on “Apache Kafka Crash Course”
  1. wtf! i was just watching peterson on another screen. i got spooked. i swear the two accounts are not connected

  2. I didn't know you before this video, but after this video? I'm gonna watch everything you upload man. Every tutorial you're selling? I'm buying! Obviously I liked & subscribed.

  3. i had to deal with Kafka in one of my project, so searched for some tutorials, As soon as I saw that you have made a video on this, i had a sense of relief that i dont have to worry about the resource atleast. Thanks for all your efforts man.

  4. For those facing issue with Mac M1, Add –platform="linux/amd64" in the kafka pull command.

  5. The explanation and illustration is clear and concise! Thanks for the comprehensive tutorial!

  6. Only worked with docker compose for me. (m1 mac).

    version: '3'

    services:
    zookeeper:
    image: wurstmeister/zookeeper
    container_name: zookeeper
    ports:
    – "2181:2181"
    kafka:
    image: wurstmeister/kafka
    container_name: kafka
    ports:
    – "9092:9092"
    environment:
    KAFKA_ADVERTISED_HOST_NAME: localhost
    KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181

  7. Great video. Thanks a lot. Can you also explain how kafka is handling the order of the messages. I understand that ther is ordering inside partitions, but what if we need queue model is tead of pub/sub, and we want messages to be consumed in the order that they happen.

  8. zookeeper is running and the server also running and creating one topic also a success and created topic is deleted then the server is automatically stopping ,what is the reason and what is the solution

  9. Kafka sounds like a vaguely middle eastern food that is now promoted by my hippy Aunt.

    Great vid man

Leave a Reply

Your email address will not be published.

Captcha loading...