Apache Kafka® 101: Partitioning



https://cnfl.io/kafka-101-module-3 | With partitioning, a single topic log is broken into multiple logs. Each of these logs can live on a different node in the Apache Kafka® cluster. The effort behind storing and processing messages as well as writing new messages can be split among many nodes in the cluster. Learn how partitioning works in this video with Tim Berglund (Senior Director of Developer Experience, Confluent).

Use the promo code KAFKA101 to get $101 of free Confluent Cloud usage: https://www.confluent.io/confluent-cloud/tryfree/?utm_source=youtube&utm_medium=video&utm_campaign=tm.devx_ch.cd-apache-kafka-101_content.apache-kafka

Promo code details: https://www.confluent.io/confluent-cloud-promo-disclaimer/?utm_source=youtube&utm_medium=video&utm_campaign=tm.devx_ch.cd-apache-kafka-101_content.apache-kafka

LEARN MORE
► How to Choose the Number of Topics/Partitions in a Kafka Cluster?: https://www.confluent.io/blog/how-choose-number-topics-partitions-kafka-cluster/?utm_source=youtube&utm_medium=video&utm_campaign=tm.devx_ch.cd-apache-kafka-101_content.apache-kafka
► Benchmarking Apache Kafka Latency at the 99th Percentile ft. Anna Povzner: https://developer.confluent.io/podcast/benchmarking-apache-kafka-latency-at-the-99th-percentile-ft-anna-povzner?utm_source=youtube&utm_medium=video&utm_campaign=tm.devx_ch.cd-apache-kafka-101_content.apache-kafka
► How to Change the Number of Partitions and Replicas of a Kafka Topic: https://kafka-tutorials.confluent.io/change-topic-partitions-replicas/ksql.html?utm_source=youtube&utm_medium=video&utm_campaign=tm.devx_ch.cd-apache-kafka-101_content.apache-kafka
► How to Read from a Specific Offset and Partition with the Kafka Console Consumer: https://kafka-tutorials.confluent.io/kafka-console-consumer-read-specific-offsets-partitions/confluent.html?utm_source=youtube&utm_medium=video&utm_campaign=tm.devx_ch.cd-apache-kafka-101_content.apache-kafka

CONNECT
Subscribe: https://youtube.com/c/confluent?sub_confirmation=1
Site: https://confluent.io
GitHub: https://github.com/confluentinc
Facebook: https://facebook.com/confluentinc
Twitter: https://twitter.com/confluentinc
LinkedIn: https://www.linkedin.com/company/confluent
Instagram: https://www.instagram.com/confluent_inc

ABOUT CONFLUENT
Confluent, founded by the creators of Apache Kafka®, enables organizations to harness business value of live data. The Confluent Platform manages the barrage of stream data and makes it available throughout an organization. It provides various industries, from retail, logistics and manufacturing, to financial services and online social networking, a scalable, unified, real-time data pipeline that enables applications ranging from large volume data integration to big data analysis with Hadoop to real-time stream processing. To learn more, please visit https://confluent.io

#kafka #kafkastreams #streamprocessing #apachekafka #confluent

source

19 thoughts on “Apache Kafka® 101: Partitioning”
  1. so it is generally correct , if not always, to say that messages in different partitions within same topic are mutually exclusive?

  2. If the number of partitions changes, does Kafka re-hash and redistribute events? If not then events with the same ID could end up in different partitions?

  3. What is the relation between events and messages?
    Messages have key-value pairs? or Events have key-value pairs?
    what exactly … Events or messages are stored in partitions with key-value pairs?

    this was a helpful video
    Take key ——> HASH Function——-> Output mod (Total no. of partitions) ——> Resulting no. is Partitions number where message going to store.
    I was unaware of this concept.
    Thank You Tim.

Leave a Reply

Your email address will not be published.

Captcha loading...