Should you read Kafka as a stream or in batch? Should you even care? | Ido Nadler and Opher Dubrovsky, Neilsen

opherdubrovskyidonadler 210928230201 thumbnail 2should you read kafka as a stream or in batch should you even care ido nadler and opher dubrovsky neilsen 1 638
opherdubrovskyidonadler 210928230201 thumbnail 4

Should you consume Kafka in a stream OR batch? When should you choose each one? What is more efficient, and cost effective?

In this talk we’ll give you the tools and metrics to decide which solution you should apply when, and show you a real life example with cost & time comparisons.
To highlight the differences, we’ll dive into a project we’ve done, transitioning from reading Kafka in a stream to reading it in batch.

By turning conventional thinking on its head and reading our multi-petabyte Kafka stream in batch using Spark and Airflow, we’ve achieved a huge cost reduction of 65% while at the same time getting a more scalable and resilient solution.

We’ll explore the tradeoffs and give you the metrics and intuition you’ll need to make such decisions yourself.

We’ll cover:

Costs of processing in stream compared to batch

Scaling up for bursts and reprocessing
Making the tradeoff between wait times and costs

Recovering from outages
And much more…

Leave a Reply

Your email address will not be published.