<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[Aman Ranjan Verma]]></title><description><![CDATA[Aman Ranjan Verma]]></description><link>https://arverma.hashnode.dev</link><generator>RSS for Node</generator><lastBuildDate>Sun, 22 Nov 2026 15:30:01 GMT</lastBuildDate><atom:link href="https://arverma.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[How a Kafka Connect Rebalance Skipped 2.5 Billion Messages and How I Fixed It]]></title><description><![CDATA[A Kafka Connect scale-down skipped a 24-hour offset window in an Iceberg pipeline. I rewound offsets and backfilled 2.5B records, no duplicates.]]></description><link>https://arverma.hashnode.dev/kafka-connect-rebalance-skipped-2-5-billion-messages</link><guid isPermaLink="true">https://arverma.hashnode.dev/kafka-connect-rebalance-skipped-2-5-billion-messages</guid><category><![CDATA[kafka]]></category><category><![CDATA[data-engineering]]></category><category><![CDATA[streaming]]></category><category><![CDATA[big data]]></category><category><![CDATA[apache iceberg]]></category><dc:creator><![CDATA[Aman Ranjan Verma]]></dc:creator><pubDate>Fri, 20 Nov 2026 11:23:31 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6b0029d977da122175a6ee8b/52ebbcef-a492-4fbb-9043-b1dcaf32b630.jpg" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>If you run high-throughput streaming pipelines using Kafka Connect and Apache Iceberg, you know that keeping data latency low while managing cluster resources is a constant challenge.</p>
<p>Recently, I hit an incident where a routine check on our streaming telemetry pipeline revealed a huge gap: an entire day's worth of data was completely missing in our Iceberg data lake, even though millions of records were sitting safely in our upstream Kafka topics.</p>
<p>Here is how JVM infrastructure crashes and a routine task scale-down over the weekend caused Kafka Connect to skip a 24-hour offset window, and how I safely rewound consumer offsets to backfill 2.5 billion records without creating duplicate data.</p>
<h2>The Incident: Missing Lakehouse Data</h2>
<p>Our pipeline streams raw application event logs from a high-throughput Kafka topic (<code>telemetry.app_events.v1</code>) into an Apache Iceberg table (<code>lakehouse.bronze.app_event_logs</code>) using the Iceberg Sink Connector.</p>
<p>While auditing our daily ingestion numbers in our analytics engine, I noticed that while Day 1 and Day 3 had full data, Day 2 was completely blank.</p>
<p>I checked our Dead Letter Queue (DLQ) topic (<code>telemetry.app_events.dlq.v1</code>), expecting to see malformed records. It was completely empty. So why did the pipeline stop ingesting on Day 2 without leaving a single message in the DLQ?</p>
<h2>Root Cause Analysis: Infrastructure Failures and the Rebalance Trap</h2>
<p>The missing data was caused by a combination of JVM worker crashes and task rebalancing mechanics.</p>
<h3>1. Infrastructure Crashes (Heap OOM and DNS Flakes)</h3>
<p>Our connector tasks had failed due to system-level infrastructure errors:</p>
<ul>
<li><strong>Java heap memory exhaustion:</strong> Sudden traffic spikes and large batch processing caused connector worker tasks to hit <code>OutOfMemoryError: Java heap space</code> limits.</li>
<li><strong>Network flakes (</strong><code>UnknownHostException</code><strong>):</strong> Temporary DNS errors prevented workers from reaching external endpoints.</li>
</ul>
<p>Because these were JVM memory and network infrastructure crashes (rather than record-level transformation bugs), Kafka Connect's error tolerance mechanisms could not intercept them. The worker tasks simply crashed into a <code>TASK_FAILED</code> state.</p>
<h3>2. The Two-Phase Commit Offset Delay</h3>
<p>The Iceberg Sink Connector uses a two-phase commit strategy. Sink tasks buffer records in worker memory, write temporary data files to object storage, and only commit Kafka consumer group offsets after a successful Iceberg catalog commit (which runs on a 3-minute interval via <code>iceberg.control.commit.interval-ms: 180000</code>).</p>
<p>When tasks crashed mid-stream, their uncommitted offsets remained un-checkpointed in Kafka, and their uncommitted Parquet files were left abandoned in storage as unindexed "orphan files".</p>
<h3>3. Scaling Down Triggered the Offset Skip</h3>
<p>To recover from the failed tasks, I manually restarted them and decided to scale down <code>tasks.max</code> from 50 tasks to 5 tasks to stabilise worker memory consumption. 50 tasks were overprovisioned at that time; this number had been inflated in the past to reduce lag during a prior incident.</p>
<p>That scale-down was the cause:</p>
<ul>
<li>Reducing <code>tasks.max</code> forced a complete rebalance of Kafka Connect workers across the cluster.</li>
<li>All 50 old worker tasks were revoked, and 5 new tasks were spawned.</li>
<li>The failed tasks crashed before they could write their checkpointed offsets back to Kafka, so the new task group initialised consumer positions using the auto-offset reset policy at the current topic head, completely skipping the uncommitted 24-hour offset gap.</li>
</ul>
<h2>Step-by-Step Resolution: The Recovery Plan</h2>
<p>To recover the missing data cleanly, I needed to rewind our consumer offsets back to the start of the missing window. However, simply replaying Kafka messages into Iceberg would create duplicate rows for records that had already landed.</p>
<p>Here is the 4-step execution plan I used to backfill the data cleanly.</p>
<h3>Step 1: Force Consumer Group Inactivity</h3>
<p>Attempting to reset offsets on a running or paused connector produced an error:</p>
<pre><code class="language-plaintext">400: Group's offsets can be reset only if group is inactive.
</code></pre>
<p>Even when paused, Kafka Connect keeps consumer threads warm and assigned to partitions. To force the consumer group into an <code>Empty</code> state:</p>
<ul>
<li>I saved our connector's full JSON configuration.</li>
<li>I deleted the connector definition from the Kafka Connect cluster.</li>
</ul>
<p>(Deleting a connector only stops the processing tasks; it never deletes topic messages or data in storage.)</p>
<h3>Step 2: Reset Consumer Offsets by Timestamp</h3>
<p>With the consumer group state shifted to <code>Empty</code>, I navigated to the Consumers tab in our Kafka UI (<code>connect-s3-iceberg-telemetry-events-sink-prod</code>) and performed the offset reset:</p>
<ul>
<li><strong>Reset strategy:</strong> TIMESTAMP</li>
<li><strong>Target time:</strong> exact timestamp where the missing data window began</li>
<li><strong>Scope:</strong> ALL partitions</li>
</ul>
<h3>Step 3: Clear Overlapping Data in Iceberg</h3>
<p>To prevent duplicate rows, I executed a targeted partition delete in SQL:</p>
<pre><code class="language-sql">DELETE FROM lakehouse.bronze.app_event_logs
WHERE emitted_ts &gt;= TIMESTAMP '2026-09-26 00:00:00 UTC';
</code></pre>
<p>Because Apache Iceberg handles deletes at the snapshot metadata level, this query finished almost instantly.</p>
<h3>Step 4: Re-Deploy with Max Parallelism</h3>
<p>I navigated back to Kafka Connect, clicked Create Connector, and pasted our saved JSON configuration. I paid close attention to two critical settings:</p>
<ul>
<li><strong>Connector name matching:</strong> I set the connector name to the exact original string (<code>s3-iceberg-telemetry-events-sink-prod</code>). Kafka Connect automatically prefixes consumer group IDs with <code>connect-</code>. If you accidentally add <code>connect-</code> to the name field during creation, it spawns a brand new consumer group and ignores your reset offsets!</li>
<li><strong>Max tasks bump:</strong> I temporarily set <code>"tasks.max": "50"</code> in the JSON config so 50 parallel worker threads could churn through the backlog.</li>
</ul>
<h2>Key Takeaways for Operating Kafka Connect</h2>
<p>Within a couple of hours, the 50 tasks completely processed the 2.5 billion message backlog. Here are the most important takeaways from this incident:</p>
<ul>
<li><strong>Infrastructure crashes bypass DLQs:</strong> Dead Letter Queues only capture record-level transformation and serialization failures. They will not catch JVM out-of-memory crashes or network host resolution errors.</li>
<li><strong>Never reconfigure tasks during an outage:</strong> If tasks fail due to memory or network issues, resolve the underlying worker resources or restart the tasks first. Changing <code>tasks.max</code> or triggering rebalances while there is uncommitted consumer lag risks jumping un-checkpointed offsets.</li>
<li><strong>Lag drops in batches:</strong> Kafka Connect consumer lag won't drop second-by-second in your monitoring dashboards. Because offsets are committed strictly on Iceberg catalog commit intervals (e.g., every 3 minutes), lag will drop in massive, stepped chunks.</li>
</ul>
<hr />
<p><em>Originally published in</em> <a href="https://medium.com/towards-data-engineering/kafka-connect-how-i-recovered-2-5-44735f7f7fb9"><em>Towards Data Engineering</em></a><em>. More notes from running Kafka and Iceberg in production are at</em> <a href="https://arverma.dev/writing/"><em>arverma.dev/writing</em></a><em>.</em></p>
]]></content:encoded></item></channel></rss>