[Fix][Connector-V2] Fix possible data loss in scenarios of request_tablet_size is less than the number of BUCKETS#8768
Conversation
|
Friendly ping, do you have time to take a look @Hisoka-X 🙏 ? |
|
Please add testcase (reproduce & verify) |
Ok. I'll try to complete it today. |
|
I added a test case, When I set the table's buckets to 3: DISTRIBUTED BY HASH(`BIGINT_COL`) BUCKETS 3At the same time, when request_tablet_size is set to a value less than 3: The In this case, the row count is 31, which is less than the expected 100. After applying the fix, the StarRocksIT#testStarRocksReadRowCount() test now passes successfully: |
|
good pr |
…blet_size is less than the number of BUCKETS (apache#8768)




Purpose of this pull request
When users explicitly set QUERY_TABLET_SIZE instead of using the default value Integer.MAX_VALUE, the returned List partitions contains partitions with the same beAddress.
To avoid establishing a connection to the BE for each split during the read process, the StarRocksBeReadClient is cached based on the beAddress from the split's partition. However, in StarRocksBeReadClient#openScanner(), some variables are not being reset, leading to the potential use of stale values from the cache. For example, if eos (end of stream) is true from a previous partition read, it can cause data loss for new partitions on the same BE.
Does this PR introduce any user-facing change?
no
How was this patch tested?
add new tests
Check list
New License Guide
release-note.