close
Skip to content

[Feature][Connector-V2] Starrocks implements multi table sink#8467

Merged
Hisoka-X merged 23 commits into
apache:devfrom
jw-itq:starrocks-supportMultiTableSink
Jan 9, 2025
Merged

[Feature][Connector-V2] Starrocks implements multi table sink#8467
Hisoka-X merged 23 commits into
apache:devfrom
jw-itq:starrocks-supportMultiTableSink

Conversation

@jw-itq

@jw-itq jw-itq commented Jan 6, 2025

Copy link
Copy Markdown
Contributor

Purpose of this pull request

#8455

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

@hailin0

hailin0 commented Jan 6, 2025

Copy link
Copy Markdown
Member

Please add e2e and update docs

@hailin0 hailin0 changed the title starrocks implements multi table sink [Feature][Connector-V2] Starrocks implements multi table sink Jan 6, 2025
@jw-itq

jw-itq commented Jan 6, 2025

Copy link
Copy Markdown
Contributor Author

Please add e2e and update docs

ok,thanks!

@github-actions github-actions Bot added the e2e label Jan 7, 2025
}

source {
MySQL-CDC {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, let me modify this. I'm not using fake-to-starrocks.conf because I want to test incremental data. I'm not using mysqlcdc_to_starrocks_with_schema_change.conf because it uses shop.sql from mysql-cdc, which doesn't contain multiple tables. I'm going to modify this now.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, there was a problem with my local code. I have fixed it now. Please help me take a look again. Thank you!

@hailin0 hailin0 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM cc @Hisoka-X

@Hisoka-X Hisoka-X merged commit 55eebfa into apache:dev Jan 9, 2025
@ASDFSA13

ASDFSA13 commented Jan 8, 2026

Copy link
Copy Markdown

这个pr有问题, 其实还是没解决
对于mysql8.0.28版本的数据库
PRIMARY KEY (id),
UNIQUE KEY uni_str (str_value_sha256,seq,ledger_date,is_delete) USING BTREE,
KEY idx_status (seller_id,market_place_id,is_delete) USING BTREE,
KEY idx_created_time (create_time) USING BTREE,
KEY idx_ledger_date (ledger_date,event_type,sync_status) USING BTREE

表结构如上,这时候代码,对于表结构如果主键id为 PRIMARY KEY 的时候, 他是只会从 UNIQUE KEY 取字段生效, 而不从 PRIMARY KEY 取字段生效

String tableSc = splitColumnsConfig.getOrDefault(tableId.catalog() + "." + tableId.table(), null);

if (StringUtils.isNotEmpty(tableSc)) {
AtomicBoolean isUniqueKey = new AtomicBoolean(false);
dialect.getUniqueKeys(jdbc, tableId).forEach(ck -> ck.getColumnNames().forEach(ckc -> {
if (tableSc.equals(ckc.getColumnName())) {
isUniqueKey.set(true);
}
}));

if (isUniqueKey.get()) { return column; }
else { log.warn("Config snapshotSplitColumn not unique key for table {}", tableId); }

}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants