close
Skip to content

[Feature][Connector-V2] Datahub support multi-table sink#9212

Merged
hailin0 merged 7 commits into
apache:devfrom
jia17:support_multitable_sink_datahub
Apr 23, 2025
Merged

[Feature][Connector-V2] Datahub support multi-table sink#9212
hailin0 merged 7 commits into
apache:devfrom
jia17:support_multitable_sink_datahub

Conversation

@jia17

@jia17 jia17 commented Apr 22, 2025

Copy link
Copy Markdown
Contributor

subtask of #5652

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

new e2e test using my Alibaba Cloud account locally.

Check list

@hailin0 hailin0 requested a review from Copilot April 22, 2025 10:58

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds support for multi-table sinks for the Datahub connector by updating the implementation classes and tests.

  • Added a new end-to-end test method (testDatahubMulti) to validate multi-table sink functionality.
  • Updated DataHubWriter and DataHubSink to implement the multi-table sink interfaces.
  • Modified DataHubSinkFactory to include the optional multi-table sink replica option.

Reviewed Changes

Copilot reviewed 4 out of 5 changed files in this pull request and generated no comments.

File Description
seatunnel-e2e/seatunnel-connector-v2-e2e/connector-datahub-e2e/src/test/java/org/apache/seatunnel/e2e/connector/datahub/DatahubIT.java Added multi-table sink E2E test method.
seatunnel-connectors-v2/connector-datahub/src/main/java/org/apache/seatunnel/connectors/seatunnel/datahub/sink/DataHubWriter.java Updated to implement SupportMultiTableSinkWriter.
seatunnel-connectors-v2/connector-datahub/src/main/java/org/apache/seatunnel/connectors/seatunnel/datahub/sink/DataHubSinkFactory.java Introduced optional multi-table sink replica configuration.
seatunnel-connectors-v2/connector-datahub/src/main/java/org/apache/seatunnel/connectors/seatunnel/datahub/sink/DataHubSink.java Modified to implement SupportMultiTableSink and return a DataHubWriter.
Files not reviewed (1)
  • seatunnel-e2e/seatunnel-connector-v2-e2e/connector-datahub-e2e/src/test/resources/fakesource_to_multi_datahub.conf: Language not supported

@hailin0

hailin0 commented Apr 22, 2025

Copy link
Copy Markdown
Member

Please upload a screenshot of the Datahub page

@jia17

jia17 commented Apr 22, 2025

Copy link
Copy Markdown
Contributor Author

QQ20250422-213843
QQ20250422-213939
QQ20250422-214145
The sink to test is :
sink {
DataHub {
endpoint = "https://dh-cn-hangzhou.aliyuncs.com"
accessId = "myid"
accessKey = "mypasswd"
project = "project_name"
topic = "${table_name}"
timeout = 3000
retryTimes = 3
}
}

@hailin0 hailin0 merged commit 7027162 into apache:dev Apr 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants