[Feature][Engine]Metalake support for data source information storage and management by wtybxqm · Pull Request #9688 · apache/seatunnel

wtybxqm · 2025-08-10T11:09:13Z

Purpose of this pull request

To address the problem: sensitive information such as data source usernames and passwords is directly written into task scripts
close #9687

Does this PR introduce any user-facing change?

yes
To enable Metalake, users need to configure three critical settings in seatunnel.sh: first set 'metalake_enable=true', then specify their 'metalake_url' and 'metalake_type'. Once activated, users can simply add a 'sourceId' field in their task script's source/sink configurations and replace sensitive credentials with secure placeholders like '${password}'.

How was this patch tested?

Integration testing has passed.

Check list

If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update plugin-mapping.properties and add new connector information in it
2. Update the pom file of seatunnel-dist
3. Add ci label in label-scope-conf
4. Add e2e testcase in seatunnel-e2e
5. Update connector plugin_config

…nel-env.cmd files; modify EnvCommonOptions class to include metalake configuration option

…metalake_enable=true

…ntroduce factory for type-based MetalakeClient creation; Fetch Metalake data by sourceId and replace Config placeholders

Hisoka-X · 2025-08-11T03:39:28Z

cc @liugddx

liugddx · 2025-08-11T06:37:54Z

Please add test cases and add documentation.

liugddx · 2025-09-03T14:41:59Z

+            <groupId>org.apache.httpcomponents</groupId>
+            <artifactId>httpclient</artifactId>
+            <version>4.5.13</version>


Is this change necessary?

we need to send http request to get metainfo from gravitino, so a http tool is necessary

liugddx · 2025-09-03T14:51:30Z

+}
+
+sink {
+  Console {


Please use the Assert connector.

I used the Assert connector instead of console

liugddx · 2025-09-03T15:08:41Z

+        boolean metalakeEnabled =
+                Boolean.parseBoolean(System.getenv().getOrDefault("METALAKE_ENABLED", "false"));
+        if (metalakeEnabled) {
+            this.seaTunnelJobConfig =
+                    getMetalakeConfig(ConfigBuilder.of(Paths.get(jobDefineFilePath), variables));
+        } else {
+            this.seaTunnelJobConfig = ConfigBuilder.of(Paths.get(jobDefineFilePath), variables);
+        }


We need to ensure that Flink/Spark/Zeta all support this feature simultaneously.

Therefore, it needs to be adapted in both the FlinkTaskExecuteCommandand SparkTaskExecuteCommand classes.

I added this feature in spark and flink engine and the test has passed

liugddx · 2025-09-03T15:10:39Z

        </dependency>
+        <dependency>
+            <groupId>org.testcontainers</groupId>
+            <artifactId>mysql</artifactId>


E2e test case need to support flink/spark/zeta, so they should be placed in the seatunnel-connector-v2-e2e.

I moved the metalakeIT to seatunnel-connector-v2-e2e

…or instead of console in the conf file for test

…; fix some errors in the docs

Hisoka-X

Thanks @wtybxqm for update! It's looks great!

Hisoka-X · 2025-09-14T14:44:16Z

+                if (entity == null) {
+                    throw new RuntimeException("No response entity");
+                }
+                ObjectMapper mapper = new ObjectMapper();


Please reuse JsonUtils

I added a readTree method in JsonUtils and reuse it here

Hisoka-X · 2025-09-14T14:45:13Z

+        String metalakeType = System.getenv("METALAKE_TYPE");
+        String metalakeUrl = System.getenv("METALAKE_URL");


Maybe user can overwrite it on job config env part is better. cc @liugddx

Maybe user can overwrite it on job config env part is better. cc @liugddx

Yes, user can retrieve variables from the environment (env), and if they are not present in the environment, they can obtain them from the System. Config envConfig = config.getConfig("env");

Hisoka-X · 2025-09-14T14:47:42Z

+                        if (value.valueType() == ConfigValueType.STRING) {
+                            String strValue = (String) value.unwrapped();
+                            Matcher matcher = pattern.matcher(strValue);
+                            if (matcher.find()) {
+                                String placeholder = matcher.group(1);


how about move this part into PlaceholderUtils with new method?

I have moved this into PlaceholderUtils

Hisoka-X · 2025-09-14T14:48:24Z

+                if (sinkObj.containsKey("sourceId")) {
+                    ConfigObject tmp = sinkObj;
+                    String sourceId = sinkObj.toConfig().getString("sourceId");
+                    JsonNode metalakeJson = metalakeClient.getMetaInfo(sourceId);
+                    for (Map.Entry<String, ConfigValue> entry : sinkObj.entrySet()) {
+                        String subKey = entry.getKey();
+                        ConfigValue value = entry.getValue();
+
+                        if (value.valueType() == ConfigValueType.STRING) {
+                            String strValue = (String) value.unwrapped();
+                            Matcher matcher = pattern.matcher(strValue);
+                            if (matcher.find()) {
+                                String placeholder = matcher.group(1);
+
+                                if (metalakeJson.has(placeholder)) {
+                                    String replaced = metalakeJson.get(placeholder).asText();


parse source and sink has same code. Let's refactor it again!

I have refactored these same code

Hisoka-X · 2025-09-14T14:49:46Z

+httpcore-4.4.13.jar
 httpcore-4.4.16.jar
+httpcore-4.4.4.jar


It's weired we introduce new two version of httpcore. Can you analyze the dependencies and use only httpcore-4.4.16.jar?

It is actually wired. I just introduce httpclient-4.5.13 which is already in the other part of the project, but the Dependency License test remind me to add other two version of httpcore

We should find the core reason before merge it. cc @liugddx

httpcore has been introduced in multiple versions. Can we unify the version?

Can we unify the version?

Sure.

I have unified the version of httpcore to 4.4.16 and removed the extra version in known-dependency.txt

Hisoka-X · 2025-09-19T02:37:05Z

+@Slf4j
+public class MetalakeConfigUtils {
+
+    public static Config getMetalakeConfig(Config jobConfigTmp) {


We should support transform too. For example, OpenAI key in LLM/Embedding etc.

I have supported transform here.

Hisoka-X · 2025-09-19T02:42:47Z

+        Config envConfig =
+                ConfigBuilder.of(Paths.get(jobDefineFilePath), variables).getConfig("env");
+        boolean metalakeEnabled =
+                envConfig.hasPath("metalake_enabled")
+                        ? envConfig.getBoolean("metalake_enabled")
+                        : Boolean.parseBoolean(
+                                System.getenv().getOrDefault("METALAKE_ENABLED", "false"));
+        if (metalakeEnabled) {
+            this.seaTunnelJobConfig =
+                    MetalakeConfigUtils.getMetalakeConfig(
+                            ConfigBuilder.of(Paths.get(jobDefineFilePath), variables));
+        } else {
+            this.seaTunnelJobConfig = ConfigBuilder.of(Paths.get(jobDefineFilePath), variables);
+        }


The part can be refactor too?

I have refactored this part

Hisoka-X

Thanks @wtybxqm . We almost fininsh this.

Hisoka-X · 2025-09-20T13:23:09Z

        return result.toString();
    }
+
+    public static String replacePlaceholders(String input, JsonNode json) {


Suggested change

public static String replacePlaceholders(String input, JsonNode json) {

public static String replacePlaceholders(String input, JsonNode supportedValues) {

Hisoka-X · 2025-09-20T13:23:35Z

+        Config update = jobConfigTmp;
+        try {
+            ConfigList sourceList = jobConfigTmp.getList("source");
+            update =
+                    update.withValue(
+                            "source",
+                            ConfigValueFactory.fromIterable(
+                                    replaceConfigList(sourceList, envConfig)));
+        } catch (IOException e) {
+            log.error("Fail to get MetaInfo", e);
+        }
+
+        try {
+            ConfigList sinkList = jobConfigTmp.getList("sink");
+            update =
+                    update.withValue(
+                            "sink",
+                            ConfigValueFactory.fromIterable(
+                                    replaceConfigList(sinkList, envConfig)));
+        } catch (IOException e) {
+            log.error("Fail to get MetaInfo", e);
+        }
+
+        try {
+            ConfigList sinkList = jobConfigTmp.getList("transform");
+            update =
+                    update.withValue(
+                            "transform",
+                            ConfigValueFactory.fromIterable(
+                                    replaceConfigList(sinkList, envConfig)));
+        } catch (IOException e) {
+            log.error("Fail to get MetaInfo", e);
+        }


I found this part can be refactor too.

…etMetalakeConfig method

Hisoka-X · 2025-09-21T02:53:44Z

+        Config envConfig = updateConfig.getConfig("env");
+        String metalakeType =
+                envConfig.hasPath("metalake_type")
+                        ? envConfig.getString("metalake_type")
+                        : System.getenv("METALAKE_TYPE");
+        String metalakeUrl =
+                envConfig.hasPath("metalake_url")
+                        ? envConfig.getString("metalake_url")
+                        : System.getenv("METALAKE_URL");
+        MetalakeClient metalakeClient = MetalakeClientFactory.create(metalakeType, metalakeUrl);


we should move this part outer replaceConfigList. The metalakeClient should only created once.

Hisoka-X

Thanks @wtybxqm ! Finally, let's update https://github.com/apache/seatunnel/blob/dev/docs/sidebars.js#L88

Hisoka-X

LGTM if ci passes. Thanks @wtybxqm !

liugddx

LGTM @wtybxqm Thank you for your contribution.

Hisoka-X · 2025-09-23T02:13:46Z

waiting test case passes.

wtybxqm added 4 commits July 20, 2025 18:57

feat:Add metalake configuration option in seatunnel-env.sh and seatun…

ca2d091

…nel-env.cmd files; modify EnvCommonOptions class to include metalake configuration option

Feat:Fetch metalake config from environment and merge into task's env

8ceffc9

feat:Fetch metalake config by sourceId and replace placeholders when …

b5b3cc0

…metalake_enable=true

feat:Add MetalakeClient interface & GravitinoClient implementation; I…

d4ec880

…ntroduce factory for type-based MetalakeClient creation; Fetch Metalake data by sourceId and replace Config placeholders

github-actions Bot added Zeta api labels Aug 10, 2025

wtybxqm added 4 commits August 21, 2025 19:30

feat:apply spotless code style

d2a84c7

feat: replace wildcard imports with explicit imports for okhttp3

f32ac80

fix: delete useless EnvOption

d545047

feat: Integration Test for Metalake

84491da

github-actions Bot added the e2e label Aug 25, 2025

wtybxqm added 7 commits August 27, 2025 19:08

feat: Integration Test

768815b

feat: apply spotless code style

a6d2327

fix: fix the error of config structure

a1148f7

feat: place okhttp3 with apache httpclient

c0b304e

fix: fix the bug of matainfo replace and remove log info

d66778e

feat: apply spotless code style and remove extra code

b741052

feat: Add docs of Metalake in zh and en

50d0158

github-actions Bot added the document label Aug 29, 2025

wtybxqm added 5 commits September 1, 2025 12:39

Merge branch 'dev' into support_metalake_development

bd027c7

push an empty commit to trigger the workflow

84a2b0b

fix: add license header to conf file

1acfb14

feat: download gravitino in test container

cfeb84a

feat: apply spotless codestyle and remove useless code

ec25bb0

liugddx reviewed Sep 3, 2025

View reviewed changes

wtybxqm added 2 commits September 6, 2025 20:51

feat: support metalake for spark and flink engine; use assert connect…

1b42050

…or instead of console in the conf file for test

fix: fix the error of the assert connector

d67545f

wtybxqm added 3 commits September 13, 2025 14:14

fix: reuse PlaceholderUtils and refactor the getMetalakeConfig method…

bba310a

…; fix some errors in the docs

fix: add license header

72a7538

fix: add capture group in pattern matcher

46d19f7

Hisoka-X reviewed Sep 14, 2025

View reviewed changes

wtybxqm added 2 commits September 15, 2025 20:06

fix: refactor MetalakeConfigUtils and use PlaceholderUtils and JsonUtils

8c96119

fix: unify the version of httpcore

4df31f8

github-actions Bot added the connectors-v2 label Sep 17, 2025

fix: remove extra version of httpcore in kwown-dependencies.txt

0f3363a

github-actions Bot removed the dependencies Pull requests that update a dependency file label Sep 17, 2025

feat: get variables from env before from system

dffb3b6

Hisoka-X reviewed Sep 19, 2025

View reviewed changes

feat: refactor getMetalakeConfig method and support transform

5bb154f

Hisoka-X reviewed Sep 20, 2025

View reviewed changes

fix: modify the arg name of replacePlaceholders method and refactor g…

dde368c

…etMetalakeConfig method

Hisoka-X reviewed Sep 21, 2025

View reviewed changes

fix: create metalakeClient only once in getMetalakeConfig method

2362402

Hisoka-X reviewed Sep 22, 2025

View reviewed changes

fix: add metalake in sidebar.js of docs

788bfda

Hisoka-X approved these changes Sep 22, 2025

View reviewed changes

github-actions Bot added approved reviewed labels Sep 22, 2025

Hisoka-X requested changes Sep 22, 2025

View reviewed changes

Comment thread ...ector-jdbc-e2e-part-7/src/test/resources/jdbc_mysql_source_to_assert_sink_with_metalake.conf

github-actions Bot removed approved reviewed labels Sep 22, 2025

liugddx approved these changes Sep 23, 2025

View reviewed changes

Hisoka-X approved these changes Sep 24, 2025

View reviewed changes

Hisoka-X merged commit 6416a17 into apache:dev Sep 24, 2025
8 checks passed

zhangshenghang mentioned this pull request Oct 7, 2025

[Hotfix] [connector-jdbc-e2e-part-7] CI failure caused by a conflict during container deletion #9922

Merged

3 tasks

		String metalakeType = System.getenv("METALAKE_TYPE");
		String metalakeUrl = System.getenv("METALAKE_URL");

	public static String replacePlaceholders(String input, JsonNode json) {
	public static String replacePlaceholders(String input, JsonNode supportedValues) {

Uh oh!

Conversation

wtybxqm commented Aug 10, 2025 • edited by Hisoka-X Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

Uh oh!

Hisoka-X commented Aug 11, 2025

Uh oh!

liugddx commented Aug 11, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Hisoka-X left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Hisoka-X left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Hisoka-X left a comment

Choose a reason for hiding this comment

Uh oh!

wtybxqm commented Aug 10, 2025 •

edited by Hisoka-X

Loading