[Feature][Jdbc] Support read multiple tables by regular expressions by yzeng1618 · Pull Request #9380 · apache/seatunnel

yzeng1618 · 2025-05-29T06:49:20Z

…ssions in PostgreSQL/mysql/oracle/sqlserver, and fix the null pointer issue in the regular expression
#9209

Purpose of this pull request

added contents about multiple tables and regular expressions in PostgreSQL/mysql/oracle/sqlserver, and fix the null pointer issue in the regular expression.

Does this PR introduce any user-facing change?

Yes.
This PR enhances the table_list parameter to directly support regex patterns for table filtering, while maintaining backward compatibility.
Here's the detailed breakdown:

New Feature: Direct Use of Regular Expressions in table_path
Purpose: Allow users to write regular expressions directly in the table_path field within table_list to filter tables.
Example Configuration:
"table_list"=[
{
"table_path"="TEST.TEST_DB_*" # Matches all tables with the "TEST_DB" prefix
}
]
This configuration matches all tables prefixed with TEST_DB (e.g., TEST_DB_2023, TEST_DB_2024).

Improvement and Fix:
Enhanced the robustness of the approximateRowCntStatement method in OracleDialect.
Fixed a null pointer error in Oracle when executing queries with empty or invalid parameters.
Key Notes:
The table_path now supports regex syntax (e.g., TEST.TEST_DB_* → matches all tables under the TEST schema with names starting with TEST_DB_).
The Oracle fix ensures stable execution of row count estimation logic, avoiding crashes due to unhandled edge cases.

How was this patch tested?

Testing Environment
OS: Linux (Ubuntu 20.04)
SeaTunnel Version: 2.3.9
Execution Mode: Flink on YARN (yarn-application)
Databases:
PostgreSQL 16.0(Source)
Iceberg (Sink)

Test Configuration:
Created a configuration file pg2iceberg.conf to read tables matching the regex TEST.TEST_DB_* from PostgreSQL:
{
env {
execution.parallelism = 1
job.mode = "BATCH"
job.name = "seatunnel_batch_job"
}

source配置
source {
JDBC {
url = "jdbc:postgresql://xxxxxxx:xxxxx/xxxxx"
driver = "org.postgresql.Driver"
user = "xxxxxxxx"
password = "xxxxxxx"
"table_list" = [
{
"table_path" = "postgres.public.test_db_2.*"
}
]
split.size = 5000
fetch_size = 2000
}
}

sink配置
sink {
Iceberg {
........
}
}
}

Execution Command:
./bin/start-seatunnel-flink-15-connector-v2.sh --config pg2iceberg.conf --deploy-mode run-application --target yarn-application --name multitable_pg2Iceberg

Check log：
Filtering tables with regex pattern: postgres.public.test_db_2.*
Found regex match table: postgres.public.test_db_20
Found regex match table: postgres.public.test_db_21
Found regex match table: postgres.public.test_db_22
Found regex match table: postgres.public.test_db_20250324
Found regex match table: postgres.public.test_db_202502
Found regex match table: postgres.public.test_db_202501
Total tables matched after filtering: 6

Check list

If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
[ * ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
[ * ] If you are contributing the connector code, please check that the following files are updated:
1. Update plugin-mapping.properties and add new connector information in it
2. Update the pom file of seatunnel-dist
3. Add ci label in label-scope-conf
4. Add e2e testcase in seatunnel-e2e
5. Update connector plugin_config

…ssions in PostgreSQL/mysql/oracle/sqlserver, and fix the null pointer issue in the regular expression

Hisoka-X

I think we need to add a option to mark whether the TablePath is a regular expression.

yzeng1618 · 2025-06-03T03:48:21Z

I think we need to add a option to mark whether the TablePath is a regular expression.

1、Simplified Path Matching Design，If you enter a path like /project/table_123, the system first checks for this exact path，If no exact match exists, it automatically treats your input as a regex pattern.
The main advantages are as follows:

Zero configuration: There is no need for marking parameters like "is_regex", and users don't have to switch modes manually.
Just enter the input and it will be matched, reducing the cognitive burden.
Intuitive behavior: It follows a logic similar to that of the grep tool, First, perform a literal match, and then enable pattern matching.

2、Like Flink CDC (where you directly use exact table names OR regex patterns without special parameters), we keep things simple.

Hisoka-X · 2025-06-06T03:02:27Z

+
+Examples:
+- `testdb.table+` - Matches all tables starting with "table" followed by numbers
+- `testdb.*` - Matches all tables in the `testdb` database


Does this expression will match database testdb2 too?

The regular expression testdb.* will only match all the tables in the testdb database and will not match the testdb2 database.

Hisoka-X · 2025-06-06T03:24:01Z

+            for (JdbcSourceTableConfig tableConfig : tablePaths) {
+                List<String> schemaTables = new ArrayList<>();
+                String tablePath = tableConfig.getTablePath();
+                String query = tableConfig.getQuery();
+                LOG.info("Processing table path: {}, custom query: {}", tablePath, query);
+                String sql;
+                if (StringUtils.isBlank(query)) {
+                    String schemaName;
+                    if (jdbcDialect.dialectName().startsWith(DatabaseTypeEnum.ORACLE.getValue())) {
+                        schemaName = tablePath.split("\\.")[0];
+                        sql = "SELECT OWNER, TABLE_NAME FROM dba_tables where OWNER=?";
+                        ps = connection.prepareStatement(sql);
+                        ps.setString(1, schemaName);
+                        rs = ps.executeQuery();
+                        while (rs.next()) {
+                            // For Oracle: schema.table
+                            String foundTable =
+                                    rs.getString("OWNER") + POINT + rs.getString("TABLE_NAME");
+                            schemaTables.add(foundTable);
+                            LOG.info("Found table in Oracle: {}", foundTable);
+                        }


Why not reuse JDBCCatalog to scan all should read tables? This way support all database type with catalog. https://github.com/apache/seatunnel/blob/dev/seatunnel-api/src/main/java/org/apache/seatunnel/api/table/catalog/Catalog.java#L102

The current approach of manually implementing table scanning in JdbcSource offers several key advantages:
Precise Control

Custom SQL

queries tailored for each database type (Oracle, MySQL, SQLServer, PostgreSQL)

Direct access to system tables for accurate table information retrieval

Detailed Logging

Comprehensive logging for debugging and troubleshooting

Database-specific logging (e.g., LOG.info("Found table in PostgreSQL: {}", foundTable))

Specialized Format Handling

Dedicated handling for different database table path formats:

MySQL: database.table
PostgreSQL: database.schema.table
Oracle: schema.table
SQLServer: database.schema.table

Flexible Pattern Matching

Two-step matching strategy in filterCapturedTablesByRegrex:First attempts exact matches
Falls back to regex pattern matching

Reduced Dependencies
Functions independently of complete Catalog implementations
Works even when some databases have incomplete Catalog implementations

Compatibility Assurance
Explicitly defines supported databases via isSupportedDatabase
Provides fallback to JdbcCatalogUtils.getTables for unsupported database types

While using the Catalog interface might lead to more unified code, the current implementation provides greater flexibility and adaptability to specific database behaviors, especially when handling differences in system table structures and naming conventions.

# Conflicts: # docs/zh/connector-v2/source/Mysql.md

yzeng1618 · 2025-06-11T08:36:07Z

The modification has been made. I hope you can review it when you are free.

hailin0 · 2025-06-12T01:49:06Z

                    .withDescription("The database names RegEx of the database to capture.");

+    Option<String> SCHEMA_PATTERN =
+            Options.key("schema-pattern")


Hisoka-X · 2025-06-19T03:35:45Z

+import java.util.Set;
+import java.util.stream.Collectors;
+
+@Disabled("Please Test it in your local environment")


why disabled the test case? You can reuse

seatunnel/seatunnel-connectors-v2/connector-jdbc/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/catalog/utils/TestResultSet.java

Line 43 in 2f96f2e

public class TestResultSet implements ResultSet {

to mock data.

davidzollo · 2025-06-21T11:55:01Z

good job. This is already close to merge.
Please add e2e

Hisoka-X · 2025-06-23T03:32:05Z

+        boolean isOracleDialect =
+                jdbcDialect.dialectName().equalsIgnoreCase("oracle")
+                        || jdbcDialect.dialectName().equalsIgnoreCase("oceanbase-oracle");


We cannot add this special processing logic. There are more than two Oracle-like databases.

yzeng1618 · 2025-06-23T09:41:37Z

good job. This is already close to merge. Please add e2e

E2E tests for MySQL and Oracle have been added.

Hisoka-X · 2025-06-24T03:04:23Z

+
+    default Optional<String> getDefaultDatabase() {
+        return Optional.empty();
+    }
+
+    default boolean useThreePartTablePath() {
+        return false;
+    }


Why not reuse

seatunnel/seatunnel-connectors-v2/connector-jdbc/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/internal/dialect/JdbcDialect.java

Line 296 in cbb777e

default TablePath parse(String tablePath) {

?

Hisoka-X

Overall LGTM except two minior problems. Thanks @yzeng1618

Hisoka-X · 2025-06-25T13:17:29Z

+    public Boolean getUseRegex() {
+        return useRegex != null ? useRegex : false;
+    }


Suggested change

public Boolean getUseRegex() {

return useRegex != null ? useRegex : false;

}

Hisoka-X · 2025-06-25T13:21:35Z

+                        boolean isRegexPath = tableConfig.getUseRegex();
+
+                        if (StringUtils.isNotEmpty(tableConfig.getTablePath())
+                                && StringUtils.isEmpty(tableConfig.getQuery())
+                                && isRegexPath) {


Suggested change

boolean isRegexPath = tableConfig.getUseRegex();

if (StringUtils.isNotEmpty(tableConfig.getTablePath())

&& StringUtils.isEmpty(tableConfig.getQuery())

&& isRegexPath) {

if (StringUtils.isNotEmpty(tableConfig.getTablePath())

&& StringUtils.isEmpty(tableConfig.getQuery())

&& tableConfig.getUseRegex()) {

Hisoka-X

Thansk @yzeng1618 ! LGTM if ci passes.

davidzollo

+1
good job

…pache#9380) Co-authored-by: zengyi <zengyi@chinatelecom.cn>

[Feature][Jdbc]added contents about multiple tables and regular expre…

39d155b

…ssions in PostgreSQL/mysql/oracle/sqlserver, and fix the null pointer issue in the regular expression

github-actions Bot added connectors-v2 labels May 29, 2025

nielifeng requested a review from Copilot May 29, 2025 07:42

This comment was marked as outdated.

Sign in to view

Hisoka-X reviewed May 29, 2025

View reviewed changes

[Feature][Jdbc] Add unit tests and update docs

a177a0b

github-actions Bot added the document label Jun 3, 2025

zengyi added 5 commits June 3, 2025 16:49

[Feature][Jdbc] Add unit tests and update docs

d5c9d9d

[Feature][Jdbc] delete zh jdbc.md

1d3dad4

[Feature][Jdbc] Fix some details

60da5b4

[Feature][Jdbc] Add null value validation

e2bf8dd

[Feature][Jdbc] Add null value validation

f08fe67

nielifeng requested a review from Copilot June 5, 2025 01:37

This comment was marked as outdated.

Sign in to view

[Feature][Jdbc] Fix some details

856ea8d

yzeng1618 requested a review from Hisoka-X June 6, 2025 03:01

Hisoka-X reviewed Jun 6, 2025

View reviewed changes

yzeng1618 requested a review from Hisoka-X June 6, 2025 05:50

zengyi added 3 commits June 7, 2025 21:54

[Feature][Jdbc] Refactor the code and adopt the catalog interface.

fa2f657

[Feature][Jdbc] fix some details

5baa84a

[Feature][Jdbc] fix some details

49d6865

github-actions Bot added the api label Jun 10, 2025

zengyi added 3 commits June 10, 2025 18:27

Merge branch 'dev' into BDPL-29578

849fa83

# Conflicts: # docs/zh/connector-v2/source/Mysql.md

[Feature][Jdbc] update docs and resolve conflicts

8331cc0

[Feature][Jdbc] update docs and resolve conflicts

e4fb226

hailin0 reviewed Jun 12, 2025

View reviewed changes

yzeng1618 requested a review from hailin0 June 12, 2025 06:05

[Feature][Jdbc] fix zh docs

8d77c8d

Hisoka-X reviewed Jun 19, 2025

View reviewed changes

[Fix bug][Connector-Iceberg] revised unit tests

72b0530

yzeng1618 requested a review from Hisoka-X June 20, 2025 23:09

[Feature][Jdbc] add e2e tests

882421c

github-actions Bot added the e2e label Jun 21, 2025

zengyi added 2 commits June 22, 2025 09:07

[Feature][Jdbc] add e2e tests

04bc2b5

[Feature][Jdbc] fix details

b4ba558

Hisoka-X reviewed Jun 23, 2025

View reviewed changes

[Feature][Jdbc] fix details

b0ee005

[Feature][Jdbc] Fix details from a universality perspective.

cbb777e

yzeng1618 requested a review from Hisoka-X June 23, 2025 13:44

Hisoka-X reviewed Jun 24, 2025

View reviewed changes

[Feature][Jdbc] Fix details from a universality perspective.

ee5a210

yzeng1618 requested a review from Hisoka-X June 24, 2025 15:02

Hisoka-X reviewed Jun 25, 2025

View reviewed changes

[Feature][Jdbc] Fix details

527a529

yzeng1618 requested a review from Hisoka-X June 26, 2025 02:30

zengyi added 4 commits June 26, 2025 11:19

[Feature][Jdbc] Fix details

7f339a2

[Feature][Jdbc] Fix details

3a86582

[Feature][Jdbc] Fix details

2a9315d

[Feature][Jdbc] Fix details

2f6c76a

Hisoka-X approved these changes Jun 26, 2025

View reviewed changes

github-actions Bot added approved reviewed labels Jun 26, 2025

davidzollo approved these changes Jun 27, 2025

View reviewed changes

davidzollo merged commit 670a52a into apache:dev Jun 27, 2025
5 checks passed

dybyte pushed a commit to dybyte/seatunnel that referenced this pull request Jul 23, 2025

[Feature][Jdbc] Support read multiple tables by regular expressions (a…

79f25a5

…pache#9380) Co-authored-by: zengyi <zengyi@chinatelecom.cn>

	public Boolean getUseRegex() {
	return useRegex != null ? useRegex : false;
	}

Uh oh!

Conversation

yzeng1618 commented May 29, 2025

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

Uh oh!

This comment was marked as outdated.

Uh oh!

Hisoka-X left a comment

Choose a reason for hiding this comment

Uh oh!

yzeng1618 commented Jun 3, 2025

Uh oh!

This comment was marked as outdated.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yzeng1618 commented Jun 11, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davidzollo commented Jun 21, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yzeng1618 commented Jun 23, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Hisoka-X left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Hisoka-X left a comment

Choose a reason for hiding this comment

Uh oh!

davidzollo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants