[Feature][connector-elasticsearch] elasticsearch source support PIT by CosmosNi · Pull Request #9150 · apache/seatunnel

CosmosNi · 2025-04-11T03:43:47Z

elasticsearch source support PIT

close #9149

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update plugin-mapping.properties and add new connector information in it
2. Update the pom file of seatunnel-dist
3. Add ci label in label-scope-conf
4. Add e2e testcase in seatunnel-e2e
5. Update connector plugin_config
Update the release-note.

Copilot

Copilot reviewed 11 out of 12 changed files in this pull request and generated 1 comment.

Files not reviewed (1)

seatunnel-e2e/seatunnel-connector-v2-e2e/connector-elasticsearch-e2e/src/test/resources/elasticsearch/elasticsearch_source_with_pit.conf: Language not supported

Copilot · 2025-04-11T12:25:41Z

+        Map<String, String> sortField = new HashMap<>();
+        sortField.put("order", "asc");
+        sort.add(Collections.singletonMap("_shard_doc", "asc"));


The variable 'sortField' is assigned but never used; consider removing it to clean up the code.

Suggested change

Map<String, String> sortField = new HashMap<>();

sortField.put("order", "asc");

sort.add(Collections.singletonMap("_shard_doc", "asc"));

hailin0 · 2025-04-11T12:26:29Z

Do you have more information on PIT vs scroll?

CosmosNi · 2025-04-11T12:46:50Z

Do you have more information on PIT vs scroll?
@hailin0
Here are the advantages of PIT compared with Scroll API:

Better data consistency: PIT creates a consistent data view, ensuring accurate results even when the index changes during pagination.
Lower resource consumption: It uses snapshots to save search results, only storing document changes after the write point, thus reducing memory consumption.
More flexible usage: Combined with Search After, PIT allows for more stable pagination and is less affected by the number of retrieval requests.

Hisoka-X · 2025-04-12T02:48:35Z

                    .withDescription(
                            "Elasticsearch query language. You can control the range of data read");
+
+    public static final Option<Boolean> USE_PIT =


How about add new enum type named search_api_type? Support scorll and pit and set scroll by default.

hailin0 · 2025-04-12T06:55:53Z

Do you have more information on PIT vs scroll?
@hailin0
Here are the advantages of PIT compared with Scroll API:

Better data consistency: PIT creates a consistent data view, ensuring accurate results even when the index changes during pagination.

Lower resource consumption: It uses snapshots to save search results, only storing document changes after the write point, thus reducing memory consumption.

More flexible usage: Combined with Search After, PIT allows for more stable pagination and is less affected by the number of retrieval requests.

Which versions of elasticsearch can be used?

CosmosNi · 2025-04-12T07:48:40Z

Do you have more information on PIT vs scroll?
@hailin0
Here are the advantages of PIT compared with Scroll API:

Better data consistency: PIT creates a consistent data view, ensuring accurate results even when the index changes during pagination.

Lower resource consumption: It uses snapshots to save search results, only storing document changes after the write point, thus reducing memory consumption.

More flexible usage: Combined with Search After, PIT allows for more stable pagination and is less affected by the number of retrieval requests.

Which versions of elasticsearch can be used?

7.10

Hisoka-X · 2025-04-14T02:28:48Z

 | source                  | array   | no       | -                                                              |
 | query                   | json    | no       | {"match_all": {}}                                              |
-| search_type             | json    | no       | Search method,sql or dsl,default dsl                           |
+| search_type             | json    | no       | Search method,SQL、PIT、SCROLL,default SCROLL                    |


search_type and search_api_type not same.
search_type : SQL, DSL
search_api_type: SCROLL, PIT

Hisoka-X · 2025-04-15T02:18:04Z

+        Map<String, String> sortField = new HashMap<>();
+        sortField.put("order", "asc");
+        sort.add(Collections.singletonMap("_shard_doc", "asc"));


Hisoka-X · 2025-04-15T02:24:48Z

+                        PIT_KEEP_ALIVE,
+                        PIT_BATCH_SIZE,


Suggested change

PIT_KEEP_ALIVE,

PIT_BATCH_SIZE,

PIT_KEEP_ALIVE,

PIT_BATCH_SIZE,

SEARCH_API_TYPE,

Hisoka-X · 2025-04-15T02:25:14Z

+        }
+        // DSL query
+        else {
+            // Check if we should use PIT API


Suggested change

}

// DSL query

else {

// Check if we should use PIT API

} else {

// Check if we should use PIT API

Hisoka-X · 2025-04-15T02:25:38Z

+            }
+            // Default scroll API
+            else {


Suggested change

}

// Default scroll API

else {

} else {

…h_source_PIT

Hisoka-X

Thanks @CosmosNi . LGTM if ci passes.

[Feature][connector-elasticsearch] elasticsearch source support PIT

4050dcd

github-actions Bot added connectors-v2 e2e labels Apr 11, 2025

[Feature][connector-elasticsearch] elasticsearch source support PIT

7d34224

github-actions Bot added the document label Apr 11, 2025

hailin0 requested a review from Copilot April 11, 2025 12:25

Copilot AI reviewed Apr 11, 2025

View reviewed changes

Hisoka-X reviewed Apr 12, 2025

View reviewed changes

[Feature][connector-elasticsearch] elasticsearch source support PIT

25b647f

Hisoka-X reviewed Apr 14, 2025

View reviewed changes

CosmosNi added 2 commits April 14, 2025 11:45

[Feature][connector-elasticsearch] elasticsearch source support PIT

1bb296f

[Feature][connector-elasticsearch] elasticsearch source support PIT

3969e90

CosmosNi requested a review from Hisoka-X April 14, 2025 06:01

Hisoka-X reviewed Apr 15, 2025

View reviewed changes

CosmosNi added 2 commits April 15, 2025 11:08

[Feature][connector-elasticsearch] elasticsearch source support PIT

856cd10

Merge remote-tracking branch 'upstream/dev' into feature_elasticsearc…

9e05e92

…h_source_PIT

Hisoka-X approved these changes Apr 15, 2025

View reviewed changes

github-actions Bot added approved reviewed labels Apr 15, 2025

hailin0 approved these changes Apr 16, 2025

View reviewed changes

hailin0 merged commit 948d588 into apache:dev Apr 16, 2025

	Map<String, String> sortField = new HashMap<>();
	sortField.put("order", "asc");
	sort.add(Collections.singletonMap("_shard_doc", "asc"));

Uh oh!

Conversation

CosmosNi commented Apr 11, 2025

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 11, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hailin0 commented Apr 11, 2025

Uh oh!

CosmosNi commented Apr 11, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hailin0 commented Apr 12, 2025

Uh oh!

CosmosNi commented Apr 12, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Hisoka-X left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants