[Improve][Docs] Update Documentation for Storing IMAP in S3 with Minio or other S3 Compatible Storage

[krutoileshii]

Search before asking

• I had searched in the feature and found no similar feature requirement.

Description

SeaTunnel’s IMAP source currently persists state (e.g., offsets, UID tracking, checkpoints) in HDFS. This proposal adds support for S3-compatible object storage as an alternative state backend.

Usage Scenario

We use S3 compatible storage and would like to have an ability to store imap in it as we do not have access to HDFS.

Related issues

Proposed Behavior

• Support storing IMAP source state in:

  • S3
  • MinIO
  • Other S3-compatible object storage

• Configuration aligns with existing SeaTunnel S3 settings:

  • Endpoint
  • Bucket
  • Path/prefix
  • Credentials

• State semantics remain consistent with the current HDFS implementation (fault tolerance and delivery guarantees unchanged).

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[krutoileshii]

I think i got this working with existing config:

map:
   engine*:
     map-store:
       enabled: true
       initial-mode: EAGER
       factory-class-name: org.apache.seatunnel.engine.server.persistence.FileMapStoreFactory
       properties:
         type: hdfs
         namespace: /seatunnel/engine/
         clusterName: seatunnel
         storage.type: s3
         s3.bucket: s3a://your-bucket
         fs.defaultFS: s3a://your-bucket
         fs.s3a.endpoint: your-endpoint
         fs.s3a.path.style.access: "true"
         fs.s3a.access.key: access_key
         fs.s3a.secret.key: secret_key
         fs.s3a.aws.credentials.provider: org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider

[nancaldw]

When IMAP data is stored in MinIO, how can I set up scheduled cleanup? In actual production scenarios, when the metadata stored in MinIO grows too large, SeaTunnel fails to start after a complete cluster restart due to the excessive metadata size.

[DanielLeens]

Thanks for adding the production detail.

The original issue here was about whether MinIO / S3-compatible storage could be used for this path at all. The new question about metadata growth, scheduled cleanup, and cluster restart failure looks like a different operational problem, so I do not think this closed thread is the best place to track it.

Could you please open a new issue for that runtime behavior and include:

1. the exact SeaTunnel version
2. the current map-store / MinIO-related config with sensitive values masked
3. the startup error stack after a full cluster restart
4. the approximate metadata size or object count when startup begins to fail
5. whether the cleanup concern is about old abandoned job metadata, active checkpoint state, or both

Those details will help us determine whether this is a missing retention mechanism, a startup-scan scalability issue, or an environment-specific configuration problem.