close
Skip to content

[Improve][Redis] Redis reader use scan cammnd instead of keys, single mode reader/writer support batch#7087

Merged
hailin0 merged 6 commits into
apache:devfrom
FuYouJ:feature_redis
Jul 3, 2024
Merged

[Improve][Redis] Redis reader use scan cammnd instead of keys, single mode reader/writer support batch#7087
hailin0 merged 6 commits into
apache:devfrom
FuYouJ:feature_redis

Conversation

@FuYouJ

@FuYouJ FuYouJ commented Jun 30, 2024

Copy link
Copy Markdown
Contributor

Purpose of this pull request

Does this PR introduce any user-facing change?

  • reader use scan cammnd instead of keys to avoid blocking the database from reading data
  • In single mode, supports batch read and batch write

How was this patch tested?

Check list

@FuYouJ FuYouJ changed the title [improve][redis]issue#7030,issue#7085,redis reader using scan replace keys command, writer support batch [improve][redis]issue#7030,issue#7085,redis reader use scan cammnd instead of keys, single mode reader/writer support batch Jun 30, 2024
@Hisoka-X Hisoka-X changed the title [improve][redis]issue#7030,issue#7085,redis reader use scan cammnd instead of keys, single mode reader/writer support batch [Improve][Redis] Redis reader use scan cammnd instead of keys, single mode reader/writer support batch Jul 1, 2024
@hailin0 hailin0 requested a review from TyrantLucifer July 1, 2024 02:48
Comment thread docs/en/connector-v2/sink/Redis.md Outdated
Comment on lines +87 to +91
>
> ### batch_size [int]
>
> Ensure the batch write size in single-machine mode; no guarantees in cluster mode.
>

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
>
> ### batch_size [int]
>
> Ensure the batch write size in single-machine mode; no guarantees in cluster mode.
>
### batch_size [int]
Ensure the batch write size in single-machine mode; no guarantees in cluster mode.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you. I was careless here

Comment thread docs/en/connector-v2/source/Redis.md Outdated

### batch_size [int]

indicates the number of keys to attempt to return per iteration.default 10

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
indicates the number of keys to attempt to return per iteration.default 10
indicates the number of keys to attempt to return per iteration, default 10

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@FuYouJ FuYouJ requested a review from Hisoka-X July 1, 2024 14:57

@Hisoka-X Hisoka-X left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall LGTM. Thanks @FuYouJ

@hailin0 hailin0 merged commit be37f05 into apache:dev Jul 3, 2024
hawk9821 pushed a commit to hawk9821/seatunnel that referenced this pull request Jul 10, 2024
Thomas-HuWei pushed a commit to zilliztech/seatunnel that referenced this pull request Jul 10, 2024
hawk9821 pushed a commit to hawk9821/seatunnel that referenced this pull request Jul 13, 2024
chaorongzhi pushed a commit to chaorongzhi/seatunnel that referenced this pull request Aug 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature][Redis]Sink support batch write [Improve][Connector-V2][Redis]using scan relpace keys operation command

3 participants