[Feature][connector-file-local]localfile supports splitting and parallel reading of large files of csv, text, and json#10142
Conversation
|
LGTM |
corgy-w
left a comment
There was a problem hiding this comment.
First, you should update the documentation.
After enabling enable_split_file, the original skip_header behavior for the file no longer works as before. Is there any way to keep it compatible?
|
| URL url = getClass().getClassLoader().getResource("test_split_csv_data.csv"); | ||
| String realPath = Paths.get(url.toURI()).toString(); | ||
| final List<FileSourceSplit> splits = localFileSplitStrategy.split("test.table", realPath); | ||
| Assertions.assertEquals(4, splits.size()); |
| URL url = getClass().getClassLoader().getResource("test_split_csv_data.csv"); | ||
| String realPath = Paths.get(url.toURI()).toString(); | ||
| final List<FileSourceSplit> splits = localFileSplitStrategy.split("test.table", realPath); | ||
| Assertions.assertEquals(2, splits.size()); |
| import java.nio.file.Paths; | ||
| import java.util.List; | ||
|
|
||
| public class SplitFileStrategyTest { |
There was a problem hiding this comment.
Add escape characters to test: when the field content contains a newline character, the line is not truncated.
e.g(csv):
1,t1,"a
b"
2,t2,"c
d"There was a problem hiding this comment.
Add escape characters to test: when the field content contains a newline character, the line is not truncated.
e.g(csv):
1,t1,"a b" 2,t2,"c d"
please see: #10185
corgy-w
left a comment
There was a problem hiding this comment.
Your change not only involves localfile, but also other files related documents need to be changed.
Connectors of different file types have different logic. Currently, only local implements AccordingToSplitSizeSplitStrategy, so only the localfile document is modified. |
Issue
#10129
Check list