[Improve][Transform] Sql transform support inner strucy query#6484
Conversation
1838568 to
3e6f188
Compare
hailin0
left a comment
There was a problem hiding this comment.
Thanks for your contribution, need to update the documentation?
|
LGTM waiting for ci passed |
Updated |
| @DisabledOnContainer( | ||
| value = {}, | ||
| type = {EngineType.SPARK}) |
There was a problem hiding this comment.
i meet an exception on spark engine, so i disable it. will check the failed reason
There was a problem hiding this comment.
Seem like the spark translation have some problem when use map type. You can create an issue and add it into your disabledReason.
| try { | ||
| String columnName = columnExp.getColumnName(); | ||
| int idx = inputRowType.indexOf(columnName); | ||
| return inputFields[idx]; | ||
| } catch (IllegalArgumentException e) { |
There was a problem hiding this comment.
Can you add a new indexOf method in SeaTunnelRowType which no IllegalArgumentException exception is thrown. Then use the result of the index to determine whether there is a corresponding fieldname. Because data processing is performed through try catch at now, the performance is particularly bad.
| try { | ||
| String columnName = columnExp.getColumnName(); | ||
| return inputRowType.getFieldType(inputRowType.indexOf(columnName)); | ||
| } catch (IllegalArgumentException e) { |
| int idx = ((SeaTunnelRowType) parDataType).indexOf(columnNames[i], false); | ||
| if (idx == -1) { | ||
| throw new IllegalArgumentException( | ||
| String.format("can't find field [%s]", fullyQualifiedName)); | ||
| } | ||
| parDataType = ((SeaTunnelRowType) parDataType).getFieldType(idx); | ||
| res = parRowValues.getFields()[idx]; |
There was a problem hiding this comment.
public int indexOf(String fieldName, boolean throwExceptionWhenNotFound) {
for (int i = 0; i < fieldNames.length; i++) {
if (fieldNames[i].equals(fieldName)) {
return i;
}
}
if (throwExceptionWhenNotFound) {
throw new IllegalArgumentException(String.format("can't find field [%s]", fieldName));
} else {
return -1;
}
}
``` Logical conflict？
cc @hailin0 @Hisoka-X
There was a problem hiding this comment.
check the field name is in this struct. if not exist, choose throw exception or return -1 index let's the caller handle.
There was a problem hiding this comment.
Thanks @zhilinli123 for ping me. I think it's different. indexOf only throw sub fieldname. The fullyQualifiedName and columnNames[i] not equals. Like field user.age, we should throw user.age not age.
Purpose of this pull request
close #6468
Now the struct query will be recognized as
table_name.column_name, I am not change this because i think there are someone still use this query pattern. just add a backfill logical to handle it.For map stuct, it should be the latest struct.
Does this PR introduce any user-facing change?
No
How was this patch tested?
add e2e test
Check list
New License Guide
release-note.