[Fix][Transform-v2] Fix custom udf throw exception by hawk9821 · Pull Request #9195 · apache/seatunnel

hawk9821 · 2025-04-17T08:41:29Z

Purpose of this pull request

custom udf throw Exception. remove seatunnel-transform-v2.jar from connecots to lib solve the problem .

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/seatunnel/transform/sql/zeta/ZetaUDF
	at java.lang.ClassLoader.defineClass1(Native Method)
	at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
	at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
	at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
	at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
	at java.security.AccessController.doPrivileged(Native Method)
	at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
	at org.apache.seatunnel.engine.common.loader.SeaTunnelBaseClassLoader.loadClassWithoutExceptionHandling(SeaTunnelBaseClassLoader.java:56)
	at org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader.loadClassWithoutExceptionHandling(SeaTunnelChildFirstClassLoader.java:97)
	at org.apache.seatunnel.engine.common.loader.SeaTunnelBaseClassLoader.loadClass(SeaTunnelBaseClassLoader.java:47)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
	at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:370)
	at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
	at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
	at java.lang.Iterable.forEach(Iterable.java:74)
	at org.apache.seatunnel.transform.sql.zeta.ZetaSQLEngine.init(ZetaSQLEngine.java:88)
	at org.apache.seatunnel.transform.sql.SQLTransform.open(SQLTransform.java:94)
	at org.apache.seatunnel.transform.sql.SQLTransform.tryOpen(SQLTransform.java:103)
	at org.apache.seatunnel.transform.sql.SQLTransform.transformTableSchema(SQLTransform.java:115)
	at org.apache.seatunnel.transform.common.AbstractSeaTunnelTransform.transformCatalogTable(AbstractSeaTunnelTransform.java:69)
	at org.apache.seatunnel.transform.common.AbstractSeaTunnelTransform.getProducedCatalogTable(AbstractSeaTunnelTransform.java:54)
	at org.apache.seatunnel.transform.common.AbstractMultiCatalogTransform.lambda$new$3(AbstractMultiCatalogTransform.java:86)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
	at org.apache.seatunnel.transform.common.AbstractMultiCatalogTransform.<init>(AbstractMultiCatalogTransform.java:88)
	at org.apache.seatunnel.transform.common.AbstractMultiCatalogFlatMapTransform.<init>(AbstractMultiCatalogFlatMapTransform.java:33)
	at org.apache.seatunnel.transform.sql.SQLMultiCatalogFlatMapTransform.<init>(SQLMultiCatalogFlatMapTransform.java:32)
	at org.apache.seatunnel.transform.sql.SQLTransformFactory.lambda$createTransform$0(SQLTransformFactory.java:51)
	at org.apache.seatunnel.api.table.factory.FactoryUtil.createAndPrepareMultiTableTransform(FactoryUtil.java:445)
	at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseTransform(MultipleTableJobConfigParser.java:459)
	at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parseTransforms(MultipleTableJobConfigParser.java:408)
	at org.apache.seatunnel.engine.core.parse.MultipleTableJobConfigParser.parse(MultipleTableJobConfigParser.java:232)
	at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.getLogicalDag(ClientJobExecutionEnvironment.java:123)
	at org.apache.seatunnel.engine.client.job.ClientJobExecutionEnvironment.execute(ClientJobExecutionEnvironment.java:191)
	at org.apache.seatunnel.core.starter.seatunnel.command.ClientExecuteCommand.execute(ClientExecuteCommand.java:165)
	at org.apache.seatunnel.core.starter.SeaTunnel.run(SeaTunnel.java:40)
	at org.apache.seatunnel.core.starter.seatunnel.SeaTunnelClient.main(SeaTunnelClient.java:34)
Caused by: java.lang.ClassNotFoundException: org.apache.seatunnel.transform.sql.zeta.ZetaUDF
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 49 more

Does this PR introduce any user-facing change?

How was this patch tested?

e2e case : ExampleUdfIT

Check list

If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update plugin-mapping.properties and add new connector information in it
2. Update the pom file of seatunnel-dist
3. Add ci label in label-scope-conf
4. Add e2e testcase in seatunnel-e2e
5. Update connector plugin_config
Update the release-note.

hailin0 · 2025-04-17T10:56:50Z

+                <include>org.apache.seatunnel:seatunnel-transforms-v2:jar</include>
+            </includes>
+            <outputFileNameMapping>${artifact.file.name}</outputFileNameMapping>
+            <outputDirectory>/lib</outputDirectory>


Why? This conflicts with pr #7218

pr #7218 does not consider the scenarios of custom udf , moved seatunnel-transform-v2.jar to connectors will throw exception.

The core reason is NoClassDefFoundError org.apache.seatunnel.transform.sql.zeta.ZetaUDF. It would be more appropriate to move interface org.apache.seatunnel.transform.sql.zeta.ZetaUDF to the api module

cc @liunaijie @Hisoka-X

Copilot

Pull Request Overview

This pull request addresses a NoClassDefFoundError in custom UDFs by updating the import paths for ZetaUDF and relocating the transform UDF jar from the connectors directory to the lib directory.

Update import statements in several UDF-related classes to use the new org.apache.seatunnel.api.transform.sql.ZetaUDF package
Adjust the container copy logic to load the transform UDF jar from the lib directory
Update the API package for ZetaUDF accordingly

Reviewed Changes

Copilot reviewed 9 out of 13 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/sql/zeta/functions/udf/DesEncrypt.java	Updated ZetaUDF import to the new API package
seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/sql/zeta/functions/udf/DesDecrypt.java	Updated ZetaUDF import to the new API package
seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/sql/zeta/ZetaSQLType.java	Added import for ZetaUDF from the API package
seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/sql/zeta/ZetaSQLFunction.java	Added import for ZetaUDF from the API package
seatunnel-transforms-v2/src/main/java/org/apache/seatunnel/transform/sql/zeta/ZetaSQLEngine.java	Added import for ZetaUDF from the API package
seatunnel-e2e/seatunnel-transforms-v2-e2e/seatunnel-transforms-v2-udf/src/main/java/org/apache/seatunnel/e2e/transform/udf/ExampleUdf.java	Provides an example implementation for custom UDF testing
seatunnel-e2e/seatunnel-transforms-v2-e2e/seatunnel-transforms-v2-e2e-udf/src/test/java/org/apache/seatunnel/e2e/transform/udf/ExampleUdfIT.java	End-to-end integration test for the custom UDF
seatunnel-e2e/seatunnel-e2e-common/src/test/java/org/apache/seatunnel/e2e/common/util/ContainerUtil.java	Updated the container copy logic to move the UDF jar to the lib directory
seatunnel-api/src/main/java/org/apache/seatunnel/api/transform/sql/ZetaUDF.java	Changed the package to the new API namespace for ZetaUDF

Files not reviewed (4)

seatunnel-e2e/seatunnel-transforms-v2-e2e/pom.xml: Language not supported
seatunnel-e2e/seatunnel-transforms-v2-e2e/seatunnel-transforms-v2-e2e-udf/pom.xml: Language not supported
seatunnel-e2e/seatunnel-transforms-v2-e2e/seatunnel-transforms-v2-e2e-udf/src/test/resources/custom_udf.conf: Language not supported
seatunnel-e2e/seatunnel-transforms-v2-e2e/seatunnel-transforms-v2-udf/pom.xml: Language not supported

liunaijie · 2025-04-22T16:10:52Z

Thanks @hawk9821 point this issue, this is a bug, good catch.

It would be more appropriate to move interface org.apache.seatunnel.transform.sql.zeta.ZetaUDF to the api module

I tested with this fix, it works. But I don’t understand why these changes can be work. Can you give me some hint? Thanks.

BTW, when do this update, it can impact the user that has implment udf. it need update the udf lib, update the class full name, we need describe this in the document or release note (Breaking Change).

hawk9821 · 2025-04-24T00:52:54Z

Thanks @hawk9821 point this issue, this is a bug, good catch.

It would be more appropriate to move interface org.apache.seatunnel.transform.sql.zeta.ZetaUDF to the api module

I tested with this fix, it works. But I don’t understand why these changes can be work. Can you give me some hint? Thanks.

BTW, when do this update, it can impact the user that has implment udf. it need update the udf lib, update the class full name, we need describe this in the document or release note (Breaking Change).

The compatibility issue was indeed not considered. It has been fixed in the way of moving transform-v2.jar to the lib .

hailin0 · 2025-04-24T04:13:57Z

Need to delete transform from plugin-mapping.properties?
https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties #L148

Hisoka-X · 2025-04-29T02:06:12Z

 seatunnel.source.GraphQL = connector-graphql
 seatunnel.sink.GraphQL = connector-graphql
 seatunnel.sink.Aerospike = connector-aerospike
-


I think we can leave a demo about how to create plugin-mapping for new custom transform.

https://github.com/apache/seatunnel/blob/dev/docs/en/transform-v2/sql-udf.md

Add this into plugin-mapping.properties.

# For custom transforms, make sure to use the seatunnel.transform.[PluginIdentifier]=[JarPerfix] naming convention. For example: # seatunnel.transform.Sql = seatunnel-transforms-v2

Hisoka-X

LGTM. Thanks @hawk9821

github-actions Bot added core SeaTunnel core module e2e labels Apr 17, 2025

hawk9821 changed the title ~~Custom udf~~ [BUG][Transform-v2]Custom udf throw exception Apr 17, 2025

hailin0 reviewed Apr 17, 2025

View reviewed changes

hawk9821 force-pushed the custom_udf branch from ee93818 to 0ef3480 Compare April 17, 2025 13:36

github-actions Bot added Transform-v2 api and removed core SeaTunnel core module labels Apr 17, 2025

nielifeng requested a review from Copilot April 18, 2025 01:31

Copilot AI reviewed Apr 18, 2025

View reviewed changes

Hisoka-X changed the title ~~[BUG][Transform-v2]Custom udf throw exception~~ [Fix][Transform-v2] Fix custom udf throw exception Apr 21, 2025

Hisoka-X reviewed Apr 21, 2025

View reviewed changes

Comment thread seatunnel-api/src/main/java/org/apache/seatunnel/api/transform/sql/ZetaUDF.java Outdated

github-actions Bot added document CI&CD core SeaTunnel core module Zeta connectors-v2 Zeta Rest API labels Apr 22, 2025

hawk9821 force-pushed the custom_udf branch from a110274 to eebf18c Compare April 22, 2025 07:14

hailin0 mentioned this pull request Apr 22, 2025

SeaTunnel Community Meeting #9174

Closed

3 tasks

hawk9821 force-pushed the custom_udf branch from eebf18c to c81ae15 Compare April 23, 2025 04:27

github-actions Bot removed document CI&CD Zeta connectors-v2 Zeta Rest API api labels Apr 23, 2025

liunaijie previously approved these changes Apr 24, 2025

View reviewed changes

github-actions Bot added approved reviewed labels Apr 24, 2025

Hisoka-X approved these changes Apr 24, 2025

View reviewed changes

Comment thread seatunnel-dist/src/main/assembly/assembly-bin.xml

Hisoka-X requested changes Apr 24, 2025

View reviewed changes

Comment thread ...ansforms-v2/src/main/java/org/apache/seatunnel/transform/sql/zeta/functions/udf/ZetaUDF.java Outdated

github-actions Bot removed approved reviewed labels Apr 24, 2025

hawk9821 dismissed liunaijie’s stale review via de6dd79 April 28, 2025 00:42

hawk9821 force-pushed the custom_udf branch from de6dd79 to 73a6158 Compare April 28, 2025 15:10

Hisoka-X reviewed Apr 29, 2025

View reviewed changes

hawk9821 force-pushed the custom_udf branch 3 times, most recently from 65b6a2b to a59e21c Compare June 13, 2025 06:52

github-actions Bot added the dependencies Pull requests that update a dependency file label Jun 20, 2025

[Bug][Transform]fix custom udf throw exception

2f93b06

hawk9821 force-pushed the custom_udf branch from b394c00 to 2f93b06 Compare June 20, 2025 09:08

Hisoka-X approved these changes Jun 23, 2025

View reviewed changes

github-actions Bot added approved reviewed labels Jun 23, 2025

corgy-w approved these changes Jun 23, 2025

View reviewed changes

corgy-w merged commit 9777ecb into apache:dev Jun 23, 2025
6 checks passed

hawk9821 deleted the custom_udf branch June 27, 2025 01:53

chncaesar pushed a commit to chncaesar/seatunnel that referenced this pull request Jun 30, 2025

[Fix][Transform-v2] Fix custom udf throw exception (apache#9195)

1bb1ada

dybyte pushed a commit to dybyte/seatunnel that referenced this pull request Jul 23, 2025

[Fix][Transform-v2] Fix custom udf throw exception (apache#9195)

8089982

Uh oh!

Conversation

hawk9821 commented Apr 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose of this pull request

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

liunaijie commented Apr 22, 2025

Uh oh!

hawk9821 commented Apr 24, 2025

Uh oh!

Uh oh!

Uh oh!

hailin0 commented Apr 24, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Hisoka-X left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

hawk9821 commented Apr 17, 2025 •

edited

Loading