close
Skip to content

[Feature][Connector-V2] Fake Source support produce vector data#7401

Merged
hailin0 merged 5 commits into
apache:devfrom
corgy-w:support-vector-data-generation-fake
Aug 20, 2024
Merged

[Feature][Connector-V2] Fake Source support produce vector data#7401
hailin0 merged 5 commits into
apache:devfrom
corgy-w:support-vector-data-generation-fake

Conversation

@corgy-w

@corgy-w corgy-w commented Aug 14, 2024

Copy link
Copy Markdown
Contributor

Purpose of this pull request

issue : #7390

  1. The Milvus connector uses column-scale as dim, so that using fake to Milvus has to configure dimensions twice.
    eg:
 FakeSource {
      row.num = 10
      vector.dimension= 4 <---
      schema = {
           columns = [
           {
              name = book_intro
              type = float_vector
              columnScale = 4 <---
              comment = "vector"
           }
       ]
      }
  }
  1. Currently vector support is incomplete, based on Milvus mainly test floatVector

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

@corgy-w corgy-w requested a review from Hisoka-X August 19, 2024 01:59

@Hisoka-X Hisoka-X left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @corgy-w ! We are close to finish!

Comment on lines +328 to +330
case BINARY_VECTOR:
case BFLOAT16_VECTOR:
case FLOAT16_VECTOR:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.


public static final VectorType<Byte> VECTOR_BINARY_TYPE =
new VectorType<>(Byte.class, SqlType.BINARY_VECTOR);
public static final VectorType<ByteBuffer> VECTOR_BINARY_TYPE =

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this change compatible with older versions?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently first connector of binary vectors is implemented in this pr with no historical version problem

@hailin0 hailin0 merged commit 6937d10 into apache:dev Aug 20, 2024
@corgy-w corgy-w deleted the support-vector-data-generation-fake branch January 28, 2026 18:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants