Vertex AI Matching Engine - Vector Similarity Search 

See many customers moving to Matching Engine, they're all happy with it. Only time to update the index could be quicker. But I guess this depends on the requirements. Getting new embeddings into the index in real time is not possible. Though there are workarounds.

@@ml-engineer Sascha, do you think online inference is possible by running Cloud Run/ Cloud Functions on a Vertex AI endpoint, getting the embeddings and then submitting to Matching Engine ANN ? I was wondering if this may be a solution ....

Hi Rubens yes embeddings models are usually hosted with Vertex AI Endpoints or with Cloud Run (if you don't need a GPU). After you did the inference you need to tell the Matching Engine Index to take the new embeddings / vector into account, stored on Cloud Storage. And that's currently the bottleneck as this indexing process takes quite a long time.

I would like to know the answer to this one too. I don't see support for metadata as pinecone does.

​@@ArmandoCuevas-sx5cf Based on my further investigations over the past week the metadata filtering is restricted to string matching only with key/value pairs (so no comparators on numeric values) and the idea is to pair the matching engine IDs with another key-value store like Bigtable where you could possibly do further complex filtering-comparing this setup to to Pinecone or Qdrant and considering the costs, I don't think I would use Matching Engine

Hi Alex Hi Armando Matching Engine as Alex already said supports string matching on metadata. cloud.google.com/vertex-ai/docs/matching-engine/filtering Pinecone is indeed more flexible on this point.

@@alexchan5643 thanks a lot, that's helpful and you're right having metadata filtering availablo is a big advantage for Pinecone.

Hi Google released public endpoints it is no longer required to use a VPC network. Therefore you not necessarily need a Cloud Run service in front. Here is the documentation for the public Matching Engine endpoint: cloud.google.com/vertex-ai/docs/matching-engine/deploy-index-public In case you are still interested in the Cloud Run approach here I have a sample implementation for a image similarity matching solution github.com/SaschaHeyer/image-similarity-search/tree/main/query-service. The critical part is in the cloudbuild.yaml that contains the reference to the VPC network github.com/SaschaHeyer/image-similarity-search/blob/main/query-service/cloudbuild.yaml Let me know if that helps

Hi Mohammad I can recommend to use Vertex AI Vector Search / Matching Engines streaming capabilities. This way you can simply send new data via SDK to the vector database. Check out my sample repo to get you started github.com/SaschaHeyer/Real-Time-Deep-Learning-Vector-Similarity-Search It's the same process likes pinecones upsert.

documentation: matching-engine -> create-manage-index?hl=en#create-index

Hello Federicoph, That is indeed a good question that is not covered in the video nor the article =). The machine-type can be defined when deploying the index. Like mentioned in the documentation deploy_index But if you actually check the the gcloud command there is nothing documented cloud.google.com/sdk/gcloud/reference/alpha/ai/index-endpoints/deploy-index So I always fall back to the actual implementation and there you can see the deploy_index method is indeed accepting a machine type. See here github.com/googleapis/python-aiplatform/blob/90bb8ef3d675af62b7cc1f0d2fdf99b476e8dde5/google/cloud/aiplatform/matching_engine/matching_engine_index_endpoint.py#L542 In your use case you can set it to the smallest machine. Also because you only have 100 vectors I recommend to use the brute force algorithm. Let me know if that helps.

Quick appendix This is reflected in the API documentation as well cloud.google.com/vertex-ai/docs/reference/rest/v1/projects.locations.indexEndpoints/deployIndex see the request body cloud.google.com/vertex-ai/docs/reference/rest/v1/projects.locations.indexEndpoints#DeployedIndex especially this part cloud.google.com/vertex-ai/docs/reference/rest/v1/DedicatedResources

Hi no they can only be set during index creation it is not possible to update them. That's because a update would require a full index re-build which in the end is the same as creating a new index.

Yes there is langchain support for Matching Engine. The Google team implemented it a few weeks ago. github.com/hwchase17/langchain/pull/3104 Currently writing an article on it that will be published in the next days. from langchain.vectorstores.matching_engine import MatchingEngine vector_store = MatchingEngine.from_components( index_id=INDEX_NAME, region=MATCHING_ENGINE_REGION, embedding=embeddings_llm, project_id=PROJECT_ID, endpoint_id=ENDPOINT_NAME, gcs_bucket_name=DOCS_BUCKET) relevant_documentation=vector_store.similarity_search(question, k=8)

To send your request to the Matching Engine you need to be "inside" of the network. This can be complicated if you want to integrate it into an service that is running outside of that network. There is one simple approach that I really like. You can implement a Cloud Run Service that is part of the VPC network and takes your requests. This Cloud Run service can be also reached from outside the network. I have implemented exactly that in one of my other articles medium.com/google-cloud/recommendation-systems-with-deep-learning-69e5c1772571

What do you mean with schema? The Matching engine does not need a schema as we just provide the embeddings. Can you rephrase 🙂 in case I misunderstood your question.

@@ml-engineer aah got it , so the embedding input vector file has to be in the format{"id":"string","embedding":[vector]}

are you referring to this section of the video? ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-KMTApM5ajAw.html

@@ml-engineer yes precisely.

Yes exactly. Alternative file formats are CSV or AVRO

Hi Lucas Pinecone is also a highly recommended product. Or you can go open source with Faiss or Annoy but this requires you to take care of the infrastructure yourself. If you want a similarity search I recommend to either go with Matching Engine or Pinecone.

Hello Anjana are you refering to the filtering functionality? cloud.google.com/vertex-ai/docs/matching-engine/filtering

@@ml-engineer Yes, the same. Could you tell me how to incoorporate that into a query? I have an idea on how to have it inserted in the index, but would be good if you could give a clarity there as well. Thanks for replying :)

Got it. Yeah it's not well documented. But if you check the proto file you can get an understanding on how you can use it when querying the matching engine. As simple as applying it to your query request namespace = match_service_pb2.Namespace() namespace.name = 'color' namespace.allow_tokens.append('red') request = match_service_pb2.MatchRequest() request.deployed_index_id = DEPLOYED_INDEX_ID request.restricts.append(namespace)

@@ml-engineer This worked!! I had not looked at the proto file in much detail, thank you so much😃

@@anjanak8303 perfect. I add this to the article I hope we can help more people having the same question.

Hi I have written a comparison article between Cloud Composer and Vertex AI Pipelines to orchestrate ML pipelines. medium.com/google-cloud/vertex-ai-pipelines-vs-cloud-composer-for-orchestration-4bba129759de In general, if you want to use Vertex AI's capabilities as part of Cloud Composer, you can simply use the Vertex AI SDK as part of your composer tasks. Though I would highly recommend switching to Vertex AI Pipelines: ru-vid.com/video/%D0%B2%D0%B8%D0%B4%D0%B5%D0%BE-gtVHw5YCRhE.html

@@ml-engineer my requirement is to orchastretate vertex ai pipeline through cloud composer via Terraform code

¯\_(ツ)_/¯

Excuse me, you briefly mention that there are workarounds to simulate real time indexing. Could you explain the options for this? Or point me to some docs. Beginner trying to work it out for a project :)

The feasibility of the solution depends on the number of new vectors you get between the indexing updates. You store the vectors that need to be indexes in the next index update round in Memorystore for fast millisecond access. Build a for example Cloud Run application that takes the vectors from Memorystore and calculate the distance yourself (it's just simple math). The same Cloud Run application also calls the Matching Engine. And in the end you combine the results if the distance is in your desired range. On long term I hope for quicker index updates using GPUs.

@@elijahdecalmer613 Google added streaming support which makes it easier to get new vectors into the index.

@@ml-engineer if I want to update the new indexes in real-time can I just add new files in the GCS folder where your vector data is stored or do I have to add it Programmatically?

Hi Majid You can define the number of neighbors you you want to retrieve when calling the matching endpoint response = my_index_endpoint.match( deployed_index_id=DEPLOYED_INDEX_ID, queries=..., num_neighbors=NUM_NEIGHBOURS )

@@ml-engineer But in this tutorial you don't query this way. Instead match_service.proto is used which has a field num_neighbours = 3. But always returns 10 neighbours

@@majidalikhani2765 yes Google changed the way to get matching results, since o released the video. No need for complex .proto file handling anymore. Just us the SDK in the same way like creating the index much easier.

Will add the new way to the notebook in the next few days and publish an additional video.

@@ml-engineer Google's documentation very poor smh. I got it working via the sdk. thanks

Hi Ayush what do you mean with your endpoint question? Recording 4 new videos about Generative AI on Google Cloud at the moment will be released in the next weeks.

You need a VPC network this is a requirement to run queries against the index. Batch prediction over the complete index is not possible. This is due to the nature of the index you only get the k_nearest neighbors.

probably getting a bit greedy here, do you have plans to upload tutorial on two tower?

no worries, love all the comments here on youtube. Yes I release an article next week It's a deep dive on how to use the two-tower algorithm + Matching Engine + Vertex AI Pipelines to build a Deep Learning Recommendation Engine.

@@ml-engineer Thanks a ton

The article is published medium.com/google-cloud/recommendation-systems-with-deep-learning-69e5c1772571

Hello Niladri thanks a lot. (Anjana in the comments had a similar question about allow tokens.) Your JSON and query are definitely correct. I don't see any issues here. Did you make sure to update the index after adding the restricts filter into the JSON?

@@ml-engineer Hey Thanks for the quick reply. Yes, I have completely overwritten the index twice now (just to be sure) but it still doesn't seem to work. Is there any requirement for the token to be on the allow list as well?

Deny alone is without allow possible. see documentation: cloud.google.com/vertex-ai/docs/matching-engine/filtering#denylist {} // empty set matches everything {red} // only a 'red' token {blue} // only a 'blue' token {orange} // only an 'orange' token {red, blue} // multiple tokens {red, !blue} // deny the 'blue' token {red, blue, !blue} // a weird edge-case {!blue} // deny-only (similar to empty-set) See the following description: When a query denylists a token, matches are excluded for any datapoint that has the denylisted token. If a query namespace has only denylisted tokens, all points not explicitly denylisted, match, in exactly the same way that an empty namespace matches with all points. So the issues has to be somewhere else

@@ml-engineer Unfortunately, this does not seem to be working :( I have looked at my JSON multiple times now and tried different variations but it still fails. Do you have any ideas?

Hi @Nilandri, did you solve your problem? If yes, can you explain how please? If no, I got the same problem with the allow-list-tokens, I opened an issue on github and on googlecloudcommunity. Thank you in advance! @ML Engineer