I'm getting fullness errors when upserting. What does it mean and how do I fix it?

While your index can still serve queries, new upserts may fail as the capacity becomes exhausted. If you need to scale your environment to accommodate more vectors, you can modify your existing index and scale it vertically or create a new index and scale horizontally. This article will describe both methods and how to scale your index effectively. 

Vertical scaling

Scaling vertically is fast and involves no downtime. This is a good choice when you can't pause upserts and must continue serving traffic. It also allows you to double your capacity instantly. However, there are a couple of considerations. You can scale to x2, x4, and x8, which means you are doubling your capacity at each step. Moving up to a new capacity will effectively double the number of pods used at each step. This also doubles the cost. If you need to scale but not by double, then consider horizontal scaling. The number of base pods specified when you initially created the pod also is static and cannot be changed. For example, if you start with ten pods of p1.x1 and vertically scale to p1.x2, this will equate to 20 pods worth of usage. Pod types (performance versus storage pods) also cannot be changed with vertical scaling. If you want to change your pod type while scaling, then horizontal scaling is the better option. So that you know, you can only scale up and cannot scale back down at this time.

See also Vertical Scaling

Horizontal scaling

Horizontal scaling is now possible using our "Collections" feature. Creating a collection is essentially an immutable snapshot of your index in time. Only the data is stored, and the original index definition is not stored. When you create an index from a collection, you will define the new definition. This allows users to scale the base pod count horizontally without scaling vertically. The main advantage here is that you can scale incrementally instead of doubling capacity as with vertical scaling. Also, you can redefine pod types if you are experimenting or know you need to use a different pod type, such as our performance-optimized pods versus our storage-optimized pods. Another advantage of this method is that you can change your metadata configuration to redefine metadata fields as indexed or stored-only. This is important when tuning your index for the best throughput. 

Here are the general steps to make a copy of your index and re-create a brand new index where you can change pod type, pod count, metadata configuration, replicas, and all typical parameters when creating a new collection. 

General steps to horizontally scale:
1.) Pause upserts
2.) Create a ‘collection’ from the current index
3.) Create an index from the collection with new parameters (pod type, pod count, metadata config, etc.)
4.) Continue upserts to the newly created index (note, the endpoint has likely changed)
5.) Delete the old index if desired

See also  Collections

Final notes

Future updates to collections will allow import/export between S3 and GCS blob storage, write streaming, and bulk data upload directly to collections.