Connecting your S3 bucket

ManyVector stores all vector data — HNSW indexes, inverted indexes, and raw vector objects — in your own S3 bucket. This guide covers the complete setup: bucket creation, IAM policy, encryption, and access logging.

Why your own bucket?

You own the bytes. ManyVector reads from and writes to your bucket using credentials you provide. You can:

• Inspect the raw objects at any time
• Revoke ManyVector’s access without losing data
• Manage retention, lifecycle policies, and compliance independently
• Pay AWS directly for storage at published S3 prices — no markup

Step 1: Create an S3 bucket

Create a dedicated bucket for ManyVector. Use the AWS CLI or console:

aws s3api create-bucket \
  --bucket my-vector-store \
  --region us-east-1

For non-us-east-1 regions, add the --create-bucket-configuration flag:

aws s3api create-bucket \
  --bucket my-vector-store \
  --region eu-west-1 \
  --create-bucket-configuration LocationConstraint=eu-west-1

Recommendation: Enable versioning for your bucket. This gives you the ability to recover from accidental deletes:

aws s3api put-bucket-versioning \
  --bucket my-vector-store \
  --versioning-configuration Status=Enabled

Step 2: Create an IAM policy

ManyVector needs a minimal set of S3 permissions. Create a policy with least-privilege access:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ManyVectorBucketAccess",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:ListBucket",
        "s3:GetBucketLocation"
      ],
      "Resource": [
        "arn:aws:s3:::my-vector-store",
        "arn:aws:s3:::my-vector-store/*"
      ]
    }
  ]
}

Save this as manyvector-policy.json and create the policy:

aws iam create-policy \
  --policy-name ManyVectorS3Access \
  --policy-document file://manyvector-policy.json

Step 3: Create an IAM user or role

For cloud-hosted ManyVector: Create an IAM user with programmatic access:

aws iam create-user --user-name manyvector-access
aws iam attach-user-policy \
  --user-name manyvector-access \
  --policy-arn arn:aws:iam::ACCOUNT_ID:policy/ManyVectorS3Access

# Generate access keys
aws iam create-access-key --user-name manyvector-access

For self-hosted ManyVector in AWS: Use an IAM role with an instance profile or EKS IRSA — keyless authentication is more secure:

# Create a role that can be assumed by EC2 or EKS
aws iam create-role \
  --role-name ManyVectorRole \
  --assume-role-policy-document file://trust-policy.json

aws iam attach-role-policy \
  --role-name ManyVectorRole \
  --policy-arn arn:aws:iam::ACCOUNT_ID:policy/ManyVectorS3Access

Step 4: Enable encryption

Enable server-side encryption with an AWS managed key (SSE-S3):

aws s3api put-bucket-encryption \
  --bucket my-vector-store \
  --server-side-encryption-configuration '{
    "Rules": [{"ApplyServerSideEncryptionByDefault": {"SSEAlgorithm": "AES256"}}]
  }'

For stricter compliance (e.g., HIPAA, FedRAMP), use SSE-KMS with a customer-managed key:

aws s3api put-bucket-encryption \
  --bucket my-vector-store \
  --server-side-encryption-configuration '{
    "Rules": [{"ApplyServerSideEncryptionByDefault": {
      "SSEAlgorithm": "aws:kms",
      "KMSMasterKeyID": "arn:aws:kms:REGION:ACCOUNT_ID:key/KEY_ID"
    }}]
  }'

Step 5: Enable access logging

Enable S3 server access logs to capture every read and write ManyVector performs:

# Create a separate bucket for logs
aws s3api create-bucket --bucket my-vector-store-logs --region us-east-1

# Enable access logging
aws s3api put-bucket-logging \
  --bucket my-vector-store \
  --bucket-logging-status '{
    "LoggingEnabled": {
      "TargetBucket": "my-vector-store-logs",
      "TargetPrefix": "access-logs/"
    }
  }'

Step 6: Configure ManyVector

from manyvector import ManyVector

client = ManyVector(
    api_key="mv-...",
    backend={
        "provider": "s3",
        "bucket": "my-vector-store",
        "region": "us-east-1",
        "access_key_id": "AKIA...",        # or use IAM role (no keys needed)
        "secret_access_key": "...",
    }
)

# Verify connectivity
print(client.ping_backend())  # {"status": "ok", "backend": "s3"}

Storage class selection

Namespace type	Recommended storage class	Cost (us-east-1)
Hot (queried constantly)	S3 Standard	$0.023/GB/mo
Warm (queried daily)	S3 Standard-IA	$0.0125/GB/mo
Cold (queried rarely)	S3 Glacier Instant Retrieval	$0.004/GB/mo

ManyVector supports per-namespace storage class configuration:

ns = client.create_namespace(
    name="archive-2024",
    dimensions=1536,
    storage_class="STANDARD_IA",
)

Monitoring costs

Your S3 bucket costs are separate from ManyVector query costs. Enable S3 Storage Lens or use AWS Cost Explorer filtered on your bucket for full visibility into what ManyVector is storing.

Typical index overhead: raw vectors are stored directly; the HNSW graph adds ~20-50% overhead depending on M parameter. A billion 1536-dimensional float32 vectors is ~5.7TB of raw data, ~7-8.5TB with HNSW overhead.