Anyscale

Use Anyscale’s serverless endpoints for Llama, Mistral, and other open-source models through Portkey.

Quick Start

Get started with Anyscale in under 2 minutes:

Python

from portkey_ai import Portkey

# 1. Install: pip install portkey-ai
# 2. Add @anyscale provider in model catalog
# 3. Use it:

portkey = Portkey(api_key="PORTKEY_API_KEY")

response = portkey.chat.completions.create(
    model="@anyscale/mistralai/Mistral-7B-Instruct-v0.1",
    messages=[{"role": "user", "content": "Hello!"}]
)

print(response.choices[0].message.content)

Add Provider in Model Catalog

Before making requests, add Anyscale to your Model Catalog:

1. Go to Model Catalog → Add Provider
2. Select Anyscale
3. Enter your Anyscale API key
4. Name your provider (e.g., anyscale)

Complete Setup Guide

See all setup options and detailed configuration instructions

---

Supported Models

Anyscale provides serverless access to popular open-source models:

Model	Description
mistralai/Mistral-7B-Instruct-v0.1	Mistral 7B instruction model
meta-llama/Llama-2-70b-chat-hf	Llama 2 70B chat
meta-llama/Llama-2-13b-chat-hf	Llama 2 13B chat
codellama/CodeLlama-34b-Instruct-hf	Code Llama 34B
mistralai/Mixtral-8x7B-Instruct-v0.1	Mixtral 8x7B

Check Anyscale’s documentation for the complete model list.

---

Next Steps

Gateway Configs

Add fallbacks, load balancing, and more

Observability

Monitor and trace your Anyscale requests

Prompt Library

Manage and version your prompts

Metadata

Add custom metadata to requests

For complete SDK documentation:

SDK Reference

Complete Portkey SDK documentation