The .servers accessor provides methods for creating, listing, starting, and stopping servers.
Some methods like .match() and the parameter classes below are backend-specific and may not be available on all backends.
Creating servers
The .create() method creates a new server configuration. By default, it downloads the model file (if not already downloaded) and selects a random, unused port for the server.
If a server with the specified configuration is already running, the existing configuration is returned, and no new server is created.
from anaconda_ai import AnacondaAIClient
client = AnacondaAIClient()
server = client.servers.create('OpenHermes-2.5-Mistral-7B/Q4_K_M')
from anaconda_ai import AnacondaAIClient
client = AnacondaAIClient()
quant = client.models.get('OpenHermes-2.5-Mistral-7B').get_quantization('Q4_K_M')
server = client.servers.create(quant)
from anaconda_ai import AnacondaAIClient
client = AnacondaAIClient()
# Get existing server ID from client.servers.list()
server = client.servers.create('server/my-server-name') # ai-catalyst
# or
server = client.servers.create('server/abc-123-uuid') # ai-navigator/anaconda-desktop
ai-catalyst supports named servers. Use server/<server-name> to reference existing servers by name.ai-navigator and anaconda-desktop:
- Use
server/<uuid> to reference existing servers by UUID (obtained from client.servers.list())
- Have automatic server matching. When calling
client.servers.create('<model>/<quant>', extra_options=...), these backends automatically search for running servers with the same model and options. If a match is found, the existing server is returned instead of creating a new one.
Server configuration parameters
The .create() method accepts an extra_options parameter to customize server behavior:
def create(
model: str,
download_if_needed: bool = True,
extra_options: Optional[Dict[str, Any]] = None,
show_progress: bool = True,
console: Optional[Console] = None
) -> Server:
Server configuration options are backend-specific. The parameters below apply to the ai-navigator and anaconda-desktop backends.
server = client.servers.create(
'OpenHermes-2.5-Mistral-7B/Q4_K_M',
extra_options={
'host': '127.0.0.1',
'port': 9999,
'ctx_size': 4096,
'n_gpu_layers': 20,
'temp': 0.7,
'top_p': 0.9
},
download_if_needed=False
)
Managing servers
New servers are not automatically started when their configuration is created. You can start or stop a server using the following methods:
server.start()
server.stop()
# Start or stop the server object returned from .create
client.servers.start(server)
client.servers.stop(server)
This approach automatically starts and stops the server: with client.servers.create('OpenHermes-2.5-Mistral-7B/Q4_K_M') as server:
openai_client = server.openai_client()
# Make requests to the server
Server attributes
