The .servers accessor provides methods for creating, listing, starting, and stopping servers.
Some methods like .match() and the parameter classes below are backend-specific and may not be available on all backends.
Creating servers
The .create() method creates a new server configuration. By default, it downloads the model file (if not already downloaded) and selects a random, unused port for the server.
If a server with the specified configuration is already running, the existing configuration is returned, and no new server is created.
from anaconda_ai import AnacondaAIClient
client = AnacondaAIClient()
server = client.servers.create('OpenHermes-2.5-Mistral-7B/Q4_K_M')
from anaconda_ai import AnacondaAIClient
client = AnacondaAIClient()
quant = client.models.get('OpenHermes-2.5-Mistral-7B').get_quantization('Q4_K_M')
server = client.servers.create(quant)
To look up an existing running server rather than create one, use .get(). Pass the server’s name or UUID as a string:server = client.servers.get('my-server-name')
Server configuration parameters
The .create() method accepts an extra_options parameter to customize server behavior:
def create(
model: str,
download_if_needed: bool = True,
extra_options: Optional[Dict[str, Any]] = None,
show_progress: bool = True,
console: Optional[Console] = None
) -> Server:
Server configuration options are backend-specific. The parameters below apply to the ai-navigator and anaconda-desktop backends.
server = client.servers.create(
'OpenHermes-2.5-Mistral-7B/Q4_K_M',
extra_options={
'host': '127.0.0.1',
'port': 9999,
'ctx_size': 4096,
'n_gpu_layers': 20,
'temp': 0.7,
'top_p': 0.9
},
download_if_needed=False
)
Managing servers
New servers are not automatically started when their configuration is created. You can start or stop a server using the following methods:
server.start()
server.stop()
# Start or stop the server object returned from .create
client.servers.start(server)
client.servers.stop(server)
This approach automatically starts and stops the server: with client.servers.create('OpenHermes-2.5-Mistral-7B/Q4_K_M') as server:
openai_client = server.openai_client()
# Make requests to the server
Server attributes
