Servers - Anaconda

The .servers accessor provides methods for creating, listing, starting, and stopping servers.

Some methods like .match() and the parameter classes below are backend-specific and may not be available on all backends.

Creating servers

The .create() method creates a new server configuration. By default, it downloads the model file (if not already downloaded) and selects a random, unused port for the server.

If a server with the specified configuration is already running, the existing configuration is returned, and no new server is created.

Create servers using one of the following approaches:

Using model string reference
Using QuantizedFile object

from anaconda_ai import AnacondaAIClient

client = AnacondaAIClient()
server = client.servers.create('OpenHermes-2.5-Mistral-7B/Q4_K_M')

from anaconda_ai import AnacondaAIClient

client = AnacondaAIClient()
quant = client.models.get('OpenHermes-2.5-Mistral-7B').get_quantization('Q4_K_M')
server = client.servers.create(quant)

To look up an existing running server rather than create one, use .get(). Pass the server’s name or UUID as a string:

server = client.servers.get('my-server-name')

Server configuration parameters

The .create() method accepts an extra_options parameter to customize server behavior:

def create(
    model: str,
    download_if_needed: bool = True,
    extra_options: Optional[Dict[str, Any]] = None,
    show_progress: bool = True,
    console: Optional[Console] = None
) -> Server:

Server configuration options are backend-specific. The parameters below apply to the ai-navigator and anaconda-desktop backends.

Common server options include: For example, to create a server with custom configuration:

server = client.servers.create(
    'OpenHermes-2.5-Mistral-7B/Q4_K_M',
    extra_options={
        'host': '127.0.0.1',
        'port': 9999,
        'ctx_size': 4096,
        'n_gpu_layers': 20,
        'temp': 0.7,
        'top_p': 0.9
    },
    download_if_needed=False
)

Managing servers

New servers are not automatically started when their configuration is created. You can start or stop a server using the following methods:

Using the server object
Using the .servers accessor
Using server as a context manager

    server.start()
    server.stop()

    # Start or stop the server object returned from .create
    client.servers.start(server)
    client.servers.stop(server)

This approach automatically starts and stops the server:

    with client.servers.create('OpenHermes-2.5-Mistral-7B/Q4_K_M') as server:
        openai_client = server.openai_client()
        # Make requests to the server

Documentation Index

​Creating servers

​Server configuration parameters

​Managing servers

​Server attributes

Creating servers

Server configuration parameters

Managing servers

Server attributes