Skip to main content

​
Usage

anaconda ai launch <MODEL> [OPTIONS]

​
Description

Launches an OpenAI-compatible inference server for a quantized model. If the model has not been downloaded, it is downloaded automatically before the server starts. By default, the server runs in the foreground and is stopped and removed when you press Ctrl+C. Use --detach to leave the server running in the background after the command exits.

​
Arguments

​
Options

​
Server options

Additional server options can be appended as --key=value pairs or --key boolean flags. These options are passed directly to the backend server.
Server options are backend-specific. The options below apply to the ai-navigator and anaconda-desktop backends and map to llama-server parameters. The ai-catalyst backend may support different options.
Common server options for ai-navigator and anaconda-desktop include:

​
Examples

Launch a server in the foreground 
anaconda ai launch OpenHermes-2.5-Mistral-7B/Q4_K_M
Launch in the background (detached) 
anaconda ai launch OpenHermes-2.5-Mistral-7B/Q4_K_M --detach
The command returns to the terminal immediately while the server runs in the background. Launch with custom server options 
anaconda ai launch OpenHermes-2.5-Mistral-7B/Q4_K_M --ctx_size=512 --jinja
Launch and open the server in a browser 
anaconda ai launch OpenHermes-2.5-Mistral-7B/Q4_K_M --show