As local AI model deployments become more common, model safety is rightfully called into question. Is this model safe? Is it good? Is it bad?
The answer is more nuanced than you might think.
Anaconda has a lot of experience here, since we’ve been building and securing open-source artifacts for years. In some ways, generative AI models are very similar to our other artifacts, like Python packages. You look at who created it, how they’ve licensed it, what assertions have been made about it, and then build and test it to verify expected functionality.
However, in just as many ways, determining the safety of an AI model is a whole new ballgame.
Why AI Models are Harder to Secure Than Software
First of all, the term “open source” takes on a new definition when it comes to AI models. With open-source software, the “source” that’s open is the code used to build it and it’s open for anyone to see. You can inspect that code, check it for safety, build and run it transparently.
However, open models are generally just distributed as “open weights,” not the full source you’d need to build them. This means they don’t specify how the model was trained or what data it was trained on. The model you download is just a big unreadable file of numbers (weights) that you can’t reproduce and mean nothing outside the context of a running model. They need some other software to run them. Since you can’t inspect the weights themselves, they’re much harder to scan, which means traditional malicious content can hide much better.
Scans of Hugging Face have shown hundreds of thousands of model safety issues across tens of thousands of models. Just this February, malicious models were found on Hugging Face that opened connections to remote servers when loaded.
Models as Unknowing Accomplices
Though malicious models are surely out there, most models are neither good nor bad. They’re just unknowing accomplices in bad behavior.
In many cases, the infrastructure and software stack around the model is given less consideration than the model itself, which can open you to risk. If you read OWASP’s 2025 Top 10 Risk & Mitigations for LLMs and Gen AI Apps, you’ll see a number of risks that don’t rely on a tampered-with model, but instead exploit the system and location running the model. Especially when used in agents, models trained to be helpful can expose system vulnerabilities.
Multiple Layers of Model Safety
Model safety requires looking at many things: The model itself, the infrastructure running it, and how it integrates with your system.
AI Catalyst addresses all three. We curate models from trusted sources and give you the information you need to determine whether the license, performance, and terms meet your specific requirements. We benchmark each model and run security evaluations to assess susceptibility to attacks, like prompt injection and jailbreaking. And we build the inference infrastructure (llama.cpp and the quantization toolchain) from verified packages, which gives you a controlled stack from model to runtime.
The Bottom Line
Just as comedian George Carlin said in one of his famous bits, “The words are innocent… It’s the context that makes them good or bad.” The same holds true here: The models are innocent. It’s the context they’re used in that creates the risk.