AI Governance Guide: Building Trust into Your Platform

• Map: Identify risks in your business context
• Measure: Assess, analyze, and track risks
• Manage: Prioritize and act based on the potential impact of each risk

Together, these activities are the heart of AI risk management. In an open-source environment, an example of these activities would be:

• Map: Use automated tools to identify vulnerabilities in open-source AI components
• Measure: Implement regular security audits to catch any new package vulnerabilities
• Manage: Prioritize the selection of well-maintained open-source libraries with clear security documentation and governance structures

AI Governance and Open-Source Environments

Some argue that open-source libraries are the heart of the AI governance problem. Others see open-source libraries as essential to AI work, enabling agentic AI, machine learning, and other AI applications. Both are correct.

In a recent Anaconda survey, The State of Enterprise Open-Source AI, respondents reported concrete benefits from open-source software: 57% cited improved efficiency or easier operations, with estimated savings of 28% in cost and 29% in time. Often, the most important benefit of using open-source solutions is that they enhance the organization’s ability to innovate and stay competitive in the market. Off-the-shelf AI tools are now available for mainstream use cases. The power of open source lies in its suitability for creating new and custom AI solutions.

A good rule of thumb is that if an AI product is needed to accomplish a particular goal within an organization that is not that organization’s primary function, buying or subscribing to a proprietary AI product makes sense. For example, if your company uses a group video meeting application, then using the AI from that application to transcribe and summarize company meetings makes sense.

Conversely, when AI capabilities can generate revenue or provide competitive differentiation in your market, open-source custom projects built in-house are the way to go. For example, if your company sells a group video meeting application, developing meeting transcription AI in-house allows you to monetize that feature and avoid losing customers to competitors. The better you make your AI, the more competitive your company becomes. Open-source software has the level of virtually infinite customizability needed to differentiate your organization’s AI capabilities.

The downside of open source is unrestricted package acquisition and its associated security risks and supply chain issues. Open-source code can be particularly vulnerable to bad actor exploitation or simple unintended vulnerabilities. Even the licenses can become a problem area. Start with an enterprise-grade AI platform that integrates governance and development capabilities while maintaining security through built-in checks and vetted open-source packages.

Technical Foundations of AI Governance Platforms

When considering which technologies to use as your AI governance tools, remember that there is no single perfect AI governance platform. One solution may provide data provenance, while another offers code-base security and dependability checking, and yet another is ideal for model and pipeline deployment and observability. Since there’s no one-size-fits-all option, select a curated open-source library of components—one that has all the packages you want and all the security you need. While AI governance frameworks are policy-driven, you need the right technology to implement them.

1. Model Security and Supply Chain Integrity

Model and code supply chain integrity must be a fundamental priority. Enterprises cannot afford security breaches from inadvertently downloaded malicious code. Organizations need to account for all software package dependencies, verify version compatibility, and automatically scan for vulnerabilities.

AI governance platforms play a vital role by offering curated, secure open-source libraries with automated vulnerability scanning and security policies. Alternatively, you could build your own, or integrate proprietary scanning tools. Whether this is built or bought, it is an essential, foundational function of AI governance.

This one crucial practice alone—using software to scan for security vulnerabilities in open-source code—is used by 61% of respondents to The State of Enterprise Open-Source AI survey and can reduce security incidents by up to 60% all by itself.

2. Data Quality and Bias Management

Data quality hygiene practices and data ethics are core aspects of data governance in general. In AI governance, they take on new importance. The data used to train an AI model determines how it will perform. If the data is biased or inaccurate, the model will also be biased or inaccurate, possibly producing results that violate ethical mandates. AI project failures are often attributed to a lack of a good quantity and quality of data in the organization. This can erode trust in the final product if the business lacks faith in the data that feeds the AI. One of the three main points in Gartner’s 2025 “A Journey Guide to Manage AI Governance, Trust, Risk and Security” is “Through 2027, most AI projects will fail to meet expectations due to a failure to focus on data governance.”

“Through 2027, most AI projects will fail to meet expectations due to a failure to focus on data governance.”

The fundamentals of enterprise data governance apply whether the data enables human decision-making through business intelligence or automated decision-making through AI agents. However, solid data quality practices are only a starting point for AI governance.

Beyond standard data quality demands, two aspects of data hygiene that require extra data governance steps specific to AI implementations include:

• Data poisoning: A type of attack where a bad actor deliberately introduces inaccurate or biased data into the training data set for an AI model to sabotage it. Close data monitoring and provenance tracking is the best way to catch this malicious behavior before it damages your AI project. Guarding against data poisoning is more of a security activity than a data quality improvement activity since the goal is to prevent it from happening. Any third-party data set used in an AI project needs to be particularly scrutinized for possible data poisoning.
• Data bias: An unbalanced tendency toward favoring or discriminating against a particular idea, person, or group in a data set. Using that same data to train AI can codify bias into the system. An example of this is using historical prison sentencing data to determine how prisoners should be sentenced in the future.

A 2024 study at the University of Washington tested LLM-based HR resumé-screening applications by changing just the names on identical resumes to names associated with male, female, black, and white groups. The resume screening application showed high bias levels, to the point of failing to recommend resumes with black male names 100% of the time.

Mitigating bias can be very tricky, since many forms of bias are not immediately obvious. Including diverse stakeholders can be key to catching problems early. Common bias elimination techniques include:

• Data balancing: Adjusting data representation to ensure balanced coverage across demographic groups. Oversampling increases representation of underrepresented groups (sometimes through synthetic data generation), while undersampling reduces overrepresented groups.
• Reweighing: Adjusting the influence of different data points during model training to ensure protected groups are appropriately represented in the model’s learning process, rather than being overshadowed by majority groups.
• Feature masking: Excluding protected characteristics like race or gender from training data. However, this approach often fails because proxy variables can reintroduce bias. For example, removing race doesn’t prevent bias if address becomes a proxy due to residential segregation patterns.

In the end, there’s no substitute for continuous monitoring and regular audit testing on the AI output for bias and accuracy.

3. Monitoring and Auditability

AI governance must include comprehensive monitoring and evaluation frameworks that include performance metrics, bias testing, and security considerations. Established processes need to assess not just AI model capabilities, but also vulnerabilities, biases, potential adversarial exploits, and unintended behaviors.

Many of the same monitoring activities necessary for compliance are also accelerators for ongoing development.

• Vulnerability scanning and risk assessment: This should be done for all AI components, before beginning to use those components in AI project development. One good method to accomplish this is with an AI project bill of materials. This doesn’t just document models, it also exposes and categorizes risks your organization faces, including:
  • Technical risks, like memory requirements, compute constraints, and performance degradation at different quantization levels.
  • Legal risks, including license type, usage restrictions, and attribution requirements.
  • Operational risks, such as version stability and integration complexity.
• Enterprise monitoring: Automated and telemetry-based analytics provide AI workflow tracking for compliance. Comprehensive logging captures all user actions and model interactions for audit purposes. System events that occurred, such as syncing jobs, and user actions that were taken, such as downloading a model, deploying a model, and logging in, should be automatically tracked. This monitoring also collects usage information and feedback for further improvements over time.
• Audit logs and compliance tracking: Enterprise security teams need comprehensive control and auditability across the entire AI asset lifecycle. Audit logging helps build complete audit trails for regular compliance reviews. Compliance officers can review audit logs of admin actions taken (policies created and updated, workflow permissions updated) while automated compliance checks validate deployments against enterprise policies.

When problems are identified due to robust monitoring, the entire AI governance team should be notified so they are informed. A person or team needs to be assigned to evaluate and remediate the problem as soon as possible.

4. Responsibility and Accountability

Depending on the size of your organization, an AI governance committee may also make sense. This could include representatives from IT, legal advisors, data scientists, subject matter experts, and others. Having these stakeholders meet regularly helps ensure alignment on regulatory requirements, organizational goals, and AI governance standards. This committee should also be responsible for developing policies for data handling, ethical requirements, deployment procedures, model lifecycles, and more.

The RACI project management system can be a sensible framework for clarifying roles and responsibilities in AI projects. RACI sets four levels of responsibility for team members in any type of project:

• Responsible: The person or team who implements the work and is directly responsible for the result.
• Accountable: The individual who approves projects, authorizes tasks, and has ultimate authority and accountability for the outcome.
• Consulted: Subject matter experts whose input shapes decisions. In AI projects, these often include legal advisors (on compliance), security teams (on vulnerabilities), and domain experts (on model appropriateness).
• Informed: Stakeholders who need to know about progress or decisions but don’t provide input.

AI Governance Maturity Levels

There are several models for AI maturity that can give you an idea of where your organization currently stands and where you need to go next. AI governance maturity should move along with your general AI maturity level.