Corporations are increasingly deploying AI models that are available for download and can be operated outside the infrastructure of the companies that originally developed them. These models can offer significant advantages to the enterprise: lower cost, greater control over deployment, the ability to keep the AI reasoning function inside the enterprise environment (which is not something offered by the frontier labs for their cutting edge models), and the opportunity to customize a model for particular business or professional uses.
These models also introduce a new form of technology supply-chain risk. Unlike conventional software, an AI model’s behavior is not determined solely by human-readable source code. Much of what a trained model has learned is embedded in billions of numerical parameters—its weights. Those weights can be downloaded, further trained, and deployed as part of a company’s internal AI model. (An example of weights could be deciding whether to take an umbrella when you go out: factors include whether it is raining now, whether it’s forecast to rain later, whether you will be inside for the trip, whether your clothing would react badly to being wet, etc. That last factor would have a higher weighting if you were dressed in formal wear for a wedding, as opposed to sneakers and jeans for a trip to the grocery store.)
That feature creates a critical governance question: What exactly has the enterprise imported into its environment, and how much is known about how the model was created and how it will behave?
“Open source” and “open weight” are not the same thing
The terminology can be confusing.
An open-weight model makes its trained model parameters—the weights—available for others to download and run. The weights represent much of what the model learned during training.
“Open-source AI” is a broader concept and addresses not only model weights, but also the ability to use, modify, and share code needed to modify software. For that reason, describing every downloadable model as “open source” can be misleading.
For enterprise risk purposes, the more useful question is often simpler: Is the company importing a pretrained model created by someone else, and if so, what does it know about its provenance, training, and behavior? These are not easy questions to answer when one is talking about a billion parameter mathematical construct. It is a haystack of haystacks.
The weights themselves can create a model-integrity risk
A trained neural network is generally composed of a model architecture together with learned parameters, or weights. Training adjusts those parameters and modifies the model’s behavior.
As a result, if an adversary were able to manipulate the training process or model parameters, malicious or unwanted behavior could potentially be embedded in the resulting model. NIST’s taxonomy of adversarial machine learning expressly recognizes model poisoning, backdoor poisoning, and related attacks that can cause integrity violations in machine-learning systems. NIST has also described robustness against supply-chain model-poisoning vulnerabilities as an important unresolved problem.
Such behavior need not appear during ordinary testing. A backdoor could theoretically be conditioned on a particular trigger or circumstance while the model behaves normally in other situations. Additionally, a deployed model that has been poisoned or backdoored won’t need to connect to the Internet to produce bad behaviors, the weights are already baked-in.
That distinction is important. The concern is not necessarily that someone outside the company is actively controlling the model. The concern is that the enterprise may have imported an extremely complex behavioral/reasoning engine whose complete characteristics are, at a minimum, difficult to inspect and perhaps even unknowable in full.
Post-training does not necessarily eliminate the upstream risk
Many organizations do not simply deploy an open-weight model unchanged. Instead, they perform additional post-training to make the model better suited to a particular industry or task.
Post-training can include supervised fine-tuning, reinforcement learning, distillation, or other techniques. These techniques can substantially alter how the model behaves.
Post-training, however, should not automatically be treated as creating a new trusted model from scratch, because unless the post-training impacted the poisoned weights, then the malicious behavior will not have been “trained away.” And since one doesn’t know which weights—out of billions—are problematic, post-training alone shouldn’t be a basis upon which to rule out model-poisoning.
Agentic systems can amplify the impact of model-poisoning
The risk also depends heavily on what surrounds the model.
A model operating as an isolated chatbot with no file access, credentials or network connectivity has a limited ability to cause direct harm. It can produce a bad output, but its ability to act on that output is constrained.
An agent harness, however, may give the model access to tools that can read files, execute commands, query databases, communicate over networks or interact with external systems. In such an architecture, the model may select or propose actions and the surrounding software gives those actions operational effect.
That means a latent model behavior becomes more significant when the model has access to valuable tools or sensitive information.
The relevant security question isn’t just:
“Do we trust this model?”
But also:
“What can this model cause the surrounding system to do?”
Enterprises should treat tool permissions, credentials, network access, execution environments and human-approval requirements as separate trust boundaries around the model.
Hosted models create a different risk
Open-weight models can also be consumed through third-party hosted services rather than downloaded and run by the enterprise.
That hosted model is a materially different architecture.
When prompts, documents or other information are transmitted to a third-party model provider, the enterprise should understand where the request is processed, which entities have access to it, what information is retained, and whether the provider or its subprocessors operate in jurisdictions presenting additional regulatory or national-security concerns.
The Department of Justice’s Data Security Program is an important consideration in this regard. The DSP currently designates China, including Hong Kong and Macau, Cuba, Iran, North Korea, Russia and Venezuela as countries of concern. But the DSP does not make every transmission of ordinary personal information to one of those countries unlawful. Its application depends on, among other things, whether a transaction involves specified government-related data or bulk U.S. sensitive personal data, the type of transaction, the parties involved, applicable thresholds and available exemptions or authorizations. (U.S. Department of Justice)
Depending on the circumstances, it can be relatively easy to hit the “bulk” data tripwire. Accordingly, enterprises should therefore evaluate the actual data flows and legal relationships rather than simply assuming that the geographic origin of a model determines the legal result.
Embedded AI can make these issues difficult to see
These issues become harder to spot where AI components are woven into third-party software, mobile applications, SDKs or enterprise products. It may not be clear to the business that AI is even being used.
An application may:
- call a remotely hosted AI service;
- contain an on-device model;
- use a third-party inference provider;
- incorporate an open-weight model that has been further trained by the software vendor; or
- combine several of these approaches.
Each architecture presents different risks.
Network and runtime analysis can help establish whether information is actually transmitted outside the environment, what endpoints receive it, and which application components cause those transmissions. But network testing alone cannot establish that a locally running model is free from latent or trigger-conditioned behavior. Conversely, examining model provenance alone does not establish where an application sends its data.
Practical steps for enterprises
Organizations adopting third-party AI models should consider extending traditional software-supply-chain controls to the model layer.
At a minimum, enterprises should consider identifying:
- The exact model and version being used, including its base-model lineage and a cryptographic hash of the approved model.
- Who developed and trained the base model, and what information is available about its training and evaluation.
- Whether the model has undergone additional post-training, by whom, using what general methodology, and whether that training modifies the base weights or adds trained adapters.
- Where inference occurs—inside the enterprise, in the enterprise’s cloud environment, or through a third-party API.
- What additional code is loaded with the model, including custom model code, libraries, serialization formats and inference runtimes.
- What the model can access and do, including file-system access, credentials, databases, network connectivity and agent tools.
- What network communications actually occur during operation, rather than relying solely on documentation or contractual descriptions.
- Whether the deployed model has undergone behavioral and adversarial testing, including testing for unusual trigger-dependent behavior and regression testing after post-training.
- Whether model artifacts and dependencies are pinned and controlled, so that an approved model cannot silently change because a repository or dependency has been updated.
- Whether outbound network access is restricted to approved destinations and whether sensitive execution environments apply least-privilege principles.
Traditional due diligence remains important. But AI requires enterprises to extend that diligence beyond source code and vendor questionnaires.
With a trained AI model, some of the system’s behavior resides in billions of parameters produced through a training process that the enterprise may not have observed and cannot practically inspect parameter by parameter.
That difficulty does not mean open-weight AI should be avoided. It means that the model itself should be treated as part of the technology supply chain and as an independent object of security, provenance and governance review.
Norton Rose Fulbright’s NT Analyzer assists clients in examining network traffic and runtime behavior in websites, mobile applications and other digital properties. Those techniques can help determine what data is actually transmitted, to which third parties, and which components are responsible for the transmission. They can form one part of a broader diligence program that also addresses model provenance, model integrity, software dependencies and agent execution boundaries.
Recent Comments