What makes AI/ML security posture different?

The AI/ML architecture differs from a standard application that allows an adversary to influence the AI/ML system behavior to perform unintended operations with the help of simple but clever worded statements. While there is active dialogue on the different attacks against AI/ML technologies, emphasis and focus on secure design improvements can help to influence positive security outcomes. The blog will discuss the key differentiations when analyzing AI/ML technologies and what options exist today to safeguard against threats.

What is different?

From a security architecture perspective, separate data and management plane (aka control plane) is an important concept that helps to enhance separation of duties in addition to segmentation itself. If the "data and management plane" do not resonate with you, data plane refers to the use of a system by a user and management plane refers to the change access managed by an admin (think of a privileged user that can cause system changes). For simplicity, I use the word "system" to describe an application with supporting infrastructure that takes user input, processes it and generates an output.

Coming back to the data and management plane, separation is best achieved when there are distinct APIs to access the planes individually. When distinct APIs exist, separation allows to filter APIs that restrict management access to trusted endpoints for admins exclusively. The advantage here is defense in depth that prevents an authorization failure to gain privileged access to the system from the data plane. When distinct APIs are not supported, the fallback controls are authentication and authorization. And for an effective authorization control, the APIs need to support fine grained entitlement model that uniquely identifies data and management plane actions.

The distinction allows to maintain separation of duties between users and admins, which otherwise would make change management challenging. A good example where you can see this in action is cloud where cloud APIs for both user and management plane are by default available over the internet. Even though a strong entitlement model would seem like a fundamental need, it is not rare to find systems where it is less than ideal to establish mature separation of duties.

When we look at AI/ML technologies, the 2 things that stand out are the free form user interaction and learning capability. These 2 aspects provide a unique opportunity for a system to interact with the user not bound by a definite set of key words (unlike an API) and also its ability to learn through interactions. The free form capability is quite different from a standard application interaction where the system has finite interactions possible due to the definite nature of the APIs. And because of the learning aspect, the system needs the ability to change itself, again something not seen in change management for a standard application.

If we focus on the nature of interaction for an AI/ML system, the interaction is not truly finite like the APIs. Sure, an AI/ML system will have specific use cases and generate specific response type, e.g. a Large Language Model (LLM) that generates text responses and not videos. But generally for a given AI/ML system, inputs are not finite. Which is to say that I can ask a question in more than 1 ways and I can ask many different types of questions. In other words, it is difficult to predetermine all inputs ahead of time because of the diverse possibilities for user input.

For a secure system, traditionally, knowing input and output syntax have been fundamental to developing effective security controls. As an example, when we consider SQL injection attack prevention, we rely on knowing the input data syntax that we expect from the user. If the input is a username, we expect alphanumeric characters of a certain length. We also enforce additional controls in the form of stored procedures and encoding. Basically, the goal of the exercise is to avoid any input from being interpreted as code by establishing context.

Establishing context is tricky

By default, a system does not distinguish between a valid and invalid request, it has to be programmed or configured to process the same request differently depending on the context. In case of SQL injection attack, it is the distinction between the SQL query executed by a database admin v/s a query injected by an adversary through the website. If you follow the premise, it is easy to understand that to prevent against all possible attacks, the system needs to have a clear definition of the "right context".

"Right context" signifies the conditions in which a system processes the input along with due checks and balances that validate the request. If an input does not meet the established criteria, it is rejected. The property enables a system to reject everything but only accept known good input. Another good example is a firewall with allows and denies rules. Therefore, for a system to establish context, predetermination of input becomes important. When you design the system in such a manner, all non-conforming or invalid inputs including attempted compromises or attacks should be unsuccessful barring other flaws (e.g. replay attacks, timing attacks, cryptography issues etc.).

Coming back to the AI/ML architecture, predetermination of input is tricky. By design, an input that interacts with the AI/ML system is not finite and therefore, it is difficult to establish rules to validate the input and determine how the system should handle it. Additionally, when an adversary sends an input to AI/ML system, traditional input validation methods will not be helpful here because the input is within the bounds of allowed characters. The adversary does not have to use special characters (like in a SQL query injection) to target the AI/ML system.

The attack surface for AI/ML differs in this regard with a standard application because instead of relying on a definite set of APIs, the AI/ML determines the input (through Natural Language Processing) to process the request. For an adversary, the opportunity lies in the (mis)interpretation of the input in a statement by the system rather than the set of characters that forms code (in a SQL injection).

Further, the system by design needs to learn which results in co-existence of user and management plane. So, an AI/ML system neither has the APIs nor the data and management plane separation, which results in an interesting situation where traditional security controls fall short. As a fall back, in the absence of known "good input", blocking "bad input" is an option, which you can relate to a sort of "deny list" of keywords. The issue with deny list approach is that there are usually ways to get around them.

Continuous learning is great but does not help the security cause

AI/ML is a learning system, which means it is expected to continue to change with time. So even if we were to create any form of guardrails, those guardrails will be based on observations at a point in time, requiring constant tweaking. Not to forget the changing threat landscape with new and emerging threats that can further diminish the effectiveness of existing guardrails. The adversary is also continuously learning and exploring new methods to bypass controls that puts additional pressure on catching up approach. And in the absence of fundamental secure design controls from the start, catching up can fall short. The 2 problems are that 1) it is difficult to make AI/ML context aware through deterministic input and 2) establish valid learning.

What can be done

There is an opportunity to restrict the blast radius of non-confirming AI/ML behavior as a fallback design control. The following principles can be helpful:

1. Ensure that AI/ML system uses a prescribed set of data sources and nothing else. You want to restrict an adversary to influence AI/ML model to source data from an unauthorized source otherwise, the system can generate incorrect responses.

2. Treat dataset trained by AI/ML system differently and carefully design data flows (e.g. replication or enrichment). You do not want to have an unvetted trained dataset as the source of information for other systems. The approach can help you to maintain data integrity for the rest of the systems while limiting data integrity issues if any, to the specific AI/ML system.

3. Restrict AI/ML system access to only what it needs to perform its function (aka least privilege). The idea is to avoid unintended access to other systems if an adversary can cause the AI/ML system to execute unauthorized actions.

The first principle ensures that AI/ML system can only use predetermined data sources while the second principle restricts other systems from accidentally using AI/ML datasets without validation. The third principle constrains downstream impact of unexpected AI/ML system behavior. The discussed principles focus on high level design constructs that aim to reduce blast radius in the event of unintended AI/ML system behavior and must be complemented by comprehensive security best practices that reduce likelihood of system compromise itself. Some of the examples are data classification and integrity for model training, data privacy, data leakage, injection attacks, model theft etc. A thorough list of considerations can be sourced from OWASP, in-depth security assessments and industry best practices to name a few.

Looking ahead

A lack of specific secure design foundations presents a problem for AI/ML because the alternative is to iteratively apply varying guardrails in response to newly discovered threats. The problem as a whole is amplified as AI/ML technologies are adopted by organizations with varying security maturity. Though the blog focused on very specific AI/ML design issues, do remember that technology security and hygiene across cloud, infrastructure and application, end of life, supply chain, patching etc. also play an important role and unfortunately, continue to be challenges for many organizations.

Where we stand today, AI/ML security requires a holistic security approach with focus on improvements through continuous assessments to find and fix vulnerabilities and also, continuous organization wide learning. The AI/ML space is rapidly changing with quick advancements and given that some of the key security approaches require rethinking, taking a cautious approach may not be a bad idea for organizations while aligning with their respective risk appetite (or establish one). At the same time, I am also eager to see how the security evolves to safeguard the AI/ML technology at a more fundamental level. I hope to see continued security dialogue in the AI/ML space.