Secure AI has three main considerations which must be monitored throughout the system’s life cycle. Its data, its model, and its predictions. AI Data security ensures that the underlying data is secured, unbiased, private and relevant. AI model security covers protection for the parameters of AI models from internal and external threats. Finally, AI prediction security protects the quality, reliability and robustness of the outputs of AI models. This post explores the various aspects of ensuring that data used in learning models are secure from internal and external threat
AI Data Security
Data is at the core of all AI systems – data which includes end-user personal data, sensitive corporate data or data from nation state critical infrastructure. This has implications for the security risk profiles of AI data providers and must be managed carefully. In defining an encompassing framework for AI data security, we should consider how data used to build AI models puts its providers at risk. We are familiar with ethical and trust requirements for processing data in AI systems such as data bias and data anonymity. In addition to these, attackers have demonstrated that AI models can be manipulated to produce unintended outcomes. Some of these attacks such as data poisoning are due to vulnerabilities caused by breaching of data minimisation and data closure concepts.
AI Data Anonymity
Data anonymity is the practice of ensuring that personal information is not connected to an individual’s identity. This can be achieved by de-identifying the data, which involves removing or masking identifying information such as names, addresses, and social security numbers. There are a number of reasons why data anonymity is important. For example, it helps protect individuals’ privacy and ensures that their personal information is not misused. It also allows organisations to share data without compromising the privacy of their customers or employees. In the context of AI, data anonymity is especially important when it comes to training machine learning models, as it helps ensure that the models are not biased against certain groups of people. It is important to note that it is not always possible to fully anonymise data, and there is ongoing research and debate about the best approaches to data anonymity in the context of AI.
AI Data Bias
Data bias occurs when the data used in learning models do not represent the realworld population they are to be deployed in. This leads to deploying models that are trained on data with biased or discriminatory information. Such negligence of data bias during training often leads to systematic tendency of a machine learning model to make certain types of errors more frequently than others. For example, an AI Anime generation application trained on a bias sample of Asian faces turns Taylor Swift into a Japanese Cartoon character and a little black girl into a monkey. Similarly, data bias can also manifest when a model is trained on data that includes biased or discriminatory information may make decisions that are unfair or unjust. For example, the US criminal justice system training risk prediction models on datasets with predominantly Black offenders. It is important to be aware of data bias and to take steps to mitigate it when training machine learning models. This can be achieved by using diverse and representative training datasets, implementing fairness constraints, and regularly evaluating the performance of the model on different groups of people.
AI Data Poisoning
Data poisoning involves manipulating the training data used in learning models to produce undesirable outcomes. For example, An attacker will infiltrate a machine learning database and insert incorrect or misleading information. As the algorithm learns from this corrupted data, it will draw unintended and even harmful conclusions. Data poisoning attacks fall into two main categories: attacks targeting availability and those targeting integrity. Availability attacks are often unsophisticated but broad, injecting as much bad data into a database as possible. After a successful attack, the machine learning algorithm will be entirely inaccurate, producing little to no true or useful insights. For example, consider a machine learning model that is trained to classify images as either “cat” or “dog.” If an adversary is able to inject images of objects that are not cats or dogs into the training dataset, the model may become confused and unable to accurately classify new images as either “cat” or “dog.”
Attacks against machine learning integrity are more complex and potentially more harmful. These leave most of a database untouched, except for an unnoticeable back door that lets attackers control it. As a result, the model will seemingly work as intended but with one fatal flaw, such as always reading one file type as benign.
These types of attack can be particularly effective because machine learning models are generally designed to optimize for performance on the training data, and may be more vulnerable to errors or biases that are introduced through the training process. There are a number of techniques that can be used to mitigate the risk of data poisoning attacks, including data sanitization, data quality checks, and adversarial training.
AI Data Closure
Data closure is a property of a machine learning model that refers to the ability of the model to make predictions based only on the data it was trained on, and not on any additional information that may be available. This is important because it allows us to understand the limits of the model’s capabilities and to ensure that the model is not making predictions based on biased or irrelevant data. For example, consider a machine learning model that is trained to predict the likelihood that a person will default on a loan. If the model is able to make accurate predictions based only on the data it was trained on (such as the borrower’s credit score and income), then it has good data closure. However, if the model is able to make accurate predictions based on additional information that it was not trained on (such as the borrower’s race or gender), then it may have poor data closure and may be biased.
AI Data Leakage
Data leakage is a phenomenon that occurs when information from the test set (or future data) is inadvertently included in the training process of a machine learning model. This can cause the model to overfit to the test data, resulting in poor performance on new, unseen data. There are a number of ways that data leakage can occur. For example, it can occur if the test set is used to select or tune the hyperparameters of the model, or if the test set is used to validate the model’s performance during the training process. Data leakage can also occur if the training data includes information that will not be available when the model is used in the real world, such as the outcome of a future event.
To prevent data leakage, it is important to carefully design the training and testing process and to ensure that the test set is only used to evaluate the model’s performance after training is complete. It is also important to ensure that the training data is representative of the data that the model will be used on in the real world.
Data Minimization
Data minimization is the practice of limiting the amount of personal data that is collected, used, or stored to the minimum necessary for the specific purpose for which it is being collected. This principle The principle stipulates that a data controler should limit the collection of personal information to what is directly relevant and necessary to accomplish a specified purpose. They should also retain the data only for as long as is necessary to fulfil that purpose.This approach is often used to protect individuals’ privacy and reduce the risk of data breaches or misuse.
The data minimisation principle is expressed in Article 5(1)(c) of the GDPR and Article 4(1)(c) of Regulation (EU) 2018/1725, which provide that personal data must be “adequate, relevant and limited to what is necessary in relation to the purposes for which they are processed”.