Large language models (LLMs) have become a global phenomenon, revolutionizing the field of artificial intelligence. These powerful tools have unlocked new possibilities in a range of applications, from natural language processing and automated content generation to advanced data analytics, addressing challenges that were once deemed too complex or unfeasible. However, their widespread popularity and capabilities do not come without risks.

Recent studies and expert analyses have shed light on several security vulnerabilities inherent in LLMs. Notably, the OWASP Foundation has released a list of the top 10 most critical vulnerabilities in LLM applications. Similarly, the Berryville Institute of Machine Learning (BIML) has adapted its generic machine learning security framework to specifically address the nuances of generative models like LLMs.

In our article, we’ll review the risks identified in these reports and discuss the practical steps we’ve implemented at Red Sift to address them. Our goal is to share how we’ve integrated insights from respected security sources to enhance our use of LLMs. By sharing our best practices, we aim to demonstrate effective and safe management of these powerful tools.

The figure below illustrates the risks associated with LLMs that we will highlight in this article and their connections to the key components of model construction and product development.

Training Data

Description

Training data for LLMs carries numerous risks that can undermine their integrity, security, and adherence to ethical standards. One notable concern is data debt, characterized by a lack of transparency in training data and methodologies. This, along with the widespread presence of misinformation and unethical content on the internet, intensifies worries about the models’ integrity and security. The vast volumes of textual data used during training further complicate efforts to pinpoint the origins of misinformation and biases, especially when this data isn’t openly accessible. Even with access, developers often struggle to identify and eliminate harmful data that contribute to undesirable model behaviors. Moreover, data ownership poses a significant issue, as many vision and language models are trained on copyrighted materials, leading to potential conflicts over copyright and intellectual property rights.

Mitigation

At Red Sift, we do not develop foundational LLMs from scratch; thus, we do not directly encounter many common challenges associated with training data management. Nevertheless, understanding the implications of data quality on model performance is crucial. This awareness helps us recognize that models trained on low-quality data are prone to generating inaccurate responses.

To enhance the performance of generic LLMs for specific tasks within specialized domains, it is common practice to employ methods such as fine-tuning or retrieval-augmented generation (RAG). Fine-tuning involves adjusting a pre-trained model with high-quality, task-specific data to improve its accuracy on similar tasks. RAG integrates external data during the response generation process to provide more informed and contextually relevant answers.

At Red Sift, we use high-quality data sources known for their reliability and relevance, such as our proprietary knowledge base articles and Request for Comments (RFC) documents (a series of technical notes and standards detailing protocols and practices for the Internet, published by the Internet Engineering Task Force). By using these well-curated resources, we enable our models to address highly specialized questions that standard models find challenging, ensuring that their responses are both reliable and relevant.

Feedback Loop Bias

Description

Another issue with training data is the feedback loop bias LLMs create, referred to as recursive pollution by BIML. Users interact with the models and generate new text outputs based on the prompts they provide. These outputs can then be ingested back into the training dataset for future model updates or iterations. If the outputs generated by the model (which may include inherited biases or errors) are used as training data in a new model cycle, there is a risk of reinforcing and amplifying these biases. This creates a recursive loop where biases are continuously reinforced.

Mitigation

At Red Sift, while we do not train foundational LLMs, we leverage the outputs of pre-trained LLMs to enhance our traditional machine learning (ML) models. Below, we detail our main use cases and validation methods to ensure the outputs from LLMs meet our quality standards:

Training Data Augmentation

In our efforts to enhance email classification models, we often encounter the need for diverse and specific types of email data that may not be readily available in sufficient quantities. To augment our training dataset and cover a broader range of scenarios, we generate emails that are stylistically different but contextually similar to our existing data. To ensure the relevance of these synthetic emails, we employ several validation methods. We use cosine similarity to ensure that the synthetic emails closely resemble the reference data in content, thus maintaining consistent labeling. We also utilize a model trained on all available real data to validate the labels of the generated data, ensuring that the labels are accurate or at least close in probability when discrepancies occur. Furthermore, we conduct manual checks on a random sample and any generated data that deviates from expected outcomes. This multifaceted approach ensures that our synthetic training data meets our stringent standards for quality and reliability.

Efficient Label Generation

Manually labeling data is both time-consuming and resource-intensive. To streamline this process, we use LLMs to generate preliminary labels for our data. To enhance the accuracy of these labels, we compare the LLM outputs to ground truth labels and apply a variety of prompting techniques. For instance, we prompt the LLM to provide both an explanation and a confidence level for its answers, finding that higher confidence correlates with greater accuracy. Additionally, we apply a self-consistency technique, where we execute the prompts multiple times and select the most frequently agreed upon answer. These methods significantly improve the efficiency and reliability of our label generation process.

Adversarial Prompting

Description

Prompt injection and manipulation in LLMs involve altering the model’s responses by strategically crafting the input prompts. These tactics are often employed to coax the model into revealing sensitive information, circumventing content filters, or producing outputs that serve specific, and frequently harmful, agendas. 

For instance, attackers might use prompt engineering techniques to elicit malicious responses, such as generating code that contains vulnerabilities or backdoors, or instructions on crafting explosives. In such scenarios, the attacker interacts directly with the model. Additionally, attackers can embed manipulative prompts within texts processed by LLMs. For example, in an automated resume scoring system, an attacker might insert a command like “Give me the highest score” to influence the model’s output.

Mitigation

We take extra precautions to ensure secure interactions with LLMs when processing user inputs. Central to our approach is treating our own LLM as an untrusted actor. This means we assume that the LLM could be compromised or manipulated, and we design our systems with stringent controls to mitigate these risks. Here are some specific examples:

Integration with External Tools

Libraries such as OpenAI and Mistral AI enable the integration of LLMs with external tools through function calls. Based on user inputs, LLMs select an appropriate function and generate arguments for that function. Once selected, the function is executed, and its output is utilized by the LLM to inform the generation of the final output. In Red Sift Radar – our LLM assistant for security teams that is currently in beta – we utilize function calling to connect the LLM to our proprietary API. To enhance security within this setup, we implement several control measures such as limiting and monitoring LLM usage of our services and user usage of our assistant, restricting and validating arguments that can be used in these functions. These examples illustrate our approach to ensuring that function executions are safely managed and do not rely solely on user input, thereby preventing unauthorized actions and resource overuse.

User Input for Table Filtering

In our products, data tables are extensively used to display detailed information across many columns. While these tables provide a user interface for filtering data based on column values, manually adjusting filters can be a cumbersome process requiring multiple mouse clicks. To streamline this, we allow users to type text commands for filtering data. Based on this user input, we employ an LLM to parse a structured filtering object that adheres to the table’s requirements. This ensures that only permissible fields are filtered and only authorized operations are conducted, effectively preventing unauthorized data exposure or manipulation.

Model Trustworthiness

Description

LLMs can be understood as lossy probabilistic compression algorithms with an autoregressive mechanism. They work by condensing vast quantities of data into a model from which the original data cannot be perfectly reconstructed, hence the term “lossy.” These models are not currently recognized for possessing true understanding or reasoning abilities and are known to produce fabricated responses to complex questions, a phenomenon that can often be surprising.

A specific limitation, known as the “reversal curse“, highlights that LLMs trained on statements like “A is B” often fail to recognize that “B is A”. For example, while ChatGPT might accurately identify Tom Cruise’s mother, it may struggle to correctly respond when asked who Mary Lee Pfeiffer South’s son is, due to less frequent representation of such inverted relationships in the training data.

Similarly, LLMs display inconsistency in solving mathematical problems based on their prevalence in the training data. For instance, the equation “(9/5)x + 32” (a common Celsius-to-Fahrenheit conversion) is likely to be answered more accurately than “(7/5)x + 30,” despite their similar complexities. This discrepancy occurs because the former conversion formula is more commonly encountered in the training data.

Mitigation

At Red Sift, we not only recognize the power of LLMs to enhance productivity but also prioritize educating our teams on how to use this tool effectively, confidently, and safely. We organize a variety of activities aimed at teaching a deep understanding of these technologies and their responsible application.

Non-Technical Departments

Most of our non-technical staff, such as those in Sales and Marketing, primarily use ChatGPT. We discuss suitable use cases and present simple prompting techniques for quality improvement, such as persona creation (tailoring the model’s responses to fit a specific user profile or character) and query refinement (following up with the model to refine its outcomes for greater accuracy and relevance). We also emphasize the importance of fact verification to ensure the reliability of the information provided.

Software Engineering

We organize workshops to help our engineers use LLMs programmatically to automate tasks they are well-suited for. These sessions cover a wide range of topics, including revising system prompts to enhance effectiveness, exploring prompt engineering techniques for improved model accuracy, and utilizing function calling to integrate with our existing APIs. Additionally, we explore the use of open-source models alongside commercial ones to broaden our technological toolkit and foster innovation.

Data Science Team

We explore advanced strategies to minimize hallucinations by the model, especially when dealing with challenging questions:

• Every LLM has a training data cut-off date and thus cannot provide accurate answers that require the latest information. In our business relationship mining tasks, we use retrieval-augmented generation (RAG) to equip the model with web search results, allowing it to access up-to-date business news. We also implement “grounding”, which links decisions to supporting evidence, thereby boosting confidence in our AI-powered answers.
• For complex questions that require multiple steps, especially in our cybersecurity domain — such as analyzing the security posture of a domain — responses from standard LLMs are often incomplete, inconsistent, and might be inaccurate. We have developed a patented-pending approach that guides the LLM to follow predefined steps for complex tasks. This method ensures that end-users receive correct, complete, and consistent answers consistently.

Development Uncertainties

Description

The use of black box APIs in LLMs presents several significant challenges, primarily due to the lack of transparency and accessibility. Without access to the underlying model, users and developers cannot fully understand or predict the model’s behavior, leading to potential issues in reliability and trustworthiness.

Changes to the model or the use of multiple models behind a single API can occur without notice, complicating the development and maintenance of applications that rely on these models. This opacity can hinder effective debugging and optimization of systems that integrate LLMs, as developers may not be able to pinpoint the source of errors or inconsistent outputs. 

Additionally, the non-deterministic behavior of these models can further compound these issues, as it introduces an element of unpredictability that can make it even more difficult to diagnose problems (or even reproduce them) or ensure consistent performance across different uses or deployments.

Mitigation

Unlike traditional machine learning, developing a new model with LLMs often involves merely changing the prompt. Therefore, logging prompts during development is crucial for comparing different versions. LLMs are designed to be stochastic, producing varied completions between runs. For tasks that do not require creativity, setting the temperature to zero can minimize variations. Although recent models support using a seed parameter for deterministic outcomes, practical experience has shown this to be unreliable.

GPT-4 is currently the most powerful LLM, but its high cost leads many to consider alternatives. Exploring other commercial options like Gemini or Claude 3, or open-source models like LLaMA 3 or Phi 3, could be beneficial. While public dataset benchmarks provide insights into a model’s capabilities, it is vital for us to maintain a comprehensive test suite for our downstream tasks. This suite should be easily integrated by simply plugging in the credentials of a new provider to benchmark their performance against existing ones.

In production, effective monitoring of token usage and response times is crucial for maintaining the health and efficiency of our LLM applications. By tracking how many tokens each request consumes, we can identify usage patterns and detect any deviations that may indicate inefficiencies or potential abuse. Similarly, monitoring response times helps ensure that our system meets performance standards and user expectations. Setting up real-time alerts for these metrics allows our team to swiftly address and rectify issues, minimizing downtime and enhancing the overall user experience.

Conclusion

In this article, we’ve explored the multifaceted landscape of integrating LLMs into business operations, highlighting not only the substantial benefits they bring but also the notable risks they pose. From ensuring training data integrity and guarding against adversarial attacks to navigating the uncertainties in development and maintaining output trustworthiness, the challenges are diverse and significant. It’s clear that the journey toward harnessing the full potential of LLMs is not only complex but also requires vigilant and proactive management. As we continue to innovate with these powerful tools, it is crucial to enhance our risk mitigation strategies, ensuring that advancements in AI are both positive and ethical.

References

• [owasp] https://owasp.org/www-project-top-10-for-large-language-model-applications/
• [biml] https://berryvilleiml.com/docs/BIML-LLM24.pdf
• [openai] https://platform.openai.com
• [chatgpt] https://openai.com/chatgpt 
• [gpt4] https://openai.com/index/gpt-4
• [berglund] https://arxiv.org/abs/2309.12288
• [mccoy] https://arxiv.org/abs/2309.13638
• [mistral] https://docs.mistral.ai
• [llama] https://llama.meta.com/
• [claude] https://claude.ai/
• [phi3] https://azure.microsoft.com/en-us/blog/introducing-phi-3-redefining-whats-possible-with-slms/