Automating Compliance Mapping: Using Local LLMs to Tame NIST and Cloud Frameworks

TLDR: Security teams often struggle to map complex internal policies against evolving frameworks like NIST SP 800-53 or cloud-specific security guides. By deploying local LLMs via Ollama and Open WebUI, you can perform Retrieval-Augmented Generation (RAG) on sensitive documentation without leaking data to third-party cloud providers. This approach turns thousands of pages of compliance noise into a searchable, queryable knowledge base that helps you identify gaps in your security architecture.

Compliance documentation is the bane of every security professional’s existence. Whether you are a pentester trying to understand the specific controls a client is failing, or a researcher attempting to map a new vulnerability to a regulatory requirement, you are likely staring at thousands of pages of PDF-based misery. The industry standard for risk management, specifically the NIST Risk Management Framework (RMF), is a labyrinth of control families, enhancements, and implementation guidance. When you add cloud-native security requirements from AWS or Azure, the cognitive load becomes unsustainable for small teams.

The Problem with Manual Compliance Mapping

Most teams handle this by manually cross-referencing spreadsheets or relying on "tribal knowledge" that disappears the moment a senior engineer leaves. This is where LLMs shine, but the standard enterprise approach—uploading sensitive internal policy documents to a public cloud-based LLM—is a non-starter for anyone who cares about data privacy. You are essentially handing your internal network architecture and security gaps to a third party.

The solution is to bring the model to the data, not the data to the model. By running a local LLM stack, you maintain full control over your information while gaining the ability to query your documentation in plain English.

Building a Local RAG Pipeline

To get started, you need a containerized environment. Using Docker, you can spin up an instance of Open WebUI and Ollama. This setup is portable, reproducible, and keeps your data off the public internet.

The workflow relies on Retrieval-Augmented Generation (RAG). Instead of training a model on your data, you create a vector database of your documents. When you ask a question, the system retrieves the most relevant chunks of text from your PDFs and feeds them to the LLM as context.

Technical Implementation

If you are on a Linux machine, you can pull the necessary images and run them with minimal configuration. For those with an NVIDIA GPU, ensure you have the NVIDIA Container Toolkit installed to offload the heavy lifting.

# Pull and run Open WebUI
docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui ghcr.io/open-webui/open-webui:main

# Run Ollama in a container
docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

Once the containers are running, you can use a tool like Docling to convert your messy PDFs into clean Markdown. Markdown is significantly easier for an LLM to parse than raw PDF text, which often contains broken tables and headers that confuse the model.

Why This Matters for Pentesters

During an engagement, you are often asked to provide a "gap analysis" or a list of recommendations. If you have a local RAG instance populated with the client’s internal security policies and the relevant NIST SP 800-53 controls, you can ask the model: "Based on the current network architecture, which NIST controls are most likely to be missing?"

The model will not give you a perfect answer, but it will give you a starting point that is grounded in the client's own documentation. It allows you to move from "I think this is a problem" to "This configuration violates the client's own policy on Access Control (AC) as defined in their internal handbook."

The Reality of Data Quality

Garbage in, garbage out remains the golden rule of AI. If your internal documentation is outdated, contradictory, or poorly formatted, the LLM will hallucinate based on that bad data. Before you feed your documents into the vector database, perform basic data cleansing:

• Deduplication: Remove multiple versions of the same policy.
• Normalization: Ensure consistent terminology across documents.
• Formatting: Use Docling to ensure tables and lists are correctly identified.

If you are working with NIST SP 800-161 for supply chain risk management, the structure of the document is critical. If the LLM cannot parse the control hierarchy, it will fail to map your findings to the correct requirement.

Moving Forward

You do not need a massive budget or a dedicated AI team to start using these tools. The barrier to entry is now just a few Docker commands and a bit of time spent cleaning your documentation. Stop treating your policy documents as static files that sit in a folder. Treat them as a dataset. By building a local RAG pipeline, you are not just making your life easier; you are creating a force multiplier that allows you to perform deeper, more informed security assessments in a fraction of the time. Start small, pick one framework, and see how much faster you can navigate the compliance landscape.