How we built an AI threat detection pipeline for Envoy Response

Every day, Envoy Response processes more than 150,000 potential threat signals from government agencies, vendor APIs, news feeds, RSS sources, and other public data providers. At that scale, volume is only part of the challenge. The signals themselves are messy, often arriving with inconsistent schemas and missing critical details such as a headline, severity level, or expiration times. The stream also contains duplicate reports, outdated incidents, minor local events, and articles that mention a location without describing an active threat.

Envoy Response was built to turn this continuous stream of messy, real-world threat signals into a clear view of actionable threat events for security teams. To achieve this, we built an event-driven pipeline architecture combining Kafka, deterministic processing, focused LLM (Large Language Models) reasoning, and OpenSearch’s geospatial querying capabilities.

The signal-to-threat architecture

Workplace security administrators traditionally had to triage large volumes of incoming signals and determine whether each one was operationally relevant, how severe the event was, how far its effects might extend, and which employees or locations could be affected. This required repeatedly interpreting source data and applying domain expertise across inconsistent formats.

LLMs allowed us to bring parts of that contextual reasoning into the processing pipeline. We use focused LLM tasks to filter irrelevant signals, infer missing context, normalize heterogeneous inputs into a consistent threat representation, and resolve ambiguous cases where multiple reports may describe the same real-world event.

Rather than relying on a single monolithic prompt invocation, we decomposed the pipeline into focused stages with clearly defined responsibilities. Each stage resolves a specific class of uncertainty, applies stage-specific validation and guardrails, and publishes a progressively more structured event for downstream processing.

Kafka connects the major stages of the pipeline and each stage does one focused task:

1. Relevance: Determine whether a signal is operationally relevant to workplace security
2. Enrichment: Convert heterogeneous source payloads into a consistent threat representation.
3. Deduplication: Identify whether a signal is new, another source reporting an existing event, or an update to an evolving threat.

The resulting threat event is then indexed in OpenSearch, where it can be filtered, clustered, and queried geospatially for the map experience.

How AI supports each stage

Relevance

Keyword filtering is not enough to determine whether a signal is relevant. A report containing the word “fire,” for example, could describe an active wildfire, a historical incident, a minor structure fire, or even an unrelated sports story.

Instead, we define a set of rules for what makes a signal relevant to workplace security, such as whether the event is active or imminent, geographically specific, and capable of affecting people, workplaces, travel, or business operations. AI evaluates the signal’s full context against these rules and determines whether it is relevant.

When the available context is incomplete, or any failures might happen we instruct the model to mark the signal as relevant and allow later stages to investigate further. We also ask the model to give a clear explanation how it came to this judgement so that we can validate, test and keep improving the system.

Enrichment

Once a signal is considered relevant, the enrichment stage transforms it into a consistent threat representation. The same decomposition continues inside this stage: category, severity, headline, description, expiration time, location resolution, and impact-radius estimation are implemented as separate focused tasks rather than one combined model response.

Each enrichment task performs a distinct function, has its own output contract, and can be validated and monitored independently. This lets us evaluate each capability separately, improve one without unintentionally affecting another, and identify the specific task responsible when something fails.

It also lets us select the right model for each task. Headline rewriting and description generation can use faster, lower-cost models, while tasks requiring more contextual reasoning can use a more capable model.

Wherever the source provides trustworthy data, we preserve and use it. LLM reasoning is introduced only when information is missing, ambiguous, or inconsistent.

• Location resolution: We first use coordinates provided by the source. If coordinates are unavailable but location metadata is present, we use a geocoding service to resolve it into coordinates. When neither is available, an LLM infers the location from the signal’s description.
• Headline and description rewriting: We use focused LLM tasks to rewrite unclear or inconsistent source content into concise headlines and descriptions for security admins. The model is instructed to ground its output only in facts present in the source description and not introduce unsupported details.
• Category and severity estimation: When the source does not provide a usable category or severity, an LLM estimates them from the signal’s description and maps the result to our defined threat taxonomy and severity criteria.
• Impact-radius estimation: When a trusted source provides a geographic boundary, we preserve that geometry. When only a point is available, an LLM estimates an initial radius, which is constrained by deterministic limits before being stored.
• Expiration-time assignment: We first use the expiration time provided by the source. If it is missing, we attempt to extract one from the description. When neither is available, we assign a predefined default based on the threat type so that stale threats do not remain active on the dashboard indefinitely.

Deduplication

Threats rarely arrive as a single clean report. We often receive duplicate or near-duplicate signals describing the same incident. A wildfire may first appear in local news, followed by a government alert and then several updates as evacuation zones, road closures, and containment levels change.

The challenge is recognizing that five reports about the Woodside Fire belong to one evolving threat, while a fire near Yosemite represents a separate event.

We solve this problem in layers. 

First, the pipeline applies deterministic checks to compare threat metadata and identify exact duplicates. When exact matching is insufficient, we use semantic search to compare the meaning of the reports rather than relying only on shared words, allowing us to identify near-duplicates written in different ways.

For cases where semantic similarity falls within an ambiguous range, an LLM compares the reports and their surrounding context to determine whether they refer to the same real-world event.

When a match is found, the pipeline updates the existing threat with the latest information and attaches the new source instead of creating another threat event.

What we gained from a decomposed pipeline

• LLM observability: We track token usage, cost, latency, and errors independently for each stage. This makes it easier to understand which capabilities are driving cost or slowing down the pipeline and optimize them separately.
• Model flexibility: AI models are evolving quickly, with new models continually improving the tradeoffs between reasoning quality, latency, and cost. Because each task is isolated, we can choose the model best suited to that task and tune cost accordingly. Rewriting a headline or summarizing a description can use a faster, lower-cost model, while ambiguous relevance or deduplication decisions may benefit from a model with stronger reasoning capabilities.
• Independent debugging and iteration of LLM tasks: Because each capability has a narrow responsibility, its prompts, inputs, outputs, and failure modes can be inspected independently. We can evaluate relevance decisions separately from location resolution or deduplication, identify which capability produced an incorrect result, and improve its prompt, model, validation, or test set without changing the rest of the pipeline. This also lets us benchmark model accuracy against latency and cost for each prompt, so we can choose the best quality-to-cost tradeoff for every task.
• Focused guardrails: Each stage has its own definition of correctness, so it can apply task-specific validation and constraints. For example, impact-radius estimates are restricted by category-specific bounds before being stored. Location resolution follows a deterministic order—source coordinates, structured location metadata, and geocoding—before relying on AI for ambiguous cases.
• Independent scaling: Each stage can scale according to its workload. Relevance processes every incoming signal, while enrichment and deduplication handle fewer but more computationally expensive operations.
• Failure isolation: Kafka creates durable checkpoints between stages. If an AI provider, geocoding service, or downstream datastore slows down, Kafka absorbs the backlog without stopping ingestion or blocking the rest of the pipeline.
• Selective replay: The event stream makes every stage independently replayable. When we improve a prompt, change a model, refine a guardrail, or update a processing rule, we can replay historical signals if needed through only the affected stage instead of retrieving the source data again or rebuilding the entire pipeline.

Leveraging workplace context to make threats actionable

One of Envoy’s key differentiators is its rich workplace context: With customer-configured integrations and appropriate permissions, Envoy Response can compare a threat’s impact area with relevant workplace context, such as office locations, people currently onsite, business travelers, and other people or locations the customer has chosen to include.

Using OpenSearch’s geospatial querying capabilities, we can quickly compare a threat’s impact area with that workplace context and identify the people and places that may be affected. By combining these two worlds, the map can now show:

• Offices inside the impact area
• Employees currently onsite
• Business travelers in the region
• VIPs requiring additional attention

This turns a threat from something an admin simply sees into something they can immediately understand and act on.

Because the impact radius is an estimate, we also give admins simple tools to expand, shrink, or reposition it. As the impact area changes, the system immediately recalculates the potentially affected people and places.

Envoy Response changes where the security admin begins. Instead of starting with scattered alerts and manually piecing together context across threat feeds, maps, HR systems, travel platforms, and spreadsheets, they start with one evolving threat, an estimated impact area, and the people and places that may be affected. The system does the initial sorting and connecting, giving admins a head start while leaving them in control of the decisions that matter most.

AI at Envoy

It's been a really fun adventure to not just leverage the latest in modern technology and learn so much so quickly, but to be able to apply it to real world, life-critical situations. This is just one example of the many things we're doing to bring scalable safety and security to physical workplaces around the world.

We're hiring! Check out our careers page. here's no shortage of cutting-edge problems we're trying to solve and we need your help!