When the Agent Found the Attacks

A 7B model can't hold 86,000 log records in context. So I gave it Python tools to query summaries instead and asked it to answer the 5 W's of an investigation. On two different datasets, it found the attacks. Here's how I set it up and what it actually found.

The Constraint

A 7B parameter model has a limited context window. Even if I could fit more tokens, inference slows down as context grows. The naive approach of "dump all the logs into the prompt" wasn't going to work.

But analysts don't read every log line either. They query, filter, pivot. They ask "show me everything this IP did" and work from summaries.

That's the approach I took: Python handles the heavy lifting (aggregation, statistics, outliers). The agent gets tools to query what it needs. Deterministic where you can, LLM where you must.

Test 1: BOTS Dataset (With IDS Alerts)

The Splunk BOTS dataset has 12 million events across firewall, IDS, DNS, and HTTP logs. It also has pre-labeled IDS alerts, which makes it a good first test. The agent has signatures to work from.

Spoiler Warning: The examples below reveal findings from the BOTS v1 dataset. If you want to work through BOTS yourself first, skip to "Test 2" below.

I gave the agent two tools: query IP and query domain. It started from alerts and pulled data as needed.

Here's what it found:

Finding: 40.80.148.42

• Who: IP address 40.80.148.42
• What: Triggered 589 IDS alerts including XSS, SQL injection, and information leak attempts
• When: August 10, 2016, with 28,119 total events
• Where: Targeting internal server 192.168.250.70
• Why: Repeated, diverse attack signatures indicate active exploitation attempt
• Verdict: MALICIOUS

It also found the ransomware infrastructure:

Finding: 85.93.43.236

• What: Triggered an alert due to a signature indicating TROJAN Ransomware/Cerber Checkin Error ICMP Response
• When: 2016-08-24 at 10:49:36
• Verdict: MALICIOUS

Two tools. Query IP, query domain. The agent traced an attack from web scanner to ransomware infection.

But this test had training wheels. The IDS told the agent what was bad before it started looking. The real question: can it identify threats from statistics alone?

Test 2: Auth.log Without Signatures

For the second test, I used a plain auth.log. No IDS. No threat intel. Just raw authentication events: failed logins, successful logins, disconnects.

The profilers computed statistics: which IPs were active, what behaviors they exhibited, which usernames were being targeted. But numbers alone don't tell an analyst what to do.

Here's what the profiler gives you:

{
  "ip": "173.192.158.3",
  "priority": "ELEVATED",
  "behaviors": {
    "AUTH_FAILURE": 303,
    "DISCONNECT": 445
  },
  "unique_usernames": 61,
  "total_events": 748,
  "first_seen": "Dec  2 04:12:40",
  "last_seen": "Dec  2 04:17:19"
}

That's useful if you know what to look for. 61 usernames in 5 minutes is a credential spray. But you have to recognize the pattern.

Here's what the agent gives you for the same IP:

Finding: IP 173.192.158.3

• Who: IP address 173.192.158.3
• What: High volume of authentication attempts with credential spraying (61 different usernames targeted) including 303 authentication failures
• When: The activity occurred within a short duration (Dec 2, 04:12:40 - 04:17:19) with frequent DISCONNECT events
• Where: The target host (limited to SSH authentication logs)
• Why: This behavior may indicate an automated script or brute force attack attempting unauthorized access
• Verdict: MALICIOUS
• Confidence: HIGH
• Recommended Actions:
  1. Block the IP address at the network level
  2. Examine firewall logs for broader context
  3. Questions remaining: What are the origins of the traffic? Are there any successful login attempts?

The agent looked at the same numbers and produced context, reasoning, and next steps. It identified the 5-minute window as significant. It recommended specific follow-up actions.

Which would you rather be handed as an analyst?

Results:

Twenty entities investigated in two minutes. All running locally on consumer hardware. No cloud API, no credits burned.

What It Missed (And How To Fix It)

The agent correctly identified distributed attack patterns across common usernames like "admin," "ftpuser," and "guest." But it missed something I noticed immediately: usernames like "D-Link," "PlcmSpIp," "vyatta," and "pi."

Those are default credentials for IoT devices. D-Link routers, Polycom VoIP phones, Vyatta network appliances, Raspberry Pis. This server was being targeted by Mirai-style botnets scanning for factory defaults.

The agent asked the right question ("Is this a valid system account?") but lacked the domain knowledge to answer it. That's expected. A 7B model trained months ago doesn't have current threat intelligence.

The fix seemed straightforward: give it access to threat intel.

• Web search tool - Let the agent look up unfamiliar usernames
• Threat intel database - Query a local database of known IoT defaults, botnet signatures, malicious IPs
• Paid platform API - At work, I use a threat intel platform that returns enrichment data via API. The agent gets the results and has everything it needs.

This experiment didn't have those integrations, so the gap is expected. The architecture supports adding them. That's the point.

The Takeaway

There's more than one way to approach this. The 7B model couldn't ingest all the data directly, so I gave it tools to query what it needed. The methodology of "deterministic where you can, LLM when you must" worked.

The process of identifying gaps and finding ways to fill them is where the learning happens. The agent missed IoT default credentials. That's a solvable problem: add a threat intel lookup. Identifying what's missing and how to fix it is more valuable than pretending everything worked perfectly.

Both tests proved the architecture is sound. The agent can reason about statistics without pre-labeled alerts. It produces findings an analyst can actually use. And it runs on consumer hardware.

I still want to train a dedicated security analyst agent for CTF competitions. But that's a different project. For this one, the goal was learning how to build an agent that could handle large datasets within the constraints of a 7B model on local hardware.

Mission accomplished.