How pattern detection works on log files

Reading time: 6 minutes

Pattern detection is the technique of recognising which parts of a log line vary (timestamps, IDs, addresses) and which parts stay the same. Once the variable parts are replaced with placeholders, lines that previously looked different become identical and can be grouped together. The result is a count of how many times each distinct event occurred.

This guide explains how the detection works, what kinds of variables it recognises, and how the ordering of detection passes affects the result.

The fundamental principle: replace variables with placeholders

Consider these two real log lines from a web server:

2026-09-27T10:00:01Z ERROR timeout reading /api/users
2026-09-27T10:00:06Z ERROR timeout reading /api/users

These two lines are saying the same thing: ERROR timeout reading a specific URL. The timestamp is different, but the pattern is identical. A pattern detector recognises the timestamp as a variable ({ISO_TS}) and the URL as a variable ({URL}), producing:

{ISO_TS} ERROR timeout reading {URL}

Both lines match this pattern. Count: 2. That is the entire idea, applied at scale across 20+ variable types.

The variable types a pattern detector recognises

A good pattern detector recognises enough variable types to cover the most common sources of variation in logs. Logdedupe's detection engine covers these categories:

Timestamps and dates

ISO 8601 timestamps (2026-09-27T10:00:01Z), syslog-style dates (Sep 27 10:00:01), Unix epoch timestamps (1727438401), and several other date formats. These are often the noisiest variable in any log, because every line has a different timestamp.

Network addresses

IPv4 addresses (192.168.1.100), IPv6 addresses (2001:db8::1), MAC addresses, and port numbers. These appear in access logs, firewall logs, and connection-related messages.

Identifiers

UUIDs (550e8400-e29b-41d4-a716-446655440000), hex identifiers (short and long), request IDs, and session tokens. These are the most common source of false negatives — two lines that are otherwise identical but differ in their request ID will not group without proper placeholder detection.

Paths and URLs

File paths (/var/log/nginx/access.log), URLs with paths (/api/v2/users/1234), and full URLs with query parameters. The detector must handle path segments that contain other variable types (a UUID in a URL path, for example).

Numbers and units

Integers, decimal numbers, durations (342ms, 1.5s), byte sizes (64MB, 1.2GB), and percentages. These appear in performance metrics, memory usage lines, and timing information.

Why the order of detection passes matters

Pattern detection is not a single regex — it is a series of passes, each looking for a specific variable type. The order determines which placeholder wins when a string could match multiple patterns:

• ▶ UUIDs before hex numbers — a string like 550e8400-e29b-41d4-a716-446655440000 is a UUID, not a sequence of hex numbers. UUID detection runs first so it takes priority.
• ▶ IPv6 before IPv4 — an address like ::ffff:192.168.1.100 contains dots and digits that could match IPv4. IPv6 runs first so the full address is captured as one placeholder.
• ▶ ISO 8601 before generic time patterns — 2026-09-27T10:00:01Z has a specific structure that should not be broken into separate date and time placeholders.
• ▶ URLs before individual path segments — a full URL like /api/v2/users/550e8400-e29b-41d4-a716-446655440000 should be one placeholder, not a path with a UUID inside it.

Logdedupe runs 15 passes in this carefully ordered sequence. The result is that every variable gets the most semantically specific placeholder available, keeping patterns readable.

What pattern detection looks like in practice

Here is a real example. A log with 16 lines containing 5 ERRORs (timeout reading various URLs), 4 WARNs (slow queries with different durations), 2 INFOs (logins from different IPs), 3 ERRORs (connection pool exhausted — identical, no variables), and 2 FATALs (out of memory on different nodes). After detection, the 16 lines collapse into 5 patterns:

Notice that the FATAL lines (2 occurrences, 12.5% of total) stand out immediately. In the raw log they were buried between ERROR lines and might have been missed.

Limitations and edge cases

Pattern detection by regex is heuristic. It works well on structured log formats (timestamps, levels, messages) and less well on arbitrary text where variables do not follow recognisable patterns. Specific limitations:

• ✘ Log lines that contain only variables (a timestamp and nothing else) cannot be meaningfully grouped because every line produces the same pattern.
• ✘ Lines where the variable does not follow a detectable format (a custom ID like ABC-123-XYZ without a clear pattern) may not be recognised and may prevent grouping.
• ✘ Multiline log messages (stack traces, exception details) are treated per-line. A stack trace becomes many separate patterns unless the lines are pre-joined.

These are not bugs — they are inherent limitations of a serverless, in-browser approach that prioritises privacy over integration depth.

Use the tool to see it work

The best way to understand pattern detection is to see it on your own logs. Open the tool, paste any structured log text, and watch the lines collapse into patterns. The tool includes sample logs you can load with one click to see the effect immediately.