Debugging CI/CD pipeline logs faster

Reading time: 5 minutes

A failed CI pipeline often produces thousands of log lines. The first ERROR in the output is buried under 200 INFO lines about downloading dependencies, caching packages, and running setup steps. The actual failure — a test assertion, a compilation error, a missing environment variable — shows up somewhere in the middle of a wall of repetition.

Scrolling through a CI log chronologically is slow, and the eye naturally skims past the one line that is different. This guide explains a better approach using pattern collapsing: reduce the whole log to the handful of distinct events it contains, and find the failure in seconds.

Why CI logs are especially noisy

CI pipeline logs have a few characteristics that make them harder to read than server logs:

• ▶ Command echo. Many runners print every command before executing it. The same 10 setup commands generate the same output on every run.
• ▶ Progress lines. Downloading, extracting, installing — each step prints status updates that are identical except for the item being processed.
• ▶ Cached output. Restoring from cache prints hundreds of lines like Cache hit: node_modules/package-a where only the package name changes.
• ▶ Step wrappers. Each pipeline step is bookended with ##[section]Starting: Build and ##[section]Finishing: Build — identical every time.

These structural repetitions are exactly what a pattern detector handles well. Every Cache hit: {PATH} becomes a single pattern with a count. The 200 identical-looking setup lines collapse into maybe 10 patterns.

How to process a CI log with Logdedupe

The steps are straightforward, and the tool is designed to work with any structured text — it does not need to understand your CI provider's format:

1. Copy the log from your CI provider. Most providers (GitHub Actions, GitLab CI, CircleCI, Jenkins) let you copy the full build log as plain text.
2. Paste it into the Logdedupe tool. Or use Open file if you downloaded the log as a .txt or .log file.
3. Click Deduplicate. The tool processes everything in your browser and shows you the distinct patterns.
4. Filter by ERROR or FAIL. Use the level filter checkboxes to show only ERROR and FATAL patterns. This narrows your view to the lines most likely to contain the failure.
5. Sort by rarity. The line that caused the pipeline to fail is often the one that appeared once or twice. Sorting by rarity surfaces it immediately.

What a collapsed CI log looks like

A typical CI log of 4,000 lines might collapse to around 30 patterns. The most common patterns (10+ occurrences each) are usually INFO-level setup steps that succeeded every time. The least common patterns (1–2 occurrences) include:

• ▶ The actual test failure. A pattern like FAIL test_api_auth: AssertionError at line 142 — appeared once, counts as 0.5% of total lines.
• ▶ A compilation error. A pattern like {PATH}:{NUMBER}:{NUMBER} error: expected ';' before '{TOKEN}' — appeared 3 times (one for each affected line).
• ▶ A timeout. A pattern like Error: Connection timeout after {NUMBER}ms — appeared 5 times, but in a 4,000-line log that is only 0.1% of the total.

The key insight: before collapsing it, you had to scroll past 3,970 lines of noise to reach those 30 relevant lines. After collapsing, you see 30 patterns ordered by whatever dimension matters to you — and the failure is visible immediately.

What about different CI providers?

The pattern detector in Logdedupe does not care which CI system produced the log. It recognises general variable types — timestamps, file paths, numbers, URLs — regardless of the surrounding format. A GitHub Actions log, a Jenkins console output, and a raw test runner output all work the same way: paste, deduplicate, read the patterns.

The only requirement is that the log is plain text. Binary output, ANSI colour codes, or formatted tables may not group cleanly. If the log contains terminal escape sequences, strip those before pasting.

Try it on your next failed build

The next time a CI pipeline fails and the log is too long to read, paste it into Logdedupe first. It takes ten seconds and costs nothing. If the patterns do not help, you have lost only a paste. If they do, you have found your failure faster than reading top to bottom.