The Ideal Build Log #001: When Staging Passed but Production Failed

Here's a neat trick: build a search filter that blocks the good stuff and lets the bad stuff through.

Yep. Managed that one.

I've been building DomainJourney, a domain-name discovery tool, with the help of AI coding agents.

Recently, I ran into a problem that reminded me why passing tests and working in production aren't always the same thing.

## What went wrong with DomainJourney's search filter?

After fixing a few issues in staging, everything looked good.

Tests passed. The search filter appeared to be working.

Time to go live.

Or so I thought.

Once I checked the production website, I discovered two problems:

- A search term that should have been blocked slipped through.

- A perfectly legitimate medical term was rejected.

In software testing, these are two different kinds of errors.

A false positive happens when a filter blocks something it should allow.

A false negative happens when a filter allows something it should block.

My search filter was managing to do both.

Not exactly the result I had in mind.

## How did I fix the search filtering problem?

Back to adjusting the filtering rules, running tests, and checking the live website.

The important part was verifying both sides of the problem.

It wasn't enough to confirm that unwanted search terms were blocked. I also needed to make sure legitimate searches weren't being rejected.

After another round of fixes and verification, both cases behaved as expected.

The exact reason the earlier tests missed these cases wasn't established in this account. But the production results made one thing clear: the testing hadn't caught everything that mattered.

## What did I learn about AI coding agents?

AI coding agents can accelerate software development, but generated code still needs to be tested and verified in the environment where people will use it.

That was the main lesson.

AI can help write code, troubleshoot problems, and implement changes much faster than I could working through everything manually.

But faster development doesn't automatically mean reliable software.

A test passing doesn't guarantee that every important scenario has been covered.

And a fix isn't really finished until the expected behavior has been verified.

For this particular issue, I came away with three reminders:

1. Test what the system should reject.

2. Test what the system should allow.

3. Verify the results in production, not just staging.

Simple enough in theory.

Apparently, still easy to get wrong.

## Why I'm also building an AI project manager

Experiences like this are part of the reason I'm building my own AI project manager.

I want AI agents to do more than write code and report that a task is complete.

The goal is a system that helps coordinate development, track problems, document decisions, and verify that the work actually achieves its intended outcome.

Because there's a difference between an agent saying "the fix is implemented" and knowing that the problem has actually been resolved.

That's something I'm continuing to work on.

## Welcome to The Ideal Build Log

This is the first entry in The Ideal Build Log, where I'll share what I'm building with AI, what goes wrong, what works, and what I learn along the way.

Not just the successful launches or polished results.

The debugging, unexpected problems, experiments, and small wins that happen behind the scenes.

And since DomainJourney started this particular story, here's a small plug.

Looking for a .com domain name for your next project?

Try DomainJourney for domain name suggestions.

More build logs to come.