You ship a feature that queries the DB and brings a lot of data into memory.
There is a separate thread that dequeues and processes logs from a queue in a loop.
The memory pressure starves or kills the logging path.
When the log queue is full, instead of dropping logs, it blocks until logs are dequeued.
So the main execution thread is fine for a few more minutes.
The server does not crash. And it's not like you are seeing any error logs.
But eventually the main thread tries to write logs, blocks on the full queue, and the program grinds to a halt.
The obvious issue - "the program stopped running".
The second-order issue - "the bad DB query created memory pressure that broke the logging path, which eventually blocked the main execution path".
The funny thing is, AI would be able to figure this out, but probably wouldn't surface this.
If AI listed every edge case that could happen, it would be untenable.
It has been trained to be a conversational chatbot and give a helpful answer. That means giving enough information to be useful, but not so much that the answer becomes noise. So it is not going to provide an exhaustive list of every possible way your system might catch on fire.
And AI still has trouble consistently distinguishing between plausible edge cases that should be considered and overly defensive ones (that just make the system more complex to reason about for no real benefit). So if it gives you a subset of ways your system might catch on fire, the most realistic one might not be in there.
So I guess what I am saying is that even though LLMs can do very good second-order thinking, they often will not surface second-order failure modes unless specifically prompted and given the right context.
It is on the human software engineer to know which scenarios are actually plausible and think through the second-order effects.