Abstract

AI generates code faster than we can even read it. If we cannot read the code fast enough, we are unlikely to catch any of the quality gaps in its code or tests. Essentially, relying on the speed of LLMs reduces our role to a passive overseer - the irnoy is that our cognitive abilities do not catch logical bugs as just a passive overseer [1]. We must find a way to become active problem sovlers again.

This matters because LLMs are probabilitic but quick. While the speed of implementation has already scaled up, can the quality keep up? No one wants to use software that works sometimes, even if that software was built using probabilistic code generation.

Manual verification is too slow and even LLMs cannot find bugs effectively [2]. We must find a way to match the speed that AI brings with the quality that we expect from software. Human thinking can be the balance the between speed and quality.

But to embed human thinking into the SDLC today, we need a different mindset. Compared to traditional example-based unit tests, we need a more robust testing mechanism; one that is unaffected by the LLM’s blind spots.

Property based testing (PBT) is a mechanism where the test engine (and not the LLM) explores problem space to identify cases where system invariants do not hold. Not only does this circumvent any bias that LLMs have, it also identifies gaps in our mental models.

Invariant-driven exploration brings forth a new division of cognitive labour. LLMs bring speed. PBT brings rigorous exploration. Humans bring comprehensive thinking involved in defining the specifications and the invariants. This gives us speed and quality both, making invariant-driven exploration a very effective strategy today.

[1] https://arxiv.org/abs/2608.23642
[2] https://dl.acm.org/doi/10.1145/3696630.3728702

Problem Statement

Software engineering was always about ensuring that we solve the problem. In the era of LLM-generated code, we are instructing the LLM in terms of what we want to build but after it is implemented, we need to rely on something stronger than LLM-generated unit tests to ensure the final implementation we built conforms to our initial expectations. Essentially ensuring that we solved the problem.

Example-based test cases worked well when we were typing the code by hand because our thinking processes were more intimately inolved in the process. But today, the risk is that because the LLM generates perfect code most of the time, we are losing the ability to detect the few times where bugs are not caught by LLM’s own test cases.

Solution: Property Based Tests

PBT offers a different route than LLM generating test cases. It starts with abstract thinking - defining what it is that must always be true in the context of the problem. The process of defining these properties makes us an active participant in the problem solving process.

Once we have the properties we define, we can utilize test engines that explore the problem space. The LLM generates the source code, the human defines the properties that should hold always and the test engine deterministically explores the problem space for cases where the property does not hold. This three way cooperation achieves both speed and quality and is driven ultimately by human thinking.

I will use a meeting scheduling application to show this process in the talk.

Intented Audience

Engineers, Engineering leaders
Level: Intermediate

Key Takeaways

• We cannot rely entirely on LLMs for ensuring quality software
• Humans just being in the prompting loop does not work because our thinking processes do not engage that way
• Abstract problem solving is required to define wht fundamental properties of what we are building
• Exploratory testing can catch bugs that you or the LLM might not think of
• The three-way coperation is a recipe for success in today’s world: LLMs bring speed, test engines explore the problem space for quality gaps, humans bring comprehensive thinking

#testing #agents #propertybasedtesting #humans #problemsolving