Beyond the Scanner

AI and automation can do a lot. But there’s still real value in an experienced tester who knows where to look, what to question and when to dig a little deeper.

Michael Jepson  |  Co-Founder, Technical Director & Lead Penetration Tester

A web application is more than the pages you can see. The APIs and services behind it need testing too.

If you spend five minutes looking at cybersecurity products online, you'll come across automated scanners, AI-powered testing platforms and some pretty bold promises. I've tried a fair few of them. Some are genuinely useful; others produce an impressive-looking report without much substance behind it.

That's not an argument against automation, I use it regularly. But just because a scan has finished doesn’t mean the job’s done.

What did the scanner actually reach?

One of the first things I check during an assessment is whether I can actually reach and test everything we've agreed is in scope.

A homepage loading or a scanner returning a long list of URLs doesn’t tell me that on its own. I've seen automated scans look as though they’ve covered a web application, while missing the APIs doing much of the work behind it.

In some modern applications, the website is little more than a front end.

The data access, business rules and functionality I need to test sit behind those API requests.

A capable scanner can test APIs too, with the right setup, documentation, accounts and access. The problem is assuming it’s covered them without checking. The same goes for areas behind a login, different user roles and workflows that take a few extra steps to reach.

An experienced tester should check what they can actually access and test, look into any gaps and explain anything they couldn’t assess. Manual testing can miss things too. Being clear about those limits is part of doing the job properly.

Coverage involves more than the visible web pages.

The weaknesses that need more thought

Automated tools are good at repetitive checks and picking up known issues. Understanding how an application is meant to work can take a bit more digging.

Take an invoicing system. A request to download an invoice might look completely ordinary.

But can a customer change an identifier and download someone else’s invoice? Can a user approve something they shouldn’t, or get around a check by doing things in a different order? These are questions about authorisation and business logic-areas covered in OWASP Top 10.

After years of testing applications, I've got quicker at seeing how the pieces fit together and spotting things that deserve a closer look. There’s no magic to it.  

It comes from practice, curiosity and following up when something doesn’t quite add up.

And what about AI?

AI can help with research, writing scripts, analysing responses and suggesting things to try next. I've seen tools that make testing more efficient, and I expect they’ll keep improving.  

I've also seen them confidently explain vulnerabilities that weren't actually there. Well known as AI hallucination.

AI can sound remarkably confident, even when it's completely wrong. That's why verification matters.

Something sounding convincing doesn’t mean it’s been demonstrated. And if a tool misses a weakness, you might get no indication that it never looked in that area.

Human testers get things wrong too. That’s why someone with the right technical experience needs to guide the testing, question the results and check that the findings stand up to scrutiny.

What would I ask before buying?

• What's automated, and how much hands-on testing is included?

• Will testing cover areas behind a login, different user roles and the APIs behind the application?

• How will the provider check for gaps in coverage and explain anything they couldn’t test?

• Who checks the findings and explains what they mean to the business?

  
  

Those answers tell me more to me than a dashboard full of alerts or an 'AI-powered' label.

The balance I believe in

For me, I don't see this as humans versus automation. Regular automated checks have their place, and good tools can save a tester a lot of time. That means more time to dig into the parts of the application that need a closer look.

At AppTest Direct, we're focusing on specialist-led penetration testing for web applications, APIs and external infrastructure, combining hands-on investigation with the right tools for the job.

My aim is simple: understand what we’re testing, check that our findings hold up and give customers clear, practical advice they can act on.