Penetration testing has been included as a checkbox in compliance frameworks and a regular line item in security budgets for a long time. What is it actually for? What value does penetration testing really bring, and why would you want to do it?

This article, the second in our Purpose and Execution of Penetration Testing series, explores the purpose of penetration testing, how consultants differ from real attackers, and why understanding systems is important when operating a modern IT environment.

The Goal of a Penetration Test

The main goal of a penetration test is to review a target (usually an application, network, organisation’s IT infrastructure, etc.) for security vulnerabilities and security hardening opportunities to reduce the likelihood of a compromise. This improves the “security posture” of the environment, and reduces the likelihood of a real attacker exploiting the application, network or organisation.

The penetration tester assesses the target using techniques that overlap with what real attackers do. This isn’t always the most efficient route, and only covers finding vulnerabilities. A good penetration test will also assess other security controls (Adrian’s 2025 Kawaiicon talk on building secure systems has great insights on types of security controls), hardening mechanisms and architectural flaws by performing more in-depth technical analysis of the system overall.

We generally communicate this information through a list of findings and summaries in a report. We usually consider and tag findings in three broad categories:

• Bug. Erroneous logic or configuration. Typically, software flaws that can be remediated by fixing the vulnerable logic or configuration. These are generally vulnerabilities that can be actively exploited for some impact.
• Hardening. Missing security controls or hardening mechanisms that reduce the likelihood of exploitation or future vulnerabilities. This includes both fundamental security controls, such as transport-layer encryption, as well as defence-in-depth controls such as rate limiting or security alerting. A test could find no bugs in a target, but could highlight additional areas where hardening can be improved to prevent bugs from occurring or being exploited in the future.
• Architecture. Weak or insecure architectural technology or design choices. These issues are inherent in the design, and are mitigated through compensating controls or redesign. I think of these as “fact of life” vulnerabilities, and are usually artefacts of the systems the client has chosen to use. The goal with these is to increase our understanding of the issue, and discuss compensating controls.

A quality penetration test should analyse a system and find actionable issues in each of those three categories. Further, the recommendations should explain how to improve the system and remediate the findings.

The Race Against Attackers

Penetration testing and cybersecurity in general is a race against real attackers. Consultants and testers are not the attackers, though we may use some of the same trade-craft. Most importantly, our actual end goal is to improve the system.

Real attackers have no such constraints. They operate without scope boundaries, they can spend weeks or months on a single target, and they exploit vulnerabilities that we might not even know exist yet, and their goals are often less than benevolent.

The purpose of penetration testing is to find issues before the attackers do, to understand the practical security posture of a system, and to provide actionable guidance on how to improve it. It’s about reducing the window of opportunity for an attacker who is actively looking at your systems.

In my hybrid assessment article, I discussed how real-life attackers aren’t limited by penetration test scope. A traditional black-box test might focus on a web application’s or network’s front door, but an attacker can exploit a myriad of other things like unknown API endpoints (like the Optus API breach), integration points (like the 700Credit data breach), or developer access that crosses boundaries (the LastPass compromise). The race against real attacks is an ongoing process. Systems evolve, new vulnerabilities are discovered, and attackers change their tactics. Penetration testing methodology needs to keep up. Scratch that, penetration testing methodology needs to actively cheat to win.

Since our goal is to improve the system, executing the penetration test is best done using any techniques we can to maximise the usefulness of our time and advice. Practically, that means leaning into the differences between consultants and attackers to tip the scales in our favor.

Consultants Are Not Attackers - This Is Good

This whole “consultants are not real attackers” discussion might seem a bit odd without some history. The penetration testing industry effectively started with pioneering hackers finding vulnerabilities in things, and trying to get them fixed. Over years this grew into the cyber security assurance industry we have today, but the “bunch of hackers” aspect sometimes lingers. It’s fairly common to have penetration testers act like some simulation of real attackers, but this mindset has its limitations in a security engineering context.

By working together with clients and using access to information real attackers don’t have, we tip the scales to find and resolve more issues across a wider breadth (this is called “coverage”) in a smaller timeframe (time spent is proportional to engagement cost).

For example, we can use access to source code to quickly determine vulnerabilities and assess whether they are a single outlier in an otherwise secure system, or a repeating pattern that requires further effort to remediate across a platform. We can discuss the realities of operating applications and systems to make sure our recommendations are actionable, instead of parroted industry best-practice one liners (a recommendation that cannot be implemented is a bad recommendation, in my opinion), and we can work together to increase our shared understanding of technology.

Speaking of shared understanding…

The Complexity Problem

Computer systems are Complex. Not “the kind of Complex that takes a while to understand”, but the kind of Complex that means no single person fully understands every component, interaction, and failure mode of every item in the chain. A practitioner who understands every aspect of IT systems from what happens when the switch under your keyboard triggers, all the way up to Cloud-provider-specific automation technology is extremely rare and possibly some form of “chosen one”.

This Complexity is the result of layers of abstraction, third-party dependencies, evolving requirements, and human decisions.

Consider a typical modern application (I’m being generous here by citing a modern application example and not a 30 year old monolith of technical debt and year-on-year features and partial rewrites):

• A web front-end built with a JavaScript framework
• API endpoints in a language like Go or .NET
• Database queries through an ORM layer
• Authentication handled by a third-party identity provider
• Infrastructure managed by Kubernetes or Terraform
• CI/CD pipelines that build and deploy everything

Each of these components introduces its own attack surface, and each attack surface has the potential for exploitable vulnerabilities. Understanding this complexity takes time and effort. Computers aren’t magic, they are perceivable, but understanding them thoroughly is a massive undertaking.

This is why good penetration testers can be a little weird. We have to stuff all of this information into our minds, figure out where to set the needle on the level of detail we care about, and distill this all down into a set of targets and eventually recommendations on how to improve a system. This process is non-deterministic, very hard to automate beyond the trivial parts of the job, and can incur a big cognitive load without the appropriate support mechanisms in place.

This is why reverse engineering and deep-dive analysis can be so valuable. In my Ducati CANBUS reversing article I decoded proprietary protocols to understand how systems communicate and attack the immobilizer logic. In the .NET dynamic debugging article we discussed line-level debugging without source code to understand exactly how user input was processed. These techniques (reverse engineering, dynamic analysis, and deep understanding) are essential to modern penetration testing and we use them to tip the vulnerability discovery scales in our favor.

The purpose of penetration testing extends into understanding the Complexities of modern computer systems, and figuring out how to make them safer through actionable findings. You should absolutely be leaving a good penetration testing engagement with a better understanding of how your systems really operate.

The Box-Checking Fallacy

One line I’ve heard a fair bit over my career is “penetration testing is just for compliance”. I have heard this often, and recently enough that it’s stuck in my brain. I’d like to use this article as a platform to complain about this take :-)

Various compliance frameworks like PCI (take your pick of which one), ISO27001, SOC2, have requirements for penetration testing. To maintain compliance, penetration testing must be completed. Testing is expensive, though, so the engagements are artificially limited in time to minimize cost outlay and “check the box”. I’ve recently seen another group’s report for a million-line code base where the poor testers were given 3 days a year to review said million lines of code (If you are reading this, my people. I see you. Stay strong. Kia kaha.)

3 days is enough time to set up the testing environment, perform a cursory review of only basic parts of the application and maybe a pass with some automated tooling, and then onto the report writing. Simply not enough time for a penetration test against a target of that scale.

The intention of the compliance framework was to reduce the likelihood of exploitation by identifying vulnerabilities and improving cybersecurity posture through robust security engineering. Did this three-day test meet the requirements of the compliance framework? Well, it depends on the auditor. Maybe. In this case, yes. Did the intent get met? No.

We reviewed the same system with a more appropriate scope and found multiple vulnerabilities that had gone undetected for years. This isn’t because the previous penetration testers on the tools were doing a bad job. Their hands were effectively tied! There was a fundamental failure in the scoping, methodology and execution process of the previous tests.

The point of these frameworks is to certify that organizations meet a certain standard. Papering over the cracks with the bare minimum of testing required is not quality security assurance.

That’s not to say there aren’t ways to improve cost effectiveness, but this requires radical transparency and even smarter engineering using the various techniques we’ve discussed so far. Not just an arbitrary deadline.

Summary

This article was intended to provide our view on the purpose of penetration testing. Finding and helping remediate vulnerabilities, improving cybersecurity posture, and building our shared understanding of technology.

There are other reasons why you may want to do technical security analysis beyond just what was discussed here, but that would require a thesis instead of an article. Working together and building our shared understanding of technology is the priority, regardless of if you’re building things or breaking things.

Continuing on with my desire to demystify penetration testing in the next article, we’ll look at what makes a quality penetration test. What to expect, the trade-offs involved, and why no penetration test will ever find all the bugs (and why that’s okay).

Have thoughts on this? Want to discuss how your next engagement could be better structured? Drop us a line at info@pulsesecurity.co.nz.