Darkbit has joined Aqua Security!


Why You Don't Need a Cloud Penetration Test... Yet

24 August 2020

For organizations moving workloads to the cloud, the primary focus is rarely on security. In most cases, the goal is to get the environment migrated or an application deployed and to the point of working. At some point down the road, making sure the environment is secure enough to run in production becomes a priority. Great, but how do we do that?


Two common initiatives organizations will undertake are penetration tests and vulnerability assessments, and despite decades of experience and examples to fall back on, there is still confusion about the differences between these two activities. In our view, these are very different projects with vastly different goals and outcomes.

Let’s try to help define these two terms:

  1. Vulnerability Assessment - Using automated tools to scan one or more networks, systems, or applications for known security vulnerabilities. The assessment involves identifying the potential existence of a vulnerability without actually exploiting the vulnerability. An analogy would be checking if all the doors and windows are locked but not actually using them to enter a building.
  2. Penetration Test - Using a combination of automated and manual tools and processes to identify known or unknown security vulnerabilities, exploit them on live systems, and see how far they can penetrate the environment. An analogy would be finding all doors and windows to be locked and picking the lock to enter the building and seeing how far inside the building a malicious person could go.

In short, a vulnerability assessment aims to show the entire attack surface area – what is theoretically possible. While a penetration test aims to illustrate how and where an attacker can actually get in.

The Driving Force

While penetration testing and vulnerability assessments are often compared as alternatives to each other, they should really be considered totally separate activities. There are reasons for each, but to truly understand which is better at a given time, we should be clear about their expected outcomes.

To help answer this question, it’s important to look at what is driving the requirement in the first place. Many organizations must comply with regulatory requirements, and many of those drive the need to perform periodic third party security assessments. Is a security assessment a penetration test? Is it a vulnerability assessment? Does the distinction even matter if the outcome we are after is regulatory compliance?

As security professionals, we are often critical of compliance driven security initiatives that look and feel more like security theater than effective and pragmatic security initiatives. However, we have to be realistic that in many cases, the presence of regulatory compliance is what is bringing us to the table to the first place.

If the only driver for a third party security assessment is compliance, then the decision is more semantic that anything else. If the ultimate goal is to reduce real risk, then it’s worth comparing some key elements of the two activities side-by-side to highlight their primary differences.

Key Differences

A vulnerability assessment, by definition, only generates findings of known issues, but it’s typically very good at achieving wide coverage of the environment within a limited budget and time window. A penetration test can be a more realistic gauge of the skills and methodologies used by a real adversary, but the cost and time-boxing of typical engagements means those skilled professionals can only dedicate a certain amount of time to each potential attack path. Coverage is sacrificed in exchange for higher quality findings with detailed explanations of how they succeeded in the specific environment.

Penetration Test Yes No
Covers broader surface area - X
Can be reliably automated - X*
False negatives X -
False positives - X
Live exploitation X -
Coverage limited by time X -
Coverage limited by budget X -

*Note: penetration testing absolutely involves automated tooling, but precise exploitation of specific systems will always require skilled, manual effort.

Vulnerability Assessment Yes No
Covers broader surface area X -
Can be reliably automated X* -
False negatives - X
False positives X -
Live exploitation - X
Coverage limited by time - X
Coverage limited by budget - X

*Note: while vulnerability assessments benefit greatly from heavily automated tooling, manual verification is almost always needed to minimize false positives and incorrect results that tools inevitably produce.


Clearly, there are trade-offs with either approach. With penetration testing, you can be more certain of exploitable vulnerabilities. However, you can never know what you don’t know. Are there other attack paths that your penetration testing team didn’t cover? With a vulnerability assesssment, you’ll likely get a lengthy report with a large number of vulnerabilities that you are left to triage and decide whether or not they present a risk to your organization.

Understanding these trade-offs is not meant to crown a winner between the two approaches. Instead, we advocate an ordered approach, based on the maturity of your organization from a security standpoint and the true return on your investment in leveraging third party teams to bolster your security posture.

The astute reader may have gathered by now that many of the inherent flaws in each approach can be mitigated by strategically executing the right engagement at the right time. For these reasons, we strongly encourage our clients not to undertake penetration testing engagements until they have performed a security posture assessment and have addressed the issues it uncovered.

What’s a Security Posture Assessment?

A security posture assessment is a vulnerability assessment adapted to modern, cloud-native environments. In cloud-native and containerized infrastructures, the majority of the environment is described and built via infrastructure-as-code (IaC) using declarative patterns against a set of APIs. Compute instances, virtual networks, firewall rules, container images, and their settings become the language by which the bulk of security settings are configured, so a more cloud-native version of a vulnerability assessment is necessary to properly validate these components for known vulnerabilities.

Instead of using a network reconnaissance tool like Nmap to discover all the exposed hosts and ports in an environment, it’s far more efficient and accurate to just query the cloud provider APIs and obtain that list. And if the cloud provider is operating the compute and management services for where workloads run, running a patch-scan might not be possible. Instead, the configuration settings of the managed service where those workloads run that pertain to security, observability, and availability can be queried and assessed. Finally, with cloud identities and permissions being assigned to systems and users at multiple levels in the stack, it’s extremely important to validate that the principle of least privilege is maintained and that known pathways for privilege escalation are identified and refactored out.

The key difference between a security posture assessment and a traditional vulnerability assessment is that we don’t have to rely on off-the-shelf software or tools to try to infer the state of a system or application. In cloud-native environments, we know the precise state of the resource in question because it’s dictated by the underlying API. This allows for a level of precision, depth, and efficiency not possible with a traditional vulnerability assessment.

When we look at security posture assessment in this specific light, it becomes feasible to anticipate a state in which posture assessment is actually a continuous activity. Instead of a point-in-time assessment, we can reset expectations around the idea that we should be able to know instantly when our environment has drifted into an insecure state. This is only possibly with a completely automated and API-driven security posture assessment.

So, when should I pen test?

Once the overall surface area is clearly understood by performing security posture assessments, it becomes more realistic to engage in tightly scoped penetration tests with clear objectives against systems known to carry higher risk of exploitation. This approach allows a penetration testing team to more quickly validate existing controls and spend the bulk of their time attempting to find and exploit the unknown issues. Back to the physical security analogy: you bring in the experts after you’ve verified the doors and windows are locked, the security system is armed, and the guards are well trained. This maximizes the time and money spent in relation to the specific benefits of each kind of test, and ultimately leads to higher quality findings and more timely guidance.