Defence in Depth as Code: Test Driven Application Security

19 September 2023

In the article Secure APIs by design we implemented a secure API built according to the six-step model presented in Secure APIs. This article will show the importance of tests with focus on security, and how to implement a set of tests that will address many of the common vulnerabilities found in the OWASP Top 10 lists.

All code that we will show here is available at our Github repository, and we encourage you to take a closer look and try it out at https://github.com/Omegapoint/defence-in-depth.

Martin Altenstedt, Omegapoint

There are many ways to implement tests. The code here is just one way to do it, not a template that will fit everyone. Hopefully, it will help explain the details and challenges we developers face and inspire you to write tests with a focus on security.

Erica Edholm, Omegapoint

Please note that even if we use C# and .NET 7 in this article, the tests and concepts are general and valid for any framework when implementing APIs.

There is also a repo using Java 17 with Spring Security available at https://github.com/Omegapoint/defence-in-depth-java

When performing penetration tests it is very common to find vulnerabilities that are represented on the OWASP Top 10 lists, note that in addition to the one for web applications there is also one specific for APIs.

Based on our experience from years of performing penetration tests and security reviews, more often than not, vulnerabilities from these lists can be traced back to unclear non-functional requirements and a lack of security-focused tests. Read more about penetration tests and security reviews in our article on Offensive application security.

If these findings were identified during development or testing, we would simply call them bugs. Wouldn’t it be great if we could catch “security bugs” just as early as all other bugs? Before the application has been in production for a while?

But even if the findings are just bugs in the backlog and addressing them is part of our secure development process, it is important to note that there is a significant difference between functional bugs and security bugs, a k a vulnerabilities.

Functional bugs in production that our test processes miss will often be reported by users, while vulnerabilities most likely won’t be discovered by users. A malicious attacker who exploits it will probably not be discovered. Intrusion detection is hard, and the vulnerability is likely to stay open for a long time. Maybe until a penetration test is performed.

Thus, it is for many applications a valid argument that finding security bugs before deploying to production is even more important than finding functional bugs.

If we think of vulnerabilities as bugs, then security is just another non-functional requirement and should be part of our quality assurance process (QA). Our industry has well-established patterns and practices for QA, where one is Test Driven Development (TDD).

With this article we will show how to apply TDD to application security. We show that a fairly simple set of tests will help us address many common vulnerabilities and find security bugs during development, not after deployment to production.

Tobias Ahnoff — Note that we are not saying Test-First, but we are saying Test-BeforeDeployToProduction.

Martin Altenstedt — It sounds obvious and simple, and most teams do implement automated tests, but we rarely see tests for security. Well-written test cases will help us identify the boundaries of our functionality. If we have a hard time writing good, clean tests then we might have a problem with our understanding of the domain.

Tobias Ahnoff — Or put in another way, by Bruce Schneier (in 1999): “You can´t secure what you don´t understand” 1. And tests will help us understand and define exactly how a function should work, not just the happy case.

Security requirements and test cases for APIs

Before we dive into test cases and code, we need a little bit of context and to elaborate about requirements from a security perspective.

Let’s say we have a system with a set of three APIs, where API3 is for system administration only.

Figure 1: API Requirements and boundaries Figure 1: API Requirements and boundaries

Then, to assert all user functionality in one line, we could define the following requirement:

This would meet any functional requirement and we would typically verify that this works in a test environment, before deploy to production.

But note that the requirement is not well-defined, since it also allows the ordinary user to access functionality intended for administrators and services. To address this, we could express the requirement as:

This sounds easy and completely obvious, but teams often fail to identify the test cases needed to avoid vulnerabilities. It is essential for security that we identify boundaries in our requirements.

Tobias Ahnoff, Omegapoint

This is what many attackers do. They simply try to misuse our functionality and break out of boundaries and execution contexts. But it is hard for teams to switch to a mindset of an attacker where they are looking for abuse of the system. If we instead think in terms of boundaries and a well-defined domain, we can minimize attack vectors without being experts on attacks.

Notice the word only, how do we verify that? A common practice is to add negative test cases, one example is:

Given the context defined above, this will assert that users can only reach API1 and API2.

Another common requirement is tenancy isolation, which could be expressed like this:

We rarely find requirements or test cases that verify tenancy isolation, but many teams identify this threat among their top concerns, and it is number one on the OWASP Top 10 for APIs.

Pontus Hanssen, Omegapoint

During penetration tests we almost always find cases of broken access control, it is No 1 on OWASP Top 10 for a reason! In my experience systems that implement multi-tenancy are even more susceptible to bugs of this class. It’s more common to properly verify that a user has the rights to perform a specific operation (e.g. edit a blog post), but more uncommon to verify that the user is allowed to perform the operation on a specific piece of data. Is the user allowed to edit this blog post that was created by someone else?

If we think in terms of requirements and the need to make them exact. Then it is easy to add automated unit, integration, and system tests to verify that our code works as expected, even if someone tries to misuse our application by buying “-1 book” or set a product id to “DROP TABLE”.

Well-defined requirements and boundaries are the very core of how we build systems that are secure by design. Read more about this in the book Secure by Design by Dan Bergh Johnsson, Daniel Deogun and Daniel Sawano.

Besides a well-defined domain and trust boundaries we also need an understanding of how to build secure APIs.

A secure API can of course be implemented in many ways, but you need to handle the following 6 steps in one way other the other for all requests.

Martin Altenstedt, Omegapoint

This is what we are looking for when we do security reviews, penetration tests and review code. Based on experience, we believe that you need to address all of these steps to get a strong solution.

Figure 2: Secure API model implemented in [Secure APIs by Design] Figure 2: Secure API model implemented in Secure APIs by Design

For the context of this article we want to highlight the following aspects:

Note that the order of steps 4 and 5 might differ depending on implementation details. We can do basic authorization, like access to operation based on scopes, at the routing level, in the API controller or in a gateway or both. But this should not replace token validation in the API and any authorization in the domain layer, within the trust boundary.

Pontus Hanssen, Omegapoint

We often see solutions where all access control is implemented at the gateway level using for example AWS API Gateway. This is often done to keep “security in one place” and focus let developers focus on business logic in their APIs.

This is a risk-filled pattern, not only does it make step 6 impossible to perform. Access control functionality must be tested using system tests on a live system, and slight misconfigurations might bring the system’s whole security model down.

The same reasoning applies to input validation.

For a strong solution we need to build defenses in multiple layers, and the core business domain should not assume that other layers has enforced security like input validation and authorization. One example of this strategy can be found in the Git repo, where we use both ASP.NET Core Authorize attributes in the application layer (the Controller) and verify access to the operation in the service layer.

Given this context and the model for a secure API, we need to identify the tests we need.

A good place to start is OWASP, especially ASVS and WSTG will help us identify requirements and test cases with focus on security.

For the context of this article we have only listed the most relevant ones. For your application, you might identify additional categories, like compliance and regulations or performance.

Implementation of automated tests

The following sections define a set of tests we believe all APIs should have. They cover the test categories above and we implement them as system tests which runs against a deployed instance (no dependencies mocked) and unit tests (with all dependencies mocked.)

Code can be found at Github for both .NET and Java.

Note that larger projects would most likely require more structure and test on different levels of integrations. The important part is what kind of tests we added to verify our model for secure APIs, this is not a complete test suite for a real-world product API.

System tests

The system tests will concern step 1 and 2 in our API model, and the test categories:

With these tests we want to assert requirements for:

For Availability in a Kubernetes environment it is a common practice to use two endpoints: Liveness and Readyness. But for other applications a “ping” endpoint might be enough.

The Liveness endpoint, allows anonymous request and will respond with a 200 OK. Since it is anonymous it is important that this endpoint does not consume any system resources or leak any internal information.

The Readyness endpoint requires authenticated requests, for REST API endpoints this typically means a valid JWT. Since the request is authenticated we can do a deeper system health check, e g database calls or requests to downstream services.

It is of course important to verify that all instances of our API are running the correct version. This can be achieved in different ways. One way is to return the version of our deployed software from either the Readyness endpoint or the Liveness endpoint (given that the software version can be considered public information).

Martin Altenstedt, Omegapoint

Note that we might also add some more advanced automated tests (sometimes called smoke tests), one example is to verify a critical business flow. This would of course also require proper authentication and access control.

To assert authentication of requests we need to add negative tests where we try to use different kinds of invalid tokens. The tests we need for that depends on your JWT-validation component, and as noted earlier this should not be custom code. Typically we should validate that expired tokens are rejected, but there are many more tests for JWT validation, see e g https://42crunch.com/7-ways-to-avoid-jwt-pitfalls/.

Tobias Ahnoff, Omegapoint

You could of course argue that JWT validation is part of Authorization and not Authentication. But for this article it is not important to distinguish between the two.

In the repo we have implemented the following tests to cover this:

Björn Larsson, Omegapoint

Error messages and stack traces provide a window into the inner workings of an application. During black-box engagements they often contain information which allows us to find and exploit more severe vulnerabilities. By not allowing an attacker to debug your application you can significantly increase the threshold for an attack.

We also need system tests for each endpoint and have included the following tests for the Products controller. Notice the negative test cases, without them we might have deployed an open API (allowing anonymous requests), because the happy case always works!

Note that you should also run system tests in production. We need to verify that our basic defenses work in production, not just during test. But make sure that these tests do not disturb the production environment or introduce attack vectors with weaker protection.

Adrian Bjugård, Omegapoint

Test clients are always a challenge. Make sure to always practice least privilege! For example, a test client that will be used for system tests in production should only provide access to the specific functions required for the tests, without the ability to provide “normal” access to the system. It is also good practice to monitor how and when this client is used, to detect misuse.

Unittests – ProductsController

The first set of unit tests are for the Controller and the GetProductById API endpoint. They will address the responsibilities of the controller, which basically is to deal with basic input validation and return the correct result and response codes. We should not have any business logic in the application layer, this should be part of the domain/service layer.

We connect this to the test categories: Data confidentiality and Integrity.

The tests can be found at Github in ProductsControllerTests.cs.

Note that they also include tests to assert the correct request and response models, to avoid the common mistakes of “Mass assignment” and “Excessive data exposure” from the OWASP TOP 10 list.

Tobias Ahnoff, Omegapoint

For “no access to data” we can choose between 403 or 401, depending on if the existence of the data is considered to be public information. If it is important to not give this information to the client then we should return 404. One example is from Open Banking where a bank account number might not be considered to expose a customer (and we could return 403), but for endpoints where we use e g a personal number this would not be the case (and we should return 404.)

Unittests – ClaimsTransformation and PersmissionService

These tests are to ensure correct logic for mapping token claims to the permission model (not the actual permission configuration, since it is a unit test where all dependencies are mocked.) They will concern step 3 and the categories: Authentication and Authorization.

We want to point out that this can be done in many ways. For many frameworks a common pattern is some sort of principal for the request context. In .NET this is called a ClaimsPrincipal. We could use this approach, it works, but often forces you to represent a rich permission model as strings (key-value pairs.)

This has some drawbacks and is a missed opportunity to utilize strong typing, compile time validation, and provide developer experience using complex types.

Pontus Hanssen, Omegapoint

Mistakes in code for access control is common, many of the things on OWASP TOP 10 concerns authorization code. A good way to limit the risk of mistakes is to use strong types instead of strings.

A stronger pattern is to introduce a Permission service. This provides us with a strongly typed centralized pattern for authorization. It is also well suited for unit testing and supports more complex scenarios where access control needs to handle external lookups, caching or be supported by some sort of backend policy service, perhaps shared within the organization (e g using OPA.)

public interface IPermissionService
{
    bool CanReadProducts { get; }

    bool CanWriteProducts { get; }

    bool CanDoHighPrivilegeOperations { get; }
        
    MarketId MarketId { get; }

    UserId? UserId { get; }

    ClientId? ClientId { get; }

    AuthenticationMethods AuthenticationMethods { get; }

    bool HasPermissionToMarket(MarketId requestedMarket);
}

Note that the first part is basically claims from the JWT, while the second part are permissions based on claims from the JWT. Often we get these permissions from a database or service lookup, but it could also be a simple in-memory data structure for more static configuration. Either way it is often done using the identity of the caller mapped to roles (RBAC.) Together with other authentication metadata (typically OIDC claims like amr or azp) and permissions given to the client (OAuth2 scopes and JWT audiences.)

In the repo we have an CanDoHighPrivilegeOperations example where a user needs to have: The admin role and used MFA at the time, with a client that has the write-scope and the audience for the products API.

We also have support for verifying access to the data by calling HasPermissionToMarket, asserting that a user should only have access to products for a given market.

Note that these test cases will vary depending on our model for identity and access control, but typically we will have some sort of roles and need to support different kinds of system integrations. These are a few examples:

Unittests – Domain layer

The final set of unit tests is for the Product domain, which we also refer to as the service layer. They concern step 4, 5 and 6 in our API model and the test categories: Authorization, Data confidentiality, Integrity and Accountability.

We use a Domain Driven Design approach, based on patterns from the book Secure by Design.

It is important to note that you don´t have to use DDD, but we believe it is a strong pattern which clearly identifies trust boundaries and will help you build systems which are secure by design.

We use the term domain layer. Which has a set of services, where any given service has a clear responsibility and implements a well-defined trust boundary. All operations are handled by services, no direct access to data using the repository layer. Regardless of if we use DDD or not, any design we use should assert the following key aspects are addressed for a given trust boundary:

In the article Secure APIs by design we elaborate more on the importance of trust boundaries. This picture is one representation of the the design of our API.

Note that even if the core business domain is our inner most trust boundary, there is also a trust boundary surrounding our backend API (represented by the thick blue hexagon.) And it is when leaving that trust boundary output encoding for databases etc is needed.

Output encoding for the web application needs to be handled in front-end code (depending on html, css or javascript context.)

This might seem like a contradiction, but the domain needs to be represented to some extent in the front-end as well, dealing with output encoding, client side input validation and enable functionality based on user permissions to provide good UX.

The Products service GetById method captures this design, complete code is found at ProductService.cs.

public async Task<(Product? product, ReadDataResult result)> GetById(ProductId productId)
{
    if (!permissionService.CanReadProducts)
    {
        await auditService.Log(DomainEvent.NoAccessToOperation, productId);

        return (null, ReadDataResult.NoAccessToOperation);
    }

    var product = await productRepository.GetById(productId);

    if (product == null)
    {
        return (null, ReadDataResult.NotFound);
    }

    if (!permissionService.HasPermissionToMarket(product.MarketId))
    {
        await auditService.Log(DomainEvent.NoAccessToData, productId);

        return (null, ReadDataResult.NoAccessToData);
    }
    
    // Here we can do more complex logic, like finding out if it is available in stores etc.

    await auditService.Log(DomainEvent.ProductRead, productId);

    return (product, ReadDataResult.Success);
}

Note how this pattern implements Zero Trust, Least Privilege and Defence in depth. Where the core business domain does not trust that other layers has enforced security like input validation and authorization.

But at the same time other layers should do authorization and input validation. It is perfectly fine to do basic parts of authorization and input validation twice (or more), as long as domain logic is kept in the domain.

An example of this is that we also do basic authorization in the application layer using .NET Authorize policies or Java Spring Security PreAuthorize tags.

Summary

With this article and repo we have showed that, with a reasonable effort, we cover several important parts of security testing and that we can use TDD thinking for application security where we meet security requirements, and address most of the issues on the OWASP Top 10 list.

None of the tests we have shown are hard to write, but they are selected with a little care and thought to verify the boundaries of our requirements. This is the hard part.

Tests will help you understand, identify and define boundaries. Automated tests will make sure your rules and defenses are applied and implemented correctly. This is vital for security.

“You can’t secure what you don’t understand” — Bruce Schneier, 1999 1

If all teams wrote these kinds of tests, then we as pentesters would have a much harder job. And the bar for attackers has risen significantly.

Martin Altenstedt — Will it scale? Will there be 50 tests for every single endpoint?

Tobias Ahnoff — Well, as always, our investments in security should be proportional to our business. Based on our experience, this implementation, with all the tests, are a reasonable level of security for many businesses. No APIs should have OWASP Top 10 vulnerabilities.

As a final summary we want to highlight the flowing key take-aways:

With this article we hope to inspire to do more tests with focus on security, maybe start with a few system tests that asserts a valid JWT.

Further reading

Defence in Depth

Defence in Depth as Code: Secure APIs by Design

Secure by Design

Offensive application security

Footnotes

  1. https://www.schneier.com/essays/archives/1999/11/a_plea_for_simplicit.html. ↩ ↩2