Securing AWS Transfer Family with AWS Web Application Firewall and Amazon API Gateway

AWS Transfer Family is a fully managed, serverless file transfer service for Amazon S3 and Amazon EFS. The service provides you with the flexibility to authenticate your file transfer client users using credentials stored in an identity provider (IdP) of your choice. You can achieve this by integrating an Amazon API Gateway endpoint backed by an AWS Lambda function that accesses your IdP. This enables you to seamlessly migrate from an existing platform without making changes to your end users’ credentials.

However, server endpoints open to the internet from any environment can be prone to port scanners and bots. Earlier this year, AWS Transfer Family launched support for Security Groups, so when you host the endpoint within your VPC, you can restrict access to known IPs only. Now, what if you do not have the list of known IPs, or you have a large list that goes beyond what security group rules can support?

To address these use cases, AWS Transfer Family recently launched support for using AWS Web Application Firewall (AWS WAF) to protect your API Gateway endpoints. This means you can leverage AWS WAF rules to protect your internet facing endpoints. The features include:

• Use AWS WAF rules to allow or deny access to end users based on their source IP addresses.
• Use rate limiting to protect against brute force attacks from common exploit tools, such as scanners and bots.
• Restrict access based on country of origin, which uses AWS WAF’s geographic matching capabilities.

In this blog, we demonstrate how to define an AWS WAF rule to block a set of IP addresses by leveraging the end users’ source IP address as a factor of authorization when authorizing access over SFTP. This solution is applicable to all protocols. In doing so, the AWS Transfer Family and AWS WAF integration not only protects you from bad actors and exploits, but it also helps you save on API Gateway and Lambda costs.

Solution architecture and overview

When an end user authenticates using your server, your server passes the end user’s credentials, protocols used, and source IP to your API Gateway endpoint in the header. Enabling AWS WAF on this API Gateway endpoint will use this IP address to apply allow or blocking rules based on a set of AWS WAF rules. We also call this a web access control list (web ACL). This approach provides you with additional security when you allow selected IP addresses, as the authorization code can potentially contain errors in the verification logic.

Each AWS WAF rule contains a statement that defines the condition, in addition to an action to take if a request meets the criteria. AWS WAF allows you to write rules on any part of the request, such as HTTP header, IP address, URI, and body. When a request meets the criteria of the condition, that’s a match. You can use AWS WAF rules to:

• Allow all requests except the requests that you specify: Use when you have known requestors you want to block.
• Block all requests except the requests that you specify: Set web ACL’s default action to “block” and then add rules to explicitly allow IPs you want. Use this AWS WAF rule configuration when you want Amazon API Gateway to serve requests from specific IP addresses and block all other requests.
• Count requests that match properties you specify: You may want to allow or block requests based on new properties in the requests. In that case, you can configure AWS WAF to count the requests that match those properties without allowing or blocking those requests. This lets you test that you didn’t accidentally configure AWS WAF to block all the traffic to your API Gateway endpoint. When you’re confident that you specified the correct properties, you can change the behavior to allow or block requests.

AWS Transfer Family supports Secure File Transfer Protocol (SFTP), File Transfer Protocol over SSL (FTPS), or FTP access to Amazon S3 or Amazon EFS. The solution proposed in this blog can be used when integrating your own identity provider using API Gateway and is available for all three protocols. If you are using endpoints only enabled for SFTP and using the service to store and manage your users’ identities (SSH keys), this solution cannot be used. The following diagram shows how this solution works.

The steps are as follows:

1. A file transfer application or user attempts to log in with either a user name and password, or a user name and private SSH key (stored local to their disk).
2. The AWS Transfer Family service passes these credentials to the API Gateway endpoint you provided when you created the SFTP server. The end user’s Source IP is passed as a ‘SourceIP’ custom header along with the requested credentials to API Gateway for authentication. In turn, AWS WAF intercepts that request.
3. When AWS WAF is enabled on an API, AWS WAF rules take precedence before other access control features, such as resource policies, IAM policies, Lambda authorizers, and Amazon Cognito authorizers. For example, if AWS WAF blocks access from a CIDR block that a resource policy allows, AWS WAF takes precedence and the resource policy isn’t evaluated. AWS WAF inspects the request containing the source IP address and user credentials. Afterward, it allows or blocks the request based on a set of AWS WAF rate-based, geographic match, or IP match rules you create. AWS WAF protects the API Gateway endpoint by filtering out requests from unknown IP addresses requesting authentication, thus blocking unknown request attempts.
4. If the AWS WAF rule evaluates and allows the incoming request, then it also invokes the API Gateway and Lambda function to authenticate and authorize the end users’ request.
5. Retrieve credentials from your identity provider (AWS Secrets Manager in the preceding example). This includes the user’s password (or hash if that is what you have stored), IAM role, and optionally the scope-down policy, in addition to logical directory mappings. If the user attempted to authenticate using SSH keys, public key will be returned instead of password.
6. Upon successful authentication, the Lambda function constructs an HTTP 200 response with the remaining key-value pairs. If you do not have a valid user login, then it returns an empty HTTP 200 response.

The following are important considerations for this solution:

• AWS WAF rules will not work if you want to block or rate limit based on authentication failures.
• If there is a networking component in front of your endpoint that does not preserve client IP addresses, you cannot apply AWS WAF rules based on end users’ Source IP addresses.

Getting started

To demonstrate the AWS WAF integration, we followed the steps mentioned in the blog post “Enable password authentication for AWS Transfer for SFTP using AWS Secrets Manager.” That blog post describes how to set up AWS Transfer Family integrated with API Gateway and Lambda to authenticate users using Secrets Manager as the custom identity provider.

We set up AWS WAF by first defining the AWS WAF rule to block bad actors, and provide IP filtering for undesirable requests from reaching the API Gateway.

1. Navigate to the AWS WAF and Shield console to create a new IP set. An IP set is a collection of IP addresses and IP address ranges that you want to use together in a rule statement. From the left navigation menu, select IP sets. Provide the IP addresses you wish to block for the AWS WAF rule. Click the Create IP set button once you’ve defined the IP addresses for your IP set.

2. Next, create a web access control list (ACL) by selecting the Web ACLs from the left navigation menu. Select the Create Web ACL option and describe the web ACL details.

3. In the next step, select Add rules and choose Add my own rules and rule groups. Configure the rule for the IP set option shown in the following screenshot.

Select the IP set created in step 1. For the IP address to use as the originating address option, choose the IP address in header radio button, and specify “SourceIP” for the Header field name. For Action, choose to block requests originating from IP addresses configured in the IP set. In this setup, the other radio button option Source IP address points to the IP address for the AWS Transfer Family service. In this case, this option does not point to the IP address of the end user connecting with the AWS Transfer Family service. Hence, we enable the radio button option IP address in header. The end user’s IP address then passes on the “SourceIP” custom forwarded header.

Keep default selections for the remaining options.

4. In this step, the rule and its associated action configured in the previous step is shown. The Default web ACL action for requests that don’t match any rules is set to Allow. The rule for the web ACL blocks traffic originating from the IP set defined in step 1.

5. Then, select the default rule priority. For this demonstration, since our web ACL has only one rule, we accept the default rule prioritization. If you have multiple rules in your web ACL, you will want to prioritize the rule evaluation order according to your needs.

6. Next, configure Amazon CloudWatch metrics for AWS WAF.

7. Review the web ACL configurations you’ve made before you create your web ACL.

8. Connect the web ACL you created to the API Gateway API that you use to authorize your users access to your server. Save the changes, and deploy the API in the “prod” stage once the changes are saved. You can also enable AWS WAF logging as an audit log, providing additional insight.

Test your AWS WAF rule

Now, test the AWS WAF rule you just created.

1. Verify the public facing IP address of the computer used for testing is the same IP address that is blocked on the web ACL rule and IP set. Navigate to your local computer and open the terminal session.
2. Connect to your server using the following command – provide the password when prompted for the user as configured in the Secrets Manager key-value pair:

• ‘user1’ is the user name configured for the server.
• ‘s-d5774523a60442e9a’ is the server ID for the server.

Even though you enter the correct password, you will see the permission denied error message. This occurs because the Source IP address matches with the blocked IP address in the IP set.

Whether or not the credentials match, Transfer Family will deny user access because the AWS WAF rule is configured to block the client’s Source IP address.

3. From looking at the AWS WAF and Shield console, we confirm the AWS WAF web ACL rule blocks the Source IP address.

Cleaning up

When you are done with the resources you created and deployed, don’t forget to clean up and check for any permissions that are no longer required. These include the AWS Transfer Family server, AWS WAF, API Gateway, and Secrets Manager.

Summary

In this blog, we discussed leveraging AWS Transfer Family and AWS WAF integration to improve security and control Amazon API Gateway costs when using an identity management system to authenticate and authorize users. This can help customers who are concerned with cost implications that may occur when bots or bounty hunters send multiple requests to the server, which in turn invokes requests to the AWS API Gateway and AWS Lambda functions.

We also demonstrated how to define an AWS WAF rule to block a set of IP addresses when authorizing access over SFTP. These same controls can also be applied when authorizing access over FTPS and FTP. Standard rates apply when you use AWS WAF. For more information, visit the AWS WAF pricing page.

Thank you for reading about how AWS Transfer Family and AWS WAF integration can be used to improve security and control Amazon API Gateway costs. Please leave a comment in the comments section if you have any questions. To learn more about AWS Transfer Family, check out the following links:

• Enable password authentication for AWS Transfer for SFTP using AWS Secrets Manager
• Using Okta as an identity provider with AWS Transfer for SFTP
• AWS Transfer Family: Working with custom identity providers