AWS Storage Blog

Enhance data access control with AWS Transfer Family and Amazon S3 Access Points

Sharing data is at the core of collaboration efforts sparking innovation. Many industries rely on secure data access solutions that scale with their business. From pharmaceutical companies exchanging research data with partners in Amazon S3, to financial customers migrating and modernizing transaction data with traditional file transfer protocols, there is a persistent ask for fine-grained access control (FGAC) in managed file transfer solutions. Organizations like these need mechanisms to simplify user data access patterns across complex datasets shared with internal departments and external partners.

In this post, I discuss how you can use AWS Transfer Family logical directories and Amazon S3 Access Points to enhance FGAC in your shared S3 datasets.

Overview of AWS Transfer Family access controls

AWS Transfer Family continues to evolve as a managed file transfer (MFT) solution with a variety of ways to configure access. Your options for implementing secure access include layering AWS native controls such as AWS Identity and Access Management (IAM) permissions, security groups, and S3 bucket policies. You can also take user authentication into your own hands by either integrating your existing authentication systems or choosing the service-managed approach. AWS Transfer Family also provides logical directories as a built-in feature that maps multiple S3 buckets and files to a virtual directory. Logical directories simplify complex folder structures for data distribution without replicating files across multiple users. With this feature, you define and lock a specific user’s visibility into the data file-structure, have chroot-capabilities, and hide S3 bucket names and paths from users.

Compatibility with Amazon S3 Access Point aliases

S3 Access Points, a feature of Amazon S3, simplifies data access for any AWS service or customer application that is storing data in S3. With each S3 Access Point, you can create a unique access control policy to easily define authorization patterns on your shared S3 datasets.

The recently launched S3 Access Points aliases allow AWS Transfer Family to absorb S3 Access Points benefits. Use these aliases interchangeably with S3 bucket names in AWS Transfer Family workflows to provide compatibility with S3 Access Points and subsequent functionality at no additional cost. Customers with shared datasets from data lakes, media archives, or user-generated content can now use these aliases to create individualized access points with tailored names and permissions across hundreds of applications.

Fine-grain access control (FGAC) with S3 Access Points and logical directories

Rather than manage permissions through a single bucket policy spanning hundreds of use cases, abstract and delegate these permissions to discrete S3 Access Points policies. This simplifies administering thousands of roles and policies with respective character limits, adds flexibility to changes in existing file-transfer folder structures, and streamlines common asymmetrical access patterns such as read/write vs. read-only. You can design and configure granular policies to meet the scale of your business across users, applications, teams, departments, organizations, and partners.

When you combine AWS Transfer Family logical directories and Amazon S3 Access Points, you simplify and scale FGAC patterns used to manage both specific users and user-group permissions. In this blog, I cover an example defining an asymmetrical access pattern using S3 Access Points and logical directories as a mechanism to minimize the management of IAM roles and S3 bucket policies.

Prerequisites

To get started, you’ll need:

Permission Definitions

I use a multi-folder dataset that is shared with end users stemming from internal teams and external organizations. The users have different expected access patterns on the same S3 dataset. Here, authorized actions are dependent on a user’s folder, role, and organization. My shared S3 dataset has many types of end-users, and I narrow it down to the following three:

  • Admin: internal user with full permissions and viewing access into the bucket contents
  • Alice: internal user with full read/write permissions, but only to specified user and shared folders
  • Ben: external user with read/write permissions into a user folder and read-only permissions on shared folder

As I use logical directories and S3 Access Points for FGAC, I define the access patterns, S3 structure, and expected user exposure.

My access pattern driven by S3 policies is shown as followed:

Access-pattern-driven-by-S3-policies

My S3 structure configuration is shown as follows:

S3-structure-configuration-is-shown

My expected user exposure driven by logical directories covers these three access patterns:

Expected-user-exposure-driven-by-Logical-Directories

Note: The preceding asymmetrical access pattern can be achieved through a variety of the previously mentioned access controls. This example intends to show the basic functionality of using S3 Access Points with logical directories to minimize the number of roles and policies needed to manage. This becomes increasingly apparent with more complex user-patterns.

Walkthrough

To achieve FGAC via S3 Access Points and AWS Transfer Family logical directories I complete the following steps and walk you through each in more detail.

  1. Create S3 Access Points and retrieve the respective alias.
  2. Define your S3 policies.
  3. Define logical directory mappings with S3 Access Points alias.
  4. Test expected user access.

Create S3 Access Point and retrieve the alias

You can create S3 Access Points and retrieve the alias using the S3 console, REST API, or AWS CLI as shown. I create an access point named `read-only-ap` for bucket `bucket-name` in account 123456789012.

aws s3control create-access-point --bucket bucket-name --name read-only-ap --
account-id 123456789012
{
"AccessPointArn":"arn:aws:s3:region:123456789012:
accesspoint/read-only-ap","Alias": "read-only-ap-METADATA-s3alias"
}

Define your S3 policies

At this step, you can get strategic in defining authorization patterns that minimize policy overhead. Rather than creating a per-user or per-role policy, my goal is to narrow the scope of the `shared` folder to read-only access for external users like Ben. Therefore, my desired permissions state is simplified with a shorter deny-list than allow-list, and I remove the IAM management headache by simplifying permissions to two types of authorized bucket actions:

  • Policy 1: A general permissive S3 bucket policy allowing read/write access to our bucket
  • Policy 2: A more restricted S3 Access Points policy for read-only access to our bucket

I assume my users Admin, Alice, and Ben each have an IAM role including an IAM policy with full access to bucket-name, and a trusted relationship between AWS Transfer Family is defined. Regardless of how over permissive a user’s IAM role may be, if routed via this sample S3 Access Points Policy (Policy 2), the user is denied on putting and deleting objects:

{ 
"Version": "2012-10-17", 
"Statement" : [
{
"Effect": "Deny",
"Principal" : { 
"*" 
},
"Action": ["s3:PutObject","s3:DeleteObject"],
"Resource":"arn:aws:s3:region:123456789012:accesspoint/read-only-ap/object/*"  
}
]
}

Granted, if there isn’t a conflict with a mapped user’s IAM role, I can design these S3 bucket policies and S3 Access Points policies to scale permissions across different authorization patterns.

Define logical directory mappings using S3 Access Points alias

Now that I used bucket policies to define what my users can do, I use logical directories to specify what users are authorized to see upon file-transfer client entry. For contrast in Admin, I create a user via the CLI with the S3 bucket path exposed as a PATH-home directory type with no chroot-capability:

Admin:

aws transfer create-user --server-id server-id --user-name admin --ssh-public-
key-body key--role role --home-directory-type PATH --home-directory /bucket-name

Using logical directories, I create users Alice and Ben to offer chroot-capabilities and granular control in folder and file visibility. For Alice, I use bucket-name as part of the logical directory target definition to make use of our more general permissive bucket policy (Policy 1).

Alice:

aws transfer create-user --server-id server-id --user-name alice --ssh-public-
key-body key --role role --home-directory-type LOGICAL --home-directory-mapping
‘[{"Entry": "/alice", "Target": "/bucket-name/internal/alice"},
{"Entry": "/shared", "Target": "/bucket-name/shared"}]’

For Ben, I replace bucket-name with the retrieved alias in HomeDirectoryMappings so that authorized actions on the shared folder are limited by the S3 Access Points policy (Policy 2).

Ben:

aws transfer create-user --server-id server-id --user-name ben --ssh-public-
key-body key --role role --home-directory-type LOGICAL --home-directory-mapping
‘[{"Entry": "/ben", "Target": "/bucket-name/external/ben"},
{"Entry": "/shared", "Target": "/read-only-ap-METADATA-s3alias/shared"}]’

Test expected user access

Using your file-transfer client of choice, you can test the following read/write commands to meet expectations. I tested each authenticated user’s read/write output using OpenSSH as a file-transfer client to test authorized actions on an SFTP-enabled server:

Admin

sftp> pwd
Remote working directory: /fgac-tf-ap
sftp> ls
external  internal  shared    // Admin has full view into folder structure
sftp> put image.jpg 
Uploading image.jpg to /fgac-tf-ap/image.jpg
image.jpg     100% 6828   160.6KB/s   00:00    
sftp> ls
external   image.jpg  internal   shared     // Admin can put file regardless of folder
sftp> rm image.jpg
Removing /fgac-tf-ap/image.jpg
sftp> ls
external  internal  shared    // Admin can remove file
sftp> cd shared
sftp> ls
image.png  
sftp> rm image.png
Removing /fgac-tf-ap/shared/image.png //Admin can remove existing files in shared folder
sftp> ls
sftp> cd ..
sftp> ls
external  internal  shared    
sftp> cd external
sftp> ls
ben  
sftp> cd ben
sftp> ls
folder1   
sftp> put image.jpg 
Uploading image.jpg to /fgac-tf-ap/external/ben/image.jpg
image.jpg     100% 6828   175.9KB/s   00:00    
sftp> ls
folder1    image.jpg  // Admin can put file in a user folder
sftp> rm image.jpg
Removing /fgac-tf-ap/external/ben/image.jpg
sftp> cd ..
sftp> ls
ben  // Admin can remove file in a user folder
sftp> cd ..
sftp> ls
external  internal  shared    
sftp> cd ..
sftp> ls
… //

Note: Some clients may allow user to traverse up the folder even if user has scoped down policy. Learn more about this, chroot and logical directories in this blog

Alice

sftp> pwd
Remote working directory: /
sftp> ls
alice   shared  // Alice can get user and shared folder
sftp> cd alice
sftp> ls
folder1  folder2 
sftp> put image.jpg
Uploading image.jpg to /alice/image.jpg
image.jpg  100% 6828   156.9KB/s   00:00 // Alice can put file in Alice folder    
sftp> ls
folder1     folder2     image.jpg  
sftp> rm image.jpg
Removing /alice/image.jpg
sftp> ls
folder1  folder2  // Alice can remove file in Alice folder    
sftp> cd ..
sftp> ls
alice   shared 
sftp> cd shared
sftp> ls 
sftp> put image.jpg
Uploading image.jpg to /shared/image.jpg
image.jpg  100% 6828   176.6KB/s   00:00 // Alice can put files in shared folder    
sftp> ls
image.jpg

Note: Some clients may allow user to traverse up the folder even if user has scoped down policy. Learn more about this, chroot and logical directories in this blog

Ben

sftp> pwd
Remote working directory: /
sftp> ls
ben    shared // Ben can get user and shared folder
sftp> cd ben
sftp> ls
folder1
sftp> cd folder1
sftp> ls
sftp>
sftp> put image.jpg
Uploading image.jpg to /ben/folder/image.jpg
Image.jpg 100% 15KB 1.9MB/s 00:00
sftp> ls
image.jpg // Ben can put file in user folder
sftp> rm image.jpg
Removing /ben/folder1/image.jpg
Sftp> ls
Sftp> //Ben can remove file in user folder
sftp> cd ../..
sftp> ls
ben    shared
sftp> cd shared
sftp> ls
image.jpg 
sftp> put ben.jpg
Uploading ben.jpg to /shared/ben.jpg
100% 6828      177.3KB/s 00:00
Couldn’t close file: Permission denied // Ben cannot put file in shared folder
sftp> rm image.jpg
Removing /shared/image.jpg
Couldn’t delete file: Permission denied // Ben cannot delete file in shared folder

As expected, the Admin user has full read/write access and full visibility into the bucket. Alice has read/write permissions into both the shared and user folders. Ben is most limited with read-only views into the shared folder and read/write access into user folder.

If your output doesn’t provide the anticipated authorization, you can troubleshoot by walking through a user’s IAM role, the S3 bucket policies, S3 Access Points policies, logical directory mappings, and refresh the client connection.

Cleaning up

To avoid incurring future charges, delete the sample resources set up as part of this post including your S3 bucket, your IAM roles, and your AWS Transfer Family server or any respective users no longer needed.

For cost details, please refer to Amazon S3 pricing and AWS Transfer Family pricing pages.

Conclusion

In this post, I walked through integrating Amazon S3 Access Points with AWS Transfer Family logical directories to scale your permission management of shared datasets. Now, imagine scaling a similar pattern to thousands of users with niche access needs in your MFT. You no longer have to manage complex IAM roles or error-prone configurations in a single bucket policy. You can now build more complex variations with solutions that span across accounts and S3 buckets, delegate access control to individual access points, or have multiple complex entry-target pairings.

Leave a comment if you found this helpful or have any questions, and happy building!