Amazon S3 object lock (retention period) in Python with IAM policy example

S3 object lock is a feature to prevent permanent deletion on S3 objects, by accident and deliberate. This article focus on retention period in governance mode. For legal hold, please refer to another article.

This article will cover

• Create S3 bucket in AWS console with object lock enabled
• IAM policy to prevent a user from bypassing governance retention
• Examples in Python – write an object to S3 and set the retention period
• Test – try to shorten the retention period and delete the object with AWS CLI

Create a new bucket in AWS S3 console

When creating an S3 bucket, tick the following options

• Versioning – “Keep all versions of an object in the same bucket.”
• Object lock – “Permanently allow objects in this bucket to be locked.”

Otherwise, it’s not possible to create an object with object lock (retention or legal hold). The Python script below will raise exception in put_object_retention()

IAM policies

This IAM policy avoids a user to bypass governance retention. AWS root user can disable governance retention and permanently delete objects.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Deny",
            "Action": "s3:BypassGovernanceRetention",
            "Resource": "*"
        }
    ]
}

Allow bucket listing

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "s3:ListAllMyBuckets",
            "Resource": "*"
        }
    ]
}

Allow access to buckets

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": [
                "arn:aws:s3:::obj-lock-test-bucket",
                "arn:aws:s3:::obj-lock-test-bucket/*"
            ]
        }
    ]
}

Write an object and put retention period in Python

#!/usr/bin/python3

import logging
from datetime import datetime
from datetime import timedelta

import boto3
from botocore.exceptions import ClientError


def upload_file(file_name, bucket, object_name=None):
    s3_client = boto3.client('s3')
    try:
        s3_client.upload_file(
                Filename=file_name,
                Bucket=bucket,
                Key=object_name,
        )

        s3_client.put_object_retention(
                Bucket=bucket,
                Key=object_name,
                Retention={
                    'Mode':'GOVERNANCE',
                    'RetainUntilDate':datetime.now() + timedelta(days=1)
                },
        )
    except ClientError as e:
        logging.error(e)
        return False
    return True

upload_file("local-filename", "obj-lock-test-bucket", "object-key")

Test it

We can get version ID with AWS CLI

aws s3api list-object-versions \
    --bucket obj-lock-test-bucket

Try to delete it, and try to bypass governance retention – it should fail.

aws s3api delete-object \
    --bucket obj-lock-test-bucket \
    --key object-key \
    --version-id JDRmmQxdmwC3NTfwvPsXOQ0dH0H.0hTS

aws s3api delete-object \
    --bucket obj-lock-test-bucket \
    --key object-key \
    --version-id JDRmmQxdmwC3NTfwvPsXOQ0dH0H.0hTS \
    --bypass-governance-retention

Try to change (extend or reduce) retention period. Also try tor bypass governance retention.

• Attempt to bypass should always fail
• Extending retention period should success
• Reducing retention period should fail

aws s3api put-object-retention \
    --bucket obj-lock-test-bucket \
    --key object-key \
    --retention Mode=GOVERNANCE,RetainUntilDate=2020-01-19

aws s3api put-object-retention \
    --bucket obj-lock-test-bucket \
    --key object-key \
    --retention Mode=GOVERNANCE,RetainUntilDate=2020-01-19 \
    --bypass-governance-retention