Processing S3 Uploads With Step Functions

Step Functions Workflow: Route S3 Uploads to Lambda or Fargate

Overview

This skill deploys an event-driven workflow using AWS CLI. When a file is uploaded to an S3 bucket, EventBridge triggers a Step Functions state machine. The state machine checks the file size and routes processing to either a Lambda function (files ≤ 6 MB) or a Fargate task (files > 6 MB).

The architecture includes:

• An S3 bucket with EventBridge notifications enabled
• An EventBridge rule that triggers Step Functions on S3 object creation
• A Step Functions state machine with a Choice state for routing
• A Lambda function for processing small files
• An ECS Fargate task for processing large files
• A VPC with two subnets, internet gateway, and security group
• An ECR repository for the Fargate container image
• Scoped IAM roles for Lambda, Step Functions, and ECS tasks

Use this skill when:

• You need to process S3 uploads with different compute based on file size
• You want a serverless workflow that can handle both small and large files
• You need Step Functions orchestration with Lambda and Fargate

Do not use this skill when:

• All files are small enough for Lambda (use S3 → Lambda directly)
• You need real-time streaming (use Kinesis)
• You don't need file-size-based routing

Prerequisites

1. AWS CLI v2 — Installed and configured. Verify with aws sts get-caller-identity.
2. Python 3.12 — For the Lambda function runtime.
3. Docker — For building and pushing the Fargate container image.

Parameters

• bucket_name (required): Name for the S3 bucket (globally unique, lowercase, 3-63 characters)
• region (required): AWS region for all resources
• ecr_repo_name (required): Name for the ECR repository
• state_machine_name (required): Name for the Step Functions state machine
• kms_key_arn (optional): ARN of a KMS key for CloudWatch Logs encryption. If not provided, create one with aws kms create-key --description "Key for CloudWatch Logs encryption" --region {region}

Constraints for parameter acquisition:

• You MUST ask for all required parameters upfront in a single prompt
• You MUST support multiple input methods (direct input, file path, URL)
• You MUST confirm successful acquisition of all parameters before proceeding
• You MUST validate that bucket_name follows S3 naming rules

Procedures

Step 0: Verify Dependencies

Constraints:

• You MUST verify the following tools are available: aws-cli, python3 (3.12+), docker
• You MUST inform the user about any missing tools with a clear message
• You MUST ask if the user wants to proceed despite missing tools
• You MUST respect the customer's decision to abort at any point
• You MUST explain to the customer what step is being executed, why, and which tool is being called

Step 1: Retrieve AWS Account ID

Constraints:

• You MUST retrieve the account ID with: aws sts get-caller-identity --query 'Account' --output text
• You MUST store the result as {account_id} for use in all subsequent steps
• You MUST abort if credentials are not configured

Step 2: Get the Default VPC and Networking

Constraints:

• You MUST retrieve the default VPC ID with: aws ec2 describe-vpcs --filters Name=isDefault,Values=true --query 'Vpcs[0].VpcId' --output text --region {region}
• If no default VPC exists, inform the user they must create one with aws ec2 create-default-vpc --region {region} or provide a VPC ID manually
• You MUST retrieve two subnet IDs from the default VPC: aws ec2 describe-subnets --filters Name=vpc-id,Values={vpc_id} --query 'Subnets[0:2].SubnetId' --output text --region {region}
• You MUST create a security group in the default VPC: aws ec2 create-security-group --group-name fargate-sg --description "Security group for Fargate tasks" --vpc-id {vpc_id} --region {region}
• You MUST configure security group egress rules to allow only HTTPS and DNS outbound. First revoke the default allow-all egress rule: aws ec2 revoke-security-group-egress --group-id {sg_id} --ip-permissions IpProtocol=-1,IpRanges='[{CidrIp=0.0.0.0/0}]' --region {region} Then add scoped rules: aws ec2 authorize-security-group-egress --group-id {sg_id} --protocol tcp --port 443 --cidr 0.0.0.0/0 --region {region} and aws ec2 authorize-security-group-egress --group-id {sg_id} --protocol udp --port 53 --cidr 0.0.0.0/0 --region {region}
• You MUST recommend VPC endpoints for S3 and CloudWatch Logs for production workloads to avoid internet-routed traffic and eliminate the need for broad egress rules
• You MUST capture {vpc_id}, {subnet1_id}, {subnet2_id}, and {sg_id} for use in later steps

Step 3: Create the ECR Repository

Constraints:

• You MUST create the repository with: aws ecr create-repository --repository-name {ecr_repo_name} --region {region}
• You MUST capture the repositoryUri from the response

Step 4: Build and Push the Container Image

Constraints:

• You MUST verify Docker is installed by running docker --version. If Docker is not installed, instruct the user to install it from https://docs.docker.com/get-docker/ and abort until it is available

• You MUST authenticate Docker with ECR: aws ecr get-login-password --region {region} | docker login --username AWS --password-stdin {account_id}.dkr.ecr.{region}.amazonaws.com

• The Dockerfile and processor code are in scripts/Dockerfile and scripts/fargate_processor.py

• You MUST build and push the image from the scripts directory:

  cd scripts
  docker build --platform linux/amd64 -t {ecr_repo_name} .
  docker tag {ecr_repo_name}:latest {account_id}.dkr.ecr.{region}.amazonaws.com/{ecr_repo_name}:latest
  docker push {account_id}.dkr.ecr.{region}.amazonaws.com/{ecr_repo_name}:latest
  cd ..

Step 5: Create IAM Roles

Follow the detailed instructions in references/iam-roles.md to create all IAM roles (Lambda, ECS task execution, ECS task, Step Functions, and EventBridge roles).

• You MUST wait at least 10 seconds for IAM role propagation

Step 6: Create the Lambda Function

Constraints:

• The function code is in scripts/lambda_function.py

• You MUST be in the skill root directory before packaging and creating the function

• You MUST package it with: python3 -c "import zipfile,io; z=io.BytesIO(); f=zipfile.ZipFile(z,'w'); f.writestr('lambda_function.py', open('scripts/lambda_function.py').read()); f.close(); open('/tmp/lambda_function.zip','wb').write(z.getvalue())"

• You MUST create the function with:

  aws lambda create-function \
      --function-name sfn-file-processor \
      --runtime python3.12 \
      --handler lambda_function.lambda_handler \
      --role arn:aws:iam::{account_id}:role/sfn-lambda-role \
      --zip-file fileb:///tmp/lambda_function.zip \
      --timeout 60 \
      --architectures x86_64 \
      --region {region}

• You MUST verify the function was created with: aws lambda get-function --function-name sfn-file-processor --region {region}

Step 7: Create the CloudWatch Log Group

Constraints:

• You MUST create the log group for Fargate: aws logs create-log-group --log-group-name /StepFunctionFargateTask --region {region}
• You MUST encrypt the log group with a KMS key: aws logs associate-kms-key --log-group-name /StepFunctionFargateTask --kms-key-arn {kms_key_arn} --region {region}

Step 8: Create the ECS Cluster and Task Definition

Follow the detailed instructions in references/ecs-task-definition.md to create the ECS cluster and register the Fargate task definition.

• You MUST capture the task definition ARN from the response

Step 9: Create the S3 Bucket with EventBridge Notifications

Constraints:

• You MUST create the bucket with: aws s3api create-bucket --bucket {bucket_name} --region {region} --create-bucket-configuration LocationConstraint={region}
• You MUST NOT include --create-bucket-configuration if region is us-east-1
• You MUST enable EventBridge notifications on the bucket: aws s3api put-bucket-notification-configuration --bucket {bucket_name} --notification-configuration '{"EventBridgeConfiguration": {}}' --region {region}
• You MUST enable default encryption on the bucket: aws s3api put-bucket-encryption --bucket {bucket_name} --server-side-encryption-configuration '{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"aws:kms"}}]}' --region {region}