troubleshooting-s3-files

Troubleshooting S3 Files

Overview

Diagnoses and resolves Amazon S3 Files issues: mount failures, IAM permissions, synchronization, conflict resolution, and performance.

For authoritative guidance, see S3 Files Troubleshooting.

Common Tasks

0. Verify Dependencies

• You MUST verify aws CLI is available with s3files subcommand support
• You MUST confirm valid AWS credentials
• You MUST ONLY check for tool existence and version — MUST NOT execute destructive or mutating commands during verification
• You MUST inform the user if any required tools are missing
• You MUST respect the user's decision to abort if tools are unavailable
• You SHOULD explain steps before executing and wait for user confirmation on write commands

1. Classify the Issue

2. Category A — Client Installation

mount.s3files: command not found means amazon-efs-utils is missing or < v3.0.0.

sudo yum -y install amazon-efs-utils  # Amazon Linux

3. Category B — Network/Security Group

Connection timeout is the #1 mount failure — almost always security groups.

Verify mount target exists in the instance's AZ:

aws s3files list-mount-targets --file-system-id fs-ID --region REGION

Cross-AZ mounting works but adds latency.

Verify security groups — most common fix:

• Mount target SG MUST have inbound TCP 2049 from compute SG
• Compute SG MUST have outbound TCP 2049 to mount target SG
• Fix: aws ec2 authorize-security-group-ingress --group-id sg-MT --protocol tcp --port 2049 --source-group sg-COMPUTE

Test connectivity:

Note: These SG troubleshooting steps also apply to EFS — use aws efs describe-mount-targets instead.

Mount hangs in isolated VPC: If the VPC has no internet access, S3 Files requires a CloudWatch Logs VPC endpoint (com.amazonaws.REGION.logs) for mount to complete.

4. Category C — IAM Permissions

File system stuck in "creating" status: S3 Files does NOT validate IAM role permissions at creation time. Wrong trust policy or missing permissions → stuck in creating with access denied in statusMessage.

Check status:

Check statusMessage. If access denied, fix the IAM role and delete/recreate.

Mount access denied: Compute role needs s3files:ClientMount. For dev/test only, AmazonS3FilesClientFullAccess is acceptable — avoid in production.

Write permission denied: Compute role needs s3files:ClientWrite

Root access denied: Compute role needs s3files:ClientRootAccess. ⚠️ Bypasses POSIX permissions — prefer access points with scoped POSIX users.

Check file system policy:

5. Category D — Synchronization

Files not appearing in S3: Writes sync within ~60 seconds. Check status:

Common ExportError values:

Monitor: PendingExports CloudWatch metric. Growing = exceeds 800 files/sec rate.

6. Category E — Conflict Resolution

Files in .s3files-lost+found-{fs-id} = sync conflict (modified via FS and S3 simultaneously). S3 wins; FS version moved to lost+found.

7. Category F — Performance

First access latency: Normal — first directory access imports metadata.

Intelligent read routing not working: Compute role needs s3:GetObject on the bucket.

Slow writes: If PendingExports growing, distribute across multiple file systems.

8. Category G — Encryption/KMS

NFS server error with encrypted FS = KMS issue. Verify key is enabled and role has KMS permissions.

9. Category H — VPC DNS

DNS resolution failure = VPC DNS settings disabled.

Both MUST be true. If not:

Troubleshooting

AWS CLI endpoint URL cannot be resolved

CLI is too old for S3 Files. Run aws --version — if v1.x, upgrade to AWS CLI v2: Installing the AWS CLI.

ECS task fails with DNS resolution error

Used efsVolumeConfiguration instead of s3filesVolumeConfiguration. Fix: use fileSystemArn in S3 Files-specific volume config.

S3 Files vs other products confusion

S3 Files is NOT Mountpoint for S3, S3 File Gateway, or File Cache. Uses aws s3files CLI, s3files: IAM actions, mount -t s3files.

Enable Debug Logs

Set logging_level = DEBUG in /etc/amazon/efs/s3files-utils.conf. Logs at /var/log/amazon/efs/mount.log.

Collect Logs for AWS Support

Security Considerations

• When diagnosing IAM issues, verify least-privilege — avoid FullAccess as a shortcut
• Without a file system policy, any VPC client can mount
• Restrict /var/log/amazon/efs/ access — logs contain S3 key names

Additional Resources

• S3 Files Troubleshooting
• S3 Files Best Practices
• S3 Files Quotas