# Guide to Access S3 Storage as Local Filesystem S3 object storage offers scalable and cost-effective storage solutions but working with it directly can be challenging when your applications expect traditional filesystem access. This guide explores two powerful tools - `rclone` and `s3fs` - that bridge this gap by mounting S3 buckets as local filesystems. ## Prerequisites Before getting started, ensure you have installed the required third-party software: * `rclone`: A versatile command-line tool for managing files on cloud storage * Installation: [https://rclone.org/install/](https://rclone.org/install/) * Supports numerous storage providers beyond S3 * `s3fs`: A FUSE-based filesystem specifically designed for S3 * Installation: [https://github.com/s3fs-fuse/s3fs-fuse](https://github.com/s3fs-fuse/s3fs-fuse) * Available in most package managers: `apt install s3fs` or `yum install s3fs-fuse` ## Configuring Your S3 Mount Tools ### Step 1: Setting Up Configuration Files Each tool requires specific configuration to connect to your S3 bucket: #### rclone Configuration Create a configuration file at `/etc/rclone.conf`: ```ini [s3-mount] type = s3 provider = AWS env_auth = false access_key_id = YOUR_ACCESS_KEY secret_access_key = YOUR_SECRET_KEY endpoint = YOUR_ENDPOINT_URL acl = private ``` See [rclone.config.example](rclone/rclone.config.example) for a complete template. #### s3fs Configuration Create a credentials file at `/etc/passwd-s3fs` with the following format: ```python ACCESS_KEY_ID:SECRET_ACCESS_KEY ``` Set appropriate permissions: ```bash chmod 600 /etc/passwd-s3fs ``` See [s3fs-passwd.example](s3fs/s3fs-passwd.example) for reference. ### Step 2: Creating Mount Scripts Create shell scripts to manage the mounting process with proper parameters: #### rclone Mount Script Create `/usr/local/bin/rclone-mount.sh`: ```bash #!/bin/bash # Configuration variables bucket="your-bucket-name" url="https://your-endpoint.com" mount_point="/mnt/s3-bucket" config_file="/etc/rclone.conf" log_file="/var/log/rclone-mount.log" log_level="DEBUG" provider="s3" # Options: vstorage, s3, etc. # Create mount point if it doesn't exist mkdir -p "${mount_point}" # Mount the bucket rclone mount \ --config "${config_file}" \ --log-file "${log_file}" \ --log-level "${log_level}" \ --allow-other \ --file-perms 0644 \ --dir-perms 0755 \ --vfs-cache-mode full \ --vfs-cache-max-size 1G \ --vfs-read-chunk-size 10M \ --daemon \ "${provider}:${bucket}" "${mount_point}" exit 0 ``` Make the script executable: ```bash chmod +x /usr/local/bin/rclone-mount.sh ``` #### s3fs Mount Script Create `/usr/local/bin/s3fs-mount.sh`: ```bash #!/bin/bash # Configuration variables bucket="your-bucket-name" url="https://your-endpoint.com" mount_point="/mnt/s3-bucket" passwd_file="/etc/passwd-s3fs" log_file="/var/log/s3fs-mount.log" log_level="debug" region="HCM03" # Your specific region # Create mount point if it doesn't exist mkdir -p "${mount_point}" # Mount the bucket s3fs "${bucket}" "${mount_point}" \ -o passwd_file="${passwd_file}" \ -o url="${url}" \ -o use_path_request_style \ -o allow_other \ -o umask=0022 \ -o dbglevel="${log_level}" \ -o curldbg \ -o endpoint="${region}" \ > "${log_file}" 2>&1 exit 0 ``` Make the script executable: ```bash chmod +x /usr/local/bin/s3fs-mount.sh ``` ### Step 3: Creating Systemd Service Units To ensure your S3 bucket mounts automatically at boot and is properly managed by systemd: #### rclone Systemd Service Create `/lib/systemd/system/rclone-mount.service`: ```ini [Unit] Description=Mount S3 Bucket using rclone After=network-online.target Wants=network-online.target [Service] Type=forking ExecStart=/usr/local/bin/rclone-mount.sh Restart=on-failure RestartSec=10 [Install] WantedBy=multi-user.target ``` #### s3fs Systemd Service Create `/lib/systemd/system/s3fs-mount.service`: ```ini [Unit] Description=Mount S3 Bucket using s3fs After=network-online.target Wants=network-online.target [Service] Type=oneshot ExecStart=/usr/local/bin/s3fs-mount.sh RemainAfterExit=yes ExecStop=/bin/fusermount -u /mnt/s3-bucket [Install] WantedBy=multi-user.target ``` ### Step 4: Enable and Start the Service Choose which tool you prefer (rclone or s3fs) and enable its service: ```bash # For rclone sudo systemctl daemon-reload sudo systemctl enable rclone-mount.service --now sudo systemctl status rclone-mount.service # For s3fs sudo systemctl daemon-reload sudo systemctl enable s3fs-mount.service --now sudo systemctl status s3fs-mount.service ``` ## Performance Considerations * **rclone**: * Offers better performance for large files * More feature-rich with built-in caching * Uses more memory but provides better throughput * Excellent for backup/sync operations * **s3fs**: * Simpler, lighter resource footprint * Better for direct file access patterns * More POSIX-compliant but slower for metadata operations * Good for applications that need basic file access ## Troubleshooting Common Issues ### Mount Failure If your mount fails to initialize: 1. **Check credentials**: Verify your access keys are correct in the configuration files ```bash cat /var/log/rclone-mount.log | grep "auth" # or cat /var/log/s3fs-mount.log | grep "auth" ``` 2. **Test connectivity**: Confirm network access to your S3 endpoint ```bash curl -I https://your-endpoint.com ``` 3. **Permissions**: Ensure your mount scripts are executable ```bash ls -la /usr/local/bin/rclone-mount.sh ls -la /usr/local/bin/s3fs-mount.sh ``` 4. **Bucket existence**: Verify the bucket name is spelled correctly and exists ```bash # For AWS S3 aws s3 ls s3://your-bucket-name # For other S3 providers, use their CLI tools ``` ### Performance Issues If you experience slow access: 1. **Increase cache size**: For rclone, modify the `--vfs-cache-max-size` parameter 2. **Adjust chunk size**: Modify `--vfs-read-chunk-size` for your workload 3. **Check network latency**: High latency to your S3 endpoint will impact performance 4. **Consider local caching**: For frequently accessed files ## References * [rclone Documentation](https://rclone.org/docs/) * [rclone S3 Configuration](https://rclone.org/s3/) * [s3fs Documentation](https://github.com/s3fs-fuse/s3fs-fuse) * [FUSE Filesystem Overview](https://www.kernel.org/doc/html/latest/filesystems/fuse.html) * [AWS S3 Documentation](https://docs.aws.amazon.com/s3/)