Introduction

DuraCloud Preserve is a project designed to make using AWS S3 as simple as possible for users who only want to care about uploading files, or integrating S3 storage with other applications, and not have to worry about esoteric configuration or infrastructure management. It also supports digital preservation use cases by managing the configuration and features available to S3 to support long term access to and preservation of files.

The goal is to make it easy for users to choose any off the shelf S3 client and interact with S3 gaining more advanced features by default. Advanced features are described in more detail throughout the user and technical documentation but in brief: versioning, inventory, replication, logging etc. is enabled as buckets are created without a user having to do anything in AWS.

Periodically checksum verification is performed to ensure that file integrity is maintained between the primary and replicated (backup) files. This builds upon the already impressive levels of durability that S3 provides by adding a recurring guarantee that files are what they are intended to be.

Additional features include generating manifest (file inventory) and storage reports and user access control via preconstructed groups that are scoped to stacks. When deployed every resource is created “within” a stack. A stack is simply a resource naming prefix and tag applied to all resources managed by the deployed components to exclusively associate them. This makes it possible to have multiple stacks within a single account and makes it so different users can belong to one or more stacks.

Lyrasis provides a hosting service for DuraCloud Preserve, handling the AWS account creation and installation, and which comes with access to a web based ui for S3, using SFTPGo. S3 can then be interacted with using the web ui or via direct AWS access credentials for broader integrations or for usage with tools like the AWS cli.

AWS resources used:

• CloudWatch
• Eventbridge
• IAM
• Lambda
• S3

Context

DuraCloud Preserve is a continuation of the DuraCloud project in a form that is intended to be more sustainable for the long term. It does this by focusing on the core mission of DuraCloud but with a significantly smaller technical footprint, made possible by leveraging AWS S3 features directly in contrast to the more abstracted approach that DuraCloud took in being open to support multiple backend storage providers.

But the goals remain the same: in the digital era, ensuring that critically important documents remain safe and available is a continual challenge. Physical computing hardware that is used to create and store documents can fail or become obsolete very quickly, providing a need for tools to ensure that these documents remain available. DuraCloud Preserve aims to address these concerns:

• How do I upload files to the storage service in a simple and reliable way?
• How do I ensure that the storage service that I am using receives a copy of my local files?
• How do I ensure that files remain intact over time?
• How do I retrieve my content once it is stored?
• How do I recover a file if it has been overwritten or corrupted?
• How do I make my content publicly accessible at a stable URL?
• How am I protected against the storage service becoming obsolete or going away?

Answers to these questions are provided throughout the rest of this documentation.

Features

This is a brief overview of the functionality that is explained more thoroughly in the user guide and technical documentation:

Access controls

Users can be standard or power users by assignment to a stack created IAM group.

• Standard users can list and upload files but cannot download or delete them.
• Restricted users can list and upload files within designated buckets but cannot download or delete them.
• Power users can list, upload, download, and delete files.

Only AWS account administrators can access replicated buckets and objects.

Audit trail

Request logs are generated for each user-created bucket. This is raw AWS provided data that can be processed using tools like DuckDB.

Checksum reports

Checksum reports are generated on a configurable schedule, comparing checksums across source and replication buckets to detect corruption. Files found to be corrupt can be restored from the verified copy. See the checksum verification documentation for more details.

Choice of region

Files can be stored in any AWS region supported by the infrastructure.

CLI available

A command-line interface (dcp) is available for advanced users. It provides access to all core functions and additional maintenance commands for tasks such as checksumming local files, reconciling bucket configuration, and transferring data between buckets.

Hosting and support

If creating an AWS account and deploying resources to it is not possible then Lyrasis provides a hosting and technical support option to handle the infrastructure for you.

Lifecycle transitions

Files are uploaded to the standard storage tier and transition to a selected storage class after a configurable interval, which can be specified for each stack deployment. Old versions of files and aborted multipart uploads are automatically deleted after a configurable period.

Manifest reports

A file manifest is generated for each user-created bucket. The raw inventory data is available in Parquet format and a consolidated, human-readable CSV file is generated listing all files with metadata including S3 URL, size, storage class, and last modified date.

Public access via CDN (Content Delivery Network)

A CloudFront distribution and bucket is created that can be used to make files publicly available. Simply upload files to it and share the public url using a specified domain.

Other buckets can be created as publicly accessible by naming them with a -public suffix. Files uploaded to such buckets will be available using a standard, unauthenticated S3 URL.

Files will be stored in the intelligent storage tier and not transitioned to Glacier; however replication will still occur and the backup copies will be stored in Glacier.

Reconciliation reports

The reconciliation report is used to detect drift in bucket configuration, providing reassurance that buckets are configured correctly and working as expected.

Replication

Files for all buckets are replicated to Glacier Deep Archive. These files are included in the checksum verification process to determine file integrity. We have dedicated documentation for how this works.

Storage reports

An HTML storage report is generated showing usage statistics across all buckets in the stack, including total file counts and storage consumed by bucket and top-level prefix. It also includes the year-to-date total of data transfer out from S3 to the internet (requires Cost Explorer to be enabled in AWS, and an active Stack cost allocation tag).

Versioning

Bucket versioning is enabled. This supports file restore for up to a configurable number of days post update which can be specified for each stack deployment.

Web UI integration with SFTPGo

There is support within the application and deployment tooling for SFTPGo integration, which provides a web based interface for S3. Users can be created that are pre-configured with appropriate access (per the access controls that have been assigned to them) and the SFTPGo user account is kept in sync as buckets are created, or via the dcp cli.

General integrations

Web applications that support use of Amazon S3 for storage

Any application or framework that can be configured to use Amazon S3 for storage can work with DuraCloud Preserve. By simply using a bucket created as part of a DuraCloud Preserve stack files will be stored with the additional benefits outlined in this documentation, including versioning, replication and checksum verification.

Some specific examples:

• Any Rails web applications using ActiveStorage.
• Archivematica Storage Service
• CollectionSpace file storage.
• DSpace Storage Layer.

Lyrasis service integrations

ArchivesSpace

ArchivesSpace itself does not manage digital content and provides no way to upload files. The public urls provided by the Duracloud Preserve CloudFront enabled bucket can be used to host files that are referenced in Digital Objects using the File URI field to make them openly accessible on the internet.

CollectionSpace

Refer to the roadmap for any upcoming work.

DSpace

The Replication Task Suite is a plugin for DSpace that adds preservation capabilities that can be accessed using the DSpace user interface. It creates archival information packages used to backup DSpace items in a self contained way that are periodically transferred to external storage, including Amazon S3. Doing the latter with a DuraCloud Preserve created bucket works equivalently to using S3 for the DSpace Storage Layer (assetstore), and if both are configured this way it enables a dual layer of protection for files (as both the assetstore and archival packages would benefit from versioning, replication and checksum verification etc.).

Other integrations

Archive-It

Create an inventory and a backup of WARC files retrieved from the Internet Archive - Archive-It service.

Checksum Verification

DuraCloud Preserve stores and replicates files using Amazon S3. Checksum verification is the process by which the system confirms that stored files have not been silently corrupted over time. Even in highly durable storage systems, subtle errors (known as “bit rot”) can alter file content without any obvious warning. By regularly comparing checksums across independent copies of each object, the system can detect and remediate corruption before it affects both copies.

How It Works

1. Upload Integrity

AWS S3 provides integrity guarantees at the point of upload. Using built-in integrity checking mechanisms, S3 validates received data and rejects any upload where the computed checksum does not match. A successful upload response from S3 confirms that the stored object matches exactly what was transmitted.

The system’s integrity guarantee begins at this point of successful upload.

The checksum and version of any stored object can be retrieved using the AWS CLI:

aws s3api head-object --bucket ${bucket} --key ${key} --checksum-mode ENABLED

Example response:

{
    "AcceptRanges": "bytes",
    "LastModified": "2026-01-24T00:22:19+00:00",
    "ContentLength": 15310515,
    "ChecksumCRC64NVME": "V+va1ramtYo=",
    "ChecksumType": "FULL_OBJECT",
    "ETag": "\"822f9ffde463633f9a56df6d90b1dbb6\"",
    "VersionId": "HnU.prnfFqU2oJKqjIibty9_cet6zTDH",
    "ContentType": "application/pdf",
    "ServerSideEncryption": "AES256",
    "Metadata": {},
    "StorageClass": "GLACIER_IR",
    "ReplicationStatus": "COMPLETED"
}

Further reading:

• Checking object integrity in Amazon S3
• Checking object integrity for data uploads in Amazon S3

2. Replication

After a successful upload, AWS S3 replication creates a copy of the object in a second independent bucket, typically within 15 minutes. The same upload integrity guarantees apply to replication, ensuring the replicated object is an exact copy of the source.

The checksum and version ID of the replica will match the source object exactly:

{
    "AcceptRanges": "bytes",
    "LastModified": "2026-01-24T00:22:19+00:00",
    "ContentLength": 15310515,
    "ChecksumCRC64NVME": "V+va1ramtYo=",
    "ChecksumType": "FULL_OBJECT",
    "ETag": "\"822f9ffde463633f9a56df6d90b1dbb6\"",
    "VersionId": "HnU.prnfFqU2oJKqjIibty9_cet6zTDH",
    "ContentType": "application/pdf",
    "ServerSideEncryption": "AES256",
    "Metadata": {},
    "StorageClass": "GLACIER",
    "ReplicationStatus": "REPLICA"
}

Note that ChecksumCRC64NVME and VersionId are identical across both objects.

Further reading:

• Meeting compliance requirements with S3 Replication Time Control
• Replicating objects within and across Regions

3. Durability

AWS S3 is designed for 99.999999999% (eleven nines) durability. Given S3’s upload integrity guarantees and its documented durability, uploaded and replicated objects can be considered correct and consistent at the point of replication with a very high degree of confidence.

Further reading: Durability in Amazon S3

4. Ongoing Verification

S3 Batch Operations are used to generate checksum reports across all objects in both the source and replication buckets. These reports are compared on a regular schedule.

If a Mismatch Is Detected

If verification finds that checksums do not match, the following steps identify and repair the corruption.

Step 1 — Check prior reports. A previously generated checksum report may already contain the expected checksum values, making it straightforward to determine which copy — source or replica — is corrupt.

Step 2 — Request object metadata. If no prior report is available, retrieve the stored checksum, value, and version directly from each object’s metadata and compare them:

aws s3api head-object --bucket ${bucket} --key ${key} --checksum-mode ENABLED

Step 3 — Download and verify locally. For a more thorough inspection, download the objects and compute checksums locally using an algorithm included in the object metadata (S3 uses CRC-64/NVME by default but other checksums may be present in addition to or instead of crc64nvme depending on how the object was uploaded):

# Retrieve the stored checksum
aws s3api head-object --bucket ${bucket} --key ${key} --checksum-mode ENABLED

# Download the file
aws s3 cp s3://${bucket}/${key} .

# Compute the checksum locally using the DuraCloud Preserve CLI
dcp checksum --file ${key}

Step 4 — Restore. Once the valid copy is confirmed, re-upload it to the source bucket to repair the corrupted object.

Important

Hosted clients: Lyrasis will handle checksum verification and file restoration on your behalf if errors are found.

Learn more about Lyrasis Hosting

Further reading:

• Compute checksums
• Examples: S3 Batch Operations completion reports
• Efficiently verify Amazon S3 data at scale with compute checksum operation

What Successful Verification Confirms

Successful verification confirms that the source and replica objects are identical to each other. Given S3’s upload integrity guarantees and its documented durability, this means objects are also identical to what was originally uploaded to a very high degree of confidence.

This strategy is considered sufficient for the vast majority of standard use cases. In the unlikely event that corruption is not automatically addressed by the S3 infrastructure, it is highly improbable that both independent copies would be corrupted in exactly the same way — which would be required to produce a false verification result.

For the strongest possible guarantee, independent verification using locally managed checksums is required. See Stricter Compliance Requirements below.

Further reading:

• Data protection in Amazon S3
• Amazon S3 Storage Classes

Checksum Reports

Checksum reports are stored in S3 for the duration of the stack’s retention policy and can be downloaded at any time.

For organizations requiring independent verification or stricter compliance, reports should be downloaded and stored locally or in a system separate from S3.

Stricter Compliance Requirements

For organizations with higher assurance requirements — such as regulated industries or formal digital preservation programs — the approach described above may not be sufficient on its own, as it is ultimately dependent on the claims of a single third-party provider (Amazon AWS). An independent audit mechanism, separate from the primary storage provider, is required for the strictest compliance standards.

Best practice for stricter compliance:

1. Generate checksums locally before uploading. Use a tool such as QuickHash to compute a checksum for each file before it is uploaded to S3.
2. Maintain a local checksum inventory. Keep a record of each filename and its corresponding checksum in a safe location. This inventory can be stored in S3, but must also exist independently.
3. Verify on retrieval. When downloading a file, recompute its checksum locally and compare it against the inventory record.

It is also important to note that DuraCloud Preserve is entirely dependent on the Amazon AWS S3 service, its regional infrastructure, and its policies. Organizations with strict independence or sovereignty requirements should factor this into their preservation planning.

Frameworks and standards for reference:

• NDSA Levels of Digital Preservation — A tiered framework for assessing digital preservation practices
• Digital Preservation Coalition — Audit and Certification — Overview of audit standards and certification options

Getting started

Whoever is responsible for deployment will provide access credentials to users.  If you are intending to connect directly to S3 using a GUI or CLI tool then you should receive an access key and secret, which serve as a username and password for interacting with S3. It is important to treat this as sensitively as you would any username and password.

If you are intending to use the web client then you should receive a username (your email address), password and the url to login. It’s completely fine to use both approaches if you’d like access to both.

You should also receive a stack name.

This will typically be in the form duracloud-$ID where $ID is an identifier assigned by those handling the deployment. It may be based on or similar to a sitecode used by your institution for its domain (e.g. INSTITUTION.edu).

It is important to know this because your user will only be able to interact with a subset of buckets in an AWS account that are prefixed with that stack name. You will also see references to stack name throughout the documentation.

Important

Before proceeding confirm you have received:

• Access key (username) and secret (password) for direct s3 access if requested
• Stack prefix (duracloud-$ID)
• Web client username, password and url if requested

S3 Client Options

In order to keep things simple for the end user, less complicated to maintain on the technical side, but also provide some flexibility over how content can be uploaded to S3, there is no prescribed user interface. Any S3-compatible client can be used to interact with the tool.

We believe this is the right choice because there are many popular, well-supported, and tested options already available. However, we provide streamlined documentation for the use of the open source program Cyberduck as a downloadable GUI option, the AWS CLI for command line usage, and the web-based browser SFTPGo for the simplest access point.

Here’s a list of clients that have been used or tested by Lyrasis staff:

• AWS CLI
• Cyberduck
• S3Browser (Windows only - we are not providing additional documentation about this option)
• SFTPGo

But there are many others and you are free to use any S3 compatible client that you prefer.

After connecting to your S3 account via your preferred method, you will see the folders already created for your account using your dcp-$ID, including:

• -managed
• -public (default bucket for files that can be accessed publicly through CloudFront)
• -request (used for making create bucket or checksum inventory requests)

AWS CLI Documentation

Step 1: Install AWS CLI

Installing or updating to the latest version of the AWS CLI

After following the instructions for your operating system, check your installation:

aws --version

Step 2: Configure Your AWS Credentials

Configuration and credential file settings in the AWS CLI

Verify your configuration:

aws sts get-caller-identity

If you have multiple AWS accounts or environments, set up a named profile and configure with your key, secret, and region (for example, us-west-2):

aws configure --profile dcp

Cyberduck Documentation

Cyberduck documentation for setting up new connections:
https://docs.cyberduck.io/cyberduck/connection/

Step-by-step Instructions for DuraCloud Preserve

1. File → Open Connection\
2. Change dropdown menu to Amazon S3
3. Type in provided Access Key ID and Secret Access Key\
4. Click Connect

Tip

• Click Go → Enclosing Folder to navigate up the file path tree one level at a time, or click in the filepath dropdown to navigate up multiple levels after your connection is set up.
• Logs and other items you download via Cyberduck will go to your Downloads folder by default. You can change this under Edit → Preferences → Transfers (General tab)

SFTPGo Documentation

Navigate to: DuraCloud Preserve

Use this web-based interface to log in, upload, and download content.

Individual users will be provided credentials by their system administrator. The first time you log in, you will be asked to change your password. You can do this from the small person icon in the upper-right corner of the screen.

Note: SFTPGo login sessions are set for 2 hours.

Upon login you will see three folders already created for you:

• managed
• public
• request

From this web-based interface, you may:

• Create new buckets by uploading a request file (see Creating Buckets)
• Upload content buckets, creating subfolder structures as needed
• Download content from buckets
• Download reports and other hosted content from the managed bucket

Tip

Before proceeding, confirm that you are able to successfully connect to S3.

Managed Resources

When you connect to your S3 account for the first time, you will see a number of pre-existing buckets.

Pre-Existing Buckets

• dcp-$ID-request — used to make requests to create new buckets. See: Creating Buckets
• dcp-$ID-managed — contains generated files such as audit logs, inventory, and reports. This bucket is read-only.
• dcp-$ID-public — default public bucket. Files uploaded here will be publicly accessible via CloudFront.

Managed Bucket Structure

Over time, the dcp-$ID-managed bucket will contain the following folders:

• audit/ — AWS audit logs, organized by bucket
• batch/ — outputs from S3 batch operations
• cloudtrail/ — S3 event logs
• feedback/ — troubleshooting files generated when errors occur
• manifests/ — raw S3 inventory files generated by AWS
• metadata/ — internal stats used by the application (checksums, inventory, storage)
  • 0000-00-00-LATEST/ — most recent version of each stat file (sorts first)
  • YYYY-MM-DD/ — date-stamped archive copies
• reports/ — reports intended for user review and download
  • 0000-00-00-LATEST/checksums/ — most recent checksum reports
  • 0000-00-00-LATEST/manifests/ — most recent inventory manifest CSVs (one per bucket)
  • 0000-00-00-LATEST/storage/ — most recent storage usage HTML report
  • YYYY-MM-DD/ — date-stamped archive copies of all reports

See the Reports page for more detail on the contents of the reports/ folder.

Creating Buckets

Buckets are folders where you store your content. To create one, you upload a plain text file containing your bucket names to a special request location. The system processes the file automatically and creates your buckets within a few minutes.

Important

You can create up to five buckets per request.

Step 1: Create your bucket list file

Open a text editor (such as Notepad) and create a plain text (.txt) file with one bucket name per line:

manuscripts
newspapers
rare-books

Save the file with any name, for example bucket-list.txt.

Naming rules

• Do not include your stack name — it is added automatically as a prefix
• Use only letters, numbers, and hyphens (-)
• Names must not start or end with a hyphen
• Names must be short enough that the full bucket name stays under 63 characters — the system prepends dcp-$ID and reserves -repl as a suffix

Tip

To make objects publicly accessible, upload directly into the -public bucket. Sub-folders can be created within the -public bucket to align with your desired asset management methods.

The following are reserved and cannot be used in names: duracloud-, -logs, -managed, -repl, -request

Step 2: Upload the file

Upload your .txt file to the buckets folder inside your dcp-$ID-request bucket. If the buckets folder does not exist, create it first. The folder must be named buckets exactly.

Cyberduck

1. Connect to S3 (see Connecting to S3).
2. Navigate to the dcp-$ID-request bucket.
3. If a buckets folder does not exist, create one: Action → New Folder.
4. Open the buckets folder and drag your .txt file in, or click Upload.

Tip

If you re-upload the same filename with updated bucket names, Cyberduck may ask you to confirm overwriting. Confirm to proceed. Existing buckets will not be overwritten or deleted by this action.

SFTPGo

1. Log in to the web interface (see Connecting to S3).
2. Navigate to the request folder.
3. If a buckets sub-folder does not exist, click New Folder and name it buckets exactly.

4. Open the buckets folder, then click Upload Files or drag your .txt file into the upload area.

5. Click Save to complete the upload.

AWS CLI

aws s3 cp bucket-list.txt s3://dcp-$ID-request/buckets/bucket-list.txt

What happens next

Processing normally takes 0–2 minutes. For each name in your file, two buckets are created:

• duracloud-$ID-<name> — your active bucket
• duracloud-$ID-<name>-repl — a Glacier Deep Archive replication bucket (list-only; you will not have access to add to or download content from this replicated bucket)

A results file is uploaded to the feedback folder in your -managed bucket. Check it to confirm all buckets were created successfully, then refresh your client or reconnect to see them.

To create more buckets, update your file with new names and upload it again.

Troubleshooting

• No buckets were created — check the feedback folder in your -managed bucket for error messages.
• One name has an error — none of the buckets in that request will be created. Fix the name(s) as necessary and upload the file again.

Uploading Files

You can upload files to any bucket you have access to using Cyberduck, SFTPGo, or the AWS CLI. For information on creating buckets, see Creating Buckets.

Important

After uploading, content is automatically mirrored in the corresponding -repl bucket (Glacier Deep Archive). You cannot upload to or manage -repl buckets directly — they are managed by the system for backup purposes. If you are using the AWS CLI or a desktop client, you can view filenames in the -repl bucket as confirmation that replication occurred, but attempting to download or access files there will result in errors. The -repl buckets are not visible to SFTPGo users.

Tip

This system is designed primarily for the long-term storage and preservation of digital assets. Frequent or repeated access to files may lead to increased costs. Use it in accordance with its preservation-focused purpose.

AWS CLI

Upload an entire folder:

aws s3 sync ./local-folder s3://duracloud-$ID-mybucket/

Upload a single file:

aws s3 cp myfile.txt s3://duracloud-$ID-mybucket/myfile.txt

Replace duracloud-$ID-mybucket with the name of the bucket you are uploading to.

For full AWS CLI documentation, see https://docs.aws.amazon.com/cli/latest/userguide/cli-services-s3-commands.html

Cyberduck

1. Connect to S3 (see Connecting to S3).
2. Navigate to the bucket you want to upload to.
3. Drag files or folder structures from Finder (macOS) or File Explorer (Windows) directly into the Cyberduck window. Alternatively, click the Upload button and browse for files.
4. Cyberduck will show a transfer log confirming whether the upload was successful. A pop-up will appear if there are any errors or authorization issues.

For full Cyberduck documentation, see https://docs.cyberduck.io/cyberduck/transfer/

SFTPGo

1. Log in to the web interface (see Connecting to S3).
2. Navigate to the bucket folder you want to upload to.
3. Drag files or folders into the “drop files here to upload” area, or click it to browse for files.
4. Review the upload queue to confirm filenames and paths.
5. IMPORTANT! Click Save in the bottom right corner to complete the upload. Your content will not be saved if you skip this step.

Note

You cannot upload an empty folder in SFTPGo, but you can use the New Folder button to create folder structures before uploading content.

Uploading very large files (1–2 GB or more) may time out, depending on your browser settings and internet connection. For large uploads, use Cyberduck or another S3-compatible tool instead.

Tip

Click on Name, Size, or Last Modified to change the sort of objects from ascending to descending order and vice versa

Tip

We have occasionally seen a generic “Error uploading files” message in SFTPGo. Closing the error and trying again has resolved it in all known cases. This may be related to an expired session.

Reports

Reports generated by the system are stored in the reports/ folder inside your duracloud-$ID-managed bucket. For a full description of all folders in the managed bucket, see Managed Resources.

NOTE: AWS runs inventory once daily, your hosting provider may have no control over that timing, so your reports might not reflect exactly what you were expecting, or you may receive errors. These errors may be because of moving content around in your account and the inventory process has not caught up with the changes you made.

Report types

Checksum verification

Automatically generated by the system. Compares checksums between your active bucket and its replication bucket to detect data integrity issues. This report runs twice a year on a schedule.

• reports/0000-00-00-LATEST/checksums/<bucket>_checksum-report.csv — most recent report
• reports/YYYY-MM-DD/checksums/<bucket>_checksum-report.csv — date-stamped archive

Each report includes rows for each object within a bucket, including folder structures, with the following fields:

If there are any errors in the “status” column, contact your hosting provider for more information.

A note about the version_id column: When you overwrite files, both versions are available for a set period, and the old version is restorable in case you accidentally overwrote a file.

Checksum inventory

A per-object listing of checksum metadata built from S3 inventory data. Unlike the verification report, this records the checksum value for each object rather than comparing source and replica. Folder structures are not included as separate rows on this report.

• reports/0000-00-00-LATEST/checksums/<bucket>_checksum-inventory.csv — most recent report
• reports/YYYY-MM-DD/checksums/<bucket>_checksum-inventory.csv — date-stamped archive

Each row includes the object key, its CRC64NVMe or SHA256 checksum (when present), and a status:

• ok — checksum metadata retrieved successfully (note: this is not checksum verification)
• not_found — object was not found
• missing_checksum — object exists but has no checksum recorded
• error — other failure

Requesting a checksum inventory

To trigger a checksum inventory for a bucket, upload a .txt file named after the bucket to the checksums/ folder in your request bucket. The file’s contents do not matter — only the filename is used.

aws s3 cp duracloud-$ID-mybucket.txt s3://duracloud-$ID-request/checksums/dcp-$ID-mybucket.txt

The report will appear in the managed bucket once processing is complete (typically within a few minutes). If you mistyped the bucket name, no report will appear, and you should double-check you spelled everything correctly.

Inventory manifest

A full listing of all objects in a bucket, generated daily by AWS.

• reports/0000-00-00-LATEST/manifests/<bucket>.csv — most recent report
• reports/YYYY-MM-DD/manifests/<bucket>.csv — date-stamped archive

The fields included in this report:

This report does not include rows for folder structures.

Storage

An interactive HTML report generated weekly showing storage usage across all buckets in your stack.

• reports/0000-00-00-LATEST/storage/<stack>.html — most recent report
• reports/YYYY-MM-DD/storage/<stack>.html — date-stamped archive

Open the .html file in your browser to view:

The top row of the report includes

• Total Buckets - how many buckets are associated with your AWS account
• Total Files - across all buckets
• Total Size - how much space all files are taking up
• Total Prefixes - how many folders you’ve created within buckets
• Capacity Used - how much of your subscription you have used (will show a % of however many TBs)
• Data Transfer Out - how much data has been transferred out (or downloaded) from your account

Additional information includes

• Big Picture (Aggregated totals) — storage usage across all buckets, graphs for bucket size and file distribution
• Per bucket totals — usage broken down by individual bucket
• Per bucket / per prefix totals — usage by folder (prefix) within each bucket

Accessing reports

These reports are available no matter how you access your AWS account. The HTML storage reports will open in your default browser after you download.

Cyberduck

1. Connect to S3 (see Connecting to S3).
2. Navigate to duracloud-$ID-managed → reports → 0000-00-00-LATEST.
3. Open the relevant subfolder (checksums, manifests, or storage).
4. Right-click the file and select Download, Download As, or Download To.

SFTPGo

1. Log in to the web interface (see Connecting to S3).
2. Navigate to managed → reports → 0000-00-00-LATEST.
3. Open the relevant subfolder (checksums, manifests, or storage).
4. To download a single file, click directly on its filename, or check the box next to it and use Actions → Download.
5. To download multiple files, check the boxes next to them and use the Actions menu → Download. Selected items will be zipped automatically.

AWS CLI

Download the latest inventory manifest for a bucket:

aws s3 cp s3://duracloud-$ID-managed/reports/0000-00-00-LATEST/manifests/duracloud-$ID-mybucket.csv .

Download the latest checksum inventory for a bucket:

aws s3 cp s3://duracloud-$ID-managed/reports/0000-00-00-LATEST/checksums/duracloud-$ID-mybucket_checksum-inventory.csv .

Download the latest storage report:

aws s3 cp s3://duracloud-$ID-managed/reports/0000-00-00-LATEST/storage/duracloud-$ID.html .

List all available reports in a folder:

aws s3 ls s3://duracloud-$ID-managed/reports/0000-00-00-LATEST/checksums/

Sync an entire dated archive locally:

aws s3 sync s3://duracloud-$ID-managed/reports/ ./reports/

• Right-click to rename files when downloading to avoid overwriting reports from previous dates.

Downloading Content

You can download content from any bucket you have access to using Cyberduck, SFTPGo, or the AWS CLI.

Important

You cannot download content from replication buckets (buckets ending in -repl). Those are managed by the system for backup purposes. If you need to recover content from a replication bucket due to accidental deletion or corruption, contact your hosting provider.

AWS CLI

Download a folder:

aws s3 sync s3://duracloud-$ID-mybucket/myfolder ./local-folder

Download a single file:

aws s3 cp s3://duracloud-$ID-mybucket/myfile.txt .

Replace duracloud-$ID-mybucket with the name of the bucket you are downloading from.

For full AWS CLI documentation, see https://docs.aws.amazon.com/cli/latest/userguide/cli-services-s3-commands.html

Cyberduck

1. Connect to S3 (see Connecting to S3).
2. Navigate to the bucket and folder you want to download from.
3. Select the file(s) or folder(s) you want to download.
4. Right-click (or Control-click on Mac) and choose a download option:
   • Download — saves to your Downloads folder (or your configured default)
   • Download As — rename the file as you save it
   • Download To — choose a different save location

Tip

If you’re attempting to download from a bucket to which you don’t have download rights, such as a replication bucket, you will get a “Bad request. Request error” message or similar.

For full Cyberduck documentation, see https://docs.cyberduck.io/cyberduck/download/

SFTPGo

1. Log in to the web interface (see Connecting to S3).
2. Navigate to the folder containing the content you want to download.
3. Check the box next to the file(s) you want to download.
4. Click Actions and select Download. Selected items will be zipped automatically. To download a single file without zipping, click directly on its filename.

Tip

Some file types (such as .jpg, .txt, and .pdf) may be viewable directly in your browser by clicking the eye icon to the right of the filename. If there is no eye icon, the item can only be downloaded.

Making Content Public

There are two ways to make content public.

Pre-created public bucket (recommended)

Each stack includes a pre-created -public bucket that is served through a CloudFront distribution with a friendly domain. This is the recommended way to make content publicly accessible.

Your administrator will provide the public domain URL, but you can also construct what a public link from this bucket will look like based on this pattern:

https://{$ID}.preserve.duracloud.org/{FOLDER STRUCTURE}{FILENAME}

So, for example, a .jpg image found in the test account’s public bucket → photographs → Cats folder structure would look like:

https://test.preserve.duracloud.org/photographs/Cats/callie_and_friend.jpg

Cyberduck

Navigate to the duracloud-$ID-public bucket and upload your files there (see Uploading Files). Files uploaded to this bucket will be publicly accessible via the CloudFront domain.

SFTPGo

Navigate to the public folder and upload your content there (see Uploading Files). Files placed here will be publicly accessible via the CloudFront domain.

AWS CLI

aws s3 cp myfile.jpg s3://duracloud-$ID-public/myfolder/myfile.jpg

Creating public buckets (not recommended)

You can also make content publicly available by designating a bucket as -public - See How to Create Buckets.

You can construct what a public link will look like in this scenario based on this pattern:

https://{BUCKET_NAME}.s3.{REGION}.amazonaws.com/{PREFIX}/{FILE}

If you have spaces in any of your folder or filenames, replace those with a + sign when forming a URL. The region information is also optional.

So, for example, an image found in the lyrasis account’s bucket public → test-01 → catpics folder structure would look like:

https://dcp-test-public.s3.us-west-2.amazonaws.com/photographs/Cats/callie_and_friend.jpg

OR, without the region information:

https://dcp-test-public.s3.amazonaws.com/photographs/Cats/callie_and_friend.jpg

Note this feature is currently available but may be restricted in the future, as it goes against AWS guidelines.

Cyberduck sharing options

Cyberduck has some additional ways to share folders and individual objects.

• Navigate to the item you wish to share.
• Right-click on Windows / control+click on a Mac or two-finger click on a touchpad and select “Copy URL” — you can also use the Action (cog) menu and select “Open URL”.
  • If you right-click and select “Copy URL,” you will have options for how you wish to copy the URL, including HTTPS or HTTP, an expiration on the link (for individual objects only), or the AWS command link.
  • You can now share the item however you wish.
  • The HTTPS and HTTP links may be formed slightly differently (with AWS information before the bucket name), but they should still provide public access to objects in your account.

Query audit and inventory data

S3 audit logs and inventory can be synced locally for ad-hoc querying with DuckDB.

Pre-reqs

• AWS cli
• DuckDB

Sync the files

Download audit and / or inventory data to a local ./data folder. For example:

mkdir -p data/audit/
mkdir -p data/inventory/

aws s3 sync s3://${stack_name}-managed/audit/ data/audit/
aws s3 sync s3://${stack_name}-managed/manifests/ data/inventory/

# also download the query setup sql files
curl -O https://artifacts.preserve.duracloud.org/query/audit.sql
curl -O https://artifacts.preserve.duracloud.org/query/inventory.sql

Query audit data with DuckDB

The log files are in the S3 server access log format: one request per line, space-delimited, with a bracketed timestamp and quoted request_uri, referer, and user_agent. DuckDB’s CSV reader can’t handle the mixed quoting, so audit.sql reads each line as a single string and pulls fields out with a regex, exposing them as the audit view.

Launch the DuckDB CLI with the view preloaded:

duckdb -init audit.sql

Then query away. For example, every request ordered by time:

SELECT event_time, bucket, remote_ip, operation, key, http_status, bytes_sent
FROM audit
ORDER BY event_time;

Standard object operations by users

The requester field is an IAM ARN. Most traffic is programmatic (for example SDK sessions named aws-go-sdk-…, service roles doing replication or batch work etc.) but when a user assumes a role via a named profile, the session name at the end of the ARN is usually the IAM username. To see just the standard object-level operations (GET, PUT, DELETE) performed by assumed-role sessions, with the obvious programmatic sessions filtered out:

SELECT
  event_time,
  regexp_extract(requester, 'assumed-role/[^/]+/(.+)$', 1) AS who,
  bucket,
  operation,
  key,
  http_status
FROM audit
WHERE operation IN ('REST.PUT.OBJECT', 'REST.GET.OBJECT', 'REST.DELETE.OBJECT')
  AND requester LIKE '%:assumed-role/%'
  AND requester NOT LIKE '%aws-go-sdk-%'
  AND requester NOT LIKE '%assume-role-from-profile-%'
ORDER BY event_time;

Service roles (e.g. replication, batch jobs) may still appear in the results. Inspect the who column and add further NOT LIKE clauses for any session names that aren’t people of interest.

Query inventory data with DuckDB

S3 inventory reports ship as Parquet, which DuckDB reads natively. inventory.sql globs every parquet file under data/inventory/ and exposes them as the inventory view. Because each daily snapshot re-reports objects that haven’t changed, the view uses SELECT DISTINCT to collapse identical rows so basic queries see one row per unique observed state.

Launch the DuckDB CLI with the view preloaded:

duckdb -init inventory.sql

List every object across all buckets:

SELECT bucket, key, size, last_modified_date, storage_class
FROM inventory
ORDER BY bucket, key;

Object count and total bytes per bucket:

SELECT bucket, COUNT(*) AS objects, SUM(size) AS total_bytes
FROM inventory
GROUP BY bucket
ORDER BY bucket;

To work with both views in the same session, pass both scripts:

duckdb -init audit.sql -cmd ".read inventory.sql"

Users and Groups

There are three different account types available with DuraCloud Preserve:

• Power users
• Standard users
• Restricted users

Power users

Power users have the most options available. They can create buckets, upload content, move content around, and delete content. The delete option is unique to power users and should likely be limited to 1-3 staff at your institution because the ability to delete is so powerful.

Standard users

Standard users can create buckets, copy content, and upload content. They cannot delete. This means if moving content around between folder structures or buckets, standard users can copy the content from one location to another but not remove it from the original location. Only a power user can do that. Standard users have access and visibility to all buckets within your DuraCloud Preserve account.

Restricted users

Restricted users have the ability to upload content to specific buckets as identified by the power user(s) for the institution. Depending on those restrictions, this may mean that a restricted user can create buckets, if they have access to the -requested bucket. If you decide not to give a restricted user access to that bucket, then they will not have the ability to create buckets (but they can still create file structures within the buckets to which they have been designatd access).

Restricted users can see all buckets associated with a DuraCloud Preserve account but won’t be able to see or interact with content to which they are not designated. For example, if your account has -archives and -special-collections buckets, but a restricted user only has access to the -archives bucket, they will see that there is a -special-collections bucket, but if they try to navigate into the bucket, they will get a permissions error, no matter which option they’re using to interact with your DuraCloud Preserve account.

Tip

When setting up your DuraCloud Preserve account, be sure to tell your hosting provider which users should have power, standard, or restricted accounts. If asking for restricted users, also let your provider know to which buckets these users should have access. You’ll also need to update your provider when creating new buckets or needing new restricted user accounts.

External documentation

Documentation provided by third-party clients recommended for providing access to your account’s managed S3 buckets:

• Amazon command line documentation
• Cyberduck documentation, including for downloading and installing their client on Mac or Windows devices
• SFTPGo documentation (mostly intended for institutions setting up their own infrastructure)

Instructions

Setup

This documentation is focused on the technical aspects of the core functionality and how to test locally using the provided cli and remotely after the functions have been deployed.

This documentation does not address user functionality or deployment concerns, for those see:

• User guide
• Deployment guide

Pre-reqs

Requirements:

• mise (installs node and rust from mise.toml)
• aws cli
• cargo-lambda
• terraform

You must have access to an AWS account. Caution: costs may be incurred.

Setup

There are mise tasks to wrap cargo (et al.) commands for convenience:

These options are used frequently:

• --function function name i.e. bucket-request
• --profile aws profile name i.e. default
• --stack resource prefix used for identification/partitioning within an aws account

Run mise tasks to list all tasks and their options.

To get started run this task to create the base infrastructure:

# choose your own values for --stack and --profile
mise run setup --stack digipres-dev1 --profile default

This task uses Terraform so it must be installed for it to work.

Of most significance for testing using the above example will create:

• digipres-dev1-s3-batch-role (i.e. ${stack}-s3-batch-role)
• digipres-dev1-s3-replication-role (i.e. ${stack}-s3-replication-role)
• digipres-dev1-request (i.e. ${stack}-request)
• digipres-dev1-managed (i.e. ${stack}-managed)
• digipres-dev1-public (i.e. ${stack}-public)
• digipres-dev1-public-repl (i.e. ${stack}-public-repl)

The managed bucket will also be assigned a policy that permits it to be a target for S3 inventory from buckets using the same stack name (prefix).

The public bucket is “special” as it works differently from regular user created public buckets owing to a CloudFront distribution that is created to provide access to the files, rather than using raw S3 urls.

Testing remotely with Lambda

The base infrastructure is sufficient for testing using the provided cli. However, no AWS Lambda functions will be deployed by the setup task. If you want to test a full stack deployment including the Lambda functions then there is a deploy task for that:

mise run deploy --stack digipres-dev1 --profile default

This will build the Lambda packages and upload them to an “artifacts” bucket that Lambda can access. Doing this will enable you to try out the remote testing instructions for each function vs. only testing via the cli. Generally speaking the cli covers most of what happens when run through Lambda with these primary differences:

• Local cli testing uses your local AWS credentials
• Deployed Lambdas use permissions provided by IAM roles
• The entrypoints are different: see the cli vs. functions folders

Testing public access via CloudFront

terraform output cloudfront_domain_name

This will output something like: d2vy8bpfecxis5.cloudfront.net.

mise run upload --bucket digipres-dev1-public --dir example --file files/buckets.txt --profile default

Then access the file in the browser, it should work:

• https://d2vy8bpfecxis5.cloudfront.net/example/files/buckets.txt

For production the other Terraform outputs can be used for setting up a custom domain using ACM, see the deployment documentation for more details.

Functions

The core service functionality is encapsulated by Lambda functions that run on a schedule or in response to S3 events:

All functions can also be run locally via the dcp CLI, which additionally provides commands for tasks not covered by Lambda. See CLI for details.

bucket-request

• Lambda trigger: S3 event (fires when a user uploads a file to the request bucket)
• Dependencies: None

Overview

This Lambda function creates S3 buckets with prefab configuration based on a list of bucket names provided in a plain text file.

Example buckets.txt

manuscripts
newspapers
rare-books

The workflow is:

1. A text file containing bucket names is uploaded to the S3 bucket named ${stack}-request
2. The Lambda function is triggered by the upload event
3. The file is downloaded and processed — either locally (for development/testing) or inside Lambda (for remote execution)
4. Buckets are created according to the prefab configuration if they don’t already exist

CLI Testing

Use mise run bucket-request to process a file locally without uploading to S3:

mise run bucket-request --file files/buckets-list.txt --stack digipres-dev1 --profile default

• --file — path to a local file containing bucket names
• --stack — the stack name (used as a prefix for created buckets)
• --profile — the AWS profile to use

You can also create a single bucket by name without a file, using the cargo CLI directly.

Important

Before testing, export your aws profile prior to using the cargo CLI.

cargo run -p dcp -- bucket-request --stack=digipres-dev1 --name=rare-books

This is useful for one-off bucket creation or quick iteration without maintaining a file.

Remote Testing

Use mise run upload to upload a file to S3 and trigger the Lambda function as it would run in production:

mise run upload --bucket digipres-dev1-request --dir buckets --file files/buckets.txt --profile default

• --bucket — the name of the S3 request bucket (typically ${stack}-request)
• --dir — the S3 directory (path) to upload into (must be buckets)
• --file — path to the local file containing bucket names
• --profile — the AWS profile to use

Output

Given the example file files/buckets.txt, two buckets should be created (assuming they do not already exist):

• digipres-dev1-private — private S3 bucket
• digipres-dev1-private-repl — private S3 bucket used as the replication destination for the above

You can verify the buckets were created using:

mise run bucket list --profile default
# Filter results by stack name using grep
mise run bucket list --profile default | grep digipres-dev1

QA testing

Aside from the happy path, here are variations to try:

• File too large
• File invalid (rename some other file buckets.txt i.e a jpg)
• Bucket names are too long or has invalid characters
• Too many bucket names (5 max, additionals are discarded)
• Bucket names are duplicates, the buckets already exist
• Errors should be uploaded to a file in the managed bucket feedback path

inventory-report

Type: Lambda function
Trigger: S3 event (manifest.json is created)
Dependencies: None

Overview

This function processes Parquet-formatted S3 inventory data into a single human-readable CSV manifest per bucket. It also generates storage usage statistics used by the storage report:

• Total number of files and total storage used
• The same, broken down by top-level prefix (folder)

Note

At least one bucket must exist with files uploaded before this function can run. It has no inventory to process otherwise.

CLI testing

Run locally against the most recently available S3 inventory for a bucket:

mise run inventory-report --bucket digipres-dev1-private --profile default

• --bucket — Bucket name to process the inventory report for (required)
• --profile — AWS profile to use (required)

Remote testing

Staging a remote test requires crafting a specific event payload and uploading matching Parquet files, which adds significant overhead. In practice it is simpler to let the infrastructure run on its normal daily schedule and inspect the logs if the report does not appear.

If the CLI works but the Lambda does not, the most likely cause is an IAM permissions issue.

To stage a full remote test:

1. Craft an event payload that references a manifest.json.
2. Upload Parquet inventory files to the location referenced in the manifest.json.
3. Upload the manifest.json to the path specified in the event payload — this must be within the event notification path (/manifests).
4. Ensure the Parquet files contain the correct stack-prefixed bucket name.

Output

When run successfully there should be four generated files:

• metadata/0000-00-00-LATEST/manifests/stats/$bucket.csv
• metadata/YYYY-MM-DD/manifests/stats/$bucket.csv
• reports/0000-00-00-LATEST/manifests/$bucket.csv
• reports/YYYY-MM-DD/manifests/$bucket.csv

To access the latest report you can do:

aws s3 cp \
    s3://digipres-dev1-managed/reports/0000-00-00-LATEST/manifests/digipres-dev1-private.csv \
    . \
    --profile default

QA testing

Confirm:

• All expected files are available.
• The report contains expected items.
• The stats are accurate.

compute-checksums

Type: Lambda function
Trigger: Scheduled EventBridge event
Dependencies: None

Overview

This Lambda function triggers S3 batch checksum jobs to verify data integrity across your buckets. It processes standard/public + replication bucket pairs together, ensuring both the source and replicated data are checksummed.

Invocation methods

Scheduled execution (production)

The Lambda is automatically triggered by a scheduled EventBridge event at regular intervals.

CLI testing

Compute checksums for a single bucket and its replication pair:

mise run compute-checksums --bucket digipres-dev1-private --profile default

Parameters:

• --bucket — Standard or public stack bucket to checksum (required)
• --profile — AWS profile (required)

Constraints:

• Only supports single bucket at a time
• Automatically paired with replication bucket
• Cannot directly specify a replication bucket

Remote trigger

Compute checksums for all stack buckets in a given stack:

mise run trigger --function compute-checksums --stack digipres-dev1 --profile default

Parameters:

• --function — Function name (compute-checksums)
• --stack — Stack name (required)
• --profile — AWS profile (required)

Behavior: Triggers jobs for ALL stack buckets in the specified stack.

Output

Function response

{
    "StatusCode": 200,
    "ExecutedVersion": "$LATEST"
}

Receipt files

For each bucket pair processed, a job receipt is uploaded to:

• metadata/0000-00-00-LATEST/checksums/receipts/{source_job_id}.json
• metadata/0000-00-00-LATEST/checksums/receipts/{repl_job_id}.json
• metadata/0000-00-00-LATEST/checksums/receipts/{source_bucket_name}.json
• metadata/{date}/checksums/receipts/{source_bucket_name}.json

Purpose: The receipt is uploaded multiple times for different discovery paths:

• Job IDs — used by the Lambda checksum report process for internal tracking
• Bucket names — used by the CLI checksum report and for easier manual access

QA testing

Confirm:

• Jobs are created without errors
• Jobs are completed successfully
• All receipt files are generated and avaiable at the expected paths

checksum-request

Trigger: S3 event (.txt file uploaded under the request bucket’s checksums/ prefix)
Dependencies: inventory-report — the manifest CSV must already exist before running this

Overview

checksum-request turns an inventory manifest CSV into a checksum inventory. For every object listed in the manifest, it issues a HEAD request, records the CRC64NVMe checksum (when present), and assigns a per-object status of ok, not_found, missing_checksum, or error. The result is uploaded as a CSV to the managed bucket under reports/*/checksums/.

The trigger file’s name (minus the extension) identifies which bucket’s inventory to process. For example, uploading checksums/digipres-dev1-private.txt processes the inventory for digipres-dev1-private.

Workflow:

1. A .txt file named <bucket>.txt is uploaded to s3://${stack}-request/checksums/
2. The Lambda function is triggered by the upload event
3. The bucket name is parsed from the trigger filename
4. The function checks for a matching inventory manifest at s3://${stack}-managed/reports/0000-00-00-LATEST/manifests/<bucket>.csv
5. If found, the inventory is processed and the checksum CSV is uploaded to the managed bucket
6. The trigger file is deleted on success — re-upload to re-trigger

CLI testing

Run locally against an existing manifest:

mise run checksum-request --bucket digipres-dev1-private --profile default

Important

If no manifest exists for the bucket, the CLI will fail with Inventory report not found. Run inventory-report first.

Remote testing

Upload a trigger file to the request bucket’s checksums/ prefix:

mise run upload --bucket digipres-dev1-request --dir checksums --file files/digipres-dev1-private.txt --profile default

Note

The trigger file’s contents are not read — only its name matters.

Output

A successful run writes two files to the managed bucket:

• reports/0000-00-00-LATEST/checksums/<bucket>_checksum-inventory.csv
• reports/YYYY-MM-DD/checksums/<bucket>_checksum-inventory.csv

To download the latest report:

aws s3 cp \
    s3://digipres-dev1-managed/reports/0000-00-00-LATEST/checksums/digipres-dev1-private_checksum-inventory.csv \
    . \
    --profile default

QA testing

In addition to the happy path, test these edge cases:

checksum-report

Trigger: CloudTrail EventBridge event (batch job status: complete or failed)
Dependencies: compute-checksums

Overview

checksum-report processes AWS Batch compute checksum job output into a single checksum report CSV per bucket, and generates checksum verification stats (e.g. total mismatches).

In production, this function is triggered asynchronously by EventBridge when a batch job reaches complete or failed status. Each bucket pair (source + replication) runs as independent jobs. Report generation requires both jobs to be complete — if the first job finishes before the second, the function exits early and waits for the second event before continuing.

Usage

CLI (local testing)

Important

compute-checksums must have already run and completed for the target bucket pair (source + replication) before running this command.

mise run checksum-report --bucket digipres-dev1-private --profile default

Remote testing

Remote testing starts the same way as compute-checksums:

mise run trigger --function compute-checksums --stack digipres-dev1 --profile default

When a compute checksum job completes, it automatically triggers checksum report generation — once per bucket job.

Note

Replication buckets with objects in glacier storage tier can take days to complete. For testing, use buckets that contain only recently created objects that haven’t yet transitioned to glacier storage.

Tracking job status

mise run job-status-by-receipt --bucket digipres-dev1-private --profile default

A status of "Active" means the job is still running.

Expected output

On success, the CLI prints a verification summary and uploads a report CSV to the managed bucket:

Checksum report complete:
        Total objects:      6
        Matches:            6
        Mismatches:         0
        Missing replica:    0
        Missing source:     0
        Failed source:      0
        Failed replication: 0

A report CSV is also uploaded to the stack’s managed bucket for long-term record keeping.

To verify the checksum report was written to S3:

aws s3 ls s3://digipres-dev1-managed/reports/$(date +%F)/checksums/

QA testing

Confirm:

• Files are uploaded
• Appropriate logging for first bucket event (exit only)
• Appropriate logging for second bucket event (continuation)

storage-report

Type: Lambda function
Trigger: Scheduled EventBridge event (weekly)
Dependencies: inventory-report

Overview

This Lambda function generates a consolidated storage report for a stack, displaying storage usage across all standard and public buckets. The report is output as a single interactive HTML file using Chart.js for visualizations.

Report sections

• Aggregated totals — Storage usage across all buckets in the stack
• Per bucket totals — Storage usage broken down by individual bucket
• Per bucket / per prefix totals — Storage usage by prefix within each bucket

Prerequisites

The storage report requires S3 inventory data to be available. Before running this function:

1. S3 inventory must be enabled for the buckets
2. At least one inventory report must have been generated and uploaded
3. The inventory-report function must have completed successfully

CLI testing

Generate a storage report for a specific stack:

mise run storage-report --stack digipres-dev1 --profile default

Parameters:

• --stack — Stack name (required)
• --profile — AWS profile (required)

Remote trigger

mise run trigger --function storage-report --stack digipres-dev1 --profile default

Parameters:

• --function — Function name (storage-report)
• --stack — Stack name (required)
• --profile — AWS profile (required)

Scheduled execution

Automatically triggered weekly by EventBridge.

Output

When successful, four files are generated:

Statistics (JSON format)

• metadata/0000-00-00-LATEST/storage/stats/{stack}.json — Latest version
• metadata/YYYY-MM-DD/storage/stats/{stack}.json — Date-stamped archive

Contains raw storage metrics for programmatic access.

Report (HTML format)

• reports/0000-00-00-LATEST/storage/{stack}.html — Latest version
• reports/YYYY-MM-DD/storage/{stack}.html — Date-stamped archive

Interactive HTML report with Chart.js visualizations for viewing in a browser.

sync-users

• Lambda trigger: S3 event (fires when a TRIGGER file is uploaded to the managed bucket under sync-users/)
• Dependencies: None

Overview

This Lambda function synchronizes IAM users with an SFTPGo server so that each user can access their stack buckets over SFTP using their AWS access keys.

Unlike the other functions, sync-users operates across stacks. A user can belong to one or more stacks (via IAM group membership), and this function discovers those relationships to grant the user access to the appropriate set of buckets.

Important

sync-users only updates existing SFTPGo users — it does not create them. SFTPGo users are provisioned separately via the users terraform module.

The workflow is:

1. An empty TRIGGER file is uploaded to s3://${stack}-managed/sync-users/TRIGGER
2. The Lambda function is triggered by the upload event
3. Eligible IAM users are discovered (those with an Email tag and one or more stack group memberships)
4. For each user, their access/secret keys are retrieved from SSM and the matching SFTPGo account is updated with access to the buckets for each stack they belong to
5. The TRIGGER file is deleted on success

The SFTPGo connection details (SFTPGO_HOST, SFTPGO_USERNAME, SFTPGO_PASSWORD) are provided via Lambda environment variables set at deploy time.

CLI testing

The CLI can sync a single user or all users. SFTPGo credentials are read from the environment. Defaults for local development are set in mise.toml; to target a real server, set SFTPGO_HOST, SFTPGO_USERNAME, and SFTPGO_PASSWORD in your shell (or in an untracked mise.local.toml) before running:

mise run sync-users --profile default

To sync a specific user only:

cargo run -p dcp -- sync-users --username=alice

Unlike other CLI commands, sync-users does not take a stack argument — it works across all eligible users in the account.

Remote testing

Upload the TRIGGER file to the managed bucket to invoke the Lambda:

mise run upload --bucket digipres-dev1-managed --dir sync-users --file TRIGGER --profile default

• --bucket — the managed bucket name (${stack}-managed)
• --dir — the S3 directory (must be sync-users)
• --file — path to an empty local file named TRIGGER
• --profile — the AWS profile to use

Create an empty TRIGGER file first if you don’t have one (touch TRIGGER, or New-Item TRIGGER on Windows).

Output

sync-users does not produce files in S3. Successful execution can be verified in the following ways:

• The TRIGGER file is removed from s3://${stack}-managed/sync-users/ after a successful run
• CloudWatch logs show per-user processing output (email, identified buckets)
• The SFTPGo admin UI shows the expected users with the expected bucket virtual folders configured

QA testing

Confirm:

• A user with no Email tag is skipped (not synced)
• A user with no stack group memberships is skipped (no buckets)
• A user with no matching SFTPGo account is skipped (sync-users does not create SFTPGo users)
• A user belonging to multiple stacks has access to buckets from each stack
• The TRIGGER file is deleted after a successful run
• A user’s SFTPGo account reflects changes when their IAM group memberships change

CLI Reference

The dcp command-line tool provides access to core operations for managing buckets, generating reports, and maintaining data integrity. This reference documents all available commands and their usage.

Commands

Usage

dcp <COMMAND> [OPTIONS]

Global options

• -h, --help — Print help message

Commands

Bucket operations

bucket-reconciliation

Check bucket configuration and report drift.

dcp bucket-reconciliation [OPTIONS]

Detects inconsistencies between local bucket configuration and remote state, useful for identifying configuration drift or missing objects.

bucket-request

Process bucket creation requests.

dcp bucket-request [OPTIONS]

Handle requests to create new buckets within the stack infrastructure.

reset

Reset stack (empty buckets, requires confirmation).

dcp reset [OPTIONS]

Caution

This is a destructive operation. Removes all content from stack buckets. Requires confirmation before proceeding.

transfer

Transfer files from source to stack destination bucket.

dcp transfer [OPTIONS]

Copy data from a source bucket to a destination bucket within the stack. Useful for migrations and data reorganization.

Checksum operations

checksum

Checksum a file.

dcp checksum [OPTIONS] <FILE>

Compute checksum for a local file to verify data integrity.

compute-checksums

Run S3 batch operations compute checksums.

dcp compute-checksums [OPTIONS]

Trigger S3 batch checksum jobs for buckets. For detailed usage, see compute-checksums documentation.

checksum-request

Build checksum inventory from S3 inventory data.

dcp checksum-request [OPTIONS]

Process S3 inventory data to create a checksum inventory for analysis and verification.

checksum-report

Generate checksum report and statistics.

dcp checksum-report [OPTIONS]

Create a report of checksum results and statistics across buckets. For detailed usage, see checksum-report documentation.

Reporting operations

inventory-report

Generate inventory report and statistics.

dcp inventory-report [OPTIONS]

Create an inventory report from S3 inventory data showing bucket contents and statistics. For detailed usage, see inventory-report documentation.

storage-report

Generate storage report.

dcp storage-report [OPTIONS]

Generate a comprehensive storage report with visualizations showing storage usage across all buckets in the stack. For detailed usage, see storage-report documentation.

User management

sync-users

Sync IAM users to SFTPGo.

dcp sync-users [OPTIONS]

Synchronize IAM users with SFTPGo for SFTP access management. For detailed usage, see sync-users documentation.

Help

help

Print help message or help for a specific subcommand.

dcp help [COMMAND]

Display general help or help for a specific command.

Common workflows

Local testing with CLI

Most development and testing uses the CLI. See development documentation for local testing patterns.

Testing with deployed Lambda

For testing with deployed Lambda functions, see the documentation for specific operations:

• Compute checksums
• Storage reports
• Inventory reports

mise task helpers

The project provides mise tasks that wrap CLI commands with common parameters:

# Example: Run compute-checksums via mise
mise run compute-checksums --bucket digipres-dev1-private --profile default

# Example: Trigger Lambda function
mise run trigger --function storage-report --stack digipres-dev1 --profile default

# Example: Run CLI command directly
dcp compute-checksums --bucket digipres-dev1-private

For all available tasks, run mise tasks.

Cleanup

# empties buckets only, resources are not destroyed
mise run reset --stack digipres-dev1 --profile default

# teardown: empties buckets and deletes everything
mise run teardown --stack digipres-dev1 --profile default

Development

Most new features follow the same progression: CLI command → perform module → Lambda → Terraform. The CLI is the fastest path to a working end-to-end against real AWS, and the Lambda is a thin entrypoint that delegates to the same perform module once the functionality is proven.

1. Add a CLI command

The CLI lives in cli/src/commands/. Each command is its own module exposing an Args struct and a run function.

• Create cli/src/commands/<new_command>.rs with pub struct Args (clap) and pub async fn run(args: Args) -> Result<(), Box<dyn std::error::Error>>.
• Register the module in cli/src/commands/mod.rs.
• Add a Commands::<NewCommand>(commands::<new_command>::Args) variant and dispatch arm in cli/src/main.rs.
• Build SDK clients directly from awsutils::config::load_defaults() + Clients::new(&sdk_config), or use app::config::load(stack) if the command is stack-scoped.
• Wire clap args/env vars (e.g. #[arg(long, env = "SFTPGO_HOST")]).

Keep the CLI thin — parse args, build config, delegate to a perform function.

2. Implement the perform module

Shared functionality lives in shared/app/src/perform/. This is where the real work happens, and it is reused by both the CLI and the Lambda.

• Create shared/app/src/perform/<feature>.rs.
• Export a PerformArgs struct (public fields) and pub async fn perform(...) -> Result<..., <Feature>Error>.
• Add the module to shared/app/src/perform/mod.rs.
• Add a <Feature>Error variant in shared/app/src/errors.rs.
• If the work is stack-scoped, accept &Config. For account-wide work (e.g. cross-stack user sync), accept &Clients instead.

Write unit tests alongside the module with test_support::TestClientBuilder for mocked SDK responses. Integration tests that hit real AWS go in shared/app/tests/<feature>.rs (gated with #[ignore] and run via mise run test-integration).

3. Add a Lambda function

Once the CLI and perform module work, wrap them in a Lambda entrypoint.

cd functions
cargo lambda new <feature-name>

Add the new crate to members in the workspace Cargo.toml.

Each Lambda crate has two files:

• src/main.rs — reads env vars (at minimum STACK), loads config, starts the runtime.
• src/event_handler.rs — validates the inbound event (bucket, prefix, filename), short-circuits on config.debug_handler(), builds PerformArgs, calls perform.

Provide a sample payload at events/sample.json and test the handler with test_support::TestClientBuilder + debug_handler=true.

From the project root:

# Build all or specified pkg (using -p)
cargo lambda build [-p $pkg]

# Run local
cargo lambda watch -p $pkg

# Invoke local with a sample payload
cargo lambda invoke -p $pkg --data-example s3-event

# Invoke local using a json file as payload
cargo lambda invoke -p $pkg --data-file functions/$pkg/events/event.json

• Event payloads

4. Wire up Terraform

The Lambda needs infrastructure: an IAM policy scoping its permissions, a trigger (S3 event or EventBridge schedule), and an entry in the dev main.tf so the artifact gets uploaded and the function gets deployed.

4a. Shared constants → terraform locals

If the Lambda needs any prefixes, filenames, or other fixed values that terraform also needs to reference, add them to shared/constants/src/lib.rs and regenerate the terraform locals:

mise run locals

This keeps Rust and Terraform aligned — never hand-edit terraform/modules/stack/_locals.tf.

4b. Function-specific IAM policy

Create terraform/modules/stack/<feature>.tf following the pattern in bucket_request.tf or storage_report.tf:

locals {
  deploy_<feature> = contains(keys(local.functions), "<feature>") ? { "<feature>" = {} } : {}
}

data "aws_iam_policy_document" "<feature>" {
  for_each = local.deploy_<feature>

  statement { ... }
}

resource "aws_iam_role_policy" "<feature>" {
  for_each = local.deploy_<feature>

  role   = aws_iam_role.lambda[each.key].name
  policy = data.aws_iam_policy_document.<feature>[each.key].json
}

The base Lambda role, log group, and error alarm are created automatically from the functions map in functions.tf and alarms.tf — you do not need to add those.

4c. Trigger

Pick one based on how the function should fire:

S3 event trigger — add a aws_lambda_permission resource scoped to the source bucket ARN in your <feature>.tf, then add an entry to the appropriate bucket in notifications.tf:

for k, _ in local.deploy_<feature> : {
  id            = "<feature>-trigger"
  lambda_arn    = aws_lambda_function.main[k].arn
  events        = ["s3:ObjectCreated:*"]
  filter_prefix = "${local.<feature>_prefix}/"
  filter_suffix = local.<feature>_file
}

Add aws_lambda_permission.<feature> to the depends_on list.

Scheduled trigger — add local.deploy_<feature> into local.scheduled_functions in scheduler.tf. The schedule itself is configured via the schedule and tz fields on the functions map entry (defaults in variables.tf).

4d. Register in the dev main.tf

Add the function to local.functions in the project-root main.tf so it gets built, uploaded to the artifacts bucket, and deployed:

<feature> = {
  bucket = local.functions_bucket
  file   = "target/lambda/<feature>/bootstrap.zip"
  env    = { SOME_VAR = local.some_value } # optional
}

4e. Apply

mise run deploy --stack <stack> --profile <profile>

Testing the new function

• CLI (local, against real AWS): cargo run -p dcp -- <subcommand> [args]
• Lambda (local watch + invoke): cargo lambda watch -p <feature> in one shell, then cargo lambda invoke -p <feature> --data-file functions/<feature>/events/sample.json (or --data-example s3-event for a built-in fixture).
• Lambda (invoked remotely with sample payload): mise run trigger --function <feature> --stack <stack> --profile <profile>
• Unit tests: cargo test -p <crate>
• Integration tests: mise run test-integration --stack <stack> --profile <profile>

Testing unreleased artifacts

mise run publish and mise run push publish versioned artifacts (keyed by date + commit hash) without touching the stable channel production stacks deploy from. Deploy a test stack pinned to that version with mise run deploy --stack <stack> --profile <profile> --version <version>. See Releases for the full workflow, including promoting to stable and rolling back.

Each feature should also get a technical doc at docs/src/technical/<feature>.md following the format of the others in that directory.

Releases

Deployments consume two kinds of build artifacts:

• Lambda zips — one bootstrap.zip per function, in the regional dcp-artifacts-{region}.
• The dcp Docker image — duracloud/dcp, used by the ECS scheduled tasks (e.g. Archive-It).

Both are published with a derived version so test builds can be deployed without disturbing what production stacks use.

Version scheme

There is no semantic versioning and nothing to bump. Every publish derives its version from git:

<version> = <UTC date>-<short commit hash>[-dirty]
            e.g. 20260710-f34103e

The -dirty suffix is added when the working tree has uncommitted changes, so a build of unreleased work can never masquerade as its base commit. The same string is used as the S3 prefix and the Docker tag.

Channels

Stacks deployed without a pinned version follow the stable channel. Publishing to the stable channel is refused from a dirty tree, so every stable artifact is traceable to a commit. Versioned copies are never deleted, which is what makes rollback possible.

Testing a build

Publish the artifacts (versioned channel only — the stable channel and any production stack are untouched):

# Lambda zips; prints the version, e.g. 20260710-f34103e
mise run publish --profile <profile>

# dcp image, if the image is part of what you are testing
mise run push --profile <profile>

Deploy a test stack pinned to that version:

mise run deploy --stack <stack> --profile <profile> --version 20260710-f34103e

--version sets both the lambda zip keys (v/<version>/...) and the dcp image tag. Terraform fails at plan time if the version was never published (the aws_s3_object data sources look the keys up), so a typo cannot deploy stale artifacts.

Promoting to stable

Once the build is verified, commit, then release from the clean tree:

mise run release --profile <profile>

This rebuilds and publishes everything to both channels: the stable keys and latest tag are updated, and an immutable versioned copy is kept alongside.

Rolling back

Redeploy pinned to any previously released version:

mise run deploy --stack <stack> --profile <profile> --version <older-version>

To roll the stable channel itself back, check out the corresponding commit and run mise run release from it.

Lyrasis hosting and support

DuraCloud Preserve is open source and freely available for anyone to deploy into their own AWS account. However, Lyrasis provides a hosted option for individuals or institutions wanting a managed service.

Benefits

• Lyrasis manages an AWS account for you, which can be transferred to your ownership at any time upon cancellation of your hosting contract.
• Setup, configuration, and monitoring are fully handled by Lyrasis.
• You receive S3 access credentials to interact with DuraCloud Preserve using any S3 client.
• Technical support is provided by experienced hosting staff.
• We provide access to a web application (SFTPGo) for file uploads.

For pricing information and other details see the Lyrasis website.