Moving storage

You can change the blob store of a running server, for example from the local disk to R2. The server copies every existing blob in the background. Reads and uploads keep working during the copy. The old store is never changed or deleted.

Start a move

1. Open Settings, group Storage.

2. Change the backend settings, for example storage to s3, and enter the endpoint, bucket and keys. See Storage.

3. Save. The server probes the new store first. If the probe fails, nothing is saved.

4. If the server holds files and the new settings point to another location, a move starts. The page shows a progress bar and a Cancel button.

The storage settings do not change yet. They change when the copy ends. The server saves the other settings of the same request at once.

A move starts only when the new settings address another location and the server holds assets. These changes save and take effect at once, with no move: only credentials, region, path style or redirect changed, or the server has no assets.

Only one move runs at a time. A second change while a move runs gets 409.

What happens during the copy

While the move copies, the server holds the old and the new store together:

• Reads and listings use the old store.
• Uploads and deletes go to the new store first, then to the old store. If either fails, the request fails.

So a file that a client uploads during the move is in both stores. You do not need to stop publishing.

During a storage move the server holds the old and the new store together. Reads and listings use the old store. Uploads and deletes go to the new store first, then to the old store, and the request fails if either fails. The copier walks the assets in order, checks the size and SHA-256 of each old blob, writes it to the new store, and saves its position every 500 assets. With no failures the move is done and the new store becomes the active storage. With failures the move is failed, the server keeps the old store, and the page lists each blob that failed. The old store is never changed or deleted by the move.

• Reads and listings

  Old store

• Uploads and deletes

  New store Old store

  If either fails, the request fails.

• The copier

  Old storeNew store

  Asset by asset, in order. Checks size and SHA-256. Saves its place every 500 assets.

done

No failures. The new store becomes the active storage.

failed

"N blobs could not be copied". The server keeps the old store and lists each failed blob.

The server holds both stores until the copy ends. Nobody has to stop publishing.

The copier walks the assets in order and saves its position after every 500 assets. For each asset it:

1. Takes the same lock as an upload and reads the asset again.
2. Skips an asset that was deleted.
3. Skips a blob that the new store already holds with the same size and a modification time that is not older than the asset.
4. Checks the size and the SHA-256 of the old blob.
5. Writes the blob to the new store.

When the move ends

• No failures. The new store becomes the active storage. The move is done.
• Failures. The move is failed with the text “N blobs could not be copied”. The server keeps the old store. The page lists each blob that failed, with a reason. A blob fails when it is missing in the old store or when its size or SHA-256 does not match.

To retry after a failure, fix the cause and save the storage settings again. The new move skips the blobs that are already copied.

Note

A missing source blob means the database lists a file that the old store no longer holds. The copier does not invent the file. Restore the blob in the old store, then start the move again.

Cancel a move

Click Cancel on the Settings page. The server stops the copier, marks the move cancelled and returns to the old store. The new store keeps the blobs it already received. You cannot cancel a move that has ended.

The same action is DELETE /api/storage/moves/{id} in the HTTP API.

Restart during a move

A shutdown leaves the move in the state copying. At the next start, before the server serves requests, the move continues from its saved position.

• If the settings already point to the new store, the server marks the move done.
• If the server cannot open or probe the new store, the move fails with that error.

Gotchas

• The old store stays. After a successful move, the old blobs are still there. Delete them yourself when you are sure the new store is right.
• Time. A move copies every blob once. The time depends on the number of blobs and the speed of both stores.
• Keys with a prefix. On S3, the new keys use s3.prefix. See Storage.
• Backups. Take a backup of the data directory before a large move. See Backups and data.