Skip to content

Use cases and best practices

This guide shows the library at work in four common jobs, then lists the practices they share.

A deposit service takes a directory of files, such as a finished upload, and stores it as an OCFL object. The first deposit for an id creates the object; each later one adds a version whose files are exactly the new deposit.

import { OcflError, type OcflObject, openStorageRoot, type User } from '@ocfl/ocfl-fs';
const root = await openStorageRoot('/srv/ocfl');
/** Makes the object's head hold exactly the files in `directory`. */
async function deposit(id: string, directory: string, user: User): Promise<OcflObject> {
for (let attempt = 1; ; attempt++) {
try {
return await depositOnce(id, directory, user);
} catch (error) {
const raced = error instanceof OcflError && (error.code === 'OBJECT_NOT_FOUND' || error.code === 'OBJECT_OUT_OF_DATE');
if (!raced || attempt === 3) {
throw error;
}
}
}
}
async function depositOnce(id: string, directory: string, user: User): Promise<OcflObject> {
try {
return await root.create(id, { message: 'First deposit', user }, async (draft) => {
await draft.import(directory);
});
} catch (error) {
if (!(error instanceof OcflError && error.code === 'OBJECT_EXISTS')) {
throw error;
}
}
const object = await root.open(id);
return object.update({ message: 'New deposit', user }, async (draft) => {
await draft.clear();
await draft.import(directory);
});
}
const object = await deposit('arcp://name,corpus/item/1', '/srv/uploads/item-1', { name: 'Alice' });
console.log(`${object.id} is at version ${object.head.number}`);
  • The service tries root.create first and catches OBJECT_EXISTS, rather than checking whether the object exists. A check followed by a create can be beaten by another writer in between; the create can’t.
  • When the object exists, it opens it straight away and updates that. It never updates an object it opened earlier, which may no longer be the head.
  • draft.clear() and then draft.import(directory) make the new version hold only the deposited files. Without clear(), files from the last version that aren’t in the deposit would stay. See Replacing in Updating objects.
  • Content already in the object is shared, not stored again. Depositing the same files twice writes no version: update returns the object it was called on, at the same version. See An update that changes nothing.
  • import copies the whole directory or nothing. Its filter option can leave out files such as .DS_Store. It refuses a symbolic link the filter doesn’t leave out with UNSUPPORTED_FILE_TYPE, and throws SOURCE_CHANGED if a file changes while it is being copied, so deposit an upload only once it is complete.

Another writer can get in between the steps. If it removes the object after create throws OBJECT_EXISTS, open throws OBJECT_NOT_FOUND. If it commits a version after open, update throws OBJECT_OUT_OF_DATE. Neither commits anything, so deposit starts again from create, which makes the object again if it was removed. Running the build again is safe because it reads nothing from the object: whatever the head holds, the new version holds only the deposit. A build that decides what to change from files it read earlier mustn’t be retried this way; see Concurrent writers and recovery.

The retry limit keeps a busy object from looping forever. UPDATE_IN_PROGRESS, when another writer holds the object’s lock, isn’t retried here: that writer may be slow or dead, and only you can judge which. See Concurrent writers and recovery.

A file server reads files from OCFL objects and sends them over HTTP. It never writes, so it takes no lock and never waits on a writer.

import { createServer } from 'node:http';
import { pipeline } from 'node:stream/promises';
import { type ByteRange, OcflError, type OcflObject, openStorageRoot } from '@ocfl/ocfl-fs';
const root = await openStorageRoot('/srv/ocfl');
const opened = new Map<string, OcflObject>();
/** The object as it is now: the one opened before while no version has been committed since, or else opened again */
async function current(id: string): Promise<OcflObject> {
let object = opened.get(id);
if (!object || !(await object.isCurrent())) {
object = await root.open(id);
opened.set(id, object);
}
return object;
}
/** Reads `bytes=start-end` or `bytes=start-`; a suffix range such as `bytes=-500` isn't handled here. */
function parseRange(header: string | undefined, size: number): Required<ByteRange> | undefined {
const match = /^bytes=(\d+)-(\d*)$/.exec(header ?? '');
if (!match?.[1]) {
return undefined;
}
return { start: Number(match[1]), end: match[2] ? Math.min(Number(match[2]), size - 1) : size - 1 };
}
// GET /?id=arcp://name,corpus/item/1&path=audio/1.wav
const server = createServer(async (request, response) => {
const url = new URL(request.url ?? '/', 'http://localhost');
const id = url.searchParams.get('id');
const path = url.searchParams.get('path');
if (!id || !path) {
response.writeHead(400).end();
return;
}
try {
const object = await current(id);
const file = object.head.files.get(path);
if (!file) {
response.writeHead(404).end();
return;
}
const size = await file.size();
const range = parseRange(request.headers.range, size);
// Rejects before any headers are sent
const body = await file.stream(range);
response.writeHead(range ? 206 : 200, {
etag: `"${file.digest}"`,
'last-modified': file.modifiedAt.toUTCString(),
'content-length': range ? range.end - range.start + 1 : size,
...(range && { 'content-range': `bytes ${range.start}-${range.end}/${size}` }),
});
await pipeline(body, response);
} catch (error) {
if (response.headersSent) {
response.destroy();
} else if (error instanceof RangeError) {
response.writeHead(416).end();
} else if (error instanceof OcflError && ['OBJECT_NOT_FOUND', 'ID_INCOMPATIBLE_WITH_LAYOUT'].includes(error.code)) {
response.writeHead(404).end();
} else {
console.error(error);
response.writeHead(500).end();
}
}
});
server.listen(8080);
  • The server opens the storage root once, and keeps each object it opens. An opened OCFL object is the object as read at one moment, so before using a kept one the server asks object.isCurrent(), and opens the object again once a version has been committed since. isCurrent() costs one stat of the root inventory, where opening reads and checks the whole inventory. It is a cheap check, not a guarantee; see OcflObject.isCurrent.
  • file.stream(range) reads only the bytes asked for. A ByteRange counts from 0 and includes its end, as an HTTP Range header does. A range that starts after it ends, or after the file’s last byte, rejects with RangeError before anything is read, which the server answers with 416.
  • A file’s digest names its content, so it makes a good ETag: the same content has the same digest in every version.
  • file.modifiedAt is when the version that gave the file its current content was committed, which makes a good Last-Modified. It comes from the inventory, not the stored file, so copying or restoring the storage root doesn’t change it. Content changed back to an earlier content counts as a change, so the date is that of the change back, not the earlier version’s.
  • await file.stream(range) opens and checks the stored file before it resolves. One that is missing or isn’t a regular file rejects with INVALID_OBJECT, and one that can’t be read rejects with the file system error, all before any headers are sent, so the server answers 500. A stored file removed once the stream is open is still read in full.
  • The stream holds the stored file open until it ends. pipeline destroys it if the client goes away; code that gets a stream and doesn’t read it should destroy() it.
  • To serve an earlier version, read its files from object.versions.get(number)?.files instead of object.head.files.

An indexing job lists every OCFL object in a storage root and records what it finds, for a search index or a catalogue.

import { openStorageRoot } from '@ocfl/ocfl-fs';
const root = await openStorageRoot('/srv/ocfl');
const index = new Map<string, { version: number; files: string[] }>();
const damaged: string[] = [];
for await (const entry of root.objects()) {
if ('error' in entry) {
const { objectRoot, id, error } = entry;
damaged.push(`${error.ocflCode ?? error.code}: ${objectRoot}${id ? ` (${id})` : ''}`);
continue;
}
const { object } = entry;
index.set(object.id, { version: object.head.number, files: [...object.head.files.keys()] });
}
console.log(`Indexed ${index.size} objects`);
for (const line of damaged) {
console.warn(line);
}
  • root.objects() yields each object as it finds it, in no particular order. Reading it lazily keeps memory use flat however large the storage root is, and breaking out of the loop stops the walk.
  • An object it can’t open is yielded as an OcflError with code INVALID_OBJECT, and the listing carries on. The job keeps these errors and reports them at the end, so one damaged object doesn’t stop the index or go unnoticed. Their ocflCode, when present, names the OCFL rule the object breaks.
  • A file system error, such as a directory it can’t read, is thrown, not yielded, and ends the listing.
  • The listing leaves out an object that has lost its declaration and a create whose writer died, without an error. See what the library doesn’t check or do.

This job runs a fixity check on every object in a storage root and reports what has changed or gone missing since it was stored.

import { type FixityProblem, openStorageRoot } from '@ocfl/ocfl-fs';
const root = await openStorageRoot('/srv/ocfl');
// Stop the whole run after six hours, between or part way through an object
const signal = AbortSignal.timeout(6 * 60 * 60_000);
let failures = 0;
for await (const entry of root.objects()) {
if ('error' in entry) {
failures++;
console.error(`${entry.objectRoot}: can't open the object (${entry.error.ocflCode ?? entry.error.code})`);
continue;
}
const { object } = entry;
let problems: readonly FixityProblem[];
try {
problems = await object.checkFixity({
signal,
onProgress: ({ checkedFiles, totalFiles }) => process.stderr.write(`\r${object.id}: ${checkedFiles}/${totalFiles}`),
});
process.stderr.write('\n');
} catch (error) {
if (error === signal.reason) {
console.error(`\nOut of time: stopped at ${object.id}`);
failures++;
break;
}
// A stored file that can't be read, such as EACCES or EIO
failures++;
console.error(`\n${object.id}: fixity check failed`, error);
continue;
}
for (const { problem, storedFile, affects, expected, actual } of problems) {
failures++;
const state = problem === 'missing' ? 'is missing' : `has changed: expected ${expected}, found ${actual}`;
const files = affects.map(({ version, path }) => `v${version}:${path}`).join(', ') || 'no file';
console.error(`${object.id}: ${storedFile} ${state}; restore it from a backup (affects ${files})`);
}
}
process.exitCode = failures > 0 ? 1 : 0;

checkFixity() reads every stored file listed in the object’s manifest again and compares its digest with the one the inventory records. It returns one FixityProblem for each stored file that fails, so an empty list means every stored file matches.

It finds:

  • a missing file: the manifest lists a stored file, and there is no regular file at its path (missing);
  • changed content: a stored file whose digest no longer matches the one recorded for it (changed).

It doesn’t find:

  • a logical path pointing at the wrong content. The check compares each stored file with its own digest. It doesn’t check that each file in a version is the content you meant it to be. If a writer recorded notes.txt against another file’s digest, every stored file still matches and the check reports nothing.
  • extra files that the inventory doesn’t list;
  • problems in the fixity block of the inventory, which it doesn’t read.

It isn’t validation: an object can pass the fixity check and still break the OCFL specification. To check that, use an OCFL validator, as described under Objects written by other tools.

A stored file that is there but can’t be read, such as EACCES or EIO, throws instead of being reported. The job catches that for each object, so one unreadable file doesn’t end the run.

checkFixity({ signal, onProgress }) can be cancelled and report how far it has got:

  • onProgress is called after each stored file is checked, whether it passed, changed or is missing, with checkedFiles and totalFiles. checkedFiles reaches totalFiles before the check resolves. If onProgress throws, the check stops and rejects with that error.
  • Aborting signal stops the check, which rejects with the signal’s reason. No stored file is still open or being read once it has rejected. The job above gives the whole run six hours, and stops at the object it was checking when time runs out.

Each problem names the stored file by its path from the storage root, as storedFile, and lists in affects every version and logical path that reads it, so you can see which of your files are hit. It is empty for a stored file no logical path reads, such as a second copy of one digest’s content. Repair it by hand: copy a good copy from a backup to storedFile under the storage root. Writing the content again with object.update doesn’t repair it, because an update that finds the content’s digest already in the object keeps the stored file it has.

Both digests are lowercase, even when an object written by another tool records them in uppercase.

  • Open the object fresh before updating it. An opened OCFL object doesn’t change when another writer commits a version. Updating one that is no longer the head throws OBJECT_OUT_OF_DATE, so open it just before the update, and open it again before you retry. Within one process, use the object each update returns for the next: see How an update works. Concurrent writers and recovery shows the retry.
  • Catch OBJECT_NOT_FOUND instead of checking whether the object exists first. Call root.open(id) and catch the error, or call root.create and catch OBJECT_EXISTS. A separate check can be out of date by the time you act on it. root.has(id) answers cheaply, without reading the inventory, when you only need to know, such as to show it; don’t follow it with a create that relies on it.
  • Handle damaged object roots while listing. root.objects() yields { object } for each object and { error, objectRoot, id? } for each object root it can’t open. Test 'error' in entry before using entry.object, and record objectRoot so each damaged object root gets looked at.
  • Run fixity checks regularly. Disks fail, and files get changed by hand, without anyone noticing until the content is read. A regular check finds this while you still have a good copy to restore from. A check reads every stored file, so its time grows with the size of the storage root: run it when the storage is quiet, or spread the objects over several runs, and keep each run’s report.

See StorageRoot, OcflObject and OcflFile in the API reference for every error each method throws, and Errors for what to do about each.