Use cases and best practices
This guide shows the library at work in four common jobs, then lists the practices they share.
A deposit service
Section titled “A deposit service”A deposit service takes a directory of files, such as a finished upload, and stores it as an OCFL object. The first deposit for an id creates the object; each later one adds a version whose files are exactly the new deposit.
import { OcflError, type OcflObject, openStorageRoot, type User } from '@ocfl/ocfl-fs';
const root = await openStorageRoot('/srv/ocfl');
/** Makes the object's head hold exactly the files in `directory`. */async function deposit(id: string, directory: string, user: User): Promise<OcflObject> { for (let attempt = 1; ; attempt++) { try { return await depositOnce(id, directory, user); } catch (error) { const raced = error instanceof OcflError && (error.code === 'OBJECT_NOT_FOUND' || error.code === 'OBJECT_OUT_OF_DATE'); if (!raced || attempt === 3) { throw error; } } }}
async function depositOnce(id: string, directory: string, user: User): Promise<OcflObject> { try { return await root.create(id, { message: 'First deposit', user }, async (draft) => { await draft.import(directory); }); } catch (error) { if (!(error instanceof OcflError && error.code === 'OBJECT_EXISTS')) { throw error; } }
const object = await root.open(id); return object.update({ message: 'New deposit', user }, async (draft) => { await draft.clear(); await draft.import(directory); });}
const object = await deposit('arcp://name,corpus/item/1', '/srv/uploads/item-1', { name: 'Alice' });console.log(`${object.id} is at version ${object.head.number}`);- The service tries
root.createfirst and catchesOBJECT_EXISTS, rather than checking whether the object exists. A check followed by a create can be beaten by another writer in between; the create can’t. - When the object exists, it opens it straight away and updates that. It never updates an object it opened earlier, which may no longer be the head.
draft.clear()and thendraft.import(directory)make the new version hold only the deposited files. Withoutclear(), files from the last version that aren’t in the deposit would stay. See Replacing in Updating objects.- Content already in the object is shared, not stored again. Depositing the same files twice writes no version:
updatereturns the object it was called on, at the same version. See An update that changes nothing. importcopies the whole directory or nothing. Itsfilteroption can leave out files such as.DS_Store. It refuses a symbolic link the filter doesn’t leave out withUNSUPPORTED_FILE_TYPE, and throwsSOURCE_CHANGEDif a file changes while it is being copied, so deposit an upload only once it is complete.
Another writer can get in between the steps. If it removes the object after create throws OBJECT_EXISTS, open throws OBJECT_NOT_FOUND. If it commits a version after open, update throws OBJECT_OUT_OF_DATE. Neither commits anything, so deposit starts again from create, which makes the object again if it was removed. Running the build again is safe because it reads nothing from the object: whatever the head holds, the new version holds only the deposit. A build that decides what to change from files it read earlier mustn’t be retried this way; see Concurrent writers and recovery.
The retry limit keeps a busy object from looping forever. UPDATE_IN_PROGRESS, when another writer holds the object’s lock, isn’t retried here: that writer may be slow or dead, and only you can judge which. See Concurrent writers and recovery.
A read-only file server
Section titled “A read-only file server”A file server reads files from OCFL objects and sends them over HTTP. It never writes, so it takes no lock and never waits on a writer.
import { createServer } from 'node:http';import { pipeline } from 'node:stream/promises';import { type ByteRange, OcflError, type OcflObject, openStorageRoot } from '@ocfl/ocfl-fs';
const root = await openStorageRoot('/srv/ocfl');const opened = new Map<string, OcflObject>();
/** The object as it is now: the one opened before while no version has been committed since, or else opened again */async function current(id: string): Promise<OcflObject> { let object = opened.get(id); if (!object || !(await object.isCurrent())) { object = await root.open(id); opened.set(id, object); } return object;}
/** Reads `bytes=start-end` or `bytes=start-`; a suffix range such as `bytes=-500` isn't handled here. */function parseRange(header: string | undefined, size: number): Required<ByteRange> | undefined { const match = /^bytes=(\d+)-(\d*)$/.exec(header ?? ''); if (!match?.[1]) { return undefined; } return { start: Number(match[1]), end: match[2] ? Math.min(Number(match[2]), size - 1) : size - 1 };}
// GET /?id=arcp://name,corpus/item/1&path=audio/1.wavconst server = createServer(async (request, response) => { const url = new URL(request.url ?? '/', 'http://localhost'); const id = url.searchParams.get('id'); const path = url.searchParams.get('path'); if (!id || !path) { response.writeHead(400).end(); return; } try { const object = await current(id); const file = object.head.files.get(path); if (!file) { response.writeHead(404).end(); return; }
const size = await file.size(); const range = parseRange(request.headers.range, size); // Rejects before any headers are sent const body = await file.stream(range); response.writeHead(range ? 206 : 200, { etag: `"${file.digest}"`, 'last-modified': file.modifiedAt.toUTCString(), 'content-length': range ? range.end - range.start + 1 : size, ...(range && { 'content-range': `bytes ${range.start}-${range.end}/${size}` }), }); await pipeline(body, response); } catch (error) { if (response.headersSent) { response.destroy(); } else if (error instanceof RangeError) { response.writeHead(416).end(); } else if (error instanceof OcflError && ['OBJECT_NOT_FOUND', 'ID_INCOMPATIBLE_WITH_LAYOUT'].includes(error.code)) { response.writeHead(404).end(); } else { console.error(error); response.writeHead(500).end(); } }});
server.listen(8080);- The server opens the storage root once, and keeps each object it opens. An opened OCFL object is the object as read at one moment, so before using a kept one the server asks
object.isCurrent(), and opens the object again once a version has been committed since.isCurrent()costs onestatof the root inventory, where opening reads and checks the whole inventory. It is a cheap check, not a guarantee; seeOcflObject.isCurrent. file.stream(range)reads only the bytes asked for. AByteRangecounts from 0 and includes itsend, as an HTTPRangeheader does. A range that starts after it ends, or after the file’s last byte, rejects withRangeErrorbefore anything is read, which the server answers with416.- A file’s
digestnames its content, so it makes a goodETag: the same content has the same digest in every version. file.modifiedAtis when the version that gave the file its current content was committed, which makes a goodLast-Modified. It comes from the inventory, not the stored file, so copying or restoring the storage root doesn’t change it. Content changed back to an earlier content counts as a change, so the date is that of the change back, not the earlier version’s.await file.stream(range)opens and checks the stored file before it resolves. One that is missing or isn’t a regular file rejects withINVALID_OBJECT, and one that can’t be read rejects with the file system error, all before any headers are sent, so the server answers500. A stored file removed once the stream is open is still read in full.- The stream holds the stored file open until it ends.
pipelinedestroys it if the client goes away; code that gets a stream and doesn’t read it shoulddestroy()it. - To serve an earlier version, read its files from
object.versions.get(number)?.filesinstead ofobject.head.files.
An indexing job
Section titled “An indexing job”An indexing job lists every OCFL object in a storage root and records what it finds, for a search index or a catalogue.
import { openStorageRoot } from '@ocfl/ocfl-fs';
const root = await openStorageRoot('/srv/ocfl');const index = new Map<string, { version: number; files: string[] }>();const damaged: string[] = [];
for await (const entry of root.objects()) { if ('error' in entry) { const { objectRoot, id, error } = entry; damaged.push(`${error.ocflCode ?? error.code}: ${objectRoot}${id ? ` (${id})` : ''}`); continue; } const { object } = entry; index.set(object.id, { version: object.head.number, files: [...object.head.files.keys()] });}
console.log(`Indexed ${index.size} objects`);for (const line of damaged) { console.warn(line);}root.objects()yields each object as it finds it, in no particular order. Reading it lazily keeps memory use flat however large the storage root is, and breaking out of the loop stops the walk.- An object it can’t open is yielded as an
OcflErrorwith codeINVALID_OBJECT, and the listing carries on. The job keeps these errors and reports them at the end, so one damaged object doesn’t stop the index or go unnoticed. TheirocflCode, when present, names the OCFL rule the object breaks. - A file system error, such as a directory it can’t read, is thrown, not yielded, and ends the listing.
- The listing leaves out an object that has lost its declaration and a create whose writer died, without an error. See what the library doesn’t check or do.
Checking fixity across a storage root
Section titled “Checking fixity across a storage root”This job runs a fixity check on every object in a storage root and reports what has changed or gone missing since it was stored.
import { type FixityProblem, openStorageRoot } from '@ocfl/ocfl-fs';
const root = await openStorageRoot('/srv/ocfl');// Stop the whole run after six hours, between or part way through an objectconst signal = AbortSignal.timeout(6 * 60 * 60_000);let failures = 0;
for await (const entry of root.objects()) { if ('error' in entry) { failures++; console.error(`${entry.objectRoot}: can't open the object (${entry.error.ocflCode ?? entry.error.code})`); continue; } const { object } = entry;
let problems: readonly FixityProblem[]; try { problems = await object.checkFixity({ signal, onProgress: ({ checkedFiles, totalFiles }) => process.stderr.write(`\r${object.id}: ${checkedFiles}/${totalFiles}`), }); process.stderr.write('\n'); } catch (error) { if (error === signal.reason) { console.error(`\nOut of time: stopped at ${object.id}`); failures++; break; } // A stored file that can't be read, such as EACCES or EIO failures++; console.error(`\n${object.id}: fixity check failed`, error); continue; }
for (const { problem, storedFile, affects, expected, actual } of problems) { failures++; const state = problem === 'missing' ? 'is missing' : `has changed: expected ${expected}, found ${actual}`; const files = affects.map(({ version, path }) => `v${version}:${path}`).join(', ') || 'no file'; console.error(`${object.id}: ${storedFile} ${state}; restore it from a backup (affects ${files})`); }}
process.exitCode = failures > 0 ? 1 : 0;checkFixity() reads every stored file listed in the object’s manifest again and compares its digest with the one the inventory records. It returns one FixityProblem for each stored file that fails, so an empty list means every stored file matches.
It finds:
- a missing file: the manifest lists a stored file, and there is no regular file at its path (
missing); - changed content: a stored file whose digest no longer matches the one recorded for it (
changed).
It doesn’t find:
- a logical path pointing at the wrong content. The check compares each stored file with its own digest. It doesn’t check that each file in a version is the content you meant it to be. If a writer recorded
notes.txtagainst another file’s digest, every stored file still matches and the check reports nothing. - extra files that the inventory doesn’t list;
- problems in the
fixityblock of the inventory, which it doesn’t read.
It isn’t validation: an object can pass the fixity check and still break the OCFL specification. To check that, use an OCFL validator, as described under Objects written by other tools.
A stored file that is there but can’t be read, such as EACCES or EIO, throws instead of being reported. The job catches that for each object, so one unreadable file doesn’t end the run.
checkFixity({ signal, onProgress }) can be cancelled and report how far it has got:
onProgressis called after each stored file is checked, whether it passed, changed or is missing, withcheckedFilesandtotalFiles.checkedFilesreachestotalFilesbefore the check resolves. IfonProgressthrows, the check stops and rejects with that error.- Aborting
signalstops the check, which rejects with the signal’sreason. No stored file is still open or being read once it has rejected. The job above gives the whole run six hours, and stops at the object it was checking when time runs out.
Each problem names the stored file by its path from the storage root, as storedFile, and lists in affects every version and logical path that reads it, so you can see which of your files are hit. It is empty for a stored file no logical path reads, such as a second copy of one digest’s content. Repair it by hand: copy a good copy from a backup to storedFile under the storage root. Writing the content again with object.update doesn’t repair it, because an update that finds the content’s digest already in the object keeps the stored file it has.
Both digests are lowercase, even when an object written by another tool records them in uppercase.
Best practices
Section titled “Best practices”- Open the object fresh before updating it. An opened OCFL object doesn’t change when another writer commits a version. Updating one that is no longer the head throws
OBJECT_OUT_OF_DATE, so open it just before the update, and open it again before you retry. Within one process, use the object each update returns for the next: see How an update works. Concurrent writers and recovery shows the retry. - Catch
OBJECT_NOT_FOUNDinstead of checking whether the object exists first. Callroot.open(id)and catch the error, or callroot.createand catchOBJECT_EXISTS. A separate check can be out of date by the time you act on it.root.has(id)answers cheaply, without reading the inventory, when you only need to know, such as to show it; don’t follow it with acreatethat relies on it. - Handle damaged object roots while listing.
root.objects()yields{ object }for each object and{ error, objectRoot, id? }for each object root it can’t open. Test'error' in entrybefore usingentry.object, and recordobjectRootso each damaged object root gets looked at. - Run fixity checks regularly. Disks fail, and files get changed by hand, without anyone noticing until the content is read. A regular check finds this while you still have a good copy to restore from. A check reads every stored file, so its time grows with the size of the storage root: run it when the storage is quiet, or spread the objects over several runs, and keep each run’s report.
See StorageRoot, OcflObject and OcflFile in the API reference for every error each method throws, and Errors for what to do about each.