API Reference

Use croissant-baker as a Python library to generate Croissant metadata programmatically — without the CLI.

MetadataGenerator

croissant_baker.metadata_generator.MetadataGenerator

Generates Croissant metadata for datasets with automatic type inference.

Discovers files, delegates format-specific logic to registered handlers via the build_croissant protocol, and assembles the final JSON-LD.

scan_report property

Coverage of the last generate_metadata() call.

One entry per file the scan found, each carrying its outcome and, where it was not described, the reason. Populated before the "No supported files found in the dataset" guard fires, so a caller catching that error can still ask why. Empty before the first call.

reference_report property

What foreign-key detection found, or None when it did not run.

Three states, because they mean different things to a reader: None for a bake that never ran the pass (detect_references off, or assembly raised before it), an empty report for one that ran and found nothing, and a populated one otherwise.

__init__(dataset_path, name=None, description=None, url=None, license=None, citation=None, version=None, date_published=None, date_created=None, date_modified=None, creators=None, publisher=None, keywords=None, in_language=None, same_as=None, sd_license=None, sd_version=None, alternate_name=None, is_live_dataset=None, temporal_coverage=None, usage_info=None, identifier=None, conditions_of_access=None, is_accessible_for_free=None, included_in_data_catalog=None, profiles=None, field_mappings=None, count_csv_rows=False, max_workers=None, detect_references=False, includes=None, excludes=None, rai_fields=None, handlers=None)

Initialize the metadata generator for a dataset.

Parameters:

Name	Type	Description	Default
dataset_path	str	Path to the directory containing dataset files.	required
name	Optional[str]	Dataset name (defaults to directory name).	None
description	Optional[str]	Dataset description.	None
url	Optional[str]	Dataset URL.	None
license	Optional[str]	License URL or SPDX identifier (e.g. "CC-BY-4.0").	None
citation	Optional[str]	Citation text, preferably BibTeX format.	None
version	Optional[str]	Dataset version string.	None
date_published	Optional[str]	Publication date in ISO format ("2023-12-15" or "2023-12-15T10:30:00").	None
date_created	Optional[str]	Creation date in ISO format.	None
date_modified	Optional[str]	Last-modified date in ISO format.	None
creators	Optional[List[Dict[str, str]]]	List of dicts with "name", "email", and/or "url" keys.	None
publisher	Optional[str]	Name of the publishing organization (schema.org/Organization).	None
keywords	Optional[List[str]]	Topical keywords for dataset discovery (schema.org/keywords).	None
in_language	Optional[List[str]]	BCP 47 language code(s) (e.g. "en"). Multiple supported.	None
same_as	Optional[List[str]]	URLs of equivalent dataset records (e.g. DOI, mirror landing pages). Multiple values supported per schema.org/sameAs.	None
sd_license	Optional[str]	License of the metadata description itself, distinct from the data license (schema.org/sdLicense).	None
sd_version	Optional[str]	Version of the metadata description, distinct from version. Defaults to None — only emitted when set.	None
alternate_name	Optional[str]	Short alias for the dataset (schema.org/alternateName).	None
is_live_dataset	Optional[bool]	Mark dataset as a live, evolving stream.	None
temporal_coverage	Optional[str]	Time period the data covers — schema.org accepts free text or ISO 8601 (e.g., "2008/2019", "2023-01-15").	None
usage_info	Optional[str]	URL of a usage/consent policy (e.g., a DUO term URL, ODRL Offer URL).	None
identifier	Optional[List[str]]	Accessions or persistent identifiers the dataset is known by (e.g. a dbGaP phs number, an EGA study accession, a DOI). Deduplicated in the order given; one value is emitted as a string, several as a list.	None
conditions_of_access	Optional[str]	How access is obtained, in free text (e.g. the data access agreement and committee for a controlled release). schema.org/conditionsOfAccess.	None
is_accessible_for_free	Optional[bool]	Whether the data can be had without payment or an access agreement. Tri-state: None leaves the key absent.	None
included_in_data_catalog	Optional[str]	URL of a catalog entry that lists this dataset. Emitted as a sc:DataCatalog node carrying the URL, the one range schema.org/includedInDataCatalog has.	None
profiles	Union[str, List[str], None]	Additional profiles the document declares in conformsTo alongside Croissant 1.1, by the names in PROFILE_CONFORMS_TO. A list, or one name as a bare string; either form may carry comma-separated names. Normalised by normalize_profiles at construction. generate_metadata refuses a document that declares a profile without the fields PROFILE_MINIMUM_KEYS lists for it.	None
field_mappings	Optional[Dict[str, Dict[str, object]]]	Per-column overrides keyed by field name. Each value is a dict with optional equivalent_property (vocab URI) and data_types (list of vocab URIs). Used to link columns to external vocabularies like Wikidata/SNOMED/LOINC.	None
count_csv_rows	bool	If True, scan each CSV fully for exact row counts. Defaults to False for performance.	False
max_workers	Optional[int]	Maximum worker threads for per-file metadata extraction. None (default) auto-sizes from the CPU count; 1 forces serial. Output is identical regardless of this value.	None
detect_references	bool	If True, run conservative foreign-key detection and emit cr:references links between RecordSets that share a key column with a name-identifiable parent table. Defaults to False.	False
includes	Optional[List[str]]	Glob patterns to include. Applied before excludes.	None
excludes	Optional[List[str]]	Glob patterns to exclude. Applied after includes.	None
rai_fields	Optional[Dict[str, object]]	Native mlcroissant RAI metadata fields, passed through to mlc.Metadata unchanged.	None
handlers	Optional[HandlerRegistry]	Which handlers to consult, and in what order. Defaults to the built-in registry. Supply one to bake with a narrower set, or with a handler the baker does not ship.	None

Raises:

Type	Description
ValueError	If dataset_path is not a directory, or a named profile is not one of PROFILE_CONFORMS_TO.

generate_metadata(progress_callback=None)

Generate complete Croissant metadata for the dataset.

Per-file metadata extraction (handler selection, whole-file SHA-256, header/schema reads) is I/O-bound and independent across files, so it runs on a thread pool sized by max_workers. Results are reassembled in discovery order before any FileObject @id is assigned, so the document is identical regardless of worker count. Nothing is reported per file as it goes: what became of each one is in :attr:scan_report, which the CLI prints under --verbose.

Parameters:

Name	Type	Description	Default
progress_callback		Optional callback with signature (completed: int, total: int, file_path: str) -> None invoked once per file as it finishes extraction.	None

Raises:

Type	Description
ValueError	If nothing in the dataset could be described, or a declared profile's minimum fields are missing from the document that was assembled.

save_metadata(output_path, validate=True)

Generate and save Croissant metadata to a file.

Parameters:

Name	Type	Description	Default
output_path	str	Path where the JSON-LD metadata file will be written.	required
validate	bool	If True (default), validates with mlcroissant before saving.	True

Raises:

Type	Description
ValueError	If validation fails or the file cannot be saved.

Scan coverage

Every file the scan finds gets one entry carrying what became of it. The report survives the "no supported files" error, so a bake that described nothing can still explain itself.

croissant_baker.scan.ScanReport dataclass

A view over resolved scan entries: what was described, and what was not.

:meth:summary_lines is for the terminal, and its length depends on how many kinds of problem occurred, never on how many files did. A diagnostic is counted there the same way: one line per code. :meth:to_dict carries the per-file detail, for --report and for downstream tools checking coverage.

total property

How many files the scan found.

described property

Entries a handler described and whose nodes were assembled.

linked property

Entries carried as another described file in a different form.

referenced property

Entries another file's handler put in the document.

undescribed property

Entries the document does not carry, in scan order.

Membership of the document, not of a record set: a file with a FileObject is in there whether it got there on its own (DESCRIBED), as another form of a described file (LINKED), or as part of a multi-file record (REFERENCED).

counts()

Number of entries per reason, in declaration order.

Over the undescribed only: a reason answers why a file is not in the document, so counting one against a file that is in there says the opposite of what it means. A linked file keeps its own reason — that is the evidence it was linked on — and is not counted here.

summary_lines()

A header plus at most one line per reason.

The header names every way into the document that happened, so a run that linked or referenced files says so rather than folding them into one count. described and not described are always stated; the two middle buckets appear only when non-zero.

to_dict()

Every discovered file with its outcome.

reason is one of a finite set a caller can branch on; detail is the sentence for a human. described, linked and referenced are the three ways into the document and sum with undescribed to total; by_reason accounts for the undescribed alone.

by_diagnostic is orthogonal to all of them, and a file's diagnostics key appears only when it has one.

croissant_baker.entries.Outcome

Bases: str, Enum

What became of a file the scan found.

str mixin so an outcome serialises to its own value in the JSON report. PENDING, READY and WOULD_PROCESS are working states; a completed bake leaves none of them behind.

croissant_baker.entries.Reason

Bases: str, Enum

Why a file was not described, as one of a finite set of categories.

The entry's detail names the file and the exception; this is what the summary counts, so terminal output stays bounded by the number of kinds of problem.

Foreign-key detection

Opt-in, via --detect-references or detect_references=True. The report is None when the pass did not run, so "found nothing" and "never asked" stay distinguishable. Shared key columns it declines to link are carried here rather than dropped.

croissant_baker.references.ReferenceReport dataclass

What the foreign-key pass linked, and what it declined to link.

Held by the generator and rendered by the CLI, the way :class:croissant_baker.scan.ScanReport is: the library states what it found, and only the CLI writes to a terminal. :meth:summary_lines is one line by default, whatever the dataset, and names a column per line only under verbose.

links = field(default_factory=list) class-attribute instance-attribute

unresolved = field(default_factory=list) class-attribute instance-attribute

summary_lines(verbose=False)

One line saying what was linked, and under verbose what was not.

The invitation to re-run is folded into that line rather than added as its own Tip:, because the coverage section above may already end in one and two competing tips read as noise.

croissant_baker.references.Unlinkable

Bases: str, Enum

Why a shared key column was reported instead of linked.

str mixin so a reason serialises to its own value, as :class:croissant_baker.entries.Reason does.

File Discovery

croissant_baker.files.discover_files(dir_path, include_patterns=None, exclude_patterns=None)

Recursively discover all files in a directory (skipping hidden directories) and return their relative paths.

Parameters:

Name	Type	Description	Default
dir_path	str	Path to the directory to scan.	required
include_patterns	Optional[List[str]]	Optional list of glob patterns to include.	None
exclude_patterns	Optional[List[str]]	Optional list of glob patterns to exclude.	None

Returns:

Type	Description
List[Path]	List of relative file paths found in the directory.

Raises:

Type	Description
FileNotFoundError	If the directory does not exist or is not a directory.
PermissionError	If the directory cannot be accessed.

Handler Interface

A handler answers three questions about one format, none of them involving compression: the pipeline resolves that first and hands over a FileSource.

can_handle(path) and extract_metadata(path) are the previous names for the first two. They still work — in both directions — and warn once per handler class. See DEVELOPMENT.md for how to write a handler.

croissant_baker.handlers.base_handler.FileTypeHandler

Bases: ABC

Abstract base class for file type handlers.

Each handler is responsible for three things:

• claims: decide if this handler owns a given file
• extract: read one file's structure from a source
• build_croissant: turn that metadata into FileSets + RecordSets

The generator owns FileObject creation and @id assignment.

claims and extract never see a compression wrapper: both are given a source built from the logical name.

build_croissant is given both names. relative_path is logical and derives identifiers, so a wrapped file and its plain twin describe one table; stored_name is the file as it sits on disk and belongs in descriptions. Use :func:~croissant_baker.handlers.utils.display_name for the latter.

can_handle(path) and extract_metadata(path) are the previous names for the first two. They still work, in both directions, and warn once per class.

Adding a new format: subclass this, implement all three methods, add the instance to builtin_handlers() in registry.py.

Subclasses set these class attributes for documentation and dispatch:

• EXTENSIONS: format suffixes this handler claims, e.g. (".csv",). Compression is stripped before a handler is asked, so ".csv.gz" is never a valid entry.
• FORMAT_NAME, FORMAT_DESCRIPTION: for the generated docs table.
• INPUT_KIND: see :class:InputKind. Defaults to STREAM.

claims(source)

Whether this handler describes the given file.

Match on source.suffix, the logical suffix: .csv for data.csv and data.csv.gz alike. Use source.peek() when the extension alone is not enough.

extract(source, **kwargs)

Describe one file's structure.

Read through source.open() or source.open_text(), which are already decompressed, and take file_name, file_size and sha256 from the source. Report only the format's own media type in encoding_format; the generator adds the compression one.

Thread-safety: may be called concurrently across files on a single shared handler instance, so keep per-call state local.

Returns:

Type	Description
dict	Extracted metadata. For tabular data this should include
dict	column_types mapping column names to Croissant types.

build_croissant(file_metas, file_ids) abstractmethod

Build Croissant FileSets and RecordSets for every file this handler read.

Called once per handler, after the FileObject loop.

Parameters:

Name	Type	Description	Default
file_metas	list[dict]	metadata dicts from extract, one per file	required
file_ids	list[str]	FileObject @ids assigned by the generator, aligned by position with file_metas	required

Returns:

Type	Description
tuple	(file_sets, record_sets). FileObjects are the generator's.
tuple	A handler that describes some of its batch but not all returns a
tuple	third element, declined: one (index, Reason, detail) per
tuple	file it passed over, indexed into file_metas. Those files are
tuple	reported as failures and the rest of the batch is still described.
tuple	Raising instead fails the whole batch.

can_handle(file_path)

Deprecated. Implement and call :meth:claims instead.

extract_metadata(file_path, **kwargs)

Deprecated. Implement and call :meth:extract instead.

croissant_baker.handlers.base_handler.InputKind

Bases: str, Enum

What a handler needs in order to read a file.

A handler declares the input it consumes rather than whether it supports compression; compression support follows from the declaration.

Sources

croissant_baker.sources.FileSource dataclass

One file, offered to a handler with compression already resolved.

Attributes:

Name	Type	Description
name	str	The logical basename. data.csv whether the file on disk is data.csv or data.csv.gz.
relative_path	Path	The logical path relative to the dataset root.
size	int	Size in bytes of the file as stored, so compressed size for a wrapped file. This is what goes in contentSize.
exists	bool	Whether the file was present when the source was built.

Compared by identity: two different files can share a logical name, a size and an existence flag, and the openers that tell them apart are closures.

name instance-attribute

relative_path instance-attribute

suffix property

The logical suffix, lowercased. .csv for data.csv.gz.

size instance-attribute

sha256 cached property

SHA-256 of the bytes as stored, so the digest identifies the artefact that was actually acquired.

open()

Open the file as binary, already decompressed.

open_text(encoding=compression.DEFAULT_TEXT_ENCODING)

Open the file as text, already decompressed.

peek(size)

Read the first size decompressed bytes, then close.

Returns fewer bytes than asked for at end of file, and b"" if the file cannot be read at all.

croissant_baker.sources.PathSource dataclass

Bases: FileSource

A source that also exposes a real path, for handlers that need one.

Built only for uncompressed files, so path never points at a wrapper — an invariant :func:make_source enforces.

Handler registry

croissant_baker.handlers.registry.HandlerRegistry

The handlers to consult, in the order to consult them.

Order is registration order, and it decides overlapping claims.

__init__(handlers=None)

register(handler)

Add handler, unless one of its class is already registered.

Identity is the class, not the instance: callers construct a fresh instance each time, so deduplicating by instance would not deduplicate.

handlers()

Every registered handler, in dispatch order.

select(file_path, relative_path=None)

Find the handler that owns file_path, or say why none does.

Every handler is asked about the logical file through a plain :class:~croissant_baker.sources.FileSource. Only a winner declaring InputKind.PATH gets a :class:~croissant_baker.sources.PathSource, and only for an uncompressed file; a compressed one is refused with its own reason.

Parameters:

Name	Type	Description	Default
file_path	Path	Path to the file, wrapper suffix included.	required
relative_path	Optional[Path]	Its path relative to the dataset root, for the source.	None

croissant_baker.handlers.registry.builtin_handlers()

The handlers the baker ships with, in dispatch order.

croissant_baker.handlers.registry.HandlerSelection dataclass

The outcome of asking the registry who owns a file.

When nothing claimed it, reason is the category the scan summary counts and refusal is the sentence a human reads.

Compression

Adding a compression is one call: dispatch strips the new suffix, streams decompress through it, encodingFormat gains its media type, and FileSet globs expand to cover it.

croissant_baker.compression.Compression dataclass

One supported compression wrapper.

media_type accompanies the format's own media type rather than replacing it. opener has the signature of :func:gzip.open.

croissant_baker.compression.register_compression(comp)

Add a compression to the registry.

Everything downstream follows: dispatch strips the new suffix, streams decompress through it, encodingFormat gains its media type, and FileSet globs expand to cover it. No handler changes.

croissant_baker.compression.compressions()

Every registered compression, in match order.