---------------------------------------------------------------------- This is the API documentation for the cane_format library. ---------------------------------------------------------------------- ## Parsing and formatting CANE Turn model output into character spans, and documents into CANE. loads(output: 'str', *, text: 'str', variant: 'Variant | str | None' = None, labels: 'Iterable[str] | None' = None, match: 'Match' = 'exact', anchor_fallback: 'bool' = False, strip_list_markers: 'bool' = False, handles: 'bool' = False, prefer_whole_words: 'bool' = False, dialect: 'Dialect' = Dialect(header='Entities:', empty='NONE', quote_open='`', quote_close='`', copula_singular='is', copula_plural='are', articles=('a', 'an'), anchor_intro='found in', part_connective='and', anchor_connective='and', record_separator='\n', handle_prefix='T'), options: 'ParseOptions | None' = None, id: 'str | None' = None) -> 'ParseResult' Parse CANE model output and ground every record in ``text``. Problems in the model output never raise: they are returned as ``result.issues``. ``NONE`` (optionally after the header) or the header alone means no entities; an empty output also gives no entities but is reported as ``empty_output``, because it usually means the generation failed. Args: output: The model's CANE output, exactly as received. text: The source text the model annotated. variant: The CANE variant the model was asked for; lines that break it are issues. labels: The allowed labels; an unknown label keeps the entity and adds an issue. match: ``"exact"``, ``"normalized"`` (Unicode NFC) or ``"casefold"``. Only this level is used. Whitespace runs always match any whitespace run. anchor_fallback: Ground the bare part when its anchor is not found. strip_list_markers: Remove leading list markers (``- ``, ``1. ``) from lines. handles: Records carry handles (``T1 `x` is a L``), which become entity IDs. prefer_whole_words: When a part could be placed at several occurrences, try those that neither start nor end inside a word (between two letters or digits) first, then the rest, each in reading order. Nothing is ever rejected. Matters mainly for Base and Disco, where no anchor picks the occurrence: in "against Northamptonshire at Northampton", `` `Northampton` `` then lands on the whole word. dialect: The surface vocabulary. options: All of the above as one ``ParseOptions``; excludes the other keywords. id: The ID of the resulting document (e.g. needed to write it as brat). dumps(doc: 'Document', *, variant: 'Variant | str', context: 'int' = 1, handles: 'bool' = False, on_unrepresentable: 'OnUnrepresentable' = 'raise', on_mismatch: 'OnMismatch' = 'warn', parse_options: 'ParseOptions | None' = None, lossy: "Literal['warn', 'ignore']" = 'warn', dialect: 'Dialect' = Dialect(header='Entities:', empty='NONE', quote_open='`', quote_close='`', copula_singular='is', copula_plural='are', articles=('a', 'an'), anchor_intro='found in', part_connective='and', anchor_connective='and', record_separator='\n', handle_prefix='T')) -> 'str' Write a document's entities as CANE text, e.g. for a few-shot example. Entities are written in reading order: by start, outer before nested, then by label and ID. Every record is parsed back as ``loads`` with ``parse_options`` would parse it. A record that would not ground on its own spans (a repeated string without an anchor, an anchor of ``context`` units that occurs more than once, a part that reads as two) is still written as is and handled by ``on_mismatch``. Two entities that would give the same record are written once, with the lossy warning. Args: doc: The document to write. variant: ``"base"``, ``"disco"``, ``"context"`` or ``"full"``. context: Anchor size in whitespace-separated units on each side (anchored variants). handles: Prefix each record with a handle ``T1``, ``T2``, … in output order. on_unrepresentable: What to do with an entity the variant cannot express (no label, a label that is not one token, or discontinuous in Base/Context): ``"raise"``, or ``"skip"``, ``"first"`` (first span only) or ``"hull"`` (one span covering the gaps), which warn. Entities with label problems are skipped by any of these. on_mismatch: A record that does not parse back on its own spans: ``"warn"`` (one ``LossyConversionWarning`` naming each), ``"raise"`` (``UnrepresentableError``), or ``"ignore"`` (the check is not run). parse_options: How the output will be parsed (``match``, ``prefer_whole_words``, ``anchor_fallback``, …); by default ``loads``' defaults. Variant and dialect must be the ones given here. lossy: ``"warn"`` once about information CANE cannot carry, or ``"ignore"``. dialect: The surface vocabulary. Raises: UnrepresentableError: An entity cannot be written and no strategy allows dropping it, or a record does not parse back and ``on_mismatch="raise"``. ParseOptions(*, variant: 'Variant', labels: 'frozenset[str] | None' = None, match: 'Match' = 'exact', anchor_fallback: 'bool' = False, strip_list_markers: 'bool' = False, handles: 'bool' = False, prefer_whole_words: 'bool' = False, dialect: 'Dialect' = Dialect(header='Entities:', empty='NONE', quote_open='`', quote_close='`', copula_singular='is', copula_plural='are', articles=('a', 'an'), anchor_intro='found in', part_connective='and', anchor_connective='and', record_separator='\n', handle_prefix='T')) -> None How model output is parsed. Every default is strict. Recorded on each ``ParseResult``. ParseResult(document: 'Document', issues: 'tuple[ParseIssue, ...]', settings: 'ParseOptions') -> None The grounded document, everything that could not be grounded, and the settings used. ParseIssue(*, kind: 'IssueKind', message: 'str', line: 'str | None' = None, line_no: 'int | None' = None, candidates: 'tuple[Span, ...]' = ()) -> None One problem found while parsing model output. Attributes: kind: What went wrong. message: Human-readable explanation. line: The raw output line, if the issue belongs to one. line_no: 1-based line number in the model output. candidates: Spans that were considered, for debugging grounding decisions. IssueKind(*values) Closed vocabulary of problems found while parsing model output. Variant(*values) The four CANE variants: one grammar, two switches (several parts, anchors). Dialect(*, header: 'str' = 'Entities:', empty: 'str' = 'NONE', quote_open: 'str' = '`', quote_close: 'str' = '`', copula_singular: 'str' = 'is', copula_plural: 'str' = 'are', articles: 'tuple[str, ...]' = ('a', 'an'), anchor_intro: 'str' = 'found in', part_connective: 'str' = 'and', anchor_connective: 'str' = 'and', record_separator: 'str' = '\n', handle_prefix: 'str' = 'T') -> None Every literal token of the CANE surface form. ``ENGLISH`` is the default. Attributes: header: First line of every non-empty output. empty: The whole output when there are no entities. quote_open: Delimiter opening a part or anchor. quote_close: Delimiter closing a part or anchor. copula_singular: Verb for one part (``is``). copula_plural: Verb for several parts (``are``). articles: Accepted articles; the formatter (`dumps`) always writes the first. anchor_intro: Phrase introducing the anchors (``found in``). part_connective: Joins parts (``and``). anchor_connective: Joins anchors (``and``). record_separator: Separates records. handle_prefix: Prefix of record handles (``T`` as in ``T1``). ENGLISH Every literal token of the CANE surface form. ``ENGLISH`` is the default. Attributes: header: First line of every non-empty output. empty: The whole output when there are no entities. quote_open: Delimiter opening a part or anchor. quote_close: Delimiter closing a part or anchor. copula_singular: Verb for one part (``is``). copula_plural: Verb for several parts (``are``). articles: Accepted articles; the formatter (`dumps`) always writes the first. anchor_intro: Phrase introducing the anchors (``found in``). part_connective: Joins parts (``and``). anchor_connective: Joins anchors (``and``). record_separator: Separates records. handle_prefix: Prefix of record handles (``T`` as in ``T1``). ## Prompts and evaluation Few-shot prompts in the CANE paper's layout, and spans for nervaluate-disco. build_prompt(text: 'str', *, examples: 'Sequence[Document]', variant: 'Variant | str', labels: 'Iterable[str] | None' = None, context: 'int' = 1, handles: 'bool' = False, on_unrepresentable: 'OnUnrepresentable' = 'raise', on_mismatch: 'OnMismatch' = 'warn', parse_options: 'ParseOptions | None' = None, lossy: "Literal['warn', 'ignore']" = 'warn', dialect: 'Dialect' = Dialect(header='Entities:', empty='NONE', quote_open='`', quote_close='`', copula_singular='is', copula_plural='are', articles=('a', 'an'), anchor_intro='found in', part_connective='and', anchor_connective='and', record_separator='\n', handle_prefix='T')) -> 'str' A few-shot prompt asking a model to annotate ``text`` in CANE. The layout follows the CANE paper: task, entity types, format and guideline, the 'Important' instruction, the worked examples, then the input. Examples are written with ``dumps`` and the same options, so the prompt teaches exactly what ``loads`` expects. Args: text: The text to annotate. examples: Annotated documents shown as few-shot examples. variant: The CANE variant. labels: The entity types in the order to list them (a set is sorted); by default the examples' labels, sorted. Required when there are no examples. context, handles, on_unrepresentable, on_mismatch, parse_options, lossy, dialect: As for ``dumps``. guideline(variant: 'Variant | str', *, context: 'int' = 1, handles: 'bool' = False, dialect: 'Dialect' = Dialect(header='Entities:', empty='NONE', quote_open='`', quote_close='`', copula_singular='is', copula_plural='are', articles=('a', 'an'), anchor_intro='found in', part_connective='and', anchor_connective='and', record_separator='\n', handle_prefix='T')) -> 'str' The format instructions for one CANE variant, worded with the dialect's tokens. to_nervaluate(docs: 'Iterable[Document]', *, unlabelled: "Literal['raise', 'skip']" = 'raise') -> 'list[dict[str, Any]]' Documents in nervaluate-disco's span format, one dict per document. Pass gold and predicted documents in the same order:: from nervaluate import Evaluator true = cane.to_nervaluate(gold_docs) pred = cane.to_nervaluate(result.document for result in results) Evaluator(true, pred, tags=labels).evaluate() Args: docs: The documents. unlabelled: Entities without a label cannot be evaluated: ``"raise"`` or ``"skip"``. ## Document model Text with entities, relations and segments; immutable and validated. Document(*, text: 'str', entities: 'Iterable[Entity]' = (), relations: 'Iterable[Relation]' = (), segments: 'Iterable[Segment]' = (), id: 'str | None' = None, metadata: 'Mapping[str, Any] | None' = None) -> 'None' A text with its annotations. On construction, missing entity IDs become ``T`` and missing relation IDs ``R`` (the next free numbers), and the document is validated: every span and segment lies inside the text, IDs are unique, and every member and trigger refers to something that exists. Entity(*, spans: 'Iterable[tuple[int, int]]', label: 'str | None' = None, id: 'str | None' = None, concepts: 'Iterable[Concept]' = (), attributes: 'Mapping[str, str | bool] | None' = None, metadata: 'Mapping[str, Any] | None' = None) -> 'None' A labelled annotation over one or more spans (more than one = discontinuous). Equality and hashing use ``(label, spans)`` only: two entities over the same characters with the same label are the same entity, whatever their IDs, concepts or attributes. Span(start: 'int', end: 'int') -> 'Span' A contiguous character range ``text[start:end]``. Requires ``0 <= start < end``. Relation(*, type: 'str', members: 'Iterable[Member | tuple[str] | tuple[str, str | None]]' = (), trigger: 'str | None' = None, id: 'str | None' = None, attributes: 'Mapping[str, str | bool] | None' = None, metadata: 'Mapping[str, Any] | None' = None) -> 'None' A relation, event or equivalence set over entities and relations. Attributes: type: The relation or event type. members: The participants. Their order is kept but not significant for equality. trigger: ID of the entity whose text expresses the relation (event trigger, verb). id: Stable identifier; filled in by ``Document`` when missing. Equality uses type, trigger and members as a multiset. Symmetry needs no flag: members without distinguishing roles are interchangeable. Member(refid: ForwardRef('str'), role: ForwardRef('str | None') = None) A participant in a relation: the ID of an entity or relation, and its role. ``role=None`` marks an unordered participant (e.g. in an equivalence set). Concept(*, id: 'str', source: 'str | None' = None, name: 'str | None' = None) -> None A link from an entity to a concept in a vocabulary. Attributes: id: The identifier exactly as the source wrote it (``"MESH:D008687"``); never parsed. source: The vocabulary, when the source names it separately. name: The concept's preferred name, when the source gives one. Segment(*, kind: 'str', span: 'tuple[int, int]', id: 'str | None' = None, metadata: 'Mapping[str, Any] | None' = None) -> 'None' A typed region of the text, such as a passage, sentence or token. ## brat standoff read_brat(path: 'str | PathLike[str]', *, verify: 'Verify' = 'raise', offset_unit: 'OffsetUnit' = 'codepoint') -> 'list[Document]' Read a brat directory, or one document given its ``.txt`` or ``.ann`` path. iter_brat(path: 'str | PathLike[str]', *, verify: 'Verify' = 'raise', offset_unit: 'OffsetUnit' = 'codepoint') -> 'Iterator[Document]' Like ``read_brat``, yielding documents one at a time (sorted by file name). parse_brat(text: 'str', ann: 'str', *, id: 'str | None' = None, verify: 'Verify' = 'raise', offset_unit: 'OffsetUnit' = 'codepoint') -> 'Document' Build a document from the contents of a ``.txt`` and an ``.ann`` file. Args: text: The ``.txt`` content, unchanged (read with ``newline=''``). ann: The ``.ann`` content. id: The document ID, usually the file stem. verify: Check every text-bound annotation against its stored text: ``"raise"`` an ``IntegrityError`` on mismatch, ``"warn"``, or ``"off"``. offset_unit: The unit the offsets count in. write_brat(docs: 'Iterable[Document]', directory: 'str | PathLike[str]', *, offset_unit: 'OffsetUnit' = 'codepoint', lossy: "Literal['warn', 'ignore']" = 'warn') -> 'None' Write each document as ``.txt`` and ``.ann`` into ``directory``. Every file name is checked and every document rendered before anything is written, so an error leaves no partial output. format_brat(doc: 'Document', *, offset_unit: 'OffsetUnit' = 'codepoint', lossy: "Literal['warn', 'ignore']" = 'warn') -> 'str' The ``.ann`` content for a document. The text is written unchanged as the ``.txt``. Relations map to line kinds by structure: a trigger makes an ``E`` line, members without roles a ``*`` line, two members with roles an ``R`` line. IDs that do not fit brat's ``T``/``E``/``R`` numbering are renumbered consistently. ## BioC JSON read_bioc_json(path: 'str | PathLike[str]', *, verify: 'Verify' = 'raise', offset_unit: 'OffsetUnit' = 'utf8', label_key: 'str' = 'type', concept_keys: 'Sequence[str]' = ('identifier',)) -> 'list[Document]' Read a BioC JSON file. See ``parse_bioc_json``. iter_bioc_json(path: 'str | PathLike[str]', *, verify: 'Verify' = 'raise', offset_unit: 'OffsetUnit' = 'utf8', label_key: 'str' = 'type', concept_keys: 'Sequence[str]' = ('identifier',)) -> 'Iterator[Document]' Like ``read_bioc_json``, yielding documents one at a time. parse_bioc_json(content: 'str', *, verify: 'Verify' = 'raise', offset_unit: 'OffsetUnit' = 'utf8', label_key: 'str' = 'type', concept_keys: 'Sequence[str]' = ('identifier',)) -> 'list[Document]' Build documents from BioC JSON: a collection, a list of collections, or a PubTator3 export. Args: content: The JSON text. verify: Check every annotation's stored text against its offsets: ``"raise"`` an ``IntegrityError`` on mismatch, ``"warn"``, or ``"off"``. offset_unit: The unit offsets count in. The BioC specification says UTF-8 bytes. label_key: The annotation infon holding the label. concept_keys: Annotation infons holding concept identifiers (kept verbatim). write_bioc_json(docs: 'Iterable[Document]', destination: 'Destination', **options: 'Any') -> 'None' Write documents as one BioC JSON collection. Options as for ``format_bioc_json``. format_bioc_json(docs: 'Iterable[Document]', *, offset_unit: 'OffsetUnit' = 'utf8', label_key: 'str' = 'type', concept_key: 'str' = 'identifier', indent: 'int | None' = None, lossy: "Literal['warn', 'ignore']" = 'warn') -> 'str' Documents as one BioC JSON collection. Annotations and relations go back where they were read from (``bioc_level``); everything else is placed in the passage containing its start, or at document level. Args: docs: The documents. offset_unit: The unit to write offsets in (UTF-8 bytes per the specification). label_key: The annotation infon for the label. concept_key: The infon for concepts that did not come from BioC. indent: JSON indentation; one line when None. lossy: ``"warn"`` once about information BioC cannot carry, or ``"ignore"``. ## CoNLL / IOB read_conll(path: 'str | PathLike[str]', *, scheme: 'Scheme' = 'bio', repair: 'bool' = False, text: 'str | None' = None, encoding: 'str' = 'utf-8', comment_prefix: 'str | None' = None) -> 'list[Document]' Read a CoNLL file. See ``parse_conll``; ``encoding`` is the file's text encoding. iter_conll(path: 'str | PathLike[str]', *, scheme: 'Scheme' = 'bio', repair: 'bool' = False, text: 'str | None' = None, encoding: 'str' = 'utf-8', comment_prefix: 'str | None' = None) -> 'Iterator[Document]' Like ``read_conll``, yielding documents one at a time. parse_conll(content: 'str', *, scheme: 'Scheme' = 'bio', repair: 'bool' = False, text: 'str | None' = None, comment_prefix: 'str | None' = None) -> 'list[Document]' Build documents from CoNLL content. Args: content: The file content. scheme: The tag scheme: ``"bio"`` (IOB2), ``"iob1"``, ``"bioes"`` or ``"bilou"``. repair: Accept invalid tag sequences (e.g. ``I-X`` after ``O``) by starting a new entity there, instead of raising. text: The true raw text, if known. Tokens are aligned onto it (only whitespace may lie between them) and offsets refer to it. Only for content with exactly one document. comment_prefix: Treat lines starting with this prefix (e.g. ``"#"``) as comments. A comment ``doc_id = X`` or ``newdoc id = X`` starts a new document with ID ``X``; other comments are kept in ``metadata["conll_comments"]``. Off by default, since ``#`` can also be a token. write_conll(docs: 'Iterable[Document]', destination: 'Destination', **options: 'Any') -> 'None' Write documents to one CoNLL file or stream. Options as for ``format_conll``. format_conll(doc: 'Document', *, scheme: 'Scheme' = 'bio', tokenizer: 'Tokenizer' = , alignment: 'Alignment' = 'strict', on_conflict: 'OnConflict' = 'raise', on_unrepresentable: 'OnUnrepresentable' = 'raise', lossy: "Literal['warn', 'ignore']" = 'warn') -> 'str' One document as CoNLL lines (ending with a blank line). Tokens and sentences come from the document's token and sentence segments when present; otherwise ``tokenizer`` splits the text and the document is one sentence. Args: doc: The document. scheme: The tag scheme to write. tokenizer: ``text -> (start, end)`` token spans, used without token segments. alignment: An entity boundary inside a token: ``"strict"`` raises, ``"expand"`` takes the whole token, ``"contract"`` drops the partial token (both warn). on_conflict: Entities sharing a token: ``"raise"``, keep the ``"outermost"`` or ``"innermost"``, or ``"skip"`` every entity involved (all but raise warn). on_unrepresentable: Discontinuous, unlabelled or sentence-crossing entities: ``"raise"``, ``"first"`` span, ``"hull"``, ``"split"`` into one entity per span, or ``"skip"`` (all but raise warn; entities with label problems are skipped). lossy: ``"warn"`` once about information CoNLL cannot carry, or ``"ignore"``. from_tokens(tokens: 'Sequence[str]', tags: 'Sequence[str | int]', *, scheme: 'Scheme' = 'bio', label_names: 'Sequence[str] | None' = None, repair: 'bool' = False, id: 'str | None' = None) -> 'Document' One sentence given as tokens and tags, e.g. a Hugging Face ``tokens``/``ner_tags`` row. Integer tags are looked up in ``label_names``. align(doc: 'Document', text: 'str') -> 'Document' Move a CoNLL document's annotations onto its true raw text. The tokens must appear in ``text`` in order, separated only by whitespace. ## Inline XML read_xml(path: 'str | PathLike[str]', **options: 'Any') -> 'list[Document]' Read an XML file. Options as for ``parse_xml``. iter_xml(path: 'str | PathLike[str]', **options: 'Any') -> 'Iterator[Document]' Like ``read_xml``, yielding documents one at a time. parse_xml(content: 'str', *, tag: 'str | None' = None, type_attr: 'str' = 'type', ref_attr: 'str' = 'ref', document_tag: 'str | None' = None, segment_tags: 'Sequence[str]' = (), exclude_tags: 'Sequence[str]' = ()) -> 'list[Document]' Build documents from inline XML. Content without a single root element (an inline fragment, as a model would write it) is one document. A ```` root, as ``write_xml`` writes, holds ````s. Args: content: The XML text. tag: The annotation element name; None means every element is an annotation named by its label. type_attr: With ``tag``, the attribute holding the label. ref_attr: The attribute linking the elements of one entity (space-separated values put one element into several entities). document_tag: Each element with this name is one document (e.g. ``"article"``). segment_tags: Structural elements to keep as segments of that kind (e.g. sentences). exclude_tags: Structural elements whose text is not part of the document text; their text is kept in ``metadata[name]``. write_xml(docs: 'Iterable[Document]', destination: 'Destination', **options: 'Any') -> 'None' Write ``……`` to a path or stream. format_xml(doc: 'Document', *, tag: 'str | None' = None, type_attr: 'str' = 'type', ref_attr: 'str' = 'ref', lossy: "Literal['warn', 'ignore']" = 'warn') -> 'str' The document text with its entities as inline elements. Raises: UnrepresentableError: In the default style, a label that is not a valid XML element name (e.g. ``protein_N/A``) or a missing label; pass ``tag=`` to write any label. ## Offset JSON read_json(path: 'str | PathLike[str]') -> 'list[Document]' Read a JSON Lines file, a single JSON document, or a JSON array of documents. iter_json(path: 'str | PathLike[str]') -> 'Iterator[Document]' Like ``read_json``, yielding documents one at a time. parse_json(content: 'str') -> 'Document' Build a document from one JSON object. write_json(docs: 'Iterable[Document]', destination: 'Destination') -> 'None' Write documents as JSON Lines, one per line, to a path or an open text stream. format_json(doc: 'Document', *, indent: 'int | None' = None) -> 'str' One document as a JSON object (on one line unless ``indent`` is given). ## Errors and warnings CaneFormatError Base class of every exception raised by cane-format. ParseError(issues: 'tuple[ParseIssue, ...]') -> 'None' Raised by ``ParseResult.raise_for_errors()``; carries every issue of the parse. IntegrityError A file's offsets do not reproduce the surface text it stores next to them. UnrepresentableError A writer or the CANE formatter was asked to write something its target cannot hold. LossyConversionWarning Information was dropped because the target format cannot hold it. ---------------------------------------------------------------------- This is the User Guide documentation for the package. ---------------------------------------------------------------------- ## Get started ### Installation `cane-format` needs Python 3.10 or newer and has no dependencies. ```bash pip install cane-format ``` With [uv](https://docs.astral.sh/uv/): ```bash uv add cane-format ``` Import it under its short name: ```python import cane_format as cane ``` The command line tool `cane-format` is installed with the package: ```bash cane-format --help ``` :::{.callout-note} ## Versions `cane-format` follows [Semantic Versioning](https://semver.org/). While the version starts with `0.`, the API may still change between minor releases; every change is listed in the [changelog](https://github.com/coree/cane-format/blob/main/CHANGELOG.md). ::: ### Quickstart This page walks through the whole loop: write annotated examples as CANE, build a few-shot prompt, and parse what the model answers back into character spans. ## An annotated document A `Document` is a text with entities. Each entity has one or more character spans and a label: ```{python} import cane_format as cane text = "After starting amoxicillin, the patient developed a rash." doc = cane.Document( text=text, entities=( cane.Entity(spans=[(15, 26)], label="DRUG"), cane.Entity(spans=[(52, 56)], label="ADR"), ), ) [(e.label, text[e.start:e.end]) for e in doc.entities] ``` In practice you read documents from a corpus instead, for example with `cane.read_brat("corpus/train")`. ## Write it as CANE `dumps` writes the document in one of the four [variants](03-cane-format.qmd). Full adds an anchor, a short verbatim snippet that tells the parser where the entity is: ```{python} print(cane.dumps(doc, variant="full")) ``` ## Build a prompt `build_prompt` puts the task, the entity types, the guidelines and your examples in the layout of the CANE paper, followed by the text to annotate: ```{python} new_text = "Ibuprofen gave her a headache." prompt = cane.build_prompt(new_text, examples=[doc], variant="full") print(prompt[prompt.index("Examples:"):]) # the part after the guidelines ``` Send the prompt to any model; this library makes no model calls. ## Parse the answer Suppose the model answers like this: ```{python} output = """Entities: `Ibuprofen` is a DRUG found in `Ibuprofen gave` `headache` is a ADR found in `a headache.` `nausea` is a ADR found in `and nausea`""" result = cane.loads(output, text=new_text, variant="full", labels={"DRUG", "ADR"}) [(e.label, e.spans) for e in result.entities] ``` Problems never raise. They are collected in `result.issues`, one per line that could not be grounded: ```{python} for issue in result.issues: print(issue) ``` `result.ok` is `False` whenever there is an issue; `result.raise_for_errors()` turns them into one exception if you prefer that. ## Save the result Write the parsed document in any supported format, for example as brat to inspect it in an annotation tool: ```python result = cane.loads(output, text=new_text, variant="full", id="doc-17") cane.write_brat([result.document], "predictions/") ``` Next: [the CANE format](03-cane-format.qmd), then [parsing](04-parsing.qmd) in detail. ## Concepts ### The CANE format CANE (Context-Anchored Natural Expression) is an output format for named entity recognition with large language models, introduced in [*CANE is a Format*](https://aclanthology.org/2026.findings-emnlp.1398/) (Findings of EMNLP 2026). Instead of character offsets, per-token tags or a marked-up copy of the input, the model writes one short statement per entity, copied verbatim from the text: ``` Entities: `metastatic breast cancer` is a DISEASE found in `metastatic breast cancer cells` `breast cancer cells` is a CELL found in `metastatic breast cancer cells caused` `headache` and `nausea` are a ADR found in `caused headache and` and `and nausea` ``` Models are good at copying visible text and bad at counting, so CANE never asks for offsets, token numbers or codes. This library turns those statements back into character spans. ## A record, piece by piece Each line is a **record** and describes one entity. The last line above, taken apart: | Piece | In the example | Meaning | |---|---|---| | parts | `` `headache` ``, `` `nausea` `` | The entity's text, copied verbatim. Several parts make one discontinuous entity. | | part connective | `and` (between the parts) | Joins the parts. | | copula | `are` | `is` or `are`. | | article | `a` | Optional. | | label | `ADR` | The entity type: one token. | | anchor intro | `found in` | Starts the anchors. | | anchors | `` `caused headache and` ``, `` `and nausea` `` | One per part, in the same order: the part with some text around it, copied verbatim, saying *where* the part is. | | anchor connective | `and` (between the anchors) | Joins the anchors. | Anchors matter when a string occurs more than once: in a text that mentions aspirin twice, `` `aspirin` is a DRUG `` could mean either, while `` `aspirin` is a DRUG found in `take aspirin again` `` means one. ## The four variants | Variant | Parts per record | Anchors | |---|---|---| | **Base** | exactly one | none | | **Disco** | one or more | none | | **Context** | exactly one | exactly one | | **Full** | one or more | one per part, in the same order | The variant is always given explicitly, when writing and when parsing: a line with an anchor is required in Full but wrong in Base, so the parser must know what the model was asked for. The same entity in two variants: ```{python} import cane_format as cane text = "It caused headache and nausea after the first dose." doc = cane.Document( text=text, entities=(cane.Entity(spans=[(10, 18), (23, 29)], label="ADR"),), ) for variant in ("disco", "full"): print(f"--- {variant}") print(cane.dumps(doc, variant=variant)) ``` ## Grammar ``` output ::= HEADER | [HEADER NEWLINE] ( EMPTY | record { NEWLINE record } ) record ::= [HANDLE] parts COPULA [ARTICLE] LABEL [ANCHOR_INTRO anchors] parts ::= QUOTE text QUOTE { PART_CONNECTIVE QUOTE text QUOTE } anchors ::= QUOTE text QUOTE { ANCHOR_CONNECTIVE QUOTE text QUOTE } ``` `[ … ]` is optional, `{ … }` repeats zero or more times. The symbols and their default spelling (every one can be changed with a [dialect](#dialects)): | Symbol | Default | `Dialect` field | |---|---|---| | `HEADER` | `Entities:` | `header` | | `EMPTY` | `NONE` | `empty` | | `QUOTE` | `` ` `` (backtick) | `quote_open`, `quote_close` | | `COPULA` | `is`, `are` | `copula_singular`, `copula_plural` | | `ARTICLE` | `a`, `an` | `articles` | | `ANCHOR_INTRO` | `found in` | `anchor_intro` | | `PART_CONNECTIVE` | `and` | `part_connective` | | `ANCHOR_CONNECTIVE` | `and` | `anchor_connective` | | `HANDLE` | `T1`, `T2`, … | `handle_prefix` | | `NEWLINE` | a line break | `record_separator` | | `LABEL` | any token without whitespace | | | `text` | any text | | The grammar allows anchors and several parts everywhere; the variant then restricts them as in the table above. ## Rules for a whole output - **One record per line.** Blank lines and whitespace at the start or end of a line are ignored. - **The header is optional when parsing.** If present, it must be the first line, spelled exactly (`entities:` is not the header). `dumps` always writes it. - **No entities** is written as `NONE` (in any case) on its own or after the header, or as the header alone. `Entities: NONE` on a single line is not one of these and is reported. - **An empty output** gives no entities too, but is reported as `empty_output`: it usually means the generation failed. - **A bad line never stops parsing.** It is reported, and the next line is read as usual. ## Rules for a record - **Parts and anchors are verbatim text.** Whitespace inside them matches any run of whitespace in the text, so `` `caused headache` `` still matches *caused headache*. - **`is` and `are` are both accepted**, whatever the number of parts. `dumps` writes `is` for one part and `are` for several. - **The article is optional**, and any article of the dialect is accepted (`is an ADR`, `is ADR`). `dumps` writes the first one, `a`. - **The label is one token**: everything up to the next whitespace, punctuation included (`is a ADR.` gives the label `ADR.`). Labels are checked only if you pass `labels=`. - **An anchor must contain its part.** The anchor is found in the text first, then the part inside it. - **Handles** (`T1 `, `T2 ` before the parts) are accepted only with `handles=True`, and then every record needs one. The handle becomes the entity's ID; a handle used twice is reported. - **There is no escaping.** A part may contain the quote character; records are split only at a closing quote, connective and opening quote (`` ` and ` ``), so only that exact sequence cannot appear inside a part. ## What a wrong line looks like Each problem becomes a `ParseIssue` with the line number and a reason, and the other lines are still read: ```{python} text = "Aspirin relieved the headache and the nausea." output = """Entities: `Aspirin` is a DRUG found in `Aspirin relieved` `headache` is a ADR `headache` and `nausea` are a ADR found in `the headache and` `fever` is a ADR found in `the fever` Aspirin is a DRUG found in `Aspirin relieved`""" result = cane.loads(output, text=text, variant="full") print([(e.label, text[e.start:e.end]) for e in result.entities]) for issue in result.issues: print(issue) ``` The [parsing guide](04-parsing.qmd) lists every issue kind and explains how a record is placed when its text occurs more than once. ## Dialects Every literal (`Entities:`, `NONE`, the backtick, `is` / `are`, `a` / `an`, `found in`, `and`, the handle prefix `T`) comes from a `Dialect`. The default is `cane.ENGLISH`; another language or other delimiters are one object away: ```{python} german = cane.Dialect( header="Entitäten:", empty="KEINE", copula_singular="ist", copula_plural="sind", articles=("ein", "eine"), anchor_intro="gefunden in", part_connective="und", anchor_connective="und", ) print(cane.dumps(doc, variant="full", dialect=german)) ``` Pass the same dialect to `loads` to parse such output. ## Guides ### Parsing model output `cane.loads(output, text=..., variant=...)` reads a model's CANE output and finds each record's place in the text. It never raises for problems in the output: everything that could not be grounded comes back as a `ParseIssue`. ```{python} import cane_format as cane text = "Take aspirin with food. If pain persists, take aspirin again." output = "Entities:\n`aspirin` is a DRUG\n`aspirin` is a DRUG\n`fever` is a ADR" result = cane.loads(output, text=text, variant="base", labels={"DRUG", "ADR"}) [(e.label, e.spans) for e in result.entities], [str(i) for i in result.issues] ``` ## Grounding: where a record lands The parser grounds what the model wrote; it never judges whether a prediction is plausible. - **Reading order.** Each record takes the earliest unused occurrence at or after the previous record's position, else the earliest unused one anywhere. Repeated mentions and nested entities land where a reader expects them. (On the CANE paper's model outputs this rule beat "earliest occurrence anywhere" in every case, including outputs grouped by label.) - **Anchors** select the occurrence when there are several. If an anchor is not in the text, or does not contain its part, the record is reported as `anchor_not_found`. - **Overlap never rejects.** Nested and overlapping entities are kept. A record that would repeat an earlier entity exactly is reported as `duplicate`. - **Discontinuous parts** are grounded together: all parts or nothing, and parts may not overlap each other. ## Strict by default Every leniency is an explicit switch, off by default, and the parser never tries a looser setting on its own: | Option | Effect | |---|---| | `labels=` | Unknown labels keep the entity and add an `invalid_label` issue. | | `match="normalized"` | Match after Unicode NFC normalisation. | | `match="casefold"` | Match ignoring case. | | `anchor_fallback=True` | Ground the bare part when its anchor is not found. | | `strip_list_markers=True` | Accept lines starting with `- ` or `1. `. | | `handles=True` | Records carry handles (`T1 …`), which become entity IDs. | | `prefer_whole_words=True` | See below. | All of them can be bundled in one `cane.ParseOptions`, which is also recorded on every result as `result.settings`. ## Choosing an occurrence Without anchors (Base, Disco), only the reading order decides, and the earliest occurrence can be inside a longer word: ```{python} text = "Maurice Tate against Northamptonshire at Northampton" record = "`Northampton` is a LOC" for prefer in (False, True): e = cane.loads(record, text=text, variant="base", prefer_whole_words=prefer).entities[0] print(f"prefer_whole_words={prefer}:", text[: e.start] + "[" + text[e.start : e.end] + "]" + text[e.end :]) ``` `prefer_whole_words=True` tries occurrences that neither start nor end between two letters or digits first. It only reorders candidates: an entity that occurs only inside a word is still found, and in scripts written without spaces nothing changes. ## Issues | Kind | Meaning | |---|---| | `empty_output` | The output is empty: the generation probably failed. | | `malformed_line` | The line is not a record of this variant. | | `not_found` | A part does not occur in the text. | | `anchor_not_found` | An anchor does not occur in the text, or does not contain its part. | | `part_count_mismatch` | The number of anchors differs from the number of parts. | | `unknown_handle` | A handle does not refer to an entity. | | `invalid_label` | The label is not in `labels=`. | | `invalid_span` | The parts cannot form a valid entity. | | `duplicate` | The record repeats an earlier entity exactly. | Each issue carries the raw line, its line number and the candidate spans that were considered, so you can see why a line failed. ### Writing CANE `cane.dumps(doc, variant=...)` writes a document's entities as CANE, typically for the examples in a few-shot prompt. - Records come in **reading order**: left to right, outer before nested. - Anchors have exactly `context` whitespace units on each side (default 1). - The header is always written, and `NONE` when there are no entities. ## Every record is checked `dumps` parses each record back as `loads` would. A record that would not land on its own spans, for example a string that also occurs earlier unannotated, is still written, but it is reported, so a few-shot example never misleads silently: ```{python} import warnings import cane_format as cane text = "take aspirin daily; then take aspirin daily" doc = cane.Document(text=text, entities=(cane.Entity(spans=[(30, 37)], label="DRUG"),)) with warnings.catch_warnings(record=True) as caught: warnings.simplefilter("always") print(cane.dumps(doc, variant="context")) print(caught[0].message) print(cane.dumps(doc, variant="context", context=2)) ``` | Option | Values | |---|---| | `on_mismatch` | `"warn"` (default), `"raise"`, or `"ignore"` (skip the check) | | `parse_options` | The `ParseOptions` you will parse with, e.g. `match="casefold"` | ## What a variant cannot express A discontinuous entity in Base or Context, an entity without a label, or a label with spaces cannot be written. By default `dumps` raises `UnrepresentableError`; `on_unrepresentable` chooses another strategy, each with a warning: | `on_unrepresentable` | Effect | |---|---| | `"raise"` | Raise `UnrepresentableError` (default). | | `"skip"` | Leave the entity out. | | `"first"` | Write only the first span of a discontinuous entity. | | `"hull"` | Write one span covering all parts and the gaps. | Relations, concepts, attributes and segments have no place in CANE; `dumps` says so once in a `LossyConversionWarning` (`lossy="ignore"` silences it). ## Prompts `build_prompt` writes the examples with `dumps` and the same options, so the prompt teaches exactly what `loads` expects: ```python prompt = cane.build_prompt( new_text, examples=train_docs[:5], variant="full", labels=["DRUG", "ADR"], ) ``` `cane.guideline(variant)` returns just the guideline paragraph, if you build prompts yourself. ### File formats Every format has the same five functions: `read_*` (a file or folder, returns a list of documents), `iter_*` (the same, one document at a time), `parse_*` (from a string), `write_*` and `format_*` (to a file or a string). | Format | Functions | Holds | |---|---|---| | brat standoff | `read_brat` … `format_brat` | entities, relations, events, attributes, normalisations, notes | | BioC JSON | `read_bioc_json` … | passages, entities, relations, concepts | | CoNLL / IOB | `read_conll` … plus `from_tokens`, `align` | tokens, sentences, flat entities | | Inline XML | `read_xml` … | entities as elements, refs for discontinuous ones | | Offset JSON | `read_json` … | everything, losslessly | ## Guarantees - **Text is never changed.** Files that carry text keep it character for character, including `\r\n`, tabs and trailing whitespace. - **Offsets are verified** against the surface text a file stores next to them (brat's text column, BioC's annotation text). A mismatch raises `IntegrityError`, with a hint when the offsets would fit another unit; `verify="warn"` or `"off"` relaxes this. - **Nothing is lost silently.** What a format cannot hold raises `UnrepresentableError`, or, with an explicit strategy, is dropped with a `LossyConversionWarning`. ## Format notes :::{.callout-warning} ## BioC offsets are UTF-8 bytes BioC JSON counts offsets in UTF-8 bytes, as the BioC specification and NCBI's data do. The reader and writer default to `offset_unit="utf8"`; all other formats count code points. ::: **brat.** `read_brat` takes a folder of `.txt` / `.ann` pairs (or one file); `T`, `R`, `E`, `*`, `A`/`M`, `N` and `#` lines are all read and written back identically. **CoNLL / IOB.** Schemes `bio`, `iob1`, `bioes` and `bilou`. CoNLL files carry tokens, not text, so the text is rebuilt (tokens joined by spaces, sentences by newlines); `read_conll(..., text=raw)` aligns the tokens onto the real text instead. `from_tokens` accepts Hugging Face style tokens and tags. Writing needs a decision for overlapping or nested entities (`on_conflict`). **Inline XML.** Labels become element names (`…`), or a generic element with a type attribute (`tag="entity"`). Discontinuous and crossing entities are written with refs. The reader also takes corpora with structural markup, such as GENIA (`document_tag`, `segment_tags`, `exclude_tags`). **Offset JSON.** The library's own lossless format, JSON Lines for collections. Use it to store documents between steps of a pipeline. ### Command line The `cane-format` command has two subcommands. ## `convert` Convert annotated documents between any two formats, including CANE: ```bash cane-format convert corpus/ train.conll --from brat --to conll --on-conflict outermost cane-format convert train.jsonl examples/ --from json --to cane-full ``` `--from` is one of `brat`, `json`, `conll`, `bioc`, `xml`; `--to` also accepts `cane-base`, `cane-disco`, `cane-context` and `cane-full` (a folder with one `.cane` file per document). The reading and writing options mirror the library's keyword arguments, for example `--offset-unit`, `--scheme`, `--on-unrepresentable` and `--context`. When writing CANE, `--on-mismatch` controls the parse-back check, and `--match`, `--anchor-fallback` and `--prefer-whole-words` give the settings the output will be parsed with. ## `parse` Parse a model's CANE output for a text and write the result in any format: ```bash cane-format parse output.cane --text doc.txt --variant full --to brat -o predictions/ ``` Issues are printed to standard error, one per line, with their line number. ## Exit codes | Code | Meaning | |---|---| | `0` | Success. | | `1` | `parse` only: the output has issues (unless `--allow-issues`). | | `2` | Any error, such as an unreadable file or an entity the target cannot hold. | Run `cane-format convert --help` or `cane-format parse --help` for every option.