Redact a large document /

A single call accepts about 10 MB of text, so most documents go through whole. Larger ones — a contract bundle, a year of exported tickets — have to be split, and splitting quietly costs accuracy in a way that is easy to miss.

Country inference reads signals out of the text: a postcode format, a phone prefix, the language of an address. A chunk that carries no such signal of its own — a page of bare reference numbers between two pages of Dutch addresses — is scored with nothing to go on, where the same numbers in the whole document would have been attributed confidently. The spans are still found; the attribution is what degrades.

from euredact import DocumentContext, redact

ctx = DocumentContext()
offset = 0

for chunk in chunks:
    result = redact(chunk, context=ctx, chunk_offset=offset)
    offset += len(chunk)
    print(result.redacted_text)

# Every signal the document produced, with spans pointing into the full text.
print(ctx.evidence())

The context accumulates the evidence each chunk contributes and scores every chunk against the document as a whole, so the page of bare numbers inherits what the rest of the document established.

chunk_offset is where this chunk starts in the whole document. Leave it out and every evidence span points into the first few hundred characters of the text, because each chunk reports offsets relative to itself. The redacted output is unaffected — this shows up later, when someone tries to highlight the evidence in the original and finds it all stacked at the top.

Detections returned from a call are always relative to the text you passed in. chunk_offset only rebases what is recorded on the context.

Caching goes off. With a context in use the result depends on the document rather than on the chunk alone, so an identical chunk cannot reuse an earlier answer. Expect the throughput of a cold run.

One context per document. Reusing a context across unrelated documents mixes their countries. That cannot cause a miss — a context influences scoring only, never which spans are found — but it can attribute a value to the wrong national scheme, which is a subtler kind of wrong. Build a fresh DocumentContext per document, and let it go when the document is done.

Cut on a boundary the text already has — a page, a section, a message — rather than a fixed character count. A split through the middle of an IBAN leaves two fragments, and neither half matches a pattern: the context cannot recover a span that was never found.

If your source has no natural boundary, overlap the chunks slightly and drop duplicate detections by span, rather than trusting a hard cut to land between identifiers.