Document hygiene at scale: finding stale, duplicated and over‑shared files

Description and use

Implemented a repeatable, low‑cost audit that identifies stale subtrees, expired sharing links, duplicates and personal‑drive data, enabling evidence‑based deletion and compliance with retention policies.

Problem

Microsoft 365 tenants accumulate large amounts of stale, duplicated and over‑shared files that are hard to locate, making it difficult to prove deletion of personal data or to assess attacker reach in a breach.

Outcome

Implemented a repeatable, low‑cost audit that identifies stale subtrees, expired sharing links, duplicates and personal‑drive data, enabling evidence‑based deletion and compliance with retention policies.

Workflow

• 1. Scheduled job walks every site and drive via Microsoft Graph delta queries, collecting metadata (path, size, owner, timestamps, sensitivity, retention labels, version count, permissions, sharing links).
• 2. Store each run as a JSON snapshot in SQLite with generated columns for indexing.
• 3. Run deterministic rules over the snapshot: exact duplicate hash, stale subtree (no change in 18 months, no retention label or legal hold), links with no expiry, external sharing on internal sites, orphaned drives.
• 4. Apply near‑duplicate detection using embeddings for files that differ in filename or date but are otherwise identical.
• 5. Run content classification on ambiguous files using a document parsing model and a classifier to detect personal info, record status, and business function.
• 6. Generate per‑owner report and proposed‑actions table.
• 7. Diff snapshots to track growth and prioritize budget.
• 8. Rollout: audit only for ≥1 month, then dry‑run deletes with ledger and owner digests, tune rules, finally enforce with reversible actions and final destruction record.
• 9. Maintain audit evidence, ledger, and retention compliance.
• 10. Review and sign‑off for sensitive records before execution.

Evidence

A scheduled job walks every site and drive through the Microsoft Graph, using delta queries so subsequent runs only pull what changed.

Source

Deterministic rules run over the snapshot first, because they are free. Exact duplicates by content hash, stale subtrees, links with no expiry, external sharing on sites marked internal, orphaned drives.

Source

Near‑duplicate detection uses embeddings rather than hashes. Two copies of the same contract that differ by a filename and a date field will never match on a hash, but they sit next to each other in vector space.

Source

The system reports and does nothing else. This is where you discover that the "stale" site is the one Finance opens each June, and that half the external links belong to an auditor who is still engaged.

Source