ProofLayer: Control Coverage Screening for Government Cybersecurity Policies via Multi-LLM Orchestration

Abstract

ProofLayer is a two-stage system for screening government cybersecurity policy documents against control catalogues: a keyword stage that maps policy text to a twenty-two control set crosswalked to NIST SP 800-53 Revision 5, and a language-model stage intended to resolve the cases keyword matching cannot, with a cost-aware orchestration layer routing between models. This paper evaluates the first stage on ten Maryland state and local government policy documents and separates what is measured from what is only described. Reading each document in full gives a mean control coverage of 62.8 percent (standard deviation 24.1). An earlier analysis of this corpus truncated each document at fifteen pages and reported 49.1 percent; seven of the ten documents exceed fifteen pages, and the two longest gain 36 and 41 points when read in full. Both figures are reported, because a screening metric that moves this far with extraction depth is itself informative. Coverage correlates with document length at  against log word count, so 48 percent of its variance is explained by length alone. Of 138 covered document-control pairs, 18.1 percent carry a negation cue within 130 characters of the match and 44.2 percent rest on a single keyword out of a mean 6.1 available. A worked error analysis gives four cases, among them a footnote URL scored as transport encryption and an incident-reporting requirement carrying a two-business-day deadline scored as absent because the keyword list lacked the word notifying. The second-stage classifier is described but not evaluated: establishing whether it resolves these failures requires expert-annotated ground truth for this corpus, which does not yet exist, and model-generated labels are declined as a substitute. The orchestration cost saving is a token-price projection rather than instrumented spend. We therefore report policy-vocabulary coverage, not compliance, and release all 220 document-control pairs with matched keywords and text snippets.