work / policy — july 2023
The detector was never the hard part.
A policy paper on AI-driven phishing, written with two co-authors. Almost every proposal in this space stops at “build a better classifier,” which is the part that already works. We spent the paper on the question nobody wants: once the model exists, who maintains it, who feeds it, who pays for it, and who gets to decide what counts as an attack?
co-authored, 3 authors · 6 pages · runner-up, watgov hackathon (university of waterloo)
delivered · runner-up · not adopted
the problem
The specific target was Business Email Compromise — the attack where somebody who sounds exactly like your CFO asks you to move money, and you do. The reason it was worth writing about in mid-2023 is that generative models had just removed its last reliable tell. Bad grammar and stilted phrasing were, for years, the free signal that let a distracted human catch a fake. That signal is gone and it is not coming back.
The paper leads on cited industry and government reporting — an 83% rise in these attacks across 2022, billions in single-year losses, and the case where two of the largest technology companies on earth were taken for roughly $121m by a fake supplier with convincing invoices. Those are their numbers, from FBI/IC3 and Verizon breach reporting, quoted as context. We measured none of them, and the paper said so.
The Google and Facebook case is the one that reframes the problem. Those are not under-resourced companies with bad email filters. They have the best security teams money can hire, and the attack still worked — because it did not attack the filter. It attacked a person who had every reason to believe the invoice.
why the existing answer fails
Policy in this space had been overwhelmingly criminalisation-driven: make the act illegal, prosecute the actor. That approach has a structural dependency it cannot satisfy — it needs to know who the actor is. Attribution across proxies and jurisdictions is unreliable at best, and the enforcement body typically finds out an attack happened because a victim filed a complaint. A regime whose input is self-reported harm cannot be preventative; it is an accounting system for damage that has already occurred.
The technical proposals had the opposite failure. They were sound — structured API access, governance over who can call a model, LLM-based defensive systems reading inbound mail — and they quietly assumed a well-resourced organisation would deploy and maintain them. Which means the protection lands exactly where the loss is most survivable and skips the small organisation for whom one successful invoice is existential. A defence distributed by ability to pay inverts the need.
what we proposed
So the paper's actual contribution is not an architecture. It is an incentive structure — three parties, each of which has to come out ahead or the system does not survive contact with reality.
Two pieces of that are doing the real work. The first is reimbursement conditional on correct deployment. It converts security from a cost centre into something with a payoff, and it makes the company's incentive point the same way as the public one — which is the only version of this that survives a budget meeting. The second is the academic maintenance loop, and it is the answer to the question that kills most shared-infrastructure proposals: who is still doing this in year four, after the launch, when it is unglamorous? Tying it to course credit means the maintenance is somebody's coursework in perpetuity rather than a line item waiting to be cut.
The oversight board is the part I would defend hardest now. A shared flagged-sender database is a blocklist with real consequences: being on it means your mail silently stops arriving. Whoever controls that definition controls who can be heard, and handing that to a single vendor, agency or company is a governance failure waiting to happen regardless of how good the model is. Putting the standard in the hands of a mixed board is not bureaucratic decoration — it is the check on the most dangerous thing the system does.
the honest part
It is a paper. It was runner-up at a hackathon judged in a weekend, it was not adopted or piloted, and it is co-authored — three names on it, and the thinking was genuinely shared.
With distance, the weakest link is the false-positive path. A shared blocklist that silently reroutes mail needs a fast, well-resourced appeal route for the wrongly flagged sender, and the paper gestures at oversight without ever specifying that mechanism. That is the gap I would fix first — it is the exact place where a system designed to protect people starts quietly harming a different set of them.
why it's on this site
This is the same instinct as CiteGuard and as the guardrails in the correspondent layer, pointed at institutions instead of at code. In all three, the model is not the thing that makes the system trustworthy — the structure around it is, and the structure has to hold when the people inside it are busy, cheap, or motivated to skip it. Writing this taught me to ask “who maintains this in year four” about my own projects, which is the question that later made me build SkillTern to outlive me and shut TurtleShell down.