‘Your Search Button Powers My Smart Home’ ⇥ tomcasavant.com

Tom Casavant:

You see, what I hadn’t considered that night when I was messing around with this website’s chat bot was that the existence of a public user facing chat bot had the requisite of having public LLM API endpoints. Normally, you probably wouldn’t care about having a /search endpoint exposed on your website, because very few (if any) people would care to abuse it. Worst case scenario is someone has an easier way of finding content on your site…which is what you wanted when you built that search button anyways. But, when your /search endpoint is actually just talking to an LLM and that LLM can be prompt injected to do what I want it to do, suddenly I want access to /search because I get free access to something I’d normally pay for.

If you have administrative access over a website and you have had reason to dig into the access logs, you have no doubt seen an avalanche of automated requests looking for common security vulnerabilities. Now imagine that but with a bunch of plain language attacks on the very expensive new website feature you added. It is going to be a wild several years as more people begin to integrate these sophisticated yet — to anthropomorphize — gullible text boxes without understanding how much it is going to cost them directly and indirectly.