Developing programming language support for end-to-end verification of neural AI agents

Abstract

Over the past decade, there has been impressive progress in developing algorithms capable of formally verifying first-order, linear specifications about small-to-medium-sized neural networks. This has opened the door to formally verifying the correctness of neural AI agents operating in complex environments. However, due to the complexity of integrating these verification algorithms with other reasoning tools and the many moving parts involved, deploying them in practice has proven more challenging than anticipated. In this talk, I will introduce Vehicle, a tool that my collaborators and I have been developing. Vehicle allows users to: (i) write specifications for AI agents in a high-level, domain-specific language; (ii) use these specifications to guide training; (iii) formally verify whether the resulting neural network satisfies the specification; and (iv) export verified specifications to interactive theorem provers where the agents can be reasoned in the context of their environment. In the talk I will highlight how we leverage type theory in unusual ways to improve the user experience, our use of real-valued logic semantics to interpret Boolean specifications, and finally some of the open theoretical challenges in the field.