Extract bookings from uploaded documents with a tool-less model call over OpenAI file inputs
Decision
Store each uploaded document in S3-compatible object storage, Cloudflare R2 in production, and extract its bookings by sending the document itself to OpenAI. A PDF is uploaded through the Files API for the one request and deleted when the request returns. An image is passed as a presigned URL to our own bucket.
The extraction request offers the model a single strict function, the effect schema of ADR-0096, and requires it to call it. The model gets no web search and no other tool. Its arguments are parsed into typed effects, resolved against the trip, and applied directly as one modification tagged with the source document, so it shows in history and can be undone like any edit. Alongside the document the model sees only the trip's purpose and destinations, each wrapped in provenance markers the prompt tells it to treat as data.
Extraction is limited per user per time window and by file size, and a second extraction of a document whose bookings are already on the trip is refused.
Why
Booking confirmations arrive as PDFs and screenshots in every layout an airline, hotel or restaurant has invented. A multimodal model reads them directly, including scanned pages and images, without a text-extraction step that loses tables and layout.
An uploaded document is untrusted content, and it holds personal data. ADR-0038 keeps neighbourhood text safe by giving the flow only one leg of the lethal trifecta. Here two legs are unavoidable, so we remove the third: the model has no tool that reaches the network, and its only output is effects on the uploader's own trip, which only that trip's members can read.
Rejected alternatives
- Extract text locally and send only text. It loses layout, fails on images and scans, and the text is just as untrusted.
- Give extraction the agent's tools or web search. That would let a document's contents reach the network.
Consequences
The model-facing effect type is the full one intent parsing uses, and extraction applies it with no filter to bookings. A planted instruction in a document can therefore make any edit to that trip that a typed request could make, including removing its destinations. Every such change is in the trip's history and can be undone.
Extraction applies its result directly, while the in-app agent's changes wait for the user to review them because it reads untrusted web pages (ADR-0100). Extraction predates that proposal flow, and we have not moved it onto it.
Each extraction sends the user's document to OpenAI, so OpenAI's data handling applies to it. Deleting the uploaded file is best effort, and a failed delete leaves it in our OpenAI account until removed.