Choose a task that can be evaluated
“Add AI” does not define a feature. “Find the relevant policy paragraph and show its source” does. Create a small set of representative questions, including ambiguous requests and cases where the material contains no answer. Decide what a useful answer looks like and when the system should abstain. Human evaluation remains necessary for meaning; simple automated checks can verify source presence, output format and forbidden actions.
Make the data boundary visible
Map which documents enter retrieval, who can read them and whether any content is sent to a provider. Permissions must survive indexing: a convenient search surface should not make restricted content public. Keep provider credentials on the server, review retention settings and avoid logging complete prompts or private documents by default. Choose the provider only after understanding those constraints and the actual operating costs.
Separate an answer from an action
A draft summary and an autonomous action have different risk. Prefer a reviewable suggestion for decisions with a meaningful consequence. Show the source, allow correction and require explicit approval before changing records or messaging people. Build a fallback for provider failures and rate limits. A graceful refusal is often a better product result than a confident unsupported answer, particularly when there is no relevant source.
Measure changes against the same examples
Keep the evaluation examples versioned with the feature. Re-run them when changing prompts, retrieval or provider settings, and record both improvements and regressions. Distinguish a synthetic demo from a connected model: a local deterministic search can prove interface behavior, but cannot prove model quality. Budget provider usage separately and define an acceptable failure behavior before scaling the workflow.