Public AI Governance Documentation

Evaluation and Monitoring

A practical AI evaluation, release, incident, and quarterly review cadence.

Last reviewed: July 9, 2026

This approach is designed for a solo-founder SaaS. It favors repeatable, evidence-producing checks over heavyweight compliance process.

Evaluation Goals

HireProxy should be evaluated for:

• Grounding: AI outputs should stay tied to candidate-provided career data, saved prep, or the submitted transcript.
• Confabulation resistance: AI should not invent metrics, credentials, dates, employers, clients, tools, or outcomes.
• Privacy boundaries: private practice answers should not become public candidate-site facts unless the user chooses to refine and publish them.
• Human oversight: users and visitors should understand the role and limits of AI output.
• Security and tenant isolation: one user's data should not be exposed to another user or public visitor.
• Voice reliability: voice practice should fail gracefully to typed fallback when transcription or speech generation fails.
• Product quality: faster spoken feedback should not replace or degrade the full detailed evaluation.
• Interview coaching quality: rehearsal scoring should emphasize direct question relevance, clear structure, candidate-owned action, result or impact, and role-relevant story fit without becoming harsh or cluttered.

Current Monitoring Signals

• Application error monitoring and provider-failure messages.
• Report-abuse submissions for misleading, false, or abusive candidate-site content.
• User-flow events for interview prep and rehearsal completion.
• Interview Mode usage events for voice transcription and answer evaluation.
• Tests for accuracy guardrails, privacy telemetry controls, public discovery, interview prep, speech/transcription behavior, and rehearsal behavior.
• Public governance pages that describe privacy, retention, incident response, and risk posture.

Lightweight Grounding Eval Set

Maintain an internal grounding and confabulation evaluation set. The public documentation describes the evaluation categories, while the exact examples, expected answers, and pass/fail notes stay internal.

The eval set should cover:

• Questions where the answer is supported by candidate-provided facts.
• Questions that ask for missing, sensitive, or high-risk facts.
• Role-fit analysis against representative job descriptions.
• Interview feedback for weak, strong, noisy, and incomplete answer transcripts.
• Attempts to override the assistant's intended scope or instructions.

For each internal example, record the input, allowed facts, facts that must not be stated, expected uncertainty behavior when relevant, model/version context, result, and reviewer notes.

Run the eval set before:

• Changing core AI instructions or accuracy guardrails.
• Upgrading or switching AI models.
• Changing data flows between private practice and public career-site content.
• Launch campaigns, partner pilots, or enterprise/outplacement sales.

Release Checklist for AI Changes

For each AI-impacting release, answer:

• Did an AI instruction, model, provider, data source, retention behavior, or public disclosure change?
• Which system card changed?
• Which NIST AI RMF functions are affected?
• Which NIST AI 600-1 risks are affected?
• Were grounding/confabulation evals run?
• Were privacy-sensitive paths or logs affected?
• Were public terms, privacy, retention, risk, or incident pages affected?
• Is user-facing behavior still clear about AI limitations?
• Are rollback or disable steps understood?

Quarterly Review Checklist

Every quarter:

• Review confirmed AI quality reports and incident register entries.
• Review provider failures and application error patterns.
• Review any model/provider changes and update the changelog.
• Re-run the grounding/confabulation eval set.
• Spot-check public career-agent answers for grounding and tone.
• Spot-check Interview Mode feedback for accuracy, harshness, and usefulness.
• Spot-check that quick spoken coaching remains concise and scoreless, with scored feedback reserved for the full written rubric.
• Verify privacy/data-retention docs still match product behavior.
• Verify sensitive surfaces remain excluded from session diagnostics.
• Verify public pages do not claim NIST certification or outcome guarantees.
• Decide whether any risk should move from accepted to mitigated.

HireProxy uses a scheduled operator review reminder for this cadence. The operator receives an email when the review opens, and the operator dashboard shows the current review checklist and proof note until the review is marked complete.

Quality Metrics

Use simple metrics that are realistic for a small company:

• Confirmed AI false-fact incidents per 100 meaningful AI interactions.
• Confirmed privacy or data-boundary incidents.
• Confirmed cross-tenant exposure incidents.
• Report-abuse volume by category.
• Eval pass rate before model or AI-instruction changes.
• Median and p95 time to spoken feedback in Interview Mode.
• Failed voice transcription rate and typed-fallback rate.

Escalation Thresholds

Investigate immediately:

• Any suspected cross-tenant data exposure.
• Any AI output that invents a credential, employer, certification, client, or material metric.
• Any private practice answer exposed publicly without user action.
• Any provider or telemetry behavior that captures private resume, job-description, voice, transcript, or account content unexpectedly.
• Any repeated report-abuse pattern from a candidate site.

Accepted Residual Risks

The following residual risks are accepted with disclosure and monitoring:

• AI can still make mistakes even with grounding rules.
• Candidate-provided data may itself be inaccurate.
• Public candidate sites can expose information candidates choose to publish.
• Transcription can be inaccurate.
• Public company information used for interview prep can be stale or incomplete.
• HireProxy depends on third-party AI and infrastructure providers.