Observability

Observability — OpenTelemetry → GCP-native

The observability stack uses OpenTelemetry for instrumentation and GCP-native backends for storage and visualization: Cloud Logging (+ Log Analytics), Cloud Trace, Cloud Monitoring, and Error Reporting. Telemetry stays inside the existing Google BAA — no second vendor boundary, no standing Elasticsearch cluster.

API OTel Setup — backend/api

Dependencies (pyproject.toml)

• opentelemetry-sdk
• opentelemetry-instrumentation-fastapi
• opentelemetry-exporter-otlp

Initialization (main.py)

• Initialize the OTel tracer + meter on application startup.
• Auto-instrument FastAPI with opentelemetry-instrumentation-fastapi.
• The OTLP endpoint is configured via OTEL_EXPORTER_OTLP_ENDPOINT.

In production, OTLP is exported to GCP's managed endpoint (telemetry.googleapis.com) → Cloud Trace / Cloud Monitoring. In development, it points at the local OTel Collector.

PHI Scrub Filter

A PHIScrubFilter stub class is included in the Python backend services (backend/api and backend/worker):

import logging
 
class PHIScrubFilter(logging.Filter):
    def filter(self, record):
        raise NotImplementedError(
            "Implement PHI scrubbing before production use"
        )

This filter is attached to the root logger in each service (backend/api and backend/worker) and will intentionally fail if any log message is emitted before the scrubbing logic is implemented. (This matches the PHIScrubFilter control in HIPAA Compliance, which covers both backends.)

TODO before production:

• Implement regex-based scrubbing for:
  • Medical Record Numbers (MRNs)
  • Dates of Birth (DOB patterns)
  • Patient names
• Test the filter with representative log data
• Verify no PHI appears in Cloud Logging, Cloud Trace spans, or Cloud Build logs

Local Development Stack — docker-compose.otel.yml

For local development, run a lightweight OTel Collector — no Elasticsearch/Kibana required:

The collector receives OTLP on 4317/4318 and, in dev, exports to the console and/or a local Jaeger UI. In production the same OTLP stream is pointed at telemetry.googleapis.com.

OTel Collector Configuration

An otel-collector-config.yml stub is provided:

• Dev: debug (console) exporter + optional otlp → Jaeger.
• Prod: googlecloud exporter (or OTLP → telemetry.googleapis.com).

Data Flow

FastAPI App (OTel SDK)
    → OTLP (gRPC/HTTP)
    → OTel Collector
    → Cloud Trace / Cloud Monitoring / Cloud Logging   (prod: telemetry.googleapis.com)
    → console / local Jaeger                            (dev)

Dashboards

Use Cloud Monitoring dashboards first. If Grafana-grade dashboards are needed, self-host Grafana on Cloud Run against Cloud Monitoring / Managed Prometheus data sources — GCP has no first-party managed Grafana, and Grafana Cloud SaaS would add a third-party boundary.

Alerting / On-Call

Alerting is GCP-native, matching the rest of the observability stack — no separate paging vendor at launch:

• Cloud Monitoring alert policies fire on metric thresholds, log-based metrics, and uptime checks.
• SLOs (Cloud Monitoring service-level objectives with burn-rate alerts) drive the signal — page on error-budget burn, not on every transient blip.
• Notification channels: alerts route to Microsoft Teams (via an incoming-webhook channel) and email. Teams is the primary real-time surface; email is the durable backstop.

Error Tracking

Error tracking is split by tier, each staying inside an existing boundary:

• Backends (backend/api, backend/worker) → GCP Error Reporting (part of Cloud Operations; groups exceptions from Cloud Run logs, in-boundary under the Google BAA).
• Clients (ui/web-app, ui/marketing, ui/ios) → PostHog error tracking — already the BAA-covered analytics/replay vendor, so it covers web + Expo/React Native with no extra boundary.
• Escalation: Sentry if richer release-health / source-map grouping is needed later (adds a vendor boundary + BAA).

LLM Observability & Evals

LLM calls (Gemini via Vertex) are traced and evaluated in PostHog (LLM analytics — prompt/response capture, latency + cost, and eval scoring), consolidating on the vendor already inside the BAA.

• Escalation: self-hosted Langfuse on Cloud Run (in-boundary) if deeper trace trees or dataset-based eval workflows are needed.