Building a Secure Self-Hosted Observability Pipeline for an AI Evaluation Platform

Description and use

A secure, self-hosted observability pipeline is live. Team members can view traces and evaluation metadata via a browser interface protected by Cloudflare Access and Tunnel. The underlying Azure VM remains invisible to the public internet, with no public IP. Secrets are managed securely via Key Vault and Managed Identity, and telemetry is redacted to prevent data leakage.

Problem

How to safely view evaluation telemetry for an AI evaluation platform without exposing the VM, databases, credentials, or internal services to the public internet.

Outcome

A secure, self-hosted observability pipeline is live. Team members can view traces and evaluation metadata via a browser interface protected by Cloudflare Access and Tunnel. The underlying Azure VM remains invisible to the public internet, with no public IP. Secrets are managed securely via Key Vault and Managed Identity, and telemetry is redacted to prevent data leakage.

Workflow

• Provision an Azure VM without a public IP address.
• Set up Cloudflare Access to require company email domain verification and one-time passcode (OTP) authentication.
• Install and configure Cloudflare Tunnel on the Azure VM to create an outbound-only connection to Cloudflare.
• Deploy Caddy as a local reverse proxy on the VM to route traffic to Phoenix UI and Evaluation API.
• Store container images in Azure Container Registry (ACR) and configure the VM to pull images using Azure Managed Identity.
• Store sensitive configuration (tunnel credentials, DB passwords, API keys) in Azure Key Vault.
• Configure the Evaluation API to emit safe operational metadata (IDs, scores, timing) to OpenTelemetry, excluding raw prompts and secrets.
• Configure OpenTelemetry Collector to apply redaction rules before forwarding telemetry to Arize Phoenix.
• Access the Phoenix UI via the Cloudflare-protected URL, authenticating with company email OTP.

Evidence

The Azure VM has no public IP address. This means the following are not exposed to the internet: SSH, Phoenix, The evaluation API, PostgreSQL, OpenTelemetry Collector ports, Container runtime ports, Internal reverse-proxy routes.

Source

The OpenTelemetry Collector also applies redaction rules before forwarding telemetry to Phoenix. This means the UI can answer operational questions... Without automatically storing potentially sensitive agent content.

Source

The platform is protected using Cloudflare Access. The user flow is: ... Cloudflare sends a one-time passcode. ... Cloudflare allows the authenticated browser session through the tunnel.

Source