Skip to content
active platform v0.1.1 Logging Lake Created 2026-06-17 · Updated 2026-07-24

OpenSearch MCP Server with Per-User Auth

  • Product: Logging Lake
  • Family: Enterprise Logging Infrastructure

Objective

Enable AI agents to query the Logging Lake OpenSearch cluster with per-user identity and access control, replacing the shared-credential model that agents typically use.

Context

The Logging Lake product uses Amazon OpenSearch Service as its backend. Agents (Claude, Kiro, etc.) need to query these indices for log search, observability, and investigation workflows.

The existing opensearch-mcp-server-py from the OpenSearch Project supports basic auth, IAM roles, and header-based auth — but all auth is server-wide or per-call with static credentials. There is no mechanism for per-user identity propagation where OpenSearch itself enforces access control based on who the human user is.

IAM-based auth (SigV4) requires the agent to hold AWS credentials, and those credentials grant a single principal's access — not the calling user's. The existing API user model (api-users.tf) creates per-user basic-auth credentials in Vault, but requires pre-provisioning and doesn't scale to arbitrary users. Any shared-credential approach loses per-user audit trails and per-user access restrictions inside OpenSearch.

Current Auth Model (as-is)

Path Auth Method Identity Source Notes
Dashboards (browser) Cloudflare Access → SAML Shibboleth via CF Access roles attribute = cognito:groups; mapped to OS backend roles
API users Basic auth (internal users) Pre-provisioned in api-users.tf Username/password in Vault; scoped index permissions
Infrastructure IAM SigV4 Jenkins, Admin, OSIS roles Mapped to all_access or security_manager
Sourcetype consumers Cognito groups → backend roles edna_cognitogroups_service Per-sourcetype read-only roles

None of these fit a user-operated agent: IAM and basic-auth credentials are shared or pre-provisioned and lose per-user attribution, and the browser SAML flow can't be driven by an MCP client.

Proposed Solution

Build an MCP server that authenticates the user through Cloudflare Access Managed OAuth, then mints a short-lived JWT — signed with a private key it holds — carrying the user's identity (sub) and Cognito group memberships (roles). OpenSearch validates these JWTs against a JWKS endpoint the server exposes, and maps the roles claim to its existing backend roles. The Dashboards flow already proves this identity model (Cloudflare Access → Cognito groups → OpenSearch backend roles); the MCP server reuses the same mapping for agents.

JWT auth is additive on the OpenSearch side — it runs alongside the existing SAML, basic-auth, and IAM domains without disrupting them. On AWS-managed OpenSearch, SAMLOptions and JWTOptions are independent fields targeting different surfaces — SAML governs Dashboards browser sign-in (the existing LOGL use), JWT governs programmatic/API requests (the MCP server) — so they coexist by design (see AWS's JWT fine-grained access control guide). JWKS URL support requires OpenSearch 2.11+; LOGL runs 3.5.

Why Cloudflare Access

  • Per-user attribution — every agent action is tied to the human operator, not a service account.
  • Agent-ready — Managed OAuth exposes the standards-based OAuth 2.0 flow (RFC 9728 discovery, dynamic client registration, PKCE) that MCP clients already speak.
  • No new user pool — leverages the existing IdP integration and the Cognito groups already used by Dashboards.
  • Zero-trust posture — device checks, geo restrictions, and session duration come for free.
  • Consistent — same auth front door as other internal tools.

Architecture

flowchart TD
    agent["AI Agent / IDE<br/>(Kiro, Claude)"]
    idp["Identity Provider<br/>(ASU Primary Sign-in / Shibboleth)"]
    access["Cloudflare Access (Managed OAuth)<br/>OAuth 2.0 AS · RFC 9728 / 7591 / 7636"]
    mcp["LOGL MCP Server<br/>(in VPC, via Cloudflare Tunnel)<br/>validates token · resolves groups ·<br/>mints OS JWT · serves JWKS"]
    os["OpenSearch Domain (eli5-logl-&lt;env&gt;)<br/>In VPC, reached via Tunnel<br/>JWT auth via JWKS URL<br/>roles claim → existing backend roles<br/>(sourcetype-*, data_analyst, all_access)"]

    agent -->|"OAuth flow (browser consent)"| access
    access -->|"authenticates user"| idp
    agent -->|"HTTPS + user-scoped Access token"| mcp
    mcp -->|"get-identity (token)"| access
    mcp -->|"HTTPS + minted Bearer JWT"| os
    os -->|"fetch signing key"| mcp

Authentication Flow

  1. The MCP server's Access application has Managed OAuth enabled (open beta as of April 2026), making Access an OAuth 2.0 authorization server.
  2. The user-operated agent runs the OAuth flow; the user authenticates and consents in-browser via ASU Primary Sign-in, and the agent receives a user-scoped Access token.
  3. The MCP server validates the Access token — signature against the Cloudflare Access JWKS, plus the aud (the app's AUD tag) and iss claims so a token issued for another Access app can't be replayed here — then resolves the user's email and Cognito groups via the Access get-identity endpoint (with a Cognito AdminListGroupsForUser lookup as fallback). Groups are not assumed to be present in the token.
  4. The MCP server mints a short-lived OpenSearch JWT (sub = email, roles = resolved groups), signed with its private key.
  5. The MCP server exposes /.well-known/jwks.json with the corresponding public key.
  6. OpenSearch validates the minted JWT against that JWKS endpoint and maps the roles claim to existing backend roles.

Key Design Decisions

  1. Reuses existing role mappings. The MCP server mints JWTs with roles containing the same Cognito group names already mapped in roles.tf. No new OpenSearch roles needed — users get the same access via MCP as they do via Dashboards.

  2. ECS Fargate in the OpenSearch VPC, fronted by Cloudflare Tunnel. The MCP server runs as an ECS Fargate task in the same VPC as the OpenSearch domain — same pattern as the existing dashboard (cloudflare-tunnel module). Running in-VPC means the server's source IP falls within the private-subnet CIDRs already allowlisted in the domain access policy (opensearch.tf), so requests satisfy both the IP-based resource policy and JWT fine-grained access control (the two are evaluated independently). OpenSearch's JWKS URL points to the MCP server's internal VPC endpoint.

  3. Group resolution outside the token. The Cloudflare Access application token does not carry Cognito groups by default (the custom_attribute in cloudflare.tf applies only to the SAML SaaS app → OpenSearch assertion, not to the agent's Access token). The MCP server resolves groups via the Access get-identity endpoint, falling back to a direct Cognito AdminListGroupsForUser lookup. The fallback path requires Cognito read permissions on the MCP server's execution role.

  4. Backend-role strings emitted verbatim. The minted JWT's roles claim must contain the exact cognito_group strings that the existing role mappings use as backend_roles (e.g., eli5-logl-admin and the edna_cognitogroups_service.*.cognito_group values for sourcetype/data_analyst roles). The MCP server emits resolved group strings unchanged — no normalization, prefixing, or case folding — since a mismatch silently maps the user to default_role instead of failing loudly.

  5. Ephemeral in-memory signing keys. Because OpenSearch fetches public keys from the JWKS endpoint by kid, the MCP server generates signing keys in memory and never persists a private key (no Secrets Manager dependency). The JWKS endpoint publishes the current and previous public keys so short-lived in-flight tokens validate across a rotation. Rotation runs on a bounded interval (e.g., hourly), not per-token — OpenSearch caches JWKS and rate-limits refreshes (default 10 per 10s) when it encounters an unknown kid. The only relying party for these tokens is the LOGL OpenSearch domain.

  6. Token lifetimes. Minted OpenSearch JWTs are short-lived (~15 minutes) to bound replay; OpenSearch validates exp with the 30s clock-skew tolerance. The Cloudflare Access session uses the existing app policy as-is. When the Access session expires, the agent re-runs the OAuth flow (browser re-consent); an expired/invalid token surfaces as an auth error that triggers re-auth.

JWKS Endpoints

Endpoint Purpose Consumer
<team>.cloudflareaccess.com/cdn-cgi/access/certs Validates the user's Cloudflare Access token MCP Server
MCP server internal /.well-known/jwks.json Validates minted OpenSearch JWTs OpenSearch Domain

Operational Requirements

As a long-running in-VPC service, the MCP server needs:

  • Health checks — ECS liveness/readiness checks so the task is replaced when unhealthy; the JWKS endpoint and OpenSearch connectivity are reasonable readiness signals.
  • Structured logging — per-request logs that record the authenticated user (email/sub), tool invoked, and target index, so MCP-level activity correlates with OpenSearch audit logs. Never log tokens or key material.
  • Metrics and alerting — emit to Datadog (org standard; see existing datadog.tf): request rate, auth failures, token-mint failures, group-resolution failures, and OpenSearch error rates, with alerts on sustained failure.
  • Query guardrails — enforce a result-size cap (the upstream server's max_size_limit) and a query timeout so an agent can't overwhelm the cluster or the agent's own context window.

Deployment

Infrastructure is defined in OpenTofu in the eli5-logging-lake repo, alongside the existing OpenSearch and Cloudflare Tunnel config: the Fargate service and task definition, the tunnel/JWKS wiring, and the domain JWTOptions change. It is applied via the repo's existing Jenkins pipeline using the established dev.tfvars / prod.tfvars per-environment pattern. The container runs on ARM64 (Graviton) for cost efficiency. The MCP server application (container image) lives where the fork-vs-standalone decision lands (see Open Questions) and is built/published by its own pipeline.

Tools In Use

  • Python (FastMCP / MCP SDK)
  • Cloudflare Access (frontend authentication)
  • Amazon OpenSearch Service (existing LOGL domain, 3.5)
  • opensearch-mcp-server-py ≥ 0.3.1 (upstream; provides stateless streaming server and all core tools)
  • PyJWT / cryptography for JWT minting
  • ECS Fargate in the LOGL VPC (deployment target)

Under Evaluation

  • Whether to fork opensearch-mcp-server-py or build standalone

Scope

In scope

  • MCP server that authenticates users via Cloudflare Access Managed OAuth
  • User group resolution via the Access get-identity endpoint, with a Cognito AdminListGroupsForUser fallback
  • JWT minting with per-user sub and roles claims
  • JWKS endpoint serving the server's public key
  • OpenSearch domain configuration for JWT auth via JWKS URL
  • Role mapping reuse: minted roles map to existing OpenSearch backend roles
  • Core tools: index listing, search, multi-search, mappings, cluster health, count, explain, shards (all ship in upstream ≥ 0.3.1)
  • Deployment to LOGL prod/non-prod
  • Audit trail: OpenSearch audit logs capture per-user identity from JWT
  • End-to-end validation: integration test that a known user resolves to the expected backend role, and confirmation that OpenSearch audit logs attribute queries to distinct user identities

Out of scope

  • Modifying the upstream opensearch-mcp-server-py project
  • Write operations to OpenSearch (initial version is read-only)
  • Multi-cluster support (single LOGL domain per deployment)
  • OpenSearch Dashboards integration
  • Custom tool development beyond core search operations
  • Headless/non-interactive agent auth — phase 1 is strictly for user-operated agents where the user authenticates in-browser (see Future Phases)

Future Phases

  • Service-token auth for headless/CI-CD agents. Cloudflare Access service tokens authenticate non-interactive clients, but their tokens carry no user identity (empty sub). A later phase could map a service token to a dedicated low-privilege OpenSearch role for automation, kept separate from the per-user path so attribution is never ambiguous.
  • Write operations to OpenSearch (e.g., index management) behind appropriately scoped roles.

Risks

Risk Likelihood Impact Mitigation
JWKS endpoint reachability from OpenSearch Low Medium Deploy MCP server in same VPC; OpenSearch already resolves internal endpoints. Alternatively, expose JWKS via existing Cloudflare Tunnel with IP allowlist
Private key management Low Medium Use in-memory ephemeral signing keys (no key at rest); publish current + previous public keys via JWKS so in-flight tokens validate across rotations
JWKS refresh rate limiting on aggressive rotation Low Medium OpenSearch caches JWKS and rate-limits refreshes (default 10 per 10s) on unknown kid. Rotate on a bounded interval (e.g., hourly), not per-token, and set cache-control headers
get-identity does not return Cognito group data Medium Medium Fall back to direct Cognito AdminListGroupsForUser lookup; grant the MCP execution role scoped Cognito read permissions
Resolved group strings don't match backend_roles Medium High Silent failure mode — user falls through to default_role with no error. Confirm the exact cognito_group string format end-to-end (resolver output vs. roles.tf backend_roles); emit verbatim; add an integration test asserting a known user lands in the expected role
Managed OAuth is open beta Low Medium Feature is in open beta as of April 2026; validate stability in non-prod before prod rollout
JWT clock skew between MCP server and OpenSearch Low Low Set jwt_clock_skew_tolerance_seconds to 30s
Existing role mappings don't cover all MCP users Medium Medium Users without a sourcetype role get default_role (cluster monitor + own tenant) — same as current Dashboards behavior

Success Criteria

  • Users authenticate via Cloudflare Access using existing IdP credentials
  • Each OpenSearch query carries a per-user JWT — no shared service account
  • OpenSearch audit logs show distinct user identities for each query
  • Users with admin groups can search all indices; restricted groups have limited access
  • JWKS endpoint is operational and OpenSearch successfully validates minted JWTs
  • Agents (Kiro, Claude Code) can use the MCP server via SSE/Streamable HTTP transport

Open Questions

  • Fork opensearch-mcp-server-py or standalone implementation? The upstream (≥ 0.3.1) exposes a streaming_server.serve(stateless=True) API suitable for embedding as the tool-execution backend inside a wrapper that handles auth separately. This favors a thin wrapper approach over a full rewrite — the wrapper handles Cloudflare Access validation, group resolution, and JWT minting, then delegates tool calls to the upstream's stateless serving layer.
  • Does the Access get-identity response include the IdP group data needed, or is the Cognito fallback the primary path in practice?
  • Verify in non-prod that adding JWTOptions via update-domain-config leaves the existing SAMLOptions block intact (both are set in the same AdvancedSecurityOptions API call).
  • What exact string does the group resolver return (get-identity IdP data and/or Cognito AdminListGroupsForUser), and does it match the cognito_group values used as backend_roles in roles.tf verbatim?
  • How does OpenSearch's identity cache (plugins.security.cache.ttl_minutes) behave on the managed domain — can it serve stale roles within the TTL, and is the setting even tunable on managed OpenSearch? (Likely minor for read-only access.)

References

Decisions

Date Decision Rationale
— — —

Changelog

Date Status Change
2026-06-17 draft Initial draft
2026-07-24 active Review pass: removed the duplicated status and product-key lines from the body so frontmatter is the single source of truth; body now carries full product and family names