Security and Authentication: JWT, OAuth2, and Secrets Management

Security architecture decisions have higher stakes than most — the cost of getting them wrong is a data breach, not a performance degradation. This post covers the trade-offs that come up in EM-level interviews: authentication approaches, identity protocols, and secrets management.

Both are valid. The choice depends on your consistency requirements and architecture.

The server stores session state. On login, the server creates a session record (in DB or Redis) and sends a session cookie. On each request, the cookie is sent and the server looks up the session.

Advantages:

• Immediate revocation. Delete the session record → user is logged out globally, right now. This is the critical advantage.
• Simple invalidation on security events — password change, suspicious activity detection → delete all sessions.
• Server controls session lifetime — extend, shorten, or terminate based on server-side logic.
• No sensitive data on the client. The session cookie is just a random ID.

Disadvantages:

• Requires shared session store for stateless/multi-instance deployments. Redis is the standard solution.
• Every request hits the session store — adds ~1ms latency per request (Redis round trip). This is usually fine.
• Doesn’t work well for mobile apps or non-browser clients where cookie management is manual.

Tokens are self-contained: they carry claims (user ID, roles, expiry) and are cryptographically signed by the server. No server-side state is needed to validate — just verify the signature.

Advantages:

• Stateless validation — no session store lookup. Validates purely from the token + signing key.
• Works naturally across domains — mobile apps, SPAs, third-party integrations.
• Embeds user context — downstream services can extract claims without calling an auth service.

Disadvantages:

• Revocation is hard. There’s no registry of valid tokens — a signed token is valid until expiry. If you need to log out a user mid-session (compromised account, password reset), you either:
  • Wait for expiry (can be hours)
  • Maintain a blocklist (now you have state again, defeating the purpose)
  • Use short expiry (5–15 minutes) with refresh tokens (complexity)
• Token size — JWTs in cookies or headers can be large (especially with many claims). Not a problem in practice, but worth knowing.
• Algorithm confusion attacks — if JWT validation is implemented incorrectly (accepting alg: none, not validating the algorithm), it’s exploitable. Use a well-tested library, never roll your own JWT validation.

When to use which:

• Web apps with server-rendered content or same-domain SPA: Session cookies. Revocation is clean, CSRF protection is straightforward with SameSite=Strict.
• Mobile apps, SPAs calling third-party APIs, OAuth2 flows: JWTs. Stateless, portable.
• Microservices: JWTs for service-to-service claims propagation. API gateway validates the JWT once; downstream services trust the claims without calling auth.
• Revocation requirement is hard: Lean toward sessions or short-lived JWTs (< 5 minutes) with refresh token rotation.

These are three different protocols solving overlapping but distinct problems.

OAuth2 is an authorization framework — it answers “can application X access resource Y on behalf of user Z?”

The flows:

• Authorization Code flow (+ PKCE for public clients): The standard flow for web apps and mobile apps. User authenticates with the auth server, gets an authorization code, app exchanges it for tokens.
• Client Credentials flow: Machine-to-machine. No user involved. Service A gets a token to call Service B.
• Device flow: For devices without browsers (CLI tools, smart TVs) — user authenticates on a separate device.

Key tokens:

• access_token: Short-lived (minutes to hours), used to call APIs. Should be opaque to clients or a JWT.
• refresh_token: Long-lived, used to get new access tokens without re-authentication.

OIDC adds an identity layer to OAuth2. It answers “who is this user?” by introducing the id_token (a JWT with user claims: sub, email, name).

Use OIDC when you need to: authenticate users via a third-party identity provider (Google, GitHub, Azure AD), implement SSO across your applications, or get standard user profile information.

OIDC vs OAuth2: OAuth2 alone tells you an app can access a resource. OIDC tells you who the user is. For user authentication (login), use OIDC. For API access delegation, use OAuth2.

SAML (Security Assertion Markup Language) is the older enterprise SSO standard. XML-based, stateful, tightly coupled to browser-redirect flows.

When you encounter SAML: Enterprise customers requiring SSO integration with their corporate identity providers (Active Directory, Okta, PingIdentity). You don’t choose SAML — your enterprise customer requires it.

SAML vs OIDC: OIDC is the modern alternative. If your enterprise customers support OIDC, use it. SAML is harder to implement correctly, XML is verbose, and the tooling is older. Many identity providers now support both.

Three options, each with different trade-offs:

Option 1: Each service validates tokens independently Every service has the JWT signing key and validates tokens itself. Simple, no single point of failure. But: every service needs the signing key (key distribution problem), every service reimplements the same validation logic (risk of inconsistency), and adding a claim or changing validation logic requires updating every service.

Option 2: API Gateway handles authentication The gateway validates the JWT. Downstream services receive a trusted header (X-User-ID, X-User-Roles) and trust it without revalidation. Centralizes auth concern, simplifies services.

The risk: If a service is reachable without going through the gateway (direct internal calls, misconfigured networking), it’ll accept requests without auth. Mitigation: network policy restricts direct access; mTLS between services.

Option 3: Service Mesh handles authentication (mTLS + SPIFFE/SPIRE) The mesh enforces mTLS between all services. Services only accept connections from other services with a valid mesh certificate. Identity is proven at the transport layer, not the application layer. Combine with JWT validation at the gateway for user identity.

The recommendation for most teams: API gateway handles JWT validation + extracts user context. Pass user identity as trusted headers to downstream services. Add mTLS via service mesh if you need service-to-service authentication beyond network policy.

Secrets (API keys, DB passwords, signing keys, certificates) are the most sensitive assets in your system. Where they live determines your blast radius when they’re compromised.

Antipattern: Secrets in code / environment variables hardcoded in deployment manifests

• Committed to Git → permanent exposure in history
• Visible to anyone with repo access
• No rotation without redeployment

The tiers of secrets management:

Tier 1: Cloud-native secret stores

• AWS Secrets Manager, Azure Key Vault, GCP Secret Manager
• Secrets stored encrypted, access controlled via IAM
• Automatic rotation for supported services (RDS passwords, for example)
• Audit trail of all access
• Injected into workloads at runtime via SDK or container init

Tier 2: HashiCorp Vault

• Self-hosted, cloud-agnostic secret store
• Dynamic secrets — generate short-lived DB credentials on demand instead of shared long-lived passwords
• Kubernetes integration — applications authenticate to Vault using their K8s service account token
• Sophisticated policy engine, full audit log
• Operational overhead of running Vault itself (though Vault’s HA mode and HCP Vault reduce this)

Tier 3: Kubernetes Secrets

• Base64-encoded (not encrypted by default) ConfigMaps with tighter access control
• Must enable etcd encryption at rest to actually secure them
• External Secrets Operator: sync from AWS Secrets Manager / Vault into Kubernetes Secrets — best of both worlds

Rotation strategy: Every secret should have a rotation plan. Database passwords rotated quarterly minimum (monthly for high-value systems). API keys with rotation support should be rotated regularly. Certificates should auto-rotate (cert-manager + Let’s Encrypt or internal CA).

Mutual TLS authenticates both client and server. Each service presents a certificate; both sides verify.

What mTLS gives you:

• Service identity — you know who’s calling, not just that the call came from within the cluster
• Encryption of inter-service traffic (important if network isn’t fully trusted)
• Defense against a compromised pod injecting traffic — without a valid certificate, connections are rejected

The implementation path:

• Manual: Generate CAs, issue certificates per service, rotate them — operationally expensive, error-prone
• Service mesh (Istio, Linkerd): mTLS is automatic. The mesh issues certificates (SPIFFE SVIDs) to each service and handles rotation. Zero application code change.

When mTLS is worth it:

• Regulated industries (PCI, HIPAA) where network-level encryption and service identity are required
• Large microservice architectures where zero-trust networking matters
• When you’re already running a service mesh (the cost is nearly zero — mTLS is built in)

When it’s overkill:

• Small teams, few services, already have network policies restricting access
• The operational overhead of managing certificates or running a service mesh isn’t justified

The honest take: if you’re running Kubernetes and have the resources to operate Istio, turn on mTLS. It’s free security. If you’re a small team running 5 services, network policies + JWT validation at the gateway is probably enough.