Operations Runbook

This runbook is for engineers and on-call operators handling production issues for Girard AI. The canonical deployment path is Coolify building the Dockerfile from the live branch. For setup details, see the Coolify deployment guide.

First Response Checklist

1. Classify the incident:
   • P0: site unavailable, auth/session failures for most users, data loss, security incident, or payments blocked.
   • P1: major feature unavailable, sustained 5xx rate, failed deploy, or background jobs stopped.
   • P2: degraded performance, isolated route failure, provider outage with a working fallback, or noisy but contained error spike.
2. Preserve evidence before changing anything:
   • Affected URL, user or organization ID, browser and device, timestamp, and screenshot or response body.
   • x-correlation-id and x-request-id response headers from the failing request.
   • Current /api/version payload and latest Coolify deployment ID.
3. Check the live app:

   curl -i "$APP_URL/api/version"
   curl -i "$APP_URL/api/health"
   curl -i "$APP_URL/api/health?strict=1"
   curl -i "$APP_URL/api/ready"
   

4. Open Coolify logs for the app container and, when relevant, the worker container running npm run worker.
5. Search Sentry for the route, digest, user ID, x-correlation-id, or x-request-id.
6. Decide whether the safest immediate action is rollback, env correction, provider failover, traffic shaping, or a hotfix.

Operational Signals

Signal	Where to check	Healthy result
Deploy freshness	GET /api/version	JSON includes the expected commit SHA and app version.
Liveness	GET /api/health	200 with status: "healthy" or status: "degraded" for non-strict dependency warnings.
Strict readiness	GET /api/health?strict=1	200 only when database and required env checks pass.
Runtime readiness	GET /api/ready	200 with ready: true, database ok, cache ok, and environment ok.
Errors	Sentry and Coolify runtime logs	Events contain route/source/runtime tags and correlation IDs when present.
Request tracing	Response headers and structured logs	x-correlation-id is stable across retries; x-request-id identifies the specific request.
API throttling	Response headers	X-RateLimit-* headers count down before any 429.
Web Vitals	Signed-in /performance dashboard	Health score, route hot spots, and recent samples reflect current traffic.
Background jobs	Coolify worker logs and Redis	Jobs are being consumed; no repeated BullMQ connection errors.

Common Incident Playbooks

App returns 5xx

Symptoms: sustained 500 or 503, dashboard crash boundary, Sentry spike, or Coolify logs showing repeated unhandled exceptions.

1. Confirm whether the latest deploy is live:

   curl -i "$APP_URL/api/version"
   

2. Capture the failing response headers:

   curl -i "$APP_URL/path-that-fails"
   

3. Use x-correlation-id and x-request-id to search Coolify logs and Sentry.
4. Check whether the issue started immediately after the latest Coolify deploy.
5. Roll back from the Coolify deployment list if the current deploy introduced the error and the previous image is known good.
6. If rollback is not viable, patch the route or boundary and add a focused regression test for the failing payload.

Health or readiness is failing

Symptoms: /api/health?strict=1 or /api/ready returns 503.

1. Compare non-strict and strict checks:

   curl -i "$APP_URL/api/health"
   curl -i "$APP_URL/api/health?strict=1"
   curl -i "$APP_URL/api/ready"
   

2. If environment is failing, inspect app runtime logs for missingKeys. The public response intentionally hides individual secret names.
3. Validate required configuration locally against lib/env/contract.ts:

   npx tsx scripts/validate-env.ts
   

4. Check these required production values first:
   • DATABASE_URL
   • JWT_SECRET
   • ENCRYPTION_KEY
   • NEXT_PUBLIC_APP_URL
   • NEXT_PUBLIC_MARKETING_URL
   • At least one of OPENAI_API_KEY, ANTHROPIC_API_KEY, or GOOGLE_GENERATIVE_AI_API_KEY
5. If database is failing, confirm the Coolify Postgres resource is running, DATABASE_URL points at the reachable internal host, and migrations have been applied with:

   npx prisma migrate deploy
   

6. If cache is failing in /api/ready, check ENABLE_REDIS_CACHE, CACHE_REDIS_URL, and REDIS_URL. Single-node fallback should still work when Redis cache is disabled.

API requests are rate limited

Symptoms: API responses return 429 with RATE_LIMITED and Retry-After.

1. Capture the rate-limit headers:

   curl -i "$APP_URL/api/some-route"
   

2. Confirm whether broad API rate limiting is enabled:
   • ENABLE_API_RATE_LIMITING or API_RATE_LIMIT_ENABLED
   • API_RATE_LIMIT_MAX_REQUESTS
   • API_RATE_LIMIT_WINDOW_MS
   • API_RATE_LIMIT_BYPASS_PATHS
3. Default bypass paths are /api/health, /api/ready, and /api/admin/health.
4. If a proxy is collapsing many users into one IP, verify Coolify forwards x-forwarded-for, x-real-ip, or cf-connecting-ip.
5. For an incident mitigation, raise the limit or add a narrow bypass path. Keep product-level upload, chat, and generation limits in place.

Sentry is not receiving errors

Symptoms: users see an error boundary, but no Sentry event appears.

1. Verify runtime DSNs:
   • Server and edge: SENTRY_DSN
   • Browser: NEXT_PUBLIC_SENTRY_DSN
2. Verify enable flags and metadata:
   • SENTRY_ENABLED
   • NEXT_PUBLIC_SENTRY_ENABLED
   • SENTRY_ENVIRONMENT or NEXT_PUBLIC_SENTRY_ENVIRONMENT
   • SENTRY_RELEASE or NEXT_PUBLIC_SENTRY_RELEASE
   • SENTRY_TRACES_SAMPLE_RATE or NEXT_PUBLIC_SENTRY_TRACES_SAMPLE_RATE
3. Check the route, digest, source, runtime, and correlation tags on captured events.
4. If errors are ignored, compare the message to the shared ignore lists in lib/error-tracking.ts.
5. If source maps are needed for a release, confirm the build plugin is enabled in the deployment environment before the next build.

Performance is degraded

Symptoms: users report slow pages, poor interaction latency, or the performance dashboard shows poor LCP, INP, CLS, FCP, or TTFB.

1. Open the signed-in /performance dashboard.
2. Check the health score, worst metric, route hot spots, sample count, and recent measurements.
3. Fetch the aggregated Web Vitals payload when debugging authenticated traffic:

   curl -i "$APP_URL/api/performance/vitals"
   

4. If samples are empty, confirm the browser is loading the Web Vitals reporter and that the page can POST beacons to /api/performance/vitals.
5. For poor LCP or TTFB, inspect database queries, external provider calls, and server logs for the same route and correlation ID.
6. For poor INP, reproduce in the browser, profile the interaction, and check whether a client component is doing heavy synchronous work.

Deploy fails or build OOMs

Symptoms: Coolify build fails, the image does not start, or logs mention memory pressure.

1. Confirm Coolify is building from deploy/coolify-live; stale branches are a common cause of "deploy succeeded but nothing changed".
2. Check build variables:
   • NEXT_BUILD_MAX_OLD_SPACE_SIZE
   • NEXT_BUILD_CPUS
   • NEXT_STATIC_GENERATION_MAX_CONCURRENCY
3. The Next build intentionally skips its internal type checker because this repository type-checks separately. Do not re-enable build-time type errors as part of incident response.
4. If memory is tight, lower build parallelism before retrying:

   NEXT_BUILD_CPUS=1 NEXT_STATIC_GENERATION_MAX_CONCURRENCY=1 npm run build
   

5. Reproduce locally with focused verification first, then npm run build if memory allows.
6. If the container starts but fails immediately, check that npm run start passes env validation before launching .next/standalone/server.js.

Background jobs are stalled

Symptoms: async work queues up, generated assets do not finish, webhook follow-up jobs do not run, or worker logs are quiet.

1. Confirm the separate Coolify worker application is running with start command npm run worker.
2. Confirm both app and worker share:
   • ENABLE_BACKGROUND_QUEUE=true
   • REDIS_URL
   • Any provider or storage env vars used by the job family.
3. Check worker logs for BullMQ connection errors, repeated job failures, or missing env validation.
4. Tune QUEUE_WORKER_CONCURRENCY only after confirming Redis and provider dependencies are healthy.
5. If Redis is unavailable, web requests continue, but queue producers no-op with warnings until Redis is restored.

AI provider or image generation outage

Symptoms: chat, agent, Studio, image, video, audio, or speech generation fails.

1. Identify the provider family from the failing route, model, or product area.
2. Check provider status and recent Coolify logs.
3. Confirm at least one core AI provider key is configured for health: OPENAI_API_KEY, ANTHROPIC_API_KEY, or GOOGLE_GENERATIVE_AI_API_KEY.
4. For Studio image generation, confirm the model-specific keys:
   • FAL_KEY for Flux and Stable Diffusion models.
   • OPENAI_API_KEY for DALL-E and GPT Image models.
   • GOOGLE_GENERATIVE_AI_API_KEY for Imagen and Gemini image models.
5. Temporarily hide or disable the affected provider family if fallback quality is better than repeated user-visible failures.

Cache, trending, or hot-path data looks stale

Symptoms: hot-path API responses do not update, route dashboards disagree with recent writes, or Studio trending data appears stale.

1. Check whether distributed cache is enabled:
   • ENABLE_REDIS_CACHE
   • CACHE_REDIS_URL or REDIS_URL
2. Inspect response cache headers such as X-API-Cache on affected API routes.
3. Confirm the cache backend in /api/ready diagnostics.
4. If a hot path is stale after a deploy, disable Redis cache temporarily or clear the relevant Redis keys from the Coolify Redis console.
5. Keep local in-memory fallback enabled for single-node recovery.

Rollback Procedure

1. Prefer Coolify's deployment rollback for a bad release. It restores the previous image faster than a revert build.
2. Confirm the rollback version:

   curl -i "$APP_URL/api/version"
   curl -i "$APP_URL/api/health?strict=1"
   

3. If the issue was caused by a database migration, do not roll back code until the schema compatibility risk is understood.
4. After service recovery, create a follow-up fix or revert commit on deploy/coolify-live.
5. Record the incident summary, root cause, mitigation, affected users, and follow-up tasks.

Escalation Criteria

Escalate to the engineering lead immediately for P0 incidents, suspected security or data exposure, destructive migration risk, payment disruption, or any rollback that might conflict with database state. Escalate to provider support when the app checks are healthy but the upstream provider is confirmed degraded or unreachable.

Post-Incident Checklist

• Sentry issue linked to the incident notes.
• Coolify deploy ID and /api/version payload recorded.
• Correlation IDs captured for representative failures.
• User impact and duration recorded.
• Regression test added for code defects.
• Runbook updated when the response required undocumented knowledge.