Operations and monitoring

Once Areca is running, you need to know whether it is healthy, whether releases are moving, and where to look when something fails. This page covers the endpoints, metrics, logs and background work the CMS exposes. For the environment variables that control them, see configuration. For what happens to a release in the queue, see the publishing pipeline.

Health endpointsCopy Anchor Link

Two endpoints report health. Neither needs a sign-in.

The Docker image uses /api/health for its built-in health check, every 30 seconds. Liveness only shows that the process is responding. Restarting a container does not fix a database or storage outage, so the built-in check does not use readiness. If you want a load balancer to stop sending traffic to an instance while a dependency is down, point it at /api/health/ready.

The readiness response lists each check. Responses contain only names, pass or fail status, timings and short, redacted reasons. Full errors are in the logs.

Two flags mean different things. ok is false only when a check fails outright, and then the response is 503. degraded is true when something needs attention but the instance can still do its work. A degraded response is still 200, and the top-level degraded flag and each check's degraded flag show what needs attention.

{
  "ok": false,
  "degraded": true,
  "checks": {
    "db": { "ok": true, "ms": 4 },
    "storage": { "ok": false, "ms": 3001, "reason": "CheckTimeout: timed out after 3000ms" },
    "redis": { "ok": true, "ms": 2, "degraded": true, "reason": "ECONNREFUSED" },
    "worker": { "ok": true, "ms": 0, "detail": { "enabled": true, "tickAgeSeconds": 1 } },
    "queue": { "ok": true, "ms": 5, "detail": { "queued": 0, "active": 1, "oldestQueuedAgeSeconds": null, "expiredLeases": 0, "schedulerAgeSeconds": 2 } },
    "mail": { "ok": true, "ms": 3, "degraded": true, "reason": "mail waiting 22 min to send (limit 15 min)", "detail": { "pending": 4, "oldestPendingAgeSeconds": 1320, "expiredLeases": 0, "notificationEventsPending": 0, "storageDeletionsPaused": 0, "schedulesExhausted": 0 } },
    "loops": { "ok": true, "ms": 0, "detail": { "staleLoops": 0 } }
  }
}

The checks are:

• db runs a trivial query against PostgreSQL.
• storage checks that the S3 bucket is reachable.
• redis pings Redis. When Redis is down, rate limits count in memory on each instance, so a Redis failure is reported as degraded and does not make the instance unready.
• worker fails if the publishing worker has not run for longer than READINESS_MAX_TICK_AGE_SECONDS. It is checked only when WORKER_ENABLED is on. Each process reports its own worker state.
• queue fails if the oldest queued build is older than READINESS_MAX_QUEUE_AGE_SECONDS, if a build lease has been expired for longer than the tick age without recovery, or if the scheduler has stopped ticking.
• mail is degraded when the oldest queued email has waited more than 15 minutes. Test sends are not counted. It never fails readiness, and a database outage is reported by db.
• loops is degraded when a worker loop has not finished a run without error for five times its interval. The mail and notification loops run every 5 seconds, so they are stale after 25 seconds. The worker runs every 3 seconds, the scheduler, recovery and cleanup loops every 15 seconds, and the hourly maintenance loop every hour. The response names the stale loops. Only processes with the worker on have loops.

Each check has a time limit set by READINESS_TIMEOUT_MS.

Readiness is degraded, not failing, in four cases: Redis is down, the oldest queued email is older than 15 minutes, a worker loop is stale, or the check itself could not run. Use the metric areca_readiness_check_up to alert on those. It is 0 for a degraded check, as well as for a failing one.

MetricsCopy Anchor Link

GET /api/metrics returns Prometheus text. It is off by default. Set METRICS_TOKEN to a value of at least 16 characters, for example from openssl rand -hex 32, and restart the CMS. Without a token the endpoint returns 404.

Requests must send the token as a bearer credential:

• A missing or wrong token returns 401.
• The endpoint is rate limited to 60 requests per minute per client IP. Keep your scrape interval at 15 seconds or more.
• Behind a reverse proxy, every request can appear to come from the proxy's address, so all scrapes share one bucket. Set RATE_LIMIT_TRUST_PROXY=true only when the proxy overwrites the forwarding headers itself. The stock Compose file does not pass this variable; see production setup.

Store the token in a file that only your Prometheus server can read:

scrape_configs:
  - job_name: areca-cms
    metrics_path: /api/metrics
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/areca-metrics-token
    static_configs:
      - targets: ['cms.example.com']

Metrics referenceCopy Anchor Link

Most metrics are gauges read from the database when Prometheus scrapes, so the values stay correct across restarts and multiple instances.

Suggested alertsCopy Anchor Link

These rules are a starting point. Adjust the thresholds to match your release cadence.

groups:
  - name: areca-cms
    rules:
      - alert: ArecaNotReady
        expr: areca_ready == 0
        for: 5m
        annotations:
          summary: "CMS readiness failing ({{ $labels.instance }})"
      - alert: ArecaDependencyDown
        expr: areca_readiness_check_up{check=~"db|storage|queue|worker"} == 0
        for: 5m
        annotations:
          summary: "CMS check {{ $labels.check }} is down"
      - alert: ArecaBuildQueueStuck
        expr: areca_build_jobs_oldest_queued_age_seconds > 900
        for: 5m
        annotations:
          summary: "A publish job has waited over 15 minutes"
      - alert: ArecaBuildJobsFailing
        expr: increase(areca_build_jobs_failed_total[1h]) > 3
        annotations:
          summary: "More than 3 build jobs failed in the last hour"
      - alert: ArecaSchedulerStalled
        expr: areca_scheduler_last_tick_age_seconds > 300
        annotations:
          summary: "Scheduler has not completed a tick in 5 minutes"
      - alert: ArecaScheduledEntriesOverdue
        expr: areca_scheduled_entries_overdue > 0
        for: 30m
        annotations:
          summary: "Scheduled entries are overdue by more than 30 minutes"
      - alert: ArecaRedisDegraded
        expr: areca_readiness_check_up{check="redis"} == 0
        for: 10m
        annotations:
          summary: "Redis unreachable; rate limiting is failing open"

Alerts on ArecaBuildQueueStuck and ArecaBuildJobsFailing tell you a release needs attention. Start with the publishing pipeline troubleshooting steps.

LogsCopy Anchor Link

The CMS writes one JSON object per line:

{"level":"error","time":"2026-10-01T13:30:20.823Z","component":"worker","msg":"job failed","jobId":"...","err":{"name":"Error","message":"...","stack":"..."}}

Set LOG_LEVEL to debug, info (the default), warn, error or silent. The stock Compose file does not pass it. Debug and info lines go to stdout. Warnings and errors go to stderr, so you can route them separately. Fields named level, time, component or msg are reserved. If a field uses one of those names, it is written with a field_ prefix.

Areca redacts secrets from logs on a best-effort basis. It removes values from variables whose names contain words such as SECRET, TOKEN, PASSWORD, API_KEY, ACCESS_KEY, ENCRYPTION_KEY or CREDENTIAL, the password in database URLs, and authorization headers. Do not rely on this to keep secrets out of your log pipeline. Keep log access limited to the people who run the instance.

Find an error from the interfaceCopy Anchor Link

When something unexpected fails, the admin interface shows Something went wrong (ref: <id>). The same reference appears in the server log next to the full error, on a line with the message unhandled server function error. Use the reference to find the matching log line.

Build logsCopy Anchor Link

Each release has a build log, which you can read from the release details. Each line is limited to 8 KiB, and each job's log to 256 Ki characters. When a log goes over the limit, the oldest lines are dropped, the log starts with a [truncated] marker, and the most recent output is kept.

StartupCopy Anchor Link

The container starts in four steps, and each waits for the one before it:

1. It waits for PostgreSQL. Each attempt has a 5-second connection and query limit.
2. It applies database migrations while holding a PostgreSQL advisory lock. Replicas never migrate at the same time. A replica that finds the lock taken waits for the other to finish, then finds nothing left to apply.
3. It makes sure the S3 bucket exists. See object storage.
4. It starts the server, then sends it one request to /api/health. The server loads its configuration on that first request, so a 500 there, for example when REDIS_URL is missing in production, stops the container with code 1 and leaves the server's error in the log.

These waits share one deadline set by STARTUP_DEADLINE_SECONDS (default 60). Retries back off from one second up to five seconds, and each retry logs the dependency, the attempt and the time left. When the deadline passes, the container logs startup failed, exiting with the reason and exits with code 1, so your orchestrator can restart it or show it as failed. A migration that has already started runs to completion, because the deadline does not interrupt it. A replica waiting for the migration lock does count against the deadline, so raise STARTUP_DEADLINE_SECONDS if a long migration makes other replicas time out.

Object storageCopy Anchor Link

Files stream from S3 to the browser. The CMS does not hold a whole file in memory to serve it. The S3 client has fixed time limits, which are not set by an environment variable:

A download that hits a limit fails its request, and the browser can retry it. A failed delete goes back into the durable cleanup queue. After four failed attempts it pauses until someone retries it, as described in paused jobs.

Bucket creation. Startup creates S3_BUCKET only when S3 definitively reports that it does not exist (a 404, NotFound or NoSuchBucket answer). Wrong credentials (403), server errors and network failures are never read as "missing". Startup retries those until the deadline and then exits, and nothing is created. Readiness checks the bucket with a read-only request and never creates it. pnpm s3:init follows the same rule.

RedisCopy Anchor Link

Production requires REDIS_URL. The CMS refuses to start without it, because Redis holds the rate-limit counters that every instance shares.

If Redis cannot be reached while the CMS runs, each instance counts requests in its own memory with the same budgets. It retries Redis every five seconds. The log records when counting moves to memory and when Redis returns. During an outage a client can use up to one budget per running instance. Readiness reports redis as degraded and does not fail.

Content Security PolicyCopy Anchor Link

Every response carries a Content-Security-Policy with a new nonce for each request, and the framework applies that nonce to the scripts and styles it writes. The policy allows:

• Scripts from this origin, and scripts with the request's nonce. There is no unsafe-inline and no unsafe-eval.
• Stylesheets from this origin and with the nonce, Google Fonts (fonts.googleapis.com) for the branding preview, and one hashed stylesheet written by the toast library.
• Inline style attributes. React writes them for layout and transitions, and they cannot run script.
• Fonts from this origin and fonts.gstatic.com, images from this origin and data: or blob: URLs, and connections to this origin only.

CSP_REPORT_ONLY=true sends the same policy as Content-Security-Policy-Report-Only. Browsers then log violations and block nothing. Use it to see what a change would break before you enforce it.

For operators, the practical effects are these. Uploaded files load, because the CMS serves them from /api/assets/file/. An external image URL pasted into rich text does not load in the editor until its file is in the library. The CMS interface cannot load third-party scripts or widgets. To find a violation, open the browser console and look for Content Security Policy messages, which name the directive and the blocked source. The development server (vite dev) uses a relaxed policy so that hot reload works. Production builds never send it.

Several instancesCopy Anchor Link

Run as many replicas as you need. The work they share is coordinated in PostgreSQL:

• Migrations run under an advisory lock (see startup).
• Builds are claimed with a 60-second lease that the running worker renews. A claim is fenced: a worker whose lease expired cannot record its result, and the job stays queued for the next tick. Recovery fails the jobs left behind by a crashed process. A job runs for one site at a time, and shared Dokploy application deployments are serialized.
• Email is leased per message for 60 seconds and renewed while it is sent. A lapsed lease is reclaimed by another replica. areca_mail_expired_leases counts the messages waiting to be reclaimed.
• Storage cleanup claims each object before deleting it, so two replicas never delete the same object.
• Retention works in bounded batches, and overlapping runs remove different rows.

Two things stay per process: the rate-limit counters during a Redis outage, and each process's worker heartbeat. Readiness and the areca_worker_* metrics describe the process that answered, so each replica reports its own loops. Set WORKER_ENABLED=false on replicas that should only serve pages.

Worker and graceful shutdownCopy Anchor Link

The publishing worker and the scheduler run inside the CMS process. They are on by default. Set WORKER_ENABLED=false only during maintenance, or when you deliberately want to pause publishing and scheduled releases.

WORKER_CONCURRENCY sets how many build jobs one process runs at a time, from 1 to 16 (default 2). Each site can have only one job running at a time, and the database enforces that across all instances. A slow deployment, which can take up to 15 minutes, does not hold up other sites or the scheduler.

Areca handles SIGTERM and SIGINT like this:

1. It stops taking new jobs.
2. It waits up to WORKER_DRAIN_TIMEOUT_MS (default 8000) for running jobs to finish.
3. It exits with code 0 if all jobs finished, or 1 if some were still running. A second signal stops it immediately.

A job still running at exit keeps its lease. After restart, the lease expires and the job is marked failed. If the job had already sent its deploy request, the release is marked uncertain and must be reconciled. See uncertain deployments.

Keep WORKER_DRAIN_TIMEOUT_MS below the container's stop grace period. Docker's default is 10 seconds, so the default drain of 8 seconds fits. If you need a longer drain, raise the stop grace period in your orchestrator as well.

Run one CMS instance, as described in production setup. The database prevents two jobs for the same site, but running several instances is not a tested setup.

Database timeoutsCopy Anchor Link

Three settings stop a slow or abandoned database operation from blocking the rest of the system:

Setting a value to 0 turns that timeout off. Database migrations run at container startup and are not subject to these limits.

Routine maintenanceCopy Anchor Link

These tasks keep an instance healthy over time:

• Back up the database and the encryption keys together. Areca does not schedule backups for you. Take a daily dump with the bucket copy, a full backup before every upgrade, and a quarterly restore drill. The dump and the bucket must come from the same day, and losing SECRETS_ENCRYPTION_KEY makes the saved SMTP password unreadable. See backup and recovery.
• Check readiness after each deploy or restart. Confirm /api/health/ready returns 200, and that the queue is empty or moving.
• Review failed releases. A failed release leaves the previous release live, but it needs a cause and a fix before the next Go live. See the publishing pipeline.
• Rotate credentials on a schedule. The database password, the S3 access keys, the site tokens and the Dokploy API keys all have a rotation procedure. See secrets.
• Clean up stored files carefully. Storage objects that are still referenced by a draft, release or archive are protected. Follow asset retention.
• Upgrade with a backup in hand. Read upgrades before you change the image version.

How long data is keptCopy Anchor Link

A background job removes old rows once an hour. Sessions, tokens, invitations, build jobs and autosaves are removed in batches of 1,000 rows. The periods are fixed in the CMS:

Releases are never removed, because rollback needs them. Stored files follow asset retention. Paused work is retried hourly, see paused jobs.

RelatedCopy Anchor Link

• Configuration: every variable named on this page.
• Publishing pipeline: the queue, the worker and recovery.
• Troubleshooting
• Capacity: sizing guidance.