Certificates and domains

This guide is for operators maintaining HTTPS on a public installation. The certificate manager discovers active Worker and tenant storage hostnames, requests certificates through Certbot, and publishes them to HAProxy. Public hosting covers initial setup. Local HTTPS covers the separate development CA and rehearsal.

Operate the certificate lifecycle

The certificate_manager Ansible role is disabled by default. When enabled it:

• installs a pinned Certbot and the configured DNS provider plugin;
• discovers non-deleting organizations and active Worker routes from authenticated internal APIs, then maintains exact Worker hostnames plus each tenant’s R2 endpoint;
• maintains explicitly configured platform certificates such as Access;
• reconciles desired certificates on a short timer and runs ordinary Certbot renewal twice daily;
• publishes a renewed full chain and private key atomically, validates the full HAProxy configuration, and reloads HAProxy only after validation succeeds;
• restores the previously served bundle if validation or reload fails and keeps working certificates in place when identity, DNS, or the CA is unavailable;
• retires only publications recorded in its own manifest after an organization reaches the deleting state, then removes the corresponding Certbot lineage so deleted tenants are not renewed forever.

Disabling the role stops its timers but deliberately preserves Certbot state, private keys, and published bundles. Disabling automation is not certificate revocation and must not be used as an uninstall shortcut on live ingress.

The manager serializes reconciliation, explicit issuance, and renewal. Certbot deploy hooks remain separate so a successful issuance can publish while the parent operation owns that lock. Root-only state lives under /var/lib/ermahgerd-certificates; the latest reconciliation result is status.json, and detailed client logs live under /var/log/ermahgerd-certificates.

For manual inspection on the ingress host:

sudo systemctl list-timers 'ermahgerd-certificates*'
sudo systemctl start ermahgerd-certificates.service
sudo /usr/local/sbin/ermahgerd-certificate-manager reconcile
sudo cat /var/lib/ermahgerd-certificates/status.json

Do not run Certbot directly against the managed directories. The configured wrapper supplies the protected environment and the deploy hook performs safe publication.

Use the public single-node path

The latitude profile is the tested hosted path for ermahgerd.io. Public DNS points the apex, portal names, *.workers, and *.r2 to the ingress address. Those wildcard DNS records route arbitrarily nested names but do not imply wildcard TLS certificates. The certificate manager requests exact names for deployed Workers and tenant R2 endpoints, so the authoritative DNS service does not need an API token and the public CA can validate every order through HTTP-01.

make latitude-up
make latitude-status
make latitude-check

The first run begins on HTTP, deploys the curated applications through Wrangler, issues exact certificates, switches public application origins to HTTPS, and verifies them with normal system trust. Later Wrangler deployments are discovered by certificate reconciliation. A newly introduced hostname can have a short issuance window before it accepts TLS; existing names remain served.

The reference DNS set is:

The admin certificate does not publish the administrator portal: HAProxy’s public application router returns 404 for that hostname. Do not turn the DNS record into administrative exposure; put the listener behind VPN or an explicit operator allowlist.

Configure another hosted TLS listener

The control-plane renderer supports haproxy_tls_port, haproxy_tls_address, and haproxy_tls_certificate_dir in Ansible inventory. TLS is disabled when the port is null. Enabling it requires existing PEM certificate/key bundles and platform_external_scheme: https; public URL generation also uses control_api_external_port and worker_base_domain.

HAProxy loads the bundles, requires a matching SNI certificate, enforces TLS 1.2 or newer, redirects plain HTTP, and preserves the public HTTPS scheme through Access to the Worker. An externally reachable listener, firewall rules, portal origins, and secure portal cookies require separate operator configuration. Changing the scheme alone does not enable TLS or provision certificates.

For environments where HTTP-01 cannot reach ingress, put a provider plugin and credentials in encrypted inventory and enable both the manager and listener. This Cloudflare DNS example shows the alternative shape; pin the plugin version selected for the deployment and store credentials in Ansible Vault or an external secret source:

worker_base_domain: workers.example.com
platform_external_scheme: https
control_api_external_port: 443
haproxy_tls_port: 443
haproxy_tls_address: 0.0.0.0
haproxy_tls_certificate_dir: /var/lib/ermahgerd-certificates/published

certificate_manager_enabled: true
certificate_manager_email: operations@example.com
certificate_manager_certbot_packages:
  - certbot==5.8.0
  - certbot-dns-cloudflare==<validated-version>
certificate_manager_dns_credentials: !vault |
  <encrypted dns_cloudflare_api_token INI content>
certificate_manager_authenticator_args:
  - --dns-cloudflare
  - --dns-cloudflare-credentials
  - /etc/ermahgerd/certbot-dns-credentials.ini
certificate_manager_platform_certificates:
  access: [access.example.com]
  landing: [example.com]
certificate_manager_reconcile_on_install: true

Converge first against the CA’s staging directory by overriding certificate_manager_acme_server. Promote to the production directory only after issuance, renewal, publication, traffic, and recovery checks succeed. Provider credentials should be restricted to TXT changes in the platform’s validation zone; they do not need general account or application privileges.

The role handles software installation, root-only provider credentials, organization discovery, issuance, renewal, and HAProxy publication. Inventory still owns public addresses, DNS records, portal origins, firewall policy, and the decision to expose the administrator portal only on a VPN or private listener. Portal TLS termination is wired for the public landing, tenant console, status, and Access origins. Administrative ingress remains an operator network concern.

What remains before broader public hosting

The nested hostname scheme works with the exact-host certificates issued by the current manager. If you choose wildcard certificates instead, each tenant needs a name such as *.t-tenant.workers.example.com; a certificate for *.workers.example.com cannot cover its nested Worker names. A customer name such as blog.customer.example requires its own certificate and an authorized mapping to the tenant’s Worker. A CNAME alone neither changes the browser’s requested hostname nor grants permission to serve it.

The remaining gates before treating the reference installation as a broader multi-tenant service are:

1. Expiry/renewal alerting and operational backup/recovery for protected certificate state.
2. A private administrator listener and operational backup/recovery planning.

Customer hostnames

Tenants manage exact hostnames from Worker → Domains; see Custom domains for the DNS workflow. The control plane owns durable, organization-scoped claims and TXT verification. It checks every resolved A/AAAA address against the configured ingress addresses. Verified claims are globally unique; pending claims do not reserve someone else’s name.

Set custom_domains_enabled: true and custom_domain_addresses to the actual ingress IP addresses in inventory. TLS must already be configured. Latitude enables this with its persistent HTTPS state. Reserved administrator origins are passed explicitly, including private administration hostnames.

The existing certificate manager discovers verified claims through its private API and requests one exact certificate per custom hostname using HTTP-01. It publishes through the normal validated HAProxy deploy hook, then acknowledges certificate coverage and expiry to the control plane before route activation. Each custom-host failure is isolated, with an hour before another issuance attempt. Certificate renewal and retirement use the same managed Certbot directories as platform certificates. Do not invoke a second unmanaged Certbot installation for customer domains.

Ingress preserves the requested hostname and selects the same tenant, Worker, version, metering, and Access backend as the platform address. Access verifies custom callback origins with the control plane and binds its one-time code to the browser state and originating hostname. Protected gateway traffic also requires the private ingress credential; client-supplied route headers are removed at public ingress.

On the prepared local HTTPS profile, make domains-check exercises private TXT verification, real HTTP-01 issuance and renewal, verified TLS, routing, and a protected application login. It also checks certificate retirement. Its temporary customer.test mappings and records are removed afterwards. make domains-browser-check covers the console setup, refresh-safe selection, and deletion confirmation; prefix it with UPDATE_DOC_SCREENSHOTS=1 to refresh the guide’s screenshot. Do not run either alongside provisioning or another HTTPS rehearsal. No public DNS is modified by these tests.

Public DNS-01 validation can be tested while the application remains private: the CA checks a public TXT record rather than contacting the application. See Let’s Encrypt challenge types and its staging environment. These are future validation steps, not work performed by make https-check.

The local check uses Pebble, a test-only ACME server, with the role’s pinned upstream Certbot client. Pebble is deliberately ephemeral and does not reproduce public CA rate limits or every validation policy. Its DNS challenge server is a fixture, not a production authoritative DNS service. Passing locally is meaningful protocol and application evidence, not certification that the platform is ready for public multi-tenant hosting.

The current manager groups exact tenant names into bounded certificates. Public CA registered-domain and duplicate-certificate limits still affect how quickly many organizations and routes can be created. Use staging during development, pace production onboarding, monitor the CA’s current published limits, and revisit certificate grouping or issuer choice before high-volume signup. Do not encode a historical Let’s Encrypt number as a permanent platform quota.