The hotel distribution industry treats zero-downtime deployment as a checkbox feature. Buy a load balancer, enable blue-green, call it done. The reality is harsher: without graceful shutdown, in-flight bookings are dropped. Without connection draining, active supplier sessions are severed mid-transaction. Without readiness verification, an unhealthy instance receives traffic before it has warmed its cache or connected to its database. The deploy script is green, but the customer experience is red.

HotelByte approaches zero downtime as a runtime discipline that spans four time-pressured phases: graceful shutdown of the old worker, readiness verification of the new worker, rollout evidence collection during promotion, and rollback readiness before the first byte of traffic shifts. Each phase has bounded timeouts, explicit failure modes, and automated recovery paths.

Time as the Enemy of Correctness

The core challenge is that “correct” changes definition as time passes. At T+0, the old worker is healthy and serving traffic. At T+1, the new worker starts but is not yet ready. At T+2, the old worker receives a shutdown signal but still has open connections. At T+3, Nginx upstream checks may remove either worker from rotation. At T+4, the deployment script classifies the outcome as SUCCESS, PARTIAL_SUCCESS, or ROLLBACK.

HotelByte’s Master/Worker process model manages these transitions with explicit bounds. The Master polls the new worker’s /ready endpoint at 1-second intervals for up to 120 seconds before declaring it healthy. The old worker receives a 30-second graceful shutdown window; if it hangs, force-termination ensures the port is released. Port availability is actively probed after unexpected exits before restart is attempted. These timeouts are not generous—they are conservative, chosen to prevent the system from entering an undefined state.

What the Stack Gives Up

The custom Master/Worker model replaces container-native orchestration primitives with a process-level supervisor. This means HotelByte does not rely on Kubernetes rolling updates or Docker Swarm restart policies for its core runtime safety. The trade-off is operational complexity: the Master process, signal handling, and port reconciliation logic must be understood and maintained by the platform team.

The 400-line change limit enforces small, reviewable diffs but constrains the scope of individual deployments. A large feature must be decomposed into multiple sequential deployments, each with its own verification gate. This is slower than a single big-bang release, but it prevents the class of failures where a massive diff introduces multiple interacting defects that are impossible to isolate.

The Four Phases in Detail

Graceful shutdown. When the old worker receives SIGTERM, it stops accepting new requests and waits for in-flight transactions to complete. The 30-second cap ensures that a hung worker does not block the port indefinitely. This is not a nice-to-have; in hotel distribution, an interrupted booking attempt can leave a reservation in an ambiguous state across multiple supplier systems.

Connection draining and readiness. The new worker must respond with HTTP 200 from /ready before the old worker is signaled to exit. The /ready endpoint reports application-level health: database connectivity, cache availability, and essential background workers. Until this succeeds, the new worker is invisible to the Nginx upstream. Nginx’s active health checks (nginx_upstream_check_module, 3-second interval) provide a second layer of defense, removing unhealthy workers before the Master detects the failure.

Rollout evidence. The deployment script probes /ready and /ping via both localhost (SSH tunnel) and external HTTP/HTTPS paths with retry logic. Network quality is assessed before file transfer. Pre-deployment backups are timestamped and rotated. The outcome is classified as SUCCESS, PARTIAL_SUCCESS, or ROLLBACK based on failure counts across non-backup servers. Gray deployments shift traffic incrementally (e.g., 1%) before full promotion.

Rollback readiness. If non-disaster-recovery nodes exceed the failure threshold, the deployment auto-rolls back to the backed-up version. The UAT canary gate requires version parity between staging and production before promotion, ensuring that the canary environment has already validated the exact binary being deployed.

Reading Path

Start with “Graceful Transitions” in the Design Principles section. It establishes the foundational idea that every lifecycle transition is a coordinated handoff, not an abrupt state change.

For the runtime mechanics, “Layer 1 — Master/Worker Process Layer” and “Layer 2 — Health Check Layer” describe the signal protocol, readiness polling, crash recovery, and Nginx upstream defense in depth.

For the deployment pipeline, “Deployment Lifecycle” walks through the eight-stage process from pre-flight gates through outcome classification, with explicit rollback triggers at each stage.

Cross-References