Application instances can be stopped deliberately. For example, application instances running in a cluster may be started and stopped based on auto-scaling behaviors.
Application shutdowns consider the need to complete work in progress by listening for SIGTERM, SIGKILL and os.Interrupt ‘events’ in a dedicated goroutine which we will refer to here as the ‘shutdown detector goroutine’.
When a signal of interest is detected, the shutdown detector goroutine passes a value through a channel being listened to in the main services loop.
Upon receiving the channel value indicating a shutdown, the application instance sends a DEPARTING message to all group-members.
Each group-member updates the process status of the departing application instance to DEPARTED upon receiving the DEPARTING message.
DEPARTING messages contain a double-incremented (+=2) incarnation number and override ALIVE and SUSPECT statuses in the receiving application instance’s local group-membership list.
The departing instance calls a Server.Shutdown(…) method with a timeout context of 5 seconds to permit work in progress to complete.
The departing instance then shuts down.
While the departing instance sends a DEPARTING message to all application instances that it knows about, this may not constitute the complete group-membership.
Processes disseminate DEPARTED statuses of departed peers for a few ping cycles in order to ensure that the information is made available as widely as possible. After a short period of dissemination, each running application instance purges the DEPARTED application instance information from its local group-membership list.