Why a Quiet TCP Reset Happens When One Service Closes While the Other Still Sends

Thesis

When a process closes a TCP socket while the peer still has unread data queued, the kernel may generate a RST packet. The reset propagates to the peer as ECONNRESET, even though both applications appear to run without crashes. The phenomenon, illustrated with a minimal C reproducer, mirrors the sporadic resets seen in production stacks such as nginx → gunicorn → Flask.

---

Key arguments

1. The minimal reproducer shows the reset is triggered by the server’s close()

• The server forks a child, writes 600 000 bytes with a single sendto(), then immediately calls close().
• The client, when started with --spam, first sends 100 bytes, then reads in a loop until recv() returns -1 and errno == ECONNRESET.
• tcpdump confirms a RST packet originates from the server right after it closes the socket.
• Strace of the server shows sendto() reports success (the kernel accepted the data into its send buffer) and the process exits cleanly; there is no explicit error path.

2. Why the kernel sends a RST on close

• TCP’s state diagram distinguishes between a full‑duplex close (FIN/ACK exchange) and a half‑duplex close where the application discards any further inbound data.
• RFC 1122, section 4.2.2.13, permits an implementation to reset the connection if data arrives after the local endpoint has called close() while unread data remains in the receive queue.
• The Linux kernel implements this rule: when a socket is closed with pending inbound data, it aborts the connection with a RST to inform the peer that the data will be lost.
• Adding a sleep(1) before close() in the server delays the RST, allowing the client to finish reading the buffered data and avoid the error, confirming the timing‑sensitive nature of the bug.

3. Real‑world manifestation in gunicorn‑nginx setups

• In production, an Nginx reverse proxy forwards an HTTP request to gunicorn. Nginx writes the request headers and body in two separate writev() calls.
• Gunicorn sometimes reads only the first recvfrom() (the headers) and never consumes the remaining body bytes because the application does not touch the request payload.
• Gunicorn then sends the response and closes the socket. If the client (Nginx) still has body bytes pending, the kernel on the gunicorn side emits a RST, which Nginx reports as ECONNRESET.
• The fix is to ensure the application drains the request body—e.g., by reading it into a dummy buffer—so that the socket is clean before close().

---

Implications

• Silent data loss: Applications that assume a successful send() guarantees delivery may be misled; the peer can still receive a reset after the sender thinks everything was sent.
• DoS surface: If a server discards large request bodies without reading them, a malicious client could force the server to generate many RSTs, consuming CPU and network resources.
• Observability: Monitoring tools that only look at exit codes may miss the underlying TCP reset; capturing RST packets or logging ECONNRESET on the client side provides a clearer picture.
• Configuration knobs: Web servers and reverse proxies often expose options to limit request size (client_max_body_size in Nginx) or to enforce full consumption of request bodies before closing.

---

Counter‑perspectives

• Some developers argue that a server should be free to close a connection once it has produced a response, regardless of pending inbound data, treating the client’s extra bytes as irrelevant. From this view, the RST is a reasonable signal that the client’s extra data will not be processed.
• Others point out that the TCP specification allows both a graceful FIN‑based shutdown and an abortive RST; the choice depends on the desired semantics. In high‑throughput services, aborting may be preferable to avoid lingering half‑closed connections that waste resources.

---

Conclusion and next steps

• Verify the hypothesis by reproducing the reset with other languages (e.g., Python’s socket.close() after a partial read) and by consulting the Linux kernel source (tcp_close_state()).
• Identify the exact component in the gunicorn stack responsible for the premature close—whether it is gunicorn itself, the Flask worker, or a middleware.
• Contribute a bug report upstream, referencing the relevant RFC sections and the minimal C example.
• In production, adopt defensive patterns: always read the full request body or explicitly shut down the read side with shutdown(sock, SHUT_RD) before closing, and configure request‑size limits to mitigate abuse.

For a deeper dive into the second part of this investigation, see the follow‑up article.