A newly disclosed OpenSSL flaw named HollowByte allows an unauthenticated attacker to exhaust server memory using specially crafted TLS requests containing as little as 11 bytes of data.
The denial-of-service issue was discovered by the Okta Red Team, which found that vulnerable OpenSSL versions allocate memory based on an attacker-controlled size declaration before the full TLS handshake message has arrived.
Although OpenSSL has released a fix, the project classified the change as a bug or hardening improvement rather than a security vulnerability. HollowByte therefore has no CVE identifier, dedicated security advisory, or explicit entry in the affected release changelogs.
OpenSSL Trusts an Attacker-Controlled Length
Every TLS handshake message begins with a four-byte header. Three of those bytes specify the expected length of the message body that follows.
Older OpenSSL versions immediately expand their receive buffer according to that claimed length. The allocation occurs before the server receives the actual message body or verifies that the declared amount of data will ever arrive.
An attacker can exploit this behavior by opening a TLS connection and sending an 11-byte payload containing a header that claims a much larger message is coming.
The vulnerable server can allocate up to 131 KB for the connection and then leave a worker waiting for data the attacker never sends. No authentication, completed TLS session, or key exchange is required.
While 131 KB may appear relatively small, the attacker can repeat the request across large numbers of connections while transmitting very little data.
Freed Memory Remains Trapped
HollowByte becomes more disruptive because of the way the GNU C Library, or glibc, manages freed memory.
When a malicious connection closes, OpenSSL releases the allocated buffer. However, glibc may retain small and medium-sized allocations for future reuse instead of immediately returning that memory to the operating system.
By repeatedly changing the false message length, an attacker can create differently sized memory blocks that the allocator struggles to reuse. This causes the heap to fragment and the server’s resident memory usage to continue increasing even after the attacking connections have been terminated.
Okta said the memory remained unavailable until the affected process was completely restarted.
This distinguishes HollowByte from a conventional connection-exhaustion attack. Closing the connections may remove the immediate network load, but it does not necessarily restore the memory consumed during the attack.
NGINX Server Killed During Testing
Okta tested HollowByte against patched and unpatched OpenSSL instances running NGINX.
In a server environment with 1 GB of RAM, the unpatched process was killed after approximately 547 MB became trapped in fragmented memory.
During testing on a system with 16 GB of RAM, the attack locked up around 25% of the server’s total memory while remaining below its configured connection ceiling.
This means standard connection-limiting protections may not be sufficient. An attacker does not necessarily need enough simultaneous connections to trigger conventional denial-of-service alerts because memory can remain consumed after earlier connections have already closed.
The practical impact depends on the operating system, memory allocator, server configuration, available resources, and how the affected application uses OpenSSL. Okta’s published testing focused on NGINX running with glibc.
OpenSSL Quietly Shipped a Fix
OpenSSL resolved the issue by changing how the TLS receive buffer grows.
Instead of immediately trusting the declared message length and allocating the full buffer, OpenSSL now expands the buffer incrementally as data actually arrives over the connection.
A client that claims a large message is coming but sends no additional data can no longer force the same immediate allocation.
The correction was included in OpenSSL 4.0.1 and backported to the following releases:
- OpenSSL 3.6.3
- OpenSSL 3.5.7
- OpenSSL 3.4.6
- OpenSSL 3.0.21
The fixed versions were published on June 9 as part of broader OpenSSL security patch releases, but HollowByte was not identified in the public release notes.
OpenSSL’s own pull request states that the issue was reported to its security team by Okta and that the project decided to handle it only as a “bug or hardening” fix.
No CVE Complicates Detection
The lack of a CVE creates an additional problem for vulnerability management teams.
Many organizations depend on CVE identifiers, security advisories, and vendor changelogs to identify affected software. Because HollowByte has none of these, standard vulnerability scanners may not clearly warn administrators that an outdated OpenSSL package remains exposed.
OpenSSL is also embedded in a wide range of software, including Apache and NGINX web servers, Node.js and Python runtimes, PHP and Ruby environments, and databases such as MySQL and PostgreSQL.
Updating the operating system’s OpenSSL package may not address applications that ship their own bundled copy of the library.
Organizations should inventory both system and application-provided OpenSSL versions, update to a fixed release, and monitor unusual increases in resident memory on TLS-terminating services.
Systems that experience unexplained memory growth should also be checked for large numbers of incomplete TLS handshakes. Restarting an affected process may recover fragmented memory, but it does not prevent the server from being targeted again.
HollowByte demonstrates how a tiny unauthenticated request can create a disproportionate operational impact when foundational software allocates resources based on untrusted input.
