OpenSSL Vulnerability: HollowByte Flaw Can Indefinitely Freeze Server Memory

| Editorial Team On 21 Jul, 2026

OpenSSL HollowByte Flaw Could Freeze Server Memory with Just 11-Byte TLS Requests

Okta's Red Team has uncovered a denial-of-service vulnerability in OpenSSL that allows an attacker to permanently fragment server memory using an 11-byte TLS request — and OpenSSL shipped the fix in June without a CVE, an advisory, or a single changelog entry pointing to it.

The flaw, named HollowByte by the researchers who found it, exposes a fundamental gap between how a bug is classified and how damaging it can actually be in production. For the millions of servers running unpatched OpenSSL builds, the risk is real and largely invisible — because nothing in a standard patch pipeline will flag it.

How HollowByte Works

Every TLS handshake message carries a 4-byte header. Three of those bytes declare the length of the body to follow. Vulnerable versions of OpenSSL allocated memory equal to that declared size the moment the header arrived — before the body appeared and before any handshake validation ran.

For an inbound ClientHello message, that ceiling is 131 KB. The worker thread then blocks, waiting on a body that never comes. No authentication. No session. No key exchange required.

On its own, that describes a classic connection-exhaustion attack — a technique as familiar to security engineers as Slowloris, the 2009 slow HTTP attack tool that paralyzed servers by holding connections open indefinitely. Understanding the different types of denial-of-service and DDoS attack techniques helps clarify why HollowByte sits in a category of its own. What makes HollowByte more dangerous is what happens after the attacker disconnects.

When the connection drops, OpenSSL frees the buffer. But on systems running glibc — the standard C library on most Linux distributions — freed memory is held for reuse rather than returned to the kernel. Okta's testing showed that by varying the declared size on every connection, an attacker prevents the allocator from reusing what it previously freed. The heap fragments. Resident set size climbs. And it stays climbed long after the attack ends.

The Scale of the Damage

In Okta's NGINX tests, a 1 GB server was killed by the out-of-memory manager with 547 MB of memory locked in fragments. On a 16 GB server, HollowByte froze 25% of system memory without ever triggering connection-limiting defenses.

"Standard connection-limiting defenses won't stop it," Okta's Red Team stated in its disclosure published Thursday.

This is a critical distinction. Most server operators rely on connection rate limiting and timeout configurations as their primary defense against slow-connection attacks. HollowByte sidesteps both. The memory damage accumulates across connections rather than within them, meaning the attack can be distributed across time and source addresses without losing effectiveness. The fragmentation persists independently of whether any single connection is blocked or dropped.

Okta published no exploit code alongside its findings. As of July 18, The Hacker News found no public proof-of-concept repository on GitHub.

Why OpenSSL Didn't Issue a CVE

OpenSSL shipped the HollowByte fix across five releases — versions 4.0.1, 3.6.3, 3.5.7, 3.4.6, and 3.0.21 — all dated June 9, 2026. Every release on those branches prior to those versions carries the flaw.

Despite that scope, OpenSSL's security team classified the fix as a "bug or hardening" issue. OpenSSL's own security policy defines four severity tiers ranging from Critical to Low. Even a Low-severity finding earns a CVE number, a changelog note, and an entry on the vulnerabilities page. HollowByte received none of the three.

For context, the same June 9 release assigned CVE-2026-34183, rated Moderate, to unbounded memory growth in the QUIC PATH_CHALLENGE handler — also a memory-exhaustion denial-of-service. A January fix for a TLS 1.3 certificate-compression bug earned CVE-2025-66199, rated Low, despite requiring four conditions to align before it could be exploited. HollowByte requires none.

OpenSSL's Justification — and Its Limits

Alexandr Nedvedicky, the OpenSSL developer who triaged HollowByte, responded to The Hacker News after initial publication. He drew a distinction between HollowByte and the QUIC flaw that received a CVE. The QUIC issue is a protocol problem because the code sets no limit on PATH_CHALLENGE frames. HollowByte, he argued, stems from a server left in blocking mode with no limits configured — "closer to a deployment choice than a flaw in the protocol."

He also noted that he triaged the report on OpenBSD, which does not use glibc, and said he had "not considered any effects possibly coming from glibc." That behavior is the center of Okta's case. He added uncertainty about whether the shipped fix resolves the fragmentation problem at all, noting that patch author Matt Caswell's fix still uses realloc and that he was "not sure if it improves anything for glibc."

This is where the classification decision becomes consequential. The severity of HollowByte is not purely a function of the code path — it is a function of the runtime environment. A triage process that excluded glibc behavior from consideration assessed a different attack surface than the one most production servers actually present. Whether that constitutes a protocol flaw or a deployment condition is a reasonable debate. Whether it warrants a CVE given the documented production impact is a harder case to dismiss.

Okta recommends upgrading regardless.

What Administrators Need to Do Now

The absence of a CVE creates a practical problem for every organization that depends on scanner alerts or advisory feeds to drive patching decisions. There is no identifier for a scanner to match and no advisory to read.

Downstream Linux distributions compound the difficulty. Red Hat's documented default is to backport fixes rather than advance version numbers, meaning a patched package can still report a pre-fix version string. What normally resolves that ambiguity is an advisory and an OVAL feed keyed to a CVE name. Neither exists here.

Robust website and application security practices include verification steps that go beyond automated scanner results — and HollowByte is a precise example of why that matters. Administrators who need to verify coverage should check for the relevant pull requests directly:

PR 30792 covers the master and 4.0 branches
PR 30793 covers 3.6, 3.5, and 3.4
PR 30794 covers 3.0

Those building OpenSSL from source should upgrade to the listed releases and restart any process that loaded the previous version.

The DTLS Gap That Remains Open

One additional exposure remains open. The fix covers TLS only. Caswell noted on the pull request that DTLS was left unaddressed because a proper fix would have been "far more invasive." A comparison of OpenSSL source at the 3.6.2 and 3.6.3 tags confirmed the DTLS handshake file is byte-identical across the fix. In version 4.0.1, the DTLS path still sizes its receive buffer from the length the peer declares. OpenSSL has not classified that path or committed to fixing it.

Whether HollowByte reaches the extended-support 1.1.1 and 1.0.2 branches remains unanswered. Nedvedicky declined to comment on that question and directed inquiries to OpenSSL's support portal.

Memory Monitoring as a Defense Layer

Given that the fragmentation effect survives the attack itself, patching alone is not sufficient as an operational response. Administrators should treat memory monitoring as an active defense layer, not a passive reporting tool. Establishing baseline resident set size for OpenSSL-linked processes — NGINX, HAProxy, custom TLS endpoints — and alerting on sustained deviation provides detection coverage that connection-rate monitoring will miss entirely. Where process restarts are operationally feasible on a scheduled basis, they serve as a practical mitigation for fragmentation accumulation on systems that cannot be immediately patched.

The role of encryption in protecting data in transit makes OpenSSL one of the most widely deployed security dependencies in existence. That ubiquity is precisely what raises the stakes when a flaw of this nature goes unclassified.

HollowByte illustrates three realities security teams can act on immediately. The absence of a CVE does not mean the absence of risk — patch verification processes should include direct source and changelog review for critical dependencies like OpenSSL. glibc memory behavior can turn a bounded allocation bug into a persistent resource drain that survives the attack itself, making restart procedures and memory monitoring as important as the patch. DTLS exposure on the same codebase remains unresolved and unclassified, and any organization running DTLS-enabled services should treat that path as unpatched until OpenSSL states otherwise.