..

The AWS S3 client fix that caused a new bug

7 min read

I recently chased down a slow memory leak in a Java service. The fix was simple and it worked. But it also uncovered a second problem that only showed up because the first fix was correct. I’m writing both down because the pair is a good reminder that “the recommended way” can have consequences you need to plan for.

The service uses the AWS SDK for Java v2 to fetch objects from S3.

The symptom

Memory on some instances kept climbing over time. Others stayed flat. When I lined the memory graphs up against traffic, the pattern was clear. The instances doing more S3 fetches grew faster. Restart the process, and the climb started again.

That shape, memory that goes up and never comes back down in proportion to how much work an instance does, usually means we’re creating something per request and never releasing it.

Root cause: a new S3 client on every request

Here’s a simplified version of what the code did:

public S3Client getS3Client() {
    var credentials = AwsBasicCredentials.create(accessKey, secretKey);
    var provider = StaticCredentialsProvider.create(credentials);

    return S3Client.builder()
            .credentialsProvider(provider)
            .region(Region.of(region))
            .build();   // a brand-new client every time this is called
}

Every S3 fetch called this method and got a fresh S3Client. That looks harmless, but an S3Client is not a lightweight object. Each one owns:

  • an HTTP connection pool (real open sockets),
  • native resources and file descriptors for those sockets,
  • some internal bookkeeping inside the SDK.

The part that surprised me is that dropping your reference to the client is not enough to free it. The client’s connection manager is registered with long-lived SDK internals, including a background thread that cleans up idle connections. That internal reference keeps the whole client reachable, so the garbage collector never collects it. To actually release the resources you have to call close(). AWS calls this out in its own docs, noting that letting the SDK manage the client’s lifecycle “helps avoid potential memory leaks if the ApacheHttpClient instance is not closed down when it’s no longer needed.” (AWS SDK for Java 2.x docs)

So every fetch left behind a client that could not be garbage-collected. The sockets and memory piled up. Busy instances piled up faster.

The fix

Create the client once and reuse it. This is what AWS recommends anyway. The SDK client is thread-safe and meant to be a long-lived singleton, and the AWS docs show exactly this with the comment // Singleton: Use the s3Client for all requests. (AWS SDK for Java 2.x docs)

// built once, e.g. in a constructor / as a singleton bean
this.s3Client = S3Client.builder()
        .credentialsProvider(provider)
        .region(Region.of(region))
        .build();

I deployed it and the leak was gone. Memory flattened out.

The new bug: “The target server failed to respond”

A while after the fix, a new error started showing up. Not on every request, just occasionally, and usually on the first request after the service had been quiet for a bit:

software.amazon.awssdk.core.exception.SdkClientException:
    Unable to execute HTTP request: The target server failed to respond
Caused by: org.apache.http.NoHttpResponseException: The target server failed to respond

I had made the code more correct and a new failure appeared. That took a minute to make sense of.

Why the fix caused it

It comes down to how connections get reused.

Before, when we made a new client per request, each request had its own empty connection pool and almost always opened a fresh TCP connection. Connections were thrown away so quickly that they never had a chance to go stale. The problem was there in theory, but we never hit it.

After the fix, the shared client keeps a persistent connection pool and reuses connections across requests. That’s normal HTTP keep-alive, and it’s the efficiency win we wanted. But it also means a connection can sit idle in the pool between requests.

Meanwhile, the other end of that connection (S3, or something in between like a load balancer or NAT gateway) closes idle connections after its own timeout. It does this quietly. Our pool doesn’t get told. So the next request grabs a connection the pool thinks is fine, sends data on it, and gets nothing back.

Here’s the sequence:

client                                  server / load balancer / NAT
  |-- request 1 ------------------------------->|
  |<------------- response 1 -------------------|
  |  (connection goes back to pool, idle)       |
  |            ... idle too long ...            |
  |            server quietly closes its side   |
  |  (pool still thinks the connection is open) |
  |                                             |
  |-- request 2 on that dead connection ------->|
  |<--- nothing comes back ---------------------|
  |-- NoHttpResponseException-------------------|

There’s a library detail underneath this too. The SDK’s sync client uses Apache HttpClient 4.x. Since version 4.4, Apache changed the automatic “is this connection still alive?” check before reuse: it “now only checks the connection if the elapsed time since the last use of the connection exceeds the timeout that has been set. The default timeout is set to 2000ms.” (PoolingHttpClientConnectionManager javadoc) Their recommended alternative is a dedicated monitor thread that evicts idle connections, because “the stale connection check is not 100% reliable.” (Apache HttpClient connection management) If you don’t configure it, you can be handed a dead connection.

The fix

Tell the client to evict idle connections and cap how long any connection lives. This is the mitigation Apache itself recommends, and the AWS SDK exposes it through the ApacheHttpClient.Builder options. (ApacheHttpClient.Builder javadoc)

var httpClientBuilder = ApacheHttpClient.builder()
        .connectionMaxIdleTime(Duration.ofSeconds(20)) // close idle conns early
        .connectionTimeToLive(Duration.ofMinutes(5))   // cap total conn age
        .useIdleConnectionReaper(true);                // the eviction thread

this.s3Client = S3Client.builder()
        .credentialsProvider(provider)
        .region(Region.of(region))
        .httpClientBuilder(httpClientBuilder)
        .build();

connectionMaxIdleTime together with useIdleConnectionReaper is the main lever. It closes idle connections before the other end does. The one rule that matters is that this value has to be lower than the idle timeout of whatever sits between you and S3, such as a load balancer or NAT gateway. If it isn’t, the eviction happens too late and you still get handed dead connections.

connectionTimeToLive is a backstop. It caps how old any connection can get, which helps when something force-closes connections after a fixed lifetime.

One more thing: a retry

Idle eviction isn’t a 100% guarantee. The eviction thread runs on a schedule, so there’s still a tiny window where a connection could go stale and get used before the next sweep. The clean way to cover that gap is a retry. A stale-connection failure on a read like getObject is safe to retry, since the retry just opens a fresh connection and succeeds. The AWS SDK has retries enabled by default and already treats transient errors and socket timeouts as retryable, so this is mostly about confirming your retry strategy is configured the way you expect. (AWS SDK for Java 2.x retry docs)

Takeaways

AWS SDK clients are meant to be reused. Build one and keep it. Don’t create one per request, because they hold real resources and won’t be garbage-collected until you close them.

Once you reuse connections, stale connections become your problem. Idle connections that the other side has already closed will bite you eventually. Configure idle eviction and a connection TTL.

Set the idle timeout below your infrastructure’s timeout. This is the part that’s easy to get wrong. The numbers only work if yours is the smaller one.

And a fix that seems to reveal a new bug often didn’t create it. The stale-connection problem was always possible. The old, wasteful code just hid it by never reusing anything.

References

  1. AWS SDK for Java 2.x, “Configure the Apache-based HTTP client” (singleton usage and the memory-leak note)
  2. Apache HttpClient, PoolingHttpClientConnectionManager javadoc (the 4.4 change to stale-connection checking and the 2000ms default)
  3. Apache HttpClient 4.5.x tutorial, “Connection management” (the eviction policy and “not 100% reliable” note)
  4. AWS SDK for Java 2.x, ApacheHttpClient.Builder javadoc (connectionMaxIdleTime, connectionTimeToLive, useIdleConnectionReaper)
  5. AWS SDK for Java 2.x, “Configure retry behavior” (retries enabled by default; transient errors and socket timeouts are retryable)