TLS negotiation failures can occur on either side of a reverse proxy, even when the public website certificate looks correct. Transport Layer Security, or TLS, protects connections through encryption and identity checks. A proxy can accept one protected connection, then create another connection with different settings.
This guide focuses on finding which connection fails and why. It covers protocol versions, cipher compatibility, server names, certificate validation, clocks, and proxy termination. Start with evidence rather than replacing certificates or weakening security settings.
Map Each TLS Connection Before Testing
A reverse proxy accepts requests and forwards them to an application server. TLS termination means the proxy handles the incoming encrypted connection. It may then contact the backend through plain HTTP or a separate TLS session.
For example, a browser might connect to a public proxy while that proxy connects securely to an internal application. Each connection has its own protocol negotiation, certificate presentation, and trust requirements.
- Identify the client, public hostname, destination port, and resolved address.
- Locate the device that terminates the public TLS session.
- Record whether the proxy uses HTTP or HTTPS upstream.
- For HTTPS upstreams, record the configured backend name and trust source.
- Include load balancers and inspection devices that create additional TLS sessions.
Keep public and backend settings separate throughout the investigation. A successful browser handshake proves nothing about the proxy’s connection to the application.
Locate the Failing Handshake With Matched Evidence
Capture one failed request with its timestamp, time zone, client address, and exact error. Compare that event with proxy and backend logs. Avoid drawing conclusions from unrelated log entries.
If the client receives an HTTP response, TLS succeeded on that client-facing connection. A gateway error may still reflect an upstream TLS failure. Conversely, a connection reset before any HTTP response needs closer investigation.
- No TCP connection: Check reachability, routing, firewall rules, and listening ports first.
- Connection established, then TLS alert: Inspect negotiation and validation evidence.
- Public handshake succeeds, upstream fails: Investigate the proxy-to-backend session.
- Only some requests fail: Compare backend nodes and address families.
Errors such as “handshake failure” describe a symptom, not a complete diagnosis. Record which endpoint reported the error and whether it sent or received the alert.
Run a Controlled TLS Negotiation Test
Use an approved diagnostic host with network access similar to the failing client. For backend tests, use the proxy host or an authorized equivalent. A workstation may have different routing and trust settings.
On systems with a suitable OpenSSL installation, this command tests a TLS listener with explicit server naming and hostname verification:
openssl s_client -connect HOSTNAME:443 -servername HOSTNAME -verify_hostname HOSTNAME -verify_return_error -showcertsReplace both hostname placeholders with the intended service name. Adjust the port only after confirming the service configuration. These tests create connections and may trigger monitoring or access controls.
Save the negotiated protocol, cipher, certificate chain, verification result, and any alert. OpenSSL’s trust store may differ from the application’s trust store. Therefore, a successful command does not prove that every client will succeed.
The -showcerts option displays certificates the server sends; it does not independently prove a valid chain. Keep private keys, passwords, and unrelated customer data out of diagnostic records.
Check Protocol and Cipher Compatibility Separately
The client and server need a common TLS version and compatible cryptographic choices. A cipher suite specifies cryptographic algorithms for the connection. TLS 1.3 handles some choices differently from TLS 1.2.
Compare the client’s supported versions with the listener’s enabled versions. Where your installed OpenSSL supports them, repeat the baseline test using -tls1_2 and -tls1_3 separately. Change only that test condition.
A version-specific failure narrows the investigation but does not identify the final cause. Cipher suites, signature algorithms, key-exchange groups, and certificate key types can also affect compatibility.
In OpenSSL, -cipher controls TLS 1.2 and earlier cipher lists. The -ciphersuites option controls TLS 1.3 suites. Confirm local tool support before using either option.
Use the Mozilla SSL Configuration Generator as a configuration reference, not a blind replacement. Prefer updating obsolete clients over restoring deprecated protocols. Test proposed changes against actual business applications before deployment.
Verify SNI and Hostname Checks on Both Connections
Server Name Indication, or SNI, lets a client name the desired service during the handshake. A shared listener uses that name to choose its TLS configuration and certificate.
SNI and hostname verification serve different purposes. SNI requests a service identity. Hostname verification checks whether the certificate covers the expected name.
A proxy may connect to an internal IP address while expecting a certificate for an internal service hostname. It must send appropriate SNI and validate the intended identity. Setting an HTTP Host header alone does not configure TLS SNI.
- Compare the requested SNI with the intended virtual host.
- Check the certificate’s Subject Alternative Name entries.
- Confirm the proxy’s upstream verification name.
- Check every backend node for consistent listener settings.
Forcing a connection to an IP address can help isolate routing or node differences. Preserve the intended server name during that test. For public endpoint selection issues, see our guide to checking the live certificate path.
Separate Certificate Chains From Client Trust
The server normally presents its leaf certificate and necessary intermediate certificates. The client builds a path from that identity to a trusted root. Servers generally do not need to send the root certificate.
A missing intermediate can break validation. However, an intact chain can still fail when the client does not trust its root. Internal applications often use a private certificate authority that public browsers do not trust automatically.
Inspect the trust store that the failing process actually uses. A proxy, Java application, container, and operating system may each use different trust sources. Updating one store may leave the others unchanged.
If the backend requires mutual TLS, it also checks a client certificate from the proxy. Confirm that requirement before treating every certificate alert as a server certificate problem.
Do not disable certificate verification as a permanent fix. Correct the chain, authorized trust source, identity, or client certificate configuration instead. Keep this investigation separate from certificate renewal unless the evidence points to validity dates.
Check Client and Server Clocks
Certificate validation depends on time. A clock error can make a valid certificate appear expired or not yet valid.
Check the actual time on the workstation, proxy, backend, and relevant virtual hosts. Compare their synchronization status with an approved time source. A time-zone display difference alone does not prove clock drift.
On a normal one-way TLS connection, the client evaluates the server certificate against the client’s clock. For an upstream connection, the proxy acts as that client. With mutual TLS, the server also validates the client’s certificate.
Server clock problems can also disrupt certificate issuance, renewal, and useful log correlation. Avoid changing clocks casually on production systems. Follow the platform’s time-correction procedure and assess effects on authentication and scheduled tasks.
After correction, repeat the same test. Record whether the certificate validity error disappears rather than assuming that synchronization solved every failure.
Confirm Proxy Termination and Upstream Settings
TLS negotiation failures often persist because administrators change the public listener while the broken settings belong to the upstream connection. Inspect both configurations explicitly.
- Does the upstream port expect TLS or plain HTTP?
- Does the proxy enable upstream SNI where the backend requires it?
- Which name does upstream certificate verification expect?
- Which trusted certificate authorities does the proxy use?
- Do all backend nodes support the required protocol and algorithms?
- Does the backend require an authorized client certificate?
Sending TLS to a plain HTTP port can produce misleading protocol errors. Likewise, sending HTTP to a TLS-only listener will not work. Confirm the intended scheme and port together.
For Nginx deployments, consult the official Nginx documentation for the relevant listener and upstream TLS directives. Other proxies use different settings and defaults. Validate configuration syntax before a controlled reload, and preserve a rollback path.
Retest One Change and Preserve the Result
Change one verified cause at a time. Repeat the original failing request and the controlled handshake test. Then check another backend node or representative client to catch inconsistent settings.
Record the failing connection, negotiated version, certificate identity, trust source, and configuration change. Include timestamps and sanitized log excerpts. Our technical troubleshooting library covers related network and server checks when TLS proves healthy.
Professional remote assistance may help when several proxies, private trust stores, or production applications complicate the path. Tech Rescue Ops LLC can help isolate the failing session and plan a narrow correction without weakening certificate checks.