When HTTP 500 Was Neither PHP nor MySQL: An Nginx Incident Review

On the morning of 20 August 2026, a production server became unstable: websites would work and then suddenly return 500 Internal Server Error. phpMyAdmin started failing at the same time. PHP, MariaDB, host load, or a recent application change were all reasonable first suspects. This time, however, the failure lived in another layer.

About 7.5 hours elapsed between the first failure and full production recovery, including pauses and parallel work with hosting support. During that window, the investigation excluded application code and data, reproduced the failure without DNS or the client network, tested a second independent website, and connected the outage to an Nginx package revision installed by Ubuntu unattended upgrades.

What was known at the start

The symptom was not limited to one page. HTTP 500 appeared across different URLs, including simple requests such as /favicon.ico. The Nginx error log repeatedly contained:

no buffer space in script copy

That alert clearly pointed toward Nginx, but it did not by itself prove a specific directive, module, or internal defect. The next job was therefore to establish the blast radius instead of forcing every new fact into the first theory.

Why checking the database was still correct

The fact that MariaDB was not responsible does not make the initial database checks wasted work. phpMyAdmin was failing too, and the control panel reported an inability to write received data to disk or the application. Storage, the database, or an intermediate component could plausibly create that picture.

The hypothesis was tested with facts: MariaDB answered, SQL executed through an independent client, no blocked query explained the outage, and the database processes did not show a failure state. At that point the team stopped changing the database. This is an important incident rule: once a layer has been sufficiently falsified, do not keep “repairing” it.

Layer-by-layer diagnosis

The system was checked from the host upward instead of replacing components at random. Each test had to remove an entire class of causes from the decision tree, not merely produce another log file.

Two sites created the decisive A/B test

The primary website used Nginx → PHP-FPM → PHP. A second, independent project used Nginx → Apache → PHP. Their application code, project configuration, and route to PHP were different, yet both returned the same HTTP 500 response with the same Nginx alert.

That observation narrowed the search dramatically. When two independent applications with different backend chains fail identically while the database and host remain stable, the lowest shared component deserves priority. In this topology, that component was Nginx.

Control experiments that separated symptom from cause

• Direct-to-server request. curl --resolve sent HTTPS traffic straight to the selected IP while preserving the domain for Host and SNI. The failure remained, excluding external DNS and the browser.
• Simple resource. HTTP 500 on /favicon.ico showed that the issue extended beyond a complex business route.
• Maintenance mode. An expected 503 in maintenance mode and a fast 500 on the normal path were two different signals, not one generic “site down” condition.
• Worker replacement. After one worker was terminated, the alert appeared in another process. The fault was not isolated to one damaged worker.
• Independent project. The same error on a second site substantially weakened any OpenCart-specific explanation.

The apt history line that changed direction

Once Nginx was identified as the shared layer, the package-manager history showed an exact change: at 06:06 Ubuntu automatically upgraded Nginx from revision .18 to .19. Timing alone was not proof, but it produced a strong hypothesis consistent with every fact already collected.

df -h
vmstat 1
mysqladmin ping
nginx -V
dpkg-query -W 'nginx*' 'libnginx*'
grep -R "upgrade nginx" /var/log/apt/history.log*
curl --resolve example.com:443:SERVER_IP https://example.com/favicon.ico

In a future incident, recent system changes should be checked earlier when a component-specific alert appears. This does not replace diagnosis; it helps prioritize the correct hypotheses sooner.

Why an immediate downgrade was not the safest move

On a production host with a control panel, multiple sites, TLS configuration, extra Nginx modules, and different backend chains, a rushed rollback can introduce a new incompatibility. A package is not a single binary: dependencies, dynamic modules, configuration, and the active worker generation all matter.

The team therefore collected evidence first and then performed one controlled update to the available 1.18.0-6ubuntu14.20 revision. Both websites, their admin areas, and phpMyAdmin recovered immediately, and the repeated HTTP 500 responses disappeared. The pattern “one shared change, simultaneous recovery across all affected paths” materially strengthens the root-cause analysis.

What the RCA proves—and what remains open

The evidence supports this careful conclusion: the production failure was isolated to the shared Nginx layer, appeared after the automatic move to package revision .19, and stopped after a controlled move to .20. The exact internal mechanism would require a separate laboratory reproduction.

A concise playbook for the next incident

1. Record the first known failure time, exact status, URL, Host, PID, and log line.
2. Separate the external route from the server with a direct request using the correct Host and SNI.
3. Test one simple resource and one dynamic application route.
4. Quickly exclude disk, memory, CPU, I/O, database health, and network reachability.
5. Find a second independent service on the same shared layer.
6. Review apt history and configuration/package modification times.
7. Capture the active configuration, package inventory, and canary responses before intervention.
8. Make one controlled change and repeat exactly the same canary requests.
9. After recovery, verify the home page, an admin path, a static asset, a dynamic path, and the second site.

Reducing risk from automatic updates

• Separate routine security maintenance from uncontrolled changes to the critical data plane; give Nginx a maintenance window and canary path.
• Keep an inventory of Nginx builds, modules, package dependencies, and the last known-good configuration.
• Maintain tested forward-update and rollback procedures before an outage occurs.
• Run synthetic health checks for a static asset, dynamic route, admin area, and a second independent site.
• Alert on new messages in the shared proxy error log as well as on the HTTP 500 rate.
• After a package change, verify that the expected worker generation is actually serving traffic.

The larger lesson

During an outage, it is easy to repair whatever is most visible: PHP code, SQL, or the page that happened to fail first. Production is a dependency chain, however, and the same symptom across two independent applications can be more valuable than hundreds of lines from one application's log.

Good diagnosis is not guessing the guilty component. It is the disciplined removal of layers, preservation of control results, and one intervention whose effect can be measured. In this case, a single line in apt history became decisive only because the evidence around it had already been built.

Read the author's short incident summary on LinkedIn.

Nginx, HTTP 500, production, DevOps, diagnostics, OpenCart