Big file downloads stops at random

After migrating a site to new server I run into a strange problem with big file downloads that just stop/break without any specific reason at anytime. Downloads are not direct than are done using PHP (I’ll write more about that soon and I’ll link to that post from in here when I do) suddenly break without any notice. Browser reports that file has been successfully downloaded, Apache doesn’t report any errors but file is broken and not downloaded entirely.

I was pulling my hair for days, since sometimes downloads were working fine but sometimes they would just stop. Things got worse in case of downloading multiple files at the same time. Sometimes one or two downloads would break, sometimes all, sometimes none. It was driving me mad. Since I have been traveling those days across Northern Europe, I have been able to test the file download from multiple locations/countries and various connection types (ADSL, Cable, WiFi hot spots) and was still experienced same problems.

The setup on that server was bit complex and it used Apache to serve dynamic content, separate web server for static content. Downloads were done using PHP or using mod_xsendfile but either way problem persisted. So I figured out that something must be timing out and that is braking the connection. Since it would happen that out of 4-5 simultaneous downloads one or two would break and few would download normally. Finally, I tried increasing Apache timeout from 10 seconds to 30 seconds and it worked out! I was so pissed of and happy at the same time… The solution was so easy and in front of my nose but I needed so much time to figure it out… I felt so stupid for a second but then I solved the problem and I’ve learned something new…

It turns out that if there were no communication between server and client for 10 seconds the Apache would simply close connection and download would break. Nothing will be reported on both sides since it’s “normal” way to end the communication. This was also going on when mod_xsendfile was used to serve/download files. The problem was even greater when internet connection was worse (WiFi) and when I was downloading more files at once since than the connection would have to be split on few files.

So if you’re making a download server for larger files make sure you setup your Apache timeout to some reasonable value (30 seconds or more). Under term “larger files” I consider everything larger than pictures, but basically 50mb file could be called “larger file” since for it’s download it is required some time and of course open connection.

Tags: apache large file download timeout, downloads stop randomly
How to whitelist your IP at CPHulk
Sysctl.conf hardening

Comments

  1. 30 seconds seems to be the magic value. I’d the same problem with a Timeout of 120. The I’ve read your post and tried 30 seconds… and it worked afterwards wonderful.

  2. I usually don’t comment on websites, but i have been fighting with a similar problem since a recent server change. And it turns out it was a similar error! I’m using xsendfile on apache2 on a newly installed server and I could not figure out why the downloads (usually around 1 or 2gb) were stopping randomly without errors. It turns out it was also the apache timeout. I could not see it at first because i was messing with the website while downloading which sometimes allowed the download to end and sometimes not.
    Anyway, thanks a lot for that post, after searching for a solution for two days, this is the only helpful post i found!

    • I know the feeling (been searching for solution for couple of days too). I’m glad that I was able to help you out!

Leave a Reply

Your email address will not be published / Required fields are marked *

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.