Big file downloads stops at random
After migrating a site to new server I run into a strange problem with big file downloads that just stop/break without any specific reason at anytime. Downloads are not direct than are done using PHP (I’ll write more about that soon and I’ll link to that post from in here when I do) suddenly break without any notice. Browser reports that file has been successfully downloaded, Apache doesn’t report any errors but file is broken and not downloaded entirely.
I was pulling my hair for days, since sometimes downloads were working fine but sometimes they would just stop. Things got worse in case of downloading multiple files at the same time. Sometimes one or two downloads would break, sometimes all, sometimes none. It was driving me mad. Since I have been traveling those days across Northern Europe, I have been able to test the file download from multiple locations/countries and various connection types (ADSL, Cable, WiFi hot spots) and was still experienced same problems.
The setup on that server was bit complex and it used Apache to serve dynamic content, separate web server for static content. Downloads were done using PHP or using mod_xsendfile but either way problem persisted. So I figured out that something must be timing out and that is braking the connection. Since it would happen that out of 4-5 simultaneous downloads one or two would break and few would download normally. Finally, I tried increasing Apache timeout from 10 seconds to 30 seconds and it worked out! I was so pissed of and happy at the same time… The solution was so easy and in front of my nose but I needed so much time to figure it out… I felt so stupid for a second but then I solved the problem and I’ve learned something new…
It turns out that if there were no communication between server and client for 10 seconds the Apache would simply close connection and download would break. Nothing will be reported on both sides since it’s “normal” way to end the communication. This was also going on when mod_xsendfile was used to serve/download files. The problem was even greater when internet connection was worse (WiFi) and when I was downloading more files at once since than the connection would have to be split on few files.
So if you’re making a download server for larger files make sure you setup your Apache timeout to some reasonable value (30 seconds or more). Under term “larger files” I consider everything larger than pictures, but basically 50mb file could be called “larger file” since for it’s download it is required some time and of course open connection.