php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #64273 Requesting nonexistent files should produce HTTP 404 Not Found
Submitted: 2013-02-22 09:04 UTC Modified: 2013-12-27 00:46 UTC
From: php-bugs-2013 at ryandesign dot com Assigned: bjori (profile)
Status: Closed Package: Website problem
PHP Version: Irrelevant OS:
Private report: No CVE-ID: None
 [2013-02-22 09:04 UTC] php-bugs-2013 at ryandesign dot com
Description:
------------
Requesting a nonexistent file from the network of PHP mirror servers should 
product an HTTP 404 Not Found error. In fact it produces an HTTP 302 Found 
redirect to a URL that produces an HTTP 200 OK code. This makes automated download 
systems think that the download was successful when it really was not. For example 
here is the type of problem this causes for the MacPorts project:

https://trac.macports.org/ticket/38162


Test script:
---------------
curl -I -L http://uk.php.net/distributions/php-5.3.222.tar.bz2


Expected result:
----------------
HTTP/1.1 404 Not Found
Date: Fri, 22 Feb 2013 08:53:01 GMT
Server: Apache/2.2.15 (CentOS)
X-Powered-By: PHP/5.3.3
Content-language: en
Set-Cookie: LAST_LANG=en; expires=Sat, 22-Feb-2014 08:53:01 GMT; path=/; 
domain=.php.net
Set-Cookie: COUNTRY=USA%2C72.183.101.32; expires=Fri, 01-Mar-2013 08:53:01 GMT; 
path=/; domain=.php.net
Status: 404 Not Found
Content-Length: 4173
Connection: close
Content-Type: text/html;charset=utf-8


Actual result:
--------------
HTTP/1.1 302 Found
Date: Fri, 22 Feb 2013 08:38:45 GMT
Server: Apache/2.2.15 (CentOS)
X-Powered-By: PHP/5.3.3
Content-language: en
Set-Cookie: COUNTRY=USA%2C72.183.101.32; expires=Fri, 01-Mar-2013 08:38:45 GMT; 
path=/; domain=.php.net
Location: http://uk1.php.net/releases/
Connection: close
Content-Type: text/html; charset=utf-8

HTTP/1.1 200 OK
Date: Fri, 22 Feb 2013 08:38:45 GMT
Server: Apache/2.2.15 (CentOS)
X-Powered-By: PHP/5.3.3
Content-language: en
Set-Cookie: COUNTRY=USA%2C72.183.101.32; expires=Fri, 01-Mar-2013 08:38:45 GMT; 
path=/; domain=.php.net
Connection: close
Content-Type: text/html;charset=utf-8


Patches

Add a Patch

Pull Requests

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2013-02-22 10:27 UTC] tyrael@php.net
Hi,

it seems that it is/was an intentional decision so that the it is more convinient 
for the humans to find what they are looking for when they request a distribution 
which isn't there:
http://git.php.net/?p=web/php.git;a=blob;f=error.php#l120
fixing the problem reported here would be trivial, but it requires a decision 
which I don't wanna take on alone.
 [2013-02-22 10:44 UTC] php-bugs-2013 at ryandesign dot com
I have no objection to you making it easier for users to find your software on 
your web site. But please do so in a way that does not increase the support burden 
for people trying to help your users install your software, namely me and other 
people who maintain PHP in package management systems.

One solution would be to serve up a custom 404 page that says "The file you 
requested was not found; go *here* for a list of available files," or some other 
wording for directing the user to the Releases page. Or even put links to the 
latest releases on the 404 page. See how Apple does their 404 page:

http://apple.com/404notfound
 [2013-02-22 10:59 UTC] tyrael@php.net
that sounds like a good compromise
(funnyfact: I've checked and trying to download a nonexistent release on apple 
like iTunes will result in an empty 404 page, so they not showing that nice 404 
page for the direct downloads)
 [2013-02-22 10:59 UTC] tyrael@php.net
-Package: Systems problem +Package: Website problem
 [2013-02-22 12:50 UTC] tyrael@php.net
-Status: Open +Status: Assigned -Assigned To: +Assigned To: tyrael
 [2013-02-22 16:46 UTC] bjori@php.net
Ryan, please use http://www.php.net/get/php-5.3.23.tar.bz2/from/this/mirror style 
urls.
These will give you the correct header and redirect you to the correct mirror if 
needed.
 [2013-02-23 10:59 UTC] php-bugs-2013 at ryandesign dot com
We would prefer not to use http://www.php.net/get/php-5.3.23.tar.bz2
/from/this/mirror style URLs. In MacPorts, each package (or "port") can
specify multiple URLs for each file to be downloaded. To that list are 
added a number of MacPorts mirror servers located around the globe. When 
a user wants to install a port, MacPorts pings each server and prefers 
to download the files from the servers with the lowest ping times. Our 
hope is this will result in faster downloads for users because they will 
connect with servers closer to them.

Currently for PHP ports we list a handful of PHP mirror servers. If we 
were to instead list only www.php.net, then our algorithm to determine the 
fastest server would only take the ping time of www.php.net itself into 
consideration. The end result might be that a user would be directed to 
one of our mirror servers, even if a PHP mirror server might be closer to 
them and deliver a faster download.
 [2013-02-23 13:19 UTC] tyrael@php.net
what bjori suggested is using
http://uk.php.net/get/php-5.3.222.tar.bz2/from/this/mirror
instead of
http://uk.php.net/distributions/php-5.3.222.tar.bz2
(or any other mirror you want to use)
as that would still fetch the same files but producing 404 when the file can't be 
found.
 [2013-02-23 14:16 UTC] php-bugs-2013 at ryandesign dot com
Oh I see. Yes that's ok then. The next issue is that MacPorts assumes
the filename to be downloaded should be appended to the URL; getting
the filename to appear elsewhere within the URL is more work. We've
done it before, on a port-by-port basis as needed, but there are
several ports that fetch PHP tarballs, so the list of PHP mirrors is
collected in a central place, and it doesn't appear as though that
central place has access to the name of the file that's going to be
downloaded, so inserting the filename into those URLs is not going to
be straightforward. I'll have to think about a good way to do that that
does not involve duplicating code in each portfile.
 [2013-02-24 21:50 UTC] bjori@php.net
I actually thought thought we had /get/<filename>/from/NEAREST/mirror support.. 
Doesn't look like it... We should probably look into it, is probably trivial to 
implement.

Our servers do ip2country so you shouldn't have to do any pre-emptive ping 
checks, we will in general redirect you to a mirror in your country which should 
be the fastest one.

The /get/<filename>/.. links do a redirect to the "backend file location", so if 
you follow the redirect and grab the last component of the url you have the 
filename.
 [2013-02-24 22:40 UTC] php-bugs-2013 at ryandesign dot com
MacPorts *does* do ping checks, for all ports; there is no switch to
turn this off on a per-port basis. And the list of servers checked
is the combination of the MacPorts mirror servers and the servers
listed for the individual port; there is no differentiation. I
explained above why listing just a single PHP download URL could
therefore result in a worse download experience for our users. We've
been through this for SourceForge before and do the same thing
there, listing various SF mirrors instead of using their own
redirector.

I'm aware that the /get/<filename>/.. links do a redirect (that's
actually another reason why we would prefer not to use them, since
reducing redirects leads to better performance). We aren't trying to
find out the filename after requesting the URL; above I was trying
to explain that actually getting the filename into the middle of the
URL is the part that's not so straightforward to do in MacPorts. But
I've discussed it with my colleagues and we've found a solution
that's not too bad.

The best solution remains that you fix your servers to return a 404
status code when I request a file that does not exist. You can't
really claim that returning a 200 status code, when you did not
return the file I asked for, is correct.
 [2013-02-24 23:02 UTC] bjori@php.net
I'm not claiming that 200 OK is not a bug.
I am however attempting to explain to you our infrastructure to see if you can 
even give your users even better experience.

We have over 100 mirrors, on average 2 mirrors in each country and do already 
implement ip2country redirecting to either a specific mirror in that country or 
the round-robin for highest availability.

I assume you list totally random handfull of mirrors, which can be disabled at 
any time in our system for all sorts of reasons and their dns entries revoked.
If you use our round-robin for those country names you have higher chance of 
succeeding, but the best possible way would be to use our system for the redirect 
as the mirrors themselves even do sanitychecks and in event of failure will 
redirect you to the nearest sane mirror.
 [2013-12-27 00:46 UTC] bjori@php.net
Automatic comment on behalf of bjori@mongodb.com
Revision: http://git.php.net/?p=web/php.git;a=commit;h=ef1389c73d9f19623338a603c904b1a4a69f1734
Log: Fixed bug #64273 (Requesting nonexistent files should produce HTTP 404 Not Found)
 [2013-12-27 00:46 UTC] bjori@php.net
-Status: Assigned +Status: Closed
 [2013-12-27 00:46 UTC] bjori@php.net
-Assigned To: tyrael +Assigned To: bjori
 [2013-12-27 00:46 UTC] bjori@php.net
The fix for this bug has been committed. Since the websites are not directly
updated from the repository, the fix might need some time to spread
across the globe to all mirror sites, including PHP.net itself.

Thank you for the report, and for helping us make PHP.net better.


 [2015-08-08 17:40 UTC] dragoonis@php.net
Automatic comment on behalf of bjori@mongodb.com
Revision: http://git.php.net/?p=web/php.git;a=commit;h=ae8ead9dde45e358a3b6f7241df740dd8a1681ad
Log: Fixed bug #64273 (Requesting nonexistent files should produce HTTP 404 Not Found)
 [2015-08-09 00:06 UTC] tyrael@php.net
Automatic comment on behalf of bjori@mongodb.com
Revision: http://git.php.net/?p=web/php.git;a=commit;h=ef1389c73d9f19623338a603c904b1a4a69f1734
Log: Fixed bug #64273 (Requesting nonexistent files should produce HTTP 404 Not Found)
 
PHP Copyright © 2001-2021 The PHP Group
All rights reserved.
Last updated: Wed Apr 21 01:01:23 2021 UTC