php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #46845 Test expected to fail, but pass, dont cause an error
Submitted: 2008-12-12 12:20 UTC Modified: 2008-12-31 17:42 UTC
From: hannes dot magnusson at gmail dot com Assigned: zoe (profile)
Status: Closed Package: *General Issues
PHP Version: 5.3.0alpha3 OS: Linux
Private report: No CVE-ID: None
 [2008-12-12 12:20 UTC] hannes dot magnusson at gmail dot com
Description:
------------
When test using XFAIL start "working" (i.e. don't fail) run-tests.php doesn't mark them in any special manners.
This means that when the fix breaks the test goes back into the "Expected fail" category without anyone ever noticing that something broke.


Patches

Add a Patch

Pull Requests

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2008-12-31 10:03 UTC] zoe@php.net
This discussion did come up when we added the XFAIL option and iirc we decided to go with the current behaviour. That said, I'm happy to change it if we can find a better solution.

Currently, as you say, we have the risk that someone could change PHP, the test could start passing, the developer forgets to remove the XFAIL section and then some time later the test starts to really fail but because there is an XFAIL section the failure is ignored.

I have checked through all the tests in PHP53 and found no passing test with an XFAIL section, so this hasn't been a problem yet.

One solution might be to make tests always FAIL until the XFAIL section is removed. This might be annoying and confusing  if you were trying to fix a bug. However it would ensure that the developer remembered to remove the XFAIL section.

Another would be to introduce another state "XFAILPASS"? for tests  which have an XFAIL section but actually pass? Or maybe "WARN"? Do you have a preference for any of these?

On balance my view is still that it's a small enough risk that this isn't worth another hack in an already much-hacked script, on the other hand if there  is a clear 'right' answer I will fix it. 


 [2008-12-31 11:14 UTC] hannes dot magnusson at gmail dot com
WARN.
I think WARN tests have their own "printout section" so it sounds good to me.
 [2008-12-31 17:42 UTC] zoe@php.net
Done. When I looked a couple of weeks ago I didn't find any XFAIL tests that were passing (maybe you had just fixed them up?). Having just fixed this I found 7 in PHP53 so I guess that demonstrates that the fix was necessary.
 
PHP Copyright © 2001-2024 The PHP Group
All rights reserved.
Last updated: Fri Mar 29 09:01:28 2024 UTC