php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login

go to bug id or search bugs for

Bug #46845 Test expected to fail, but pass, dont cause an error
Submitted: 2008-12-12 12:20 UTC Modified: 2008-12-31 17:42 UTC
From: hannes dot magnusson at gmail dot com Assigned: zoe (profile)
Status: Closed Package: *General Issues
PHP Version: 5.3.0alpha3 OS: Linux
Private report: No CVE-ID: None
View Developer Edit
Welcome! If you don't have a Git account, you can't do anything here.
If you reported this bug, you can edit this bug over here.
(description)
Block user comment
Status: Assign to:
Package:
Bug Type:
Summary:
From: hannes dot magnusson at gmail dot com
New email:
PHP Version: OS:

 

 [2008-12-12 12:20 UTC] hannes dot magnusson at gmail dot com 
Description:
------------
When test using XFAIL start "working" (i.e. don't fail) run-tests.php doesn't mark them in any special manners.
This means that when the fix breaks the test goes back into the "Expected fail" category without anyone ever noticing that something broke.

Patches

Pull Requests

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2008-12-31 10:03 UTC] zoe@php.net 
This discussion did come up when we added the XFAIL option and iirc we decided to go with the current behaviour. That said, I'm happy to change it if we can find a better solution.

Currently, as you say, we have the risk that someone could change PHP, the test could start passing, the developer forgets to remove the XFAIL section and then some time later the test starts to really fail but because there is an XFAIL section the failure is ignored.

I have checked through all the tests in PHP53 and found no passing test with an XFAIL section, so this hasn't been a problem yet.

One solution might be to make tests always FAIL until the XFAIL section is removed. This might be annoying and confusing  if you were trying to fix a bug. However it would ensure that the developer remembered to remove the XFAIL section.

Another would be to introduce another state "XFAILPASS"? for tests  which have an XFAIL section but actually pass? Or maybe "WARN"? Do you have a preference for any of these?

On balance my view is still that it's a small enough risk that this isn't worth another hack in an already much-hacked script, on the other hand if there  is a clear 'right' answer I will fix it.
 [2008-12-31 11:14 UTC] hannes dot magnusson at gmail dot com 
WARN.
I think WARN tests have their own "printout section" so it sounds good to me.
 [2008-12-31 17:42 UTC] zoe@php.net 
Done. When I looked a couple of weeks ago I didn't find any XFAIL tests that were passing (maybe you had just fixed them up?). Having just fixed this I found 7 in PHP53 so I guess that demonstrates that the fix was necessary.
 
PHP Copyright © 2001-2025 The PHP Group
All rights reserved. 		Last updated: Sat Oct 25 09:00:02 2025 UTC