php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Request #36398 "make test" should be fully scriptable [Do not prompt for email address]
Submitted: 2006-02-15 06:56 UTC Modified: 2020-09-23 09:40 UTC
Votes:5
Avg. Score:3.8 ± 0.7
Reproduced:5 of 5 (100.0%)
Same Version:1 (20.0%)
Same OS:1 (20.0%)
From: nickj-phpbugs at nickj dot org Assigned: cmb (profile)
Status: Closed Package: Testing related
PHP Version: 5.1.2 OS: Linux
Private report: No CVE-ID: None
View Developer Edit
Welcome! If you don't have a Git account, you can't do anything here.
If you reported this bug, you can edit this bug over here.
(description)
Block user comment
Status: Assign to:
Package:
Bug Type:
Summary:
From: nickj-phpbugs at nickj dot org
New email:
PHP Version: OS:

 

 [2006-02-15 06:56 UTC] nickj-phpbugs at nickj dot org
Description:
------------
I have a little shell script to download PHP snapshots, extract them, configure them, and build them. I would like to add automatically running "make test" on them.

However, it does not appear that this can be done at the moment, because "make test" prompts the user for information after running tests, like so:

================================================
You may have found a problem in PHP.
We would like to send this report automatically to the
PHP QA team, to give us a better understanding of how
the test cases are doing. If you don't want to send it
immediately, you can choose "s" to save the report to
a file that you can send us later.
Do you want to send this report now? [Yns]: 

Please enter your email address.
(Your address will be mangled so that it will not go out on any
mailinglist in plain text):
================================================

It would be really useful if "make test" could be fully scripted, so that it did not ask for whether to send email, and what email address to send it from.

For example, if you could do this, then that would be ideal:
make test --email-report --email-address=me@somewhere.com


Patches

Pull Requests

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2006-02-15 08:48 UTC] derick@php.net
You can already do this, set the env variable TEST_PHP_ARGS to "-q" and it should skip the question. Use -h to get all available options.
 [2006-02-16 06:19 UTC] nickj-phpbugs at nickj dot org
Thank you - that sort of helps, and I realise now that I should have phrased my request better (so I have reopened, and rephrased it to hopefully be clearer).

What I mostly wanted was to run the tests, and then email the results off, without prompting.

I can enable/disable the prompt now with:
TEST_PHP_ARGS="-q" make test

And I can see the available options with:
TEST_PHP_EXECUTABLE="/root/tmp/php-5.1-dev/php5.1-200602150330/sapi/cli/php" sapi/cli/php run-tests.php --help

However, unless I am mistaken, I can't see any option to specify the email address to use, such as for example: TEST_PHP_ARGS="--email-results --use-email=my@email.com" make test

My underlying assumption here is that you folks want and use the output of "make test" in some way. If that's not the case, then of course, don't make it an option. However, if you are using this, then wouldn't it be good to be able to script it so that it could automatically email off the results of make test, without having to prompt the user? Then you could (for example) have the build farm automatically email the qa.php.net list with the results of "make test" after every build of every snapshot. Maybe you already do this in some other way, but if not, it seems like it could be useful to me.

You could also update http://qa.php.net/running-tests.php with this information, so that people could "set and forget" to run the tests and email the results, without prompting them.

P.s. there is a very small typo in the help for "make test". It says "-q   Quite, no user interaction"; but you want it to say "Quiet", not "Quite".
 [2006-02-16 07:35 UTC] tony2001@php.net
Thanks, I've just fixed the typo.
As for the request - I'm pretty sure you can add it to run-tests.php and send a patch for us to review.
Though, I don't know if we really want php-qa list to get filled with results of the tests executed automatically every X minutes.
 [2006-02-17 06:53 UTC] nickj-phpbugs at nickj dot org
No worries! There is a patch now available for this at:
http://www.files.nickj.org/php/run-tests-patch.txt

Also, the run-test.php script is not currently E_STRICT clean. I have included in the patch some simple changes I had to make to get it to run with E_STRICT enabled.

Note that there are 3 remaining things I did not know how to fix for clean E_STRICT output, so even with this patch run-tests.php is still not totally E_STRICT clean. The three remaining things were: 

1) Was this error:
Error! type: 8; File: /root/php-5.1-dev/php5.1-200602150330/run-tests.php; Line: 93; Message: ob_end_clean(): failed to delete buffer. No buffer to delete.
... which was generated by this line under E_STRICT:
while(@ob_end_clean());

2) Was the use of date() in this line:
$output_file = $CUR_DIR . '/php_test_results_' . date('Ymd_Hi') . '.txt';

Which gives this error under E_STRICT:
Error! type: 2048; File: /root/php-5.1-dev/php5.1-200602150330/run-tests.php; Line: 246; Message: date(): It is not safe to rely on the system's timezone settings. Please use the date.timezone setting, the TZ environment variable or the date_default_timezone_set() function. In case you used any of those methods and you are still getting this warning, you most likely misspelled the timezone identifier. We selected 'Australia/Melbourne' for 'EST/11.0/DST' instead

Not sure how to say "just use the system default" nowadays, given that we probably shouldn't hard-code a timezone, as users from anywhere could be running this test script.

3) Was the "select_stream" call in the "system_with_timeout" function, which generates this error under E_STRICT:
Error! type: 2; File: /root/php-5.1-dev/php5.1-200602150330/run-tests.php; Line: 893; Message: stream_select(): 230 is not a valid stream resource
 [2008-08-05 12:21 UTC] Bjorn dot Wiberg at its dot uu dot se
Shouldn't the patch be incorporated into run-tests.php? As it has been around for a long time.
 [2008-08-07 06:58 UTC] nickj-phpbugs at nickj dot org
Yes, it would be nice if after calling for a patch, if the submitted patch could be either accepted or rejected, rather than ignored for 2.5 years ;-)
I gather run-tests.php may be rewritten sometime (I think I saw it somewhere as a possible Google SoC project), and if this happens including functionality to send the results using an email address specified on the command line could be nice to include. Presumably then someone (or some bit of software) can collate the output and help to pinpoint consistent problems, or when things break on particular platforms, and give a summary of the health of PHP on different platforms, and so forth.
 [2016-12-30 23:34 UTC] cmb@php.net
-Package: Feature/Change Request +Package: Testing related
 [2020-09-23 09:40 UTC] cmb@php.net
-Status: Open +Status: Closed -Assigned To: +Assigned To: cmb
 [2020-09-23 09:40 UTC] cmb@php.net
Thanks for the patch, and sorry that it has been missed, but …

> Though, I don't know if we really want php-qa list to get filled
> with results of the tests executed automatically every X minutes.

No, we really don't want that.  Instead it would be more helpful,
if individual test fails would be investigated by the testers, and
reported as separate tickets, ideally accompanied by a respective
pull request.
 
PHP Copyright © 2001-2024 The PHP Group
All rights reserved.
Last updated: Thu Oct 31 23:01:28 2024 UTC