php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Doc Bug #73111 make prof-use don't work -> still profiling code
Submitted: 2016-09-19 08:21 UTC Modified: 2016-10-03 16:10 UTC
From: spam2 at rhsoft dot net Assigned:
Status: Open Package: Compile Failure
PHP Version: 7.0.12 RC1 OS: Linux
Private report: No CVE-ID: None
View Add Comment Developer Edit
Welcome! If you don't have a Git account, you can't do anything here.
You can add a comment by following this link or if you reported this bug, you can edit this bug over here.
(description)
Block user comment
Status: Assign to:
Package:
Bug Type:
Summary:
From: spam2 at rhsoft dot net
New email:
PHP Version: OS:

 

 [2016-09-19 08:21 UTC] spam2 at rhsoft dot net
Description:
------------
make prof-gen
make test
make prof-use

the "make prof-use" just takes a second while i would expect a full compile using the gcda-files filled with informations wude running "make test" and that part works becuase while "make test" is running the timestamps of that files are changing

after that there is still the profiling code in the binaries because running it on another machine at httpd startup you see these messages in the apache-errorlog

profiling:/home/builduser/rpmbuild/BUILD/php-7.0.11/ext/zlib/.libs/zlib.gcda:Skip
profiling:/home/builduser:Cannot create directory
profiling:/home/builduser/rpmbuild/BUILD/php-7.0.11/ext/pcre/.libs/php_pcre.gcda:Skip
profiling:/home/builduser:Cannot create directory
profiling:/home/builduser/rpmbuild/BUILD/php-7.0.11/ext/libxml/.libs/libxml.gcda:Skip
profiling:/home/builduser:Cannot create directory
profiling:/home/builduser/rpmbuild/BUILD/php-7.0.11/ext/date/lib/.libs/interval.gcda:Skip
profiling:/home/builduser:Cannot create directory
profiling:/home/builduser/rpmbuild/BUILD/php-7.0.11/ext/date/lib/.libs/parse_iso_intervals.gcda:Skip
profiling:/home/builduser:Cannot create directory
profiling:/home/builduser/rpmbuild/BUILD/php-7.0.11/ext/date/lib/.libs/unixtime2tm.gcda:Skip
profiling:/home/builduser:Cannot create directory
profiling:/home/builduser/rpmbuild/BUILD/php-7.0.11/ext/date/lib/.libs/tm2unixtime.gcda:Skip
profiling:/home/builduser:Cannot create directory
profiling:/home/builduser/rpmbuild/BUILD/php-7.0.11/ext/date/lib/.libs/timelib.gcda:Skip
profiling:/home/builduser:Cannot create directory
profiling:/home/builduser/rpmbuild/BUILD/php-7.0.11/ext/date/lib/.libs/parse_tz.gcda:Skip
profiling:/home/builduser:Cannot create directory
profiling:/home/builduser/rpmbuild/BUILD/php-7.0.11/ext/date/lib/.libs/parse_date.gcda:Skip
profiling:/home/builduser:Cannot create directory
profiling:/home/builduser/rpmbuild/BUILD/php-7.0.11/ext/date/lib/.libs/dow.gcda:Skip
profiling:/home/builduser:Cannot create directory
profiling:/home/builduser/rpmbuild/BUILD/php-7.0.11/ext/date/lib/.libs/astro.gcda:Skip
profiling:/home/builduser:Cannot create directory
profiling:/home/builduser/rpmbuild/BUILD/php-7.0.11/ext/date/.libs/php_date.gcda:Ski

Test script:
---------------
make prof-gen
make test
make prof-use

yes, i know that "make test" is probably not the best thing for PGO but a good start to get the build-process implemented and then replace it with running custom php-code and "make test" at the end with the final build

Expected result:
----------------
a final PGO build without the profiling code

Actual result:
--------------
binaries which stil contain profiling code after "make prof-use"

Patches

Add a Patch

Pull Requests

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2016-09-19 11:43 UTC] spam2 at rhsoft dot net
"make --always-make" seems to help partially

echo "PROFILE-GUIDED-OPTIMIZATION: Stage 1"
make %{?_smp_mflags} prof-gen
/rpmbuild/PHP-PGO/profile.sh $PWD
echo "PROFILE-GUIDED-OPTIMIZATION: Stage 2"
make --always-make %{?_smp_mflags} prof-use

but:

Zend/zend_language_scanner.l:2406:1: error: the control flow of function 'lex_scan' does not match its profile data (counter 'indirect_call') [-Werror=coverage-mismatch]
Zend/zend_language_scanner.l:2406:1: error: the control flow of function 'lex_scan' does not match its profile data (counter 'time_profiler') [-Werror=coverage-mismatch]
cc1: some warnings being treated as errors
Makefile:1721: recipe for target 'Zend/zend_ini_scanner.lo' failed
make[1]: *** [Zend/zend_ini_scanner.lo] Error 1
cc1: some warnings being treated as errors
Makefile:1717: recipe for target 'Zend/zend_language_scanner.lo' failed
make[1]: *** [Zend/zend_language_scanner.lo] Error 1
make[1]: Leaving directory '/home/builduser/rpmbuild/BUILD/php-7.0.11'
Makefile:302: recipe for target 'prof-use' failed
make: *** [prof-use] Error 2
Fehler: Fehler-Status beim Beenden von /var/tmp/rpm-tmp.1UuVW4 (%build)
 [2016-09-19 11:49 UTC] nikic@php.net
Without having looked into it, I suspect that the error you got in your last comment is because --always-make forces a regeneration of the lexer and you have a different re2c version installed which was used to generate the lexer in the repo, so you end up with minor differences in control flow. It will likely work if you run the whole thing (including profile generation) a second time.
 [2016-09-19 12:20 UTC] spam2 at rhsoft dot net
the problem is that "make prof-use" don't do anything while "-Wno-coverage-mismatch" is a dirty workaround for now - "-always-make" should not be needed

that sequence should work out-of-the-box instead "make prof-use" thingink "oh, already built, nothing to do"

make prof-gen
make test
make prof-use
 [2016-09-19 19:04 UTC] spam2 at rhsoft dot net
BTW:

pleae take a look at "-fprofile-generate leads in root-owned files"
https://bugzilla.redhat.com/show_bug.cgi?id=1377249

i had a fight the whole last night including some hours at moring in the office without understand what happens, it seems now disappeared but running "rpmbuild" as restrcited user and ending in root-owned files and all sorts of errors when other things are still running as the correct restricted user is very strange

finally i had 10 successful builds with PGO, the perofrmance of our own CMS system is increased by 7% with PHP 7.0.11 compared to normal builds - really great but the root-thing and need of "make --always-make" don't make me terrible happy while i can only explain the "make prof-use" NOOP which is topic of this bugreport
_______________________

currently PGO is done with that snippet while "profile.sh" first starts CLI-autotests with the temporary binary and after that starts a webserver on port 9000, fetching a URL-list and fetch all URLs known by the cms 10 times to fill up the profiledata for "make prof-use"

# build php with 'profile-guided-optimization' when enabled
%if %pgo_build
 echo "PROFILE-GUIDED-OPTIMIZATION: Stage 1"
 make %{?_smp_mflags} prof-gen
 /rpmbuild/PHP-PGO/profile.sh $PWD
 echo "PROFILE-GUIDED-OPTIMIZATION: Stage 2"
 make --always-make %{?_smp_mflags} prof-use
%else
 make %{?_smp_mflags}
%endif
_______________________

comment from GCC maintainers

The *.gcda files aren't written by gcc, but by the application you're testing (when compiled/linked with -fprofile-generate).  It uses whatever uid the process has.  Perhaps you are dlopening your -fprofile-generate built library into some suid app or something similar?
In any case, there is nothing that can be done about it on the gcc side
 [2016-09-29 14:59 UTC] spam2 at rhsoft dot net
-PHP Version: 7.0.11 +PHP Version: 7.0.12 RC1
 [2016-09-29 14:59 UTC] spam2 at rhsoft dot net
7.0.12 RC1 does the same nonsense and skip the "make -j8 prof-use" completly which leads in steop over to "make install" and in fact installs binaries with the profile-code leading in overhead error messages at runtime without the "--always-make" hack 
______________________________________

./configure
make -j8 prof-gen
<run application code>
make -j8 prof-use
make install
______________________________________

+ make -j8 prof-use
CCACHE_DISABLE=1 make PROF_FLAGS=-fprofile-use all
make[1]: Entering directory '/home/builduser/rpmbuild/BUILD/php-7.0.12'

Build complete.
Don't forget to run 'make test'.

make[1]: Leaving directory '/home/builduser/rpmbuild/BUILD/php-7.0.12'
+ exit 0
Ausf├╝hrung(%install): /bin/sh -e /var/tmp/rpm-tmp.F1EgEt
+ umask 022
+ cd /home/builduser/rpmbuild/BUILD
+ '[' /home/builduser/rpmbuild/BUILDROOT/php-7.0.12-1.fc24.20160929.1647.rh.x86_64 '!=' / ']'
+ rm -rf /home/builduser/rpmbuild/BUILDROOT/php-7.0.12-1.fc24.20160929.1647.rh.x86_64
++ dirname /home/builduser/rpmbuild/BUILDROOT/php-7.0.12-1.fc24.20160929.1647.rh.x86_64
+ mkdir -p /home/builduser/rpmbuild/BUILDROOT
+ mkdir /home/builduser/rpmbuild/BUILDROOT/php-7.0.12-1.fc24.20160929.1647.rh.x86_64
+ cd php-7.0.12
+ install -m 0755 -d /home/builduser/rpmbuild/BUILDROOT/php-7.0.12-1.fc24.20160929.1647.rh.x86_64/etc /home/builduser/rpmbuild/BUILDROOT/php-7.0.12-1.fc24.20160929.1647.rh.x86_64/etc/httpd/conf /home/builduser/rpmbuild/BUILDROOT/php-7.0.12-1.fc24.20160929.1647.rh.x86_64/usr/lib64/httpd/modules /home/builduser/rpmbuild/BUILDROOT/php-7.0.12-1.fc24.20160929.1647.rh.x86_64/usr/share/php
+ install -m 0644 /home/builduser/rpmbuild/SOURCES/opcache-zendoptimizer.php /home/builduser/rpmbuild/BUILDROOT/php-7.0.12-1.fc24.20160929.1647.rh.x86_64/usr/share/php/zendoptimizer.php
+ install -m 0644 /home/builduser/rpmbuild/SOURCES/php-httpd-dummy.conf /home/builduser/rpmbuild/BUILDROOT/php-7.0.12-1.fc24.20160929.1647.rh.x86_64/etc/httpd/conf/httpd.conf
+ INSTALL_ROOT=/home/builduser/rpmbuild/BUILDROOT/php-7.0.12-1.fc24.20160929.1647.rh.x86_64
+ make install
 [2016-09-29 15:54 UTC] nikic@php.net
Just checked the makefile, it seems pretty clear that the intended command sequence is

    make prof-gen
    # train
    make prof-clean
    make prof-use

Your invocation is missing the "make prof-clean", which is what removes the compiler artifacts so that "make prof-use" will actually recompile.
 [2016-09-29 17:14 UTC] spam2 at rhsoft dot net
That indeed seems to work, but when you Google for build php with pgo "make prof-clean" is nowhere (Intel pages and others) mentioned and looking at the rm-calls it's pretty obvious that "make prof-use" should fire them implicit
 [2016-09-29 23:15 UTC] spam2 at rhsoft dot net
back at home - that sequence works:
________________________________________

make %{?_smp_mflags} prof-gen
/usr/bin/bash /rpmbuild/PHP-PGO/profile.sh $PWD
make prof-clean
make %{?_smp_mflags} prof-use
________________________________________

the size difference of the resulting binaries with no other changes is dramatically

-rwxr-xr-x 1 root root 5,8M 2016-09-29 17:20 /usr/bin/php
-rwxr-xr-x 1 root root 4,9M 2016-09-29 18:19 /usr/bin/php
________________________________________

after thinking again about "--always-make forces a regeneration of the lexer and you have a different re2c version installed which was used to generate the lexer in the repo, so you end up with minor differences in control flow" i tried ot add "--always-make" to the "make prof-gen" call

that seems to make some differnce BUT the reduced binary size is completly gone (retried 2 times, with --always-make only in the first and also in both calls)

it's not really related to this bugreport, but given that bison/re2c are seem to do something but the binaries end as large as with the missing "make prof-clean" maybe it's better to leave --always-make completly out of the game

+/home/builduser/rpmbuild/BUILD/php-7.0.12/Zend/zend_ini_parser.y:264.1-12: warning: deprecated directive, use '%pure-parser' [-Wdeprecated]
+ %pure_parser
+ ^^^^^^^^^+bison -y --defines -l /home/builduser/rpmbuild/BUILD/php-7.0.12/ext/json/json_parser.y -o /home/builduser/rpmbuild/BUILD/php-7.0.12/ext/json/json_parser.tab.c^^^
 [2016-09-30 01:10 UTC] yohgaki@php.net
Make this a doc issue?
This page might be the page.
http://jp2.php.net/manual/en/faq.build.php
 [2016-09-30 01:24 UTC] spam2 at rhsoft dot net
probably, but on the other hand "make prof-use" without "make prof-gen" before don't make any sense and the same for missing "make prop-clean" after running sample scripts, hence "make prop-clean" should be implicit

additional you can't fix all the other references not mentioning "make prof-clean" while i wonder how they all pretend to got a sucessful build r did they just imagine optimization and not realize that their final binary still contains the profiling code and spits in logs all day long?

https://software.intel.com/en-us/blogs/2015/10/09/pgo-let-it-go-php

http://www.phpclasses.org/blog/post/321-PHP-7-Performance-Improvements-May-Increase-if-You-Optimize-it-to-Your-Application-using-to-PGO.html
 [2016-09-30 01:39 UTC] yohgaki@php.net
Reasonable. I'm +1 for making

make prof-clean

implicit.
 [2016-09-30 15:58 UTC] nikic@php.net
-Assigned To: +Assigned To: dmitry
 [2016-09-30 15:58 UTC] nikic@php.net
@dmitry: As you implemented these make targets, was there a particular reason why prof-clean is a separate step (rather than being automatically triggered by prof-use)?
 [2016-10-03 07:17 UTC] dmitry@php.net
-Status: Assigned +Status: Feedback -Type: Bug +Type: Feature/Change Request
 [2016-10-03 07:17 UTC] dmitry@php.net
The general PGO build instruction: 

make prof-gen
<training>
make prof-clean
make prof-use

but it's also possible to build with PGO only particular files (e.g. zend_execute.c). In last case implicit prof-clean makes more problems than profit.

I wouldn't change anything.
 [2016-10-03 15:21 UTC] spam2 at rhsoft dot net
-Status: Feedback +Status: Assigned
 [2016-10-03 15:21 UTC] spam2 at rhsoft dot net
"The general PGO build instruction" is to find where?

i personally wasted hours to get it work somehow and 10 days for realize "make prof-clean" exists by this bugreport - a terrible situation and it's not obvious when you type "php profile guided optimization" in Google because non of the prominent pages are mentioning it while for "but it's also possible to build with PGO only particular files" you anyway need to to something outside of the normal "./configure; make prof-gen; make prof-clean; make prof-use"

not mentioning "make prof-clean"
https://software.intel.com/en-us/blogs/2015/10/09/pgo-let-it-go-php

not mentioning "make prof-clean"
http://www.phpclasses.org/blog/post/321-PHP-7-Performance-Improvements-May-Increase-if-You-Optimize-it-to-Your-Application-using-to-PGO.html
 [2016-10-03 16:10 UTC] nikic@php.net
-Status: Assigned +Status: Open -Type: Feature/Change Request +Type: Documentation Problem -Assigned To: dmitry +Assigned To:
 [2016-10-03 16:10 UTC] nikic@php.net
Switching category to doc problem.
 
PHP Copyright © 2001-2019 The PHP Group
All rights reserved.
Last updated: Sat Oct 19 06:01:27 2019 UTC