PHP :: Bug #14333 :: Apache processes consume all CPU

Bug #14333	Apache processes consume all CPU
Submitted:	2001-12-03 19:21 UTC	Modified:	2002-01-02 14:05 UTC
From:	tauren at servlets dot net	Assigned:
Status:	Closed	Package:	Apache related
PHP Version:	4.0.6	OS:	RH 6.2 SMP
Private report:	No	CVE-ID:	None

View Add Comment Developer Edit

Welcome! If you don't have a Git account, you can't do anything here.
You can add a comment by following this link or if you reported this bug, you can edit this bug over here.

php.net Username: php.net Password:

Quick Fix:	(description)
	Block user comment
Status:		Assign to:
Package:
Bug Type:
Summary:
From:	tauren at servlets dot net
New email:
PHP Version:		OS:

New/Additional Comment:

[2001-12-03 19:21 UTC] tauren at servlets dot net

I've been experiencing the exact same problem that is described in bug #14290 and #8446 for quite some time now but did not know which Apache module was causing it.  Up until now, I've had a cron job that simply kills off (with a -9) any httpd processes that are using 99% or more cpu time.

Today I've been trying to track down what exactly is causing the problem.  I've eliminated all extra Apache modules and did not experience the problem.  When I added PHP back in, the problem started immediately.  Within one minute of starting Apache back up, the high-CPU processes started appearing again.

The Apache "server-status" didn't indicate that ANY php script had been hit.  The processes just start going out of control after some time.  In fact, there isn't even a single *.php* file on the server.  I really don't think this is happening because of a PHP script being run.

I'm currently testing this out on a Red Hat 6.2 Linux (SMP) box with dual CPUs.  From the sound of things in the other bug reports (#14290 and #8446), the problem only seems to be happening on SMP servers.  I did not compile with any extra PHP modules except for the core PHP 4.0.6.

I haven't really done a lot with PHP, so I'm not sure how to help debug this problem.  But I do want a stable Apache environment with PHP support for my hosting customers.  If there is anything I can do to help debug things, please let me know.  

I've read the page on using gdb, but I'm think this is a different kind of situation.  Apache isn't crashing, but certain processes are going "out-of-control".  Is there a way to get a backtrace of a particular process while it is still running?

Until this problem can be resolved, I'm going to have to remove PHP from my servers.  I really don't want to have to do this, but the instabilities are becoming too much to handle and very hard to explain to our customers.

Please let me know what I can do to help debug and solve this problem.  

Thanks!
Tauren

Patches

Add a Patch

Pull Requests

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports

[2001-12-03 19:35 UTC] tauren at servlets dot net

Actually, there is a sample PHP script installed on the server in several user directories.  But this is the ONLY file with a *.php* extension within our web root directories. It is called "hello.php3" and it looks like this:

<html><head><title>PHP Test</title></head>
<body>
<?php echo "Hello World<P>"; ?>
</body></html>

I really don't think this script is being run.  I don't think that any script is being run to cause the problem.

Thanks,
Tauren

[2001-12-03 20:56 UTC] sniper@php.net

I'm also running PHP / Apache in a SMP machine but I never
have noticed anything like this. Some things we need to know about
your system:

- Kernel/glibc version
- Apache version 
- PHP configure line

And you should also try with PHP 4.1.0RC5:

http://www.php.net/~zeev/php-4.1.0RC5.tar.gz

--Jani

p.s. This might sound stupid but did you do './apachectl stop ; ./apachectl start' or did you use './apachectl restart' ?

[2001-12-04 00:59 UTC] tauren at servlets dot net

It's interesting that you aren't experiencing the same problems.  I wonder why I (and several others) are having trouble.  Here's the info you've requested.

Kernel/glibc version: 
  Red Hat Linux release 6.2 (Zoot)
  Kernel 2.2.14-5.0smp on a 2-processor i686

Apache version:
  Apache/1.3.22 (Unix) mod_ssl/2.8.5 OpenSSL/0.9.6

Please note that I'm not manually building Apache with PHP.  I've built Apache manually many times, but by the time I wanted to use PHP, I was using ApacheToolbox (ATB) to build my Apache distributions.  ATB is a script that eases the download and configuration of Apache with many supported modules.  It really simplifies the building of Apache+ModSSL+PHP+OtherStuff.  It can be downloaded at www.apachetoolbox.com.  Below is the PHP configure line that it generates and lets me edit before PHP is configured.

PHP configure line:
  CPPFLAGS=-I/usr/local/include
  LDFLAGS=-L/usr/local/lib
  ./configure --prefix=/usr/local \
  --with-apache=/root/atb/Apachetoolbox-1.5.45/apache_1.3.22 \
  --enable-exif \
  --enable-track-vars \
  --with-calendar=shared \
  --enable-safe-mode \
  --enable-magic-quotes \
  --enable-trans-sid \
  --enable-wddx \
  --enable-ftp \
  --with-mysql \
  --with-ldap \

ATB then generates this Apache configuration line and lets me edit it before configuring Apache:

export CFLAGS=""
export LIBS=""
export INCLUDES=""
./configure --prefix=/usr/local/apache \
--enable-suexec \
--suexec-caller=nobody \
--enable-module=so \
--enable-module=access \
--disable-module=auth_db \
--disable-module=digest \
--enable-module=imap \
--enable-module=mime \
--enable-module=setenvif \
--disable-module=usertrack \
--enable-module=auth \
--disable-module=cern_meta \
--disable-module=expires \
--enable-module=log_config \
--disable-module=proxy \
--disable-module=vhost_alias \
--disable-module=auth_anon \
--enable-module=cgi \
--disable-module=headers \
--disable-module=log_referer \
--enable-module=rewrite \
--enable-module=userdir \
--enable-module=asis \
--enable-module=autoindex \
--disable-module=example \
--disable-module=log_agent \
--enable-module=negotiation \
--enable-module=status \
--enable-module=actions \
--disable-module=auth_dbm \
--enable-module=dir \
--enable-module=include \
--disable-module=mime_magic \
--disable-module=unique_id \
--enable-module=alias \
--disable-module=auth_digest \
--enable-module=env \
--disable-module=info \
--disable-module=mmap_static \
--disable-module=speling \
--enable-module=ssl \
--activate-module=src/modules/php4/libphp4.a \

A couple other things...  The MM library (mm-1.1.3) is being used with Apache.  I've been using this with my Apache builds for years and it hasn't caused any trouble before.  Also, Apache is built as a static binary with DSO support.  PHP is built into it, not as a DSO.

I'll give PHP 4.1.0RC5 a try and let you know what happens.

Also, yes I am using ./apachectl stop and start, not restart or graceful.  Doesn't hurt to be thorough with your questions...

Thanks for your help!
Tauren

[2001-12-04 02:22 UTC] derick@php.net

Can you possibly try php-4.1.0RC5 from www.php.net/~zeev/ and make sure you do a clean build. My guess is that your build is slightly broken. Please also try it as an static apache module.

regards,
Derick

[2001-12-04 08:11 UTC] tauren at servlets dot net

I just reconfigured and installed Apache 1.3.22 with PHP 4.1.0RC5 support.  When I did this, I also added --enable-debug to the PHP configure command-line.  I added a CFLAGS="-g" and --without-execstrip to the apache configure command-line.

Once I installed and restarted Apache, I waited about 3 or 4 minutes before one of the Apache processes started using up 88% of the CPU.  This process grew to consume 99% within the next minute or so.

I then ran gdb and attached it to the PID.  The following is the output, including a backtrace:

[root@s1 apache_1.3.22]# gdb /usr/local/apache/bin/httpd 28070
GNU gdb 19991004
Copyright 1998 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux"...
 
/root/atb/Apachetoolbox-1.5.45/apache_1.3.22/28070: No such file or directory.
Attaching to program: /usr/local/apache/bin/httpd, Pid 28070
Reading symbols from /lib/libpam.so.0...done.
Reading symbols from /lib/libdl.so.2...done.
Reading symbols from /usr/local/lib/libgd.so...done.
Reading symbols from /usr/local/lib/php/t1libs/lib/libt1.so.1...done.
Reading symbols from /usr/local/lib/libfreetype.so.6...done.
Reading symbols from /usr/local/lib/libz.so.1...done.
Reading symbols from /usr/local/lib/libjpeg.so.62...done.
Reading symbols from /lib/libcrypt.so.1...done.
Reading symbols from /lib/libresolv.so.2...done.
Reading symbols from /lib/libm.so.6...done.
Reading symbols from /lib/libnsl.so.1...done.
Reading symbols from /lib/libdb.so.3...done.
Reading symbols from /lib/libc.so.6...done.
Reading symbols from /lib/ld-linux.so.2...done.
Reading symbols from /lib/libnss_files.so.2...done.
Reading symbols from /lib/libnss_dns.so.2...done.
Reading symbols from /usr/lib/gconv/ISO8859-1.so...done.
Reading symbols from /usr/local/apache/libexec/libjrun.so...done.
Reading symbols from /usr/local/apache/libexec/mod_caucho.so...done.
0x4020922e in chunk_free (ar_ptr=0x4029dd40, p=0x8cc3b98) at malloc.c:3121
3121    malloc.c: No such file or directory.
(gdb) bt
#0  0x4020922e in chunk_free (ar_ptr=0x4029dd40, p=0x8cc3b98) at malloc.c:3121
#1  0x40208f9a in __libc_free (mem=0x8ccf458) at malloc.c:3023
#2  0x815765a in shutdown_memory_manager (silent=0, clean_cache=1) at zend_alloc.c:485
#3  0x80c85a5 in php_module_shutdown () at main.c:1007
#4  0x80c855c in php_module_shutdown_wrapper (sapi_globals=0x8367ba0) at main.c:972
#5  0x80c6a15 in php_child_exit_handler (s=0x83aa8a8, p=0x8cf6578) at mod_php4.c:816
#6  0x81c4ac1 in ap_child_exit_modules (p=0x8cf6578, s=0x83aa8a8) at http_config.c:1701
#7  0x81ca9c5 in clean_child_exit (code=0) at http_main.c:537
#8  0x81cc9e7 in just_die (sig=10) at http_main.c:3068
#9  0x81cca09 in usr1_handler (sig=10) at http_main.c:3078
#10 0x401cdc48 in __restore () at ../sysdeps/unix/sysv/linux/i386/sigaction.c:127
#11 0x81cab96 in accept_mutex_on_sysvsem () at http_main.c:841
#12 0x81cdd36 in child_main (child_num_arg=17) at http_main.c:4301
#13 0x81ce35c in make_child (s=0x83aa8a8, slot=17, now=1007468333) at http_main.c:4723
#14 0x81ce6d5 in perform_idle_server_maintenance () at http_main.c:4902
#15 0x81cec65 in standalone_main (argc=2, argv=0xbffffb34) at http_main.c:5139
#16 0x81cf233 in main (argc=2, argv=0xbffffb34) at http_main.c:5415

I then quit gdb and killed the high-CPU process (kill -9 28070).  There were around 10 idle httpd processes at that time.  The httpd process with the highest PID (#28066) then immediately started using up excessive CPU time.  Doing a backtrace of it resulted in the same kind of output:

0  0x40209233 in chunk_free (ar_ptr=0x4029dd40, p=0x8cc3b98) at malloc.c:3121
#1  0x40208f9a in __libc_free (mem=0x8ccf458) at malloc.c:3023
#2  0x815765a in shutdown_memory_manager (silent=0, clean_cache=1) at zend_alloc.c:485
#3  0x80c85a5 in php_module_shutdown () at main.c:1007
#4  0x80c855c in php_module_shutdown_wrapper (sapi_globals=0x8367ba0) at main.c:972
#5  0x80c6a15 in php_child_exit_handler (s=0x83aa8a8, p=0x8cf6578) at mod_php4.c:816
#6  0x81c4ac1 in ap_child_exit_modules (p=0x8cf6578, s=0x83aa8a8) at http_config.c:1701
#7  0x81ca9c5 in clean_child_exit (code=0) at http_main.c:537
#8  0x81cc9e7 in just_die (sig=10) at http_main.c:3068
#9  0x81cca09 in usr1_handler (sig=10) at http_main.c:3078
#10 0x401cdc48 in __restore () at ../sysdeps/unix/sysv/linux/i386/sigaction.c:127
#11 0x81cab96 in accept_mutex_on_sysvsem () at http_main.c:841
#12 0x81cdd36 in child_main (child_num_arg=16) at http_main.c:4301
#13 0x81ce35c in make_child (s=0x83aa8a8, slot=16, now=1007468328) at http_main.c:4723
#14 0x81ce6d5 in perform_idle_server_maintenance () at http_main.c:4902
#15 0x81cec65 in standalone_main (argc=2, argv=0xbffffb34) at http_main.c:5139
#16 0x81cf233 in main (argc=2, argv=0xbffffb34) at http_main.c:5415

Every time I kill the CPU consuming process, the highest PID apache process takes over doing the same thing and consumes 95%+ CPU time immediately.

Looking at a normal httpd process (with 0.0% CPU), the output looks like this:

#0  0x4025c15e in __select () from /lib/libc.so.6
#1  0x81cab80 in accept_mutex_on_sysvsem () at http_main.c:837
#2  0x81ce35c in make_child (s=0x83aa8a8, slot=14, now=1007468327) at http_main.c:4723
#3  0x81ce6d5 in perform_idle_server_maintenance () at http_main.c:4902
#4  0x81cec65 in standalone_main (argc=2, argv=0xbffffb34) at http_main.c:5139
#5  0x81cf233 in main (argc=2, argv=0xbffffb34) at http_main.c:5415

Here is the output of a process that is active, but still not consuming hardly any CPU:

#0  0x40255ab4 in __libc_read () from /lib/libc.so.6
#1  0x81ab484 in ssl_io_hook_read (fb=0x83b65e8, 
    buf=0x83b6628 "GET /lemonnier/forum/immagini/testatina_forum.jpg HTTP/1.1\r\nAccept: */*\r\nReferer: http://www.japhost.com/lemonnier/forum/ConclusioniSecondaria/LetMinis.html\r\nAccept-Language: it\r\nAccept-Encoding: gzip"..., len=4096) at ssl_engine_io.c:339
#2  0x81e48ac in ap_hook_call_func (ap=0xbfffd864, he=0x83b30d8, hf=0x83b6570) at ap_hook.c:649
#3  0x81e3fc1 in ap_hook_call (hook=0x8336fad "ap::buff::read") at ap_hook.c:382
#4  0x81c01a0 in ap_read (fb=0x83b65e8, buf=0x83b6628, nbyte=4096) at buff.c:285
#5  0x81c1d9e in buff_read (fb=0x83b65e8, buf=0x83b6628, nbyte=4096) at buff.c:329
#6  0x81c1d48 in saferead_guts (fb=0x83b65e8, buf=0x83b6628, nbyte=4096) at buff.c:702
#7  0x81c084f in read_with_errors (fb=0x83b65e8, buf=0x83b6628, nbyte=4096) at buff.c:753
#8  0x81c0ba7 in ap_bgets (buff=0xbfffd948 "", n=8192, fb=0x83b65e8) at buff.c:906
#9  0x81d09d8 in getline (s=0xbfffd948 "", n=8192, in=0x83b65e8, fold=0) at http_protocol.c:839
#10 0x81d0cf1 in read_request_line (r=0x8d045d0) at http_protocol.c:962
#11 0x81d134e in ap_read_request (conn=0x83b9640) at http_protocol.c:1125
#12 0x81ce0c2 in child_main (child_num_arg=12) at http_main.c:4544
#13 0x81ce35c in make_child (s=0x83aa8a8, slot=12, now=1007468293) at http_main.c:4723
#14 0x81ce6d5 in perform_idle_server_maintenance () at http_main.c:4902
#15 0x81cec65 in standalone_main (argc=2, argv=0xbffffb34) at http_main.c:5139
#16 0x81cf233 in main (argc=2, argv=0xbffffb34) at http_main.c:5415

One more interesting thing is that most of the time, the high-CPU PID is not listed in an Apache /server-status list of processes.  If it is listed, it has a Mode of Operation of "Waiting for connection".

Lastly, sometimes only one high-CPU processes appears and consumes 95%+ CPU.  Other times, multiple high-CPU processes appear.  When this happens, they each consume less CPU.  For instance, right now there are 3 httpd processes, each consuming 50%+ CPU time.  All three process' backtraces looked the same as the first backtrace listed above (the high-CPU processes).

Note that I have several additional modules included in Apache during these tests.  This is a production server, so I can't do any long-term tests that do not include essential Apache mods.  In addition to many of the "standard" modules, the following modules were enabled in Apache:

	mod_perl - static
	WebDAV - static
	mod_auth_ldap - static
	mod_gzip - static
	mod_layout - static
	mod_headers - static
	PHP - static
	  mod_php, v4.1.0RC5, --enable-debug
	GD
	mod_ssl - static
	mod_jrun (JRun servlet engine) - as DSO
	mod_caucho (Resin servlet engine) - as DSO

I've been able to duplicate this problem in both PHP 4.0.6 and PHP 4.1.0RC5 with only the bare minimum modules included.  I'm pretty confident that the problem would still occur even if Apache was configured with only PHP.  The backtraces do not seem to indicate any of the modules being suspicious except for PHP and the core apache.

I hope this information is useful.  Please let me know if there is anything else I can provide.

Tauren

[2001-12-04 08:22 UTC] derick@php.net

You can try this:

Edit the ./configure line for PHP that APT generated:

remove: --with-mysql
add:    --with-mysql=/usr

(if libmysqlclient.so is in /usr/lib/)

and the rebuild both PHP and apache.

Derick

[2001-12-04 17:57 UTC] tauren at servlets dot net

I can't find the libmysqlclient.so file on my system (and definitely not in /usr/lib).  MySQL-3.23.40-1
was installed from RPM, not from ApacheToolbox.  I don't think I've installed the MySQL-devel package.

Do you think this has something to do with the PHP problem?  I could try compiling it without the --with-mysql to see if that helps at all.

Were the backtraces helpful at all?

Thanks again for the help!

[2001-12-05 01:26 UTC] derick@php.net

If you use other module for apache, then PHP only, you _must_ use the MySQL libraries that are installed on your system, but not the bundled once. You should indeed try if --without-mysql would work for you. (Simply leaving out --with-mysql does not work, it's on by default).

Derick

[2001-12-08 05:14 UTC] tauren at servlets dot net

I just installed the MySQL-devel-3.23.40-1.i386.rpm RPM.  It installed the following files:

[root@s1 lib]# rpm -q -l MySQL-devel
/usr/bin/comp_err
/usr/bin/mysql_config
/usr/include/mysql
/usr/include/mysql/chardefs.h
/usr/include/mysql/dbug.h
/usr/include/mysql/errmsg.h
/usr/include/mysql/history.h
/usr/include/mysql/keymaps.h
/usr/include/mysql/m_ctype.h
/usr/include/mysql/m_string.h
/usr/include/mysql/my_config.h
/usr/include/mysql/my_global.h
/usr/include/mysql/my_list.h
/usr/include/mysql/my_net.h
/usr/include/mysql/my_no_pthread.h
/usr/include/mysql/my_pthread.h
/usr/include/mysql/my_sys.h
/usr/include/mysql/mysql.h
/usr/include/mysql/mysql_com.h
/usr/include/mysql/mysql_version.h
/usr/include/mysql/mysqld_error.h
/usr/include/mysql/raid.h
/usr/include/mysql/readline.h
/usr/include/mysql/sslopt-case.h
/usr/include/mysql/sslopt-longopts.h
/usr/include/mysql/sslopt-usage.h
/usr/include/mysql/sslopt-vars.h
/usr/include/mysql/tilde.h
/usr/lib/mysql
/usr/lib/mysql/libdbug.a
/usr/lib/mysql/libheap.a
/usr/lib/mysql/libmerge.a
/usr/lib/mysql/libmyisam.a
/usr/lib/mysql/libmyisammrg.a
/usr/lib/mysql/libmysqlclient.a
/usr/lib/mysql/libmysqlclient.la
/usr/lib/mysql/libmystrings.a
/usr/lib/mysql/libmysys.a
/usr/lib/mysql/libnisam.a

There isn't a libmysqlclient.so in this list, but there is a libmysqlclient.a.  However, it is in /usr/lib/mysql.  

What line should I use to configure PHP?  Is it still:
 --with-mysql=/usr

Or would it be:
 --with-mysql=/usr/lib/mysql

Or something else?

Thanks!

[2001-12-08 10:12 UTC] sniper@php.net

What would help us to narrow down the possible places where
this bug might be is that you build PHP with minimal amount of
extensions ie. remove all the options and leave only the apache
option there.

--Jani

[2001-12-08 10:14 UTC] derick@php.net

The RPm is called MySQL-shared BTW.

[2001-12-08 19:19 UTC] tauren at servlets dot net

Thanks for the help.  I've installed the MySQL-shared RPM and now have /usr/lib/libmysqlclient.so.  

Also, I've recompiled with a minimal PHP configuration:

CPPFLAGS=-I/usr/local/include
LDFLAGS=-L/usr/local/lib
./configure --prefix=/usr/local \
  --with-apache=/root/atb/Apachetoolbox-1.5.45/apache_1.3.22 \
--enable-exif \
--enable-track-vars \
--with-calendar=shared \
--enable-safe-mode \
--enable-magic-quotes \
--enable-trans-sid \
--enable-wddx \
--enable-ftp \
 --with-mysql=/usr \
 --with-ldap \

My Apache configuration is also very minimal.  The only things I have included are the basics plus mod_ssl and mod_php:

export CFLAGS=""
export LIBS=""
export INCLUDES=""
./configure --prefix=/usr/local/apache \
 --enable-suexec \
--suexec-caller=nobody \
--suexec-docroot=/home \
--suexec-logfile=/var/log/httpd/cgi.log \
--suexec-uidmin=500 \
--suexec-gidmin=500 \
--suexec-safepath=/usr/local/bin:/usr/bin:/bin \
--enable-module=so \
--enable-module=access \
--disable-module=auth_db \
--disable-module=digest \
--enable-module=imap \
--enable-module=mime \
--enable-module=setenvif \
--disable-module=usertrack \
--enable-module=auth \
--disable-module=cern_meta \
--disable-module=expires \
--enable-module=log_config \
--disable-module=proxy \
--disable-module=vhost_alias \
--disable-module=auth_anon \
--enable-module=cgi \
--disable-module=headers \
--disable-module=log_referer \
--disable-module=rewrite \
--enable-module=userdir \
--enable-module=asis \
--enable-module=autoindex \
--disable-module=example \
--disable-module=log_agent \
--enable-module=negotiation \
--enable-module=status \
--enable-module=actions \
--disable-module=auth_dbm \
--enable-module=dir \
--enable-module=include \
--disable-module=mime_magic \
--disable-module=unique_id \
--enable-module=alias \
--disable-module=auth_digest \
--enable-module=env \
--disable-module=info \
--disable-module=mmap_static \
--disable-module=speling \
 --enable-module=ssl \
 --activate-module=src/modules/php4/libphp4.a \

However, as within a minute of installing and running this Apache, I start getting runaway httpd processes (94% CPU usage).

It doesn't look like the libmysqlclient.so is the issue.  I'm going to try installing with the --without-mysql option to see if that makes a difference.

[2001-12-08 19:34 UTC] tauren at servlets dot net

This time I compiled PHP with this configuration:

CPPFLAGS=-I/usr/local/include
LDFLAGS=-L/usr/local/lib
./configure --prefix=/usr/local \
  --with-apache=/root/atb/Apachetoolbox-1.5.45/apache_1.3.22 \
--enable-exif \
--enable-track-vars \
--with-calendar=shared \
--enable-safe-mode \
--enable-magic-quotes \
--enable-trans-sid \
--enable-wddx \
--enable-ftp \
 --without-mysql \ 
 --without-ldap \

Apache was just the same as before.  However, I am still getting the same runaway process problem.

Any other ideas or suggestions for me to try?

[2001-12-12 09:12 UTC] sniper@php.net

Like I asked before, configure/compile PHP without ANY configure
options. (only --with-apxs)

And updating kernel (and compiling it by yourself) is not bad idea either.

[2001-12-12 19:40 UTC] yohgaki@php.net

I also suggest to check RedHat resources to see if there is any known problems with their SMP kernel. There are issues like this for SMP kernel occasionally.

[2002-01-02 14:05 UTC] lobbin@php.net

No feedback. Closing.

	php.net \| support \| documentation \| report a bug \| advanced search \| search howto \| statistics \| random bug \| login
go to bug id or search bugs for


Copyright © 2001-2024 The PHP Group All rights reserved.	Last updated: Thu Apr 18 00:01:28 2024 UTC