php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Request #29571 [PATCH] Many uploads will slow down Linux machine
Submitted: 2004-08-08 14:42 UTC Modified: 2022-04-08 07:34 UTC
From: k at ailis dot de Assigned: ilutov (profile)
Status: Closed Package: *General Issues
PHP Version: * OS: Linux
Private report: No CVE-ID: None
Welcome back! If you're the original bug submitter, here's where you can edit the bug or add additional notes.
If this is not your bug, you can add a comment by following this link.
If this is your bug, but you forgot your password, you can retrieve your password here.
Password:
Status:
Package:
Bug Type:
Summary:
From: k at ailis dot de
New email:
PHP Version: OS:

 

 [2004-08-08 14:42 UTC] k at ailis dot de
Description:
------------
This is a mix of "performance problem" and "feature request". So feel free to reclassify this report as "Feature request" if you think it fits better.

In a  large project we have the problem that many users are uploading many large files to our PHP server. PHP can handle this very well but the problem is, that all files are stored in the same directory. Our upload directory sometimes contains more than 20000 files. This produces high loads because accessing these files is very I/O intensive because of the flat directory hierarchy. 

You already have addressed this problem with session files by introducing a "directory depth". Its possible to use "2;/var/lib/php/uploads" as session.save_path which is a very nice feature. Exactly this feature would be a cool thing if it were also implemented for "upload_tmp_dir".

Here is a small patch as an example how this feature could be implemented. We are using this patch on our servers and it works great. This patch not only implements this feature for upload_tmp_dir but for all other modules which are using the php_open_temporary_file() function.

--- php_open_temporary_file.c.orig	Sun Aug  8 14:29:06 2004
+++ php_open_temporary_file.c	Sun Aug  8 14:29:12 2004
@@ -106,6 +106,10 @@
 {
 	char *trailing_slash;
 	char *opened_path;
+	char *p;
+	char *sub_dirs;
+	size_t dirdepth, i;
+	int rand_value;
 	int fd = -1;
 #ifndef HAVE_MKSTEMP
 	int open_flags = O_CREAT | O_TRUNC | O_RDWR
@@ -125,6 +129,30 @@
 	if (!(opened_path = emalloc(MAXPATHLEN))) {
 		return -1;
 	}
+	
+	/* Check subdirectory depth and point p to real path name */
+	if ((p = strchr(path, ';')))
+	{
+	    dirdepth = (size_t) strtol(path, NULL, 10);
+	    p++;
+	}
+	else
+	{
+	    dirdepth = 0;
+	    p = (char *) path;
+	}
+    
+	/* Build subdirectory tree if needed */
+	if (dirdepth) {
+	    if (!(sub_dirs = emalloc(dirdepth * 2 + 1))) return -1;
+	    sub_dirs[0] = 0;
+	    for (i = 0; i < dirdepth; i++)
+	    {
+	        strcat(sub_dirs, "x/");
+	        rand_value = php_rand() & 0xf;
+	        sub_dirs[i * 2] = rand_value < 10 ? rand_value + 48 : rand_value + 87;
+	    }
+	} else sub_dirs = "";
 
 	if (IS_SLASH(path[strlen(path)-1])) {
 		trailing_slash = "";
@@ -132,7 +160,10 @@
 		trailing_slash = "/";
 	}
 
-	(void)snprintf(opened_path, MAXPATHLEN, "%s%s%sXXXXXX", path, trailing_slash, pfx);
+	(void)snprintf(opened_path, MAXPATHLEN, "%s%s%s%sXXXXXX", p, trailing_slash, sub_dirs, pfx);
+	
+	/* Free memory if it was used */
+	if (dirdepth) efree(sub_dirs);
 
 #ifdef PHP_WIN32
 	if (GetTempFileName(path, pfx, 0, opened_path)) {
@@ -140,7 +171,7 @@
 		 * which means that opening it will fail... */
 		VCWD_CHMOD(opened_path, 0600);
 		fd = VCWD_OPEN_MODE(opened_path, open_flags, 0600);
-	}
+l1	}
 #elif defined(NETWARE)
 	/* Using standard mktemp() implementation for NetWare */
 	file_path = mktemp(opened_path);



Patches

Add a Patch

Pull Requests

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2015-01-08 23:08 UTC] ajf@php.net
-Package: Feature/Change Request +Package: *General Issues -PHP Version: 4.3.8 +PHP Version: *
 [2015-01-08 23:08 UTC] ajf@php.net
Seems reasonable.
 [2022-04-08 07:34 UTC] ilutov@php.net
-Status: Open +Status: Closed -Assigned To: +Assigned To: ilutov
 [2022-04-08 07:34 UTC] ilutov@php.net
Thanks for the patch! I'm not sure if this is still relevant with more modern file systems. If it is, please create a new issue or preferably a patch on GitHub.
 
PHP Copyright © 2001-2024 The PHP Group
All rights reserved.
Last updated: Fri Apr 19 07:01:27 2024 UTC