php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #44070 Zip is unusuable for large numbers of files or large data volume
Submitted: 2008-02-07 16:25 UTC Modified: 2008-02-07 16:43 UTC
From: mike at thetroubleshooters dot dk Assigned: pajoye (profile)
Status: Not a bug Package: Zip Related
PHP Version: 5.2.5 OS: Linux Fedora 8
Private report: No CVE-ID: None
Welcome back! If you're the original bug submitter, here's where you can edit the bug or add additional notes.
If this is not your bug, you can add a comment by following this link.
If this is your bug, but you forgot your password, you can retrieve your password here.
Password:
Status:
Package:
Bug Type:
Summary:
From: mike at thetroubleshooters dot dk
New email:
PHP Version: OS:

 

 [2008-02-07 16:25 UTC] mike at thetroubleshooters dot dk
Description:
------------
I have a system where I archive a very large amount of data, both the number of files can exceed 32000 and the volume of data can be much greater than 500M, then the ZipArchive is unusable, because it appears that it is impossible to get the ZipArchive to flush data, so that the memory usage, and file handle usage does not become excessive.

Reproduce code:
---------------
Do a simple loop to add 32000 files of 25k in size, and it is very easy to reproduce.

Using Addfile fails silently, because you run out of filehandles.

Using AddFromString fails with an out of memory error.

Expected result:
----------------
I expected that the ZipArchive would start to flush data to the temporary file, at some stage, to prevent excessive memory or filehandle usage.

Actual result:
--------------
Using the AddFromString, quickly fills memory, and eventually will exceed the memory available, this is unfortunately a side effect of not processing the data as it is added.

Program fails with an out of memory error.

Using AddFile, will eventually run out of filehandles, unless the amount of filehandles are increased to enormous levels, which is not practical, nor a good idea.

Program fails with no message in any logfile.

In either case the script fails, which is annoying.

I know that I am pushing the system to the limits with what I am doing, and likely the problem won't hit a lot of people.  I read in another bug report, that data is not processed until close.

The Main problem with the problem is that the addFile fails silently - that cost me a lot of time to diagnose.

I recommend adding a flush() to the zipArchive, as this would solve the problem, if the problem cannot be solved easily by internal code, this would allow the coder to periodically flush it.   

Ofc.  A open and close can possibly be used, but seems very clumsy.

Patches

Add a Patch

Pull Requests

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2008-02-07 16:43 UTC] pajoye@php.net
Hi!

Thanks for your report!

See the bug in pecl.php.net/zip and here. The "problem" was already reported.

"Ofc.  A open and close can possibly be used, but seems very clumsy."

yes it is but it is the way to go for now.

duplicate > bogus.
 
PHP Copyright © 2001-2024 The PHP Group
All rights reserved.
Last updated: Fri Apr 26 05:01:30 2024 UTC