php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #44070 Zip is unusuable for large numbers of files or large data volume
Submitted: 2008-02-07 16:25 UTC Modified: 2008-02-07 16:43 UTC
From: mike at thetroubleshooters dot dk Assigned: pajoye (profile)
Status: Not a bug Package: Zip Related
PHP Version: 5.2.5 OS: Linux Fedora 8
Private report: No CVE-ID: None
View Add Comment Developer Edit
Anyone can comment on a bug. Have a simpler test case? Does it work for you on a different platform? Let us know!
Just going to say 'Me too!'? Don't clutter the database with that please !
Your email address:
MUST BE VALID
Solve the problem:
39 - 26 = ?
Subscribe to this entry?

 
 [2008-02-07 16:25 UTC] mike at thetroubleshooters dot dk
Description:
------------
I have a system where I archive a very large amount of data, both the number of files can exceed 32000 and the volume of data can be much greater than 500M, then the ZipArchive is unusable, because it appears that it is impossible to get the ZipArchive to flush data, so that the memory usage, and file handle usage does not become excessive.

Reproduce code:
---------------
Do a simple loop to add 32000 files of 25k in size, and it is very easy to reproduce.

Using Addfile fails silently, because you run out of filehandles.

Using AddFromString fails with an out of memory error.

Expected result:
----------------
I expected that the ZipArchive would start to flush data to the temporary file, at some stage, to prevent excessive memory or filehandle usage.

Actual result:
--------------
Using the AddFromString, quickly fills memory, and eventually will exceed the memory available, this is unfortunately a side effect of not processing the data as it is added.

Program fails with an out of memory error.

Using AddFile, will eventually run out of filehandles, unless the amount of filehandles are increased to enormous levels, which is not practical, nor a good idea.

Program fails with no message in any logfile.

In either case the script fails, which is annoying.

I know that I am pushing the system to the limits with what I am doing, and likely the problem won't hit a lot of people.  I read in another bug report, that data is not processed until close.

The Main problem with the problem is that the addFile fails silently - that cost me a lot of time to diagnose.

I recommend adding a flush() to the zipArchive, as this would solve the problem, if the problem cannot be solved easily by internal code, this would allow the coder to periodically flush it.   

Ofc.  A open and close can possibly be used, but seems very clumsy.

Patches

Add a Patch

Pull Requests

Add a Pull Request

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2008-02-07 16:43 UTC] pajoye@php.net
Hi!

Thanks for your report!

See the bug in pecl.php.net/zip and here. The "problem" was already reported.

"Ofc.  A open and close can possibly be used, but seems very clumsy."

yes it is but it is the way to go for now.

duplicate > bogus.
 
PHP Copyright © 2001-2024 The PHP Group
All rights reserved.
Last updated: Thu Apr 25 19:01:33 2024 UTC