php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #78808 [LMDB] MDB_MAP_FULL: Environment mapsize limit reached
Submitted: 2019-11-13 10:21 UTC Modified: 2020-01-03 17:53 UTC
Votes:1
Avg. Score:5.0 ± 0.0
Reproduced:1 of 1 (100.0%)
Same Version:1 (100.0%)
Same OS:1 (100.0%)
From: progr-d at yandex dot ru Assigned: cmb (profile)
Status: Closed Package: DBM/DBA related
PHP Version: 7.3.11 OS: macOS 10.13.6
Private report: No CVE-ID: None
 [2019-11-13 10:21 UTC] progr-d at yandex dot ru
Description:
------------
Presently php not provide any way to change `mapsize` for lmdb databases and they limited by size.

How can I fix this? Now, probably, I can only recompile the lmdb library...


PS: build php in macos with lmdb support: https://gist.github.com/saippuakauppias/fac2a33dfe420adb4a6cf19c257eef46

Test script:
---------------
<?php
$lmdb_h = dba_open('test.db', 'c', 'lmdb');
for ($i = 0; $i < 50000; $i++) {
    dba_insert('key' . $i, 'value '. $i, $lmdb_h);
}
dba_close($lmdb_h);


Actual result:
--------------
➜ php prefill_lmdb.php
PHP Warning:  dba_insert(key18505,value 18505): MDB_MAP_FULL: Environment mapsize limit reached in /Users/usr/prefill_lmdb.php on line 6

Warning: dba_insert(key18505,value 18505): MDB_MAP_FULL: Environment mapsize limit reached in /Users/usr/prefill_lmdb.php on line 6
[1]    42370 segmentation fault  php prefill_lmdb.php

Patches

Pull Requests

Pull requests:

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2019-11-13 11:30 UTC] cmb@php.net
From the docs[1]:

| Default size of memory map. This is certainly too small for any
| actual applications. Apps should always set the size explicitly
| using mdb_env_set_mapsize().

So yes, there likely should be some dba_open() argument to allow
for a custom map size.

[1] <http://www.lmdb.tech/doc/group__internal.html#gac08cad5b096925642ca359a6d6f0562a>
 [2019-11-13 11:46 UTC] progr-d at yandex dot ru
Yes, I found a description about this:

> The size should be a multiple of the OS page size. The default is 10485760 bytes. The size of the memory map is also the maximum size of the database. The value should be chosen as large as possible, to accommodate future growth of the database. This function should be called after mdb_env_create() and before mdb_env_open(). It may be called at later times if no transactions are active in this process. Note that the library does not check for this condition, the caller must ensure it explicitly.

(from http://www.lmdb.tech/doc/group__mdb.html#gaa2506ec8dab3d969b0e609cd82e619e5 )


It turns out that this needs to be done here: https://github.com/php/php-src/blob/PHP-7.2.0/ext/dba/dba_lmdb.c#L61

---------------

Strange thing: 10485760 bytes is 10MB, but my Test script create file only with 1MB size (you can check it). 

---------------

> there likely should be some dba_open() argument

dba_open ( string $path , string $mode [, string $handler [, mixed $... ]] )

contains the fourth argument, but nowhere is there a description of how to use it: https://www.php.net/manual/en/function.dba-open.php
 [2019-11-13 13:34 UTC] cmb@php.net
> contains the fourth argument, but nowhere is there a description
> of how to use it

Arguments following the $handler are converted to string and are
passed to the individual DBA drivers, and are driver dependent.
This is indeed not yet documented; let alone which drivers support
which parameters.

> It turns out that this needs to be done here:

Yes.  info->argc and info->argv provide the driver specific
arguments which have been passed to dba_open().
 [2019-11-13 17:38 UTC] progr-d at yandex dot ru
I can send a patch that will add mdb_env_set_mapsize() at the specified location, but with a fixed size (e.g. 1GB). But as far as I understand it is much more correct to give users the opportunity to change this behavior themselves (through the fourth argument). And for this, you already need to write tests (in which I am not strong for PHP tests).

For macOS homebrew users I made hotfix (hacker-way style), which increases the size of the database in the library itself (by the way, there is an error - it really is 1MB in size, and the documentation says 10MB): https://gist.github.com/saippuakauppias/95b6f7b139b3d117b7dac4aeead9f5b4
 [2019-11-13 19:07 UTC] progr-d at yandex dot ru
I haven’t written anywhere that I need to do this right now and forget about other problems. I just talked about what I met myself and shared solution methods.
If you have such a toxic community here that because of every bug you will write such offensive words - I probably will not send anything else.
 [2019-11-13 19:58 UTC] nikic@php.net
Deleted comments by "henry dot paradiz at gmail dot com". A bunch of spam has recently been submitted under this email.
 [2019-11-14 10:33 UTC] cmb@php.net
The following pull request has been associated:

Patch Name: Fix #78808: [LMDB] MDB_MAP_FULL: Environment mapsize limit reached
On GitHub:  https://github.com/php/php-src/pull/4910
Patch:      https://github.com/php/php-src/pull/4910.patch
 [2020-01-03 17:52 UTC] cmb@php.net
Automatic comment on behalf of cmbecker69@gmx.de
Revision: http://git.php.net/?p=php-src.git;a=commit;h=c05a069adfcca56763d5db06afce8801382477a5
Log: Fix #78808: [LMDB] MDB_MAP_FULL: Environment mapsize limit reached
 [2020-01-03 17:52 UTC] cmb@php.net
-Status: Open +Status: Closed
 [2020-01-03 17:53 UTC] cmb@php.net
-Assigned To: +Assigned To: cmb
 
PHP Copyright © 2001-2024 The PHP Group
All rights reserved.
Last updated: Thu Nov 21 08:01:29 2024 UTC