Actually at the moment this is a known issue which we cannot fix appropriate right away. But we are working on the matter.

I have a similar but different reproduction using two dimensional arrays instead of explicitly setting a reference.

class ArrayAccessImpl implements ArrayAccess {
  private $data = array();

  public function offsetUnset($index) {}

  public function offsetSet($index, $value) {
    $this->data[$index] = $value;
  }

  public function offsetGet($index) {
    return $this->data[$index];
  }

  public function offsetExists($index) {
    return isset($this->data[$index]);
  }
}

$data = new ArrayAccessImpl();
$data['element'] = array();
$data['element']['element2'] = "hi";

Thank you for your bug report. This issue has already been fixed
in the latest released version of PHP, which you can download at 
http://www.php.net/downloads.php

We found out that this is not solvable without blowing up the interface and creating a BC or providing an additional interface to support references and thereby creating an internal nightmare - actually i don't see a way we can make that work ever. Thus we decided to enforce the original design and disallow references completley. Also the error message should have changed with 5.0.4 or is being changed with 5.0.5/5.1 at least

Excuse me if I'm being obtuse here but this doesn't really solve the problem and renders the ArrayAccess interface useless for anything other than 1-dimensional arrays. So you can't do:
$data = new ArrayAccessImpl(); // using class in example above
$data['level_1'] = array();
$data['level_1']['level_2'] = 'some data';

Suggestion:
1. Change the ArrayAccess interface to support references - so as to fix the problem.
2. Reduce the severity of the error triggered for class declarations that don't implement the ArrayAccess methods with references from "fatal" to "warning" (so that the application does not terminate).
3. If I'm not amiss very little will need to be changed in the SPL to then handle data passed to & from the (ArrayAccess) methods whether passed by reference or not. This will "maintain" BC.
4. Once the old implementation of the ArrayAccess methods have been deprecated you can remove the interface check specific to ArrayAccess which will then revert to a fatal.
* The fatal error I'm talking about would be: "Declaration of ArrayAccessImpl::offsetSet() must be compatible with that of ArrayAccess::offsetSet()"... and for the other methods.
* You could do this as a major revision. I would rather know that I need to update the classes that implement ArrayAccess before upgrading to say PHP.v6 rather than have to live with this limitation. Basically I will now not use ArrayAccess again because of this bug.

why not to add new interface RecursiveArrayIterator, and add return by reference 
in it?