- &$value is a refernce used in the foreach loop.
- If the unset is commented out, the reference to $stuff[1] as $value on the second iteration of the loop works.
- The call to unset() on the first iteration breaks the reference

Actaul Output:
array(1) { [1]=> string(3) "two" }

Expected Output
array(1) { [1]=> string(3) "This should appear below in the var_dump() because $value is passed by reference" }

<?php
$stuff = array('one','two');
foreach ($stuff as $key => &$value) {
	if($key==0){
	    unset ($stuff[$key]);
	}else{
	    $value='This should appear below in the var_dump() because $value is passed by reference';
	}
}
var_dump($stuff);
?>

You are modifying array in the foreach loop.
Consider using for/while instead of foreach.
See this code:
<?php
$stuff = array('one','two');
foreach ($stuff as $key => &$value) {
    $value='This should appear below in the var_dump() because $value is passed by reference';
}
var_dump($stuff);
?>

Although I can use a workaround for this problem (and in the meantime I have), this is still a bug in php isn't it?

After using unset() on the array, in the loop, the reference breaks.

Even more fun:  Try this variant and you'll see that $fluff not only gets reset to NULL, it gets completely unset.

<?php
$stuff = array('one','two');
$fluff = &$stff;
var_dump($stuff);
var_dump($fluff);
foreach ($stuff as $key => &$value) {
    if($key==0){
        unset ($stuff[$key]);
    }else{
        $value='This should appear below in the var_dump() because $value
is passed by reference';
    }
}
var_dump($stuff);
var_dump($fluff);
?>

Yes, the above example fails because of the obvious
typo there:
-$fluff = &$stff;
+$fluff = &$stuff;

After that change it works as expected. No bug here.