Description:
------------
while testing phpDocumentor 1.3.0RC2 (which is due out today), I discovered a hang at a token_get_all(), but only if I parse two large files in a row.

if I parser CHMdefaultConverter.inc and Converter.inc at the same time, the token_get_all() call simply hangs infinitely.  If I parse CHMdefaultConverter by itself, the code works perfectly.

The problem is that T_DOC_COMMENT is destroyed, but should be treated in the same way as T_COMMENT

This patch fixes the bug:

Index: ext/tokenizer/tokenizer.c
===================================================================
RCS file: /repository/php-src/ext/tokenizer/tokenizer.c,v
retrieving revision 1.25
diff -u -r1.25 tokenizer.c
--- ext/tokenizer/tokenizer.c	17 Jan 2004 17:44:27 -0000	1.25
+++ ext/tokenizer/tokenizer.c	9 Feb 2004 18:22:57 -0000
@@ -345,6 +345,7 @@
 			case T_OPEN_TAG_WITH_ECHO:
 			case T_WHITESPACE:
 			case T_COMMENT:
+                       case T_DOC_COMMENT:
 			case T_CLOSE_TAG:
 				destroy = 0;
 				break;


Reproduce code:
---------------
parse both phpDocumentor/Converter.inc and phpDocumentor/Converters/CHM/default/CHMdefaultConverter.inc using phpDocumentor 1.3.0RC2 in the latest snapshot.

Expected result:
----------------
works

Actual result:
--------------
hangs