php.net |  support |  documentation |  report a bug |  advanced search |  search howto |  statistics |  random bug |  login
Bug #22744 Split hangs on something that 'file' does quickly
Submitted: 2003-03-17 09:04 UTC Modified: 2003-04-24 04:30 UTC
Votes:1
Avg. Score:5.0 ± 0.0
Reproduced:0 of 0 (0.0%)
From: tim at timcrider dot com Assigned:
Status: Not a bug Package: Performance problem
PHP Version: 4.3.2RC1 OS: Red Hat 8.0
Private report: No CVE-ID: None
Welcome back! If you're the original bug submitter, here's where you can edit the bug or add additional notes.
If you forgot your password, you can retrieve your password here.
Password:
Status:
Package:
Bug Type:
Summary:
From: tim at timcrider dot com
New email:
PHP Version: OS:

 

 [2003-03-17 09:04 UTC] tim at timcrider dot com
split hangs for some reason, but doing the same logical process with other functions returns a very quick result.

The problem is that I'm trying to process the mysqlbinlog output and parse it for easier management. The logical process is as follows.

$sys = `mysqlbinlog logfile`;

This part works fine, but when I try:

$lstk = split("\n", $sys);

the script hangs for well over 5 minutes (I kill the script at that time. However this code produces the desired results in less than 2 seconds:

 $mybin     = "/usr/local/mysql/bin/mysqlbinlog";
 $mylogpath = "/var/lib/mysql/";

 $dolog = $mylogpath."towely-bin.010";

 $tmpfname = tempnam ("/tmp", "mybinlogger");   
 $fp = fopen($fp, $tmpfname);
 fwrite($fp, `$mybin $dolog`);
 fclose($fp);
 
 $lstk = file($tmpfname);
 unset($tmpfname);
 print count($lstk)."\n";

I am using this from the command line version of php 4.3.2 RC1. Here is my php -v:

PHP 4.3.2-RC1 (cli) (built: Mar 14 2003 13:28:33)
Copyright (c) 1997-2003 The PHP Group
Zend Engine v1.3.0, Copyright (c) 1998-2003 Zend Technologies
    with the ionCube PHP Accelerator v1.3.3r2, Copyright (c) 2001-2002, by Nick Lindridge


 If you need any other information please feel free to contact me at tim@timcrider.com

Patches

Pull Requests

History

AllCommentsChangesGit/SVN commitsRelated reports
 [2003-03-18 12:41 UTC] wez@php.net
Your second test script look bogus to me.

Have you considered using http://www.php.net/manual/en/ref.exec.php, which gives the lines as an array anyway?

How big is the logfile?  Do you have magic_quotes enabled?
If you leave it running, how long does it take to complete and how much memory does it use?


 [2003-03-18 13:02 UTC] tim at timcrider dot com
I apologize the second script is bogus. Here is the fix:

---[ works.php ]---
#!/bin/php
<?php
 $mybin     = "/usr/local/mysql/bin/mysqlbinlog";
 $mylogpath = "/var/lib/mysql/";

 $dolog = $mylogpath."towely-bin.010";
 $lstk = exec("mysqlbinlog $dolog");

 print_r($lstk);

 print count($lstk)."\n";
?>
(towely) phpbug# more test.php 
#!/bin/php
<?php
 $mybin     = "/usr/local/mysql/bin/mysqlbinlog";
 $mylogpath = "/var/lib/mysql/";

 $dolog = $mylogpath."towely-bin.010";

 $tmpfname = tempnam ("/tmp", "mybinlogger");   
 $fp = fopen($tmpfname, "w");
 fwrite($fp, `$mybin $dolog`);
 fclose($fp);
 
 $lstk = file($tmpfname);
 unset($tmpfname);
 print count($lstk)."\n";
?>
--[ End works.php ]---

---[ broken.php ]---
#!/bin/php
<?php
 $mybin     = "/usr/local/mysql/bin/mysqlbinlog";
 $mylogpath = "/var/lib/mysql/";

 $dolog = $mylogpath."towely-bin.010";
 
 $sys = `mysqlbinlog $dolog`; 
 $lstk = split("\n", $sys);
 
 print_r($lstk);
 
 print count($lstk)."\n";
?>
---[ End broken.php ]---
Using exec does not give me the output that I need to do this. exec only returns the last line of the output. I'm trying to get all of the output, which has line breaks at the end of every row.

I did a dump of this server using 'php -i > php_ini.txt
and can be seen here.

http://timcrider.com/php_ini.txt

If there is anything else you need me to do let me know.

 Tim
 [2003-03-18 13:03 UTC] tim at timcrider dot com
I apologize again. This is what 'works.php' SHOULD look like:


---[ works.php ]---
#!/bin/php
<?php
 $mybin     = "/usr/local/mysql/bin/mysqlbinlog";
 $mylogpath = "/var/lib/mysql/";
 
 $dolog = $mylogpath."towely-bin.010";
 
 $tmpfname = tempnam ("/tmp", "mybinlogger");
 $fp = fopen($tmpfname, "w");
 fwrite($fp, `$mybin $dolog`);
 fclose($fp);
 
 $lstk = file($tmpfname);
 unset($tmpfname);
 print count($lstk)."\n";
?>
--[ End works.php ]---
 [2003-03-18 15:07 UTC] wez@php.net
http://www.php.net/manual/en/function.exec.php
The second parameter will be filled with each line of the command.

I'm still interested to hear what the size of the data is:
echo strlen($sys);


 [2003-03-18 15:27 UTC] tim at timcrider dot com
Here is a quick size test I did:

---[ size.php ]---
#!/bin/php
<?php
 $mybin     = "/usr/local/mysql/bin/mysqlbinlog";
 $mylogpath = "/var/lib/mysql/";

 $dolog = $mylogpath."towely-bin.010";

 $sys = `mysqlbinlog $dolog`;
 print "Backtick :: ".strlen($sys)."\n";

 ob_start();
 system("mysqlbinlog $dolog");          
 $sys = ob_get_contents();
 ob_end_clean();
 print "System :: ".strlen($sys)."\n";
 
 $sys = exec("mysqlbinlog $dolog", $lstk);
 print "Exec :: ".strlen($sys)."\n";
 print count($lstk)."\n";
?>
---[ End size.php ]---

Here's the output

---[ Begin Output ]---
(towely) phpbug# ./sizes.php 
Backtick :: 3120815
System :: 3120815
Exec :: 79
30626
--[ End Output ]---

If there code in particular you'd like me to run, you could email it to me @ tim at timcrider dot com  and I'll post the output of those tests here.
 [2003-04-01 00:40 UTC] pollita@php.net
3MB is not a trivial amount of data to do an operation like split() on.  It's also worth noting (per your summary) that file() does not read all the data into a single var and perform a split, that sort of wasteful processing is left to scripts.

The first response I'd give (which addresses what you're asking for) would be to say that you should be using explode(), not split().  Using regex for this task is like using a hammer to close a CD tray.

But that's not the best way to do this anyway.

You notice how count($lstk) == 30626 ? Have you looked at the contents of the $lstk array?

I think print_r($lstk) might surprise you.  Hint: Wez already told you what you'd find.
 [2003-04-24 04:30 UTC] sniper@php.net
Normal.

 
PHP Copyright © 2001-2024 The PHP Group
All rights reserved.
Last updated: Sun Dec 22 11:01:30 2024 UTC