I also tested using ssh2_exec to simply do a file tansfer using cat > filename, however I am limited to 128k of transfer per call (so multiple cat >> filename are necessary after closing & opening new streams after 128k are written)  With this method I achieved 45KB/sec connected on 127.0.0.1 compared to around 30KB/sec when using SFTP or SCP.

Is there a way to increase the size of the write buffers or just do some magic to make it faster?

BTW, my reproduce code above is bugged, here is something that actually works:


#!/usr/local/php5/bin/php
<?php
$src_file = 'test/data.src';
$dst_file = 'mirrored/data.dst';

// get filesize
$filesize = filesize($src_file);

// establish connection to FTP over SSH
$connection = ssh2_connect('127.0.0.1', 22);
ssh2_auth_password($connection, 'user', 'password');

$time = microtime(true);

// ** SFTP ** 
/*
$sftp = ssh2_sftp($connection);
print "Connected to FTP over SSH\n";
copy($src_file, 'ssh2.sftp://' . $sftp . $dst_file);
*/

// ** SSH + cat **
/*
$fp_in = fopen($src_file, 'r');
$stream = ssh2_exec($connection, 'cat > ' . $dst_file);

$written_bytes = 0;
while ($written_bytes < $filesize) {
	$written_bytes += stream_copy_to_stream($fp_in, $stream, 131072);

	if ($written_bytes < $filesize) {
		fclose($stream);
		$stream = ssh2_exec($connection, 'cat >> ' . $dst_file);
	}
}
fclose($fp_in);
fclose($stream);
*/
$time = microtime(true) - $time;
print 'Time: ' . round($time, 2) . ' :: ' . round($filesize / 1024 / $time, 2) . "KB/sec\n";
?>

Just a note so you know I'm not ignoring this.  A couple other people have reported similar problems.  It's the #1 boadblock to a stable release at this point.

As I was going through the PHP module code, I noticed the stream_close handling does a php_sleep(2).

I am unsure if it is really needed because it means closing a SFTP connection always takes more than 2 seconds.  For the purpose of testing transfer speed, I commented out this part and re-run my code with shell execute of 'cat >' and I reached transfer speed of around 350KB/sec.. which is  a lot better already.  

Now I would greatly prefer using a SFTP wrapper with the function 'copy', it is much cleaner.

Side note:  there should be a simple function to close a SSH / SFTP / SCP connection other than fclose() or unset()

Sorry for bugging you with my problems :)

Yeah, I can't for the life of me remember why I put that sleep in there.  Probably leftover debugging cruft.

Of course as you noticed the transfer is still more than a little slow.  I've identified a few places in the library where the same buffer gets duplicated and shufled around unnecessarily, they'll be trimmed down.

As to a full fledged ssh2_sftp_close() / ssh2_disconnect() type function:  Yes that could be done.  It would mean registering each channel with the session resource and each file handle with their respective SFTP resource, then itterating through that array/subarray structure to close them all down.

The trouble with that approach is allowing the connection variable to fall out of scope without triggering the shutdown mechanism unless no children are active.  If children were active when it fell out of scope, but they all go inactive later then the cleanup process does have to be triggered.  Again, doable, but hairy from an internal standpoint.  

I took the approach that made the most sense to me, from a top-down model.  Create resources (connections, channels, handles), then destroy resources (handles, channels, connections).

If you'd like to supply a patch which provides this functionality I'd be willing to review and possibly include it.  I'm not personally planning on working on that functionality, but who knows...

Do you think the sleep command can be safely removed?

Also, have you figured out why data transfer is so slow, is it a PHP only issue?

Thanks a lot for your help

[ignore] updated my email address to something that actually works

I was wondering why I saw so many
nanosleep({0, 250000000}, NULL)         = 0
in my stack trace.  After some investigating, I found this is caused by this:
libssh2/src/packet.c: libssh2_blocking_read()
...
ret = recv(session->socket_fd, buf + bytes_read, count - bytes_read, LIBSSH2_SOCKET_RECV_FLAGS(session));
... this line very often return -1 because of socket error: EAGAIN (normal non blocking msg)

for test purposes, I modified the function to use select() calls and the speed increased from 30k/sec to 150k/sec.  Basically I avoid all the 0.25 sec sleep that occures everytime a EAGAIN occures.  If you add this up for a normal buffer length of 8k, it means a maximum of 32kb/sec if each read returns -1 once.

Here is the new beginning I modified:

/* {{{ libssh2_blocking_read
 * Force a blocking read, regardless of socket settings
 */
static int libssh2_blocking_read(LIBSSH2_SESSION *session, unsigned char *buf, size_t count)
{
        size_t bytes_read = 0;
        int polls = 0;

        /* added socket select code */
        struct timeval timeout;
        timeout.tv_sec = 30;
        timeout.tv_usec = 0;
        int socket_count;

        while (bytes_read < count) {
                int ret;


                /* added socket select code */
                fd_set read_socket;
                FD_ZERO(&read_socket);
                FD_SET(session->socket_fd, &read_socket);
                socket_count = select(session->socket_fd+1, &read_socket, NULL, NULL, &timeout);

                if (socket_count <= 0) {
                        return -1;
                }


That's that.  I realize that your socket handling is not quite optimal and I suspect a similar problem also occures when establishing a connection.  In my debugging, I found thousands of recv returning EAGAIN before the connection was succesfully completed.

I was wondering if your streams support the set_buffer function as this would also help reduce the problem.

I hope this helps you complete your debugging and cleans the way for a stable package!

Please try out libssh2 0.10 and pecl/ssh2 0.8

No feedback was provided. The bug is being suspended because
we assume that you are no longer experiencing the problem.
If this is not the case and you are able to provide the
information that was requested earlier, please do so and
change the status of the bug back to "Open". Thank you.

if you dont want to use fclose($stream), you can patch it!

for closing SFTP subsystems, Publickey Subsystems and Tunnels; use ssh2_subsys_close()

open ssh2.c file.

1) Add following lines before " PHP_FUNCTION(ssh2_fingerprint) " and after " /* }}} */ "


/* {{{ proto stream ssh2_subsys_close(stream channel)
 * Close the SFTP subsystem/Publickey Subsystem or a Listener and return a true on success, false on error
 */
PHP_FUNCTION(ssh2_subsys_close)
{
	zval *zsubsys;

	/* check & parse parameters */
	if (zend_parse_parameters(ZEND_NUM_ARGS() TSRMLS_CC, "r", &zsubsys) == FAILURE) {
		RETURN_FALSE;
	}

	/* decrease the refcount and call destructor if needed */
	if (zend_list_delete(Z_LVAL_P(zsubsys)) == FAILURE) {
		RETURN_FALSE;
	}

	/* destruct pointer */
	zval_ptr_dtor(&zsubsys);

	RETURN_TRUE;
}
/* }}} */


2) Declare the ssh2_subsys_close() function again by adding a line after the existing PHP_FE() line.

find following line :

	PHP_FE(ssh2_connect,						NULL)

after add:

	PHP_FE(ssh2_subsys_close,					NULL)


----

for closing channels (ssh2_exec, ssh2_shell) use ssh2_channel_close()

1) Add following lines to the end of "ssh2_fopen_wrappers.c" file


/* ******************
   * Channel Closer *
   ****************** */

/* {{{ proto stream ssh2_channel_close(stream channel)
 * Close a SSH2 Channel
 */
PHP_FUNCTION(ssh2_channel_close)
{
	php_stream *channel;
	zval *zchannel;

	/* check & parse parameters */
	if (zend_parse_parameters(ZEND_NUM_ARGS() TSRMLS_CC, "r", &zchannel) == FAILURE) {
        RETURN_FALSE;
    }

	php_stream_from_zval(channel, &zchannel);

	/* close php stream */
	php_stream_close(channel);

	/* destruct pointer */
	zval_ptr_dtor(&zchannel);

	RETURN_TRUE;
}
/* }}} */


2) Declare the ssh2_channel_close() function again by adding a line after the existing PHP_FE() line.

find following line (in ssh2.c) :

	PHP_FE(ssh2_fetch_stream,					NULL)

after add:

	PHP_FE(ssh2_channel_close,					NULL)


find following line (in php_ssh2.h) :

PHP_FUNCTION(ssh2_fetch_stream);

after add:

PHP_FUNCTION(ssh2_channel_close);