SVNSCC Performance with PowerBuilder

( SVNSCC , PowerBuilder 11, 1.5.1.1, WIN XP SP2 )

Type: Public Status:Closed Created: 12 Jan 09 01:00 Updated: 06 May 09 05:00

--> Bryan Moore (user) at 06 May 09 05:00 writes

I think that the inclusion of all the applib child folders in the 2nd
status call was the reason for it taking 68 seconds. As mentioned before,
we modified our directory structure to place all those objects in the pbl's
directly under the applib folder in a new child/leaf folder; and, that
dramatically improved the performance.
Though there may be other settings changes or implementation techniques
that could further improve performance, this single change in the structure
of the folders for our pbls -- that is, to place each pbl in a "leaf"
(having no children) folder -- made a dramatic performance improvement and
allowed us to go to production with the SVNSCC solution for our
PowerBuilder application source control. We've now been using SVNSCC for 3
months to develope our monthly production releases and are satisfied with
the results, including the performance. We can consider this issue closed.
Thank you for your support.

--> Bryan Moore (user) at 16 Mar 09 03:00 writes

2.5 megs is not unreasonable if it was including all the child folders --
109 pbl's and about 4500 objects.

We are currently on SVN 1.4.2 (server) and Apache 2.0.59

--> Alexander Somov (admin) at 10 Mar 09 03:00 writes

The only thing that looks a bit strange is a huge traffic which is
generated during the second call to svn_client_status2() - about 2.5
megabytes! Our tests have shown a much lower traffic consumption. Which
version of server do you use? Our test server is based on svn 1.5.1 and
Apache 2.2.9.

--> Alexander Somov (admin) at 09 Mar 09 03:00 writes

Let's look at the example of how it works. I will use the log that you sent
earlier.

1. PowerBuilder calls SccQueryInfoEx for one file,
"C:\svn\test0113\testscc.txt". The plug-in checks that SVN URL is valid (it
takes about 1 second), then it calls svn_client_status2() for
"C:\svn\test0113" (about 9.5 seconds). Then the plug-in returns status for
"C:\svn\test0113\testscc.txt".

2. PowerBuilder makes pause for about 2.5 seconds. It's too long, so the
plug-in reset its cache and will check the status again during the next
operation.

3. PowerBuilder calls SccQueryInfoEx for 7 files:
"c:\svn\test0113\entrack.pbt",
"c:\svn\test0113\applib\n_cst_socket_data.sru",
"c:\svn\test0113\applib\n_cst_wininet_ftp.sru",
"c:\svn\test0113\applib\n_cst_winsock.sru",
"c:\svn\test0113\applib\urptab_socket_server.sru",
"c:\svn\test0113\applib\urptabpg_socket_server_interface.sru",
"c:\svn\test0113\applib\wrp_socket_server.srw".

As you can see, one of them located in "c:\svn\test0113", and the others in
"c:\svn\test0113\applib". So the plug-in fetches status for both folders.
It takes about 8.5 seconds for the first one and about 68 seconds for the
second. Note that all this time we were waiting for the completion of the
svn_client_status2() function, so there is nothing to optimize.

So what we can do here? I think, nothing. By increasing the cache storage
time we can get a performance gain of about 10%, but only in this
scenario.

You can also try to adjust the SccMaxArraySize parameter in the PB.INI
file. By default PowerBuilder limits SCC requests to batches of 25 objects.
Our plug-in has no such limitations and can handle up to several thousand
files at one time.

--> Bryan Moore (user) at 06 Mar 09 03:00 writes

I ran the command-line status and found the following :

* running against a PBL in the root folder takes about 30 seconds
* running against a PBL in a leaf folder takes about 1 second

We have about about 110 PBL's and 4500 objects in our application.

These are both much quicker than what we are seeing with PB/SVNSCC.

The log messages make me think that a status refresh is requested with each
object. If each status request operates against the whole folder and that
pbl has many objects then we have an n-squared increase in the amount of
time required to refresh the status of a PBL.

Can you confirm?

--> Alexander Somov (admin) at 04 Mar 09 03:00 writes

Your assumption is right. The plug-in fetches status for all files in
directory even when IDE asks us just for several files. This is due to
Subversion API which allows to get status only for directories (or for a
single file, but there's no performance difference for these operations).

Most likely, you are faced with the problem because you are using https://
protocol through a slow network connection. In our tests, we found that
native svn:// protocol can operate up to 10 times faster, but it is not so
secure and may not be suitable for your purposes.

I suggest you download the command-line client from the Subversion
website:

http://subversion.tigris.org/files/documents/15/44582/svn-win32-1.5.5.zip

It requires no installation. Just unzip the archive to any directory. You
should specify full path to exe when you run it, for example:

C:\svn\test0113\applib>C:\TEMP\svn-win32-1.5.5\bin\svn.exe status -uv

Please check how long does it take to get file status using native svn
client. Our plug-in will never work faster than Subversion itself, because
it just translates SCC API calls to SVN calls (sometimes one to many, if
required). We have some kinds of optimization, for example, we store the
fetched status in the cache for 1 second after the last operation. But
PowerBuilder sometimes makes a pause for a few seconds between calls to the
SCC API functions, so this cached info is not used.

About Tortoise. I think that the equivalent command would be "Check for
modifications", and then click on "Check repository" button.

--> Bryan Moore (user) at 28 Feb 09 02:00 writes

I'm getting an error message :
'svn' is not recognized as an internal or external command
when attempting to run svn st -uv in my applib directory.

I do not have command-line svn installed. We connect via https: to a
remote/secure server. What is the equivalent Tortoise command? Or, should
I ask our source control admin team to run this command on the Subversion
server?

Thanks.

Craig

--> Bryan Moore (user) at 02 Feb 09 02:00 writes

I will do as you instruct, but first consider the following....
The difference is that the pbl is in the applib directory and that applib
is a root folder for several folders each having pbl's in them.
I worked around the problem by creating a new subfolder in the applib
directory and moving the pbl from the applib folder to the new folder.
I'm sure you can reproduce if you place a pbl in a root folder with
several/many pbl's in child folders below it.
Thank you.
_______________________________________
Craig Kilstrom
Specialist, IT App Development
Inventory Asset Management (ENTRACK)
One IT Applications, Corporate Apps, NSC
(Work) 614-249-2470
(Fax) 614-677-2168
Nationwide IS On Your Side
From:
support@pushok.com
To:
kilstrm@nationwide.com
Date:
02/02/2009 08:21 AM
Subject:
[pst2148] SVNSCC Performance with PowerBuilder
http://www.pushok.com/tickets_addmodify.php?id=2148
Could you please check how long it will take to execute \'svn st -uv\'
command in your c:/svn/test0113/applib directory and, if possible, traffic
consumption? We can see in your log that this operation takes more than 1
minute and eats about 1.5 Mb of traffic, but we cannot reproduce such
behavior in our environment (we got few seconds and about 15 Kbytes on
3000 files using \'https\' access method).

--> Alexander Somov (admin) at 02 Feb 09 02:00 writes

Could you please check how long it will take to execute 'svn st -uv'
command in your c:/svn/test0113/applib directory and, if possible, traffic
consumption? We can see in your log that this operation takes more than 1
minute and eats about 1.5 Mb of traffic, but we cannot reproduce such
behavior in our environment (we got few seconds and about 15 Kbytes on 3000
files using 'https' access method).

--> Bryan Moore (user) at 26 Jan 09 01:00 writes

While I was able to check out a couple of files using the debug dll's,
several attempts at checking in these files all failed. Checking in using

the normal dll's worked (but slowly -- this is the problem).

Here's the debug text files :

I did come up with a work-around. Moving all the files now located in a
parent folder to a new child folder resulted in dramatic speed
improvements. I think this verifies my assumption that the slow
performance is due to the status refresh of every file in the folder even
when only a single file(/object) is being checked in or out.

Please let me know how I can help more with your debugging.

Thank you.

debuglog.zip

--> Alexander Somov (admin) at 23 Jan 09 01:00 writes

I will try to deal with this problem as soon as possible. Right now you can
collect information on what exactly happens, using the debug DLLs. Download
them here:

http://www.pushok.com/files/temp/sccapi_svn1300d.zip
http://www.pushok.com/files/temp/svncom_151d.zip
http://www.pushok.com/files/temp/PushokSVNSCC_1513d.zip

Unzip to overwrite existing. You will find debuglog.txt and
protocollog.txt near the DLL. Send them to me for analysis.

Thank you in advance!

--> Bryan Moore (user) at 23 Jan 09 01:00 writes

We've heard nothing since this change in assignment.
This ticket was incorrectly opened with a "Low" priority. Multiple
requests to change the priority have gone unanswered.
Please update the ticket priority and provide a status.
Thank you.
_______________________________________
Craig Kilstrom
Specialist, IT App Development
Inventory Asset Management (ENTRACK)
One IT Applications, Corporate Apps, NSC
(Work) 614-249-2470
(Fax) 614-677-2168
Nationwide IS On Your Side
From:
To:
kilstrm@nationwide.com
Date:
01/19/2009 06:57 AM
Subject:
[pst2148] SVNSCC Performance with PowerBuilder
http://www.pushok.com/tickets_addmodify.php?id=2148
Responsible person for your ticket changed
from 'Oleg Akolupin' to 'Alexander Somov'

--> Bryan Moore (user) at 23 Jan 09 01:00 writes

We've isolated the problem to source files in a root folder. Because
SVNSCC checks the status of all files in the folder (and its' children)
even when only an individual object is being accessed, the response time is
extremely slow.

--> Bryan Moore (user) at 20 Jan 09 01:00 writes

Please update Priority to "Normal".
Thank you.
_______________________________________
Craig Kilstrom
Specialist, IT App Development
Inventory Asset Management (ENTRACK)
One IT Applications, Corporate Apps, NSC
(Work) 614-249-2470
(Fax) 614-677-2168
Nationwide IS On Your Side
From:
To:
kilstrm@nationwide.com
Date:
01/19/2009 06:57 AM
Subject:
[pst2148] SVNSCC Performance with PowerBuilder
http://www.pushok.com/tickets_addmodify.php?id=2148
Responsible person for your ticket changed
from 'Oleg Akolupin' to 'Alexander Somov'

--> Bryan Moore (user) at 12 Jan 09 01:00 writes

We are experiencing very slow response times from Subversion using SVNSCC.

Cannot isolate a situation where it is worse but here is an example.

When attempting to undo checkout status of an object, the status window
displays the following as it chugs along for a few minutes :

Fetching status for : c:/svn/entrack
Fetching status for : c:/svn/entrack/applib
Fetching file list for: C:/svn/ENTRACK/APPLIB
Fetching status for : C:/svn/ENTRACK/APPLIB

From these messages it appears to be making a list of and checking the
status on every object in the folder. It may not be doing exactly what
it's saying, but this would account for the extremely slow performace. Why
wouldn't it isolate the checks to the selected object instead of doing them
on the entire folder?

As I mentioned, this is only a single example of the performance problems
we're experiencing. Could document more if needed.

Thank you for looking into this. We are evaluating for use of SVNSCC as
the source control agent for our PowerBuilder applications.

Rate this ticket:

Not useful at all

Partially useful

Useful

Very useful

Ticket

SVNSCC Performance with PowerBuilder