TF14043: An error occurred computing the delta. Not enough storage is available to process this command |
|
Author |
Message |
Grant Holliday
|
Posted: Team Foundation Server - Version Control, TF14043: An error occurred computing the delta. Not enough storage is available to process this command |
Top |
We're attempting to migrate a fairly sizeable SoruceSafe database into a dual-tier TFS server configuration using VSSConverter.
After running analyze.exe, VSSConverter Analyze, then half-way through VSSConverter Migrate we start seeing errors in the EventLog.
The app-tier server had been running for about 3 weeks, so we've restarted it and are starting a fresh VSSConverter Migrate.
Searching for the win32 error "Not enough storage space is available" seems to indicate that it's run out of memory. However this is a Dual Xeon hyper-threaded box with 3.5Gb of memory and plenty of free disk space.
The 2.0 Framework machine.config has <processModel autoConfig="true" />
Can you provide any clues as to what might cause this
TF53010: An unexpected condition has occurred in a Team Foundation component. The information contained here should be made available to your site administrative staff. Technical Information (for the administrative staff): Date (UTC): 31/08/2006 11:06:36 PM Machine: APPTIERSERVER Application Domain: /LM/W3SVC/3/Root/VersionControl-1-128014889023460494 Assembly: Microsoft.TeamFoundation.Common, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a; v2.0.50727 Process Details: Process Name: w3wp Process Id: 1020 Thread Id: 1272 Account name: DOMAIN\USERNAME
Detailed Message: TF14043: An error occurred computing the delta for item $/TeamProject/Folder/Path.wsdl version 42403.System.ComponentModel.Win32Exception: Not enough storage is available to process this command at Microsoft.TeamFoundation.VersionControl.Server.PatchLibrary.handlePatchError() at Microsoft.TeamFoundation.VersionControl.Server.PatchLibrary.CreateFile(String oldFileName, String newFileName, String patchFileName) at Microsoft.TeamFoundation.VersionControl.Server.ContentManager.ComputeSingleDelta(ContentComponent db, PendingDelta delta, String[] tempFiles) at Microsoft.TeamFoundation.VersionControl.Server.ContentManager.ComputePendingDeltas() Exception Message: Not enough storage is available to process this command (type Win32Exception)
Exception Stack Trace: at Microsoft.TeamFoundation.VersionControl.Server.PatchLibrary.handlePatchError() at Microsoft.TeamFoundation.VersionControl.Server.PatchLibrary.CreateFile(String oldFileName, String newFileName, String patchFileName) at Microsoft.TeamFoundation.VersionControl.Server.ContentManager.ComputeSingleDelta(ContentComponent db, PendingDelta delta, String[] tempFiles) at Microsoft.TeamFoundation.VersionControl.Server.ContentManager.ComputePendingDeltas()
Visual Studio Team System14
|
|
|
|
|
Richard Berg MSFT
|
Posted: Team Foundation Server - Version Control, TF14043: An error occurred computing the delta. Not enough storage is available to process this command |
Top |
Is the data tier disk equally healthy looking That's where deltas are stored.
|
|
|
|
|
Grant_Holliday
|
Posted: Team Foundation Server - Version Control, TF14043: An error occurred computing the delta. Not enough storage is available to process this command |
Top |
The data tier is healthy. No errors or warnings in the event log.
It's a Dual-Proc 3.4Ghz (hyper-threading disabled) with 3.5Gb RAM and SAN storage. Plenty of disk space available.
|
|
|
|
|
Buck Hodges
|
Posted: Team Foundation Server - Version Control, TF14043: An error occurred computing the delta. Not enough storage is available to process this command |
Top |
The reverse delta patches are created on the application tier by using mspatch and then stored in the data tier. There's a little bit of information on it at http://blogs.msdn.com/jimlamb/articles/tfsfaq.aspx.
What happens when I check in an update to a file in version control
TFS uses the reverse delta model as most version control systems do. However, there’s a few notable things that TFS does. First the creation of the reverse deltas is done asynchronously as a background task. Initially the new copy is uploaded and saved. Sometime later (usually not very long), a background job will run and convert the older version from a full copy to a reverse delta (assuming the delta is actually smaller than the original – usually, but not always the case).
Because recreation of old versions can be time consuming (especially if you have to apply many reverse deltas), TFS stores the copies of the most recently used files in a cache on the application tier (AT) machine. This mostly eliminates the performance difference between accessing the most recent version vs. older versions.
Note: The reverse delta mechanism works for text and binary.
One thing that's not mentioned there is that we only compute deltas for files that are 16 MB or smaller.
My best guess would be that the "$/TeamProject/Folder/Path.wsdl" file is triggering some bug in either the patch library itself or the way we invoke it.
Does this error get reported repeatedly for the same file
Does it happen on any other files or just this one
Buck
|
|
|
|
|
Grant_Holliday
|
Posted: Team Foundation Server - Version Control, TF14043: An error occurred computing the delta. Not enough storage is available to process this command |
Top |
We've now incurred this error 3 times. However all three times the Application Event Log on the App-Tier has filled up and we've lost the original errors.
There has never been anything of interest in the data-tier event log.
However, last night we enabled perfmon on the application-tier while trying to migrate again.
Between 22:23 and 22:40 the available megabytes dropped by 1Gb.
At 22:50 VSSConverter.exe failed with the following errors in VSSConverter.log
[VersionControl, Error, 18, 2006/09/04 22:50:44.736] Exception: System.Web.Services.Protocols.SoapException Message: TF10217: Team Foundation was unable to complete your request due to an unexpected error. For more details, see the event log on the Team Foundation Server. Stack Trace: at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall) at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters) at Microsoft.TeamFoundation.VersionControl.Proxy.Repository.CheckIn(String workspaceName, String ownerName, String[] serverItems, Changeset info, CheckinNotificationInfo checkinNotificationInfo, CheckinOptions checkinOptions, Failure[]& failures) at Microsoft.TeamFoundation.Converters.VersionControl.Common.HatterasWrapper.CheckIn(PreparedComits comit) Help Link: BaseExceptionMessage: TF10217: Team Foundation was unable to complete your request due to an unexpected error. For more details, see the event log on the Team Foundation Server.
[VersionControl, Error, 18, 2006/09/04 22:50:44.736] SubCode.Code.Name: OutOfMemoryException [VersionControl, Error, 18, 2006/09/04 22:50:44.736] Exception: Microsoft.TeamFoundation.Converters.VersionControl.Common.ConverterHatterasException Message: TF10217: Team Foundation was unable to complete your request due to an unexpected error. For more details, see the event log on the Team Foundation Server. Stack Trace: at Microsoft.TeamFoundation.Converters.VersionControl.Common.HatterasWrapper.CheckIn(PreparedComits comit) at Microsoft.TeamFoundation.Converters.VersionControl.Common.HatterasWrapper.ActivityCheckIn(Object stateInfo) Help Link: BaseExceptionMessage: TF10217: Team Foundation was unable to complete your request due to an unexpected error. For more details, see the event log on the Team Foundation Server. InnerException: Microsoft.TeamFoundation.Converters.VersionControl.Common.ConverterHatterasException InnerException Message: TF10217: Team Foundation was unable to complete your request due to an unexpected error. For more details, see the event log on the Team Foundation Server. InnerException Stack Trace: at Microsoft.TeamFoundation.Converters.VersionControl.Common.HatterasWrapper.CheckIn(PreparedComits comit) at Microsoft.TeamFoundation.Converters.VersionControl.Common.HatterasWrapper.ActivityCheckIn(Object stateInfo) InnerException Help Link: InnerExceptionBaseExceptionMessage: TF10217: Team Foundation was unable to complete your request due to an unexpected error. For more details, see the event log on the Team Foundation Server.
[VersionControl, Error, 18, 2006/09/04 22:50:44.814] Exception: System.InvalidOperationException Message: Client found response content type of 'text/html; charset=utf-8', but expected 'application/soap+xml'. The request failed with the error message: -- < xml version="1.0" > <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd"> <html xmlns="http://www.w3.org/1999/xhtml" > <head> <title>Server Unavailable </title> </head> <body> <h1><span style="font-family:Verdana;color: #ff3300">Server Application Unavailable </span></h1> <p> <span style="font-family:Verdana;"> The web application you are attempting to access on this web server is currently unavailable.A Please hit the "Refresh" button in your web browser to retry your request. </span></p> <p> <b>Administrator Note: </b> An error message detailing the cause of this specific request failure can be found in the application event log of the web server. Please review this log entry to discover what caused this error to occur. </p> </body> </html>
--. Stack Trace: at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall) at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters) at Microsoft.TeamFoundation.VersionControl.Proxy.Repository.UndoPendingChanges(String workspaceName, String ownerName, ItemSpec[] items, Failure[]& failures) at Microsoft.TeamFoundation.Converters.VersionControl.Common.HatterasWrapper.UndoPendingChanges(ItemSpec[] undoItems, Workspace ws) Help Link: BaseExceptionMessage: Client found response content type of 'text/html; charset=utf-8', but expected 'application/soap+xml'. The request failed with the error message: -- < xml version="1.0" > <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" "http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd"> <html xmlns="http://www.w3.org/1999/xhtml" > <head> <title>Server Unavailable </title> </head> <body> <h1><span style="font-family:Verdana;color: #ff3300">Server Application Unavailable </span></h1> <p> <span style="font-family:Verdana;"> The web application you are attempting to access on this web server is currently unavailable.A Please hit the "Refresh" button in your web browser to retry your request. </span></p> <p> <b>Administrator Note: </b> An error message detailing the cause of this specific request failure can be found in the application event log of the web server. Please review this log entry to discover what caused this error to occur. </p> </body> </html>
--.
[VersionControl, Warning, 18, 2006/09/04 22:50:44.814] Repository is down. Retrying... [VSS, Error, 3, 2006/09/04 22:50:47.814] Exception: System.Runtime.InteropServices.COMException Message: Error HRESULT E_FAIL has been returned from a call to a COM component. Stack Trace: at Microsoft.VisualStudio.SourceSafe.Interop.IVSSItems.get_Item(Object sItem) at Microsoft.TeamFoundation.Converters.VersionControl.Vss.VssWrapper.GetItemFromAncestor(VSSItem item, ActionMetadata info) Help Link: BaseExceptionMessage: Error HRESULT E_FAIL has been returned from a call to a COM component. *** lots of these COMExceptions until 22:54 ** [VersionControl, Warning, 3, 2006/09/04 22:59:54.605] Reached max limit for queue: PendChangeQueue, thread: Main Thread [VersionControl, Warning, 18, 2006/09/04 23:25:51.044] Repository is back ** this is probably when the app-pool restarted **
We have confirmed that there's no system job that happens at 22:00. And in any case, we have seen this same failure (OutOfMemoryException) at other points in the day with our previous migrate attempts.
Comparing the logs of all three attempts, I pieced together the last "Source changeset" and TFS changeset messages before the failure (hoping they were the same):
- Source: 3054, TFS: 40656, Time before failure: 8H 27M (* I can't explain why this one took longer, we rebooted after this attempt as it had been running for 3 weeks)
- Source: 3084, TFS: 46815, Time before failure: 5H 35M
- Source: 3081, TFS: 81943, Time before failure: 5H 3M
Our next steps are:
- Increase the size of the application event log to a large number
- Turn on tracelevel 4 for VSSConverter
- Empty the application-tier cache
- Try the migration again.
- Raise a support call with the extended logs.
Can you think of anything else that would be worth looking in to
|
|
|
|
|
Grant_Holliday
|
Posted: Team Foundation Server - Version Control, TF14043: An error occurred computing the delta. Not enough storage is available to process this command |
Top |
Hi Buck,
To answer your two questions:
> Does this error get reported repeatedly for the same file > Does it happen on any other files or just this one
No, the error is repeating. Each entry is a different file. Here is an excerpt from 22:46.
Detailed Message: TF14043: An error occurred computing the delta for item $/TeamProject/Stream/Application_/Application.GenManifest/AssemblyInfo.cs version 81763
Detailed Message: TF14043: An error occurred computing the delta for item $/TeamProject/Stream/Application_/Application.GenManifest/Application.GenManifest.csproj version 81763.
Detailed Message: TF14043: An error occurred computing the delta for item $/TeamProject/Stream/Application_/Application.GenManifest/GenManifestMain.cs version 81763.
Detailed Message: TF14043: An error occurred computing the delta for item $/TeamProject/Stream/Updater/Application.GenManifest/GenManifestMain.cs version 81780.
Detailed Message: TF14043: An error occurred computing the delta for item $/TeamProject/Stream/Updater/Application.GenManifest/GenManifestMain.cs version 81803
Detailed Message: TF14043: An error occurred computing the delta for item $/TeamProject/Stream/Application_/Application.Interceptor/AssemblyInfo.cs version 81763.
Detailed Message: TF14043: An error occurred computing the delta for item $/TeamProject/Stream/Application_/Application.RunOnce/AssemblyInfo.cs version 81763
Detailed Message: TF14043: An error occurred computing the delta for item $/TeamProject/Stream/Application_/Application.Service/AssemblyInfo.cs version 81763
Thanks,
Grant
|
|
|
|
|
Mario Rodriguez
|
Posted: Team Foundation Server - Version Control, TF14043: An error occurred computing the delta. Not enough storage is available to process this command |
Top |
Hi Grant:
I recommend that you contact Product Support so we can help you further with this problem since it seems that is an issue where we will need more information and data from you.
thanks, mario
|
|
|
|
|
Christopher J. Cooper
|
Posted: Team Foundation Server - Version Control, TF14043: An error occurred computing the delta. Not enough storage is available to process this command |
Top |
Cold you migrate the Db in smaller pieces using the mappings and settings files
|
|
|
|
|
Grant_Holliday
|
Posted: Team Foundation Server - Version Control, TF14043: An error occurred computing the delta. Not enough storage is available to process this command |
Top |
Thanks Mario. We have now raised a PSS call.
I will let you know the outcome.
|
|
|
|
|
Buck Hodges
|
Posted: Team Foundation Server - Version Control, TF14043: An error occurred computing the delta. Not enough storage is available to process this command |
Top |
This is a bit of a shot in the dark, but given the number of files that are triggering the problem, you may want to check the version control db to make sure some disk error hasn't occurred. Here's the final bit from a thread where bizarre things were happening and were traced back to the db index being corrupted on disk due to a power failure.
From the thread http://forums.microsoft.com/MSDN/ShowPost.aspx PostID=550091&SiteID=1&PageID=1:
I think we've gotten to the bottom of the issue. Here's what Chandru used to determine the problem.
The TFSVersionControl database is corrupt. The table tbl_Version has a corrupt index (IX_tbl_VersionedItem_InitialVersionFrom) and is missing entries for some items – hence you are seeing null values when any query uses the particular index.
Usually such corruption occurs due to a faulty disk or a power outage. Serge noted that a power outage had occurred not long before the problem started happening.
The following command detects database corruption.
run dbcc checkdb('TFSVersionControl')
Scroll down and you should see allocation errors (in red).
In this case, the corruption in the index was fixed using the following procedure.
1. Backup your existing database
2. Put the database in single user mode by running:
alter database TFSVersionControl
SET SINGLE_USER
2. Repair the corruption, by running:
dbcc checkdb('tfsversionControl_bkp', REPAIR_REBUILD)
3. Restore the database to multi_user mode by running:
alter database TFSVersionControl
SET MULTI_USER
4. Verify that version control works correctly.
Buck
|
|
|
|
|
Grant_Holliday
|
Posted: Team Foundation Server - Version Control, TF14043: An error occurred computing the delta. Not enough storage is available to process this command |
Top |
Thanks Buck - it was worth a try.
CHECKDB found 0 allocation errors and 0 consistency errors in database 'TfsVersionControl'.
|
|
|
|
|
Grant_Holliday
|
Posted: Team Foundation Server - Version Control, TF14043: An error occurred computing the delta. Not enough storage is available to process this command |
Top |
Just to update, we have tried another migrate of the same VSS, and then a migrate of a smaller VSS and we received this exception on the app-tier (for each migration) at a random point during the migration.
TF53010: An unexpected condition has occurred in a Team Foundation component. The information contained here should be made available to your site administrative staff. Technical Information (for the administrative staff): Date (UTC): 12/09/2006 6:10:51 PM Machine: EPTFS001 Application Domain: /LM/W3SVC/3/Root/VersionControl-3-128025267317385077 Assembly: Microsoft.TeamFoundation.Common, Version=8.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a; v2.0.50727 Process Details: Process Name: w3wp Process Id: 4808 Thread Id: 5916 Account name: DOMAIN\User
Detailed Message: TF53017: Request processing is being shut down due to the exception shown below. Web Request Details Url: http://server:8080/VersionControl/v1.0/repository.asmx [method: POST] User Agent: Team Foundation (VSSConverter.exe, 8.0.50727.147) Headers: Content-Length=451049&Content-Type=application%2fsoap%2bxml%3b+charset%3 dutf-8%3b+action%3d%22http%3a%2f%2fschemas.microsoft.com%2fTeamFoundatio n%2f2005%2f06%2fVersionControl%2fClientServices%2f03%2fQueryItems%22&Acc ept-Encoding=gzip&Accept-Language=en-AU&Expect=100-continue&Host=eptfs00 1%3a8080&User-Agent=Team+Foundation+(VSSConverter.exe%2c+8.0.50727.147)& X-TFS-Version=1.0.0.0 Path: /VersionControl/v1.0/repository.asmx Local Request: False Host Address: 10.X.X.X User: DOMAIN\USER [authentication type: NTLM]
Exception Message: Exception of type 'System.OutOfMemoryException' was thrown. (type OutOfMemoryException)
Exception Stack Trace: at System.Data.SqlClient._SqlMetaDataSet..ctor(Int32 count) at System.Data.SqlClient.TdsParser.ProcessMetaData(Int32 cColumns, TdsParserStateObject stateObj) at System.Data.SqlClient.TdsParser.Run(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj) at System.Data.SqlClient.SqlDataReader.ConsumeMetaData() at System.Data.SqlClient.SqlDataReader.NextResult() at Microsoft.TeamFoundation.VersionControl.Server.VersionedItemComponent.Qu eryItems(String serverItem, Int32 changesetId, Int32 deletionId, RecursionType recursive, DeletedState deleted, ItemType itemType, IList items, String& queryPath, String& filePattern) at Microsoft.TeamFoundation.VersionControl.Server.VersionSpec.QueryItems(It emSpec itemSpec, Workspace localWorkspace, VersionedItemComponent db, DeletedState deletedState, ItemType itemType, IList items, String& queryPath, String& filePattern) at Microsoft.TeamFoundation.VersionControl.Server.DeferredQuery.Execute(Wor kspace localWorkspace, VersionedItemComponent db) at Microsoft.TeamFoundation.VersionControl.Server.DeferredQuery.Execute(Lis t`1 queries, Workspace localWorkspace, VersionedItemComponent db) at Microsoft.TeamFoundation.VersionControl.Server.ItemSet.QueryItems(IPrinc ipal userPrincipal, Workspace localWorkspace, ItemSpec[] items, VersionSpec version, DeletedState deleted, ItemType itemType, Boolean generateDownloadUrls) at Microsoft.TeamFoundation.VersionControl.Server.Repository.QueryItems(Str ing workspaceName, String workspaceOwner, ItemSpec[] items, VersionSpec version, DeletedState deletedState, ItemType itemType, Boolean generateDownloadUrls)
PSS advised us that "The server ran out of memory while running QueryItems()." and asked how many files and folders are involved in the migration.
Our VSS database includes: 91,986 files 8,782 folders 1.27Gb total.
The SourceSafe\data folder itself is: 353,467 files 35 folders 2.66Gb total.
I also tried a migration of our smaller VSS database overnight, and it failed with the same exception.
The size of that database is: 38,242 files 4,065 folders 941Mb total.
The SourceSafe\data folder is: 120,826 files 407 folders 1.37Gb
The output of http://yourtfsserver:8080/VersionControl/v1.0/administration.asmx/QueryRepositoryInformation is the following: UserCount="182" GroupCount="1536" WorkspaceCount="456" ShelvesetCount="0" FileCount="1288939" FolderCount="130117" MaxChangesetID="154659" PendingChangeCount="13"
It would also be worth noting that we have tried this migration 3-4 times using the same Team Project name, and deleting it with TfsDeleteProject after each attempt. But of course, TFS still maintains all the old changesets.
Grant
|
|
|
|
|
|
|