I've a strange situation on a physical Windows Server 2008 R2 Server. It hosts Citrix 6.5 (uses SQL 2008 back end), Terminal Services Licensing (for citrix) and IIS for an application that's hosted. Backups in DPM 2012 R2 started failing so I've been troubleshooting using wbadmin on the server affected and noticed the following: If I run wbadmin start backup -backuptarget:\\Backup01\recover\backuptest -allcritical it fails with this error message:
A Volume Shadow Copy Service operation failed. Please check "VSS" and "SPP" application event logs for more information.
ERROR - A Volume Shadow Copy Service operation error has occurred: (0x800423f4)
The writer experienced a non-transient error. If the backup process is retried, the error is likely to reoccur.
The event log shows these four errors:
Event 521, Backup
The backup operation that started at '2015-10-29T15:20:44.890000000Z' has failed because the Volume Shadow Copy Service operation to create a shadow copy of the volumes being backed up failed with following error code '2155348129'. Please review the event details for a solution, and then rerun the backup operation once the issue is resolved.
Event 16387, SPP
Shadow copy creation failed because of error reported by ASR Writer. More info: The system cannot find the path specified. (0x80070003).
Event 12290, VSS
Volume Shadow Copy Service warning: ASR writer Error 0x80070003. hr = 0x00000000, The operation completed successfully.
.
Operation:
PrepareForBackup event
Context:
Execution Context: ASR Writer
Execution Context: Writer
Writer Class Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
Writer Name: ASR Writer
Writer Instance ID: {edaf3834-5fd7-4216-a3f3-c93a45992fac}
Error-specific details:
ASR Writer: The system cannot find the path specified. (0x80070003)
Event 8193, VSS
- Volume Shadow Copy Service error: Unexpected error calling routine IVssAsrWriterBackup::GetAsrMetadata. hr = 0x80070003, The system cannot find the path specified.
.
Operation:
PrepareForBackup event
Context:
Execution Context: ASR Writer
Execution Context: Writer
Writer Class Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
Writer Name: ASR Writer
Writer Instance ID: {edaf3834-5fd7-4216-a3f3-c93a45992fac}
Error-specific details:
ASR Writer: The system cannot find the path specified. (0x80070003)
The following VSS Writers also go into a [7] failed state/Timed Out:
- IIS Metabase Writer
- WMI Writer
- IIS Config Writer
- TermServLicensing
- SqlServerWriter
- System Writer
If I restart all of the services associated with these writers, the state returns to [1] stable until a BMR backup runs again. I can runwbadmin start backup -BackupTarget:\\Backup01\recover\backuptest -include:c:or wbadmin start backup -BackupTarget:\\Backup01\recover\backuptest -include:\\?\Volume{ab7be5ee-33c9-11e4-adf9-806e6f6e6963}and it completes successfully.
If I run wbadmin start backup -BackupTarget:\\Backup01\recover\backuptest -include:C:,\\?\Volume{ab7be5ee-33c9-11e4-adf9-806e6f6e6963}it fails with the same error message as above. The \\?\... drive is a 100MB System Reserved partition that is Active. C: holds the operating system.
What I've tried so far:
- Set VSS shadowstorage to unbounded for both partitions
- Installed all windows/SQL updates
- Re-registered the VSS writer DLLs
cd /d %windir%\system32 net stop vss net stop swprv regsvr32 /s ole32.dll regsvr32 /s oleaut32.dll regsvr32 /s vss_ps.dll vssvc /register regsvr32 /s /i swprv.dll regsvr32 /s /i eventcls.dll regsvr32 /s es.dll regsvr32 /s stdprov.dll regsvr32 /s vssui.dll regsvr32 /s msxml.dll regsvr32 /s msxml3.dll regsvr32 /s msxml4.dll vssvc /register net start swprv net start vss
- Confirmed automount was enabled in Dispart.
- Restarted the server
- Being a production server, the only thing I haven't tried is to run a BMR with the SQL service stopped.
What do I try next short of rebuilding this thing?