Jobs stuck indefinitely in Queued Status
We have had an ongoing issue for about 2 months now, and since we have had a clean build (3 times now) for Backup Exec 2012. We have opened numerous cases with Symantec to resolve this, and they claim the first time that HotFix 209149 (see http://www.symantec.com/business/support/index?page=content&id=TECH204116 ) corrects the issue. The issue is also noted by seeing Robotic Element errors stating that OST media is full and it brings the "virtual" slot for the PureDisk or OST storage device offline. Restarting the services in BE only makes the problem worse and causes a snowball effect whereby jobs constantly error in the ADAMM log files. Essentially, the jobs never can get a concurrency/virtual slot and they stay Queued forever. I have seen others on this Forum with this problem, and while the Forum administrator seems to mark them as "Solved", they are not - because I see the threads drop off with no resolution identified. Are other people having this problem? If so, how are you overcoming it? Once it starts the environment is essentially dead in the water because the jobs never start (they sit Queued forever) - save for one concurrency which for our size environment is only 1/4 the need we have. We use CAS and 4 MMS servers with 2008 R2 with all patches applied, PureDisk 32TB volume on each MMS, Data Domain OST connection to DD-670 OST with OST-Plug 2.62 (28TB), replicated catalogs, and duplicate jobs for optimized deduplication between MMSs. We run BE 2012 SP3 clean - we reinstalled with SP3 slipstreamed because Symantec said this problem could be fixed through database repair by them manually or by reinstall...we chose reinstall (even though they did offer to "fix" the issue with database repair). We chose reinstall to validate whether SP3 truly fixes the issue. It is clear to us it does not. We are looking for anyone else who has had this problem to report into this forum. Thank you, Dana14KViews4likes15Comments
Duplication fails with database system error (220)
Hi there. I'm running Netbackup 7.1 on a RHEL 5 and I'm having the error "database system error (220)" when the duplication process runs. Here's the full log: 16/Jun/2011 15:19:43 - begin Duplicate 16/Jun/2011 15:19:43 - requesting resource backupserver-hcart3-robot-tld-0 16/Jun/2011 15:19:43 - requesting resource backupserver-hcart3-robot-tld-0 16/Jun/2011 15:19:43 - granted resource FC0281 16/Jun/2011 15:19:43 - granted resource HP.ULTRIUM3-SCSI.001 16/Jun/2011 15:19:43 - granted resource backupserver-hcart3-robot-tld-0 16/Jun/2011 15:19:43 - granted resource FC0330 16/Jun/2011 15:19:43 - granted resource HP.ULTRIUM3-SCSI.000 16/Jun/2011 15:19:43 - granted resource backupserver-hcart3-robot-tld-0 16/Jun/2011 15:20:54 - Info bpdbm (pid=20149) cataloguing file number 10000 path /System State/Active Directory/CN=Configuration/CN=DisplaySpecifiers/CN=40B/CN=msExchOmaConnector-Display/classDisplayName 16/Jun/2011 15:21:29 - Info bpdbm (pid=20149) cataloguing file number 20000 path /System State/Active Directory/CN=Configuration/CN=DisplaySpecifiers/CN=413/CN=serviceAdministrationPoint-Display/ 16/Jun/2011 15:22:21 - Info bpdbm (pid=20149) cataloguing file number 30000 path /System State/Active Directory/CN=Configuration/CN=DisplaySpecifiers/CN=816/CN=msExchIMGlobalSettingsContainer-Display/ 16/Jun/2011 15:23:00 - Info bpdbm (pid=20149) cataloguing file number 40000 path /System State/Active Directory/CN=Configuration/CN=Services/CN=Microsoft Exchange/CN=Universidade de Lisboa/CN=Addressing/CN=Display-Templates/CN=40F/displayName 16/Jun/2011 15:24:10 - Info bpdbm (pid=20149) cataloguing file number 50000 path /System State/Active Directory/CN=Configuration/CN=Services/CN=Microsoft Exchange/CN=Universidade de Lisboa/CN=ELC Folders Container/CN=Deleted Items/msExchELCFolderType 16/Jun/2011 15:25:15 - Info bpdbm (pid=20149) cataloguing file number 60000 path /System State/Active Directory/CN=Configuration/CN=Sites/CN=Subnets/CN=194.117.13.0\24/description 16/Jun/2011 15:25:40 - Info bpdbm (pid=20149) cataloguing file number 70000 path /System State/Active Directory/DC=alunos/OU=DBA/OU=LIC/CN=Ana Maria Domingues Vieites/extensionAttribute1 16/Jun/2011 15:26:06 - Info bpdbm (pid=20149) cataloguing file number 80000 path /System State/Active Directory/DC=alunos/OU=DBA/OU=Licenciatura/CN=Ana Sofia Briosa e Scheltinga/givenName 16/Jun/2011 15:26:10 - Info bpdbm (pid=20149) cataloguing file number 90000 path /System State/Active Directory/DC=alunos/OU=DBA/OU=Licenciatura/CN=Diana Mileu Miranda Antunes/mailNickname 16/Jun/2011 15:26:16 - Info bpdbm (pid=20149) cataloguing file number 100000 path /System State/Active Directory/DC=alunos/OU=DBA/OU=Licenciatura/CN=Joana Santos Pinheiro Anastácio/cn 16/Jun/2011 15:28:05 - Info bpdbm (pid=20149) cataloguing file number 200000 path /System State/Active Directory/DC=alunos/OU=DEGGE/OU=Licenciatura/CN=Jorge Miguel Nogueira Ponte/showInAddressBook 16/Jun/2011 15:29:35 - Info bpdbm (pid=20149) cataloguing file number 300000 path /System State/Active Directory/DC=alunos/OU=DG/OU=Licenciatura/CN=Joana Sofia Ferreira Simões/msExchMailboxGuid 16/Jun/2011 15:31:59 - Info bpdbm (pid=20149) cataloguing file number 400000 path /System State/Active Directory/DC=alunos/OU=DI/OU=Licenciatura/CN=Tiago Alexandre da Costa Rodrigues/homeDrive 16/Jun/2011 15:33:49 - Info bpdbm (pid=20149) cataloguing file number 500000 path /System State/Active Directory/DC=alunos/OU=Externos/OU=Erasmus/CN=Giulia Bacchini/proxyAddresses 16/Jun/2011 15:41:54 - Error bpduplicate (pid=20142) db_IMAGE() failed: database system error (220) 16/Jun/2011 15:41:54 - Error bpduplicate (pid=20142) Status = no images were successfully processed. 16/Jun/2011 15:41:54 - Error bpduplicate (pid=20142) Duplicate of backupid alunos-dc01.hostname_1308108619 failed, database system error (220). 16/Jun/2011 15:41:54 - Error bpduplicate (pid=20142) Status = no images were successfully processed. 16/Jun/2011 15:41:54 - end Duplicate; elapsed time 0:22:11 no images were successfully processed (191) Can anyone give me a help here? Thanks. Regards, Ricardo VilhenaSolved9.4KViews0likes15Comments
Backup Exec 2012 backup very slow (catalog problem?)
We've been having some problems with the speed of our backup. The setup is the following: physical BE2012 server coonected to disk storage via fibre and connected to VmWare enviroment via Fibre as well. When backup up our file and print server to DeDupe location on our disk storage it takes a very long time (+19hr) for 900GB of data. It seems to hang on updating catalog. History: Everything working fine and backupjob of all servers (1.4TB) runnning at speeds of between 4000MB/min to 6000MB/min. Then from one day to the other (no windows or BE updates done) the speed slowed down to between 600MB/min and 900MB/min. Things allready tried: delete and recreate the backup job. update to SP1a. Thick all the boxes under the catalog tab and set to trucate after 4 weeks. run database maintenance at a time when no backup jobs are running. split up the job into different jobs (this alleviated the problem a bit) After splitting the job up so that the exchange server, the SQL server and the file and print server had different jobs I noticed that the exchange and SQL were backing up at normal speeds (+4000MB/min) but the file and print server was still taking 18Hrs to backup. a bit more digging and a couple of days later I noticed that the data of file and print was backing up at speeds of +4000MB/min itself but it was hanging (going verrry slow) on updating the catalog at the end of the job while the size stayed unchanged. My counterpart in our other location the other side of Australia is having the same problem for the exact same setup. (his problem started about 2 weeks before mine) He has been testing with going from full daily backups to incremental but the same problem stays. We do not have a bottleneck problem with our disks, speed of our servers or any other hardware but it seems to come backup to the updating of the catalog for the file and print server. (my counterpart has also tried to uninstall and reinstall BE2012 but no sollution there either) Does anyone have an idea how to clean, compact or do anything else with the catalog to get it back to normal speed? (catalog folder is currently 9GB with 16GB available) Thanks in advanceSolved8.8KViews2likes37Comments
Backups were failing with 84 & 87 error intermittently - Destination Storage is Pure Disk Pool
Backups were failing with 84 & 87 error intermittently - Destination Storage is Pure Disk Pool Again the same backup is getting suuccessful in the next successive using different job ID Error Code: 84 *************************************************************************************************************************************** 12/25/2014 00:10:07 - Info bpbrm (pid=13539) INF - Client read timeout = 1800 12/25/2014 00:10:07 - connecting 12/25/2014 00:10:08 - Info bpbrm (pid=13539) accepted connection from client 12/25/2014 00:10:08 - Info dbclient (pid=4951) Backup started 12/25/2014 00:10:08 - Info bpbrm (pid=13539) bptm pid: 13540 12/25/2014 00:10:08 - Info bptm (pid=13540) start 12/25/2014 00:10:08 - Info bptm (pid=13540) using 524288 data buffer size 12/25/2014 00:10:08 - Info bptm (pid=13540) setting receive network buffer to 524288 bytes 12/25/2014 00:10:08 - Info bptm (pid=13540) using 256 data buffers 12/25/2014 00:10:08 - connected; connect time: 0:00:00 12/25/2014 00:10:10 - Info bptm (pid=13540) start backup 12/25/2014 00:12:17 - Info bptm (pid=13540) backup child process is pid 16293 12/25/2014 00:12:17 - begin writing 12/25/2014 00:12:18 - Info dbclient (pid=4951) dbclient(pid=4951) wrote first buffer(size=262144) 12/25/2014 00:13:02 - Critical bptm (pid=13540) Storage Server Error: (Storage server: PureDisk:) mtstrm_write_segment: Fatal error occured in Multi-Threaded Agent: Cr_ErrnoException in void mtstrm::server::Cr_Send::ThreadSend(Pd_Thread<mtstrm::server::Cr_Send>*): CRSegmentBatchStreamPutComp V-454-95 12/25/2014 00:13:02 - Critical bptm (pid=13540) image write failed: error 2060001: one or more invalid arguments 12/25/2014 00:13:04 - Critical bptm (pid=13540) Storage Server Error: (Storage server: PureDisk:) mtstrm_close_write_channel: Fatal error occured in Multi-Threaded Agent: Cr_ErrnoException in void mtstrm::server::Cr_Send::ThreadSend(Pd_Thread<mtstrm::server::Cr_Send>*): CRSegmentBatchStreamPutComp V-454-96 12/25/2014 00:13:04 - Critical bptm (pid=13540) sts_close_handle failed: 2060001 one or more invalid arguments 12/25/2014 00:13:16 - Error bptm (pid=13540) cannot write image to disk, Invalid argument 12/25/2014 00:13:16 - Info bptm (pid=13540) EXITING with status 84 <---------- 12/25/2014 00:13:16 - Info dbclient (pid=4951) done. status: 6 12/25/2014 00:13:16 - Info (pid=13540) StorageServer=PureDisk:; Report=PDDO Stats (multi-threaded stream used) for (): scanned: 49667 KB, CR sent: 0 KB, CR sent over FC: 0 KB, dedup: 100.0%, cache disabled 12/25/2014 00:13:17 - Info dbclient (pid=4951) done. status: 84: media write error 12/25/2014 00:13:17 - end writing; write time: 0:01:00 media write error (84) *************************************************************************************************************************************** Error Code: 87 *************************************************************************************************************************************** 12/25/2014 00:12:23 - Info bpbrm (pid=7499) INF - Client read timeout = 1800 12/25/2014 00:12:23 - Info bpbrm (pid=7499) accepted connection from client 12/25/2014 00:12:24 - Info dbclient (pid=736) Backup started 12/25/2014 00:12:24 - connected; connect time: 0:00:00 12/25/2014 00:12:24 - Info bpbrm (pid=7499) bptm pid: 7500 12/25/2014 00:12:24 - Info bptm (pid=7500) start 12/25/2014 00:12:24 - Info bptm (pid=7500) using 524288 data buffer size 12/25/2014 00:12:24 - Info bptm (pid=7500) using 256 data buffers 12/25/2014 00:12:26 - Info bptm (pid=7500) start backup 12/25/2014 00:17:02 - Info bptm (pid=7500) backup child process is pid 17130 12/25/2014 00:17:02 - begin writing 12/25/2014 00:17:03 - Info dbclient (pid=736) dbclient(pid=736) wrote first buffer(size=65536) 12/25/2014 00:17:04 - Info dbclient (pid=736) done. status: 0 12/25/2014 00:18:56 - Info bptm (pid=7500) waited for full buffer 1 times, delayed 71 times 12/25/2014 00:18:59 - Critical bptm (pid=7500) Storage Server Error: (Storage server: PureDisk:) mtstrm_close_write_channel: Fatal error occured in Multi-Threaded Agent: Close Write Channel command failed: Cr_ErrnoException: dref model has not been initialized yet V-454-96 12/25/2014 00:19:00 - Critical bptm (pid=7500) sts_close_handle failed: 2060017 system call failed 12/25/2014 00:19:00 - Error bptm (pid=7500) cannot write image to disk, media close failed with status 2060017 12/25/2014 00:19:05 - Info bptm (pid=7500) EXITING with status 87 <---------- 12/25/2014 00:19:05 - Info (pid=7500) StorageServer=PureDisk:; Report=PDDO Stats (multi-threaded stream used) for (): scanned: 4835 KB, CR sent: 0 KB, CR sent over FC: 0 KB, dedup: 100.0%, cache disabled 12/25/2014 00:20:13 - Info dbclient (pid=736) done. status: 87: media close error 12/25/2014 00:20:13 - end writing; write time: 0:03:11 media close error (87) *************************************************************************************************************************************** Also attached the error part in the bptm log from media server Client Read & Connect timeout for the respective Media servers - 1800 sec IS it due to any DPS/Timeout polling time out issue or any other issue ?Solved8.7KViews0likes5Comments
BackupExec 2014 - No writable idle devices available
Hello, I'm needing some help with what to me is a strange issue. I have several jobs which were queued but not running even though they should have completed. I found thatthe spoold service was stuck or hung in a starting condition.After searching the forums before I tried anything, I re-started the server and this time the spoold service is no longer hung. All services are running correctly. Now I'm getting a Job Status of Ready; No writable idle devices are available. Also when I go to the storage tab I find that everything is online except Deduplication disk storage 0001 of the Robotic library says it is offline. I am using BE2014 v14.1 rev 1786 64-bit on a Dell PowerEdge R320 Server running Windows 2012 R2 Server Standard (full installation) 64-bit. I appreciate any help that I can get for this issue. I'm not sure what other information anyone may need. I'll be happy to try andsupply any information needed. Thanks in advance for your assistance!Solved7.9KViews0likes10Comments
Duplications fail with media open error(83)
Hello, we're having random issues where a few duplications fail each day out of an easy hundred or so. Here's our current configuration: - NetBackup 7.1.0.3 on master, all media servers, and most clients - NetBackup Disk Appliance 5020, set at production site and set at DR site, all running v1.4.1.1, and 6.6.3.49048 - Master server running on Windows 2008 R2 SP1 64bit. Media servers are Windows 2003 32bit. We are backing up to our local production appliances and duplicating to the DR appliances. Several times a day, duplication jobs fail with the "media open error(83)" status message. It's appears to be the same client, dupe job runs and fails 4 times within a short period, then waits 10 hours and tries and fails again. Here is a copy of the backup status log: 6/12/2012 12:40:22 PM - requesting resource LCM_KMC_Pool_1_All 6/12/2012 12:40:22 PM - granted resource LCM_KMC_Pool_1_All 6/12/2012 12:40:22 PM - started process RUNCMD (8096) 6/12/2012 12:40:22 PM - requesting resource @aaaan 6/12/2012 12:40:22 PM - reserving resource @aaaan 6/12/2012 12:40:22 PM - reserved resource @aaaan 6/12/2012 12:40:22 PM - granted resource MediaID=@aaaan;DiskVolume=PureDiskVolume;DiskPool=KMC_Pool_1;Path=PureDiskVolume;StorageServer=khapbak001.ketthealth.com;MediaServer=khw2k046 6/12/2012 12:40:23 PM - Info bpdm(pid=4356) started 6/12/2012 12:40:23 PM - started process bpdm (4356) 6/12/2012 12:40:48 PM - Critical bpdm(pid=4356) get image properties failed: error 2060013: no more entries 6/12/2012 12:40:48 PM - Error (pid=8096) ReplicationJob::Replicate: Replication failed for backup id khfsvs34_1339468206: media open error (83) 6/12/2012 12:40:48 PMReplicate failed for backup id khfsvs34_1339468206 with status 83 6/12/2012 12:40:48 PM - end operation media open error(83) I've searched the forums and knowledgebase to no avail. Any ideas? Thanks!Solved7.5KViews1like21Comments
Deduplication Folder and "BLANK (WORM)" media
Hi all, I'm facing by monthswith strage problems related to the BackupExec Deduplication Option. I'm doing B2D Jobs to collect data in parallel from Servers to the Deduplication folder. After that Duplicate Jobs start copying data to TAPE for long term retention. It happens that B2D jobs run fine but the related Duplicate job returns an error: "Duplicate V-79-57344-1541 - The requested media was mounted, but it has been erased and does not contain any backup sets. " Taking a look at the job log I understand that there was a problem with the virtual media in the Deduplication folder. The specific media is not marked a DSK, but is marked as "BLANK (WORM)". Trying to take an inventory of the Deduplication folder causes other media to be marked a "BLANK (WORM)". Has anyone seen a similar issue?? Thanks, Marco6.9KViews0likes10Comments
BE 2012 and Data Domain w/ DDBoost
Background: So we have Backup Exec 2012 and two Data Domain DD620's. We have all the required licensing including the Data Domain DD Boost and replication licenses. I have installed the Open Storage Technology (OST) plugin 2.5.0.3 as suggested for the version of BE. Both DD boxes are visable and connected to my BE server. I have foud some documentation and a video that explains some of what I am looking for but not entirely and most of it is for BE2010. The BE 2012 documentation from DD is not up to date yet. I am trying to setup backups as my tape drive has had enough and is not working properly. Anyone I have talked to just says you backup to it, then I continue finding little things that need to be done. No verify, no precompression, some said no deduplication in the job, other said server side dedupe because DDBoost manages it with OST plugin. Too much information and not enough people knowing what to tell me. The DD weekend support guys don't know the software titles they just know the hardware. I need help now unfortunately... Question: What settings do I use for Backups? I have Active Directory, Exchange, SharePoint, and SQL agents.6.4KViews1like32Comments
Cannot run garbage collection anymore. crcollect is missing...
I have a NetBackup 5220 appliance, which was upgraded to version 2.6 several months ago. I was trying to run a manual garbage collection today, following the usual instructions. Problem is, when I get to step 5, and try to run crcollect, is says the executable is missing. This worked for me before. Did someone move the executable? s-17812:/usr/openv/pdde/pdcr/bin # ls -la total 156 drwxr-xr-x 3 root bin 4096 Aug 13 14:01 . drwxr-xr-x 5 root bin 4096 Aug 13 14:01 .. drwxr-xr-x 2 root bin 4096 Aug 13 14:01 .bin -rwxr-xr-x 1 root bin 241 May 30 11:37 adler32 -rwxr-xr-x 1 root bin 241 May 30 11:37 ca_test -rwxr-xr-x 1 root bin 247 May 30 11:37 cacontrol -r-xr-xr-x 1 root bin 2606 May 30 11:37 cdrinit.sh -rwxr-xr-x 1 root bin 247 May 30 11:37 crcontrol -rwxr-xr-x 1 root bin 250 May 30 11:37 crdoreader -rwxr-xr-x 1 root bin 232 May 30 11:37 crfp -rwxr-xr-x 1 root bin 235 May 30 11:37 crget -rwxr-xr-x 1 root bin 238 May 30 11:37 crstat -rwxr-xr-x 1 root bin 241 May 30 11:37 crstats -rwxr-xr-x 1 root bin 244 May 30 11:37 dcpcrypt -rwxr-xr-x 1 root bin 238 May 30 11:37 dcscan -rwxr-xr-x 1 root bin 259 May 30 11:37 dedupecatutil -r-xr-xr-x 1 root bin 4853 May 30 11:37 dr_createlist.sh -rwxr-xr-x 1 root bin 241 May 30 11:37 filedel -rwxr-xr-x 1 root bin 241 May 30 11:37 fileget -rwxr-xr-x 1 root bin 241 May 30 11:37 fileput -rwxr-xr-x 1 root bin 235 May 30 11:37 genpo -rwxr-xr-x 1 root bin 229 May 30 11:37 md5 -rwxr-xr-x 1 root bin 241 May 30 11:37 nstpack -rwxr-xr-x 1 root bin 238 May 30 11:37 pdconf -rwxr-xr-x 1 root bin 238 May 30 11:37 pddeDR -rwxr-xr-x 1 root bin 241 May 30 11:37 pddecfg -rwxr-xr-x 1 root bin 250 May 30 11:37 pdstresscr -rwxr-xr-x 1 root bin 238 May 30 11:37 prdate -rwxr-xr-x 1 root bin 247 May 30 11:37 refdbutil -rwxr-xr-x 1 root bin 481 May 30 11:37 spad -rwxr-xr-x 1 root bin 235 May 30 11:37 spadb -rwxr-xr-x 1 root bin 241 May 30 11:37 spauser -rwxr-xr-x 1 root bin 247 May 30 11:37 splogscan -rwxr-xr-x 1 root bin 489 May 30 11:37 spoold -rwxr-xr-x 1 root bin 232 May 30 11:37 stat -rwxr-xr-x 1 root bin 238 May 30 11:37 stconv -rwxr-xr-x 1 root bin 235 May 30 11:37 wsget -rwxr-xr-x 1 root bin 235 May 30 11:37 wslogSolved5.4KViews0likes4Comments
NIC Teaming Slow Throughput, any ideas?
Hey guys, We currently have 2Gig set up through NIC teaming for our backup exec setup.When we have backups running, it barely reaches 35-40% of the bandwith. I am using Backup Exec 2010. Has anyone else here set up NIC teaming before? What are some things that might cause slow performance? Are there any specific settings in Backup Exec that may help? Thanks!5.1KViews3likes9Comments
Deduplication Folder and "BLANK (WORM)" media
Hi all, I'm facing by monthswith strage problems related to the BackupExec Deduplication Option. I'm doing B2D Jobs to collect data in parallel from Servers to the Deduplication folder. After that Duplicate Jobs start copying data to TAPE for long term retention. It happens that B2D jobs run fine but the related Duplicate job returns an error: "Duplicate V-79-57344-1541 - The requested media was mounted, but it has been erased and does not contain any backup sets. " Taking a look at the job log I understand that there was a problem with the virtual media in the Deduplication folder. The specific media is not marked a DSK, but is marked as "BLANK (WORM)". Trying to take an inventory of the Deduplication folder causes other media to be marked a "BLANK (WORM)". Has anyone seen a similar issue?? Thanks, MarcoBE 2012 and Data Domain w/ DDBoost
Background: So we have Backup Exec 2012 and two Data Domain DD620's. We have all the required licensing including the Data Domain DD Boost and replication licenses. I have installed the Open Storage Technology (OST) plugin 2.5.0.3 as suggested for the version of BE. Both DD boxes are visable and connected to my BE server. I have foud some documentation and a video that explains some of what I am looking for but not entirely and most of it is for BE2010. The BE 2012 documentation from DD is not up to date yet. I am trying to setup backups as my tape drive has had enough and is not working properly. Anyone I have talked to just says you backup to it, then I continue finding little things that need to be done. No verify, no precompression, some said no deduplication in the job, other said server side dedupe because DDBoost manages it with OST plugin. Too much information and not enough people knowing what to tell me. The DD weekend support guys don't know the software titles they just know the hardware. I need help now unfortunately... Question: What settings do I use for Backups? I have Active Directory, Exchange, SharePoint, and SQL agents.NIC Teaming Slow Throughput, any ideas?
Hey guys, We currently have 2Gig set up through NIC teaming for our backup exec setup.When we have backups running, it barely reaches 35-40% of the bandwith. I am using Backup Exec 2010. Has anyone else here set up NIC teaming before? What are some things that might cause slow performance? Are there any specific settings in Backup Exec that may help? Thanks!