High CPU utilization due to many BPBKAR process
Hi All, I havea query, Today found a server in which CPU utilization was high upto 95%. When checked on the server, found many BPBKAR process were running there. That was obvious because backup for that server running, Is there a way we can minimize CPU utilization while "BPBKAR" process and backup keep running ? Means without affecting backup, can we make CPU utilization normal. Please advice and suggest...Solved16KViews1like6Commentsall devices stuck at "discovering devices
I have an issue with my media servers where all devices get stuck at "discovering devices" and service restart or reboot of the server. We are running BE2010R3 on windows server 2008r2 platform. I know back in the day there was a regedit for diodrivers as a temp fix, but need to get this resolved permanently ASAP.Solved16KViews1like5CommentsJobs stuck indefinitely in Queued Status
We have had an ongoing issue for about 2 months now, and since we have had a clean build (3 times now) for Backup Exec 2012. We have opened numerous cases with Symantec to resolve this, and they claim the first time that HotFix 209149 (see http://www.symantec.com/business/support/index?page=content&id=TECH204116 ) corrects the issue. The issue is also noted by seeing Robotic Element errors stating that OST media is full and it brings the "virtual" slot for the PureDisk or OST storage device offline. Restarting the services in BE only makes the problem worse and causes a snowball effect whereby jobs constantly error in the ADAMM log files. Essentially, the jobs never can get a concurrency/virtual slot and they stay Queued forever. I have seen others on this Forum with this problem, and while the Forum administrator seems to mark them as "Solved", they are not - because I see the threads drop off with no resolution identified. Are other people having this problem? If so, how are you overcoming it? Once it starts the environment is essentially dead in the water because the jobs never start (they sit Queued forever) - save for one concurrency which for our size environment is only 1/4 the need we have. We use CAS and 4 MMS servers with 2008 R2 with all patches applied, PureDisk 32TB volume on each MMS, Data Domain OST connection to DD-670 OST with OST-Plug 2.62 (28TB), replicated catalogs, and duplicate jobs for optimized deduplication between MMSs. We run BE 2012 SP3 clean - we reinstalled with SP3 slipstreamed because Symantec said this problem could be fixed through database repair by them manually or by reinstall...we chose reinstall (even though they did offer to "fix" the issue with database repair). We chose reinstall to validate whether SP3 truly fixes the issue. It is clear to us it does not. We are looking for anyone else who has had this problem to report into this forum. Thank you, Dana14KViews4likes15CommentsJobs in queued state: Waiting in NetBackup scheduler work queue on server :
Hello Everybody, We are seeing this all the jobs. I have checked other threads related to same problem but none of them helped. Nothing is moving in the environment, almost every backup is failing with 196 after staying in queued state. We opened a severity one case with support they are asking to increase the memory, we have ordered the new memory. But I am seeing no sighns that server is running out of memory or something like that. Anytime there is nearly 3 GB available of physical memory. Below is the output from one job. 8/22/2014 9:32:39 AM - Info nbjm(pid=12846) Waiting in NetBackup scheduler work queue on server 8/22/2014 9:34:41 AM - Info nbjm(pid=12846) Waiting in NetBackup scheduler work queue on server 8/22/2014 9:36:41 AM - Info nbjm(pid=12846) Waiting in NetBackup scheduler work queue on server 8/22/2014 9:38:41 AM - Info nbjm(pid=12846) Waiting in NetBackup scheduler work queue on server 8/22/2014 9:40:41 AM - Info nbjm(pid=12846) Waiting in NetBackup scheduler work queue on server 8/22/2014 9:42:41 AM - Info nbjm(pid=12846) Waiting in NetBackup scheduler work queue on server 8/22/2014 9:44:41 AM - Info nbjm(pid=12846) Waiting in NetBackup scheduler work queue on server 8/22/2014 9:46:41 AM - Info nbjm(pid=12846) Waiting in NetBackup scheduler work queue on server Secondly I tried this command as given in another thread just to check the scheduler queue but no use. nbpemreq -jobs screen all Maximum output size exceeded, see log We have already tried restarting the NBU, rebooting the master server but this things creeeps up every time. Everything is so slow now. If we triggering a job using bpbackup it takes good 20 minutes to show up in activity monitor. Tapes drives are free but still NBU is not able to assign them for the waiting job and those jobs are just waiting for the drives to continue. I am not sure which logs to provide. Let me know which logs and what other info is needed to look into this. Environment info: NBU 7.5.0.5 running on Solaris 10 20+ media server sharing same DD and Tape library SLP Duplication is used12KViews1like23CommentsMIssing PATH: for drives
I ahve a LINUX Media server, and the drive path when i run a tpconfig shows missing. How do I get NB to see the paths once again? Enter option: Id DriveName Type Residence Drive Path Status **************************************************************************** 1 Sep1-71 hcart2 TLD(3) DRIVE=71 MISSING_PATH:4:0:0:2:SG11029122 DOWN 2 Sep1-70 hcart2 TLD(3) DRIVE=70 MISSING_PATH:4:0:0:1:SG11029121 DOWN 3 Sep1-84 hcart2 TLD(3) DRIVE=84 MISSING_PATH:2:0:0:3:SG11029135 UP 4 Sep1-83 hcart2 TLD(3) DRIVE=83 MISSING_PATH:2:0:0:2:SG11029134 UP 5 Sep1-82 hcart2 TLD(3) DRIVE=82 MISSING_PATH:2:0:0:1:SG11029133 DOWN 6 Sep1-69 hcart2 TLD(3) DRIVE=69 MISSING_PATH:4:0:0:0:SG11029120 DOWN 7 Sep1-81 hcart2 TLD(3) DRIVE=81 MISSING_PATH:2:0:0:0:SG11029132 DOWN Currently defined robotics are: TLD(3) robot control host = nbuf1a EMM Server = nbuf1aSolved12KViews1like6CommentsEvent 1023 Windows cannot load the extensible counter DLL Backup Exec.
I am getting the following events from our newly build server 2012 r2 server with BE 15. I am not sure what is causing this issue, but it appears to be related to BE and performance counters. Log Name: Application Source: Microsoft-Windows-Perflib Date: 4/24/2015 11:02:17 AM Event ID: 1008 Task Category: None Level: Error Keywords: Classic User: N/A Computer: XXXXXXX Description: The Open Procedure for service "BITS" in DLL "C:\Windows\System32\bitsperf.dll" failed. Performance data for this service will not be available. The first four bytes (DWORD) of the Data section contains the error code. Event Xml: <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> <System> <Provider Name="Microsoft-Windows-Perflib" Guid="{13B197BD-7CEE-4B4E-8DD0-59314CE374CE}" EventSourceName="Perflib" /> <EventID Qualifiers="49152">1008</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2015-04-24T18:02:17.000000000Z" /> <EventRecordID>3405</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>Application</Channel> <Computer>XXXXXXX</Computer> <Security /> </System> <UserData> <EventXML xmlns="Perflib"> <param1>BITS</param1> <param2>C:\Windows\System32\bitsperf.dll</param2> <binaryDataSize>4</binaryDataSize> <binaryData>02000000</binaryData> </EventXML> </UserData> </Event> Log Name: Application Source: Microsoft-Windows-Perflib Date: 4/24/2015 10:53:43 AM Event ID: 1023 Task Category: None Level: Error Keywords: Classic User: N/A Computer: XXXXX Description: Windows cannot load the extensible counter DLL Backup Exec. The first four bytes (DWORD) of the Data section contains the Windows error code. Event Xml: <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event"> <System> <Provider Name="Microsoft-Windows-Perflib" Guid="{13B197BD-7CEE-4B4E-8DD0-59314CE374CE}" EventSourceName="Perflib" /> <EventID Qualifiers="49152">1023</EventID> <Version>0</Version> <Level>2</Level> <Task>0</Task> <Opcode>0</Opcode> <Keywords>0x80000000000000</Keywords> <TimeCreated SystemTime="2015-04-24T17:53:43.000000000Z" /> <EventRecordID>3403</EventRecordID> <Correlation /> <Execution ProcessID="0" ThreadID="0" /> <Channel>Application</Channel> <Computer>XXXXXX</Computer> <Security /> </System> <UserData> <EventXML xmlns="Perflib"> <param1>Backup Exec</param1> <binaryDataSize>4</binaryDataSize> <binaryData>7E000000</binaryData> </EventXML> </UserData> </Event>12KViews0likes4CommentsVirtual Machines going out of network during snapshot backup
Has anyone experienced the Virtual machines going out of network during the backup using "VMware" policy. Apparently , the Windows VMs losing connectivity for a little while and coming back online and the Linux VMs becoming unresponsive.Solved12KViews1like7Commentsbpbkar waited xxxx times for empty buffer, delayed xxxx times
Hello, I am getting detailed report as below; and the network KB per second is down 3762 from original 42399. Any idea? 03/09/2013 03:00:00 - granted resource DAILY1 03/09/2013 03:00:00 - granted resource HP.ULTRIUM4-SCSI.000 03/09/2013 03:00:00 - granted resource ajva5000-hcart-robot-tld-0 03/09/2013 03:00:00 - estimated 494449958 kbytes needed 03/09/2013 03:00:00 - Info nbjm (pid=960) started backup job for client ajva5000, policy m5000, schedule Daily-Differential-Inc on storage unit ajva5000-hcart-robot-tld-0 03/09/2013 03:00:01 - Info bpbrm (pid=27395) ajva5000 is the host to backup data from 03/09/2013 03:00:01 - Info bpbrm (pid=27395) reading file list from client 03/09/2013 03:00:01 - Info bpbrm (pid=27395) starting bpbkar on client 03/09/2013 03:00:01 - Info bpbkar (pid=27399) Backup started 03/09/2013 03:00:01 - Info bpbrm (pid=27395) bptm pid: 27400 03/09/2013 03:00:01 - Info bptm (pid=27400) start 03/09/2013 03:00:01 - started process bpbrm (pid=27395) 03/09/2013 03:00:01 - connecting 03/09/2013 03:00:01 - connected; connect time: 0:00:00 03/09/2013 03:00:02 - Info bptm (pid=27400) using 65536 data buffer size 03/09/2013 03:00:02 - Info bptm (pid=27400) using 30 data buffers 03/09/2013 03:00:02 - Info bptm (pid=27400) start backup 03/09/2013 03:00:02 - Info bptm (pid=27400) Waiting for mount of media id DAILY1 (copy 1) on server ajva5000. 03/09/2013 03:00:02 - mounting DAILY1 03/09/2013 03:00:56 - Info bptm (pid=27400) media id DAILY1 mounted on drive index 0, drivepath /dev/rmt/0cbn, drivename HP.ULTRIUM4-SCSI.000, copy 1 03/09/2013 03:00:56 - mounted DAILY1; mount time: 0:00:54 03/09/2013 03:00:57 - positioning DAILY1 to file 25 03/09/2013 03:02:39 - positioned DAILY1; position time: 0:01:42 03/09/2013 03:02:39 - begin writing 03/09/2013 12:13:40 - Info bpbkar (pid=27399) bpbkar waited 1219344 times for empty buffer, delayed 1219737 times 03/09/2013 12:13:40 - Info bptm (pid=27400) waited for full buffer 447 times, delayed 16121 times 03/09/2013 12:13:49 - Info bptm (pid=27400) EXITING with status 0 <---------- 03/09/2013 12:13:49 - Info bpbrm (pid=27395) validating image for client ajva5000 03/09/2013 12:13:50 - Info bpbkar (pid=27399) done. status: 0: the requested operation was successfully completed 03/09/2013 12:13:50 - end writing; write time: 9:11:11 the requested operation was successfully completed (0) Prior to this; I got an error message in /var/adm/messages Mar 8 23:01:15 ajva5000 scsi: [ID 107833 kern.warning] WARNING: /pci@2,600000/pci@0/scsi@8 (mpt1): Mar 8 23:01:15 ajva5000 Connected command timeout for Target 4. Mar 8 23:01:15 ajva5000 scsi: [ID 107833 kern.warning] WARNING: /pci@2,600000/pci@0/scsi@8 (mpt1): Mar 8 23:01:15 ajva5000 Target 4 reverting to async. mode Mar 8 23:01:15 ajva5000 mpt: [ID 675377 kern.warning] WARNING: ID[SUNWpd.mpt.sync_wide_backoff.6013]Solved11KViews1like11CommentsBest practice “Maximum I/O streams per volume” with Disk Pools
My backups are failed with below error - awaiting resource CSD-SYD-STU-PD Reason: Maximum I/O stream count has been reached for disk volume, Media Server: ABC1257, Robot Number: NONE, Robot Type: NONE, Media ID: N/A, Drive Name: N/A, Volume Pool: CSD-SYD, Storage Unit: CSD-SYD-STU-PD, Drive Scan Host: N/A Limit has been reached for the logical resource csd-nbumaster.NBU_POLICY.MAXJOBS.CSD-SYD-CA client backup was not attempted because backup window closed(196) Environment: Netbackup 7.6.0.2& Windows Server 2008. Current Setting : Limit I/O streams = 32 per volume Maximum concurrent jobs=32 Maximum fragment size=51200 MBSolved10KViews1like5CommentsVery Slow VM Backups
Hi, We are currently using Network based VM backups rather than SAN based, the Backup Host is a Windows 2008 server running NBU7.5.0.5 and has full access to the ESX servers, I have several policies set up including VIP's for covering the VM's. The back bone is a 1Gb LAN, the Backup Host is SAN attached to an HP VLS12000 which is emulating LTO4 drives, this is on 4Gb F/C When backing up the VM's I am only seeing on average 1.5M/s throughput, we have standard clients that are using the conventional NBU method whuch reach over 40M/s through the same medium. Just wondered if there were any tuning or other paramaters that we need to look at with the VM backups to speed them up. KevSolved9.7KViews1like22Comments