08-17-2014 06:02 AM
Hello Friends ,
Some of the weekly RMAN backup which is configured as SAN client failling with status code 83 below is one of the failed job details .
8/15/2014 10:16:10 AM - Info nbjm(pid=8568) starting backup job (jobid=634117) for client q001_vip, policy BV RMAN_PRD_Weekly, schedule Weekly-Application-Backup
8/15/2014 10:16:10 AM - Info nbjm(pid=8568) requesting STANDARD_RESOURCE resources from RB for backup job (jobid=634117, request id:{B11FBC96-EDCB-47F5-9066-743BB21F9270})
8/15/2014 10:16:10 AM - requesting resource FT1-FT2-HO-STU
8/15/2014 10:16:10 AM - requesting resource mst.NBU_CLIENT.MAXJOBS.q0_vi
8/15/2014 10:16:10 AM - requesting resource mst.NBU_POLICY.MAXJOBS.BV-_RMAN_QBRN_PRD_Weekly
8/15/2014 10:16:11 AM - granted resource mst.NBU_CLIENT.MAXJOBS.q0_vi
8/15/2014 10:16:11 AM - granted resource mst.NBU_POLICY.MAXJOBS.BV-_RMAN_QBRN_PRD_Weekly
8/15/2014 10:16:11 AM - granted resource MediaID=@aaaa5;DiskVolume=DD890_HO_LSU;DiskPool=OST_DD_HO_DISK_POOL;Path=DD890_HO_LSU;StorageServer=ddho;MediaServer=ftho2
8/15/2014 10:16:11 AM - granted resource ftho2_ho_stu1
8/15/2014 10:16:11 AM - granted resource TRANSPORT
8/15/2014 10:16:14 AM - estimated 0 Kbytes needed
8/15/2014 10:16:14 AM - Info nbjm(pid=8568) started backup (backupid=q0_vi_1408083373) job for client q0_vi, policy BV--_RMAN_QBRN_PRD_Weekly, schedule Weekly-Application-Backup on storage unit ftho2_ho_stu1
8/15/2014 10:16:15 AM - started process bpbrm (31159)
8/15/2014 10:16:16 AM - connecting
8/15/2014 10:16:18 AM - connected; connect time: 00:00:02
8/15/2014 10:16:42 AM - end writing
8/15/2014 10:18:50 AM - Info bpbrm(pid=31159) q0_vi is the host to backup data from
8/15/2014 10:18:50 AM - Info bpbrm(pid=31159) reading file list from client
8/15/2014 10:18:51 AM - Info bpbrm(pid=31159) listening for client connection
8/15/2014 10:18:52 AM - Info bpbrm(pid=31159) INF - Client read timeout = 1200
8/15/2014 10:18:53 AM - Info bpbrm(pid=31159) accepted connection from client
8/15/2014 10:18:53 AM - Info bphdb(pid=12507) Backup started
8/15/2014 10:18:53 AM - Info bpbrm(pid=31159) bptm pid: 31186
8/15/2014 10:18:53 AM - Info bptm(pid=31186) start
8/15/2014 10:18:53 AM - Info bptm(pid=31186) using 262144 data buffer size
8/15/2014 10:18:53 AM - Info bptm(pid=31186) using 32 data buffers
8/15/2014 10:18:53 AM - Info bptm(pid=31186) USING 262144 data buffer size for FT
8/15/2014 10:19:13 AM - Error bptm(pid=31186) Could not open FT Server pipe: pipe open failed (589841)
8/15/2014 10:19:13 AM - Info bptm(pid=31186) EXITING with status 83 <----------
8/15/2014 10:19:16 AM - Info bphdb(pid=12507) done. status: 83: media open error
media open error(83)
Kindly suggest .
Thanks You !!!
Solved! Go to Solution.
09-13-2014 11:43 PM
Hi Friends,
I got below error from nbftsrvr log on FT media server .
09/02/2014 22:13:07.206 [Debug] NB 51216 nbftsrvr 199 PID:2564 TID:139838317016832 File ID:199 [No context] 1 [SOS_MemAllocPool] SC PAL DMA-able Scatter/Gather Region Vector Pool exhausted all 161 entries used.
If the FT media server is Linux, the number of FT pipes * number of data buffers should not exceed 161 for NetBackup 6.5.5 and prior, and should not exceed 322 for 6.5.6 and above. but i am using NBU 7.5.0.5 however it's exhausting on 161 , Cause
it would show up the value as 161 but not 322 as there are two SGRVpool buffers
because Linux doesn't have room to allocate 322 SGRV buffers contiguously from the kernel non-paged pool. 161 are allocated for odd numbered pipes in SGRVpool[0] and another 161 are allocated for even numbered pipes in SGRVpool[1].
NUMBER_DATA_BUFFERS_FT can be used for overriding NUMBER_DATA_BUFFERS for SAN Client jobs. For a Fibre Transport media server, the recommendation is not to exceed 16.
So, it is anyways recommended to not use the value more than 16 which is 32 in our case
http://www.symantec.com/business/support/index?page=content&id=TECH62007
I reduce the No of data buffer from 32 to 16 now it is working fine .
Dear Riaan,
You were near around in your first post, thanks for your all help .
08-17-2014 06:56 AM
Does it ever working fine? or its a new configuraiton?
please have a look at below T/N and see if that helps
http://www.symantec.com/business/support/index?page=content&id=TECH206660
08-17-2014 07:06 AM
Yes, it was working earlier even in second attempt it completed successfully.
i face this issue on the day of weekly backup only some random client fails with same error .
I have doubt that some thing need to be adjusted in tuning parameters of FT media server
currently buffer size is 262144 and no of buffers 32 .
any advice ????
Thank You !!
08-17-2014 08:44 AM
08-17-2014 09:13 AM
no buddy it is not appliance, for how many no of jobs runing at the same time i will update you tomorrow .
and backup is happning on disk not on tape .
Thank You!!
08-17-2014 11:02 PM
Hi ,
During which time how many backup was runing you want me to check ???
like these all are RMAN backup and as it comes in multile streems .
so you want me to check how many job was runing when that particuler streem failed or how many job was rining when parent job failed .?/
please tell me .
Thaks You .
08-18-2014 12:15 AM
I meant, if you kick off your job(s) that write to the FT media server. There should not be more than 32 active at any one time.
08-18-2014 12:31 AM
Hi,
No need of tuninng FT Pipes 32 is the max limit , for the I/O it accepts.
Backups to Disk may be logs written in full try to clean it up.
If it is netbackup 7.1.x environment where dedupe configured as MSDP try below commands and link
http://www.symantec.com/business/support/index?page=content&id=TECH172281
http://www.symantec.com/business/support/index?page=content&id=TECH180659
try the above links.
09-13-2014 11:43 PM
Hi Friends,
I got below error from nbftsrvr log on FT media server .
09/02/2014 22:13:07.206 [Debug] NB 51216 nbftsrvr 199 PID:2564 TID:139838317016832 File ID:199 [No context] 1 [SOS_MemAllocPool] SC PAL DMA-able Scatter/Gather Region Vector Pool exhausted all 161 entries used.
If the FT media server is Linux, the number of FT pipes * number of data buffers should not exceed 161 for NetBackup 6.5.5 and prior, and should not exceed 322 for 6.5.6 and above. but i am using NBU 7.5.0.5 however it's exhausting on 161 , Cause
it would show up the value as 161 but not 322 as there are two SGRVpool buffers
because Linux doesn't have room to allocate 322 SGRV buffers contiguously from the kernel non-paged pool. 161 are allocated for odd numbered pipes in SGRVpool[0] and another 161 are allocated for even numbered pipes in SGRVpool[1].
NUMBER_DATA_BUFFERS_FT can be used for overriding NUMBER_DATA_BUFFERS for SAN Client jobs. For a Fibre Transport media server, the recommendation is not to exceed 16.
So, it is anyways recommended to not use the value more than 16 which is 32 in our case
http://www.symantec.com/business/support/index?page=content&id=TECH62007
I reduce the No of data buffer from 32 to 16 now it is working fine .
Dear Riaan,
You were near around in your first post, thanks for your all help .
09-15-2014 01:29 AM
It seems that Riaan pointed you in the right direction, right?
Please be fair and clear the solution for your pown post and mark Riaan's post as solution.
Thanks!
09-15-2014 02:00 AM
Hi Marianne,
Riaan was near around but in his second post he mentioned it should not be more than 32 and it was 32 in my setup.
however backup was failling , then i loged a case with symantec and i got this solution and details , so i shared that how this issue solved .
it is not about i marked my post as solution , and i should mark to Riaan. the answer what i marked it is the details how issue been resolved and got the cause after involving symantec.
Like, If the FT media server is Linux, the number of FT pipes * number of data buffers should not exceed 161 for NetBackup 6.5.5 and prior, and should not exceed 322 for 6.5.6 and above. but i am using NBU 7.5.0.5 however it's exhausting on 161 ,
in above line it says no data buffers should not exceed 161 for 6.5.5 and prior and should not exceed 322 for 6.5.6 and above , we are using 7.5.0.5 so it should have to support by 322 , but in log we got error exhausting on 161 .
this all been helped by Symantec TSE so shared that how issue solved and what was the cause .
if still you think i should mark solution to Riaan, i don't have any issue . but complete information is available in the post what i shared solution provided by symantec .
Thanks You !!
09-15-2014 02:02 AM
i can share symantec case ID and mail if you want .