backup job failing

[syedzeeshan]

dear all ,

i m facing backup failuer netbakcup generic errors in the backup window

however if i remove the backup job to another time it works fine

secenrio is

backup server connect 10 GB

with switch 

clients are connected through switch 1gb each

4gb link with san swich

and tape lib are connected to 8bg with san swtich

the total backup utlization is 2.gb at max

8 gb still remaining for backup server 

any suggestions really appreciated

[Mark_Solutions]

Need a lot more detail please - see this post:

https://www-secure.symantec.com/connect/forums/suggested-required-information-when-starting-posts

[mph999]

When logging a new post, please be sure to include the following information :

1.   The exact error message (if an error is given)
2.   During which type of operation does the problem occur ?  For example, Backup, Duplication, Vault, Restore etc.
3.   At what point in the job does the issue occur, for example, right at the beginning.
4.   Which system(s) are involved? For example, Master Server, Media Server , what are operating systems and patch levels of the operating system.
5.   Please send in the contents of the details tab in the Activity Monitor for the job.
6   Has this worked previously?  If so, when was the last time it DID work?
7.  Have there been any environmental; system; or configuration changes?
8.  When did the problem begin?
9.  How often does the problem occur?
10. Is a similar configuration working properly?

11. What version of NetBackup are you running on the servers

12.  If applicable, are you running the job to/from disk or tape, if tape, is it a VTL.

Please also explain what troubleshooting steps you have undertaken so far.

[Marianne]

"i m facing backup failuer netbakcup generic errors in the backup window"

What do you consider to be 'generic errors'?

There is nothing quick and easy about errors - each status code is different and all of them have different troubleshooting steps. You can either use the Troubleshooting Guide or the Troubleshooter button in Activity Monitor. NBU 7.1 has a separate 'Status Code' manual.

[MN_Pankaj]

Since you mention that "the backup job to another time it works fine" - continue to take backup at the time when it works. If we have to investigate why the backup fails at other times, provide output of the job details from the activity monitor at the very least.

[syedzeeshan]

The exact error message (if an error is given)

error 2,6

2.   During which type of operation does the problem occur ?  For example, Backup, Duplication, Vault, Restore etc.

taking backup

3.   At what point in the job does the issue occur, for example, right at the beginning.

after 30-50 mins

sometime the backup jobs doesnt even fire

4.   Which system(s) are involved? For example, Master Server, Media Server , what are operating systems and patch levels of the operating system.

master/media server

and media server

6   Has this worked previously?  If so, when was the last time it DID work?

yes it work well

7.  Have there been any environmental; system; or configuration changes?

no
8.  When did the problem begin?

1month ago
9.  How often does the problem occur?

everyday when backup job run at the night with all other backups

10. Is a similar configuration working properly?

11. What version of NetBackup are you running on the servers

6.5

12.  If applicable, are you running the job to/from disk or tape, if tape, is it a VTL.

tape lt0-4

Please also explain what troubleshooting steps you have undertaken so far 

i have changed the backup time windows into morning since then backup jobs are completed successfully

my point of discussion here is to identify the problem

while in the nigth backups jobs get fired .......few critical sql,oracle backups jobs are not completing

some time give error 2,6, usually 1 but i go back to restore there is no backup

after changing the time into morning same backup job completed with status 1/0 ,however i can restore

the probelm lies in the backup window of night ....where all the backup jobs fired

what is alarming is i have a STRONG BACKUP INFRASTRUCUTRE

10GB- master and switch

1gb-with clients

4gb-link san switch

tapelibraries conntect to--8gb fiber link

how come backups jobs are getting choked

[Marianne]

Please copy the text in Activity Monitor job details for one of the failed jobs and post here.

Status 2 is normally seen when SQL backup fails and status 6 for other database agent failed backup such as Oracle.

There is no 'generic' or 'quick fix' for these errors. Each failure needs to be looked at separately.

The first thing to do is to download all relevant manuals for your environment.

Full 6.5 list: http://www.symantec.com/docs/TECH52878

For troubleshooting, you will need the Troubleshooting manual as well as the manual for each database agent in your environment.
Go to the troubleshooting section of the manual of the agent that is failing and create all the log directories that are listed in this section.
After the next failure, check these logs. For tips on how to read logs, see this post: https://www-secure.symantec.com/connect/forums/how-use-logs#comment-6505511

If the reason for the failure is not clear after looking at the logs, start a new forum discussion for each failure and supply all relevant info. It will be good if you also add relevant log(s) containing errors as attachment(s).

Suggestion:
if the problem seems to be with the timing of the jobs, have a careful look at Activity Monitor to see what time most backups run and try to spread backup start times more evenly. It sometimes require to sit and look at Activity Monitor through the night to get a better understanding of what is actually happening - how tape drives are used, how many jobs are active, how many are queued (and why), if throughput is sufficient and if multiplexing and multistreaming can be used to better utilize resources.

[syedzeeshan]

hi marianne ,

thanks for your suggestions i really appreciate it

my question stands still .........what is the reason after having

16 tape drives and fiber optic link still the backup window get choaked at the night .

i have increase the buffer size and other performance enchancing tasks

he probelm lies in the backup window of night ....where all the backup jobs fired

what is alarming is i have a STRONG BACKUP INFRASTRUCUTRE

10GB- master and switch

1gb-with clients

4gb-link san switch

tapelibraries conntect to--8gb fiber link

how come backups jobs are getting choked

[Marianne]

Hopefully the 16 tape drives are spread across multiple media servers?

We have no insight into your NBU config as in STU config, amount of pools, amount of different retention levels, etc... etc...

As I've said in my previous post - the easiest way to understand what is happening in the middle of the night, is to be THERE.... To have Activity Monitor open, to see how many jobs are activated, how many are queued and why and to check effect on system resources (memory, cpu, network, etc).

1Gb network connection to accept all data from clients will be a major limiting factor.

[syedzeeshan]

hi maria ,

16 drives are shared ,i have shared storage units groups

weekly/daily job are around 50,the queued jobs are half 50% always

1gb connection client connected to switch

backup server is 10 GB  ........the overall utlization is 1.2 maximum

i was thinking about the option of  TCP Chimney  but this will  but below given link 

http://www.symantec.com/business/support/index?page=content&id=TECH60844 

please suggest

[Kernel_Panic · Accepted solution]

Open a call with Symantec, don´t care if you have the biggest network ever, they will suggest you to use SAS appcritical tool while your backup is suppoused to run. If the results of the appcritial are in red, you´ll have to check it anyway.

[syedzeeshan]

thanks alot guys for your help and support .