Raw backup is in hung status

[Raaavan]

Hi All,

Raw backup of file system is in hung status below are the details & observation that i found but not reached to any specific conclusion please assist

Master server - NBU 7.5.0.4 Solaris 10

SAN client - NBU AIX 7.5.0.4

below raw backup is hang status ( i am saying in hang status since old backup job is killed but still showing in active status so i killed OS process now only single job is in active & hung since not changing its KB )

I suspect below two things - There is low I/O write on the system see Topas output

- df -gt though it is raw backup but respective some logical mount points are showing 100 % full but cant fine any TN or anything which can justify the same

bpbkar log are attached please assist if any one tracked any thing or came across same issue

Policy Name:       DAKC_W_M_CDTCMBKP_RAW

  Policy Type:         Standard
  Active:              yes
  Effective date:      03/18/2009 13:33:06
  Client Compress:     no
  Follow NFS Mounts:   no
  Cross Mount Points:  no
  Collect TIR info:    no
  Block Incremental:   no
  Mult. Data Streams:  yes
  Client Encrypt:      no
  Checkpoint:          yes
       Interval:       15
  Policy Priority:     0
  Max Jobs/Policy:     Unlimited
  Disaster Recovery:   0
  Collect BMR info:    no
  Residence:           cdtcmbkp-hcart-robot-tld-0
  Volume Pool:         DAKC_weekly_pool
  Server Group:        Weekly
  Keyword:             (none specified)
  Data Classification:       -
  Residence is Storage Lifecycle Policy:    no
  Application Discovery:      no
  Discovery Lifetime:      0 seconds
ASC Application and attributes: (none defined)

  Granular Restore Info:  no
  Ignore Client Direct:  no
Enable Metadata Indexing:  no
Index server name:  NULL
  Use Accelerator:  no
  HW/OS/Client:  RS6000        AIX5          CDTCMBKP

  Include:  NEW_STREAM
            /dev/rarchivelognewlv
            /dev/rdb2dump_newlv
            /dev/rindex_newlv
            /dev/rsystem_newlv
            /dev/rdata10lv
            /dev/rtable_newlv
            NEW_STREAM
            /dev/rlbosdata
            /dev/rdata11lv
            /dev/rdb2datalv
            /dev/rdb2data1lv

  Schedule:              M_CDTCMBKP
    Type:                Full Backup
    Frequency:           every 7 days
    Maximum MPX:         3
    Synthetic:           0
    Checksum Change Detection: 0
    PFI Recovery:        0
    Retention Level:     10 (12 years)
    Number Copies:       1
    Fail on Error:       0
    Residence:           cdtcmbkp-hcart2-robot-tld-2
    Volume Pool:         DAKC_monthly_pool
    Server Group:        Monthly
    Residence is Storage Lifecycle Policy:         0
    Schedule indexing:     0
    Daily Windows:

  Schedule:              W_CDTCMBKP
    Type:                Full Backup
    Frequency:           every 7 days
    Maximum MPX:         3
    Synthetic:           0
    Checksum Change Detection: 0
    PFI Recovery:        0
    Retention Level:     3 (1 month)
    Number Copies:       1
    Fail on Error:       0
    Residence:           cdtcmbkp-hcart2-robot-tld-2
    Volume Pool:         DAKC_weekly_pool
    Server Group:        Weekly
    Residence is Storage Lifecycle Policy:         0
    Schedule indexing:     0
    Daily Windows:

[Raaavan]

missed out below screen shot

Filesystem    GB blocks      Used      Free %Used Mounted on
/dev/hd4           1.25      0.56      0.69   45% /
/dev/hd2          17.25      8.53      8.72   50% /usr
/dev/hd9var        1.00      0.21      0.79   21% /var
/dev/hd3           3.00      0.07      2.93    3% /tmp
/dev/hd1           1.00      0.63      0.37   63% /home
/proc                 -         -         -    -  /proc
/dev/hd10opt       5.00      3.66      1.34   74% /opt
/dev/lv00          0.25      0.01      0.24    4% /var/adm/csd
/dev/perflv        5.00      4.00      1.00   80% /perf
/dev/data1lv     136.00    101.92     34.08   75% /data1
/dev/hd11admin      0.25      0.00      0.25    1% /admin
/dev/livedump      0.25      0.00      0.25    1% /var/adm/ras/livedump
/dev/netbackuplv     10.00      6.97      3.03   70% /netbackup
/dev/archivelognewlv     49.00      2.72     46.28    6% /SH/archivelog
/dev/cmdata2lv    502.00    502.00      0.00  100% /SH/data2
/dev/cmdata3lv    502.00    502.00      0.00  100% /SH/data3
/dev/cmdata4lv    502.00    502.00      0.00  100% /SH/data4
/dev/cmdata5lv    502.00    502.00      0.00  100% /SH/data5
/dev/db2dump_newlv     98.00     58.21     39.79   60% /SH/db2dump
/dev/index_newlv     49.00      0.17     48.83    1% /SH/index
/dev/system_newlv     49.00      0.01     48.99    1% /SH/system
/dev/table_newlv    150.00     38.91    111.09   26% /SH/table
/dev/cmdata6lv    502.00    502.00      0.00  100% /SH/data6
/dev/cmdata7lv    502.00    502.00      0.00  100% /SH/data7
/dev/data10lv    502.00    175.52    326.48   35% /SH/data8
/dev/data11lv    502.00    175.49    326.51   35% /SH/data9
/dev/lbosdata    400.00     84.96    315.04   22% /SH/lbosdata
/dev/db2datalv    192.00    180.61     11.39   95% /SH/db2data
/dev/db2data1lv     99.50     78.71     20.79   80% /SH/db2data1
 

[Marianne · Accepted solution]

No errors in bpbkar log. Just warnings at the end of each partition as they finished:

00:30:50.904 [1442082] <8> bpbkar process_file: WRN - Short read at block 1052770181. Read 62976 bytes when attempting to read 65024 bytes, in file /dev/rdata10lv.

Seems backup was still running according to last entry in bpbkar or else a new file for the new day was created.
bptm log on the media server will tell us at what speed data was received from the client and until what time.

Is this the first time that a raw backup for this client is done?
Or did previous backups complete sooner?

Your last post is showing a list of mounted filesystems.
Never a good idea to perform raw backup of active filesystem.
See this section in NBU Admin Guide I:

Use raw partition backups only if you can ensure that the files

have not changed in any way during the backup. Or, in the case

of a database, if you can restore the database to a consistent state

by using transaction log files.

Before backing up file systems as raw partitions, unmount the file

system. Unmounting the file system allows buffered changes to

be written to the disk. Also, it prevents the possibility of any

changes in the file system during the backup. Use the

bpstart_notify and the bpend_notify scripts to unmount

and remount the backed-up file systems.