Getting Media Write Error (84) attempting to write to DataDomain

[MelB]

I am using Netbackup 7.1.0.3 and was successful backing snapshots to the DataDomain (DD) until recently.  The space was at 100 percent and would not clear any space over a couple of weeks (was getting 84 errors then.  I then began to expire some data in order to get more space and now have cleared more than 2/3 of the contents but still cannot write to the DD.

[revarooo]

Question: How did you 'expire some data' ?

[MelB]

Under NetBackup Management in NetBackup, I filtered for all of the backups going to the DataDomain. I then selectively expired some backups on policies.

[MelB]

Here is the output I get from my backup attempt: 1/8/2013 1:52:13 PM - Info nbjm(pid=3504) starting backup job (jobid=7759) for client ,, policy , schedule Weekly_Full 1/8/2013 1:52:14 PM - estimated 0 Kbytes needed 1/8/2013 1:52:14 PM - Info nbjm(pid=3504) started backup job for client , policy , schedule Weekly_Full on storage unit , 1/8/2013 1:52:16 PM - started process bpbrm (4140) 1/8/2013 1:52:22 PM - Info bpbrm(pid=4140) , is the host to backup data from 1/8/2013 1:52:22 PM - Info bpbrm(pid=4140) reading file list from client 1/8/2013 1:52:23 PM - connecting 1/8/2013 1:52:26 PM - Info bpbrm(pid=4140) starting bpbkar32 on client 1/8/2013 1:52:26 PM - connected; connect time: 00:00:03 1/8/2013 1:52:28 PM - Info bpbkar32(pid=5600) Backup started 1/8/2013 1:52:28 PM - Info bptm(pid=3056) start 1/8/2013 1:52:28 PM - Info bptm(pid=3056) using 262144 data buffer size 1/8/2013 1:52:28 PM - Info bptm(pid=3056) setting receive network buffer to 1049600 bytes 1/8/2013 1:52:28 PM - Info bptm(pid=3056) using 30 data buffers 1/8/2013 1:52:32 PM - Info bptm(pid=3056) start backup 1/8/2013 1:52:33 PM - Critical bptm(pid=3056) image open failed: error 2060046: plugin error 1/8/2013 1:52:40 PM - end writing media write error(84)

[Nicolai]

You need to dig deeper. The plugin error pretty much a generic faliure.

How do you write to the Data Doman via BOOST ?

[RamNagalla]

once you expire the images from the Netbackup, Data domain requies to have the cleaning run to get the space.

did the cleaning run in the data domain after you expire the images?

what is the avaliable space in DD Now?

Show us the output of below command in DD

# filesys show space

Generally cleaning runs in data domin at the schedule time, if you need the immediate space to be avaliable you need to trigger the cleaning manually in DD, once you expire the images in Netbackup.

els you need to wait untill DD get cleaned at the regular schedule time.

[Yasuhisa_Ishika]

Run "filesys clean start" on DD to start cleanup process if you want to cleanup expired data on DD immediately.

[Nicolai]

Or use the web "Enterprise Manger" gui. 

[wellssh]

I am getting the same error.

Our environment is as follows:

W2K3 SP2 Ent. x64 Master.....Media server is the same. NBU v8.5.0.4

Data Domain is a DD890, which had it's software updated mid-year last year to 5.1.1.0-291218.  OST plugin is version 2.4.1.0.

Have a policy configured for W2K8 servers.  There are 8 clients in this policy with ALL_LOCAL_DRIVES configured to be backed up.  These clients are still running NBU 7.0.1.

NBU Master and Media server were upgraded to NBU 7.5.0.4 last month, Jan 2013.

Shortly after the upgrade to 7.5.0.4, the backups for one particular client started failing with Status 84.

The BPTM log file excerpt is as follows:

"06:37:15.658 [12084.10176] <16> 703249:bptm:12084:wilbs001: D:\Veritas\NetBackup\bin\\ost-plugins\libstspiDataDomain.dll:stspi_write_image STS_EPLUGIN Bad file descriptor
06:37:15.658 [12084.10176] <32> do_write_image: sts_write_image failed: byteswritten = 0, len = 131072
06:37:15.658 [12084.10176] <32> do_write_image: sts_write_image failed: error 2060046 (1811939328 131072 0)
06:37:15.704 [12084.10176] <32> write_data: image write failed: error 2060046:
06:37:15.704 [12084.10176] <8> 703249:bptm:12084:wilbs001: [2F34:27C0] ddcl_ddcp_commit(): Commit forcing close of ddcp file wilddbs001-wilbs001-lsu01/vwilsqldb214_1360202425_C1_F1:1360202425:W2K8:4:0:: on hostname wilddbs001-local; pid=12084
06:37:15.704 [12084.10176] <16> 703249:bptm:12084:wilbs001: D:\Veritas\NetBackup\bin\\ost-plugins\libstspiDataDomain.dll:stspi_get_image_prop_v10 STS_EPLUGIN Bad file descriptor
06:37:15.704 [12084.10176] <32> bp_sts_close_image: sts_get_image_prop failed: error 2060046:
06:37:15.704 [12084.10176] <8> delete_image_disk: image close failed: error 2060022: software error
06:37:15.704 [12084.10176] <2> delete_image_disk_sts_impl: file vwilsqldb214_1360202425_C1_F1 does not exist
06:37:15.704 [12084.10176] <2> delete_image_disk_sts: Deleting disk header for ÿÿÿÿÿÿÿÿ

06:37:15.720 [12084.10176] <2> ConnectionCache::connectAndCache: Acquiring new connection for host wilbs001, query type 161
06:37:15.736 [12084.10176] <2> vnet_pbxConnect: pbxConnectEx Succeeded
06:37:15.736 [12084.10176] <2> logconnections: BPDBM CONNECT FROM 172.17.190.55.4012 TO 172.17.190.55.1556 fd = 11424
06:37:15.736 [12084.10176] <8> vnet_check_vxss_client_magic_with_info: [vnet_vxss_helper.c:871] Ignoring VxSS authentication 2 0x2
06:37:15.798 [12084.10176] <2> db_end: Need to collect reply
06:37:19.298 [12084.10176] <16> write_data: cannot write image to disk, Invalid argument
06:37:19.298 [12084.10176] <4> write_backup: Calling close_all_ft_pipes
06:37:19.298 [12084.10176] <2> wait_for_sigcld: waiting for child 11048 to exit, timeout is 3600
06:37:19.298 [12084.10176] <2> bptm: EXITING with status 84 <----------
06:37:19.298 [12084.10176] <2> cleanup: Detached from BPBRM shared memory"

The other 7 clients in this policy are backing up fine.

I had this issue last year, with a handfull of other clients, and was told to upgrade the Data Domain software, which we did and the issues stopped.  However, I seriously doubt that is the situation here.

Thoughts?

Thanks,

Sven

[RamNagalla]

hi,

the first recommendation is to upgrade your OST plug in to 2.5.2.x

06:37:15.704 [12084.10176] <16> 703249:bptm:12084:wilbs001: D:\Veritas\NetBackup\bin\\ost-plugins\libstspiDataDomain.dll:stspi_get_image_prop_v10 STS_EPLUGIN Bad file descriptor

you would need to look ddfs.log if you still having this issue afer OST plug-in upgrade.

[RamNagalla]

when you say only one client is having the issue , did you check if any other backups are active on other clients to same DD and media server?

did you try the backup when there is no load on Network?

[wellssh]

I couldn't find OST v2.5.2, but I did upgrade to OST 2.5.1.0, which seems to have done the trick.  That one client that was failing is now backing up just fine.

Thanks,

Sven

[Yasuhisa_Ishika]

Seven,

Rather than posting to MelB's disscussion, you should open new disscussion for your trouble.

[MelB · Accepted solution]

Hello all,

I apologize for the late comment, but I solved the problem.  I discovered that one of my team members was working on the replication on the unit and did not mention to me that they were making configuration changes on the DataDomain 580 unit.  When they attempted to configure replication with a distant unit, it broke the OST connection between my Netbackup Media Server and DataDomain.  I have since broke the replication connection between the distant and local DataDomain units and was able to bring up OST again and I am not getting the (84) errors anymore.  Thanks for all of your help!

MelB