NBU 7.5.04 - Image Cleanup Status 1
For the last week or so we've noticed that our Image Cleanup jobs are not completing successfully. The status is 1 with the following job details:
4/2/2014 12:53:19 AM - Info bpdbm(pid=4876) image catalog cleanup
4/2/2014 12:53:19 AM - Info bpdbm(pid=4876) Cleaning up tables in the relational database
4/2/2014 12:53:19 AM - Info bpdbm(pid=4876) deleting images which expire before Wed Apr 02 00:53:19 2014 (1396414399)
4/2/2014 12:53:43 AM - Info nbdelete(pid=7664) deleting expired images. Media Server: wilbs003.wilm.ppdi.com Media: @aaaag
4/2/2014 12:54:13 AM - Error nbdelete(pid=7664) Cannot obtain resources for this job : error [25]
4/2/2014 12:54:15 AM - Info nbdelete(pid=7664) deleting expired images. Media Server: wilbs001 Media: @aaaai
4/2/2014 12:54:45 AM - Error nbdelete(pid=7664) Cannot obtain resources for this job : error [25]
4/2/2014 12:54:47 AM - Info nbdelete(pid=7664) deleting expired images. Media Server: wildisk Media: D:
4/2/2014 12:54:54 AM - Error nbdelete(pid=7664) Fragments were not removed. Server wildisk is invalid (37 )
4/2/2014 12:54:54 AM - Warning bpdbm(pid=4876) nbdelete failed with status (37)
4/2/2014 12:54:54 AM - Info bpdbm(pid=4876) deleted 20 expired records, compressed 0, tir removed 0, deleted 38 expired copies
the requested operation was partially successful(1)
The job was successfully completed, but some files may have been
busy or unaccessible. See the problems report or the client's logs for more details
Any ideas on where to start/look, and can I perform the same processes manually on the Master server?
Thanks,
Sven
Interesting issue. You might want to create the bpdm log on the media servers to see if we ever connect. A 25 typically is a socket connection error(normally reverse hostname lookup).
Another thing that might be helpful is to check the database tables to see if you still have an entity named wildisk in the tables.
To do that, from the master run: (Windows) <install_path>Veritas\NetBackup\bin\nbdb_unload <path>
(Unix) /usr/openv/db/bin/nbdb_nload <path>
** Create a directory somewhere to dump the data too - insert that into 'path' in the command.
Run findstr or grep to see if wildisk is in any of the tables. If that entity is in the tables, try to use nbemmcmd -deletehost (in the NetBackup\bin\admincmd directory).
Fields required are:
nbemmcmd -deletehost -machinename <string> -machinetype <api | app_cluster | client | cluster | disk_array | foreign_media | index_server | master | media | ndmp | remote_master | replication_host | virtual_machine>
*** If you don't know the type of machine - try running nbemmcmd -listhosts to if the type of machine is listed in that output.
Also note if there are images still assigned to that host, the deletion will fail (which I suspect since we're trying to clean it).
If this information doesn't help, I'd suggest you open a support case and have someone help you fix the invalid references in the database.
Deb