Did a catalogue restore, but seem to have missing information.
So I did a catalogue restore (full one) which honestly I felt went sideways. It said it finished partially but the only errors I could find were IOC_CHFLAG_SET errors. I had my profiles, on the file system the DB was backup to it's 375GB range. So I assumed all the data was loaded. When I login to the JavaGUI, ALL my volume pools are gone ALL my tapes are in 1 pool now. Is this normal? Because on the old system We had volume pools for each job which had tapes in it (We don't backup daily, we archive data so it goes on tape once and that's it so I believe that's why they used volume pools when they set it up ages ago). Either way. It just feels to me like I'm missing Data, and I'm not sure what to do now. I've opened a ticket but I'm running out of hours I can say "it's good let's test" vs "we need to roll back" (Let's just say it's been a long week getting to this point).
Notification emails not being sent on NetBackup appliance after 10.3/5.3 upgrade
TLDR: Not getting job failure notification or catalog backups emails from a Netbackup appliance after an upgrade to 5.3/10.3 but the email test notifications are working fine? Check the "mailq" command from a maintenance shell and see if mail is stuck in there. Then, check the postfix service on the appliance from the maintenance shell. The service might have stopped/died at some point. There may be a permissions error on a certain library: libpostfix-global.so -- Change permissions if needed and restart the postfix service from the maintenance shell. Not TLDR: I'm guessing this is one of those "one-offs" that likely won't happen to anyone else, however, if someone else ever sees something similar, here's what I went through: I recently upgraded a customer of mine. 2 x 5240 (edit: not 5230s! My bad) appliance, each acting as a Master/Media, with AIR. It was at version 3.1/8.1 , so it was a two-phase upgrade. To 4.1/9.1 first, did some quick backup and recovery testing, then 5.3.0.1/10.3.0.1 (current as of Sept 2024). Afterwards I ran the firmware upgrade for each appliance, successfully. After the upgrade, it looked like no backup job failure email notifications were being received from one appliance, and no catalog emails either. The other appliance was behaving normally. Both appliance notifications for job failures and catalog backups worked before the upgrade. Doing an "email test" from the Appliance CLISH under Settings > Notifications > Alert Configuration worked though -- No problem. There were no changes in email configuration in the administration console either in host properties, etc. All looked like it did before. Appliance CLISH hardware & software tests ran fine. bpps check of NBU services from the maintenance shell looked fine. It's been a while since I've had to dig into something like this, so it took me a while to figure out. This is opinion, impression, and anecdotal evidence only, and I do not claim to know what lies deep in the hearts and brains of Veritas engineers and developers. This was a weird one, but I'll admit I don't play with NBU appliances as much as I once did. First off I checked the SMTP configuration in the appliance WEBUI as I saw there was a note about the SMTP password sometimes needing to be re-entered after an upgrade. (https://www.veritas.com/support/en_US/article.100061042). But, no issues there. I checked the verbose bpdbm and bpbrm logs, and messages log for any errors or anything weird with mailing out notifications. I didn't see anything weird. I didn't think anything would change in the admin console, but I went through and checked the host properties and email settings for the master servers. All were exactly as they were before. As mentioned above, when I did an email test from the appliance CLISH... it worked. Came through without issue. I went into the maintenance shell and sent some mail manually from the command line, using mailx/sendmail. It worked fine, and arrived immediately. On a whim though and I don't know why I thought of it, I ran a "mailq" command from that maintenance shell: 107 emails in the queue. At that point I was asking myself "how is that possible when alert test emails and mailing from the command line are working immediately?" I didn't find any direct references to regarding appliances (and if someone has those, I'd love to see them), but I did see a few vague references and long-dead symantec tech articles, about checking postfix config on BYOS Linux NBU installs for unrelated issues, and it made me wonder. I went looking for a postfix service via 'service postfix status'. It was down. I then ran "journalctl -u postfix" and saw that the postfix service had died after the upgrades and subsequent reboots a few days ago. The error was: loading shared libraries libpostfix-global.so: permission denied In my case, I was able to do a "service postfix restart" from the elevated maintenance shell and it started cleanly and immediately processed the whole mail queue successfully. You may end up having to do some manual permissions changes to that library if that doesn't work. (if you're not comfortable with this or mucking around in the maintenance shell in general, please contact Veritas Support) My theory, and it's not like I can easily prove it, is this: The appliance itself uses sendmail (or another MTA) for sending out hardware failure notifications, test emails, etc. The NBU installation/application environment, for lack of a better term, seems to use its own postfix MTA with configuration inferred from initial appliance settings. When the postfix service errored out, job failure and catalog notifications just filed into the mail queue with no notification. The appliance itself is not aware of the postfix service. So it doesn't come up as a problem in the appliance software tests from the CLISH. The NBU environment itself does not care about the postfix service either. It sends mail out on job failures or other conditions, and because it ends up in the queue, there's no error, it doesn't care, nothing strange gets logged. I could be right, wrong, or anything in between on this. However, it fixed the problem for me. If someone has some further information and sources for the appliances and how this is configured, I'd love to see it. No logs to post as I do not have permission to do so from a sensitive environment. May you never encounter this. I've done upgrades on appliances before and never encountered this. But if you do, I hope this provides something to reference.NBU 10.4 Upgrade License XML Parsing Error
Running an upgrade from 9.1.0.1 to 10.4 on our master server running RHEL8 but the install script is stuck at the enter license file step due to the error: "State: Invalid ( Operation failed due to an XML parsing error. ) Key is direct from our support rep. Alternatively, can I cancel this step without restarting to add later as we've seen, it will be prompt at GUI's first launch once the upgrade completes. TIA
Veritas Education Services NetBackup 10.0 administration training, now available
Check out the newest NetBackup update: Veritas NetBackup 10.0: Administration and Veritas NetBackup 10.0 Advanced Administrationare now available for registration! Stay current with our updated courses with all of NetBackup's latest capabilities. In NetBackup 10.0 Administration training, you will learn the general principles, configuration, and management of NetBackup, including how to best utilize the NetBackup tools and interfaces, effectively monitor backup and restore operations, and ensure that the data recovery objectives are met. Acquire the skills to make your data protection strategy successful today. Additionally, in NetBackup 10.0 Advanced Administration, you will learn advanced NetBackup topics, including NetBackup performance, security, disaster recovery, application, database protection on physical and virtual machines, protecting data backed up to and from the Cloud, and much more. These courses are available in Onsite Instructor-Led Training (ILT), Virtual Instructor-Led Training (VILT), and Learning Lab formats. More information on training delivery methods, class availability and registration instructions can be found here.How to protect Azure Stack Hub and Azure Stack HCI
There are a lot of misunderstandings about Azure Stack solutions and how to protect each one of them. I want to help you better understand each solution within Azure Stack portfolio and how to protect them using Veritas NetBackup. As you already know, Veritas NetBackup is an enterprise-class backup and recovery suite that provides unified data protection for multi-cloud, virtual and physical environments that can be globally managed from a single console, including all Azure Stack solutions. The usual misunderstanding is to think Azure Stack is a solution itself, but Azure Stack is a portfolio of products comprised of three distinct offerings – Azure Stack Edge, Azure Stack Hub, and Azure Stack HCI. Azure Stack solutions help you extend Azure services and capabilities to your environment of choice – from the data center to edge locations and remote offices. The confusion kicks in mostly between Azure Stack Hub and Azure Stack HCI and how to protect them. https://docs.microsoft.com/en-us/azure-stack/operator/compare-azure-azure-stack?view=azs-2108 The most important feature of HCI is that it can run “disconnected” from Azure, in other words, HCI is just like your branch office server. It is a box that contains compute, power, storage, and network connections and holds Hyper-V based virtualized workloads and it has the option to connect to some Azure services. And so, as it is like a Hyper-V server, you can use Veritas NetBackup Hyper-V policy type to protect it along with all the features Veritas NetBackup already provided for Hyper-V: NetBackup for Hyper-V uses snapshot technology to keep virtual machines 100% available to users. NetBackup for Hyper-V creates quiesced Windows snapshots using Volume Shadow Copy Service (VSS) and Windows Management Instrumentation (WMI). NetBackup for Hyper-V performs full backups and file-level incremental backups of the virtual machine. With the WMI backup method, it also performs block-level incremental backups and Accelerator backups. Can restore the full virtual machine from the following: Full backups of the VM. Block-level incremental backups of the VM. Accelerator backups of the VM. Can restore individual files of the virtual machine from the following: Full backups of the VM. File-level incremental backups of the VM. Block-level incremental backups of the VM. Accelerator backups of the VM. Can restore to the original virtual machine, to other locations on the Hyper-V server, or to a different Hyper-V server. For details on how to protect Hyper-V, please check the NetBackup for Hyper-V Administrator’s Guide on the following URL: https://www.veritas.com/content/support/en_US/doc/21357025-151824041-0/v21357050-151824041 On the other hand, Azure Stack Hub is really the on-premise extension of the Azure public cloud. Almost everything you can do in the public cloud, you could also deploy on Hub: from VMs to apps, all managed through the Azure portal or even Powershell, including things like configuring fault and updated domains. In this case, you can still use the same deployment of Veritas NetBackup to protect your in-cloud workloads. The cloud data protection framework leverages the CloudPoint infrastructure to drive faster proliferation of cloud providers, since v8.3, CloudPoint can protect assets in AWS, AWS Outpost, Azure, Azure Stack hub and GCP clouds. Features includes: Automatic discovery of cloud assets: NetBackup retrieves the cloud assets pertaining to the cloud accounts every 2 hours by default. This period is configurable. Protection of intelligent cloud groups based on a set of filters called queries. All the assets satisfying the query conditions will automatically be protected. Protection of Microsoft Azure resources using resource groups. NetBackup Accelerator backups. Application consistent snapshots. You can perform original location and alternate location restores. Incremental snapshots. Backup from snapshot. Restore from snapshot copy, replica copy, backup copy or duplicate copy. Restore VMs to an alternate configuration, to a different region, to a different subscription, and restore VMs or disks to a different resource group. Granular files and folders restore For details on how to protect Cloud Assets, please check the NetBackup Web UI Cloud Administrator’s Guide on the following URL: https://www.veritas.com/content/support/en_US/doc/150074555-150074602-0/v130722342-150074602Deployment of Veritas NetBackup on Azure Kubernetes Service (AKS)
As customers adopt a multi-cloud strategy (public, private, and hybrid), the need to align services with key cloud characteristics, such as deployment, cost management, and scaling, is increasing. Microsoft Azure Kubernetes Service (AKS) is a fully managed container orchestration service used to deploy, scale, and manage Docker containers and container-based applications in a cluster environment. AKS enables the provisioning, scaling, and upgrading of resources per requirement or demand without downtime in the Kubernetes cluster. In addition to the traditional deployment of Veritas NetBackup in the cloud, customers can deploy NetBackup’s infrastructure in a native, extensible format for greater performance and resiliency and to optimize costs when using AKS. NetBackup powered by Cloud Scale Technology delivers automation, artificial intelligence, and an elastic architecture to improve the agility and data security of any cloud, at any scale. Optimized for the cloud, NetBackup simplifies data management and delivers unmatched cyber resiliency, control, and visibility. As a prerequisite to deploying Veritas NetBackup on Azure Kubernetes Services, you will need to: Create Azure Container registry Download Container images to the Azure Container registry Create a user-assigned managed identity The container images can be found here: https://azuremarketplace.microsoft.com/en-us/marketplace/apps/veritas.netbackup You will have to download all six images to the same Azure Container Registry: NetBackup Operator NetBackup Server MSPD Meta Data MSDP Controller MSDP Engine MSDP Operator Figure 1.: Azure Marketplace – Veritas NetBackup powered by Cloud Scale Technology – Container Images Once you downloaded all images, you will use the Veritas NetBackup powered by Cloud Scale Technology – Deployment on AKS, to deploy the images in an integrated manner. Now you can start configuring the parameters for each NetBackup component, you will go through the Basics, Cluster Configuration, Primary Server Details, Media Server Details, and Storage Server Details. The typical deployment, which consists of one primary server, one media server with two replicas and one MSDP (Storage Server) with four replicas will take approximately 45 minutes until the environment is fully up and running. After that, your Veritas NetBackup on Azure Kubernetes Service is ready to start protecting your workloads. For more information about Veritas NetBackup deployment on Azure Kubernetes Service, check the following links: https://www.veritas.com/content/dam/www/en_us/documents/technical-documents/TB_netbackup_deployment_on_azure_k8s_V1568.pdf https://www.veritas.com/content/support/en_US/doc/154778686-154778690-1
