cancel
Showing results for 
Search instead for 
Did you mean: 

Backup Anomaly detection in NetBackup

swapnildombale
Level 2
Employee

Introduction

  • NetBackup has in-build capability to identify the unusual pattern in the backup meta data. This unusual behavior could be because of any Ransomware attack on the source client system.
  • Every day we read about companies who have been hacked, and their data has been held for ransom or threatened with exposure of sensitive data. So this solution in NetBackup can help to identify such events early to take any actions.

What is Anomaly Detection

  • Now let’s understand what Anomaly detection is. Anomaly detection is the ability to detect any unusual pattern or any unusual data or any unusual events. These are unusual behavior when compared to the normal behavior which indicates some underlying problems such as Ransomware attack.
  • Identifying such unusual changes earlier will help to take prompt actions against ransomware attack.

 

swapnildombale_0-1651756113624.png

                                 Normal Data  

 

swapnildombale_1-1651756113627.png

                             Anomalous Data

 

  • From above images, we can identify that for a data plot of ‘Number of files’ vs ‘Image size’, we can see in first figure, data seems to be having normal trend. In second figure, the highlighted point in Red is deviated much away from the rest of normal data points. So just by looking at this graph we can see that there is something unusual behavior in that backup with huge number of backup files getting added. This is what we detect as the Anomaly.

 

Impact on Backup metadata in Ransomware attack

In the event of attack on the system, after taking backup from that system we can correlate the attack with the backup metadata.

  • File contents encryption
    • If attack encrypts file, then in this case the Deduplication rate would go down.
  • File Renames
    • Number of files will increase in the backup
  • Files deletions
    • Number of files will decrease
    • Backup size will decrease
  • New extensions files added
    • Number of files will increase
    • Backup size (and total data transferred) will increase.

 

So with such scenarios with come up with the list of backup metadata attributes which we can use to identify any deviation in the normal trend.

  1. Backup images size
  2. Number of files
  3. Data transferred
  4. Deduplication rate
  5. Total time for job completion.

 

How Anomalies get detected in NetBackup?

Let’s us take below example to explain the detection logic. In this example data for 3 variables Image size, Number of files and Deduplication rate has been plotted. The data has been Standardized to make it in the same scale.

 

swapnildombale_2-1651756113642.png

 

                                   Fig1

 

swapnildombale_3-1651756113654.png

                                     Fig 2

 

  • Figure 1
    • Here we have plotted 100 observations on a standard deviation scale.
    • We can find each observation are reachable to each other by a distance ‘d’.
    • So, all such observations are clubbed together in a cluster of similar observations.
    • Since all observations are within the cluster, we can say that there is no outlier or anomaly in this data set.
  • Figure 2
    • We can see that the marked cluster of observations which are near to each other by a distance ‘d’.
    • Also, we can see one observation which is much away from rest of the observations. Hence that observation is not part of the cluster, and such observations are considered as anomalous.

So based on this explanation lets see how NetBackup detects the Anomaly in the backup job.

Baseline

  • NetBackup Anomaly detection at first captures the last 3 months of Image data as a baseline. This ensures we have enough observations available to start the detection.
  • Detection works for each combination
    • Policy
    • Client
    • Policy Type
    • Schedule
    • Destination storage
    • Asset ID
  • Once the historical data captured, for the subsequent backups completed, NetBackup tries to fit the new backup metadata into the cluster using training data for the same client-policy combination mentioned above.
  • Detection requires minimum 30 observations i.e., minimum of 30 backups should be done before doing detection. If for a given client-policy combination mentioned above there are not enough observations, detection logic will wait till there are minimum 30 observations. Till that point, all backups done are considered as training data only.

 

Detection

  • NetBackup has 3 different services for
    • Training data gather
    • Detection
    • Alerting
  • As mentioned in Baseline section, NetBackup first captures the Historical data and then it switches to detection mode.
  • Below is the overall flow for the Anomaly detection solution in NetBackup.

 

 

swapnildombale_0-1651756478121.png

  • For a given client-policy combination (mentioned in Baseline section above), if there are minimum 30 observations then the detection for that backup job starts.
  • As mentioned earlier, detection logic forms different clusters of observations which are near to each other. And if a given observations does not fit in any of the cluster, then it is considered as Anomaly. Refer Fig 2 above.
  • For the very first time, the detection mechanism finds the suitable cluster diameter, the distance ‘d’ mentioned in Fig 1 above. This distance ‘d’ will help to form the clusters. For first time detection, it uses all training data and do the iterative processing to find the suitable distance ‘d’ such that all the training data are within the clusters.
    • This process of finding the suitable distance ‘d’ to form cluster is largely dependent on the input training data for a given client-policy combination.
    • If there are too many variations in the training data, then this distance ‘d’ would be larger than for the data with minimum variation.
    • For example, if the backup size for given client-policy combination mentioned in Baseline section varies between say 10 GB to 12 GB, this will have minimum variation versus if the backup image size varies between 10 GB to 30 GB, which will have high variation.
    • Depending on the data, there could be multiple clusters formed from the training data.
    • See below figure.

                  

                                         swapnildombale_2-1651756824869.png

 

 

  • This cluster information gets stored and gets calibrated every after 14 days to accommodates the variation of the future data.
  • At overall, below are the high-level flow to detect Anomaly in NetBackup

Anomaly Score calculations for detected Anomaly

  • Once the detection is done for the given job, detection logic finds the severity for the given Anomaly.
    • Severity is nothing but the measure of how much the anomalous observations deviated from the normal trend.
    • The higher is the deviation the higher is the score.
    • Here we calculate the nearest cluster from the Anomalous observation (in standard deviation scale). And this ‘distance’ (Euclidian distance) we use as the Score.

                                      swapnildombale_3-1651756916783.png

 

  • Above fig. explains the Score calculations logic. This diagram contains 3 clusters. And from the Anomalous observation, it calculates the distance to all clusters to find out the nearest cluster.
  • In this example, Cluster 2 is nearest with distance ‘d2’.  Hence for this Anomaly Score is ‘d1’

 

False Positive treatment

  • Anomaly can get generated even if there is no real Ransomware attack. For example, if user carry out some activity on the system deleting huge number of files, then there would be anomaly detected on that client.
  • Since user is aware of this activity and there would be chance that in future this kind of activity will get happen again, user can mark this Anomaly as False positive.
  • Marking Anomaly as False positive will not generate ‘similar’ Anomaly in future for same client-policy combination mentioned in Baseline section                          swapnildombale_4-1651756994704.png

 

  • As per above fig. user has one Anomaly marked as false positive. Marking that there would be new cluster gets created and any new observation falling in that cluster will be treated as non-anomalous.

 

 

How sensitivity settings work.

  1. For Anomaly generated such as mentioned in Fig1 which is close to the cluster but still the Anomaly since it is away from rest of other observations more than distance ‘d’.
  2. In this case by increasing the sensitivity configuration parameter  by certain percentage will actually increases the distance ‘d’ so that new observation can be reachable from other observation in the cluster.
  3. This can be viewed as new cluster with increased size as mentioned in Fig2 and this Anomaly will be no longer considered as Anomaly

 

swapnildombale_5-1651757068016.png

                                     Fig 1

 

swapnildombale_10-1651756113713.png

                                    Fig2

 

 

 

 

 

 

 

 

 

 

3 REPLIES 3

Azhar4
Level 4

Very good explanation.thanks for sharing

quebek
Moderator
Moderator
   VIP    Certified

Hello

Good one - but is there a way to post these anomalies on OpsCenter? So far I did not touch NB IT Analytics so can't tell if its there. For sure it is not in Opscenter in 9.1.0.1 version. Having a lot NBU domain is it hard to visit their webUI on daily basis just to check if there was anomaly or not... also its a pity it is disabled by default (I am speaking about 9.1 NBU family we are not yet ready for 10).

 

Koehler
Level 1
Employee

Hello @quebek,

Version 10.0 is the last release of NetBackup OpsCenter. Please see https://www.veritas.com/support/en_US/article.100052742 for details on the transition to NetBackup IT Analytics.

Regards,
Sebastian Koehler