Hello folks,
Recently I am doing a stress test about DI data colluection.
The environment is based on DI 6.1 and the source data is on a Netapp sharing folder from CIFS that I put all the activity behavior in the same folder.
I used below queries to get the customized result,
FROM activity
GET user.name,
user.domain,
path.permissions.type AS permission_type,
formatedate(timestamp, "YYYY/MM/DD HH:mm:ss") AS event_time,
operation AS event,
path.absname AS file_path,
rename_target.absname AS renamed_path,
formatedate(path.last_accessed, "YYYY/MM/DD HH:mm:ss") AS last_accessed_time,
formatedate(path.last_modified, "YYYY/MM/DD HH:mm:ss") AS last_modified_time,
ipaddr AS client_ip,
(path.size/1024) AS size_kb,
path.type AS object_type,
path.extensions AS file_type,
formatedate(path.created_on, "YYYY/MM/DD HH:mm:ss") AS file_creation_time,
path.msu.type AS Share_type
FORMAT path.permissions.type AS CSV
SORTBY timestamp DESC
In order to test the data collection accuracy on this server, I compiled a python script to simulate high load file behaviorto check the records of data collections.
My python script is below,
import os
import sys
import shutil
import msvcrt
import glob
import time
path = imput('Please imput the assigned path?')
print ("Your path is",path)
os.chdir(path)
print("Start date & time " + time.strftime("%c"))
for n in range(0,1600):
#create a new folder
os.mkdir("test_folder)
#create 100 new file .txt and write contents
for i in range(0,100):
fo = open("test"+str(i)+".txt","w+")
fo = write("DI is a good product\n")
fo.close()
#copy 50 files to new rename newfilex
for j in range(0,50):
shutil.copy("test"+str(2*j)+".txt","newfile"+str(2*j+1)+".txt")
#rename the file from named test$ to Veritasfile$
for k in range(0,50):
os.rename("test"+str(2*k)+".txt","Veritas"+str(k)+"123825147.txt")
#move function
for l in range(0,50):
os.rename("test"+str(l*2+1)+".txt","test_folder/test"+str(l*2+1)+".txt")
#remove Veritas$ files
for s in glob.glob("Veritas*.txt"):
os.remove(s)
#remove newfile$ files
for t in glob.glob("newfile*.txt"):
os.remove(t)
#remove folder and included files
shutil.rmtree("test_folder")
time.sleep(120);
print("Finished date & time " + time.strftime("%c"))
The strange thing is, after I ran the script for 1600 times, the total records not consistent with 1 unit run.
If I ran above for loop (n=1), I got the records in DI report is 602 rows.
Then, after I changed to 1600 loops, theoritically the total rows would be 602*1600 = 963200 but the total amount of data collected is 831554
I would whether so much data is dropped event event I found the first several times records are accordance with 602 per unit.
I'm not sure if it's related with Netapp async mode?
I ran the report after the script finished for around 10 hours.
Could you please give me some suggestions?
