Forum Discussion

drahrig's avatar
drahrig
Level 4
10 years ago

DQL report problem

I am attempting to create a Sensitive files with violated policies report from the template. I have uploaded a .csv with the policies named, however when the report run and I look at the output, no policy is listed. I have included the script, any suggestions?

 

FROM   path

GET      device.name,

             msu.name,

             absname,

             activity_count,

             dlp_policies

 

IF           issensitive=1

 

AND       path.msu.name = "xxx"

              path.msu.name = "yyy"

              path.msu.name = "zzz"

 

FORMAT    dlp_policies AS CSV;

Data Insight
Information Governance
Reporting
Windows Server (2003-2008)
  • drahrig's avatar
    drahrig
    9 years ago

    The issue was resolved when we increased the memory allocated to the sensitivefilesjob on the management server.

10 Replies

Sort By
  • Rod_p1's avatar
    Rod_p1
    Level 6
    9 years ago

    Assumptions:
    You have greater RAM available than the setting you use and it is extra in the machine.
    Default setting in a 64bit machine is 4GB of RAM (4096)

    Command should be performed on the node assigned as Management Server (MS) hosting the console.

    Note: the MS node should always be number 1
     

    Error is:

    V-378-1312-807: Error importing Sensitive file paths from Data Loss Prevention Server (Details: OutOfMemoryError detected. SOLUTION: Set higher memory limit for DLP scan by issuing following command on the Management Server [default (mb):512]: configdb -o -T node -k 1 -J sensitivefilejob.max.memory -j <new_limit_mb>)

    Remedy:

    (Perform these steps in a DOS window opened as Administrator with permissions to run the binaries)

    Check for the inclusion of any prior changes in the Object Attributes:

    C:\Program Files\Symantec\DataInsight\bin>configdb -p -T objattr -k 1 |findstr sensitive

    Modify the setting to  a larger number to test the download (Example 12GB):

    C:\Program Files\Symantec\DataInsight\bin>configdb -o -T node -k 1 -J sensitivefilejob.max.memory -j 12288

    Check to verify the change:

    C:\Program Files\Symantec\DataInsight\bin>configdb -p -T objattr -k 1 |findstr sensitive

    node:1:sensitivefilejob.max.memory=12288

    Run the job to initiate the download:

    Note: the job name is case sensitive.

    C:\Program Files\Symantec\DataInsight\bin>configcli list_jobs|findstr Dlp
    | DlpSensitiveFilesJob|10/20/15 12:00 AM|10/21/15 12:00 AM|      0 00 00 * * ? *|

    C:\Program Files\Symantec\DataInsight\bin>configcli execute_job DlpSensitiveFilesJob
    Job started

    The error should be removed and the download successful. Continue testing with a larger value until there is no longer an error and the download is successful if the first attempt still results in a dataset larger than the allocated memory.

  • drahrig's avatar
    drahrig
    Level 4
    9 years ago

    The issue was resolved when we increased the memory allocated to the sensitivefilesjob on the management server.

  • Rod_p1's avatar
    Rod_p1
    Level 6
    9 years ago

    Some new information after an internal discussion today over your issue Dave.

    Seems in a properly configured DLP / DI integration we create new tables that can be polled when creating a 'PATHS' view for DQL.

    You can check the values are present in the C:\DataInsight\data\conf\policy.db.### database.

    The quick procedure to query (Never change anything in the databases!) the database would be to determine the tables and see what is present.

    Example:
    Do not use limits if you wish to see them all and output to CSV if you wish to have a list.

    DQLquery_0.jpg

    You can ignore the double-byte characters in my example I do not have the correct character set loaded.

    We should expect that you have the table populated for the Policy names to show up.

    My data is limited in the lab to provide a sample.

    Report Output Sample
    path_rowid device_name msu_name absname activity_count dlp_policies
    1 XXX.XXX.89.2 scaleshare10 \\XXX.XXX.89.2\scaleshare10\dfs_final1\data_insight-DataTest 1 "SNA"  
    2 XXX.XXX.89.2 scaleshare10 \\XXX.XXX.89.2\scaleshare10\dfs_final1\diiii\sna1.csv 1 "HR"  
    3 XXX.XXX.89.2 scaleshare10 \\XXX.XXX.89.2\scaleshare10\dfs_mismatch\dir1\f1 1 "dashboard"
    14 XXX.XXX.89.2 scaleshare10 \\XXX.XXX.89.2\scaleshare10\backup\Rod_test_dfs\dir1\file1 1 "Sensitive Policy"
    28 XXX.XXX.89.3 anushare4 \\XXX.XXX.89.3\anushare4\texas.txt 1 "Mgmt Policy"
    97 XXX.XXX.89.3 ShareExportTest \\XXX.XXX.89.3\ShareExportTest\two_paths_issue\report_output.zip 2 "Mgmt Policy"

     

     

     

     

     

     

    Rod

     

  • Rod_p1's avatar
    Rod_p1
    Level 6
    9 years ago

    Some new information after an internal discussion today over your issue Dave.

    Seems in a properly configured DLP / DI integration we create new tables that can be polled when creating a 'PATHS' view for DQL.

    You can check the values are present in the C:\DataInsight\data\conf\policy.db.### database.

    The quick procedure to query (Never change anything in the databases!) the database would be to determine the tables and see what is present.

    Example:
    Do not use limits if you wish to see them all and output to CSV if you wish to have a list.

    DQLquery_0.jpg

    You can ignore the double-byte characters in my example I do not have the correct character set loaded.

    We should expect that you have the table populated for the Policy names to show up.

    My data is limited in the lab to provide a sample.

    Report Output Sample
    path_rowid device_name msu_name absname activity_count dlp_policies
    1 XXX.XXX.89.2 scaleshare10 \\XXX.XXX.89.2\scaleshare10\dfs_final1\data_insight-DataTest 1 "SNA"  
    2 XXX.XXX.89.2 scaleshare10 \\XXX.XXX.89.2\scaleshare10\dfs_final1\diiii\sna1.csv 1 "HR"  
    3 XXX.XXX.89.2 scaleshare10 \\XXX.XXX.89.2\scaleshare10\dfs_mismatch\dir1\f1 1 "dashboard"
    14 XXX.XXX.89.2 scaleshare10 \\XXX.XXX.89.2\scaleshare10\backup\Rod_test_dfs\dir1\file1 1 "Sensitive Policy"
    28 XXX.XXX.89.3 anushare4 \\XXX.XXX.89.3\anushare4\texas.txt 1 "Mgmt Policy"
    97 XXX.XXX.89.3 ShareExportTest \\XXX.XXX.89.3\ShareExportTest\two_paths_issue\report_output.zip 2 "Mgmt Policy"

     

     

     

     

     

     

    Rod

     

  • Rod_p1's avatar
    Rod_p1
    Level 6
    9 years ago

    Dave do you have policies showing up in the dashboard report for the filers you are querying?

    Since you are using this list it is assumed yes.

    Assuming we have proper policies we would expect them to be populated.

    Are there any errors in the reportrun log for the report?


    Rod

     

     

  • drahrig's avatar
    drahrig
    Level 4
    10 years ago

    I have used the list from the DI dashboard as well as a list provided by the DLP admin. Both return the same results: dlp_policies """","""","""". The policies are not being populated into the column. 

  • Rishi_Thaper's avatar
    Rishi_Thaper
    Level 4
    10 years ago

    Hi Drahig,

    The following query comes back with policy list for me. Can you check if  policy names correctly appear in Data Insight dashboard / context map? Also, what version of DI are you using?

     

    FROM  path

    GET    device.name,
               msu.name,
               absname,
               activity_count,
               dlp_policies

    IF        issensitive=1 
               AND msu.name IN ( "xxx", "yyy", "zzz")

    FORMAT dlp_policies AS CSV;

     

    Regards,

    Rishi

     

  • drahrig's avatar
    drahrig
    Level 4
    10 years ago

    When I run the report the msu's show up, however, the dlp filed is populated with "" instead of the policies.

  • Phil_Rosier's avatar
    Phil_Rosier
    Level 4
    10 years ago

     

    Hi Drahrig

    I think the list of MSU's should be tied together as an OR function, such that the report will create a record if there is a sensive file exists in either msu "xxx", "yyy" or "zzz". Please see below.

    FROM  path

    GET    device.name,
               msu.name,
               absname,
               activity_count,
               dlp_policies

    IF        issensitive=1 
               AND (msu.name = "xxx"
               OR msu.name = "yyy"
               OR msu.name = "zzz")

     

    FORMAT dlp_policies AS CSV;

     

    Alternatively you could specify the list of msu's as a list.

    FROM  path

    GET    device.name,
               msu.name,
               absname,
               activity_count,
               dlp_policies

    IF        issensitive=1 
               AND msu.name IN ( "xxx", "yyy", "zzz")

    FORMAT dlp_policies AS CSV;

     

    Or if it's a long list, you can import it as a csv file

    FROM  path

    GET    device.name,
               msu.name,
               absname,
               activity_count,
               dlp_policies

    IF        issensitive=1 
               AND msu.name IN FILE ( "msu_list.csv")

    FORMAT dlp_policies AS CSV;

     

    Kind regards

    Phil