Forum Discussion
Hi T_N,
Because of this message i think that can be something with your Multi-Threaded Agent. Are you using MSDP right? If yes take a look below.
If Im not mistaken there is a parameter called SessionCloseTimeout that you can increase..According to this log the value is setted fot 1200s...Maybe this can help you.
Log.
07/17/2017 22:31:37 - Critical bptm (pid=214108) Storage Server Error: (Storage server: PureDisk:cdcnetbudd6) mtstrm_close_write_channel: Fatal error occured in Multi-Threaded Agent: Close Write Channel command failed: Cr_ErrnoException: Timed out after waiting 1200s to send command Close Write Channel to mtstrmd V-454-96
Sorry for long post, I would like to post only the TNS, but both are not available...I needed to copy from my Evernote...this is a problem that is happening with some Veritas Technotes....you can see if you want.
https://www.veritas.com/support/en_US/article.000072903
https://www.veritas.com/support/en_US/article.000072906
About the MSDP Deduplication Multi-Threaded Agen
Beginning with the NetBackup 7.6 release, the MSDP deduplication process can use a Multi-Threaded Agent for most data sources. The Multi-Threaded Agent runs alongside the deduplication plug-in on both the clients and the media servers. The agent uses multiple threads for asynchronous network I/O and CPU core calculations. During a backup, this agent receives data from the deduplication plug-in through shared memory and processes it using multiple threads to improve throughput performance. When inactive, the agent uses minimal resources.The NetBackup Deduplication Multi-Threaded Agent improves backup performance for both client-side deduplication and media server deduplication.The Deduplication Multi-Threaded Agent uses the default configuration values that control its behavior. You can change those values if you want to do so. The following table describes the Multi-Threaded Agent interactions and behaviors. It also provides links to the topics that describe how to configure those interactions and behaviors.
The mtstrm.conf configuration file controls the behavior of the Deduplication Multi-threaded Agent. The default values balance performance with resource usage.
A procedure exists that describes how to configure these parameters.
The pd.conf file resides in the following directories:
(UNIX) /usr/openv/lib/ost-plugins/
(Windows) install_path\Veritas\NetBackup\bin\ost-plugins
See Configuring the Deduplication Multi-Threaded Agent behavior.
The mtstrm.conf file is comprised of three sections. The parameters must remain within their sections. For descriptions of the parameters, see the following sections:
The mtstrm.conf file resides in the following directories:
UNIX: /usr/openv/lib/ost-plugins/
Windows: install_path\Veritas\NetBackup\bin\ost-plugins
Logging parameters
The following table describes the logging parameters of the mtstrm.conf configuration file.
Table: Logging parameters (mtstrm.conf file)
Parameter
Description
LogPath | The directory in which the mtstrmd.log files are created. Default values:
|
Logging | Specify what to log: Default value: Logging=short,thread. Possible values: minimal: Critical, Error, Authentication, Bug short : all of the above plus Warning long : all of the above plus Info verbose: all of the above plus Notice full : all of the above plus Trace messages (everything) none : disable logging To enable or disable other logging information, append one of the following to the logging value, without using spaces: ,thread : enable thread ID logging. ,date : enable date logging. ,timing : enable high-resolution timestamps ,silent : disable logging to console |
Retention | How long to retain log files (in days) before NetBackup deletes them. Default value: Retention=7. Possible values: 0-9, inclusive. Use 0 to keep logs forever. |
LogMaxSize | The maximum log size (MB) before NetBackup creates a new log file. The existing log files that are rolled over are renamed mtstrmd.log.<date/time stamp> Default value: LogMaxSize=500. Possible value: 1 to the maximum operating system file size in MBs, inclusive. |
Process parameters
The following table describes the process parameters of the mtstrm.conf configuration file.
Table: Process parameters (mtstrm.conf file)
Parameter
Description
MaxConcurrentSessions | The maximum number of concurrent sessions that the Multi-Threaded Agent processes. If it receives a backup job when the MaxConcurrentSessions value is reached, the job runs as a single-threaded job. By default, the deduplication plug-in sends backup jobs to the Multi-Threaded Agent on a first-in, first-out basis. However, you can configure which clients and which backup policies the deduplication plug-in sends to the Multi-Threaded Agent. The MTSTRM_BACKUP_CLIENTS and MTSTRM_BACKUP_POLICIES parameters in the pd.conf control the behavior. Filtering the backup jobs that are sent to the Multi-Threaded Agent can be very helpful on the systems that have many concurrent backup jobs. See MSDP pd.conf file parameters. Default value: MaxConcurrentSessions= (calculated by NetBackup; see the following paragraph). NetBackup configures the value for this parameter during installation or upgrade. The value is the hardware concurrency value of the host divided by the BackupFpThreads value (see Table: Threads parameters (mtstrm.conf file)). (For the purposes of this parameter, the hardware concurrency is the number of CPUs or cores or hyperthreading units.) On media servers, NetBackup may not use all hardware concurrency for deduplication. Some may be reserved for other server processes. For more information about hardware concurrency, see the pd.conf file MTSTRM_BACKUP_ENABLED parameter description. See MSDP pd.conf file parameters. Possible values: 1-32, inclusive.
| ||
BackupShmBufferSize | The size of the buffers (MB) for shared memory copying. This setting affects three buffers: The shared memory buffer itself, the shared memory receive buffer in the mtstrmd process, and the shared memory send buffer on the client process. Default value: BackupShmBufferSize=2 (UNIX) or BackupShmBufferSize=8 (Windows). Possible values: 1-16, inclusive. | ||
BackupReadBufferSize | The size (MB) of the memory buffer to use per session for read operations from a client during a backup. Default value: BackupReadBufferSize=32. Possible values: 16-128, inclusive. | ||
BackupReadBufferCount | The number of memory buffers to use per session for read operations from a client during a backup. Default value: BackupReadBufferCount=3. Possible values: 1 to 10, inclusive. | ||
BackupBatchSendEnabled | Determines whether to use batch message protocols to send data to the storage server for a backup. Default value: BackupBatchSendEnabled=1. Possible values: 0 (disabled) or 1 (enabled). | ||
FpCacheMaxMbSize | The maximum amount of memory (MB) to use per session for fingerprint caching. Default value: FpCacheMaxMbSize=20. Possible values: 0-1024, inclusive. | ||
SessionCloseTimeout | The amount of time to wait in seconds for threads to finish processing when a session is closed before the agent times-out with an error. Default value: 180. Possible values: 1-3600. | ||
SessionInactiveThreshold | The number of minutes for a session to be idle before NetBackup considers it inactive. NetBackup examines the sessions and closes inactive ones during maintenance operations. Default value: 480. Possible values: 1-1440, inclusive. |
Threads parameters
The following table describes the threads parameters of the mtstrm.conf configuration file.
Table: Threads parameters (mtstrm.conf file)
Parameter
Description
BackupFpThreads | The number of threads to use per session to fingerprint incoming data. Default value: BackupFpThreads= (calculated by NetBackup; see the following explanation). NetBackup configures the value for this parameter during installation or upgrade. The value is equal to the following hardware concurrency threshold values.
For more information about hardware concurrency, see the pd.conf file MTSTRM_BACKUP_ENABLED parameter description. |
BackupSendThreads | The number of threads to use per session to send data to the storage server during a backup operation. Default value: BackupSendThreads=1 for servers and BackupSendThreads=2 for clients. Possible values: 1-32, inclusive. |
MaintenanceThreadPeriod | The frequency at which NetBackup performs maintenance operations, in minutes. Default value: 720. Possible values: 0-10080, inclusive. Zero (0) disables maintenance operations. |
Regards,
Thiago Ribeiro
I moved that backup to another netbackup appliance, it's still running. I think it may be deduplication jobs running that cause the error. Veritas support " The Transparent Huge Pages feature is enabled by default in RHEL/CentOS 6 or 7. The kernel will always attempt to satisfy a high-order memory allocation using hugepages. If no hugepages are available, the kernel will try to defrag memory to get hugepages. This defrag effort is time-consuming when system is under memory pressure and will cause high latency to user-land processes.". He asked me to :
Disable THP without rebooting & Disable THP at boot time by adding transparent_hugepage=never to the end of the kernel line and rebooting . I will reboot that appliance on next thursday and let see what happens :)