Restores queuing for 60+ minutes before beginning execution, Netbackup 7.5.0.3
We have just upgraded from 7.0 to 7.5.0.3 on our master and media servers. They are running on Sun Solaris version 10. Previously we had no issues with restores, but now it seems that most of our restores are remaining in a queued state after being submited from anywhere from 30 to 60 or minutes before they begin execution. Once they begin running, they restore fine with normal transfer rates and complete successfully. Has anyone every come across this issue before and do you have any suggestions on where to look to reduce the "queued" time ? The backups seem to run as they did prior to the upgrade, it seems to only be a problem with the restores.1.6KViews2likes6CommentsNetbackup exluding many folders form backup
Hello Team, Thanks in advance for all your help. I have Netbackup server and from it we are taking file server backup. During restoring of files, we found many files and folders are not backuped up with Netbackup. After analyzing, I found issue with NTFS permission. I have one Security groups which dont have access to that folders which I have used for backup. Now thing is I have many file servers and I dont have any details on which folder this group dont have permission. So looking for detailed backup report from Netbackup where files are folder are not included in backup due to permission issues. Please help on this. Regards, Yogesh Makwana928Views0likes3CommentsACL permissions status when restore on different OS
Hello Team, 1)) We have a win 2003 client backup on tape with NB version 6.5. We restore this data in same domain on same client. ACL permissions are restored without any issue. 2) We brought this tape to a new domain where nbu version is 7.6.0.4 and operating system is Win 2012. We import the tape and restored data: a) On UNC path of win 2012 server - ACL permissions were not restored b) On local drive of win 2012 server - ACL permissions were not restored. 3) In 7.6.0.4 netbackup, if I try to restore a data from same domain client, it restores the ACL permissions without any issue. My question is is this behavior so, because we have different operating systems? Has anyone encountered this issue...Or any expert comments. Thanks!Solved2.3KViews0likes10CommentsNeed help in restoring a file from an Imported Tape.
I am using Netbackup 7.0. Master server is linux / Client is Windows Server 2008. I restored a file from an imported Tape, but it does not appear in the folder. 15:50:05 9/21/2015: Restore Started 15:50:10 (610.xxx) Restore job id 610 will require 1 image. 15:50:10 (610.xxx) Media id P00580 is needed for the restore. 15:50:10 (610.xxx) Media id P00426 is needed for the restore. 15:50:14 (610.001) Restoring from image created Sun Jul 5 01:08:33 2015 15:50:14 (610.001) INF - If Media id P00426 is not in a robotic library administrative interaction may be required to satisfy this mount request. 15:50:41 (610.001) INF - Waiting for mount of media id P00426 on server utbak1.solint.net for reading. 15:52:10 (610.001) INF - Waiting for positioning of media id P00426 on server utbak1.solint.net for reading. 15:53:00 (610.001) INF - Beginning restore from server utbak1.solint.net to client utbak1.solint.net. 15:53:02 (610.001) /telecom/b9/17/85/512306.crexendo.com/storage/recordings/f72434ed-6b43-47c8-8d3c-2d27948211d4-1434058444168.aac 15:53:02 (610.001) Changed /telecom/b9/17/85/512306.crexendo.com/storage/recordings/f72434ed-6b43-47c8-8d3c-2d27948211d4-1434058444168.aac to /E/jimmy/f72434ed-6b43-47c8-8d3c-2d27948211d4-1434058444168.aac 15:53:02 (610.001) (610.001) INF - TAR EXITING WITH STATUS = 0 15:53:02 (610.001) (610.001) INF - TAR RESTORED 1 OF 1 FILES SUCCESSFULLY 15:53:02 (610.001) (610.001) INF - TAR KEPT 0 EXISTING FILES 15:53:02 (610.001) (610.001) INF - TAR PARTIALLY RESTORED 0 FILES 15:53:02 (610.001) Status of restore from image created Sun Jul 5 01:08:33 2015 = the requested operation was successfully completed 15:53:03 (610.xxx) INF - Status = the requested operation was successfully completed.Solved810Views1like2CommentsWindows Failover cluster backup fails
I currently have a 2 node Windows cluster that is hosting a DFS file share as the clustered resource. My NetBackup solution is a single netbackup appliance serving as the master server, and the NBU windows client installed on both nodes within the cluster, and the client name I am using for the backup policy is the virtual name of the cluster. I have no issues backing up data when the cluster resides on either node, as I also have no issues running a restore. My problem arises when I have the cluster failover during the middle of a running job. The job fails and will not restart. Any advice would be very helpful. Thanks!Solved1.8KViews1like8Commentsrun bprestore command line specify different restore path or folder
I cannot seem to figure out how to restore a folder that is getting backed up, to a different path other than the original. netbackup is making backups of /home/dir1/ this command works fine, restoring /home/dir1 to its original location. /usr/openv/netbackup/bin/bprestore -L /tmp/restore.log /home/dir1/ but, how can I specify that I want /home/dir1 tape backup to be saved to say /home/tmp/ folder? Is that possible? I know that via the GUI I can specify a different path, but i'd like to do it via command line. Thanks!Solved3.4KViews0likes3CommentsI want to know if I can certify that the coldbackup can be used as a recovery
Hi experts. I want to know if I can certify that the coldbackup can be used as a recovery 5/11/2014 14:02:19 - Error bpbrm (pid=5407) from client vzwpr-churn-01: ERR - Cannot open file /oracle/agent12c/sbin/nmb. Errno = 13: Permission denied 05/11/2014 14:02:19 - Error bpbrm (pid=5407) from client vzwpr-churn-01: ERR - Cannot open file /oracle/agent12c/sbin/nmo. Errno = 13: Permission denied 05/11/2014 14:02:19 - Error bpbrm (pid=5407) from client vzwpr-churn-01: ERR - Cannot open file /oracle/agent12c/sbin/nmhs. Errno = 13: Permission denied 05/11/2014 14:02:47 - Info bpbkar (pid=23189) 4999 entries sent to bpdbm 05/11/2014 14:02:57 - Info bptm (pid=5412) waited for full buffer 917 times, delayed 2353 times 05/11/2014 14:02:59 - Info bptm (pid=5412) EXITING with status 0 <---------- 05/11/2014 14:02:59 - Info bpbrm (pid=5407) validating image for client vzwpr-churn-01 05/11/2014 14:03:00 - Info bpbkar (pid=23189) done. status: 1: the requested operation was partially successful 05/11/2014 14:03:00 - end writing; write time: 0:00:43 the requested operation was partially successful (1)Solved1.4KViews0likes10CommentsCannot restore system state (status:2808)
Hello I had a problems with system state restore. It always end with status 2808. This is the process status. Colud someone help me? 3/13/2014 3:30:48 PM - begin Restore 3/13/2014 3:30:54 PM - restoring image SCOM-2_1394304982 3/13/2014 3:30:59 PM - requesting resource @aaaab 3/13/2014 3:30:59 PM - granted resource MediaID=@aaaab;DiskVolume=PureDiskVolume;DiskPool=dp_disk_bme02;Path=PureDiskVolume;StorageServer=bme02;MediaServer=bme02 3/13/2014 3:31:00 PM - Info bprd(pid=9272) Restoring from copy 1 of image created 03/09/14 01:56:22 from policy Netbackup-WIN-PHY 3/13/2014 3:35:46 PM - restored image SCOM-2_1394304982 - (tar did not find all the files to be restored(185)); restore time 0:04:52 3/13/2014 10:35:29 PM - Info bpbrm(pid=12759) SCOM-2 is the host to restore to 3/13/2014 10:35:29 PM - Info bpbrm(pid=12759) reading file list from client 3/13/2014 10:35:33 PM - connecting 3/13/2014 10:35:33 PM - Info bpbrm(pid=12759) starting bptm 3/13/2014 10:35:36 PM - Info tar(pid=12108) Restore started 3/13/2014 10:35:36 PM - connected; connect time: 0:00:03 3/13/2014 10:35:36 PM - Info bpbrm(pid=12759) bptm pid: 12764 3/13/2014 10:35:37 PM - Info bptm(pid=12764) start 3/13/2014 10:35:37 PM - started process bptm (12764) 3/13/2014 10:35:37 PM - Info bpdm(pid=12764) reading backup image 3/13/2014 10:35:38 PM - Info bptm(pid=12764) using 30 data buffers 3/13/2014 10:35:38 PM - Info bptm(pid=12764) spawning a child process 3/13/2014 10:35:38 PM - Info bptm(pid=12764) child pid: 12765 3/13/2014 10:35:59 PM - begin reading 3/13/2014 10:36:05 PM - Warning bpbrm(pid=12759) from client SCOM-2: WRN - System needs to be rebooted for restored object to take effect: System State:\Task Scheduler\TasksStore 3/13/2014 10:36:05 PM - Warning bpbrm(pid=12759) from client SCOM-2: WRN - System needs to be rebooted for restored object to take effect: System State:\System Files\ 3/13/2014 10:39:03 PM - Warning bpbrm(pid=12759) from client SCOM-2: WRN - error writing object: System State:\System Files\System Files 3/13/2014 10:39:03 PM - Warning bpbrm(pid=12759) from client SCOM-2: WRN - error writing byte: -826329088 3/13/2014 10:39:03 PM - Warning bpbrm(pid=12759) from client SCOM-2: WRN - desired byte count: 65536 3/13/2014 10:39:04 PM - Warning bpbrm(pid=12759) from client SCOM-2: WRN - actual byte count: 29790 3/13/2014 10:39:05 PM - Warning bpbrm(pid=12759) from client SCOM-2: WRN - System needs to be rebooted for restored object to take effect: System State:\Automated System Recovery\ 3/13/2014 10:39:06 PM - Warning bpbrm(pid=12759) from client SCOM-2: WRN - System needs to be rebooted for restored object to take effect: System State:\Automated System Recovery\BCD\BCD 3/13/2014 10:39:06 PM - Warning bpbrm(pid=12759) from client SCOM-2: WRN - System needs to be rebooted for restored object to take effect: System State:\Windows Management Instrumentation\ 3/13/2014 10:39:06 PM - Warning bpbrm(pid=12759) from client SCOM-2: WRN - System needs to be rebooted for restored object to take effect: System State:\Windows Management Instrumentation\WMI 3/13/2014 10:39:06 PM - Warning bpbrm(pid=12759) from client SCOM-2: WRN - System needs to be rebooted for restored object to take effect: System State:\Registry\ 3/13/2014 10:39:09 PM - Warning bpbrm(pid=12759) from client SCOM-2: WRN - System needs to be rebooted for restored object to take effect: System State:\Registry\Registry 3/13/2014 10:39:09 PM - Warning bpbrm(pid=12759) from client SCOM-2: WRN - System needs to be rebooted for restored object to take effect: System State:\COM+ Class Registration Database\COM+ REGDB 3/13/2014 10:39:10 PM - Warning bpbrm(pid=12759) from client SCOM-2: WRN - System needs to be rebooted for restored object to take effect: System State:\_SharedHardlinkData_\c189f0.10000.3b59 3/13/2014 10:39:10 PM - Info bpbrm(pid=12759) from client SCOM-2: TRV - last message being suppressed after 10 occurrences 3/13/2014 10:40:07 PM - Info bptm(pid=12764) waited for empty buffer 4517 times, delayed 12645 times 3/13/2014 10:40:07 PM - Error bpbrm(pid=12759) from client SCOM-2: ERR - unable to create object for restore: System State:\_SharedHardlinkData_\c189f0.10000.12749 (WIN32 -536836931: Unknown error) 3/13/2014 10:40:07 PM - Error bpbrm(pid=12759) from client SCOM-2: ERR - unable to create object for restore: System State:\_SharedHardlinkData_\c189f0.10000.12753 (WIN32 -536836931: Unknown error) 3/13/2014 10:40:07 PM - end reading; read time: 0:04:08 3/13/2014 10:40:07 PM - Info bptm(pid=12764) completed reading backup image 3/13/2014 10:40:07 PM - Info bptm(pid=12764) EXITING with status 0 <---------- 3/13/2014 10:40:07 PM - Error bpbrm(pid=12759) from client SCOM-2: ERR - unable to create object for restore: System State:\_SharedHardlinkData_\c189f0.20000.12447 (WIN32 -536836931: Unknown error) 3/13/2014 10:40:07 PM - Info tar(pid=12108) done. status 5 3/13/2014 10:40:08 PM - Info bme02(pid=12764) StorageServer=PureDisk:bme02; Report=PDDO Stats for (bme02): read: 7321429 KB, CR received: 7338479 KB, CR received over FC: 0 KB, dedup: 0.0% 3/13/2014 10:40:08 PM - Info tar(pid=12108) done. status: 185 3/13/2014 10:40:08 PM - Info tar(pid=12108) done. status: 185: tar did not find all the files to be restored 3/13/2014 10:40:08 PM - Error bpbrm(pid=12759) client restore EXIT STATUS 185: tar did not find all the files to be restored 3/13/2014 3:35:53 PM - Warning bprd(pid=9272) Restore must be resumed prior to first image expiration on 4/9/2014 1:56:22 AM 3/13/2014 10:35:29 PM - Info bpbrm(pid=12759) SCOM-2 is the host to restore to 3/13/2014 10:35:29 PM - Info bpbrm(pid=12759) reading file list from client 3/13/2014 10:35:33 PM - connecting 3/13/2014 10:35:33 PM - Info bpbrm(pid=12759) starting bptm 3/13/2014 10:35:36 PM - Info tar(pid=12108) Restore started 3/13/2014 10:35:36 PM - connected; connect time: 0:00:03 3/13/2014 10:35:36 PM - Info bpbrm(pid=12759) bptm pid: 12764 3/13/2014 10:35:37 PM - Info bptm(pid=12764) start 3/13/2014 10:35:37 PM - started process bptm (12764) 3/13/2014 10:35:37 PM - Info bpdm(pid=12764) reading backup image 3/13/2014 10:35:38 PM - Info bptm(pid=12764) using 30 data buffers 3/13/2014 10:35:38 PM - Info bptm(pid=12764) spawning a child process 3/13/2014 10:35:38 PM - Info bptm(pid=12764) child pid: 12765 3/13/2014 10:35:59 PM - begin reading 3/13/2014 10:36:05 PM - Warning bpbrm(pid=12759) from client SCOM-2: WRN - System needs to be rebooted for restored object to take effect: System State:\Task Scheduler\TasksStore 3/13/2014 10:36:05 PM - Warning bpbrm(pid=12759) from client SCOM-2: WRN - System needs to be rebooted for restored object to take effect: System State:\System Files\ 3/13/2014 10:39:03 PM - Warning bpbrm(pid=12759) from client SCOM-2: WRN - error writing object: System State:\System Files\System Files 3/13/2014 10:39:03 PM - Warning bpbrm(pid=12759) from client SCOM-2: WRN - error writing byte: -826329088 3/13/2014 10:39:03 PM - Warning bpbrm(pid=12759) from client SCOM-2: WRN - desired byte count: 65536 3/13/2014 10:39:04 PM - Warning bpbrm(pid=12759) from client SCOM-2: WRN - actual byte count: 29790 3/13/2014 10:39:05 PM - Warning bpbrm(pid=12759) from client SCOM-2: WRN - System needs to be rebooted for restored object to take effect: System State:\Automated System Recovery\ 3/13/2014 10:39:06 PM - Warning bpbrm(pid=12759) from client SCOM-2: WRN - System needs to be rebooted for restored object to take effect: System State:\Automated System Recovery\BCD\BCD 3/13/2014 10:39:06 PM - Warning bpbrm(pid=12759) from client SCOM-2: WRN - System needs to be rebooted for restored object to take effect: System State:\Windows Management Instrumentation\ 3/13/2014 10:39:06 PM - Warning bpbrm(pid=12759) from client SCOM-2: WRN - System needs to be rebooted for restored object to take effect: System State:\Windows Management Instrumentation\WMI 3/13/2014 10:39:06 PM - Warning bpbrm(pid=12759) from client SCOM-2: WRN - System needs to be rebooted for restored object to take effect: System State:\Registry\ 3/13/2014 10:39:09 PM - Warning bpbrm(pid=12759) from client SCOM-2: WRN - System needs to be rebooted for restored object to take effect: System State:\Registry\Registry 3/13/2014 10:39:09 PM - Warning bpbrm(pid=12759) from client SCOM-2: WRN - System needs to be rebooted for restored object to take effect: System State:\COM+ Class Registration Database\COM+ REGDB 3/13/2014 10:39:10 PM - Warning bpbrm(pid=12759) from client SCOM-2: WRN - System needs to be rebooted for restored object to take effect: System State:\_SharedHardlinkData_\c189f0.10000.3b59 3/13/2014 10:39:10 PM - Info bpbrm(pid=12759) from client SCOM-2: TRV - last message being suppressed after 10 occurrences 3/13/2014 10:40:07 PM - Info bptm(pid=12764) waited for empty buffer 4517 times, delayed 12645 times 3/13/2014 10:40:07 PM - Error bpbrm(pid=12759) from client SCOM-2: ERR - unable to create object for restore: System State:\_SharedHardlinkData_\c189f0.10000.12749 (WIN32 -536836931: Unknown error) 3/13/2014 10:40:07 PM - Error bpbrm(pid=12759) from client SCOM-2: ERR - unable to create object for restore: System State:\_SharedHardlinkData_\c189f0.10000.12753 (WIN32 -536836931: Unknown error) 3/13/2014 10:40:07 PM - end reading; read time: 0:04:08 3/13/2014 10:40:07 PM - Info bptm(pid=12764) completed reading backup image 3/13/2014 10:40:07 PM - Info bptm(pid=12764) EXITING with status 0 <---------- 3/13/2014 10:40:07 PM - Error bpbrm(pid=12759) from client SCOM-2: ERR - unable to create object for restore: System State:\_SharedHardlinkData_\c189f0.20000.12447 (WIN32 -536836931: Unknown error) 3/13/2014 10:40:07 PM - Info tar(pid=12108) done. status 5 3/13/2014 10:40:08 PM - Info bme02(pid=12764) StorageServer=PureDisk:bme02; Report=PDDO Stats for (bme02): read: 7321429 KB, CR received: 7338479 KB, CR received over FC: 0 KB, dedup: 0.0% 3/13/2014 10:40:08 PM - Info tar(pid=12108) done. status: 185 3/13/2014 10:40:08 PM - Info tar(pid=12108) done. status: 185: tar did not find all the files to be restored 3/13/2014 10:40:08 PM - Error bpbrm(pid=12759) client restore EXIT STATUS 185: tar did not find all the files to be restored 3/13/2014 3:35:58 PM - end Restore; elapsed time: 0:05:10 3/13/2014 10:35:29 PM - Info bpbrm(pid=12759) SCOM-2 is the host to restore to 3/13/2014 10:35:29 PM - Info bpbrm(pid=12759) reading file list from client 3/13/2014 10:35:33 PM - connecting 3/13/2014 10:35:33 PM - Info bpbrm(pid=12759) starting bptm 3/13/2014 10:35:36 PM - Info tar(pid=12108) Restore started 3/13/2014 10:35:36 PM - connected; connect time: 0:00:03 3/13/2014 10:35:36 PM - Info bpbrm(pid=12759) bptm pid: 12764 3/13/2014 10:35:37 PM - Info bptm(pid=12764) start 3/13/2014 10:35:37 PM - started process bptm (12764) 3/13/2014 10:35:37 PM - Info bpdm(pid=12764) reading backup image 3/13/2014 10:35:38 PM - Info bptm(pid=12764) using 30 data buffers 3/13/2014 10:35:38 PM - Info bptm(pid=12764) spawning a child process 3/13/2014 10:35:38 PM - Info bptm(pid=12764) child pid: 12765 3/13/2014 10:35:59 PM - begin reading 3/13/2014 10:36:05 PM - Warning bpbrm(pid=12759) from client SCOM-2: WRN - System needs to be rebooted for restored object to take effect: System State:\Task Scheduler\TasksStore 3/13/2014 10:36:05 PM - Warning bpbrm(pid=12759) from client SCOM-2: WRN - System needs to be rebooted for restored object to take effect: System State:\System Files\ 3/13/2014 10:39:03 PM - Warning bpbrm(pid=12759) from client SCOM-2: WRN - error writing object: System State:\System Files\System Files 3/13/2014 10:39:03 PM - Warning bpbrm(pid=12759) from client SCOM-2: WRN - error writing byte: -826329088 3/13/2014 10:39:03 PM - Warning bpbrm(pid=12759) from client SCOM-2: WRN - desired byte count: 65536 3/13/2014 10:39:04 PM - Warning bpbrm(pid=12759) from client SCOM-2: WRN - actual byte count: 29790 3/13/2014 10:39:05 PM - Warning bpbrm(pid=12759) from client SCOM-2: WRN - System needs to be rebooted for restored object to take effect: System State:\Automated System Recovery\ 3/13/2014 10:39:06 PM - Warning bpbrm(pid=12759) from client SCOM-2: WRN - System needs to be rebooted for restored object to take effect: System State:\Automated System Recovery\BCD\BCD 3/13/2014 10:39:06 PM - Warning bpbrm(pid=12759) from client SCOM-2: WRN - System needs to be rebooted for restored object to take effect: System State:\Windows Management Instrumentation\ 3/13/2014 10:39:06 PM - Warning bpbrm(pid=12759) from client SCOM-2: WRN - System needs to be rebooted for restored object to take effect: System State:\Windows Management Instrumentation\WMI 3/13/2014 10:39:06 PM - Warning bpbrm(pid=12759) from client SCOM-2: WRN - System needs to be rebooted for restored object to take effect: System State:\Registry\ 3/13/2014 10:39:09 PM - Warning bpbrm(pid=12759) from client SCOM-2: WRN - System needs to be rebooted for restored object to take effect: System State:\Registry\Registry 3/13/2014 10:39:09 PM - Warning bpbrm(pid=12759) from client SCOM-2: WRN - System needs to be rebooted for restored object to take effect: System State:\COM+ Class Registration Database\COM+ REGDB 3/13/2014 10:39:10 PM - Warning bpbrm(pid=12759) from client SCOM-2: WRN - System needs to be rebooted for restored object to take effect: System State:\_SharedHardlinkData_\c189f0.10000.3b59 3/13/2014 10:39:10 PM - Info bpbrm(pid=12759) from client SCOM-2: TRV - last message being suppressed after 10 occurrences 3/13/2014 10:40:07 PM - Info bptm(pid=12764) waited for empty buffer 4517 times, delayed 12645 times 3/13/2014 10:40:07 PM - Error bpbrm(pid=12759) from client SCOM-2: ERR - unable to create object for restore: System State:\_SharedHardlinkData_\c189f0.10000.12749 (WIN32 -536836931: Unknown error) 3/13/2014 10:40:07 PM - Error bpbrm(pid=12759) from client SCOM-2: ERR - unable to create object for restore: System State:\_SharedHardlinkData_\c189f0.10000.12753 (WIN32 -536836931: Unknown error) 3/13/2014 10:40:07 PM - end reading; read time: 0:04:08 3/13/2014 10:40:07 PM - Info bptm(pid=12764) completed reading backup image 3/13/2014 10:40:07 PM - Info bptm(pid=12764) EXITING with status 0 <---------- 3/13/2014 10:40:07 PM - Error bpbrm(pid=12759) from client SCOM-2: ERR - unable to create object for restore: System State:\_SharedHardlinkData_\c189f0.20000.12447 (WIN32 -536836931: Unknown error) 3/13/2014 10:40:07 PM - Info tar(pid=12108) done. status 5 3/13/2014 10:40:08 PM - Info bme02(pid=12764) StorageServer=PureDisk:bme02; Report=PDDO Stats for (bme02): read: 7321429 KB, CR received: 7338479 KB, CR received over FC: 0 KB, dedup: 0.0% 3/13/2014 10:40:08 PM - Info tar(pid=12108) done. status: 185 3/13/2014 10:40:08 PM - Info tar(pid=12108) done. status: 185: tar did not find all the files to be restored 3/13/2014 10:40:08 PM - Error bpbrm(pid=12759) client restore EXIT STATUS 185: tar did not find all the files to be restored MS-Windows policy restore error(2808)Solved1.8KViews1like1CommentMajor Data loss situations - OSX Mavericks + WD drive software + external Hard drives
Please be aware that there are serious data loss issues ongoing with Mac OSX Maverics and Western Digital drive sofware. Both the Apple support and WD forums have mutiple customer reports of total external drive data loss. The data loss is not limited to systems activly using the WD software and drives, other vendor external hard drives are being lost. Personally I just lost 1TB of Time machine backups on a Iomega external drive. I did have a WD myBook for a while about 2 years ago but was not aware the WD drive software was still active on this machine. Current advice from Western Digital is to uninstall the WD drive software before going to Mavericks. Uninstaller is here ..... http://support.wdc.com/product/download.asp?groupid=126&sid=214&lang=en This situation as already caused a lot of digital pain, loss of data and as a techincal person makes me sad in so many ways. Clive468Views2likes0Comments