cancel
Showing results for 
Search instead for 
Did you mean: 

Suddenly unable to write to AWS S3 Mar 28-30, 2020

Dale_Mahalko_LH
Level 4

I have been using Backup Exec 20 and AWS S3 IA for about two years. All of a sudden on March 28-30, 2020, I am experiencing problems.

I have several jobs failing with a "Robotic Library" error, which is meaningless as there is no robotic library, error V-275-550.

,

Also I see:

Storage device "Amazon S3:31" reported an error on a request to write data to media.

0x17   V-79-57344-34028 - backup storage read/write error occurred.

,

I see there is an upgrade available for Backup Exec 20.6. After applying it, I am still having problems.

I checked the Amazon account and I don't see any errors or warnings.

Does anyone know what log files I should be checking to find the real cause of whatever is going on here?

 

5 REPLIES 5

Gurvinder
Moderator
Moderator
Employee Accredited Certified

can you run a test with cloud connect optimizer and check on the number of writes that are optimal for a single job to run. The error looks to be due to drop in connection. If you want to debug, use SGMON.exe --> select bengine, device and media ( from settings , select verbose for device and mdia), beserver. Launch it before the job. The debug is very verbose so turn it off after this run. You can search with 'ERR' and review that part/time of the job log where the error is seen. When multiple resources are backed up then it may be that backup of some resource failed and even though the error at the end is what you see at the bottom of the job log but inorder for you to look when the error occurred, will need to expand the job log and review that section when the error occurred and then check in SGMON log, that part.

 

It seems to work for a while, but then it stops working. A short run will succeed followed by a failure until I restart the Windows server, and then it works for a short time again and fails.

I see there are a bunch of errors in Logs\ADAMM.log but many of these error numbers can't be found.

Some example blobs:

[07064] 03/31/20 12:23:13.471 LSU Stats: 4194303GB Capacity, 0GB Reserved, 0GB Used, 0GB LogicalUsed, 1.0:1 DedupeRatio - S3:google:google.com:OurSite-gcloud
[05136] 03/31/20 12:23:30.755 LSU Stats: 4194303GB Capacity, 0GB Reserved, 0GB Used, 0GB LogicalUsed, 1.0:1 DedupeRatio - S3:amazon:amazon-ia.com:Our-Site-backup
[01444] 03/31/20 15:38:37.936 DeviceIo: ostaspi: sts_get_lsu_prop_byname S3:amazon:amazon-ia.com Our-Site-backup error 2060019
[01444] 03/31/20 15:40:37.666 DeviceIo: ostaspi: sts_get_lsu_prop_byname S3:amazon:amazon-ia.com Our-Site-backup error 2060019
[01444] 03/31/20 15:42:37.558 DeviceIo: ostaspi: sts_get_lsu_prop_byname S3:amazon:amazon-ia.com Our-Site-backup error 2060019
[01444] 03/31/20 15:44:36.728 DeviceIo: sts_delete_image BEOST_SDrv29 BEOST_SDrv 2060017
[01444] 03/31/20 15:44:36.728 DeviceIo: ostutil: DeleteOstImage, delete_image BEOST_SDrv29 failed, rc=2060017
[01444] 03/31/20 15:46:36.743 DeviceIo: ostaspi: sts_get_lsu_prop_byname S3:amazon:amazon-ia.com Our-Site-backup error 2060019
[01444] 03/31/20 15:46:36.743 DeviceIo: sts_delete_image BEOST_SMap BEOST_SMap 2060001
[01444] 03/31/20 15:46:36.743 DeviceIo: ostutil: DeleteOstImage, delete_image BEOST_SMap failed, rc=2060001
[01444] 03/31/20 15:46:36.743 DeviceIo: sts_create_image BEOST_SMap BEOST_SMap 2060001
[01444] 03/31/20 15:46:36.743 DeviceIo: ostutil: S3:amazon:amazon-ia.com !!!SlotMap Our-Site-backup:BEOST_SMap create failed 2060001
[01444] 03/31/20 15:48:36.405 DeviceIo: ostaspi: sts_get_lsu_prop_byname S3:amazon:amazon-ia.com Our-Site-backup error 2060019
[01444] 03/31/20 15:49:09.320 Reclaim incomplete OST image OST00000488-4B4A533AECF101BB from S3 Increm SPRV-Full
[01444] 03/31/20 15:49:25.553 PvlDrive::FindDismountSlot() - OST media placed in available slot 487, Changer Id = 1034 Drive "OurSite Amazon S3:29"

,

,

[05244] 03/31/20 16:17:09.280 DeviceIo: beost: Critical:
[05244] 03/31/20 16:17:09.280 DeviceIo: sts_open_server S3:amazon:amazon-ia.com AMAZON-S3-LOGIN-NAME 2060205
[05244] 03/31/20 16:17:09.280 DeviceIo: ostaspi: sts_open_server S3:amazon:amazon-ia.com as AMAZON-S3-LOGIN-NAME error 2060205
[05244] 03/31/20 16:17:09.280 DeviceIo: ostaspi: Not able to connect to server S3:amazon:amazon-ia.com

 

[05244] 03/31/20 16:21:07.644 DeviceIo: beost: Critical:
[05244] 03/31/20 16:21:07.644 DeviceIo: sts_open_server S3:amazon:amazon-ia.com AMAZON-S3-LOGIN-NAME 2060205
[05244] 03/31/20 16:21:07.644 DeviceIo: ostaspi: sts_open_server S3:amazon:amazon-ia.com as AMAZON-S3-LOGIN-NAME error 2060205
[05244] 03/31/20 16:21:07.644 DeviceIo: ostaspi: Not able to connect to server S3:amazon:amazon-ia.com
[05244] 03/31/20 16:23:06.819 DeviceIo: sts_delete_image BEOST_SDrv29 BEOST_SDrv 2060017
[05244] 03/31/20 16:23:06.819 DeviceIo: ostutil: DeleteOstImage, delete_image BEOST_SDrv29 failed, rc=2060017

,

,

[05244] 03/31/20 16:25:05.993 DeviceIo: beost: Critical:
[05244] 03/31/20 16:25:05.993 DeviceIo: sts_open_server S3:amazon:amazon-ia.com AMAZON-S3-LOGIN-NAME 2060205
[05244] 03/31/20 16:25:05.993 DeviceIo: ostaspi: sts_open_server S3:amazon:amazon-ia.com as AMAZON-S3-LOGIN-NAME error 2060205
[05244] 03/31/20 16:25:05.993 DeviceIo: ostaspi: Not able to connect to server S3:amazon:amazon-ia.com
[05244] 03/31/20 16:25:05.993 DeviceIo: sts_delete_image BEOST_SMap BEOST_SMap 2060001
[05244] 03/31/20 16:25:05.993 DeviceIo: ostutil: DeleteOstImage, delete_image BEOST_SMap failed, rc=2060001
[05244] 03/31/20 16:25:05.993 DeviceIo: sts_create_image BEOST_SMap BEOST_SMap 2060001
[05244] 03/31/20 16:25:05.993 DeviceIo: ostutil: S3:amazon:amazon-ia.com !!!SlotMap Our-Site-backup:BEOST_SMap create failed 2060001

,

,

[05548] 03/31/20 18:02:11.115 DeviceIo: beost: Critical:
[05548] 03/31/20 18:02:11.115 DeviceIo: sts_open_server S3:amazon:amazon-ia.com AMAZON-S3-LOGIN-NAME 2060205
[05548] 03/31/20 18:02:11.115 DeviceIo: ostaspi: sts_open_server S3:amazon:amazon-ia.com as AMAZON-S3-LOGIN-NAME error 2060205
[05548] 03/31/20 18:02:11.115 DeviceIo: ostaspi: Not able to connect to server S3:amazon:amazon-ia.com
[05548] 03/31/20 18:02:11.617 PvlDrive::DisableAccess() - ReserveDevice failed
Drive = 1062 "OurSite Amazon S3:28"
ERROR = 0x0000045D (ERROR_IO_DEVICE) -> ERROR = 0xE000813C (3758129468)

 

Meanwhile on the backup server, I have a testing tool installed called S3 Browser by NetSDK Software.

I can use it to logon to my Amazon S3 account using the same credentials I am using with Backup Exec, and it is able to open and browse the files and folders without any problems.

What is going on? I need backups to work, and not act stupid like this.

It has been running fine until suddenly this week it is screwing up. I have not changed anything.

 

 

 

Gurvinder
Moderator
Moderator
Employee Accredited Certified

can you check on the internet connection. These issues are mostlty related to drop in the connection. S3 browser is to browse and upload download, backups are written by BE in a different format. Also, run the cloud connect optimizer and check the results. Try to set that as the number of write and run backup. See how it goes.