Backup Exec Azure Archive and Glacier Integration

New Features for Backup Exec

  1. Azure Archive Storage Functionality
  2. S3 tier to Glacier.

Downloads :

Sdfs 3.6.0.12 :

http://www.opendedup.org/downloads/SDFS-3.6.0.12-Setup.exe

Supported OST version 2.2.7 :

http://www.opendedup.org/downloads/windows-ost-2.2.7.zip

Supported Storage Types

  • AWS S3
  • AWS S3 with Glacier
  • Google Cloud Storage
  • Azure Blob
  • Azure Blob Cool Tier
  • Azure Blob Archive Tier

Azure Archive Storage

SDFS now supports tiering to Azure Archive and Cold storage as part of a data lifecycle policy.

The data lifecycle policy is built into SDFS and keeps track of recently read and written data and ensures that this data does not get tiered off to archive storage. SDFS accomplishes this by utilizing two components:

  1. A refresh handler within SDFS that notifies the SDFS azure subsystem when a block of data has been read or claimed by another backup/file
  2. An azure blob table, Azure Table Service, per bucket that keeps state of all data that has not been moved to archive storage and the timestamp when it was last accessed. The table name is the bucket name with “table” appended to it.

When data is written or read in SDFS, it notifies the refresh handler. The refresh handler then periodically, every minute, updates the azure blob table associated with the bucket with a new timestamp associated with the block accessed.

The azure refresh handler then checks for archives tracked in the Azure blob table for timestamps that are less than the azure-tier-in-days xml parameter and moves that data to the desired tier.

Data is restored from azure archive on data reads. This process is triggered when it is determined that block required for a backup restore is located on archive storage. The SDFS Azure subsystem then notifies the filesystem that the data is archived. The filesystem then identifies all the blocks required to restore the entire backup and requests the SDFS azure subsystem restore the blocks.

The SDFS Filesystem then waits for all blocks to be restored before continuing with the read. This might take 4-14 hours (https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-storage-tiers).

Requirements:

  • The Azure account used has to have the ability to create a Table storage bucket
  • Azure Archive Storage Requires a storage account that is using GPv2. According to Microsoft documentation:

You may only tier your object storage data to hot, cool, or archive in Blob Storage or General Purpose v2 (GPv2) accounts. General Purpose v1 (GPv1) accounts do not support tiering. However, customers can easily convert their existing GPv1 or Blob Storage accounts to GPv2 accounts through a simple one-click process in the Azure portal. GPv2 provides a new pricing structure for blobs, files, and queues, and access to a variety of other new storage features as well. Furthermore, going forward some new features and prices cuts will only be offered in GPv2 accounts. Therefore, customers should evaluate using GPv2 accounts but only use them after reviewing the pricing for all services as some workloads can be more expensive on GPv2 than GPv1. See Azure storage account options to learn more.

Setup:

To enable azure archive storage run the mksdfs command line with “–azurearchive-in-days=<number of days>”

To verify the setup the xml config should look something like this in the extended-config tag:

<extended-config allow-sync=”false” azure-tier-in-days=”30″ block-size=”30 MB” delete-unclaimed=”true” io-threads=”16″ local-cache-size=”10 GB” map-cache-size=”100″ read-speed=”0″ service-type=”azureblob” storage-tier=”archive” sync-check-schedule=”4 59 23 * * ?” sync-files=”true” upload-thread-sleep-time=”10000″ user-agent-prefix=”APN/1.0 Veritas/1.0 BackupExec/” write-speed=”0″/>

The possible options for storage tier are archive and cool.

Upgrading and existing volume to Azure Archive

It is possible to upgrade and existing volume to Azure Archive. Data that is already in the volume will not be moved but new data will be subject to Azure Archive Policy.  Use the following steps to set this up:

  1. Stop BackupExec Processes
  2. Shutdown all sdfs volumes

sdfscli –shutdown –nossl –port <port number> –password <password>

E.g.

sdfscli –shutdown –nossl –port 6442 –password password123

  1. Edit the XML configuration for the volume that required for the setup. These xml configurations are in the sdfs programs directory under “etc”.
  2. Add the following options to the xml config. In the example below data will be moved to Azure Archive after 30 days.

<extended-config allow-sync=”false” azure-tier-in-days=”30″ block-size=”30 MB” delete-unclaimed=”true” io-threads=”16″ local-cache-size=”10 GB” map-cache-size=”100″ read-speed=”0″ service-type=”azureblob” storage-tier=”archive” sync-check-schedule=”4 59 23 * * ?” sync-files=”true” upload-thread-sleep-time=”10000″ user-agent-prefix=”APN/1.0 Veritas/1.0 BackupExec/” write-speed=”0″/>

  1. Restart BackupExec.

FOR TESTING USE THE FOLLOWING CONFIG

To test this feature more quickly you can add the parameter tier-immediately to move data more quickly to the cloud. If tier-immediately is set to true, azure-tier-in-days will tier in minutes based on the value. In the example below azure will tier in 15 minutes.

<extended-config allow-sync=”false” azure-tier-in-days=”15″ block-size=”30 MB” delete-unclaimed=”true” io-threads=”16″ local-cache-size=”10 GB” map-cache-size=”100″ read-speed=”0″ service-type=”azureblob” storage-tier=”archive” sync-check-schedule=”4 59 23 * * ?” sync-files=”true” tier-immediately=”true” upload-thread-sleep-time=”10000″ user-agent-prefix=”APN/1.0 Veritas/1.0 BackupExec/” write-speed=”0″/>

The image below is what your bucket should look like when data is tiered to archive storage.

This is what it should look like in a blob properties when data is being retrieved from azure archive:

Here is the report of a backup with verify when the data has been moved to Azure archive. It took 9 hours to read the verify

AWS Glacier Storage

 

SDFS supports tiering to Glacier storage as part of a data lifecycle policy.

The data lifecycle policy is managed by AWS through s3 lifecycle policies.  SDFS builds on top of this and keeps track of recently read and written data and ensures that this data does not get tiered off to glacier storage. SDFS accomplishes this by utilizing two components A refresh handler within sdfs that notifies the sdfs s3 subsystem when a block of data has been read or claimed by another backup/file.

When data is written or read in sdfs, it notifies the refresh handler. The refresh handler then periodically, every minute, updates metadata of the s3 block object in question to update its timestamp, thus ensuring the S3 storage policy does not move data to glacier..

Data is restored from Glacier on data reads. This process is triggered when it is determined that block required for a backup restore is located on glacier storage. This typically happens on a read. The SDFS aws subsystem then notifies the filesystem that the data is archived. The filesystem then identifies all the blocks required to restore the entire backup and requests the SDFS azure subsystem restore the blocks.

The SDFS Filesystem then waits for all blocks to be restored before continuing with the read. This might take 4 hours

Setup:

To enable glacier storage run the mksdfs command line with “–glacier-in-days=<number of days>”

To verify the setup the xml config should look something like this in the extended-config tag:

<extended-config allow-sync=”false” block-size=”30 MB”  delete-unclaimed=”true” glacier-archive-days=”30″ io-threads=”16″ local-cache-size=”10 GB” map-cache-size=”200″ read-speed=”0″ refresh-blobs=”true” simple-metadata=”false” simple-s3=”false” sync-check-schedule=”4 59 23 * * ?” sync-files=”true” upload-thread-sleep-time=”10000″ use-basic-signer=”false” write-speed=”0″/>

To restore Data from Glacier using no standard Tier use the parameter glacier-tier=”Expedited”

Possible options for glacier tiers are:

  • Expedited – Allows you to quickly access your data when occasional urgent requests for a subset of archives are required. For all but the largest archived object (250 MB+), data accessed using Expedited retrievals are typically made available within 1–5 minutes.
  • Standard – Allows you to access any of your archived objects within several hours. Standard retrievals typically finish within 3–5 hours. This is the default tier.
  • Bulk – The lowest-cost data access option in Amazon Glacier, enabling you to retrieve large amounts, even petabytes, of data inexpensively in a day. Bulk access typically completes within 5–12 hours.

(from https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPOSTrestore.html)

Below is what the S3 Storage Lifecycle Policy looks like

Upgrading and existing volume to Glacier

It is possible to upgrade and existing volume to glacier using the following steps:

  1. Stop BackupExec Processes
  2. Shutdown all sdfs volumes

sdfscli –shutdown –nossl –port <port number> –password <password>

E.g.

sdfscli –shutdown –nossl –port 6442 –password password123

  1. Edit the XML configuration for the volume that required for the setup. These xml configurations are in the sdfs programs directory under “etc”.
  2. Add the following options to the xml config. In the example below data will be moved to glacier after 30 days.

<extended-config allow-sync=”false” block-size=”30 MB” delete-unclaimed=”true” glacier-archive-days=”30″ io-threads=”16″ local-cache-size=”10 GB” map-cache-size=”200″ read-speed=”0″ refresh-blobs=”true” simple-metadata=”false” simple-s3=”false” sync-check-schedule=”4 59 23 * * ?” sync-files=”true” upload-thread-sleep-time=”10000″ use-basic-signer=”false” user-agent-prefix=”APN/1.0 Veritas/1.0 BackupExec/” write-speed=”0″/>

  1. Restart BackupExec.

FOR TESTING USE THE FOLLOWING CONFIG

To test this feature more quickly you can add the parameter tier-immediately to move data more quickly to the cloud. With glacier-zero-day, data will be a candidate for movement to cloud on the next UTC day as long as glacier-archive-days=”0” as well.

<extended-config allow-sync=”false” block-size=”30 MB” delete-unclaimed=”true” glacier-archive-days=”0″ glacier-tier=”expedited” glacier-zero-day=”true” io-threads=”16″ local-cache-size=”10 GB” map-cache-size=”200″ read-speed=”0″ refresh-blobs=”true” simple-metadata=”false” simple-s3=”false” sync-check-schedule=”4 59 23 * * ?” sync-files=”true” upload-thread-sleep-time=”10000″ use-basic-signer=”false” user-agent-prefix=”APN/1.0 Veritas/1.0 BackupExec/” write-speed=”0″/>

Performing a Full Recovery of Data From the Cloud

All data in the cloud bucket can be recovered if the Backup Exec System is rebuilt or can be recovered on a second node if the primary BE Server is down permanently.

To perform a complete recovery you will need the following:

  1. A windows server of similiar size and cpu
  2. The System need to have the same physical storage layout used by SDFS. As an example, if the SDFS metadata was cached to E:\ on the old system, the new system should also have an E:\ drive
  3. A backup of the xml config of the volume and the ostconfig.xml found in the etc sub directory.

Steps to perform full recovery.

  1. Make sure the system is completely unmounted and removed on the primary node
  2. Install sdfs and copy the ostconfig.xml and the volume-cfg.xml to the “etc” directory in the program files\sdfs directory
  3. Edit the sdfs config set the following in volume-cfg.xml for the <volume> tag attributes:
    1. sync-files=”true”
    2. rebuild-hashtable=”true”
  4. Mount the volume and wait for all the volume to mount completely
  5. Create a storage device in BE.

KNOWN ISSUES

Local System Time Must Match Internet Time

Cloud Storage Providers require times to be in sync for authentication. It is advisable to have an internet time service enable on the BE server for connecting to cloud storage.

Data archived before backup completes can cause slow verifies

If the backup does not complete in the time before data is is moved to Azure archive or Glacier then the verify will take a significant amount of time. This is because the image will have been partially moved to the archive so a restore will be required to read data.

To work around this issue make sure the archive time is set to something larger than the time it takes to do a backup and verify. As an example, if a backup and verify takes 40 Minutes use the following parameters in your Azure Config

<extended-config allow-sync=”false” azure-tier-in-days=”60″ block-size=”30 MB” delete-unclaimed=”true” io-threads=”16″ local-cache-size=”10 GB” map-cache-size=”100″ read-speed=”0″ service-type=”azureblob” storage-tier=”archive” sync-check-schedule=”4 59 23 * * ?” sync-files=”true” tier-immediately=”true” upload-thread-sleep-time=”10000″ user-agent-prefix=”APN/1.0 Veritas/1.0 BackupExec/” write-speed=”0″/>

Data Archive restores pause other jobs on other storage units in progress

Based on testing, Backup Exec seems to pause all jobs while Archive restores are in progress. It has not been verified if this is the case in all setups.

Glacier and Azure Archive Jobs that span media take significantly longer to restore

BE beaks up images into 50GB OST files by default. When a read to an OST that has been archived to Azure or Glacier, SDFS restored all chunks for that specific file. If the restore spans multiple OST files each file will be restored in sequence. This means that restores can take a 2x if a BE restore spans 2 ost files or 3x if it spans 3. To address this issue, set the fragment size to something larger than the largest backup so all data will be restored at once.

Azure Archive and Glacier LSU’s that are offline for some time can take a long time to bring online and might crash BE

BE requires BEOST_SInf.img and BEOST_SMap.img to be available for reading when a storage unit comes online. If the data that backs these files is moved to glacier then these files will take a long time to read or BE will time out and fail to read the image. It this occurs a workaround is to stop BE, make sure all drives are umounts. Then, mount the volume manually  with glacier-tier=”expedited”  using :

mountsdfs -v <volume-name> -m <drive letter> -e <volumepasswor>

E.g.

mountsdfs -v pool0 -m z -e password123

Once the volume is mounted attempt to copy BEOST_SInf.img and BEOST_SMap.img a folder your “c:\” drive to kick or a glacier restore event. Once its copied, unmount the volume by initiating :

sdfscli –shutdown –nossl –port <port number> –password <password>

E.g.

sdfscli –shutdown –nossl –port 6442 –password password123

Once unmounted start BE services.

Logging Diagnostics for Azure Archive and Glacier:

When data is being restored from archive storage you will see the following messages in the log. The messages below state that the archives blocks/MjkyOTI0MDcwNjE4MDY3Ng== and the like are being restored from archive storage.

2018-01-21 15:53:56,422 [sdfs] [org.opendedup.mtools.RestoreArchive] [100] [Thread-66]  - will restore MjkyOTI0MDcwNjE4MDY3Ng==
2018-01-21 15:53:56,527 [sdfs] [org.opendedup.mtools.RestoreArchive] [100] [Thread-66]  - will restore MzYyOTAzMzkxMjUwMDQ1ODY=
2018-01-21 15:53:56,850 [sdfs] [org.opendedup.mtools.RestoreArchive] [100] [Thread-66]  - will restore MjkxNjAxODcyMTcyMzQ2NjU=
2018-01-21 15:53:56,964 [sdfs] [org.opendedup.mtools.RestoreArchive] [100] [Thread-66]  - will restore MzExMDc2NjU2MDIwOTU1NjI=
2018-01-21 15:53:57,031 [sdfs] [org.opendedup.mtools.RestoreArchive] [100] [Thread-66]  - will restore NDg2MTg3ODMwMDQ0MDE2ODI=

The following messages will show up if a blob is being ignored for restore because its already either being restored or is not archived

2018-01-21 15:54:55,408 [sdfs] [org.opendedup.mtools.RestoreArchive] [129] [Thread-66]  - archive restore will check MjM2NzE5MTkwNzQwNjIzMTY=
2018-01-21 15:54:55,421 [sdfs] [org.opendedup.mtools.RestoreArchive] [137] [Thread-66]  - not restored MjM2NzE5MTkwNzQwNjIzMTY=
2018-01-21 15:54:55,421 [sdfs] [org.opendedup.mtools.RestoreArchive] [129] [Thread-66]  - archive restore will check MjA2Nzc5OTkxOTQyMDIzNjA=
2018-01-21 15:54:55,433 [sdfs] [org.opendedup.mtools.RestoreArchive] [137] [Thread-66]  - not restored MjA2Nzc5OTkxOTQyMDIzNjA=
2018-01-21 15:54:55,433 [sdfs] [org.opendedup.mtools.RestoreArchive] [129] [Thread-66]  - archive restore will check MzQ0MTg2MjExNjYwODc2NTU=

When an archive restore task is completed for a backup image the following will show up in the logs

2018-01-21 15:54:55,433 [sdfs] [org.opendedup.mtools.RestoreArchive] [129] [Thread-66]  - took [500] Minutes to import [6314]