Azure PowerShell: Copy Blobs Between Storage Accounts

Here’s a PowerShell script that leverages the Azure PowerShell cmdlets to copy all blobs in a source container to another storage account’s container. This script leverages the asynchronous Start-AzureStorageBlobCopy and then checks the status of the blobs being copied until they are done. It also reports the percent complete to the console.

# Whether or not to overwrite existing files
$ForceCopy = $false
 
# The source context
$SourceResourceGroupName = "MyRG1"
$SourceStorageAccountName = "storageaccount1"
$SourceStorageAccountKey = (Get-AzureRmStorageAccountKey -ResourceGroupName $SourceResourceGroupName -Name $SourceStorageAccountName)[0].Value
$SourceStorageContext = New-AzureStorageContext –StorageAccountName $SourceStorageAccountName -StorageAccountKey $SourceStorageAccountKey
 
# The destination context
$DestinationResourceGroupName = "MyRG2"
$DestinationStorageAccountName = "storageaccount2"
$DestinationStorageAccountKey = (Get-AzureRmStorageAccountKey -ResourceGroupName $DestinationResourceGroupName -Name $DestinationStorageAccountName)[0].Value
$DestinationStorageContext = New-AzureStorageContext –StorageAccountName $DestinationStorageAccountName -StorageAccountKey $DestinationStorageAccountKey
 
# The container we are copying
$SourceContainerName = "mycontainer"
$DestinationStorageContainerName = $SourceContainerName
 
# Create the destination container if it doesn't exist
$DestinatonExists = (Get-AzureStorageContainer -Name $DestinationStorageContainerName -Context $DestinationStorageContext -ErrorAction SilentlyContinue)
If (-Not $DestinatonExists)
{
  # Note the Permission Off
  New-AzureStorageContainer -Name $DestinationStorageContainerName -Context $DestinationStorageContext -Permission Off
}
 
# Copy all blobs from the source to the destination, excluding those that exist, unless the $ForceCopy flag is set
$Blobs = Get-AzureStorageBlob -Context $SourceStorageContext -Container $SourceContainerName
$BlobCpyAry = @()
ForEach ($Blob in $Blobs)
{
  $Msg = ""
  $ExistingBlob = $null
 
  If ($ForceCopy -eq $false) 
  {
    $ExistingBlob = Get-AzureStorageBlob -Context $DestinationStorageContext -Blob $Blob.Name -Container $DestinationStorageContainerName -ErrorAction SilentlyContinue
  }
 
  If ($ExistingBlob -eq $null)
  {
    $BlobCopy = Start-AzureStorageBlobCopy -Context $SourceStorageContext -SrcContainer $SourceContainerName -SrcBlob $Blob.Name -DestContext $DestinationStorageContext -DestContainer $DestinationStorageContainerName -DestBlob $Blob.Name -Force
    $BlobCpyAry += $BlobCopy
    $Msg = "Will copy " + $Blob.Name
  } 
  Else
  {
    $Msg = "File exists - Not copying " + $Blob.Name 
  }   
 
  Write-Output $Msg 
}
 
#Check Status until all copying activities are complete...
$HasPending = $true
while ($HasPending -eq $true)
{
  $HasPending = $false
 
  foreach ($BlobCopy in $BlobCpyAry)
  { 
    $CopyState = $BlobCopy | Get-AzureStorageBlobCopyState
    if ($CopyState.Status -eq "Pending") 
    {
      $Msg = $CopyState.Source.AbsolutePath + " " + $CopyState.Status + " {0:N2}%" -f (($CopyState.BytesCopied/$CopyState.TotalBytes)*100) 
      Write-Output $Msg
      $HasPending = $true
    }
  }
 
  if ($HasPending -eq $true) 
  {
    Write-Host "Sleeping for 15s..."
    Start-Sleep -s 15
  }
}
 
Write-Host "Done copying!"

There are considerations to be made when copying blobs en masse. See the Storage Performance Checklist for some of those considerations. In particular, the checklist notes that “when copying between storage accounts, there is no time guarantee on when the copy will complete. If your application needs to complete a blob copy quickly under your control, it may be better to copy the blob by downloading it to a VM and then uploading it to the destination.”

Hope this helps!

This entry was posted in Azure, PowerShell. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *