Streamlining Data Management with PowerShell

This is the 8th (and the last) post from the Data Manipulations with PowerShell series, where we are diving into the specific phase of data operations processes and apply/use PowerShell along the way.

Previous posts:

1. Empower your Data Retrieval with PowerShell
2. Make your Data Processing more efficient with PowerShell
3. Data Transformation Made Easy with PowerShell – Part 1, Part 2
4. PowerShell for Data Analysis: A Practical Guide to Extracting Insights from Your Data
5. Making Your Data Come Alive: A Practical Guide to Data Visualization with PowerShell
6. Data Export with PowerShell: Best Practices and Advanced Techniques

In the context of Data Manipulations with PowerShell, Data management is a holistic/horizontal phase that should be kept on the backburner all the time. In this post we look into data management activities such as backup and restore operations, data migration, and data archiving and how PowerShell can help us here.

Out of the Box

Out of the box, we are presented with a few options that can help us to cover majority of our use cases. Cmdlets that might help us are: Get-ChildItem, Copy-Item & Move-Item to move & and copy the data respectively. Get-Acl & Set-Acl to control access permissions to the data.

Data listing, copying and moving cmdlets are shipped in Microsoft.PowerShell.Management module.

Get-ChildItem – gets the items in one or more specified locations. If the item is a container, it gets the items inside the container, known as child items.

Copy-Item – copies an item from one location to another location in the same namespace.

Move-Item – moves an item, including its properties, contents, and child items, from one location to another location.

Example #1: Migrate data

Copy-Item -Path C:\Code\Source\ -Destination C:\Code\Dest\ -Recurse

Example #2: Archive data based on timestamp

$Source = "C:\Code\Source"
$Archive = "E:\Archive"
$DaysToArchive = 180

$FilesToArchive = Get-ChildItem -Path $Source | Where-Object { $_.LastAccessTime -lt (Get-Date).AddDays(-$DaysToArchive) }

$FilesToArchive | ForEach-Object {Move-Item -Path $_.FullName -Destination $Archive}

Example #3: Grant data access permissions

$Data = "C:\Code\Source"
$UserAccount = "strwrs\lskywalker"
$Permission = "FullControl"

$Acl = Get-Acl $Data
$AccessRule = New-Object System.Security.AccessControl.FileSystemAccessRule($UserAccount, $Permission, "ContainerInherit,ObjectInherit", "None", "Allow")
$Acl.AddAccessRule($AccessRule)
Set-Acl -Path $Data -AclObject $Acl

Community Provided

Modules with *data migration*, *data backup*, *data archive* – 2300. Around 30 of them can be applied to our use case. I would like to highlight few of them:

• ArchivialPowerShell by Keith Babinec – data backup software for Windows that automatically archives your local data to a cloud storage provider.
• PowerShell.FileTransformations by HT – transforms, converts, and/or secures data files.
• Poor.Backup.System by LCU (aka MS-LUF) – simple powershell module to backup file or folder using Powershell Core.
• PSBackup by Thomas Barratt (aka thbiv) – running backup jobs described in PSD1 files.
• cdisk by Cristian – wipe all data on a drive.
• DataWiper by c.dek. – used for data cleanup, and data scrambling for confidential files.
• CAT by Joyful Craftsmen CAT Team – Framework for creating automated tests for your DATA.
  Super easy to start with, you can create your first test in 5 minutes.
• EncryptionShell by Pietro Ciaccio – simple collection of cmdlets for encrypting and decrypting data.
• PSFSTools by Matteo Guadrini – various task on a file server, require execution complexity; how and when to archive, create folders assigned to certain groups or delete files older than one month or delete folders that are no longer used.
• 7Zip4Powershell by Thomas Freudenberg (aka thoemmi) – creating and extracting 7-Zip archives.

Summary

I’ll be honest, PowerShell is not the best tool to cover these activities BUT, in our case, we assume it is the only tool at our disposal. With the artificial limitation that we apply on ourselves, we have to be creative enough and come up with the ways to do the task at hand. And as I’ve mentioned multiple times throughout this series, PowerShell’s out of the box building blocks and grain of productivity can get us far along in our journey.

Data icons created by Freepik – Flaticon.

Thanks a lot for reading.