My AWS Certification Journey begins

It’s been a long time but I have decided to explore getting a tech certification.  I believe the last time I took a certification course or exam was nearly 12 years ago when I joined a previous employer, NetApp.  Either way, my journey began this past Friday when I decided to jump in and challenge myself again.  Plus, I am attending AWS Re:Invent in a months time and figured what better way to make the best use of my time leading up to the event.  As I go through my journey I will try to document my work and what I am doing to prepare.  I will likely use my blog as a means to write up my notes.  There is no better way to reinforce what you learn than by writing about it.  Being in Technical Marketing for a number of years now, that is one thing I have learned.  If you need to come up to speed on anything, you can really ramp up your retention of topics by writing about and presenting to others the details of that topic.

The first thing I did was sign up for a free AWS account to use as a place of practice.  For a vast majority of the course I found, certification candidates won’t come close to exceeding the free tier.  At least that is my view anyway.  Even so, I did set a notification when my usage would begin to charge me, so I can plan accordingly.  From there I picked out an online course to help me go through the process.  The site I chose was acloud.guru.  For a fairly low fee of $80, I purchased a (4) course exam prep for the entry level Associate Certification.  I know what you are thinking, that seems quite low and too good to be true.  So far I have gone through a few of the modules and the moderator makes it clear that you have to work outside of the course to prepare.

For $80 I figured, if it works out great, if not I haven’t lost much other than my investment of time.  Which lets be honest, it won’t be a waste of time because I will have learned something new and not spent that equivalent amount of time scrolling through useless social media posts.

**Sidebar…speaking of social media and time wasters.  If you are the type of person who tells yourself you don’t have time (and I definitely say that), I challenge you to look deeper into how you are spending your time.  If we have time to read through our friends humble brags / rants on Facetwinsnaptachatgram and respond with our own humble brags and rants, then we have time to do things like challenge yourself to learn something new.  Something that will actually provide real value in your life.

Back to our regularly scheduled programming…What this means is that you are working through the course, following along with the labs using your own AWS account and at the same time they provide you reference documents to review as well.  Reference docs that are available on AWS.  Docs like FAQs, White Papers, etc.

So off I go into the world of AWS.  I almost forgot to mention the exam that I plan to prep for.   My quest is to pass the Solutions Architect Associate exam at Re:Invent.  As I progress through the course, I will post notes in my blog.

Have a great day!

 

Nutanix Powershell – List VMs…list unprotected VMs and protect VMs by adding to Protection Domain

Recently I came across an inquiry that asked how to protect VMs running on a Nutanix cluster by adding them to a protection domain.  I was intrigued on how to do this via a script.  So I set about on a quest to build a menu driven Powershell script, which if you have seen some of my previous scripts when I worked for another Tech company, these are scripts I like to write.  Here is what I came up with:

First, Pre-Requisites for my example script are below:

  • Nutanix Powershell cmdlets installed – If you do not currently have them installed it is very easy to install them onto your Windows host.  Here are the 10K foot steps:
    • Login to your Nutanix cluster and click on the login icon in the upper right corner of the browser (the one with the username)
    • Select “Download Cmdlets Installer”
    • Run the installer from the host you would like the Nutanix Powershell cmdlets to be available on
  •  Nutanix Protection Domain created in advance of testing the script (not outlined in this post, but it’s very easy to do)
  •  Test VMs you can work with
  •  Additional cmdlet installed that will pull a stored credential from the Windows Credential Manager (see an older post I did on how to set this up located HERE , go to the section “Lets Get Started…Storing Credentials”)

The script that I wrote can be expanded upon or changed to suit your needs.  So please feel free to change or modify whatever you want.  Lets jump in:

Start out by creating a stored credential in the Windows Credential Manager.  This will be used later in the script and will keep you from having to store passwords in scripts or having to be concerned with typing one in each time it’s run.  The input details for the stored credential should be the username and password of the Nutanix cluster (i.e.: Prism)  you will be working with as part of this script.  To get to Credential Manager, just click on the Windows icon on your Windows 10 desktop and type “Credential”, then select “Credential Manager”.  Click on “Windows Credentials” and then add a new Windows credential.

The first part of the script just confirms that you have specified a Windows Credential Manager entry.  From this, the script will pull the credential username and secured password from the Credential Manager store.  Absent the added variable, the script will stop and advised you to add in a credential entry:

# Script command to use credential manager to auto login the intended user for the script.
# Relies on the ability for the user to have a local entry in their credential manager and
# the use of an add-in module that pulls from the credential store the login details. This
# keeps from having to manually add credentials or store them in a file locally.
#
#
param(
[string]$wincredstor
)

if([string]::IsNullOrEmpty($wincredstor))
{
write-host “”
write-host “You must supply Windows credential manager target URI, for example:” -ForegroundColor Green
write-host “”
write-host “PS C:\data\scripts> .\ntnxvmmgt.ps1 marctmesvm” -ForegroundColor Red
write-host “”
write-host “Where marctmesvm is an entry created in the local Windows clients” -ForegroundColor Green
write-host “Windows Credential Store.” -foregroundcolor green
write-host “”
write-host “For more details on Windows Credential Manager see the following:” -ForegroundColor Green
write-host “http://windows.microsoft.com/en-us/windows7/what-is-credential-manager” -ForegroundColor Green
write-host “”
return
}

Set-ExecutionPolicy bypass

$ntnxuser = get-storedcredential $wincredstor | select -expand UserName
$ntnxpwd = get-storedcredential $wincredstor | select -expand Password

From there the script will ensure the Nutanix Powershell cmdlets are loaded and available.  This will ensure that you don’t have to modify any Windows associated Powershell variables to ensure the cmdlets are available.

#
# Load the Nutanix Snap-ins to ensure cmdlets are available
$ntnxpsLoaded = Get-PSSnapin -Name NutanixCmdletsPSSnapin -ErrorAction SilentlyContinue | % {$_.Name}
if ($ntnxpsLoaded -eq $null){Add-PSSnapin NutanixCmdletsPSSnapin}

After that it will setup a function that can be called over and over to ensure you are logged into the cluster.  Currently you will have to hardcode in the IP address for the individual cluster Prism IP.  If you have a cluster virtual IP, it can be used.  I will work to make this a passable parameter from the Powershell CLI, but know for now you will need to add in your “server” IP.

#
# Function created to call when connect to the SVM is necessary
# Easier than extra coding of the connect-nccontroller cmdlet
# Make sure to change the server from x.x.x.x to one within your environment
function Connectntnx
{
Connect-ntnxcluster -server x.x.x.x -username $ntnxuser -password $ntnxpwd -AcceptInvalidSSLCerts | format-table -property name,version | out-null
}

# Test variable output
#write-host “$ntnxuser”
#write-host “$ntnxpwd”

The script continues from there and builds the menu that will be used throughout to provide the admin with the specifics on what workflow they would like to accomplish.  For now it only references a single working option, with references for “Future”.  I did this to fill out text and not just have a single entry.  Ideas for additional entries are: create / delete / update VMs, Restore VMs…etc.  For now only Option 1 has additional code that works at this time.

#
# Start off by asking what tasks they are interested in completing
# In order to add additional tasks, add in additional write-host
# lines and then create a function to allow the option to be run
#

Write-host
Write-host “This script will allow you to do the following for a given cluster:”
Write-host “”
Write-host ” **List All VMs, all unprotected VMs or Add VMs to a Protection Domain” -foregroundcolor white
write-host “”

Do {
Write-host “#####################################################################” -foregroundcolor Green
Write-host “#####################################################################” -foregroundcolor Green
Write-host “”
Write-host “Please select from the following options:” -foregroundcolor cyan
Write-host “Option 1: List ALL VMs, all unprotected VMs or add a VM to a Protection Domain on a particular cluster” -foregroundcolor White
write-host “Option 2: Future…DOES NOT WORK” -ForegroundColor Magenta
write-host “Option 3: Future…DOES NOT WORK” -ForegroundColor Magenta
Write-host “”
Write-host “”
Write-host “#####################################################################” -foregroundcolor Green
Write-host “#####################################################################” -foregroundcolor Green
Write-host “”
$initoptions = Read-host “Please enter 1, 2 or 3”

write-host

} until ($initoptions -eq “1” -or $initoptions -eq “2” -or $initoptions -eq “3”)

# Call Connect to test
Connectntnx

From there the remaining of the script are the endpoints depending on options chosen, that initiate workflows.  The switch statements within option #1 were easier to do that a series of if…then…else type statements.  I tried to comment the script as much as possible.  Should make following the flow fairly easily.

##########################
# If they select option 1
# jump to here
##########################

if ($initoptions -eq 1)

{

#
# List VMs on Nutanix Cluster
#

Do {

Write-host “This will list current AHV VMs, list all the unprotected VMs and Add VMs to a protection domain”
write-host “”
write-host “All = Display all VMs” -ForegroundColor Cyan
write-host “Unprotected = Display all unprotected VMs” -ForegroundColor Cyan
write-host “Protectvm = Protect a single unprotected VM” -ForegroundColor Cyan
write-host “”
$ntnxsav = read-host “Please enter All, Unprotected or Protectvm”

Write-host
} until ($ntnxsav -eq “all” -or $ntnxsav -eq “unprotected” -or $ntnxsav -eq “protectvm”)

#
# Using Powershell switch statement to keep from having a large if/else series of statements
# It is comparing the values for $list, $unprotect or $protect
#
# Requires that user type in list, unprotect or protect otherwise the above will loop continuing to ask the use to enter
# the correct statement. The switch statement is a compare of the value of the response to the available “functions” below it.
#
# Only valid responses are: add or remove
#
#

switch ($ntnxsav)
{
all{

#
# Calling the connectnx function
# This will connect to the Nutanix cluster hostname or IP when the script was started
#
write-host “”
write-host “Connecting to Nutanix cluster $ntnxclus as the user $ntnxuser” -foregroundcolor green
#
# Commented out for now, no need as the user is connected to the cluster earlier
#Connectntnx
#
# Retrives the names of the VMs, listing only the vmname, host running on and VMID

write-host “Time”
Get-Date
write-host “”
write-host “Displaying all the VMs and the NTNX host they are running on” -ForegroundColor Magenta
write-host “They are sorted alphabetically by VMName” -ForegroundColor Magenta

get-ntnxvm | where hypervisortype -eq kkvm | select vmname,hostname | Sort-Object vmname | ft -AutoSize

write-host “”
Read-host “Press the ENTER key to return to main menu…”

}
unprotected{
#
# Calling the connectnc function
# This will connect to the Nutanix cluster hostname or IP when the script was started
#
write-host “”
write-host “Connecting to Nutanix cluster $ntnxclus as the user $ntnxuser” -foregroundcolor green
write-host “”

#
# Retrives the names of the VMs, listing only the vmname, host running on

write-host “Start time” -foregroundcolor green
Get-Date
write-host “”
write-host “The current unprotected VMs on the cluster is:” -foregroundcolor green
write-host “”

get-ntnxunprotectedvm | where hypervisortype -eq kkvm | select vmname -ExpandProperty vmname | ft -AutoSize

}
Protectvm{
#
# Protecting a specific VM, will change later to all unprotected VMs
#
write-host “Listing all the currently unprotected VMs on this cluster” -ForegroundColor Magenta
write-host “”
get-ntnxunprotectedvm | where hypervisortype -eq kkvm | select vmname -ExpandProperty vmname | ft -AutoSize

#
# Select the name of the VM that you want to protect
#
#
write-host “”
write-host “Enter the name of the VM you want to Protect” -ForegroundColor Magenta
$ntnxvmname = read-host “Type in just the VM Name”
write-host “”

# Display all the available NTNX Protection Domains
write-host “Listing the current Protection Domains on the cluster” -ForegroundColor Magenta
Get-NTNXProtectionDomain | select name -ExpandProperty name

write-host “”
write-host “Enter the name of the Protection Domain you want the VM placed in” -ForegroundColor Magenta
# Capture the protection domain user input
$ntnxpd = read-host “Type in the cluster Protection Domain”

Add-NTNXProtectionDomainVM -pdname $ntnxpd -names $ntnxvmname
write-host “”
# confirm took by pulling vmdetails and listing only the vmname and the NTNX PD

write-host “Display the VM name and it’s new setting for Protection Domain” -ForegroundColor Magenta
get-ntnxvm | where vmname -eq $ntnxvmname | select vmname, protectiondomainname
}

}

}

As always, feel free to comment and if you have better ideas for the script, definitely open to new ideas.  Below is example output as the script is being run:

blogpvmprotect1

blogpvmprotect2

Becoming a better soccer player…

Outside of my day job in the IT field, I coach a youth travel soccer team for one of the largest soccer clubs in the country.   As part of coaching I am often asked questions like “My child has a desire to play at a higher level do you have any suggestions on what they can work on?” to recently “My son has a desire to play in college, what can we do now to prepare and help him or her meet that goal?”.  Below are some of the typical things I respond with.  As you read this you will find its geared towards the sport of soccer.  However you can insert in any other sport.  It does not matter the sport that one participates in, the details around what you can do are the same.

  • Individual training – Find an outside trainer to work with.  Unfortunately the older a player gets, the more expectations will be that their technical skills with the ball are already there.  Unfortunately club team training will not provide the level of interaction and focused time on technical skills with the ball needed to refine and grow technical ability as players get older.  There are often several local private individuals or businesses that specialize in more personalized training activities.  Additionally many clubs have very good coaches who will work with players to conduct individual training on the side.  This is easily the single biggest investment you can make beyond playing on a club travel team.  There is no substitute for more focused and lower trainer to player ratio engagement.
  • Work on his or her own – Just going to training a few times a week and maybe an extra trainer will only get a player so far.  Your son or daughter needs to develop a habit of putting in the extra time outside of organized activities.  Encourage them to set goals and then as they go along, measure whether they are taking the necessary actions to meet them.  When working on their own, it needs to be time spent not just kicking the ball around, but focused and purposed training of skills.  The more comfortable he or she is with the ball and can execute at speed, the better the rest of their game will be.   In addition, stay in shape so that when soccer starts back up they aren’t focused on spending time getting their endurance back and struggling to participate because they spent the off time watching TV or playing video games.  During the time between tryouts and when team training starts back up, continue to participate in activities that keep one active.  A bonus are activities where they can work with the ball at the same time.  A great soccer workout I recommend to many young and even adolescent players is the “1000 Touches” series of drills.  There are many versions of this online and through sites like Youtube a player can easily learn the skills without the assistance of a coach.Keep in mind that results don’t happen overnight.  If you look at a professional athlete today that is at the top of their sport, they got there because they put in hours upon hours upon hours of work.   Ronaldo and Messi didn’t wake up one morning and suddenly were two of the best soccer players in the world.  It takes time and hard work to be successful and great at something.
  • Be Flexible and open – Have an understanding that a coach will see something more in an individual than maybe the player sees or has a desire to do.  I have watched players who wanted so badly to be a midfield attacking player, but their  coaches saw the player as more of a central defending player.  The players fight that notion and it only hurts their development.  They later realize that their coaches were doing what was in their best interest.  It’s only after this realization that a player’s abilities on the field really blossom.  Players need to embrace what a coaches sees.  It  could mean the difference between a good vs. a great player when you are done playing.  If a coach says he sees a player as X, then work as hard as one can to learn X.  Research the position and look at what skills one needs to have in order to play that position at a high level.  This does not mean you can’t continue to have a desire to play or be a different player.  I encourage that, but you can’t have blinders on and be unwilling to see what somebody else see’s in you as well.
  • Live…eat and breath soccer as much as possible –  Watch it on TV, follow professional players that play similar positions.  Go and watch games of teams in your club that are above the one you play on.  Learn what they do well or differently than you and how the coaches work with them to improve.  Find a player you can compare yourself to on those higher teams and find a way to compare aspects of the game to your own skills.  I once heard Anson Dorrence, the great women’s team coach at UNC, say to a parent regarding a recruit he once talked to “If you tell me you love soccer and don’t watch it, then you don’t love soccer”.  Those who want to play at a high level or even in college have to LOVE the game.  Playing is only a part of loving the game.
  • Don’t focus so much on what team one is on or how much playing time you get – Don’t get caught up in “i am on blue , but really should be on red”….or….”why do I only get X mins but so and so gets Y mins”.  Instead focus on what you can do to move to the Red team or get Y minutes.  Talk to your coaches and ask for feedback, with the understanding that you may or may not get feedback you want to hear.  As a player gets older, the expectations will be that the player initiates communication with the coach and becomes active in their own development.  The players have to want it, not the parents, as they get older.  Let me repeat that again….the PLAYER has to want it and NOT THE PARENTS!  This is often something overlooked by many parents.  You can’t will or push your child into something you want more than they do.
  • Playing on a lower team in the club and getting more minutes is ok – This goes with the above point about not focusing on what team you are on or the amount of minutes you get.  If you make a lower team but will get more minutes as a result, this is a win.  Making a higher team and sitting on the bench, getting little to no minutes means less “practice” via putting to use the skills you are learning in a game.  There is no substitute for touches on the ball, that includes both training and game activities.  All the while, focus on what you can do to get on that higher team and get minutes on the field next time.  Far too often parents and players get wrapped up in what team they make.  This then feeds into a potential for a negative attitude because you did not make the team you felt you deserved.  Which in the end can impact how much you develop during the time you are playing on that team.  Accept where you are but work hard to move up!  Accepting doesn’t mean settling!
  • Attend College soccer ID camps (if you have a desire to play in college) – As a player gets into high school, start looking into college ID camps.  You might even start to get emails from various programs.  Some of the camps even provide a post camp assessment, which would help in setting goals and where they need to improve.  Getting on a colleges radar will be huge, these camps are a way to help with that.
That was a bit long, but these are my thoughts on the drive and desire to be a better athlete and what that translates to.  At the end of the day, players don’t get better by just “showing up” and participating.  Becoming a better athlete happens because you want it and you put in the work necessary.  The same goes for life…you get out of it what you put in!

Nutanix Community Edition Install – Single Node Cluster

What follows is what steps I went through in order to setup and configure my Nutanix Community Edition single node cluster.  I hope this helps anybody looking to install Nutanix CE.  Please note that Nutanix Community Edition is not meant for production use.  When combined with a few Intel NUC devices, it makes for a very nice and quiet home lab.  A solution that is also much more friendly on the electricity budget than a few howling desktop (or worse server) class machines.  Not to mention the space savings.

My Hardware:

  • Intel NUC – NUC615SYH – I purchased mine barebones from Amazon for less than $400.  I opted for the Tall version that would support 2.5″ HDD or SSD.
  • 32GB Memory – This is the max that the NUC can support and given that you will surrender memory to the Nutanix CVM, its best to max out the device
  • 2.5″ SSD 240GB – I opted for a cheaper drive from Patriot because at the time I was trying to build a home lab on the cheap.  The requirements for Nutanix CE are at least a drive of 200GB or more.  If I were to go back and do it again I would opt for something like the WD Blue 500GB SSD.  If all you are doing is building a home test lab, there is no need to go all out on expensive storage.  The WD Blue 500GB SSD goes for less than $150
  • 250GB M.2 SSD – I went with the Samsung V-NANA SSD 850 EVO.  I got it for $99
  • 32GB USB thumb drive – this will serve as the boot drive for the Nutanix CE software

Now in my case I was only building a home lab to be able to run a few VMs and test out Nutanix Prism.  So I purchased (2) each of the above.  The minimum sized Nutanix CE cluster you can build is a single node cluster.  For my purposes this was sufficient and I plan on replicating between the single node clusters for data protection.  Outside of that you could of course build a 1, 3 or 4 (max) node cluster.  Again for a home lab, I didn’t need much beyond a couple single nodes.

Now that we have the hardware out of the way, here are the simple steps to go about creating your cluster:

Download the Necessary Software

  1. Login to the Nutanix Portal – https://my.nutanix.com
  2. Once logged in click on “Community Edition”
    screen-shot-2017-01-27-at-1-37-59-pm
  3. On the right-side of the screen you will see “Download Software” link under “Announcements”. Select that link
  4. On the page that pops up select “Solved Go To Solution” in the blue box at the top
  5. From this new page you will be presented with a download link for the software.  Select the link and save the file to a location on your workstation

Create the bootable USB drive ( used a Windows machine with Rufus for this step )

  1. If you don’t already have it, download Rufus – click here
  2. Insert the thumb and open the Rufus app
  3. Device should populate with your thumb drive, verify
  4. I Set the following:
    • Partition scheme and target system type = MBR partition scheme for BIOS or UEFI
    • File System = FAT32
    • Cluster size = 8192 bytes
    • Format Options = DD Image
    • Click the disk icon next to DD Image and navigate to the Nutanix CE edition download
    • Leave all other checkboxes as they are
  5. Click Start – will take a few minutes to complete.  Now is the time to finish setting up your hardware in the NUC…install the memory, SSD, M.2 SSD,etc

Install Nutanix CE onto your NUC

Pre-reqs – You will need IP (2) addresses, one each, for the host and the CVM.  Make sure you have those ready, along with their respective subnet masks and gateways.  The single node cluster will also ask for a DNS server.  I have noted below what I used, as I don’t have an extensive lab infrastructure setup at home.

  1.  Insert the USB drive into an available slot on your NUC
  2. Connect up your monitor (note: the NUC only has support for HDMI connections)
  3. Connect a USB keyboard
  4. You will be presented with a few screens with a Nutanix banner, they will all self advance.  No need to select anything
  5. If you purchased the Intel NUC that I used, then you will need to modify a file that is part of the install.  The reason is that the NUC, depending on CPU, only has 2 COREs and the Install will look for a system with a minimum of 4 cores.  Take the following steps to adjust the install.  If you don’t you won’t get very far as the check for CPU cores will fail
    • Login as: root | nutanix/4u
    • Navigate to /home/install/phx_iso/phoenix
    • Edit the minimum_reqs.py (note: vi works well)
      • Locate the MIN_CORES entry and change from 4 to 2
    • Save file and type “exit”
  6. At login prompt type install and hit enter
  7. Follow the prompts, it is fairly straight forward.  This took about 10minutes on each of my NUCs.  Since this is a single node cluster install, make sure to select the single node option provided during setup and input a DNS server.  You don’t need your own, you can use publicly available DNS servers if you want.  In order to complete the install I used 8.8.8.8 which is the public Google DNS server.

Complete Setup

  1. Once the install is complete on your workstation open a web browser and connect to the IP address of the CVM.
  2. You will be presented with a login screen for Prism
    screen-shot-2017-01-27-at-2-06-19-pm
  3. Login is: admin | admin
  4. You will be prompted to change the password upon login
  5. Next is a prompt for your Nutanix NEXT credentials
  6. Once logged in the cluster will have a name of “unnamed”, to change the cluster name do the following:
    • Select the gear icon in the upper right corner of the screen
      screen-shot-2017-01-27-at-2-17-14-pm
    • Select “Cluster Details”
      screen-shot-2017-01-27-at-2-18-31-pm
    • Under “Cluster Name” enter a new name and click “Save”
  7. Now you should have a functioning cluster, albeit single node.

This is a great way to test drive Nutanix.  The community edition is a fully functional version of the same software that would be running in a big Enterprise Data center.  Good luck and as I have time I will try to post more about Nutanix Community Edition.

One, two, three…replicate me

Have you ever wondered why you see commercial airplanes with multiple engines?  Next time you are sitting at the airport waiting for your flight, take a look out the window.  Some (ok many) have two, an odd one might still be around with three or even some of the larger really long haul planes have four engines.  The reason for this difference has to do with several factors:

  • the age of plane and its intended route – older planes that flew long haul flights, over something like the Atlantic ocean, were required to have different number or size of engines for redundancy purposes.  This was to ensure a flight could safely continue in the event engine trouble developed or at worst fly long enough to a diversion airfield close enough that on less power the plane could make it to.
  • the amount of weight the plane holds – between people and cargo, a fully loaded plane could need a considerable amount of thrust in order to take off.  Take for example the 747, fully loaded to the gills, it needs all of its available engine power just to get off the ground
  • the altitude for flight – if you have ever been on a rather unpleasant flight that is getting thrown all over the place because of weather, you can thank the smaller planes with less capable engines for that.  Having a larger engine and/or more of them, allows for higher flight and thus the potential to avoid bad weather by finding smoother air higher up in the sky

Now over the years the various regulatory agencies have changed requirements and frankly the engines themselves improved dramatically.  In the early days of flight the engines were not considered the most reliable and stable.  As engine reliability improved, regulations around the engine configuration changed.

So what does this have to do with IT and more importantly the title of this blog…I am thinking the same thing myself, kidding of course.  Like the airplane and consequently the airline industry, IT architectures build in redundancy as well.  For this article that comes in the form of protection in the event of a failure involving disk resources.

Disks are typically considered the most likely resource to fail when dealing with IT environments.  Now over the years the technology has changed significantly, going from spinning media (aka HDDs) to flash based media.  The data protection mechanisms themselves have evolved.

To RAID or not to RAID…that is the question…

When I started out in the storage world I came to know RAID and in particular RAID-4 and RAID-6 during my time at a pretty large and rather “famous” NAS storage company.  Rather than re-write what has been written about numerous times, Google is your friend or if you want here are a few articles that explain RAID:

In short RAID is a mechanism by which to group multiple disks together for performance reasons and to provide data protection in a storage architecture.  This can be done entirely in software or utilizing hardware components to manage the disks.  With software the disks are connected directly to the motherboard, not utilizing dedicated hardware RAID controllers with a software component managing the disks (i.e.: Storage Spaces from Microsoft).  The other side of the equation is hardware based whereby the disks are all connected to some sort of RAID controller and that controller is managing each of the disks in some sort of RAID group.  Each has their own merits and potential use cases, I will leave that investigation to searches via Google.

When the traditional SAN/NAS based storage market emerged many years ago, RAID was the data protection model of choice.  As the HCI market emerged, the players in this market began looking into how to take their systems to extreme scale, reduce the administrative tasks associated with storage and finally make the system more resilient through self-healing.  In effect change the conversation and not just do something that was done before and throw a few new things on top.  This investigation for Nutanix had them looking into what the hyperscalers were doing…the Google’s, Facebook’s etc of the world.  These companies were once consumers of the traditional SAN/NAS architectures but they eventually branched off and began building their own infrastructures.  The area of focus here is around how to protect the data once its written to disk.

For Nutanix this took the route of developing the Nutanix Distributed File System and incorporating data protection mechanisms involving data replication or RF, over your traditional RAID concepts.  Data Protection via RF involves replication of data across nodes, otherwise known as RF (or Replication Factor), typically referenced as RF2 or maybe RF3 (for two and three copies respectively).  For example, a system that implements RF2 would store (2) copies of data within a cluster to ensure the data is protected.  When a virtual machine issues a write, a copy of that write will be consumed by one node in the cluster and in parallel the same data is sent over to another node to complete the 2nd copy (RF3 would create in parallel 2 additional copies over the primary).

Not all HCI players in the market took the same route.  RAID still exists in some manner by competitors in the HCI market.  Now we can debate back and forth the benefits of RF vs. RAID, but a few key points to keep in mind when thinking about HCI that incorporates RAID for persistent or local data protection:

  1. Drive sizes and rebuild times – RAID can definitely do the job it was designed for.  The problem however is that in an HCI environment that contains spinning media, the time to rebuild when encountering a failure can be significant.  The underlying capability of a spinning HDD hasn’t changed nearly as much as the overall capacity of a drive has.  Back when drives were in the sub-terabyte range, a RAID rebuild wasn’t entirely unreasonable.  However, now that drives are in the multi-terabyte range the re-build time after failures can extend into many hours (or maybe into days).  If you did some quick back of the napkin math with say a 4TB drive you are looking at roughly 7 to 10 hours for a re-build depending on the speed of the drive.  The back of the napkin math is: Drive Capacity in MB divided by Seq Write Speed in MB/sec = approximate rebuild time in seconds.  For example a 4TB drive is 4,000,000MBs and if that drive had a sustained sequential IO rate of 150MB/sec you are looking at roughly a 7.5 hour rebuild time.  That doesn’t even take into account other system work on the already degraded RAID group that could slow down the rebuild time.  ***Keep in mind the following: a RAID re-build will work from parity in the RAID group to re-construct the data from the failed disk.  This means hammering the disks in the raid group for a rebuild.  So not only is the rebuild occuring from a smaller subset of the overall cluster availability, it is also taking away cycles that could otherwise be used to process normal IO.  Sure it might be a lower priority task but its still cycles spent re-building.
  2. Administrative tasks are not reduced – In a RAID architecture, you still incur the administrative overhead.  RAID will take care of rebuilding the data through the use of hot spares, but the admin will have to take time out of their day at some point and run down to the data center to replace the failed disk.  Replacing the failed disk ensures there are still hot spares available.  With Nutanix the admin doesn’t have to take time out of their day to run to the data center.  The system will self-heal through replicating the data through-out the cluster, using all the other available nodes to assist in the rebuild.  Once the data is re-replicated, then the system is back to the same level of redundancy as it was prior to the failure.   Sure the drive will still need to be replaced, but it doesn’t turn into a fire drill that requires somebody take action sooner than later.  This allows a task like drive replacement to happen on a scheduled basis or batched into other trips to the data center.  This makes the admins more efficient in how they go about their day.

    Think about it this way: do you think that Google or Facebook stop what they are doing every time there is a drive failure and run down to the data center in order to take care of a failed disk?  Now the response might be “well we are not Facebook or Google…”, which might be true.  However, the same level of concern over how time is used shouldn’t matter whether a company is or is not Google or Facebook.  What about the failure that occurs in a remote office without IT staff?

  3. Drive size vs. drive count – When you factor in RAID rebuild times, you have to take into consideration what size HDDs you want to include in the system.  In order to ensure you don’t run into a huge RAID re-build window, you have to likely look into getting a larger number of smaller drives.  The same goes for the size of the disk group / raid group associated with the volume or container (or whatever naming construct is used for the data store).  The larger the disk group, the longer the potential rebuild time.

All too often the conversation around a solution covers performance and price at the solution level.  The conversation should also include a consideration around “how much is time worth?”

The building is on fire: Difference between random and sequential IO

Like my last post, it’s been a fairly long time since I have posted anything to my blog.  I recently had a conversation with somebody around things technology and the topic of explaining aspects of system performance came up.  That discussion and my subsequent answer gave me an idea for a blog post.  Hope you enjoy….

The question posed was something along the lines of “Explain sequential and random IO to somebody who doesn’t understand system performance in the IT world?”.  I thought about it for a moment and below is what I answered with (well not initially but its where I ended up):

Imagine there are (2) buildings on fire and you need to quickly put out the fire.  One group decides to put out the fire using a hose connected to a hydrant and provide a steady stream of water they will spray on the fire.  The other group decides to go old school and use buckets and a bucket brigade, unfortunately however they don’t have enough buckets to keep a steady stream of water coming (i.e.: 1 bucket for every 3 or 4 people..or maybe not all the buckets are the same size).  The group with the fire hose I would equate to sequential IO, each and every drop of water is coming one after another with no break or delay.  They don’t have to wait for water to come from somewhere…open a valve and out comes water.  The group with the buckets, they are a bit more like random IO.  The delivery of water won’t be consistent as there could be a delay in receiving a new full bucket as the buckets move through the line.  The bucket crew  will take longer to save the building.  There will be times when a person has to wait for a full bucket to circle back to the hydrant and back down again.

Now in reality the bucket crew will have a mixed workload of random and sequential.  It’s likely there will be times when full buckets arrive one after another.  Other times the person at the front is watching the empty bucket work its way back to the source to be filled and return.  Crew #1 with the fire hose and hydrant again equals sequential IO and the crew with the buckets is more like random IO.  It’s clear that crew #2 is not the crew you want dispatched to save a building on fire.

So how can crew #2 work more efficiently or improve their overall performance…besides giving them a fire hose and pointing them to the nearest hydrant.  We can also integrate IO size into the same analogy.  The bucket crew could use bigger buckets, increasing their “IO size” and thus providing them the ability to get more water on the fire during each arrival of a new full bucket.  However despite this great idea, it could cause the crew to get tired (or move slower) as they move a heavier object through the line and possibly slow down the bucket line.  It could create a backup of full buckets because the person at the front throwing the water on the fire can’t move as fast, creating a “full bucket queue”.

What to do then?  If increasing the bucket size (thus the IO size) isn’t possible how about creating a “water cache”.   The bucket crew could place a giant tub at the front and give the guy at the front of the line his own bucket.  Then the people behind him in line, the bucket crew, work to keep the giant tub full.  This tub would serve as a water cache, to remove or reduce the time (latency) waiting on a new full bucket.

To take it even one step further, you could also add an additional man at the front of the line, to help with throwing full buckets of water on the fire.  This begins to get into the idea of how the number of threads can impact the ability of a system to perform.  With that aside, the guy at the front of the line is just a single resource or “single threaded”, the same with the guy using the fire hose.  Adding additional resources to the front of the bucket brigade creates additional threads and can increase the overall effectiveness when attempting to save the building.  Now if you do that, you would have to think about how fast new full buckets arrive to the front.  If you add a 2nd guy at the front to help with tossing water on the fire, but don’t change the rate at which new full buckets arrive (or keep the “water cache” full), then one or both of the guys could be waiting and adding the additional resource did little to help.

I am sure there are additional aspects you could incorporate into this analogy if we thought about it long enough.  I hope you enjoyed this explanation on some aspects of system performance.  It makes sense to me (hopefully readers as well).  In the IT world its easy to forget or take for granted our understanding of how technology works.  Being able to explain something complex in terms that someone not from the same field can easily follow along, is a critical skill in anyones job.  Feel free to comment if you have additional thoughts on how to refine this explanation.  Thanks for stopping by!

 

NetApp Insight 2015…All the cool kids are going!

So it’s that time of year…no not the start of the NFL season.  While exciting as the start of the NFL season may be, it’s when the hype begins for NetApp Insight ( http://www.netapp-insight.com/ ) .  You are not familiar with NetApp Insight?  It is NetApp’s yearly technical conference where a lot of awesome technical people get up on stage and talk about all the goodness that NetApp has to offer.  I have been lucky enough to present at the last two Insight conferences and will be returning this year for a 3rd year.

There are a lot of fantastic breakout sessions and labs.  I however am partial to the two breakout sessions listed below.  They happen to be what I will be presenting this year.

1744: A Fistful of Windows File Services – An NDO Deep Dive

Find out how NetApp® provides non-disruptive operations for Microsoft® Windows® File Services, in particular the SMB protocol. Learn what happens during cluster activities like migrating a LIF to another node in the cluster, moving a volume to another node, or relocating an aggregate. We also explore how these activities relate to the different SMB protocol versions and how Clustered Data Ontap® can continue to provide access to your data.

2179: Microsoft Windows File Services Strikes Back – Not just for Home Directories

The Microsoft® SMB protocol has changed significantly over the years. SMB is still a rock star with traditional project file sharing and home directories, but there is so much more to it and Clustered Data OnCommand® is all in. Come see what NetApp® has to offer in Microsoft Windows File Services. We will briefly cover what arrived in Clustered Data ONTAP 8.3 and 8.3.1. Learn about ODX and why we are different. Discover what Dynamic Access Control is and how NetApp was the first storage vendor to implement support for this new Microsoft feature. Learn about features like SMB Witness and Auto Location, and why they matter to you. Come see how Windows File Services on Clustered Data ONTAP can help you.

You might notice a slight theme to my session titles, other than the words “Windows File Services”…they are loosely based on movie titles.  Come see how I weave in Clint Eastwood’s character in the movie “A Fistful of Dollars” into my NDO session and Luke Skywalker’s quest in becoming a Jedi Knight into my “Windows File Services Strikes Back” session.

To round out your NAS session schedule, also check out NFSDudeAbides ( https://whyistheinternetbroken.wordpress.com/2015/07/29/netapp-insight-2015/ )sessions on Multiprotocol and SecD:

1884 –  Unlocking the Mysteries of Multiprotocol NAS in Data ONTAP

Having trouble understanding multiprotocol? Can’t figure out the difference between authentication and authorization? Do you think a name mapping rule is something doctors provide to new parents at the hospital? Wondering what a “CIFS/SMB” and “NFS” is anyway? This session can help! We cover all things NAS in clustered Data ONTAP, from the basics to the best practices, and try to help make multiprotocol a little clearer for storage administrators.

1881 –  Authentication in Clustered Data ONTAP: SecD Deep Dive

Authentication in clustered Data ONTAP uses the security daemon (SecD) for connectivity to external dependencies for NAS protocols such as Active Directory, LDAP, NIS and DNS. This session will cover how SecD works in clustered Data ONTAP 8.3 and beyond, as well as how to troubleshoot issues and what kind of statistics are available for SecD and external name services.

If you are attending, look for the above when building your breakout session schedule.  Hope to see you at NetApp Insight 2015 both in the US and EMEA.  Have a great day!

Anatomy of an SMB connection…Part 1: What Languages do you speak….

This is the first in a series of articles that I hope will help to give you a good base understanding of what happens when you are attempting to connect to a shared resource via the SMB protocol.  In Part 1: What Languages do you speak, we will look at how the protocol version is determined when attempting to connect to an SMB share.

languages

Hello…Hallo…Aloha…Ciao…

The header above are all ways to say hello in various languages.  Many people likely know or could quickly learn to say hello when speaking to somebody in another language.  The learning part isn’t the issue, it’s determining which one to use…which version of “hello” do you use?  The same can be said for the SMB protocol.  In order for a client and server to know which commands they can send, they will need to make sure they can both speak the same language.  You could certainly, as you meet somebody, attempt to say hello by rolling through all the ways you know to say hello until you get a response.  Which could make for it’s own fun when suddenly the other person hears a word they understand, assumes you speak their language and begins a conversation that you don’t understand at all…but I digress, hopefully you get the point.

As an alternative you might look at that same person and instead of trying to scroll through the various versions of hello you know, instead you might say something like “Hello, do you speak….?”.  Now I know what you are thinking, if they don’t understand the language you are speaking then how will they even know to respond or how to respond.  That is exactly the point.  When you you say “Do you speak…” you are effectively telling the other person what language you speak without saying “I speak English!”.  If they speak English, they very well might respond “Hello, yes!”

Negotiate Protocol Exchange – Dialects

In the initial phase of establishing a connection to an SMB share the same process of trying to determine which languages the client and server speak has to occur.  This process occurs in the “Negotiate Protocol” exchange.  It’s a series of SMB packets sent between the client and server that helps to establish the SMB version (among other things) that will be used during the time the client is connected.  Depending on the SMB version that initiates the conversation, it will dictate how many of these exchanges will need to occur.

The negotiate protocol request is the first SMB frame exchanged and would be similar to the above example where somebody says “Hello, do you speak….?”.  It’s where the client offers up the languages that they speak to the other party, with the hopes the other party is fluent in one of them.  Below is an example of what this looks like when you capture a packet trace of a connection attempt by a client to a server:

negprotoreqYou might notice that there is more than a single request by the client.  We will dig a bit further into the initial SMB request to understand why this happens.  To know why you see more than one request by the client, lets look at the actual details of the request.  Below are the contents of that first Negotiate Protocol Request sent by the client:

negprotoreqdialectsHere we see that the client sent 8 dialects or languages that it can speak.  Going back to the original “hello” introduction between two people, this would be like giving somebody you meet a piece of paper that has written on it the languages you speak.  In this particular case this is a Windows 8 client, it can understand a considerable number of SMB dialects. Some rather old and several new ones.  The key here are the multiple SMB 2.x entries.  This means the client supports SMB 2, 2.1 and possibly 3.x.  Given that SMB 2 and 3 have sub dialect versions, the client isn’t 100% sure, so it just informs the server that it supports at least those versions.  The SMB 2.??? is a catch all for the additional versions beyond 2.0.

Further Clarification Needed

The server upon receiving this request will then examine the list of dialects the requester sent.  From the list it will then determine if it can talk to the client making the request.  In this particular example, the server obviously speaks at least SMB2 because in Negotiate Protocol Response was sent as an SMB2 response.  However, because there are multiple versions of SMB 2 and 3, the server isn’t sure which one to select.  So instead of just randomly picking one, it replies to the request selecting the dialect of 2.???.  This is seen in the initial SMB2 response, note the Dialect: 0x02fff below:

negprotorespsmbquestion

This response from the server will signal to the client that it needs to issue an additional request to further clarify which dialect to use.  In this 2nd request  the client gets a little more specific on the versions of SMB2 or 3 that it understands, hoping the server picks one from this list.

negprotoreqsmb23Now we see that the client tells the server “I can talk SMB2, SMB 2.1, SMB 3 or SMB 3.02”.  Now the server has a much clearer picture on the languages that the client can understand.  The server again will review the list and select the highest SMB dialect that it supports AND has enabled.  The enabled is key to understand.  If for any reason a version of the SMB protocol has been disabled, even if the running OS would support a particular version, it won’t be selected.  For this exchange the server selects SMB 3:

negprotorespsmb3

In Conclusion

With the dialect established, now the client and server can continue communicating with each other.  They will know which SMB commands each is capable of understanding.  Obviously there is more that goes on in this phase of establishing a connection to a share, but one of the more important is making sure that the client and server will be capable of communicating via the same language with each other.  As the series goes on we will explore more about the various phases of a share connection.  Hope you found this useful!  Have a great day and thanks for taking the time to read this post.

DO15: MMC Like Management via PowerShell Script – Part 3: Manage CIFS Sessions

It’s a been a little while since I have posted something on managing Clustered Data ONTAP from NetApp.  If you are not familiar with this series, the goal is to provide folks with a means to manage their Clustered Data ONTAP systems through the incredibly powerful scripting available in PowerShell.  If you are interested, you can find Parts 1 and 2 in the series here:

I definitely recommend reviewing those posts prior to moving forward.  There are some details, particularly in Part 1 of the series, you will need to understand and have configured to utilize the script discussed in this article.  Now that we have covered the basics, lets jump into the script for this part in the series.  You will want to download the latest copy of the code from my GitHub here – pwrshellmmc.ps1

***Note for this blog post in the series I decided to leave most of the snippets of code out.  I try to heavily comment the code enough that you can easily open up the script itself to determine what is going on.  For this part in the series, managing CIFS/SMB sessions starts at line 766 in the script.

What does the script do

whatudohere

The script will now contain a 3rd option upon start-up that will allow you to view and close open CIFS sessions.  Just like the other options within the script it is very “menu” or choose your own adventure driven.  Here is a screenshot of the new option added:

part3options

If you have the need to view open CIFS sessions, the script will provide you the option of viewing all the open sessions or just look for a single connected client.  The view output will only provide you with a small subset of the possible details that you can pull for a given client session.  The script uses the Data ONTAP PowerShell Toolkit cmdlet “get-nccifssession”.  The view option will pull the following details: windows user, IP address, connected time, number of open files, is the session SMB signed, the network address the client is connected to on the SVM itself and the protocol version used.  This information is the same whether it’s retrieving all the sessions or just a single session.  I should also note that the information returned will be SVM wide.  This means it will show you the current connected clients over an CIFS/SMB session for an SVM, no matter which node in the cluster the connection happens to be located on.

parksclosed

For the close part I only scripted the ability to close a single session.  Down the road I will work on building into the script a way to close multiple sessions without having to run the script multiple times.  The script starts out by building a hash table of all the currently connected sessions and stores them in a variable called “$sesshash”.  I then pipe the cmdlet through the “group-object” to group the items together.  It will key off the address (or IP Address) from the get-nccifssession output to group details for each session together.

$sesshash = get-nccifssession | select windowsuser,address,lifaddress,node,sessionid | Group-Object -Property address -ashashtable

This variable will be used later to close out a session requested by a user running the script.  In the hash table will be stored the windows user, ip address, network address on the SVM where the client is connected, the name of the physical cluster node the user is connected to and the id associated with the individual session.  Again in the “choose your own adventure model” the user will be prompted to provide the IP address of the connected client they wish to close out.  Prior to requesting the user for  input the script will present a list of clients currently connected to CIFS server running on the SVM.  This list is also an SVM wide list and reflects all clients connected to CIFS no matter which node happens to be housing the request.  Once the user enters in an IP address, it will use the information stored in the hashtable variable to close out the session.  The $sessselect listed in the code snippet below is the IP address the user is prompted for and what will be used to close out the session.  I chose IP address as its most likely to be unique for a given connection. When storing the details in a hash table, they are grouped together based on the IP address (see above the $sesshash code snippet).

Close-NcCifsSession -node $sesshash.item("$sessselect").node -Address $sesshash.item("$sessselect").address -SessionId $sesshash.item("$sessselect").sessionid

That’s pretty much it!  As always I welcome feedback and if you know of a better or quicker method to accomplish the same, please share.  As always, all of my scripts are use at your own risk and there is no warranty (or support) stated or implied by myself or my employer.  The scripts are cobbled together from things I find out on the interwebs and through my own testing / tinkering.  Please take the time to thoroughly test them in your environment to ensure they produce the desired results before moving them into a production environment.

Have a great day and thanks for reading!

Clustered Data ONTAP Episode 8.3.1 – SMB Encryption

Yesterday NetApp released clustered Data ONTAP 8.3.1.  Here is a great blog post “Clustered Data ONTAP 8.3.1 is now available” by Justin Parisi that explains very well the features included.  In this blog post I will spend a little time going over a new feature included for Windows File Services –  SMB encryption.

SMB encryption was introduced in the SMB3 protocol.  It allows for the securing of data in transit between a client and a server.  It utilizes the AES-CCM algorithm in order to encrypt the data, which also has the added benefit of data integrity through signing.  One thing to make clear is that this feature does not secure data at rest on disk.  At rest storage disk encryption is a supported feature in clustered Data ONTAP, however it is not what will be covered in this article.

The feature can be enabled two different ways.  The first way I will talk about is enabling it at the SVM level.  If applied at the SVM level it will require that all clients accessing SMB shares on the CIFS server be capable of negotiating encryption.  This is a global setting that effectively requires its use.  The second method by which it can be enabled is on a per share basis.  If you have a mixed environment that contains clients who do and do not support SMB Encryption, this provides the most flexibility to using SMB encryption.  This approach of both per SVM or per share is how Microsoft implemented the feature when the released SMB3.

So how do I enable SMB encryption

  • Enable at the SVM level:
     vserver cifs security modify –vserver  -is-smb-encryption-required <true|false> (default: false)
  • Enable at the share level:
     vserver cifs share properties add –vserver  -share-name  -share-property encrypt-data

If you are not a command line junkie and prefer the GUI, no fear you can use OnCommand System Manager….which is built-in to clustered Data ONTAP as of 8.3.  Here are a few screen shots showing how to do this via System Manager:

  • Enable at the SVM level:
    System Manager Global SMB Encryption

Enable at the Share Level
System Manager Encryption per shareThe beauty of this is that enabling this is a simple check box or CLI command.  Unlike other encryption methods where you may have to manage and maintain encryption keys, with SMB encryption that is not necessary.  The key management is done within the protocol itself. It’s a self-contained security mechanism and if you are interested in how the keys are generated review sections 3.1.4.2 and 3.2.5.3 of the SMB2/3 specification located here “Server Message Block Protocol Versions 2 and 3”.

When is encryption started

A client and server will negotiate the use of encryption as part of the process of establishing a connection to a share.  It will depend on which of the above is configured, as to when encrypted frames will begin.  If you are familiar with the process that a client and server go through in order to establish a connection to a share, you likely know there are several phases in the beginning of the conversation that will establish the guidelines for the conversation.  If you are not here is a VERY VERY, as in viewed from the Moon, high level of the first few frames:

  • Negotiate Protocol – this is where the client and server determine with protocol or language to speak (among a few other things)
  • Session Setup – this is where user credentials are passed and a user is authenticated
  • Tree Connect – this is where the client tells the server which share they would like to connect to

No matter which setting you chose the ‘Negotiate Protocol’ and ‘Session Setup” exchanges are not encrypted.  If you have enabled SMB encryption at the SVM level, then everything after the ‘Session Setup’ is encrypted.  Here is an example taken from my lab environment:

sesssetupencryptIf SMB encryption is enabled at the share level, then the encrypted traffic will begin after the tree connect.  Here is an example from my lab environment:

treeconnectencryptAs you can see from the above screen shots, the conversations differ slightly.  In the case of the per SVM, you will never see the details regarding the share connection (ie: the tree connect).  So once encryption has been negotiated at the user authentication phase, everything after that is encrypted.  When you look at the per share setting, the determination to use encryption is established at the share connection itself.  Either way, the data frames on either model will be secured.  The use cases for this feature besides the obvious of “I want to secure my data while in flight” are really data accessed over a WAN or an environment that you have untrusted networks.

So if the need to secure your SMB traffic while in flight has been something you have been waiting for from NetApp, jump in and take it for a spin.  Throw in our recent announcements around All-Flash FAS or AFF, our Data Fabric vision for the Cloud…it’s full speed ahead for clustered Data ONTAP.  As always, thanks for taking the time to read this post and have a great day!