Cluster OS Rolling upgrade is a new feature in Windows Server 2016 failover clustering. If a Windows server 2012 failover clusters running the Hyper-V or Scale-Out File Server roles, you can add Windows Server 2016 nodes without taking the failover cluster offline. In earlier Windows Server versions, you had to create a new cluster with the new operating system on all nodes, and then you had to move the clustered services over to the new cluster. This created downtime and made it difficult to quickly complete upgrades. With Windows Server 2016, cluster nodes do not require additional hardware to upgrade, you can take one node at a time offline, upgrade the operating system, and bring the node back into the existing cluster. This changes the cluster to a mixed operating system mode. The process is fully reversible- Windows Server 2016 nodes can be removed from the cluster and Windows Server 2012 R2 nodes can be added to the cluster in this mixed mode. After you upgrade all nodes, you can change the functional level of the entire cluster from the Windows Server 2012 R2 version to Windows Server 2016 version.
Cluster OS Rolling Upgrade Process
This section describes the workflow for performing Cluster OS Rolling Upgrade. Image copy from TechNet.

Before you start Cluster OS Rolling Upgrades, first we should backup cluster database and workload data. Then, verify that Cluster Aware Updates (CAU) is currently running by using
Get-CauRun cmdlet. If CAU is currently running, use
CAUStop-CauRun cmdlet to stop Cluster Aware Update (CAU).
Stopping Cluster Aware Update (CAU):
If you are running Cluster Aware Update (CAU), verify use the PowerShell,
Get-CauRun cmdlet. To stop CAU use the PowerShell,
Stop-CauRun cmdlet. But in this demo, you have not installed Cluster Aware Updates (CAU). So it’s
WARNING: No Updating Run is currently in progress on cluster.
Verify that all cluster nodes state are Up before remove Node from the Cluster:
Using the cmdlet,
Get-ClusterNode to verify the cluster nodes State.
Pause Node | Drain Roles Hyper-V nodes from the cluster:
In the
Failover Cluster Manager Windows, select
Roles and make sure all the VMs are migrated to another node in the cluster. Here, you are
removing KTM-HOST1 node from the cluster.

In the
Failover Cluster Manager Windows, in the left-hand pane, select
Nodes. In the middle pane,
right click on the Node,
KTM-HOST1, then select
Pause | Drain roles option to drain the node.
Evict the Node from the Cluster:
Once the
node is
drained,
right-click on the
node, KTM-HOST1, select
More Actions and then click
Evict to remove the selected node from the cluster.

In the
Evict node KTM-HOST1 dialog box,
Are you sure you want to evict node KTM-HOST1 from the cluster? Click
Yes.

In the
cluster event details, verity that the cluster node
KTM-HOST1 has been
evicted from the failover cluster.
Clean OS Install:
First remove the currently evicted node from the domain.
Clean Installation of Windows Server 2016:
Format the system drive of the evicted node and perform a clean Installation of Windows Server 2016.
Configure Server Name and Add the Server to the Domain:
Configure the server name as per your previous name and Add the server to the Domain.
Install Hyper-V Role, Multipath I/O, Failover Clustering Feature:
After you have
joined KTM-HOST1 to the
domain, msserverpro.com, you can install
Hyper-V role,
Multipath I/O, and
Failover Clustering feature on KTM-HOST1.
Configure Network and Storage Connectivity:
Configure network connectivity and virtual switch as per previous configuration.
Connect to Shared Storage:
Connect the server to shared storage as per previous configuration. For share storage connection I will share on the coming article.
Add the Node to the original Cluster:
After completed installation and configurations, it’s time to add Hyper-V Host to the failover clustering.
In the
Failover Cluster Manager Windows, in the left side, right click on
Nodes, select
Add Node…

On the
Before You begin page, click
Next.

On the
Selected Servers page, type the required server name under
Enter server name box and click
Add

On the
Validation Warning page, select
No, I do not require support from Microsoft for this cluster, and therefore do not want to run the validation tests. When I click Next, continue adding the nodes(s) to the cluster and click
Next.

On the
Confirmation page, make sure check mark is selected in
Add all eligible storage to the cluster and click
Next.

On the
Configure the Cluster page,
please wait while the cluster is configured….

On the
summary page, click
View report and click
Finish.

Verify the
KTM-HOST1 Operating System is Microsoft
Windows Server 2016 Datacenter and
KTM-HOST2 Operating System is
Microsoft Windows Server 2012 R2 Datacenter.

On the
Failover Cluster Manager Windows, click
Cluster Events to view the cluster
Event Details message, shows that
Failover Cluster runs in mixed mode, which is
compatible version of the cluster service software.
It is recommended that the same version of the cluster service software be installed on all nodes in the cluster”
Verify the ClusterFunctionalLevel, from a Windows Server 2016 cluster node that belongs to a failover cluster:
Verify the OS Version and the current ClusterFunctionalLevel
If the
ClusterFunctionalLevel value set to
8, then the cluster at
Windows Server 2012 R2.
Now Remove NODE 2, Windows Server 2012 R2 from the cluster:
Ensure that the
VMs are migrated from NODE2 to NODE1 in the failover cluster
before you can evict NODE 2 from the failover cluster.

Verify all the VMs are in NODE1,
KTM-HOST1.
Suspend the node from the cluster using PowerShell:
Remove the node from the cluster using PowerShell:
In the
Cluster Events Details, make sure that
“Cluster node KTM-HOST2 has been
evicted from the failover cluster”.

Remove the Server from the Domain, Clean Installation of Windows Server 2016, and Configure Server Name and Add the Server to the Domain, msservepro.com.
Add the Node to the original Cluster (node rejoins the cluster) using PowerShell
We can also add the node to the cluster by using the PowerShell cmdlet
Add-ClusterNode.

After the Windows Server 2016 node is successfully added to the cluster, you can (optionally) move some of the cluster VMs to the newly added node in order to rebalance the workload across the cluster using
Move-ClusterVirtualMachineRole cmdlet given below:
Move-ClusterVirtualMachineRole -Name KTM-DB1 -Node KTM-HOST2 |
Updating the Cluster Functional Level:
Until the Update-ClusterFunctionalLeve cmdlet to run, cluster runs in mixed mode, this process is fully reversible and Windows Server 2012 R2 nodes can be added to this cluster and Windows Server 2016 nodes can be removed.
ClusterFunctionalLevel
8= Windows Server 2012 R2 Functional Level (compatibility or mixed-OS mode)
9= Windows Server 2016 Functional Level
Use
Get-Cluster | Select ClusterFunctionalLevel to display state
Once all cluster members are running Windows Server 2016, use
Update-ClusterFunctionalLevel PowerShell cmdlet (one-way trip) is run, new features are available.
Finally, Microsoft recommends finishing this Cluster OS Rolling upgrade process within 1 month. That’s because having this mixed-OS mode is not optimized in terms of performance. For more details you can check in the TechNet;
https://technet.microsoft.com/en-us/windows-server-docs/failover-clustering/cluster-operating-system-rolling-upgrade
Summary:
Windows Server 2016 makes upgrading clusters easy with rolling upgrades. I hope this article help you Windows Server 2016 Cluster OS Rolling Upgrade configuration.