Microsoft Azure AZ-801 — Section 8: Implement and manage Storage Spaces Direct

48. Implement a failover cluster on-premises, hybrid, or cloud-only

Let’s talk about the concepts of failover clustering with on-premises hybrid and cloud scenarios.

So, failover clustering is a feature that’s been around for a very long time in which it allows us to have multiple servers that are sharing a certain resource., maybe, it’s SQL, maybe, it’s Exchange, maybe, it’s Hyper-V, maybe, it’s some kind of file related services, maybe, it’s DHCP. There’s various things here that you could use clustering for. And the idea would be that you have multiple servers that are sharing this resource. And in a scenario where there’s a failure, the person who’s connected to this service would experience little to no performance decrease of downtime if clustering is done, right. At least they shouldn’t really experience any downtime.

And there’s different scenarios in which this is used. There’s what’s called active-active where all the all the machines in the cluster are receiving clients at the same time. And if a node fails, if a server fails, then it fails over to one of the other servers, the client would be moved and the client wouldn’t even know. There’s also active-passive where you have one server that is accepting all the traffic. And then the passive server is just there in case that active server fails.

So, cluster in clustering. The terminology that’s used, by the way, is the term nodes. So, if you hear that term nodes, that is a server, that is part of the failover cluster. All right. So, if I say I have two nodes, that means I have two servers that are taking part in the cluster. If I say I have four nodes, that’s four servers that are taking part in the in the cluster. The cluster service that spans amongst the different machines is communicating in real time. That is how the servers know. If there is a scenario where one of the servers fails, it’s like a heartbeat based system that it uses.

Now, one of the things that failover clustering utilizes in Windows server environments is what’s called a CSV. That’s a Cluster Shared Volume. The idea of a Cluster Shared Volume is where you have a virtual hard drive that is linked to all the different nodes in the cluster. So, there’s various ways you can go about doing this from using what’s called an iSCSI scenario, which is the one Microsoft kind of puts the most stock in.

There’s also Fibre Channel scenarios. You get Fibre Channel hardware that you can use. There’s what’s known as a Scale-Out File Server. You can set up a file server and it can be a virtual hard drive. It’s hosted on another file server.

So, there’s various solutions for doing this. But the idea is it creates what’s known as a distributed namespace. That means the cluster will be seen by clients as a single machine.

So, imagine you’ve got four machines that are part of this cluster, right? Four nodes that are part of the cluster server. One server, two server, three server, four. Well, to a client, it would all be seen as one server. So, instead of, let’s say, NYC-SVR1, NYC-SVR2, NYC-SVR3, and NYC-SVR4, maybe, this is hosting a SQL database and so it might just be seen as NYC-SQL and that’s the name of the of the service. The namespace. So, clients would think there is a server called NYC-SQL, right? They don’t realize that it’s really four different servers that are sharing this SQL service. Right? So, speaking of which, failover clustering is very popular with databases. It’s also popular with Hyper-V because you could have virtual machines spanning multiple Hyper-V servers. And if you cluster those Hyper-V servers, then you’re basically providing failover for all the virtual machines, the guest operating systems, if you will, that are running on that Hyper-V server. So, that’s a very popular solution. Here’s a visual example for you of on-premises failover clustering.

Now, when you look at this, they’re actually doing this more along the lines of what’s known as stretch clustering or geo-dispersing of clusters. But let’s just focus on the left side first. So, you have the Redmond Site. You’ll see that you have two servers listed there – SR-SRV01 and then SR-SRV02, which I’m just going to call Server1 and server two. Let’s just keep it simple. So, you have Server1, server two in Redmond and then right there towards the bottom, you see that cylinder. It says, “SAN/JBOD”, and that is a storage area, network-based scenario. And JBOD is an acronym that just means just a bunch of disk and they’re using an iSCSI-based connection. So, this is iSCSI-based virtual disk, it could be hosted anywhere. It could be sitting on Server1 or server two or it could be hosted on a completely different server somewhere. It also could be hosted on a network attached storage (NAS) device. So, that disk could be anywhere. The goal, though, is you would want to make sure there was very low latency between those two servers.

Now, if you look over in the Bellevue Site, you have the exact same scenario, but you have you two other servers called SR-SRV03, SR-SRV04, and they’ve got network connectivity, hopefully with very-very low latency. But you’ll notice it’s using what’s called asymmetric shared storage. And I’m not going to get into all the depth on this right now. What that basically means is, maybe, the Redmond Site is the main active area for this, the active data store for this. The Bellevue Site might just be passive in this case. And so the Bellevue version of this data is just simply there in case that there is a failure in the Redmond Site. So, clients would be connecting to the servers in Redmond. And, let’s say, there’s a power failure or flooding or something that occurs, you know, earthquake. I mean, I don’t know, but, well, maybe, not earthquake that usually kind of big. Let’s just say there’s coming of some kind of downtime that occurs in Redmond. Then traffic can fail over to the Bellevue Side. All right. And now the one thing I’ll also say, you’ll notice this is asymmetric shared stores. What that basically means is that when changes occur, they’re going to occur in Redmond first. And as soon as they are committed and verified in Redmond, they will replicate to Bellevue. So, that is what asynchronous replication is.

And there’s also another term called synchronous, where the changes will occur in both places simultaneously. But that’s not what we’re seeing in this picture. This would be, like I said, more of an active-passive based scenario. All right. But that’s an on-premises type scenario for what’s called geo-dispersing of clusters or stretch clustering.

There’s also, of course, you can use utilize Azure Failover Concepts. And it’s really, to be honest with you, almost the exact same scenario. You’ll see that you’ve got two virtual machines in this case. In there, they’re running server and you’ve got a file share witness, which I’ll explain this idea of witnesses here in just a moment. But basically you’ve got your data storage there, right? You’ve got a shared disk and the two VMs are communicating with each other. And if there’s a failure, the service fails over to that.

And the other thing you’ll see in this example is they’ve got an Azure Load Balancer and that’s one of the things you can do with clustering as well especially if you’re doing what’s called active-active, where all the servers are accepting a load balancer and will distribute that load evenly for you. The load balancer also helps monitor in a situation where one of the servers fails.

So, for example, if VM1 fails the Azure Load Balancer because it is doing like a heartbeat type scenario, it notices that VM1 is down, it’ll no longer send load to VM1. It would send load to VM2. So, that’s the idea.

Now, the other neat thing about this because of Azure is this can all work in a hybrid -based scenario as well. In other words, I could I can have on-premises servers like you see here. And then I could also have VMs hosted in the cloud.

Now, granted, you want bandwidth. If you’re going to do that, most of the time companies will do what is called ExpressRoute with Azure. ExpressRoute means we have a dedicated line through a telecommunication company directly into Azure. And so if our on-premises version goes down, we’ve still got it up in Azure. And so, Azure can completely do clustering as well, obviously.

Now, that whole witness thing you see there that gets into something called the quorum. Quorum a is a legal term. You’ll hear basically it involves agreeing in voting. It’s like when you have a jury that has to vote on whether or not somebody’s guilty or not guilty or whatever. That’s the idea. Well, this is a voting system that clustering uses. And each node in a cluster gets a certain number of votes. And the voting simply means that this decides whether or not the cluster stays on line. And usually what you’ve got to do is you’ve got to have half of the services online for the cluster to remain working.

So, if we come over here, back over to this picture here, you can see that for the Redmond Site to stay online, we need at least half of our services up and running. Right. And that’s pretty simple because there’s only two, right? What if there are four? What if there are six? Right. Then if there are four, you’d have to have at least two of the servers up and running, usually. If there were six, you’d have to have at least three of the servers up and running. If you didn’t, then it would fell over to Bellevue, if that’s how many servers you were dealing with. All right. The quorum is what decides that.

Now, the tricky part about that. There’re different ways of using the quorum. And I’ll explain that in just a moment. Ultimately, though, these communications that are happening between all the servers, they utilize this thing called quorum to make that decision whether or not should the cluster remain active, remaining its accepting connections or should it fail. And again, in a geographically-dispersed type scenario like this stretch clustering, the idea would be if Redmond doesn’t have enough services up and running, then let’s failover to Bellevue or vice versa. Right. If Bellevue was the main site or whatever.

So, then you’ll see the term here called disk witness or file share witness. This is going to be used in a situation. So, a witness is a disk that the clustered servers can communicate with. And if they can communicate with that disk, that disk gets a vote on whether or not to keep the service up and running. And it’s usually used in a scenario where there is an even number of votes, for example. Looking back over here at the Azure failover concept, you’ll notice there’s a file called a file share witness. It is just instead of a hard drive being what’s called the witness, it’s a file share, maybe, on a file server that gets to be the witness and it gets a vote on whether or not the cluster service stays up and running. And it’s used if there is an even number of servers in this case, there is an even number of servers. And so when there is an even number of servers, you need something else to be the deciding vote on whether or not to keep the service up. Right. And that’s the idea. If there were only three servers, for example, then the way this would work is as long as two of the three servers were up and running, the cluster service would keep going and accepting clients. If two of the servers failed by default, then the cluster would go down and the idea would be, well, there’s only one server left, it’s taking too much load. We needed to shut the servers down. It’s going to be inactive. And you might say, “Well, what if one server is enough to handle all that load?” Well, I’ll explain that in just a second.

But the other thing that can happen and the reason why the file share witness is so important is what happened if for some reason VM1 and VM2 in this scenario could no longer see each other or if you go over here to this diagram and you say, “Well, Server1 and server two, something has occurred on the network and they can’t talk to each other.” Both of them are going to think that they’re down. And so they’re going to both want to fail over and take connections. What will happen, though, is that disk that you see there, that would act as the deciding factor too. Do both VMs have connectivity with that file share witness? Chances are probably not one of them won’t if there’s some kind of a connectivity issue. So, it could be that VM1 has lost connectivity on the network, but VM2 still has it. So, VM2 can still communicate with the file share witness. So, therefore, VM2 would take the failover. VM1 would realize it lost connectivity with the file share witness, so it’s not going to accept load anymore until whatever the problem is it gets fixed. That’s the idea.

Now, when you work with quorum, there are different strategies for how quorum gets configured in clustering. Now, by default in these newer versions of server, it uses what’s known as a dynamic quorum, which basically means that when you set clustering up it, it does a check and it decides what’s going to be the best model based on how many servers you have and all that. And if you’ve got an even number of servers, then it will use a file share witness. And if you have an odd number of servers, it doesn’t need a witness, doesn’t need that extra vote. So, if you look here, this kind of sums it all up. Node majority recommended for clusters with an odd number of nodes. This, by the way, is what you can manually set it to if you can just leave it set to dynamic quorum, which is the default. But you can manually set it as well. So, if node majority tells you it can sustain failures of half the nodes rounding up minus one, for example, a seven node cluster can sustain three node failures.

Another option would be to do a node and disk majority. This means you’re going to have a disk that’s going to get a vote. And so recommended for clusters with an even number of nodes can sustain the failures of half the nodes rounding up. If the disk witness remains online. For example, a six node cluster in which the disk witness is online would sustain three node failures, so it can sustain failures of half the nodes minus one of the disk witness. And then the node in file share majority is the same thing. It’s just instead of a dedicated hard drive that’s for the quorum, you’re using a file share which could be stored on a file server somewhere on the network if you want. And then the last one is node majority disk only. And this one is in a situation where you where you know that one server can handle all the load.

So, imagine if you had three servers normally if you went with any other option, if two of your servers failed, then that one server, even though it is completely, perfectly in good shape, that one server would go would, would get deactivated because it assumes that one is not enough to handle the load. But what if one server is technically enough to handle the load of three in a situation where, maybe, things are going to get slow, but at least things will still be up and running? For example, let’s say, it’s a SQL Server SQL database or something and we’ve dispersed the load amongst three servers active-active and two of the servers die. Well, the SQL database is going to go slow. It’s going to get slow all of a sudden, but, maybe, that one server can still at least keep the servers going up. So, the question would be some service or slow service is better than no service. Or is it? That’s the question you’d have to answer. And so if that’s the case, node majority would be the better way to go. It can sustain all nodes except one failure. So, if you had three servers, for example, two servers could still fail and that one server would remain active.

So, that is the idea of a core. And hopefully now you have an understanding too, of the idea of failover clustering and why it is it is a good thing and you can configure it yourself. I will also tell you I’m not getting into this in this video a whole lot, but I would like to say that if you’re doing Azure VMs and all that, there’s a lot of automation to this to where you don’t have to configure it. But if you are doing this on-premises and you’re mixing on-premises with the cloud, there’s definitely going to be some manual configuration that you’re going to have to do in order to set up failover clustering.

But ultimately, failover clustering is a great solution when it comes to providing that that redundancy, that fault tolerance and redundancy that we need when it comes to protecting our services in our environment.