Deploying VPC Prefix Lists with CloudFormation


Brett Gillett

25 minute read

Brett Gillett

25 minute read

Video Transcription

Okay, let’s let’s get started here.

What we’re gonna do tonight is actually work with relatively new feature. We’re gonna have a look at how we can create VPC prefix lists in Cloud Formation. I actually did a really short demo on this feature weeks ago when this was first announced at the end June. I guess it’s more than a couple weeks ago now, right?

I did a quick demo on this where we created a simple VPC prefix list through the console, because at the time, we couldn’t find a reference in CloudFormation on how to do this programmatically.

I was digging through some of the AWS news alerts just the other day and noticed that I was able to now do this through CloudFormation. So what we’re gonna do today is just a really simple demo, first we’ll build a simple Prefix List and then associate it with a VPC Security Group.

I had a look as we’re getting ready and I still can’t find a reference for route tables, but you can add the prefixes manually, but you can’t as far as I could tell today. Still, can’t do that, programmatically.

So for the time being, we’re gonna set up a simple prefix list for a couple of branch offices as a demo, and then we’ll associate that prefix list with a security group. That’s kind of the goal for todays walkthrough.

Let me just flip over here to the actual editor, right?

I’m ready to go. So I made myself a simple little project here, and we’ll just create file and I’m just gonna call it prefix YAML. Actually, I hate typing this. I’ve got it over here. I’m gonna grab my little starter script, right.

So we’re just gonna create a simple prefix list and, the very first thing we’re going to set up are some parameters, we’ll come back to those in a second and then resources.

So obviously, this is where we’re gonna spend the majority of our time today, and we’re just going to set up a logical name here. We’ll just call it prefix list. Actually, et’s make it a little more descriptive - we’ll call it branch office list.

We’ll define its type. Here we go and will define its properties. There’s a number of things here that are required. Some of them, some of them. Well, we’re gonna keep it really simple to get started. So we’ll set up the required pieces and then we’ll just add some tags where we need. We’re gonna create an IPv4 list, and then we have to add the actual entries.

I’m gonna leave this blank for now. We’ll come back to that in a second.

The interesting part of a prefix list is you have to set a maximum number of entries, and once that is set its immutable.

You can’t change it. So I looked in at the service limits or quotas - I guess they’re called now. And there were a couple things in there. I was just looking to see if there was an upper limit to the max entries and I couldn’t find it.

So, for our example, today, we’re just gonna get number up and say 25 and you also have to give it a name. And we’re just gonna use on intrinsic function here and jamming whatever we call this stack at the ends and we’ll call it, uh, branch pl. So let’s go back to the entries.

There’s two pieces of information we need for the entries. The first is the range. So let’s say this is our first branch office, and then we can give it a description. I can’t remember now if when I looked in the documentation, if description was required or not, but it makes a lot of sense to put it in, because if you’re anything like me in literally a week from now, I’d go back and look at this and have really no clue what the heck this was for. So I’m gonna go and give this a name will just say branch office number one.

Then you just define up to the maximum entries that you’ve created below Description Branch Office. Wow, I can’t type there we go. Branch office number two will add one more. Just so if we’ve got a good example here. So we’ve got three branch offices set up in the prefix list.

Here we go. Now. I mentioned that we’ll add some tags and this is just a standard tagging set up. So we tend to use a set of standard tags. So we’re going to say who owns this and I’ll just put myself as the default owner, right? How do you contact me? If you have a question about something in the template and then we usually use environment.

We’re going to say in here string and we’re going to say the default is, let’s say development and we’ll use allowed values because it’s so we would do it if we’re actually building this for somebody.

So production, let’s say development. I don’t know, Let’s say test. So we’ve got our three environments. I’m not quite sure why This is complaining. Why it’s showing me this. I think what’s happening here is it’s not finding the new prefix list definition.

I’ve got the AWS toolkit. Maybe I need update in PyCharm for this to work. So it I think it’s probably just not Yeah, see, it doesn’t know what this is, so that’s why I’m getting that error. We’ll see what happens when we try to do the deploy and then down here, we’ll go ahead and set up.

All right, we’ll do some references back to those parameters, uh, owner email, and then what? They say environment, right? I want to do a rough. Yeah, there we go. Okay, so at this point, in theory, we have a resource that defines our prefixed list.

So I think what we’ll do now is we’ll just deploy this first, and troubleshoot it. If I’ve got any typos in here or anything like that, we’ll find out if it likes this line 21 or not. And then what we’re gonna do is extend this by adding it to an actual will create a security group and then refer to this ah prefix list in that guy’s.

Let me just change my view here. Go back to the browser and we’re gonna go here, and we’re just gonna create a stack and we’ll do the upload of a template and I’ll find my prefix. So this will be our first indication if we did anything wrong.

Okay, so no giant red bar. So we’re good there if we get time here. I’m not really planning for this to be very long tonight, so maybe what we could do after we get this done is show you how we would do this if we were actually doing it for a customer.

Normally, we would dump all this into get lab and use a series of Well, a simple see I script. Really? And run. Um, some CLI commands for CloudFormation. So maybe we’ll do that at the end. But we’re going to call this prefix demo and here are my parameters that I set. You know, here’s my drop down list of my allowed values. I’ll leave it as development.

Doesn’t really matter, will click next again. We would do this, programmatically, but I’m gonna follow kind of the way we would do this for a customer. And we would always tag the stack itself, right? Trying to follow best practice and will save the environment for this is development.

Normally, we would have a tagging policy or something like that in the AWS organization service. So we want to make sure that you know, if we’re gonna tag, the resources will tag the stack as well. So we’ve got that. We’re not gonna change anything else down here in the advanced options.

But you might want a stack policy in those kinds of things were going to say. Next we get our preview screen, we’re going to scroll to the bottom and will create a stack. So let’s see what this does as we’re waiting, I’ll get another tab open and will jump over to the VPC service here in a second.

There we go. All right. Success. Pretty, basic templates. So I’m not surprised that we got lucky there and got that to deploy the first time. So if we go over to VPC go down the left-hand side of the console here, you see manage prefix lists, you’ll see a couple in here already.

Those are AWS managed prefix lists - one for S3 and one for DynamoDB. And then here’s our new prefix list.

So if we highlight this, you can see the definitions, right?

Its IPv4 and prefix lists supports both v4 and v6. You can’t mix and match so it has to be one address family per prefix list. So if you did have a v6 in your environment, you have to create two separate prefix lists. It’s also version controlled, which is pretty cool so you could switch back if you’ve made a mistake, our Max entries, who owns it all that kind of great stuff.

There’s the unique identifier for this prefix list, you can see the entries. So here are three branch offices right now. It’s not associated to anything. You can also share a prefix list through the resource access manager. We won’t poke around at that tonight, but you can do this so you can share these prefix lists across your entire organization.

There’s the tags that we set up right? Just kind of following best practise about tagging all the resources where it’s possible.

Let’s jump back into PyCharm and what we’re gonna do now is add a simple security group and use a reference to this branch office list we just created as a way to add an entry or at it.

All right, let me just drag this, so I can see my editor. We’ll come up with a really creative name as well. We’ll just call it Branch Security Group Zero, and we’ll just define this as a security group and then to find the properties so I’ve got Oh, actually, one other thing. We’re gonna blow this away, and we’re gonna actually deploy it from scratch the whole thing to see it work.

This is normally how I might build out a template. I build it in stages to make sure that everything works rather than having to troubleshoot a large file. So if I want to deploy this all together in one shot, I know that the branch office prefix list has to exist before the security group.

I’m going to set up a DependsOn attribute here just to make sure that when we deploy this ah, second time from scratch that confirmation instead of deploying the branch office prefix list and our security group potentially in parallel that it knows to do the prefix list first, right?

I just refer here to the logical name of that. And now we can go ahead and define, the attributes off this security groups. Then we have to go ahead and create the actual ingress rules. Here we go. This will give this a description and will say, allow TCP 22 from branch offices. Nothing too fancy, Right? And will say from Port 22 to Port 22. The IP Protocol is TCP. This is the new reference that I noticed just a couple days ago in the security ingress security group Ingress rules.

Now this is where we’re going to refer back to the branch office. Our security group has to be associated to something, So we need a VPC ID. What we’re gonna do is we’re gonna another parameter up here and we’re going to say VPC ID.

And we’re gonna I’m gonna cut and paste this type because again, I’ll mess this up. We’re going to use a special parameter type in AWS parameter type. I don’t wanna have to type out the idea of this vpc right, because most likely gonna mess it up when I try to do it.

Istead, this will give us a list of all the VP sees in the region where we’re gonna deploy this and it gives us a drop down list. I’ll just steal all these tags from up here. It will dump that down there. I’m know. I’m sure this is user, but PyCharm always does this weird indent.

At this point, we’ve defined are prefixed list. And now we’ve defined a security group associated a security group with our VPC and in this security group were saying, allow 22 basically SSH, right, inbound from the prefix list which includes right now these three ranges right are fake ranges for our subnets or sorry our branch offices.

Let’s save this. I’ll flip back to my browser. Let’s drag this work and see easily and we’ll go back to CloudFormation.

We’re gonna do is something a little different here just to demonstrate it. If we were deploying this for real we would always generate a change set. if you’re not familiar, a change set is away to similuate the deployment - like a dry run.

What we’re gonna do is we’re gonna compare the currently deployed template against the changes that we’ve just made to the template and CloudFormation will tell us what it’s gonna change. Is it gonna add some things were gonna modify something. And really, what we’re looking for here is potential situations where a resource has to be replaced that we don’t want it to bereplaced.

You’ll see here in a second, we’re going to create a change set and we’ll say, replace the current template and will just re upload my prefix template and say next. Okay, no errors again. Here’s the new parameter that we have to pass through. So here’s my drop down, Right. So I use that AWS parameter set up, and I’m just going to put it in our super important Minecraft VPC and we’ll click next.

All of this remains the same. We’re not gonna worry about changing any of this. Our summary slide and then you’ll see down here Change set. So what’s gonna happen is it’s going to do that comparison for us. Shouldn’t take very long. So you see, pending here we’re fresh a couple times.

It’s already done so you can see what it’s saying is it’s basically has an ad action. We’re going to create a new ecurity group and really, what I’m looking for over here This is the column that I’m most concerned about. I want to know Is this change going to cause a replacement happen? It says true, false or conditional? Something like that.

I’m really looking for false here. If it says conditional, we would dig into it a little bit more just to make sure we understand exactly what’s gonna happen and for happy with this, we can actually execute the change set and then I’ll drop us back to the event. This probably shouldn’t take very long. There’s our security group and we’re done right.

Now if we go back over here and we’ll filter on our Minecraft VPC just to limit what we can see and we go down to our security groups, we should have that new VPC created. What did I call it now? Already I forget right there, right at the very top. Missed it.

Prefix demo Branch SG and you can see in our inbound rules what we have now instead of having to create the source like 10.10.0.0, 10.20.0.0, 10.30.0.0 you get the idea. Now we just refer to this prefix list, which includes all of those branch offices.

Bow we could go ahead and associate that security group with an easy to instance, right? The only thing that’s running in here that should be running is that Minecraft server will go up, will say, uh, networking change security group and we will add this guy.

We’re now allowing inbound ssh from that range or several ranges of IP addresses that we defined in the prefix list. So I think this is really a fantastic way to simplify things and also make it easier. I was talking to somebody today and they were having a problem where team members, because, you know, it’s just everybody’s in a rush to get things done.

They were just creating let’s say less than optimal security groups, a lot of any-any rules, and the thought was that potentially, by using prefix lists, now they don’t have to think about the range is that they need. They could just simply say, Oh, I want to allow off won’t allow access from the branch offices or I want to allow access from customer X right? Then you could just have those prefix lists pre-defined.

The users could either create templates like we did to access them quickly, or you could put them in your CloudFormation templates to do it programmatically. I prefer CloudFormation templates, but that’s just because I’ve been writing templates for a very long time now.

That’s pretty much it. In a nutshell on how you could take a prefix list, define it in a confirmation template and then associate it with a in my in our example here, a security group. I was hoping that we could do a route table as well, but unfortunately, when I was looking through the documentation this afternoon, I didn’t notice yet a reference in the route table documentation that allows you to point to ah prefix list as a target, right.

I’m guessing at some point that will happen because you can do it here already right If we go back, to the VPC console, we can see your target now. How did I do this before? Maybe I’m getting confused. Maybe I’m in the wrong spot for this.

I could have sworn that it was supported by rote tables as well. But I could be wrong, so it might be something you want to look up?

I mentioned at the beginning that we would normally deploy this through, uh, CLI commands. So let’s have a look at that and see if we can get that working as well, rather than doing this through the management console.

All right. So only drag this browser window out of the way again. Yeah. I must be looking in the wrong spot cause I’ve got the announcement for prefixed lists still open here and it does say that security groups and route tables are used to control access customers, blah, blah, blah.

You can group multiple CIDRs in a single object and use it as a reference group for both security groups and route tables.

So I’m just mustn’t have seen it there in that console of you somehow. So you could do you can do both. So what’s the easiest way to do this? Let’s just create another file and we’ll just call it deploy SH for now. What we would normally do is validate the templates before deploying them so we could just use AWS CloudFormation validate template.

Then I can’t spell Jeez, and refer to the file. In this case, it’s just prefixed dot Jamel and I have, ah, number off profiles set up in my AWS config file. Might my credentials. So I’m going to use this profile. Assuming that that is going to work, we could then deploy it.

We’re going to say deploy And this I definitely have to look up because there’s a number of options. Just bear with me a sec. Let me show you what I’m looking at. So we would normally use the deploy Command.

You can see it’s got a number of options here on it. Okay, and I’ll show you kind of what we normally do for this. So we’re gonna say template file and again pointed to the local prefix CMO file that we’ve got here and then we’re going.

We don’t have to worry about overriding any parameters, but if we wanted to, we could. We could say something like Parameter overrides and we’ll say environments just to demonstrate this equals Let’s say we made it development, I think the first time.

So let’s go production now and then we want to do no execute change set. So essentially, what this allows us to do is don’t actually execute the deploy. Just create the change set. This is also a great way to practice some separation of duty. You could have one person or a group of people that are responsible for creating templates on deploying those templates.

Then maybe the project owner reviews those proposed changes through the change set process before the changes were actually deployed. So this is going to generate a change set. There’s the proper spelling. The other thing that we want to add here is no fail on empty change set. Now, we don’t really need this in this particular situation, but if we were putting this in a larger see I script in our get lab deployment where we’re going through and saying deploy the VPC, then deploy the security groups, then deploy I don’t know the databases.

Let’s say the RTs DB instances and we’ve deployed that already. So we’ve got the infrastructure all set up and we want to make a change to I don’t know, a security group if we didn’t have the no fail on empty change set. What happens in the sea I script is let’s say the VPC goes first and there’s no changes detected. It would actually create a failed change. Set it well. It’s going to create a failed change that regardless, but it would basically send an error code back and stop our build process. So we use this no fail on empty change set as a way to say,

Hey, we understand that there’s no changes for the VPC. That’s great, Just keep going and then it’s gonna go to the next template in our deploy, which would be, let’s, say, our security group and work through that good and then we’ve got well, we would we would set tags up. I’m not gonna do that here, but we would do the same thing, right?

That’s the way we can actually tag the the CloudFormation Stack itself. So these are the two scripts that we would normally kind of create. The other thing we would do in here is is something like cloud formation. Oh, you know what? I missed something here. I need to give it a stack name. There’s the template and it’s stack name. And we called this already forget. Give me a sec. When I called just prefix demo. I think it was or something that Yeah, prefix demo.

So I don’t want to deploy a new stack. I want basically to make changes to the existing stack. So in this case, I’ll just hard code the name in here. And then what we like to do if we’re deploying this in production is we would normally want to set termination protection on the CloudFormation stack. So we can do that as well in a simple CLI script.

If I can quickly find it here. You see if I can I think this is it. Yeah, okay. Enable termination protection and then give the stack name. So in this case, we’ve got it. Prefix demo. The other thing that we would do here is just to demonstrate this. We normally use waiters so I can go in and say AWS CloudFormation. Um, wait. And I want to say that stack exists. So before setting termination protection, wait for the stack to exist. And again, you know, in the CLI script.

We would use these all as, um, variables, right? Just to make it a little easier. So there we go. We’ve got our validate script, and then we’ve got three CLI commands that should deploy this.

So let’s see what happens here. I’m actually gonna just try to run this right here in the command line. So we’re gonna do the now that have written them down. Oh, there we go. So I forgot this, right? We need to use the correct profile.

Otherwise it’s gonna deploy in the wrong account. So profile is co-dev, and we’ll just grab these, then apply them to these commands as well.

So let’s do this just now that we’ve got them typed out this way. If we run into any problems, we can try it again.

Is not gonna let me paste. Um, copy. Here we go. They’re OK. So it didn’t like the way that I’ve put in the reference to the file. So let me just find that this is the validate template command - sorry, Validate Template. It helps if you actually type the full command.

Let’s try this again. There we go. Last time I tried the control, see, it didn’t work. So let’s do this way. Oh, yeah. I’m using cross account access to get toward development account, so I just got to get token here. Bear with me a sec.

So I think I messed that up. Let’s try to see what happens, all right? I’m not sure what happened there. That wasn’t the output I was expecting. I’ve just recently upgraded updated to the version two of the AWS CLI. Let’s actually do this too. This might give us something that’s a little easier to look at.

So this is what I was expecting. So no errors, right? I don’t know why it’s doing the page there for some reason, but that means the validate template worked. And now what we’re gonna do is we’re gonna run are deployed commands.

Let me copy that. You actually just pull this up here a bit and then running this paste and I’ll do the same thing.

We got an error invalid template path, so I must have done the same thing. And I just find the deploy command template file dash dash, template file. There’s a slight difference. And I always forget what it is between the validate command in the deploy commands.

Let me just get the half words located. Hang on one sec. I mean, I’m gonna cheat, and we’re gonna grab this out of an existing file, so just give me a second. Let’s see where I messed this up. Okay? We’ve got them to find like this in another script.

So let’s try this and see template file. Okay, The rest of it looks pretty good, so we’ll do the same thing. Copy. Go back down here and paste and a record. Okay. Oh, must have values. So yeah, we were not actually setting the VPC ID.

So you go like this. And in this case, I do have to hard code it because I don’t have the drop down list. So we’re gonna find the VPC ID of that Minecraft VPC, and we’ll just paste that in. Okay, so there’s that. Try this again.

We’ll grab that line. It doesn’t like the keyboard shortcut for that for some reason. Copy. There we go. So now what’s happening? Let me just flip back to the browser. What’s happening is it just generated a change set. So if we go over to the stack and here’s our prefixed demo, if we go in, what we’re gonna find is there is no change sets because we didn’t actually change anything in that template.

But we know that it works now. Okay, so let’s go through. The rest of the process will set up, uh, termination protection. And this will give you an indication of kind of how we do this.

Normally, I think this one’s relatively painless command. Let’s see if I at least get one of these three commands, right? No, I didn’t either Enable Terminate off. It’s supposed to be protection. Try it like that. There we go.

Ok, so Now, if we’re still not sure why it’s doing that, so that works. Okay, so we’re good. So what we’re gonna do is we’re gonna actually make a change now and we’ll run. We’ll just run this deploy command again to simulate the, um, the actual change set.

So if we go back to prefix, probably the easiest thing to do is let’s say we bring on a new branch location, so and its range is 10.40.40.0/24 and we’ll give it a name. There we go. So if we were doing this in our CI script, the first thing that we would do would be to validate that template. And what we should get back here is just the JSON.

They’re saying unable to proceed. Now what we’ll do is we’ll run the deploy command change sets, and as that’s running, we’ll flip back to the browser. Did I type something wrong here?

This I was expecting to see, actually fire a change set file CO dev, I’m just having a look at the command prefix demo.

I wonder if we just try something here. I might have my I might have My profile set to Northern Virginia is the default. Yeah, OK, so that’s the problem. See, it’s creating it over here. So let’s go back, to Canada and what we’ll do is we’re just will override the setting in our credentials and conig file, and I’ll just say, region ca-central-1.

Now what? I’m hoping to see if we go to prefix demo. Go to change set - there we go. That’s what I was expecting to see.

So now we have a change set created and we can go in here and you can see Now we’re modifying the branch office list and we’re also modifying the security group. And as I was mentioning, this is the most important column over here.

The replacement column equals false and if you wanted to actually see what it’s doing, you can go under the JSON document and you can go through and see kind of what’s been changed. So we’re gonna execute this always takes longer when you’re waiting for it.

It’s just cleaning up. Here we go. Clean up process and the updates completes. Now, if we go back over the VPC we look at manage prefixed lists, it’s like this. Now you can see we’ve got version two of our prefixed list and here’s the entries.

So now we’ve added Oh, and I messed up that range pretty good, but you get the idea, right? So it gives you a little bit of an idea on how to define a prefix list as part of a CloudFormation template.

Then how to add that that prefix list once you’ve defined it into a security group and also a little ad hoc look at how we would normally put this into a get lab. See I script as a way to deploy their push these updates automatically when somebody pushes an update into the get lab solution or the get lab repository, I should say so. That’s kind of what I wanted to run through today.

Thanks very much for joining me. Appreciate that. I know everybody’s busy - I hope you learned something and you can follow us here if you’re not already following that be great.

The other thing is we have a YouTube channel, which I’m gonna post in the ah, the channel page here. So if you’re interested in catching some other videos or longer kind of tutorials and things like that, I suggest subscribing to our YouTube channel as well. Thanks, everybody. Bye for now.


Orbit

Like what you read? Why not subscribe to the weekly Orbit newsletter and get content before everyone else?