Aws Azure Pdf.pdf

AWS & MS AZURE MATERIAL (EC2 / DB SERVICES / VPC) (AZURE STORAGE AND DATABASES)

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected]

EC2 ADMINISTRATION Contents 1. Introducing EC2! a. EC2 use cases 2. Introducing images and instances a. Understanding images b. Understanding instances c. EC2 instance pricing options d. Note e. Working with instances f. Configuring your instances g. Launching instances using the AWS CLI h. Note 3. Cleaning up! 4. Recommendations and best practices 5. Understanding EBS volumes a. Tip b. EBS volume types

Database Services in Amazon Web Services Table of Contents 1. An overview of Amazon RDS a. RDS instance types b. Multi-AZ deployments and Read Replicas 2. Working with Amazon RDS a. Getting started with MySQL on Amazon RDS 3. Recommendations and best practices 4. Database Services Use Cases 5. Creating a database with automatic failover a. Getting ready b. How to do it... c. How it works... d. There's more... Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected]

6. Migrating a database a. Getting ready b. How to do it... c. How it works... d. There's more...

AWS NETWORKING WITH VPC Contents 1. Introducing Amazon Web Services a. AWS architecture and components 2. An overview of Amazon VPC a. VPC limits and costs 3. Working with VPCs a. VPC deployment scenarios b. Getting started with the VPC wizard c. Launching instances in your VPC 4. Planning next steps 5. Best practices and recommendations 6. VPC Cloud Formation Recipes 7. Building a secure network a. Getting ready b. How to do it... c. How it works... d. There's more... 8. Creating a NAT gateway a. Getting ready b. How to do it... c. How it works... 9. Network logging and troubleshooting a. Getting ready b. How to do it... c. How it works... d. There's more...

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected]

AZURE STORAGE AND DATABASES Contents 1. 2. 3. 4.

An introduction to Microsoft Azure Storage Why Azure Storage? Terminologies Azure Storage types a. Durability b. Performance c. Persistency 5. Azure Storage accounts a. General-purpose storage accounts b. Azure Storage Account tips c. Creating an Azure Storage account 6. Automating your tasks a. Azure PowerShell b. Azure command-line interface 7. Delving into Azure Storage 8. Azure Storage services a. Blob storage b. Table storage c. Queue storage d. File storage 9. Mount the Azure File share with File Explorer 10. Mount the Azure File share with PowerShell 11. Use Azure Files with Linux 12. Prerequisites for mounting an Azure file share with Linux and the cifsutils package 13. Mount the Azure file share on-demand with 14. Create a persistent mount point for the Azure file share with /etc/fstab 15. Prerequisites for mounting an Azure File share on macOS 16. Mount an Azure File share via Finder 17. Mount an Azure File share via Terminal 18. Storage design for highly available applications a. RA-GRS b. Azure Backup Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected]

c. Azure Site Recovery d. Premium Storage 19. Automating tasks a. Creating Blob storage using PowerShell b. Creating Blob storage using the Azure CLI 2.0 c. Creating Table storage using PowerShell d. Creating Table storage using the Azure CLI 2.0 e. Creating Queue storage using PowerShell f. Creating Queue storage using the Azure CLI 2.0 20. Azure Storage for VMs 21. An introduction to Azure VMs a. Azure VMs series 22. Storage considerations for Azure VMs a. Managed versus unmanaged disks b. VM disks 23. Capturing VMs a. Sysprepping the VM b. Capturing the VM with managed storage c. Capturing the VM with unmanaged storage 24. Automating the tasks a. Creating an Azure VM using the Azure CLI 2.0 b. Adding data disks to an Azure VM using the Azure CLI 2.0 c. Resizing Azure VM disks using the Azure CLI 2.0 d. Capturing the VM using the Azure CLI 2.0 25. Further information 26. Implementing Azure SQL Databases 27. An introduction to Azure SQL Database 28. Why Azure SQL Database? 29. Service tiers a. Elastic database pools b. Single databases c. Service tier types 30. Azure SQL Database business continuity a. How business continuity works for Azure SQL Database 31. Automating the tasks a. Creating an Azure SQL Database using the Azure CLI 2.0 b. Creating an SQL Server-level firewall rule using Azure CLI 2.0 Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected]

32. SQL Database (IaaS/PaaS) a. Azure SQL Database (PaaS) b. SQL on Azure VMs (IaaS) 33. Azure SQL elastic database pools a. Creating an elastic database pool b. Adding a database to the elastic database pool 34. Active geo-replication a. Implementing active geo-replication 35. Automating the tasks a. Creating an elastic database pool using Azure CLI 2.0 b. Adding an additional database to the elastic database pool using Azure CLI 2.0

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected]

EC2 Administration Contents Introducing EC2! ...................................................................................................................................... 2 EC2 use cases ....................................................................................................................................... 2 Introducing images and instances ............................................................................................................ 3 Understanding images .......................................................................................................................... 3 Understanding instances....................................................................................................................... 6 EC2 instance pricing options ............................................................................................................... 8 Note....................................................................................................................................................... 9 Working with instances ...................................................................................................................... 11 Configuring your instances ................................................................................................................ 11 Launching instances using the AWS CLI .......................................................................................... 12 Note..................................................................................................................................................... 15 Cleaning up! ........................................................................................................................................... 15 Recommendations and best practices .................................................................................................... 16 Understanding EBS volumes ................................................................................................................. 17 Tip ....................................................................................................................................................... 18 EBS volume types .............................................................................................................................. 18

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected]

Introducing EC2! EC2 is a service that basically provides scalable compute capacity on an on-demand, pay-peruse basis to its end users. Let's break it up a bit to understand the terms a bit better. To start with, EC2 is all about server virtualization! And with server virtualization, we get a virtually unlimited capacity of virtual machines or, as AWS calls it, instances. Users can dynamically spin up these instances, as and when required, perform their activity on them, and then shut down the same while getting billed only for the resources they consume. EC2 is also a highly scalable service, which means that you can scale up from just a couple of virtual servers to thousands in a matter of minutes, and vice versa—all achieved using a few simple clicks of a mouse button! This scalability accompanied by dynamicity creates an elastic platform that can be used for performing virtually any task you can think of! Hence, the term Elastic Compute Cloud! Now that's awesome!

With virtually unlimited compute capacity, you also get added functionality that helps you to configure your virtual server's network, storage, as well as security. You can also integrate your EC2 environment with other AWS services such as IAM, S3, SNS, RDS, and so on. To provide your applications with add-on services and tools such as security, scalable storage and databases, notification services, and so on and so forth.

EC2 use cases Let's have a quick look at some interesting and commonly employed use cases for AWS EC2: •

Hosting environments: EC2 can be used for hosting a variety of applications and software, websites, and even games on the cloud. The dynamic and scalable environment allows the compute capacity to grow along with the application's needs, thus ensuring better quality of service to end users at all times. Companies such as Netflix, Reddit, Ubisoft, and many more leverage EC2 as their application hosting environments.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected]

•

Dev/Test environments: With the help of EC2, organizations can now create and deploy large scale development and testing environments with utmost ease. The best part of this is that they can easily turn on and off the service as per their requirements as there is no need for any heavy upfront investments for hardware.

•

Backup and disaster recovery: EC2 can be also leveraged as a medium for performing disaster recovery by providing active or passive environments that can be turned up quickly in case of an emergency, thus resulting in faster failover with minimal downtime to applications.

•

Marketing and advertisements: EC2 can be used to host marketing and advertising environments on the fly due to its low costs and rapid provisioning capabilities.

•

High Performance Computing (HPC): EC2 provides specialized virtualized servers that provide high performance networking and compute power that can be used to perform CPU-intensive tasks such as Big Data analytics and processing. NASA's JPL and Pfizer are some of the companies that employ the use of HPC using EC2 instances.

Introducing images and instances Almost all IT companies today run their workloads off virtualized platforms such as VMware vSphere or Citrix XenServer to even Microsoft's Hyper-V. AWS, too, got into the act but decided to use and modify a more off the shelf, open sourced Xen as its virtualization engine. And like any other virtualization technology, this platform was also used to spin up virtual machines using either some type of configuration files or some predefined templates. In AWS's vocabulary, these virtual machines came to be known as instances and their master templates came to be known as images.

Understanding images As discussed earlier, images are nothing more than preconfigured templates that you can use to launch one or more instances from. In AWS, we call these images Amazon Machine Images (AMIs). Each AMI contains an operating system which can range from any modern

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected]

Linux distro to even Windows Servers, plus some optional software application, such as a web server, or some application server installed on it. It is important, however, to understand a couple of important things about AMIs. Just like any other template, AMIs are static in nature, which basically means that once they are created, their state remains unchanged. You can spin up or launch multiple instances using a single AMI and then perform any sort of modifications and alterations within the instance itself. There is also no restriction on the size of instances that you can launch based on your AMI. You can select anything from the smallest instance (also called as a micro instance) to the largest ones that are generally meant for high performance computing. Take a look at the following image of EC2 AMI:

Secondly, an AMI can contain certain launch permissions as well. These permissions dictate whether the AMI can be launched by anyone (public) or by someone or some account which I specify (explicit) or I can even keep the AMI all to myself and not allow anyone to launch instances from it but me (implicit). Why have launch permissions? Well, there are cases where some AMIs can contain some form of propriety software or licensed application, which you do not want to share freely among the general public.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected]

In that case, these permissions come in really handy! You can alternatively even create something called as a paid AMI. This feature allows you to share your AMI to the general public, however, with some support costs associated with it. AMIs can be bought and sold using something called as the AWS Marketplace as well—a one stop shop for all your AMI needs! Here, AMIs are categorized according to their contents and you as an end user can choose and launch instances off any one of them. Categories include software infrastructure, development tools, business and collaboration tools, and much more! These AMIs are mostly created by third parties or commercial companies who wish to either sell or provide their products on the AWS platform. AMIs can be broadly classified into two main categories depending on the way they store their root volume or hard drive: •

EBS-backed AMI: An EBS-backed AMI simply stores its entire root device on an Elastic Block Store (EBS) volume. EBS functions like a network shared drive and provides some really cool add on functionalities like snapshotting capabilities, data persistence, and so on. Even more, EBS volumes are not tied to any particular hardware as well. This enables them to be moved anywhere within a particular availability zone, kind of like a Network Attached Storage (NAS) drive. We shall be learning more about EBS-backed AMIs and instances in the coming chapter.

•

Instance store-backed AMI: An instance store-backed AMI, on the other hand, stores its images on the AWS S3 service. Unlike its counterpart, instance store AMIs are not portable and do not provide data persistence capabilities as the root device data is directly stored on the instance's hard drive itself. During deployment, the entire AMI has to be loaded from an S3 bucket into the instance

store, thus making this type of deployment a slightly slow process. The following image depicts the deployments of both the instance store-backed and EBS-backed AMIs. As you can see, the root and data volumes of the instance store-backed AMI are stored locally on the HOST SERVER itself, whereas the second instance uses EBS volumes to store its root device and data.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected]

The following is a quick differentiator to help you understand some of the key differences between EBbacked and Instance store-backed AMIs:

Understanding instances Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected]

So far we have only being talking about images; so now let's shift the attention over to instances! As discussed briefly earlier, instances are nothing but virtual machines or virtual servers that are spawned off from a single image or AMI. Each instance comes with its own set of resources, namely CPU, memory, storage, and network, which are differentiated by something called as instance families or instance types. When you first launch an instance, you need to specify its instance type. This will determine the amount of resources that your instance will obtain throughout its lifecycle. AWS currently supports five instance types or families, which are briefly explained as follows: •

General purpose: This group of instances is your average, day-to-day, balanced instances. Why balanced? Well, because they provide a good mix of CPU, memory, and disk space that most applications can suffice with while not compromising on performance. The general purpose group comprises the commonly used instance types such as t2.micro, t2.small, t2.medium, and the m3 and m4 series which comprises m4.large, m4.xlarge, and so on and so forth. On average, this family contains instance types that range from 1 VCPU and 1 GB RAM (t2.micro) all the way to 40 VCPUs and 160 GB RAM (m4.10xlarge).

•

Compute optimized: As the name suggests, these are specialized group of instances that are commonly used for CPU-intensive applications. The group comprises two main instances types, that is, C3 and C4. On an average, this family contains instances that can range from 2 VCPUs and 2.75 GB RAM (c4.large) to 36 VCPUs and 60 GB RAM (c4.8xlarge).

•

Memory optimized: Similar to the compute optimized, this family comprises instances that require or consume more RAM than CPU. Ideally, databases and analytical applications fall into this category. This group consists of a single instance type called R3 instances, and they can range anywhere from 2 VCPUs and 15.25 GB RAM (r3.large) to 32 VCPUs and 244 GB RAM (r3.8xlarge).

•

Storage optimized: This family of instances comprises specialized instances that provide fast storage access and writes using SSD drives. These instances are also used for high I/O performance and high disk throughput applications. The group also comprises two main instance types, namely

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected]

the I2 and D2 (no, this doesn't have anything to do with R2D2!). These instances can provide SSD enabled storage ranging from 800 GB (i2.large) all the way up to 48 TB (d2.8xlarge)—now that's impressive! •

GPU instances: Similar to the compute optimized family, the GPU instances are specially designed for handling high CPU-intensive tasks but by using specialized NVIDIA GPU cards. This instance family is generally used for applications that require video encoding, machine learning or 3D rendering, and so on. This group consists of a single instance type called G2, and it can range between 1 GPU (g2.2xlarge) and 4 GPU (g2.8xlarge). To know more about the various instance types and their use cases, refer to http://aws.amazon.com/ec2/instance-types/.

EC2 instance pricing options Apart from the various instance types, EC2 also provides three convenient instance pricing options to choose from, namely on-demand, reserved, and spot instances. You can use either or all of these pricing options at the same time to suit your application's needs. Let's have a quick look at all three options to get a better understanding of them.

On-demand instances Pretty much the most commonly used instance deployment method, the on-demand instances are created only when you require them, hence the term on-demand. On-demand instances are priced by the hour with no upfront payments or commitments required. This, in essence, is the true pay-as-you-go payment method that we always end up mentioning when talking about clouds. These are standard computational resources that are ready whenever you request them and can be shut down anytime during its tenure. By default, you can have a max of 20 such on-demand instances launched within a single AWS account at a time. If you wish to have more such instances, then you simply have to raise a Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected]

support request with AWS using the AWS Management Console's Support tab. A good use case for such instances can be an application running unpredictable workloads, such as a gaming website or social website. In this case, you can leverage the flexibility of on-demand instances accompanied with their low costs to only pay for the compute capacity you need and use and not a dime more!

Note On-demand instance costs vary based on whether the underlying OS is a Linux or Windows, as well as in the regions that they are deployed in.

Reserved instances Deploying instances using the on-demand model has but one slight drawback, which is that AWS does not guarantee the deployment of your instance. Why, you ask? Well to put it simply, using on-demand model, you can create and terminate instances on the go without having to make any commitments whatsoever. It is up to AWS to match this dynamic requirement and make sure that adequate capacity is present in its datacenters at all times. However, in very few and rare cases, this does not happen, and that's when AWS will fail to power on your on-demand instance. In such cases, you are better off by using something called as reserved instances, where AWS actually guarantees your instances with resource capacity reservations and significantly lower costs as compared to the on-demand model. You can choose between three payment options when you purchase reserved instances: all upfront, partial upfront, and no upfront. As the name suggests, you can choose to pay some upfront costs or the full payment itself for reserving your instances for a minimum period of a year and maximum up to three years. Consider our earlier example of the t2.micro instance costing $0.0013 per hour. The following table summarizes the upfront costs you will need to pay for a period of one year for a single t2.micro instance using the reserved instance pricing model:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected]

Reserved instances are the best option when the application loads are steady and consistent. In such cases, where you don't have to worry about unpredictable workloads and spikes, you can reserve a bunch of instances in EC2 and end up saving on additional costs

Spot instances Spot instances allow you to bid for unused EC2 compute capacity. These instances were specially created to address a simple problem of excess EC2 capacity in AWS. How does it all work? Well, it's just like any other bidding system. AWS sets the hourly price for a particular spot instance that can change as the demand for the spot instances either grows or shrinks. You as an end user have to place a bid on these spot instances, and when your bid exceeds that of the current spot price, your instances are then made to run! It is important to also note that these instances will stop the moment someone else out bids you, so host your application accordingly. Ideally, applications that are non-critical in nature and do not require large processing times, such as image resizing operations, are ideally run on spot instances. Let's look at our trusty t2.micro instance example here as well. The on-demand cost for a t2.micro instance is $0.013 per hour; however, I place a bid of $0.0003 per hour to run my application. So, if the current bid cost for the t2.micro instance falls below my bid, then EC 2 will spin up the requested t2.micro instances for me until either I choose to terminate them or someone else out bids me on the same—simple, isn't it? Spot instances compliment the reserved and on-demand instances; hence, ideally, you should use a mixture of spot instances working on-demand or reserved instances just to be sure that your application has some compute capacity on standby in case it needs it.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected]

Working with instances Okay, so we have seen the basics of images and instances along with various instance types and some interesting instance pricing strategies as well. Now comes the fun part! Actually deploying your very own instance on the cloud! In this section, we will be using the AWS Management Console and launching our very first t2.micro instance on the AWS cloud. Along the way, we shall also look at some instance lifecycle operations such as start, stop, reboot, and terminate along with steps, using which you can configure your instances as well. So, what are we waiting for? Let's get busy!

Configuring your instances Once your instances are launched, you can configure virtually anything in it, from packages, to users, to some specialized software or application, anything and everything goes! Let's begin by running some simple commands first. Go ahead and type the following command to check your instance's disk size: df –h

Here is the output showing the configuration of the instance:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected]

You should see an 8 GB disk mounted on the root ( /) partition, as shown in the preceding screenshot. Not bad, eh! Let's try something else, like updating the operating system. AWS Linux AMIs are regularly patched and provided with necessary package updates, so it is a good idea to patch them from time to time. Run the following command to update the Amazon Linux OS: sudo yum update -y

Why sudo? Well, as discussed earlier, you are not provided with root privileges when you log in to your instance. You can change that by simple changing the current user to root after you login; however, we are going to stick with the ec2-user itself for now. What else can we do over here? Well, let's go ahead and install some specific software for our instance. Since this instance is going to act as a web server, we will need to install and configure a basic Apache HTTP web server package on it. Type in the following set of commands that will help you install the Apache HTTP web server on your instance: sudo yum install httpd # sudo service httpd start # sudo chkconfig httpd on

Launching instances using the AWS CLI So far, we have seen how to launch and manage instances in EC2 using the EC2 dashboard. In this section, we are going to see how to leverage the AWS CLI to launch your instance in the cloud! For this exercise, I'll be using my trusty old CentOS 6.5 machine

Stage 1 – create a key pair aws ec2 create-key-pair --key-name \ > --output text > .pem

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected]

Once the key pair has been created, remember to change its permissions using the following command: chmod 400 .pem

Stage 2 – create a security group # aws ec2 create-security-group --group-name <SG_Name> \ > --description "<SG_Description>"

For creating security groups, you are only required to provide a security group name and an optional description field along with it. Make sure that you provide a simple yet meaningful name here:

Once executed, you will be provided with the new security group's ID as the output. Make a note of this ID as it will be required in the next few steps.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected]

Stage 3 – add rules to your security group With your new security group created, the next thing to do is to add a few firewall rules to it. We will be discussing a lot more on this topic in the next chapter, so to keep things simple, let's add one rule to allow inbound SSH traffic to our instance. Type in the following command to add the new rule: aws ec2 authorize-security-group-ingress --group-name <SG_Name> \ > --protocol tcp --port 22 --cidr 0.0.0.0/0

To add a firewall rule, you will be required to provide the security group's name to which the rule has to be applied. You will also need to provide the protocol, port number, and network CIDR values as per your requirements:

Stage 4 – launch the instance With the key pair and security group created and populated, the final thing to do is to launch your new instance. For this step, you will need a particular AMI ID along with a few other key essentials such as your security group name, the key pair, and the instance launch type, along with the number of instances you actually wish to launch. Type in the following command to launch your instance: # aws ec2 run-instances --image-id ami-e7527ed7 \ > --count 1 --instance-type t2.micro \ > --security-groups <SG_Name> \ > --key-name

And here is the output of the preceding commands: Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected]

Note In this case, we are using the same Amazon Linux AMI (ami-e7527ed7) that we used during the launch of our first instance using the EC2 dashboard.

The instance will take a good two or three minutes to spin up, so be patient! Make a note of the instance's ID from the output of the ec2 run-instance command. We will be using this instance ID to find out the instance's public IP address using the EC2 describe-instance command as shown:

# aws ec2 describe-instances --instance-ids

Cleaning up! Spinning up instances is one thing; you should also know how to stop and terminate them! To perform any power operations on your instance from the EC2 dashboard, all you need to do is select the particular instance and click on the Actions tab as shown. Next, from the Instance State submenu, select whether you want to Stop, Reboot, or Terminate your instance, as shown in the following screenshot:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected]

It is important to remember that you only have instance stopping capabilities when working with EBS-backed instances. Each time an EBS-backed instance is stopped, the hourly instance billing stops too; however, you are still charged for the EBS volume that your instance is using. Similarly, if your EBS-backed instance is terminated or destroyed, then by default the EBS root volume attached to it is also destroyed, unless specified otherwise, during the instance launch phase.

Recommendations and best practices •

First and foremost, create and use separate IAM users for working with EC2. DO NOT USE your standard root account credentials!

•

Use IAM roles if you need to delegate access to your EC2 account to other people for some temporary period of time. Do not share your user passwords and keys with anyone.

•

Use a standard and frequently deployed set of AMIs as they are tried and tested by AWS thoroughly.

•

Make sure that you understand the difference between instance store-backed and EBS-backed AMIs. Use the instance store with caution and remember that you are responsible for your data, so take adequate backups of it.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected]

•

Don't create too many firewall rules on a single security group. Make sure that you apply the least permissive rules for your security groups.

•

Stop your instances when not in use. This will help you save up on costs as well.

•

Use tags to identify your EC2 instances. Tagging your resources is a good practice and should be followed at all times.

•

Save your key pairs in a safe and accessible location. Use passphrases as an added layer of security if you deem it necessary.

•

Monitor your instances at all times. We will be looking at instance monitoring in depth in the coming chapters; however, you don't have to wait until then! Use the EC2 Status and Health Check tabs whenever required.

Understanding EBS volumes First up, let's understand EBS volumes a bit better. EBS volumes are nothing more than block-level storage devices that you can attach to your EC2 instances. They are highly durable and can provide a host of additional functionalities to your instances, such as data persistence, encryption, snapshotting capabilities, and so on. Majority of the time, these EBS volumes are used for storing data for a variety of applications and databases, however you can use it just as a normal hard drive as well. The best part of EBS volumes is that they can persist independently from your instances. So powering down an instance or even terminating it will not affect the state of your EBS volumes. Your data will stay on it unless and until you explicitly delete it. Let's look at some of the key features and benefits that EBS volumes have to offer: •

High availability: Unlike your instance store-backed drives, EBS volumes are automatically replicated by AWS within the availability zone in which they are created. You can create an EBS volume and attach it to any instance present in the same availability zone; however, one EBS

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected]

volume cannot be attached to multiple instances at the same time. A single instance, however, can have multiple EBS volumes attached to it at any given time. •

Encryption capabilities: EBS volumes provide an add-on feature using which you can encrypt your volumes using standard encryption algorithms, such as AES-256, and keys as well. These keys are autogenerated the first time you employ encryption on a volume using the AWS Key Management Service (KMS). You can additionally even use IAM to provide fine-grained access control and permissions to your EBS volumes.

•

Snapshot capabilities: The state of an EBS volume can be saved using point-in-time snapshots. These snapshots are all stored incrementally on your Amazon S3 account and can be used for a variety of purposes, such as creating new volumes based on an existing one, resizing volumes, backup and data recovery, and so on.

Tip EBS volumes cannot be copied from one AWS region to another. In such cases, you can take a snapshot of the volume and copy the snapshot over to a different region using the steps mentioned at http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-copy-snapshot.html.

EBS volume types There are three different types of EBS volumes available today, each with their own sets of performance characteristics and associated costs. Let's briefly look into each one of them and their potential uses: •

General purpose volumes (SSD): These are by far the most commonly used EBS volume types as they provide a good balance between cost and overall performance. By default, this volume provides a standard 3 IOPS per GB of storage, so a 10 GB general purpose volume will get approximately 30 IOPS and so on so forth, with a max value of 10,000 IOPS. You can create general purpose volumes that can range in size from 1 GB to a maximum of 16 TB. Such volumes

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected]

can be used for a variety of purposes, such as instance root volumes, data disks for dev and test environments, database storage, and so on. •

Provisioned IOPS volumes (SSD): These are a specialized set of SSDs that can consistently provide a minimum of 100 IOPS burstable up to 20,000 IOPS. You can create Provisioned IOPS Volumes that range in size from a minimum of 4 GB all the way up to 16 TB. Such volumes are ideally suited for applications that are IO intensive, such as databases, parallel computing workloads such as Hadoop, and so on.

•

Magnetic volumes: Very similar to traditional tape drives and magnetic disks, these volumes are a good match for workloads where data is accessed infrequently, such as log storage, data backup and recovery, and so on. On an average, these volumes provide up to a 100 IOPS with an ability to burst up to 1,000 IOPS. You can create Magnetic volumes that range in size from a minimum of 1 GB all the way up to 1 TB.

Getting started with EBS Volumes Now that we have a fair idea of what an EBS Volume is, let's look at some simple ways that you can create, attach, and manage these volumes. To view and access your account's EBS Volumes using AWS Management Console, simply select the Volumes option from the EC2 dashboard's navigation pane, as shown here:

Each EBS-backed instance's volume will appear here in the Volume Management dashboard. You can use this same dashboard to perform a host of activities on your volumes, such as create, attach, detach, and monitor performance, to name a few.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected]

You can view any particular EBS Volume's details by simply selecting it and viewing its related information in the Description tab, as shown. Here, you can view the volume's ID, Size, Created date, the volume's current State as well as its Attachment information, which displays the volume's mount point on a particular instance. Additionally, you can also view the volume's health and status by selecting the Monitoring and Status Checks tab, respectively. For now, let's go ahead and create a new volume using the volume management dashboard.

Creating EBS volumes From the Volume Management dashboard, select the Create Volume option. This will pop up the Create Volume dialog box as shown here:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected]

Fill in the details as required in the Create Volume dialog box. For this tutorial, I went ahead and created a simple 10-GB general purpose volume: •

Type: From the Type drop-down list, select either General Purpose (SSD), Provisioned IOPS (SSD), or Magnetic as per your requirements.

•

Size (GiB): Provide the size of your volume in GB. Here, I provided 10 GB.

•

IOPS: This field will only be editable if you have selected Provisioned IOPS (SSD) as the volume's type. Enter the max IOPS value as per your requirements.

•

Availability Zone: Select the appropriate availability zone in which you wish to create the volume. Remember, an EBS volume can span availability zones, but not regions.

•

Snapshot ID: This is an optional field. You can choose to populate your EBS volume based on a third party's snapshot ID. In this case, we have left this field blank.

•

Encryption: As mentioned earlier, you can choose whether or not you wish to encrypt your EBS Volume. Select Encrypt this volume checkbox if you wish to do so.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected]

•

Master Key: On selecting the Encryption option, AWS will automatically create a default key pair for the AWS's KMS. You can make a note of the KMS Key ID as well as the KMS Key ARN as these values will be required during the volume's decryption process as well.

Once your configuration settings are filled in, select Create to complete the volume's creation process. The new volume will take a few minutes to be available for use. Once the volume is created, you can now go ahead and attach this volume to your running instance.

Attaching EBS volumes Once the EBS volume is created, make sure it is in the available state before you go ahead and attach it to an instance. You can attach multiple volumes to a single instance at a time, with each volume having a unique device name. Some of these device names are reserved, for example, /dev/sda1 is reserved for the root device volume. You can find the complete list of potential and recommended device names at http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/device_naming.html. The following screenshot shows the option of attaching a volume:

To attach a volume, select the volume from the Volume Management dashboard. Then select the Actions tab and click on the Attach Volume option. This will pop up the Attach Volume dialog box, as shown:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected]

Type in your instance's ID in the Instance field and provide a suitable name in the Device field as shown. In this case, I provided the recommended device name of /dev/sdf to this volume. Click on Attach once done. The volume attachment process takes a few minutes to complete. Once done, you are now ready to make the volume accessible from your instance.

Accessing volumes from an instance Once the volume is attached to an instance, you can basically format it and use it like any other block device. To get started, first up connect to your running instance using putty or any other SSH client of your choice. Next, type in the following command to check the current disk partitioning of your instance: sudo df -h

Next, run the following command to list out partitions on your current instance. You should see a default /dev/xvdapartition along with its partition table and an unformatted disk partition with the name /dev/xvdf as shown in the following screenshot. The /dev/xvdf command is the newly added EBS volume that we will need to format in the upcoming steps: sudo fdisk -l

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected]

Once you have verified the name of your newly added disk, you can go ahead and format with a filesystem of your choice. In this case, I have gone ahead and used the ext4 filesystem for my new volume: sudo mkfs -t ext4 /dev/xvdf

Now that your volume is formatted, you can create a new directory on your Linux instance and mount the volume to it using your standard Linux commands: sudo mkdir /my-new-dir # sudo mount /dev/xvdf /my-new-dir

Here is the screenshot of creating new directory using the preceding commands:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected]

Here's a useful tip! Once you have mounted your new volume, you can optionally edit the Linux instance's fstabfile and add the volume's mount information there. This will make sure that the volume is mounted and available even if the instance is rebooted. Make sure you take a backup copy of the fstab file before you edit it, just as a precautionary measure.

Detaching EBS volumes Detaching EBS volumes is a fairly simple and straightforward process. You will first need to unmount the volume from your instance and then simply detach it using Volume Management dashboard. Run the following command to unmount the EBS volume from the instance: sudo umount /dev/sdf

Note Make sure you are unmounting the correct volume from the instance. Do not try and unmount the /dev/sda or any other root partitions.

Once the volume is successfully unmounted from the instance, detach the volume by selecting the Detach Volume option from the Actions tab, as shown here:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected]

Managing EBS volumes using the AWS CLI You can create, attach, and manage EBS volumes using the AWS CLI as well. Let's go ahead and create a new EBS volume using the AWS CLI. Type in the following command: # aws ec2 create-volume \ --size 5 --region us-west-2 --availability-zone us-west-2a \ --volume-type gp2

Note The --volume-type attribute accepts any one of these three values: gp2: General Purpose instances (SSD) io1: Provisioned IOPS volumes (SSD) standard: Magnetic volumes

The following code will create a 5 GB General Purpose volume in the us-west-2a availability zone. The new volume will take a couple of minutes to be created. You should see a similar output as the following screenshot. Make a note of the new volume's Volume ID before proceeding to the next steps.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected]

Now that the new volume is created, we can go ahead and attach it to our instance. Type in the following command: # aws ec2 attach-volume \ --volume-id vol-40993355 \ --instance-id i-53fc559a \ --device /dev/sdg

The following command will attach the volume with the volume ID vol-40993355 to our instance (i53fc559a), and the device name will be /dev/sdg. Once again, you can supply any meaningful device name here, but make sure that it abides by AWS's naming conventions and best practices as mentioned in http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/device_naming.html. Once the volume is attached to your instance, the next steps are pretty easy and straightforward. First format the new volume with a suitable filesystem of your choice. Next up, create a new directory inside your instance and mount the newly formatted volume on it. Voila! Your volume is now ready for use.

You can detach and delete the volume as well using the AWS CLI. First up, we will need to unmount the volume from the instance. To do so, type in the following command in your instance: Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected]

unmount /dev/sdg

Make sure you are unmounting the correct volume and not the root partition. Once the volume is unmounted, simply detach it from the instance using the following AWS CLI code: aws ec2 detach-volume \ --volume-id vol-40993355

The output of the preceding command is as follows:

Finally, go ahead and delete the volume using the following AWS CLI code: # aws ec2 delete-volume \ --volume-id vol-40993355

Remember that you cannot delete volumes if they are attached or in use by an instance, so make sure that you follow the detachment process before deleting it.

Backing up volumes using EBS snapshots Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected]

We do know for a fact that AWS automatically replicates EBS volumes so that your data is preserved even in case the complete drive fails. But this replication is limited only to the availability zone in which the drive or EBS volume was created, which means if that particular availability zone was to go down for some reason, then there is no way for you to back up your data. Fortunately for us, AWS provides a very simple yet highly efficient method of backing EBS volumes, called as EBS snapshots. An EBS snapshot in simple terms is a state of your volume at a particular point in time. You can take a snapshot of a volume anytime you want. Each snapshot that you take is stored incrementally in Amazon S3, but, you will not be able to see these snapshots in your S3 buckets; they are kind of hidden away and stored separately. You can achieve a wide variety of tasks using snapshots. A few are listed as follows: •

Create new volumes based on existing ones: Snapshots are a great and easy way to spin up new volumes. A new volume spawned from a snapshot is an exact replica of the original volume, down to the last detail.

•

Expand existing volumes: Snapshots can also be used to expand an existing EBS Volume's size as well. It is a multistep process, which involves you taking a snapshot of your existing EBS volume and creating a larger new volume from the snapshot.

•

Share your volumes: Snapshots can be shared within your own account (private) as well publicly.

•

Backup and disaster recovery: Snapshots are a handy tool when it comes to backing up your volumes. You can create multiple replicates of an existing volume within an AZ, across AZs that belong to a particular region, as well as across regions, using something called an EBS Snapshot

copy mechanism. To create a snapshot of your volumes, all you need to do is select the particular volume from the Volume Management dashboard. Click on the Actions tab and select the Create Snapshot option, as shown here:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected]

Tip It is really a good practice to stop your instance before taking a snapshot if you are taking a snapshot of its root volume. This ensures a consistent and complete snapshot of your volume at all times.

You should see the Create Snapshot dialog box as shown in the following screenshot. Provide a suitable Name and Description for your new snapshot. An important thing to note here is that this particular snapshot is not supporting encryption, but why? Well, that's simple! Because the original volume was not encrypted, neither will the snapshot be encrypted. Snapshots of encrypted volumes are automatically encrypted. Even new volumes created from an encrypted snapshot are encrypted automatically. Once you have finished providing the details, click on Create to complete the snapshot process:

You will be shown a confirmation box, which will display this particular snapshot's ID. Make a note of this ID for future reference.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected]

The new snapshot will take a good 3–4 minutes to go from Pending to Completed. You can check the status of your snapshot by viewing the Status as well as the Progress fields in the Description tab, as shown here:

Once the snapshot process is completed, you can use this particular snapshot and Create Volume, Copy this snapshot from one region to another, and Modify Snapshot Permissions to private or public as you see fit. These options are all present in the Actions tab of your Snapshot Management dashboard:

But for now, let's go ahead and use this snapshot to create our very first AMI. Yes, you can use snapshots to create AMIs as well. From the Actions tab, select the Create Image option. You should see the Create Image from EBS Snapshot wizard as shown here. Fill in the required details and click on Create to create your very first AMI:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected]

The details contain the following options: •

Name: Provide a suitable and meaningful name for your AMI.

•

Description: Provide a suitable description for your new AMI.

•

Architecture: You can either choose between i386 (32 bit) or x86_64 (64 bit).

•

Root device name: Enter a suitable name for your root device volume. Ideally, a root device volume should be labelled as /dev/sda1 as per EC2's device naming best practices.

•

Virtualization type: You can choose whether the instances launched from this particular AMI will support Paravirtualization (PV) or Hardware Virtual Machine (HVM) virtualization.

•

RAM disk ID , Kernel ID: You can select and provide your AMI with its own RAM disk ID (ARI) and Kernel ID (AKI); however, in this case I have opted to keep the default ones.

•

Block Device Mappings: You can use this dialog to either expand your root volume's size or add additional volumes to it. You can change the Volume Type from General Purpose (SSD) to Provisioned IOPS (SSD) or Magnetic as per your AMI's requirements. For now, I have left these to their default values.

Click on Create to complete the AMI creation process. The new AMI will take a few minutes to spin up. In the meantime, you can make a note of the new AMI ID from the Create Image from EBS Snapshot confirmation box, as shown:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected]

You can view your newly created AMI under the AMIs option from the EC2 dashboard's navigation pane:

So, here we have it! You very own AMI, created and ready to use. s

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected]

Database Services in Amazon Web Services Table of Contents An overview of Amazon RDS....................................................................................... 3 RDS instance types............................................................................................... 5 Multi­AZ deployments and Read Replicas..............................................................7 Working with Amazon RDS......................................................................................... 9 Getting started with MySQL on Amazon RDS.......................................................11 Recommendations and best practices........................................................................33 Database Services Use Cases...................................................................................34 Creating a database with automatic failover................................................................34 Getting ready...................................................................................................... 35 How to do it......................................................................................................... 36 How it works....................................................................................................... 40 There's more....................................................................................................... 41 Migrating a database................................................................................................ 41 Getting ready...................................................................................................... 41 How to do it......................................................................................................... 42 How it works....................................................................................................... 50 There's more....................................................................................................... 51

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 1 *

An overview of Amazon RDS Before we go ahead and dive into the amazing world of RDS, it is essential to  understand what exactly AWS provides you when it comes to database­as­a­service  offerings and how can you effectively use them. To start off, AWS provides a bunch of  awesome and really simple­to­use database services that are broadly divided into two  classes: the relational databases, which consist of your MySQL and Oracle databases, and the non­relational databases, which consist of a propriety NoSQL database  similar to MongoDB. Each of these database services is designed by AWS to provide  you with the utmost ease and flexibility of use along with built­in robustness and fault  tolerance. This means that all you need to do as an end user or a developer is simply  configure the databases service once, run it just as you would run any standard  database without worrying about the internal complexities of clustering, sharding, and  so on, and only pay for the amount of resources that you use! Now that's awesome,  isn't it! However, there is a small catch to this! Since the service is provided and maintained  by AWS, you as a user or developer are not provided with all the fine tuning and  configuration settings that you would generally find if you were to install and configure  a database on your own. If you really want to have complete control over your  databases and their configurations, then you might as well install them on EC2  instances directly. Then you can fine­tune them just as you would on any traditional  OS, but remember that in doing so, you will have to take care of the database and all  its inner complexities.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 2 *

With these basic concepts in mind, let us go ahead and learn a thing or two about  Amazon Relational Database Service (RDS). Amazon RDS is a database service  that basically allows you to configure and scale your popular relational databases such as MySQL and Oracle based on your requirements. Besides the database, RDS also  provides additional features such as automated backup mechanisms, point­in­time  recovery, replication options such as multi­AZ deployments and Read Replicas, and  much more! Using these services you can get up and running with a completely  scalable and fault tolerant database in a matter of minutes, all with just a few clicks of  a button! And the best part of all this is that you don't need to make any major  changes to your existing applications or code. You can run your apps with RDS just as you would run them with any other traditional hosted database, with one major  advantage: you don't bother about the underlying infrastructure or the database  management. It is all taken care of by AWS itself! RDS currently supports five popular relational database engines, namely MySQL,  Oracle, Microsoft's SQL Server, PostgreSQL, and MariaDB as well. Besides these,  AWS also provides a MySQL­like propriety database called Amazon Aurora. Aurora is  a drop­in replacement for MySQL that provides up to five times the performance that a standard MySQL database provides. It is specifically designed to scale with ease  without having any major consequences for your application or code. How does it  achieve that? Well, it uses a combination of something called as an Aurora Cluster  Volume, as well as one Primary Instance and one or more Aurora Replicas. The  Cluster Volume is nothing more than virtual database storage that spans across  multiple AZs. Each AZ is provided with a copy of the cluster data so that the database  is available even if an entire AZ goes offline. Each cluster gets one Primary Instance  that's responsible for performing all the read/write operations, data modifications, and  so on. With the Primary Instance, you also get a few Aurora Replicas (also like  Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 3 *

Primary Instances). A Replica can only perform read operations and is generally used  to distribute the database's workload across the Cluster. You can have up to 15  Replicas present in a Cluster besides the Primary Instance, as shown in the following  image:

Note: You can also read more on Amazon Aurora  at http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Aurora.html  .

With this basic information in mind, let us now understand some of RDS's core components  and take a look at how RDS actually works.

RDS instance types To begin with, RDS does operate in a very similar way as EC2. Just as you have EC2  instances configured with a certain amount of CPU and storage resources, RDS too has  instances that are spun up each time you configure a database service. The major difference  between these instances and your traditional EC2 ones is that they cannot be accessed 

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 4 *

remotely via SSH even if you want to. Why? Well, since it's a managed service and everything is provided and maintained by AWS itself, there is no need for you to SSH into them! Each  instance already has a particular database engine preinstalled and configured in it. All you  need to do is select your particular instance type and assign it some storage, and voila! There  you have it! A running database service of your choice in under 5 minutes! Let's have a quick  look at some of the RDS instance types and their common uses: 

Micro instances ( db.t1.micro ): Just as we have micro instances in our EC2  environments, the same is also provided for RDS as well. Each database micro instance is  provided with just 1 CPU and approximately 600 MB of RAM, which is good enough if you  just want to test RDS or play around with it. This instance type, however, is strictly not  recommended for any production­based workloads at all. Along with this particular micro  instance, RDS also provides a slightly better instance type in the form of a db.m1.small,  which provides 1 CPU with a slightly better 1.7 GB RAM.



Standard instances ( db.m3 ): Besides your micro instances, RDS provides a standard  set of instance types that can be used on a daily basis for moderate production workloads.  This class of instance provides up to 8 CPUs and about 30 GB of RAM as well, but more  importantly, these instances are specially created for better network performance as well.



Memory optimized ( db.r3 ): As the name suggests, this instance class provides really  high­end, memory optimized instances that are capable of faster performance and more  computing capacity as compared to your standard instance classes. This instance class  provides a maximum of 32 CPUs with a RAM capacity of up to 244 GB along with a  network throughput of 10 GB/second.



Burst capable ( db.t2 ): This instance class provides a baseline performance level with  the ability to burst to full CPU usage if required. This particular class of database instance,  however, can only be launched in a VPC environment. The maximum CPU offered in this  category is up to 2 CPUs with approximately 8 GB of RAM.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 5 *

Along with an instance type, each RDS instance is also backed by an EBS volume. You  can use this EBS volume for storing your database files, logs, and lots more! More  importantly, you can also select the type of storage to go with your instances as per your  requirements. Here's a quick look at the different storage types provided with your RDS  instances: 

Magnetic (standard): Magnetic storage is an ideal choice for applications that have a light to moderate I/O requirement. A magnetic volume can provide up to 100 IOPS  approximately on average with burst capability of up to hundreds of IOPS. The disk sizes  can range anywhere between 5 GB to 3 TB. An important point to note here, however, is  that since magnetic storage is kind of shared, your overall performance can vary  depending on the overall resource usage by other customers as well.



General purpose (SSD): These are the most commonly used storage types from the lot  and are generally a good choice of storage if you are running a small to medium­sized  database. General purpose or SSD­backed storage can provided better performance as  compared to your magnetic storage at much lower latencies and higher IOPs. General  purpose storage volumes can provide a base performance of three IOPS/GB and have the  ability to burst up to 3,000 IOPS as per the requirements. These volumes can range in size from 5 GB to 6 TB for MySQL, MariaDB, PostgreSQL, and Oracle DB instances, and from  20 GB to 4 TB for SQL server DB instances.



Provisioned IOPs: Although general purpose volumes are good for moderate database  workloads, they are not a good option when it comes to dedicated performance  requirements and higher IOPs. In such cases, provisioned IOPs are the best choice of  storage type for your instances. You can specify IOPs anywhere between the values 1,000 all the way up to 30,000 depending on the database engine you select as well as the  amount of disk size that you specify. A MySQL, MariaDB, PostgreSQL, or Oracle database

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 6 *

instance with approximately 6 TB of storage can get up to 30,000 IOPs. Similarly, an SQL  server DB instance with approximately 4 TB of disk size can get up to 20,000 IOPs. You cannot decrease the storage of your RDS instance once it is allocated to it.

Multi­AZ deployments and Read Replicas We all know the importance and the hard work needed to keep a database, especially  the one running a production workload up and running at all times. This is no easy feat, especially when you have to manage the intricacies and all the tedious configuration  parameters. But thankfully, Amazon RDS provides us with a very simple and easy­to­ use framework, using which tasks such as providing high availability to your  databases, clustering, mirroring, and so on are all performed using just a click of a  button! Let's take high availability for example. RDS leverages your region's availability zones  and mirrors your entire primary database over to some other AZ present in the same  region. This is called as a Multi­AZ deployment and it can easily be enforced using  the RDS database deployment wizard. How does it work? Well it's quite simple  actually. It all starts when you first select the Multi­AZ deployment option while  deploying your database. At that moment, RDS will automatically create and maintain  another database as a standby replica in some different AZ. Now if you use a MySQL,  MariaDB, Oracle, or PostgreSQL as your database engine, then the mirroring  technology used by RDS is AWS propriety. Whereas, if you go for an SQL server  deployment, then the mirroring technology used is SQL server mirroring by default.  Once the standby replica database instance is created, it continuously syncs up with  the primary database instance from time to time, and in the event of a database failure 

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 7 *

or even a planned maintenance activity, RDS will automatically failover from the  primary to the standby replica database instance within a couple of minutes:

Note: Amazon RDS guarantees an SLA of 99.95 percent! To know more about the RDS SLA agreement, refer to http://aws.amazon.com/rds/sla/  .

However remarkable and easy multi­AZ deployment may be, it still has some minor  drawbacks of its own. Firstly, you can't use a multi­AZ deployment for scaling out your  databases, and, secondly, there is no failover provided if your entire region goes down. With  these issues in mind, RDS provides an additional feature for our database instances called as  Read Replicas.

Read Replicas are database instances that enable you to offload your primary database  instance's workloads by having all the read queries routed through them. The data from your  primary instance is copied asynchronously to the read replica instance using the database  engine's built­in replication engine. How does it all work? Well it's very similar to the steps  required for creating an AMI from a running EC2 instance! First up, RDS will create a  snapshot based on your primary database instance. Next, this snapshot is used to span a  read replica instance. Once the instance is up and running, the database engine will then start the asynchronous replication process such that whenever a change is made to the data in the  primary, it gets automatically replicated over to the read replica instance as well. You can then connect your application to the new read replica and offload all your read queries to it! As of 

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 8 *

date, RDS supports only MySQL, MariaDB, and PostgreSQL database engines for read  replicas:

Note: You can create up to five Read Replicas for a given database instance.

You can additionally use these Read Replicas as a failover mechanism as well by deploying  read replicas in a different region altogether. The only downside to this is that you will have to  manually promote the replica as a primary when the latter fails. 

Working with Amazon RDS In this section, we are going to create our very first scalable database using the Amazon RDS  service. For simplicity, I will be deploying a simple MySQL database using the RDS  Management Console; however, you can use any of the database engines provided by RDS  for your testing purposes, including Oracle, MariaDB, PostgreSQL, as well as SQL Server.  Let's first examine our use case up to now:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 9 *

We have also created a separate private subnet in each AZ for hosting our database  instances. These subnets are named US­WEST­PROD­DB­1 (192.168.5.0/24) and US­ WEST­PROD­DB­2 (192.168.6.0/24), respectively. Another extremely important point here  is that the communication between the public subnets and the private subnets is also set up  using a combination of network ACLs as well as security groups With our subnets in place, the next thing to do is jot down the database's essential  configuration parameters as well as plan whether you want to leverage a Multi­AZ deployment and Read Replicas for your deployment or not. The configuration parameters include the  database name, the database engine's version to use, the backup and maintenance window  details, and so on. For this deployment, I will be deploying my database using the Multi­AZ  deployment option as well. Do note, however, that the Multi­AZ deployment scheme is not 

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 10 *

included in the AWS Free Tier eligibility and, hence, you will be charged for the same. To  know more about the costs associated with your RDS services, refer  to https://aws.amazon.com/rds/pricing/  . Once you have thoroughly planned out these details,  you can go ahead and start off with the actual deployment of the database.

Getting started with MySQL on Amazon RDS You can access and manage your RDS deployments by using the AWS CLI, the AWS  SDK, as well as the AWS Management Console. For this activity, we will be using the  AWS Management Console. Log on to your AWS account using the IAM credentials,  and from the AWS Management Console, select the RDS option from  the Database group, as shown in the following screenshot:

Next, from the RDS Management Dashboard, select the option Subnet Groups from the  navigation pane. A Subnet Group is an essential step toward setting up the security of your  database. For starters, a subnet group is a logical grouping or cluster of one or more subnets  that belong to a VPC; in this case, the cluster is of our two database subnets (US­WEST­ PROD­DB­1 and 2).When we first launch a DB Instance in a VPC, the subnet group is  responsible for providing the database instance with an IP address from a preferred subnet  present in a particular availability zone. To get started, provide a suitable Name and Description for your DB Subnet Group as shown in the following screenshot. Next, from the VPC ID drop­down list, select a VPC of your  choice. In my case, I have selected the US­WEST­PROD­1 VPC (192.168.0.0/16). Once your  VPC is selected, you can now add the required set of subnets to your DB Subnet Group. To 

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 11 *

do so, first select the preferred Availability Zone and its corresponding Subnet ID. Click  on Add to add your subnet to the DB Subnet Group:

Now as a good practice, provide at least two subnets that are present in different AZs for your  DB Subnet Group. For example, in my case, I have provided two private subnets that are  present in us­west­2a (US­WEST­PROD­DB­1) and us­west­2c (US­WEST­PROD­DB­2),  respectively. Click on Create when done. With this step complete, you can now go ahead and  create your first RDS database instance in your VPC!

Creating a MySQL DB instance Creating an RDS database instance involves a simple four­step process. To begin  with, select the Instances option from the navigation pane, as shown in the following  screenshot. Next, select the Launch DB Instance button to bring up the DB Launch  Wizard:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 12 *

Step 1 – Select Engine To get started, select the appropriate database engine of your choice. For our  scenario, I have selected the MySQL database; however, feel free to select any of the  database engines as per your requirements.

Step 2 – Production? Now here comes the fun part! RDS basically allows you to create a database based  on your requirements; for example, a production database with multi­AZ support and  provisioned IOPS storage or a simpler database that has none of these add­on  features. With Multi­AZ deployments, your DB instance is guaranteed with a monthly  uptime SLA of 99.95 percent! However, because of such high SLAs, Multi­AZ  deployments are not covered under the AWS Free Tier usage scheme. Click on Next  Step to continue, as shown in the following screenshot:

Step 3: Specify DB Details The next page of the wizard will help you configure some important settings for your  DB instance: 

License Model: Select the appropriate database License Model as per your database  engine's selection. MySQL databases have only one license model; that is, general­public­

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 13 *

license. Other propriety databases such as Oracle and SQL servers offer two license  modes: Licenses Included and the BYOL (Bring Your Own Licenses) model. With  licenses included, AWS provides the required license keys for your databases, so you  don't have to separately purchase one. Alternatively, you can even use the BYOL model to provide your own licenses or obtain new ones from the database provider itself. 

DB Engine Version: Select the appropriate DB Engine Version as per your requirements. RDS provides and supports a variety of database engine versions that you can choose  from. In this case, I have selected the MySQL database engine version 5.6.23 as shown:



DB Instance Class: From this dropdown list, select the appropriate class of DB instance  you wish to provide for your DB instance. For a complete list of supported instance class  types, refer  to http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.DBInstanceClass. html  .

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 14 *



Multi­AZ Deployment: Select Yes from the dropdown list to ensure a multi­AZ deployment for your database. Selecting No will create your DB instance only in a single availability  zone.



Storage typ e: Select an appropriate storage option from the dropdown list. In this case, I  have opted for General Purpose (SSD); however, you can also select  between Magnetic and Provisioned IOPS as well.



Allocate storage: Allocate some storage for your database instance. You can provide  anywhere between 5 GB to 6 TB.

With these basic configurations out of the way, configure your database's settings  as shown in the following screenshot:

Here are the parameters you need to provide in Settings panel: 

DB Instance Identifier: Provide a suitable name for your DB instance. This name will be a unique representation of your DB instance in the region it is getting deployed in. In my  case, I have provided the name US­WEST­PROD­DB.



Master Username: Provide a suitable username for your MySQL database. You will use  this username to log in to your DB instance once it is deployed.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 15 *



Master Password: Provide a strong password for your DB instance. You will use this  password to log in to your DB instance once it is deployed. You can provide a password  that's up to 41 characters long; however, do not provide the following characters in it:  (@, " , /).

With the settings configured, click on Next Step to proceed with your database's  configuration.

Step 4: Configure Advanced Settings The final step of configuring your database instance can be split up into three parts.  The first part involves the setting up of the DB instance's Network & Security, that  includes selecting the VPC along with the Subnet Group that we created a while  back. The second part involves configuring various database options such as the  database name, the database port number on which the application can connect to it,  and so on. The final part consists of the  database's Backup and Maintenance window details. Let's have a quick look at each  part a bit more in detail: 

VPC: Select the name of the VPC that will host your MySQL DB instance. You can  optionally select the option Not in VPC as well if you wish to deploy your DB instance in a  standard EC2 Classic environment.



Subnet Group: Select the newly created Subnet Group from the dropdown list, as shown  in the following screenshot:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 16 *



Publicly Accessible: You can optionally set your DB instance to have public connectivity  by selecting Yes from the Publicly Accessible dropdown list; however, as best practice,  avoid making your DB instances public at all times.



Security Group(s): There are two levels of security group that you can use here. The first  is a DB security group that is basically used to control access to your DB instances that are outside a VPC. When working with DB security groups, you only need to specify the subnet CIDR associated with your DB instance, and no DB port or protocol details are required.  The second is your traditional VPC security group that can be used to control access to  your DB instances that are present in a VPC. Here, however, you need to specify both  inbound and outbound firewall rules, each with associated port numbers and supported  protocols.



You can select one or more security groups here for your DB instance; in my  case, I have selected a VPC security group as shown in the previous  screenshot. Just remember to open up only the required ports whenever you  work with VPC security groups. In this case, I have opened up ports 3306  (MySQL) and 1433 (SQL Server).

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 17 *

Moving on to the second part of the Advanced Settings, we will now set up  the Database Options as shown in the following: 

Database Name: Provide a suitable database name here. RDS will not create and initialize any database unless you specify a name here.



Database Port: Provide the port number using which you wish to access your database.  MySQL's default port number is 3306:

Note: You will not be able to change the database port number once the DB instance is created.



DB Parameter Group: DB parameter groups are logical groupings of database engine  configurations that you can apply to one or more DB instances at the same time. RDS  creates a default DB parameter group that contains mostly AWS specific configuration  settings and default values. You cannot edit the default DB parameter group, so in order to make changes, you will have to create a DB parameter group of your own. In this case, I  have left it as the default value.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 18 *



Option Group: This option is similar to DB parameter groups in that they too provide and  support few additional configuration parameters that make it easy to manage databases;  for example, MySQL DB Instances support for Memcached and so on. RDS currently  supports option groups for Oracle, MySQL, and SQL Server database engines. To know  more about option groups, refer  to http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithOption Groups.html  .



Enable Encryption: RDS provides standard AES­256 encryption algorithms for encrypting data at rest. This includes your DB instance, its associated Read Replicas, DB Snapshots,  as well as the automated backups. An important point to note here is that encryption is not  supported on the t2.micro DB instances.

For encryption to work, you will need your DB instance to be one of the following  instance classes: Instance Type

Supported Instance Class

General purpose (M3) current generation

db.m3.medium db.m3.large db.m3.xlarge db.m3.2xlarge

Memory optimized (R3) current generation

db.r3.large db.r3.xlarge db.r3.2xlarge db.r3.4xlarge db.r3.8xlarge

Burst capable (T2) current generation

db.t2.large

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 19 *

In this case, we are not going to encrypt our DB instance, so select No from  the Enable Encryption field as shown in the previous screenshot. The final part of the Advance Settings page is the Backup and Maintenance window selection. Using this section, you can configure automated backups for your database  as well as provide designated maintenance windows for the same. You can set  the Backup Retention Period as well as the Backup window's Start  Time and Duration, as shown in the following screenshot. In my case, I have opted  for the backups to be taken at 12:00AM UTC. If you do not supply a backup window  time, then RDS will automatically assign a 30­minute backup window based on your  region. For example, the default backup time block for the US West (Oregon) region is 06:00 to 14:00 UTC. RDS will select a 30­minute backup window from this block on a  random basis:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 20 *

The same can be set for your Maintenance window as well. An additional feature  provided here is that you can choose whether or not the database should receive  automated minor version updates from AWS or not. These minor updates for the  database engine will be automatically installed on the database based on their  availability as well as the maintenance window's time frame. You can make  changes in these settings even after your DB instance is created; however,  remember that the backup window should not overlap the weekly maintenance  window for your DB instance. Once you have configured the settings, click  on Launch DB Instance to complete the launch process. The DB instance will take a good 2 to 3 minutes to spin up depending on whether  you have opted for the Multi­AZ deployment or not. You can check the status of  your newly created DB instance using the RDS management dashboard, as shown Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 21 *

in the following screenshot. Simply check the Status column for all the stat us  changes that occur while your DB instance is created:

Let's take a quick look at some of the states that a DB instance goes through  during its lifecycle: 

Creating: This is the first stage of any DB instance's lifecycle where the instance is  actually created by RDS. During this time, your database will remain inaccessible.



Modifying: This state occurs whenever the DB instance enters any modifications either set by you or by RDS itself.



Backing­up: RDS will automatically take a backup of your DB instance when it is first  created. You can view all your DB instance snapshots using the Snapshots option on the  navigation pane.



Available: This status indicates that your DB instance is available and ready for use. You  can now access your database remotely by copying the database's endpoint.

Note: To read the complete list of DB instance status messages, refer  to http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Overview.DBInstance.Status.html  .

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 22 *

Connecting remotely to your DB instance Once your DB Instance is in the Available state, you can now access your database  remotely from any other EC2 instance or even remotely from your desktop if you have  set the security groups right. In my case, I have launched a new EC2 instance in my  VPC, as shown in the following screenshot. The first thing to do is make sure you have the required MySQL client packages  installed on your web server EC2 instance. To do so, simply type in the following  commands as shown: sudo yum install mysql

With the client installed, you can now access your remote RDS database using the  following command: mysql -u -h -p

Substitute the values with your master username and password that you set for the  database during the Create DB Instance phase. Here,  is the  Endpoint (.xxxxxxxxxxxx.us-west-2.rds.amazonaws.com:3306 )  that is provided by each DB instance when it is created. If all goes well, you should  see the MYSQL command prompt. Go ahead and run a few MYSQL commands and  check whether your database was created or not: > show databases;

You can additionally connect your database with tools such as the MySQL workbench  as well. Just remember to provide your database's endpoint in the hostname field  followed by the master username and password. With the database connected 

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 23 *

successfully, you can now run a few simple tests just to make sure that the DB  instance is accessible and working as expected.

Testing your database In this section, I'm going to show you a simple exercise, using which you can test the  configurations and the working of your database, as well as your DB instance. First  up, log in to your database using the following command as done earlier: # mysql -u -h -p

Next, let's go ahead and create a simple dummy table called doge. Type the following  command in your MySQL prompt: CREATE TABLE doge ( idint(11) NOT NULL auto_increment, namevarchar(255), description text, sizeenum('small','medium','large'), date timestamp(6), PRIMARY KEY (id) );

Fill in some data in your newly created table using the following INSERT commands: INSERT INTO doge (name,description ,size,date) VALUES('Xena','Black Labrador Retreiver','medium',NOW()); INSERT INTO doge (name,description ,size,date) VALUES('Betsy','Browndachshund','medium',NOW()); INSERT INTO doge (name,description ,size,date) VALUES('Shark','Mix bread- Half dachshund','small',NOW());

Modifying your DB instances Once your DB Instances are created and launched, you can further modify them using two methods. The first method is by using the AWS CLI, where you can use  Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 24 *

the modify-db-instance command along with a bunch of options to specify and assign  new parameters and values to your DB instances. For example, we need to expand  the storage as well as the instance class of our DB instance so that it can  accommodate the growing database's needs. To do so, type in the following  command: # aws rds modify-db-instance --db-instance-identifier us-west-prod-db \ --allocate-storage 100 \ --db-instance-class db.m1.large

The preceding command will update the DB instance with the identifier us-west-proddb with 100 GB of disk space and change its instance class to db.m1.large as well. 

The CLI provides a host of additional parameters as well which you can use to  configure almost any aspect of your DB Instance, such as the master user's password, the preferred backup and maintenance window, the database engine versions, and so  on. You can find the complete list of parameters and their descriptions  at http://docs.aws.amazon.com/cli/latest/reference/rds/modify­db­instance.html  . Note: Changing the instance class of a DB instance will result in an outage, so plan the changes in  advance and perform them during the maintenance window only.

The second method of modifying the DB instances is by using the RDS Management  dashboard itself. Select your DB instance, and from the Instance Actions dropdown  list, select the Modify option, as shown in the following screenshot:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 25 *

Using the Modify page, you can change almost all configuration parameters of your  DB instance just as you would by using the CLI. You can optionally set the changes to take effect immediately as well by selecting the Apply Immediately checkbox. Note,  however, that by doing so, your DB instance will try to accept the made changes  instantly, which can cause outages and even performance degradation at certain  times. So as good practice, avoid setting this checkbox unless absolutely necessary.  Changes made otherwise are reflected in your DB instance during its next scheduled  maintenance window.

Backing up DB instances RDS provides two mechanisms using which you can perform backups of your  database instances as per your requirements. The first is an automated backup job  that can be scheduled to run at a particular backup job interval, preferably when the  database is at its least utilization point. This is something that we configured sometime back while creating our DB Instance. The second is a manual database instance  snapshot that you can perform at any point in time. Here's a look at both the  techniques in a bit more detail: 

Automated backup: Automated backups are conducted periodically by RDS on a daily  user configured backup window. These backups are kept stored by RDS until the backup's 

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 26 *

retention period doesn't expire. By default, your first new database instance will have these backups enabled for ease of use. You can use these backups to restore your database to  any point in time, down to the last second. The only thing that you need to be aware of is  the slight freeze in storage IO operations that occurs when RDS actually performs the  backups. 

DB snapshots: DB snapshots are point in­time snapshots that are created by you as and  when required. To create a DB Instance snapshot, select the Take Snapshot option from  the Instance Actions dropdown list, as shown in the following screenshot:

This will bring up the Take Snapshot page where all you need to do is provide a  suitable name for your snapshot and click on Take Snapshot to complete the  process. Alternatively, you can also use the AWS CLI for performing a manual DB instance  snapshot. Type in the following command: # aws rds create-db-snapshot -i -s <SNAPSHOT_NAME>

Once you have taken your DB instance snapshot, you can view them on the RDS  Management dashboard under the Snapshots page, as shown in the following screenshot:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 27 *

The snapshot dashboard allows you to perform various operations on your DB snapshots  including copying DB snapshots from one region to another for high availability, restoring the  state of a DB instance based on a particular snapshot, as well as options to migrate your  MySQL database completely over to the Amazon Aurora database engine!

Creating Read Replicas and promoting them We have already discussed the concept of read replicas in some depth, and how they  can be useful for offloading the read operations from your primary DB instance as well as providing a mechanism using which you can create and set up Read Replicas  across AWS regions. In this section, we are going to check out a few simple steps  using which you can create and set up read replicas for your own environment using  the RDS Management dashboard. To get started, first select your newly created database from the RDS dashboard, as  shown in the following screenshot. Next, using the Instance Actions tab, select  the Create Read Replica option:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 28 *

This will bring up the Create Read Replica DB Instance page, as shown in the following  screenshot. The page is pretty self­explanatory and easy to configure. Start off by selecting  an appropriate DB Instance Class from the dropdown list. You can alternatively select a high­ end DB instance class here as compared to the primary DB instance. Next, select a Storage  Type for your Read Replica DB instance. In this case, I have opted to go for the General  Purpose (SSD) volumes:

Next, select your primary DB instance as the source for your Read Replica using  the Read Replica Source dropdown list, and provide a suitable and unique name for  your Read Replica in the DB Instance Identifier field, as shown in the preceding  screenshot.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 29 *

Now comes the fun part where you actually get to specify where you wish to deploy  your Read Replica DB instance. Remember that you can have a maximum of five read replicas for a single primary DB instance, so ideally have your replicas spread out  across the AZs that are present in your operating region or even have them residing in a different region altogether. Select an appropriate Destination Region and its  corresponding Availability Zone. In this case, I have opted for the same region (US  West (Oregon)) as well as same AZ (us­west­2a) as my primary DB instance. Besides the placement of your replica instance, you will also need to make it a part of  your existing DB Subnet Group. Select the same subnet group as provided for your  primary DB instance from the Destination DB Subnet Group field, as shown in the  following screenshot. Leave the rest of the fields to their default values and click on  the Create Read Replica option:

Here's what will happen next. First up, RDS will start off by taking a snapshot of your primary  DB instance. During this process, the DB instance will face a brief moment of IO freeze which  is an expected behavior. Here's a handy tip that you can use to avoid the IO freeze! Deploy  your DB instances using the multi­AZ deployment. Why? Well, because when it comes to  taking the snapshot, RDS will perform the snapshot on the primary DB instance's standby  copy, thereby not affecting any performance on your primary DB instance. Once the snapshot 

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 30 *

is taken, RDS will start to spin up a new Read Replica based on your specified configurations, as shown in the following screenshot. During the replica's creation phase, your primary DB instance will change from a backing up  state to modifying, and ultimately back to available status once the Replica is launched. Each Replica will behave as a DB instance on its own; hence, each of them will be provided with a  unique DB endpoint as well. Refer to the following screenshot as an example of multiple  Replicas:

In case you create multiple Replicas at the same time using the same primary DB instance,  then RDS will only perform the snapshot activity once, and that too at the start of the first  replica's creation process. You can even perform the same process using the AWS CLI's rdscreate-db-instance-read-replica command, as shown in the following: # aws rds-create-db-instance-read-replica -s

In this case, RDS will create a new Replica DB instance based on your supplied database  identifier value while keeping all the configurations same as that of the primary DB instance.  To know more about the various options and related operations that you can perform using  this command, refer  to http://docs.aws.amazon.com/AmazonRDS/latest/CommandLineReference//CLIReference­ cmd­CreateDBInstanceReadReplica.html  .

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 31 *

Once your Read Replica instance is created and functioning, you can promote it as a primary  DB instance as well. This feature comes in real handy when you have to perform a DB failure  recovery, where your primary DB instance fails and you need to direct all traffic to the newly  promoted Read Replica, and so on. To promote any Read Replica instance, all you need to  do is select it from the RDS Management dashboard and select the Promote Read  Replica option from the Instance Action drop­down list. This will bring up the Promote Read Replica page, as shown in the following screenshot:

Enable the automatic backups as well as fill out the Backup Retention Period and Backup  Window details as per your requirements. Click on Continue to proceed to the next page.  Acknowledge the Replica instance's promotion and click on Yes, Promote to complete the  process. During this process, your Read Replica instance will reboot itself once, post which it will  become available as a standalone DB instance. You can then perform all sorts of activities on  this DB instance, such as taking manual snapshots and creating new Read Replica instances  from it as well. # aws rds promote-read-replica \ --backup-retention-period 7 \

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 32 *

--preferred-backup-window 00:00-00:30

The preceding command will promote the  to a standalone primary  DB instance. It will also set the automated backup retention period to 7 and configure  the backup window for half an hour between 00:00UTC and 00:30 UTC.

Logging and monitoring your DB instance AWS provides a variety of tools and services to track and monitor the performance of  your DB instances—the most popular and commonly used being Amazon CloudWatch itself. Besides this, RDS, too, comes with a list of simple tools that you can use to  keep an eye on your DB instances. For example, you can list and view the DB  instance's alarms and events by simply selecting the DB instance from the RDS  Management dashboard, as shown in the following screenshot:

You can additionally view the DB instance's security group and snapshot events using  this page as well. RDS will store the events for a period of 14 days, after which they  are deleted. The DB instance quick view page also displays the DB instance's  memory as well as storage utilization in near real time. Each of these fields has a  custom threshold that RDS sets. If the threshold value is crossed, RDS will 

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 33 *

automatically trigger notifications and alarms to inform you about the same. You can  also view the database's Read/Write IOPS value using this page. RDS also provides a page using which you can view the DB Instance's real time  performance graphs. To do so, simply select Launch DB Instance and the Show  Monitoring option, as shown in the following screenshot:

Each graph can be further expanded by selecting it. You can optionally view graphs  for the past hour or a later duration by selecting the appropriate time from the Time  Range dropdown list. Furthermore, RDS also allows you to view your database's essential logs using the  RDS Management dashboard. Select your DB instance, and from the dashboard,  select the Logs option. This will bring up the Logs page, as shown in the following  screenshot:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 34 *

You can use this page to view as well as download the appropriate logs as per your  requirements. RDS obtains logs from the database at short, regular intervals (mostly 5 minutes) and stores them in files that rotate as well. Selecting the Watch option  adjoining a log file will display the log file in real time within your browser. You can  view up to 1,000 lines of your logs at a time using this feature.

Cleaning up your DB instances Once you have completed work with your DB instances, it is equally important to clean up your environment as well. You can delete a DB instance at any time you want using both the RDS Management dashboard and the AWS CLI. To delete a DB instance using the RDS Management dashboard, select  the Delete option from the Instance Actions dropdown list. You will be prompted  to Create a final Snapshot? for your DB instance before you proceed, as shown in  the following screenshot:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 35 *

It is strongly recommended that you create a snapshot of your DB instance before you go ahead and delete it. Once you select the Delete option, RDS will delete the DB  instance along with all the automated backups that were taken earlier. The manual  snapshots, however, are not deleted and thus can be used to restore your DB  instance to its original state if you want to revert to your original settings. To delete a DB instance using the AWS CLI, simply type in the following command  and replace   with the name of your DB instance: # aws rds delete-db-instance \ --final-db-snapshot-identifier MyFinalDBSnapshot

The command will delete your DB Instance but will additionally first create a snapshot  for it by the name of MyFinalDBSnapshot.

Recommendations and best practices Here are some key recommendations and good practices to keep in mind when  working with RDS: 

To begin with, always monitor your DB instances for overall performance and usage.  Leverage CloudWatch and its metrics to set thresholds and customized alarms that notify  you in case of any issues.



Additionally, you can also enable event notifications for your DB instance that will inform  you of all the events that occur with your instance.



Leverage Multi­AZ deployments in conjunction with Read Replicas to increase the overall  availability of your database along with its performance.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 36 *



Always enable automatic snapshots for your DB instances. Also take manual snapshots of  your DB instances before performing any maintenance activities on them.



For a small to medium­sized database, set the storage type of your DB instance to General Purpose (SSD).



If your database has a high performance requirement, then do use the DB instances with  Provisioned IOPS.



Tune your options group as well as your parameters group to improve your database's  overall performance.



Secure your DB instances with a proper security group and encryption facilities. Also  remember to assign and use IAM users with specific rights and privileges.

Database Services Use Cases Creating a database with automatic  failover In this recipe, we're going to create a MySQL RDS database instance configured in  multi­AZ mode to facilitate automatic failover.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 37 *

Database with automatic failover

Getting ready The default VPC will work fine for this example. Once you are comfortable with  creating databases, you may want to consider a VPC containing private subnets that  you can use to segment your database away from the Internet and other resources (in  the style of a three tier application). Either way, you'll need to note down the following: 

The ID of the VPC

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 38 *



The CIDR range of the VPC



The IDs of at least two subnets in your VPC. These subnets need to be in different  Availability Zones, for example, us-east-1a and us-east-1b

How to do it... Create a new CloudFormation template. We're going to add a total of 12 parameters  to it: 1. The first three parameters will contain the values we mentioned in the Getting  ready section: 2. 3. 4. 5. 6. 7. 8. 9. 10. 11.

VPCId: Type: AWS::EC2::VPC::Id Description: VPC where DB will launch SubnetIds: Type: List Description: Subnets where the DB will launch (pick at least 2) SecurityGroupAllowCidr: Type: String Description: Allow this CIDR block to access the DB Default: "172.30.0.0/16"

12. We're also going to add the database credentials as parameters. This is good practice as it means we're not storing any credentials in our infrastructure source code. Note that the  password contains the NoEcho parameter set to true. This stops CloudFormation from  outputting the password wherever the CloudFormation stack details are displayed:

DBUsername: Type: String Description: Username to access the database MinLength: 1 AllowedPattern: "[a-zA-Z][a-zA-Z0-9]*" ConstraintDescription: must start with a letter, must

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 39 *

be alphanumeric DBPassword: Type: String Description: Password to access the database MinLength: 1 AllowedPattern: "[a-zA-Z0-9]*" NoEcho: true ConstraintDescription: must be alphanumeric

13. The next block of parameters pertains to cost and performance. They should be mostly  self­explanatory. Refer to the AWS documentation on database instance types should you  wish to change the instance class for this example. We're supplying a default value of 10  GB for the storage size and choosing a magnetic ( standard) volume for the storage  type. gp2 offers better performance, but it costs a little more:

DBInstanceClass: Type: String Description: The instance type to use for this database Default: db.t2.micro DBStorageAmount: Type: Number Description: Amount of storage to allocate (in GB) Default: 10 DBStorageType: Type: String Description: Type of storage volume to use (standard [magnetic] or gp2) Default: standard AllowedValues: - standard - gp2

14. We need to set some additional parameters for our database. These are the MySQL  engine version and port. Refer to the AWS documentation for a list of all the available  versions. We are setting a default value for this parameter as the latest version of MySQL  at the time of writing:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 40 *

DBEngineVersion: Type: String Description: DB engine version Default: "5.7.11" DBPort: Type: Number Description: Port number to allocate Default: 3306 MinValue: 1150 MaxValue: 65535

15. Finally, we are going to define some parameters relating to backup and availability. We  want our database to run in multi­AZ mode, we set this to true by default. We also set a  backup retention period of 1 day by default; you might want to choose a period larger than  this. If you set this value to 0, backups will be disabled (not recommended!):

DBMultiAZ: Type: String Description: Should this DB be deployed in Multi-AZ configuration? Default: true AllowedValues: - true - false DBBackupRetentionPeriod: Type: Number Description: How many days to keep backups (0 disables backups) Default: 1 MinValue: 0 MaxValue: 35

16. We're done with the parameters for this template; we can now go ahead and start defining  our Resources. First of all, we want a security group for our DB to reside in. This security  group allows inbound access to the database port from the CIDR range we've defined:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 41 *

ExampleDBSecurityGroup: Type: AWS::EC2::SecurityGroup Properties: GroupDescription: Example security group for inbound access to DB SecurityGroupIngress: - IpProtocol: tcp CidrIp: !Ref SecurityGroupAllowCidr FromPort: !Ref DBPort ToPort: !Ref DBPort VpcId: !Ref VPCId

17. Next, we need to define a DBSubnetGroup resource. This resource is used to declare  which subnet(s) our DB will reside in. We define two subnets for this resource so that the  primary and standby servers will reside in separate Availability Zones:

ExampleDBSubnetGroup: Type: AWS::RDS::DBSubnetGroup Properties: DBSubnetGroupDescription: Example subnet group for example DB SubnetIds: - Fn::Select: [ 0, Ref: SubnetIds ] - Fn::Select: [ 1, Ref: SubnetIds ]

18. Finally, we define our RDS instance resource. We specify it as being a MySQL database  and the rest of the properties are made up of the parameters and resources that we've  defined previously. Lots of !Ref is required here: ExampleDBInstance: Type: AWS::RDS::DBInstance Properties: AllocatedStorage: !Ref DBStorageAmount BackupRetentionPeriod: !Ref DBBackupRetentionPeriod DBInstanceClass: !Ref DBInstanceClass DBSubnetGroupName: !Ref ExampleDBSubnetGroup Engine: mysql EngineVersion: !Ref DBEngineVersion MasterUsername: !Ref DBUsername

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 42 *

MasterUserPassword: !Ref DBPassword MultiAZ: !Ref DBMultiAZ StorageType: !Ref DBStorageType VPCSecurityGroups: - !GetAtt ExampleDBSecurityGroup.GroupId

19. For good measure, we can add an output to this template that will return the hostname for  this RDS database: Outputs: ExampleDbHostname: Value: !GetAtt ExampleDBInstance.Endpoint.Address

20. You can provision the database via the CloudFormation web console or use a CLI  command like so: aws cloudformation create-stack \ --stack-name rds1 \ --template-body \ file://06-create-database-with-automatic-failover.yaml \ --parameters \ ParameterKey=DBUsername,ParameterValue=<username> \ ParameterKey=DBPassword,ParameterValue=<password> \ ParameterKey=SubnetIds,"ParameterValue='<subnet-id-a>, \ <subnet-id-b>'" \ ParameterKey=VPCId,ParameterValue=

How it works... In a multi­AZ configuration, AWS will provision a standby MySQL instance in a  separate Availability Zone. Changes to your database will be replicated to the standby  DB instance in a synchronous fashion. If there is a problem with your primary DB 

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 43 *

instance AWS will automatically failover to the standby, promote it to be the primary  DB, and provision a new standby. You don't have access to query standby databases directly. So you can't use it to  handle all of your read queries, for example. If you wish to use additional database  instances to increase read capacity, you'll need to provision a read­replica. We'll  cover those in a separate recipe. Backups will always be taken from the standby instance, which means there is no  interruption to your DB availability. This is not the case if you opted against deploying  your DB in multi­AZ mode. When you deploy this example it will take roughly 20 minutes or more for the stack to  report completion. This is because the RDS service needs to go through the following  process in order to provision a fully working multi­AZ database: 

Provision the primary database



Back up the primary database



Provision the standby database using the backup from the primary



Configure both databases for synchronous replication

Note: WARNINGBe careful about making changes to your RDS configuration after you've started writing

data to it, especially when using CloudFormation updates. Some RDS configuration changes  require the database to be re­provisioned, which can result in data loss. We'd recommend using  CloudFormation change sets, which will give you an opportunity to see which changes are about  to cause destructive behavior. The CloudFormation RDS docs also provide some information on  this.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 44 *

There's more... 

You can define a maintenance window for your RDS instance. This is the time period when AWS will perform maintenance tasks such as security patches or minor version upgrades.  If you don't specify a maintenance window (which we don't in this example), one is chosen  for you.

Migrating a database In this recipe, we will use Database Migration Service (DMS) to move an external  database into Relational Database Service (RDS). Unlike many of the other recipes, this will be performed manually through the web  console. Most database migrations are one­off, and there are many steps involved. We suggest that you first perform the process manually via the console before automating it, if  required (which you can do with the AWS CLI tool or SDKs).

Getting ready For this recipe you will need the following: 

An external database



An RDS database instance



The source database in this example is called employees, so substitute your  own database name as required.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 45 *



Both databases must be accessible from the replication instance that will be  created as part of the recipe. The simplest way to do this is to allow access to  the databases from the Internet, but obviously this has security implications.

How to do it... 1. Navigate to the DMS console:

2. Click on Create Migration to start the migration wizard:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 46 *

3. Specify the details for your replication instance. Unless you have a specific VPC  configuration, the defaults will be fine:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 47 *

4. While waiting for the replication instance to be ready, fill out the source and target endpoint information, including server hostname and port, and the username and password to use  when connecting:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 48 *

5. Once the instance is ready, the interface will update and you can proceed:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 49 *

6. In order to confirm and create the source and target endpoints, click on the Run test button for each of your databases:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 50 *

7. After the endpoints have been successfully tested and created, define your task. In this  recipe, we will simply migrate the data (without ongoing replication):

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 51 *

8. For simplicity, drop the tables in the target database (which should be empty) to ensure  parity between the databases:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 52 *

9. Finally, define the mappings between the two databases. In this case, we will migrate all  the tables (by using the wildcard %) in the employees database on the source:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 53 *

10. Once you click Add selection rule you will see your rule in the selection rules list:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 54 *

11. Once the task is defined you have finished the wizard. You will then see the task being  created:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 55 *

12. Once the status of the task is Ready you can select it and click on  the Start/Resume button:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 56 *

13. When complete, you will see the task's details updated in the console:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 57 *

How it works... At a high level, this is what the DMS architecture looks like:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 58 *

Both the Source and Target databases are external to DMS. They are represented  internally by endpoint resources that are references to the databases. Endpoints can  be reused between different tasks if needed. This recipe starts by defining the replication instance details. Keep in mind that the  DMS migration process works best when the migration/transform between the two  databases is kept in memory. This means that for larger jobs you should allocate a  more powerful instance. If the process needs to temporarily write data to disk (such as swap) then the performance and throughput will be much lower. This can have flow­on effects, particularly for tasks that include ongoing replication. Next, the two endpoints are defined. It is very important to verify your endpoint  configuration by using the built­in testing feature so that your tasks do not fail later in  the process. Generally, if the connectivity test fails, it is one of two main issues: 

Network connectivity issues between the replication instance and the database. This is  particularly an issue for on­premise databases, which are usually specifically restricted  from being accessed externally.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 59 *



User permissions issues: For example, in the case of MySQL, the root user cannot be  used to connect to the database externally, so this default user cannot be used.

Defining the task involves defining your migration type. The recipe uses the simplest  type; migrate tables. This means that the data will be copied between the two  databases, and will be complete when the data is propagated. We also get to define  the behavior on the target database. For simplicity, we have configured the task to  drop the tables in the target database ensuring that the two databases look as similar  as possible, even if the tables are renamed, or the table mappings change. For the  task table mappings we use the wildcard symbol % to match all tables in the source  database. Obviously, you could be more selective if you only wanted to match a  subset of your data. Once the replication instance, endpoints, and task are defined the wizard ends and  you are returned to the DMS console. After the task is finished creating it can be  started. As it is a migrate existing data­type task, it will complete once all the data has been  propagated to the target database.

There's more... This is obviously a simple example of what DMS can do. There are other features and  performance aspects that you should consider in more advanced scenarios.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 60 *

Database engines While this example uses two MySQL databases, it is possible to migrate from one  database engine to a complete database engine, for example, Oracle to MySQL.  Unfortunately, this can be a complex process, and while this functionality is very useful it is beyond the scope of this recipe. Due to the differences in the various engines,  there are some limitations on what you can migrate and transform. Note: See the AWS Schema Conversion Tool documentation for more details on what can be migrated between different database engines.

Ongoing replication There are also some limits around the ongoing propagation of data—only table data  can be migrated. Things such as indexes, users, and permissions cannot be replicated continually.

Multi­AZ For ongoing replication tasks, you may want to create a multi­AZ replication instance  so that the impact of any interruptions of services are minimized. Obviously you will  need to have a similarly configured (such as multi­AZ) RDS instance as your target to  get the full benefit! Note: For best performance, when setting up your replication instance you should make sure it is in  the same AZ as your target RDS instance.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 61 *

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 62 *

AWS Networking with VPC Contents Introducing Amazon Web Services...............................................................................3 AWS architecture and components........................................................................3 An overview of Amazon VPC..................................................................................... 10 VPC limits and costs........................................................................................... 21 Working with VPCs................................................................................................... 22 VPC deployment scenarios..................................................................................22 Getting started with the VPC wizard.....................................................................23 Launching instances in your VPC........................................................................40 Planning next steps.................................................................................................. 42 Best practices and recommendations.........................................................................45 VPC Cloud Formation Recipes.................................................................................. 46 Building a secure network......................................................................................... 46 Getting ready...................................................................................................... 47 How to do it......................................................................................................... 48 How it works....................................................................................................... 55 There's more....................................................................................................... 56 Creating a NAT gateway........................................................................................... 57 Getting ready...................................................................................................... 57 How to do it......................................................................................................... 58 How it works....................................................................................................... 59 Network logging and troubleshooting..........................................................................60 Getting ready...................................................................................................... 61 How to do it......................................................................................................... 61 How it works....................................................................................................... 63 There's more....................................................................................................... 64

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 1 *

Introducing Amazon Web Services Amazon Web Services or AWS is a comprehensive public cloud computing platform that  offers a variety of web­based products and services on an on­demand and pay­per­use basis.  AWS was earlier a part of the e­commerce giant Amazon.com, and it wasn't until 2006 that  AWS became a separate entity of its own. Today, AWS operates globally with data centers  located in USA, Europe, Brazil, Singapore, Japan, China, and Australia. AWS provides a  variety of mechanisms, using which the end users can connect to and leverage its services,  the most common form of interaction being the web­based dashboard also called as AWS  Management Console.

AWS architecture and components Before we begin with the actual signup process, it is important to take a look at some  of the key architecture and core components of services offered by AWS.

Regions and availability zones We do know that AWS is spread out globally and has its presence across USA,  Europe, Asia, Australia, and so on. Each of these areas is termed as a region. AWS  currently has about 10 regions, each containing multiple data centers within  themselves. So what's with all these regions and why do they matter? In simple terms, the resources that are geographically close to your organization are served much 

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 2 *

faster! For example, an organization running predominantly from USA can leverage  the USA's regions to host their resources and gain access to them must faster. For most of the AWS services that you use, you will be prompted to select a region in  which you want to deploy the service. Each region is completely isolated from the  other and runs independently as well. Note: AWS does not replicate resources across regions automatically. It is up to the end user to  set up the replication process.

A list of regions and their corresponding codes is provided here for your reference. The code  is basically how AWS refers to its multiple regions:

Region North America

Name

Code

US East (N. Virginia)

us­east­1

US West (N. California)

us­west­1

US West (Oregon)

us­west­2

South America

Sao Paulo

sa­east­1

Europe

EU (Frankfurt)

eu­central­1

EU (Ireland)

eu­west­1

Asia Pacific (Tokyo)

ap­northeast­1

Asia Pacific (Singapore)

ap­southeast­1

Asia Pacific (Sydney)

ap­southeast­2

Asia Pacific (Beijing)

cn­north­1

Asia

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 3 *

Each region is split up into one or more Availability Zones (AZs) and pronounced as A­Zees. An A Z is an isolated location inside a region. AZs within a particular region connect to other  AZs via low­latency links. What do these AZs contain? Well, ideally they are made up of one  or more physical data centers that host AWS services on them. Just as with regions, even  AZs have corresponding codes to identify them, generally they are regional names followed  by a numerical value. For example, if you select and use us­east­1, which is the North Virginia region, then it would have AZs listed as us­east­1b, us­east­1c, us­east­1d, and so on:

AZs are very important from a design and deployment point of view. Being data  centers, they are more than capable of failure and downtime, so it is always good  practice to distribute your resources across multiple AZs and design your applications  such that they can remain available even if one AZ goes completely offline. An important point to note here is that AWS will always provide the services and  products to you as a customer; however, it is your duty to design and distribute your  applications so that they do not suffer any potential outages or failures. Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 4 *

AWS platform overview The AWS platform consists of a variety of services that you can use either in isolation  or in combination based on your organization's needs. This section will introduce you  to some of the most commonly used services as well as some newly launched ones.  To begin with, let's divide the services into three major classes: 

Foundation services: This is generally the pillars on which the entire AWS  infrastructure commonly runs on, including the compute, storage, network, and databases.



Application services: This class of services is usually more specific and generally used  in conjunction with the foundation services to add functionality to your applications. For  example, services such as distributed computing, messaging and Media Transcoding, and  other services fall under this class.



Administration services: This class deals with all aspects of your AWS environment,  primarily with identity and access management tools, monitoring your AWS services and  resources, application deployments, and automation.

Let's take a quick look at some of the key services provided by AWS. However, do note that  this is not an exhaustive list:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 5 *

We will discuss each of the foundation services.

Compute This includes the following services: 

Elastic Compute Cloud (EC2): When it comes to brute computation power and scalability, there must be very few cloud providers out there in the market that can match AWS's EC2  service. EC2 or Elastic Compute Cloud is a web service that provides flexible, resizable,  and secure compute capacity on an on­demand basis. AWS started off with EC2 as one of  its core services way back in 2006 and has not stopped bringing changes and expanding  the platform ever since. The compute infrastructure runs on a virtualized platform that  predominantly consists of the open sourced Xen virtualization engine. We will be exploring  EC2 and its subsequent services in detail in the coming chapters.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 6 *



EC2 Container Service: A recently launched service, the EC2 Container Service, allows  you to easily run and manage docker containers across a cluster of specially created EC2  instances.



Amazon Virtual Private Cloud (VPC): VPC enables you to create secure,  fully customizable, and isolated private clouds within AWS's premises. They provide  additional security and control than your standard EC2 along with connectivity options to  on premise data centers.

Storage This includes the following services: 

Simple Storage Service (S3): S3 is a highly reliable, fault tolerant, and fully  redundant data storage infrastructure provided by AWS. It was one of the first services  offered by AWS way back in 2006, and it has not stopped growing since. As of April 2013,  an approximate 2 trillion objects have been uploaded to S3, and these numbers are  growing exponentially each year.



Elastic Block Storage (EBS): EBS is a raw block device that can be attached to  your compute EC2 instances to provide them with persistent storage capabilities.



Amazon Glacier: It is a similar service offering to S3. Amazon Glacier offers long­term  data storage, archival, and backup services to its customers.



Amazon Elastic File System: Yet another very recent service offering introduced by  AWS, Elastic File System(EFS) provides scalable and high­performance storage to EC2  compute instances in the form of an NFS filesystem.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 7 *

Databases This includes the following services: 

Amazon Relational Database Service (RDS): RDS provides a scalable, high­ performance relational database system such as MySQL, SQL Server, PostgreSQL, and  Oracle in the cloud. RDS is a completely managed solution provided by AWS where all the  database heavy lifting work is taken care of by AWS.



Amazon DynamoDB: DynamoDB is a highly scalable NoSQL database as a  service offering provided by AWS.



Amazon Redshift: Amazon Redshift is a data warehouse service that is designed to  handle and scale to petabytes of data. It is primarily used by organizations to perform real­ time analytics and data mining.

Networking This includes the following services: 

Elastic Load Balancer (ELB): ELB is a dynamic load balancing service provided by AWS  used to distribute traffic among EC2 instances. You will be learning about ELB a bit more  in detail in subsequent chapters.



Amazon Route 53: Route 53 is a highly scalable and available DNS web service provided  by AWS. Rather than configuring DNS names and settings for your domain provider, you  can leverage Route 53 to do the heavy lifting work for you.

Distributed computing and analytics This includes the following services:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 8 *



Amazon Elastic MapReduce (EMR): As the name suggests, this service provides users  with a highly scalable and easy way to distribute and process large amounts of data using  Apache's Hadoop. You can integrate the functionalities of EMR with Amazon S3 to store  your large data or with Amazon DynamoDB as well.



Amazon Redshift: This is a massive data warehouse that users can use to store, analyze, and query petabytes of data.

Content distribution and delivery Amazon CloudFront is basically a content delivery web service that can be used to  distribute various types of content, such as media, files, and so on, with high data  transfer speeds to end users globally. You can use CloudFront in conjunction with  other AWS services such as EC2 and ELB as well.

Workflow and messaging This includes the following services: 

Amazon Simple Notification Service (SNS): SNS is a simple, fully managed push  messaging service provided by AWS. You can use it to push your messages to mobile  devices (SMS service) and even to other AWS services as API calls to trigger or notify  certain activities.



Amazon Simple Email Service (SES): As the name suggests, SES is used to send  bulk e­mails to various recipients. These e­mails can be anything, from simple notifications  to transactions messages, and so on. Think of it as a really large mail server that can scale as per your requirements and is completely managed by AWS! Awesome, isn't it!

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 9 *

Monitoring Amazon CloudWatch is a monitoring tool provided by AWS that you can use to monitor any and  all aspects of your AWS environment, from EC2 instances to your RDS services to the load on  your ELBs, and so on. You can even create your own metrics, set thresholds, create alarms, and a whole lot of other activities as well.

Identity and access management AWS provides a rich set of tools and services to secure and control your infrastructure on the  cloud. The most important and commonly used service for this is identity and access  management (IAM). Using IAM, you can, as an organizational administrator, create and manage  users, assign them specific roles and permissions, and manage active directory federations as  well. We will be using a lot of IAM in the next chapter, which covers this topic in greater depth.

An overview of Amazon VPC So far we have learnt a lot about EC2, its features, and uses, and how we can deploy  scalable and fault tolerant applications using it, but EC2 does come with its own sets  of minor drawbacks. For starters, you do not control the IP addressing of your  instances, apart from adding an Elastic IP address to your instance. By design, each  of your instances will get a single private and public IP address, which is routable on  the Internet—again, something you cannot control. Also, EC2 security groups have the capability to add rules for inbound traffic only; there is no support for providing any  outbound traffic rules. So, although EC2 is good for hosting your applications, it is still  not that secure. The answer to all your problems is Amazon VPC!

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 10 *

Amazon VPC is a logically isolated part of the AWS cloud that enables you to build  and use your own logical subnets and networks. In a simpler sense, you get to build  your own network topology and spin up instances within it. But what actually separates VPC from your classic EC2 environment is the ability to isolate and secure your  environment. Using VPCs, you can choose which instances are accessible over the  Internet and which are not. You can create a bunch of different subnets within a single VPC, each with their own security policies and routing rules. VPCs also provided an  added layer of protection by enforcing something called as Network Access Control  Lists (ACLs) besides your standard use of security groups. This ensures that you  have total control over what traffic is routed in and out of your subnets and the VPC as well. VPCs also provide an added functionality using which you can connect and extend  your on­premise datacenters to the AWS cloud. This is achieved using an IPsec VPN  tunnel that connects from your on premise datacenter's gateway device to the  VPC's Virtual Private Gateway, as shown in the following image:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 11 *

An important point to note here is that a VPC is still a part of the AWS Cloud. It is not  physically separate hosting provided by AWS, it simply is a logically isolated part of the EC2  infrastructure. This isolation is done at the network layer and is very similar to a traditional  datacenter's network isolation; it's just that we as end users are shielded from the  complexities of it. Note: To know more about VPN and virtual private gateways, refer  to http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_VPN.html  .

Subnets Perhaps the most important part of the VPC, the subnets are nothing more than a  range of valid IP addresses that you specify. VPC provides you with two different  subnet creation options: a publically or Internet routed subnet called as a public  subnetand an isolated subnet called as a private subnet. You can launch  your instances in either of these subnets depending on whether you wish your  instances to be routed on the Internet or not. Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 12 *

How does it all work? Pretty simple! When you first create a VPC, you provide it with a set of IP addresses in the form of a CIDR, for example, 10.0.0.0/16. The /16 here  indicates that this particular VPC can support up to 65,536 IP addresses (2^(32­16) =  65,536, IP address range 10.0.0.0 ­ 10.0.255.255)! Now that's a lot! Once the VPC's  CIDR block is created, you can go ahead and carve out individual subnets from it. For  example, a subnet with the CIDR block 10.0.1.0/24 for hosting your web servers and  another CIDR block 10.0.5.0/24 for your database servers and so on and so forth. The idea here is that from the 65,536 IP address block, we carved out two subnets,  each supporting 256 IPs (/24 CIDR includes 256 IP addresses in it). Now you can  specify the subnet for the web servers to be public, as they will need to be routed to  the Internet and the subnet for the database servers to be private as they need to be  isolated from the outside world. To know more about CIDRs and how they work, refer  to https://en.wikipedia.org/wiki/Classless_Inter­Domain_Routing  .

There is one additional thing worth mentioning here. By default, AWS will create a VPC for  you in your particular region the first time you sign up for the service. This is called as the  default VPC. The default VPC comes preconfigured with the following set of configurations: 

The default VPC is always created with a CIDR block of /16, which means it supports  65,536 IP addresses in it.



A default subnet is created in each AZ of your selected region. Instances launched in these default subnets have both a public and a private IP address by default as well.



An Internet Gateway is provided to the default VPC for instances to have Internet  connectivity.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 13 *



A few necessary route tables, security groups, and ACLs are also created by default that  enable the instance traffic to pass through to the Internet. Refer to the following figure:

You can use this default VPC just as any other VPC by creating additional subnets in it,  provisioning route tables, security groups, and so on

Security groups and network ACLs We have talked a lot about security groups in the past two chapters already. We do  know that security groups are nothing but simple firewall rules that you can configure  Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 14 *

to safeguard your instances. You can create a maximum of 100 security groups for a  single VPC, with each Security Group containing up to 50 firewall rules in them. Also,  it is very important to remember that a Security Group does not permit inbound traffic  by default. You have to explicitly set inbound traffic rules to allow traffic to flow to your  instance. However, all outbound traffic from the instance is allowed by default. Network ACLs are something new. These provide an added security measure over  security groups as they are instance specific, whereas Network ACLs are subnet  specific. Unlike your security groups, however, you can both allow and restrict inbound and outbound traffic using ACL rules. Speaking of ACL rules, they are very much  similar to your Security Group rules, however, with one small exception. Each ACL  rule is evaluated by AWS based on a number. The number can be anything from 100  all the way up to 32,766. The ACL rules are evaluated in sequence starting from the  smallest number and going all the way up to the maximum value. The following is a  small example of how ACL rules look: Inbound ACL rules Rule No.

Source IP

Protocol

Port

Allow/Deny

100

0.0.0.0/0

All

All

ALLOW

*

0.0.0.0/0

All

All

DENY

Rule No.

Dest IP

Protocol

Port

Allow/Deny

100

0.0.0.0/0

all

all

ALLOW

*

0.0.0.0/0

all

all

DENY

Outbound ACL rules

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 15 *

These are the ACL rules created by AWS for your default VPC; as a result, this particular ACL is called as the default Network ACL as well. What do these rules mean? For starters, the rule number 100 for both the inbound and outbound ACL specifies the traffic to flow from any  protocol running on any port in and out of the subnet. The * is also considered as a rule  number and is a must in all ACLs. It basically means that you drop any packets that do not  match the ACL's rules. We will be checking out ACLs and security groups in action later on in  this chapter when we create our very own VPC for the first time.

Routing tables Route tables are pretty straightforward and easy to implement in a VPC. They are  nothing but simple rules or routes that are used to direct network traffic from a subnet.  Each subnet in a VPC has to be associated with a single route table at any given time; however, you can attach multiple subnets to a single route table as well. Remember the default VPC and the default subnets? Well, a similar default route table is also created when you first start using your VPC. This default route table is called  as the main route table and it generally contains only one route information that  enables traffic to flow within the VPC itself. Subnets that are not assigned to any route  tables are automatically allocated to the main route table. You can edit and add  multiple routes in the main route table as you see fit; however, you cannot modify the  local route rule. The following an example of a main route table viewed from the VPC  Management dashboard:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 16 *

As you can see, there are a couple of entries made in this table. The first is the local route rule that allows traffic to flow within this particular subnet (10.0.0.0/16). The second route is  something called as a route for VPC endpoints. This is a private connection made between  your VPC and some other AWS service; in this case, the service is S3. Let's look VPC  endpoints a little closely.

VPC endpoints VPC endpoints basically allow you to securely connect your VPC with other AWS  services. These are virtual devices that are highly available and fault tolerant by  design. They are scaled and managed by AWS itself, so you don't have to worry about the intricacies of maintaining them. All you need to do is create a VPC endpoint  connection between your VPC and an AWS service of your choice, and voila! Your  instances can now communicate securely with other AWS services. The instances in  the VPC communicate with other services using their private IP addresses itself, so  there's no need to route the traffic over the Internet. Note: AWS currently only supports VPC endpoint connections for Amazon S3. More services are  planned to be added shortly.

When an endpoint is created, you first need to select either of your VPC's route tables. The traffic between your VPC instances and the AWS service will be routed using this  particular route table. Similar to any other route information, a VPC endpoint route  also contains a Destination field and a Target field. The Destination filed contains  the AWS service's prefix list ID, which is generally represented by the following ID: pl­ xxxxxxxx. The Target field contains the endpoint ID, which is represented in the  following format: vpce­xxxxxxxx.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 17 *

In the following route table example, the prefix list ID (pl­68a54001) represents the S3  service whereas the target vpce­80cd2be9 represents the endpoint ID: VPC Endpoint Route Destination

Target

10.0.0.0/16

Local

pl­68a54001

vpce­80cd2be9

Endpoints also provided an additional feature using which you can control and govern  access to the remote AWS service. This is achieved using something called  as endpoint policies. Endpoint policies are nothing more than simple IAM­based resource policies that are  provided to you when an endpoint is first created. AWS creates a simple policy  document that allows full access to the AWS service by default. The following is a  sample endpoint policy that is created by AWS for the S3 service: {

"Statement": [

"Action": "*", "Effect": "Allow", "Resource": "*", "Principal": "*" } ] }

{

To know more about the endpoint policies and features, refer  to http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc­endpoints.html  .

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 18 *

Internet Gateways Internet Gateways, as the name suggest, are primarily used to provide Internet  connectivity to your VPC instances. All you have to do is create and attach an Internet  Gateway device to your VPC and add a route entry in your public subnet's route table  to point to the Internet Gateway! That's it! The default VPC comes with an Internet  gateway already deployed in it. So, any instance that you launch from the default  subnet obtains Internet connectivity automatically. This does not apply for non­default  VPCs, however, as an instance launched in a non­default subnet does not receive a  public IP address by default. You would have to either assign one to the instance  during the launch phase or modify your non­default subnet's public IP address  attributes. Once you have created and attached an Internet Gateway to your VPC, you will also  have to make sure that the public subnet's route table has an entry for this gateway. Plus, you will also have to create the correct set of security groups and network ACL  rules to allow your subnet's traffic to flow through the Internet. The following is an  example of a VPC's route table showing the route for a subnet's traffic to the Internet  Gateway (igw­8c3066e9):

Besides the Internet connectivity, Internet Gateways also perform NAT on the instance's  private IPs. The instances in a subnet are only aware of their private IP addresses that they  use to communicate internally. The Internet Gateway maps the instance's private IP with an 

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 19 *

associated public or Elastic IP and then routes traffic outside the subnet to the Internet.  Conversely, the Internet Gateway also maps inbound traffic from the Internet to a public or  Elastic IP and then translates it to the instance's private IP address. This is how your  instances receive Internet from within a VPC, which brings us to yet another interesting topic  called as NAT instances.

NAT instances So, we have just learnt that the Internet Gateway NATs the IP addresses of instances  placed out in the public subnet so that they can communicate with the Internet, but  what about instances in the private subnets? How do they communicate with the  Internet without having direct Internet connectivity via the gateway? That's where a NAT instance comes into play. A NAT Instance is a special instance  created inside your public subnet that NATs outbound traffic from instances based in  your private subnet to the Internet. It is important to note here that the NAT instance  will only forward the outbound traffic and not allow any traffic from the Internet to reach the private subnets, similar to a one way street. You can create a NAT Instance out of any AMI you wish; however, AWS provides few  standard Linux­based AMIs that are well suited for such purposes. These special AMIs are listed out in the community AMIs page and all you need to do is filter out  the amzn-ami-vpc-nat AMI from the list and spin up an instance from it. The following example depicts the traffic flow from a private subnet (10.0.1.0/24) to the NAT instance inside a public subnet (10.0.0.0/24) via a route table:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 20 *

In the preceding example, outbound traffic from the public subnet's route table is routed to the  Internet Gateway (igw­8c3066e9) while the outbound traffic from the private subnet's route  table is routed to the NAT instance. Along with the route tables, it is also essential that you  correctly populate the Security Group for your NAT instance. The following is a simple NAT  instance Security Group example for your reference:

NAT instance ­ inbound security Rules Source

Protocol

Port

Remarks

10.0.1.0/24

TCP

80

Permit inbound HTTP traffic from private subnet

10.0.1.0/24

TCP

443

Permit inbound HTTPS traffic from private subnet

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 21 *

NAT instance ­ inbound security Rules *

TCP

22

Permit SSH login to NAT instance from remote N/W

Note: The * replace the  field with the IP address of your local desktop machine.

The following are the outbound security rules:

NAT Instance ­ outbound security rules Source

Protocol

Port

Remarks

0.0.0.0/0

TCP

80

Permit HTTP access to Internet for the NAT instance

0.0.0.0/0

TCP

443

Permit HTTPS access to Internet for the NAT instance

DNS and DHCP Option Sets VPCs provide an additional feature called as DHCP Option Sets using which you can  set and customize the DNS and DHCP for your instances. The default VPC comes  with a default DHCP Options Set that is used to provide the instances with a dynamic  private IP address and a resolvable hostname. Using the DHCP Options Set, you can  configure the following attributes for your VPC:

Domain Name Servers (DNS): You can list down up to four DNS servers here of your own  choice or even provide the Amazon DNS server details. The Amazon DNS server is provided  in your VPC and runs on a reserved IP address. For example, if your VPC has the subnet  of 10.0.0.0/16, then the Amazon DNS Server will probably run on the IP 10.0.0.2. You can  additionally provide the Amazon DNS Server's IP address, 169.254.169.253, or the  value AmazonProvidedDNS as required. Values entered here are automatically added to  your Linux instances /etc/resolv.conf file for name resolution.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 22 *

Domain name: You can either provide your own domain name value or choose to use the  default AWS domain name values using this option. The default AWS domain names can be  provided only if you have selected AmazonProvidedDNS as your DNS server. For example,  instances launched in the US West region with the Amazon DNS server value will get a  resolvable private DNS hostname as us-west-2.compute.internal. 

NTP servers: You can list up to four NTP server IP addresses using the DHCP Options  Set wizard. Note, however, that this will only accept IP address values and not FQDNs  such as pool.ntp.org .



NetBIOS name server: You can list down up to four NetBIOS name servers as well;  however, this field is optional.



NetBIOS node type: You can specify the NetBIOS node value, which can either be 1, 2, 4, or 8. AWS recommends that you specify 2 as broadcast, and multicasts are not currently  supported.

You can create and attach only one DHCP option set with a VPC at a time. AWS uses the  default DHCP option if you do not specify one explicitly for your VPC. Instances either running or newly launched will automatically pick up these DNS and DHCP settings, so there is no  need for you to restart or relaunch your existing instances.

VPC limits and costs Okay, so far we have understood a lot about how the VPC works and what its  components are, but what is the cost of all this? Very simple, it's nothing! VPC is a  completely free of cost service provided by AWS; however, you do have to pay for the  EC2 resources that you use, for example, the instances, the Elastic IP addresses,  EBS volumes, and so on. Also, if you are using your VPC to connect to your on  premise datacenter using the VPN option, then you need to pay for the data transfers 

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 23 *

over the VPN connection as well as for the VPN connection itself. AWS charges $0.05 per VPN connection hour. Besides this, VPCs also have a few limits set on them by default. For example, you  can have a maximum of five VPCs per region. Each VPC can have a max of one  Internet gateway as well as one virtual private gateway. Also, each VPC has a limit of  hosting a maximum of up to 200 subnets per VPC. You can increase these limit by  simply requesting AWS to do so. To view the complete list of VC limits, refer  to http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/VPC_Appendix_Limits. html  .

Working with VPCs VPC deployment scenarios VPC provides a simple, easy­to­use wizard that can spin up a fully functional VPC  within a couple of minutes. All you need to do is select a particular deployment  scenario out of the four scenarios provided and configure a few basic parameters such as subnet information, availability zones in which you want to launch your subnets,  and so on, and the rest is all taken care of by AWS itself. Let's have a quick look at the four VPC deployment scenarios: 

VPC with a single public subnet: This is by far the simplest of the four deployment  scenarios. Using this scenario, VPC will provision a single public subnet with a default  Internet Gateway attached to it. The subnet will also have a few simple and basic route 

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 24 *

tables, security groups, and network ACLs created. This type of deployment is ideal for  small­scaled web applications or simple websites that don't require any separate  application or subnet tiers. 

VPC with public and private subnets (NAT): Perhaps the most commonly used  deployment scenario, this option will provide you with a public subnet and a private subnet  as well. The public subnet will be connected to an Internet gateway and allow instances  launched within it to have Internet connectivity, whereas the private subnet will not have  any access to the outside world. This scenario will also provision a single NAT instance  inside the public subnet using which your private subnet instances can connect with the  outside world but not vice versa. Besides this, the wizard will also create and assign a  route table to both the public and private subnets, each with the necessary routing  information prefilled in them. This type of deployment is ideal for large­scale web  applications and websites that leverage a mix of public facing (web servers) and non­public facing (database servers).



VPC with public and private subnets and hardware VPN access: This deployment  scenario is very much similar to the VPC with public and private subnets, however,  with one component added additionally, which is the Virtual Private Gateway. This  Virtual Private Gateway connects to your on premise network's gateway using a  standard VPN connection. This type of deployment is well suited for organizations that  wish to extend their on premise datacenters and networks in to the public clouds while  allowing their instances to communicate with the Internet.



VPC with a private subnet only and hardware VPN access: Unlike the previous  deployment scenario, this scenario only provides you with a private subnet that can  connect to your on premise datacenters using standard VPN connections. There is no  Internet Gateway provided and thus your instances remain isolated from the Internet.  This deployment scenario is ideal for cases where you wish to extend your on premise 

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 25 *

datacenters into the public cloud but do not wish your instances to have any  communication with the outside world.

Getting started with the VPC wizard Before we go ahead and deploy our VPC, let's first have a quick look at our use case.  We need to create a secure website hosting environment for our friends at All-AboutDogs.com, complete with the following requirements: 

Create a VPC (US-WEST-PROD-1 - 192.168.0.0/16 ) with separate secure  environments for hosting the web servers and database servers.



Only the web servers environment ( US-WEST-PROD-WEB - 192.168.1.0/24 ) should  have direct Internet access.



The database servers environment ( US-WEST-PROD-DB - 192.168.5.0/24 ) should be  isolated from any direct access from the outside world.



The database servers can have restricted Internet access only through a jump server (NAT Instance). The jump server needs to be a part of the web server environment.



The web servers environment should full have access to Amazon S3.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 26 *

To get started with VPC, we first have to log in to the AWS Account using your IAM  credentials. Next, from the AWSManagement Console, select the VPC option from under  the Networking group

This will bring up the VPC Dashboard using which you can create, manage, and  delete VPCs as per your requirements. The VPC dashboard lists the currently  deployed VPCs, Subnets, Network ACLs, and much more under  the Resources section. You can additionally view and monitor the health of your VPC service by viewing the  status provided by the Service Healthdashboard, as shown in the following  screenshot. In my case, I'm operating my VPC out of the US West (Oregon) region.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 27 *

The VPC Dashboard also lists any existing VPN connections that you might have set  up earlier. You can view the VPNConnections, Customer Gateways information as  well as the Current Status of the VPN connection by using this dashboard.  Remember that a VPN connection has a cost associated with it when it is provisioned  and in the available state. You can additionally use this dashboard and even launch  your instances directly into a VPC using the Launch EC2 Instances option. These  instances will most probably be launched in your default VPC in case you haven't  already created another one. With all this said and done, let's go ahead and create our VPC using the VPC Wizard.  Select the Start VPC Wizard option. The wizard is a simple two­step process that will  guide you with the required configuration settings for your VPC. You will be prompted  to select any one out of the four VPC scenarios, so with our use case in mind, go  ahead and select the VPC with Public and Private Subnets option. Do note that this  will create a /16 network with two /24 subnets by default, one public subnet and the  other a private subnet. You can always create more subnets as required once the  VPC is created.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 28 *

Also worth mentioning here is that this VPC scenario will create and launch a NAT  instance as well in the public subnet. This instance will be powered on automatically  once your VPC is created, so be aware about its existence and power it down unless  you want to get charged for it. The second step of the wizard is where you get to configure your VPC network and subnets.  Fill in the following details as required: 

IP CIDR block: Provide the IP CIDR block address for your VPC's network. Ideally,  provide a /16 subnet that will provide you with a good 65,531 IP addresses to use.



VPC name: Provide a suitable name for your VPC. In this case, I have standardized and  used the following naming convention: --

; so in our case, this translates to US-WEST-PROD-1. 

Public subnet: Now, since we are going with a public and private subnet combination  scenario, we have to fill in our public subnet details here. Provide a suitable subnet block  for your instances to use. In this case, I have provided a /24 subnet which provides a good  251 usable IP addresses:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 29 *



Availability Zone: Here's the fun part! You can deploy your subnets in any availability  zone available in that particular region. Now, US­WEST (Oregon) has three AZs and you  can use any of those there. In my case, I have gone ahead and selected us­west­2a as  the default option.



Public subnet name: Provide a suitable public subnet reference name. Here, too, I have  gone ahead and used the standard naming convention, so this particular subnet gets  called as US­WEST­PROD­WEB, signifying the web server instances that will get  deployed here.



Private subnet, Availability zone, Private subnet name: Go ahead and fill out the  private subnet's details using the similar IP addressing and naming conventions.  Remember that although you can set up your private subnet in a different AZ, as compared to the public subnet, ideally doing that is not recommended. If you really want to set up a  failover­like scenario, then create a separate public and private subnet environment in a 

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 30 *

different AZ altogether, for example, us-west-2c. So, even in case us-west-2a suffers  an outage, which by the way can happen, your failover subnets will still be functioning out  of the us­west­2c AZ. Refer to the following screenshot:

Next up, we specify the details of our NAT instance: 

Instance type: Select your NAT instance type from the available dropdown menu. In my  case, I have gone ahead and selected the t2.micro instance type as that is the smallest  type available. Do remember that selecting any other option will incur additional costs as  only the t2.micro instance type is covered under the free tier eligibility.



Key pair name: Select an already existing key pair from the dropdown list. Make sure you  have this particular key pair stored safely on your local computer, as without it you will not  be able to SSH into the NAT instance. You can alternatively create a new key pair here as  well using the same EC2 Management Console.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 31 *

Moving on, we now add the S3 endpoints to our particular subnet. You can add the endpoint  to either your private or public subnets, or both of them, depending on your requirements. 

Subnet: As per our VPC's requirements, the S3 endpoint is only made available to the  public subnet, so go ahead and select the Public subnet option from the dropdown list.



Policy: You can either choose to allow any user or service within the newly created VPC to access your S3 or specify a custom IAM access policy as you see fit. In our case, let's go  ahead and select Full Access for now.



Enable DNS hostnames: Enabling this option will provide your instances with the ability to resolve their DNS hostnames on the Internet. Select the Yes option and continue.



Hardware tenancy: Although a VPC runs off a completely isolated network environment,  the underlying server hardware is still shared by default. You can change this tenancy  option by selecting either the default or dedicated option from the dropdown list provided.

Once all the required information is filled out, go ahead and click on the Create  VPC option. The VPC creation takes a few seconds to complete. You can even view  your new VPC's default routes, security groups, and Network ACLs being created.  Toward the end you will notice your NAT instance powering on, and after a few  seconds of deployment your VPC is now ready for use! Here's a handy tip for all first timers! As soon as your NAT instance is created, you  can go ahead and change its default instance type to t1.micro from the EC2  Management Dashboard. To do so, first open up the EC2 Management Dashboard in a new tab on your  browser. You should see an instance in the running state, as shown in the following 

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 32 *

screenshot. First up, stop your instance using the Actions tab. Select the Instance  State option and click on Stop. Wait for the instance state to change to Stop before  you proceed any further. Next, select the instance, and from the Actions tab,  select Instance Settings and then Change Instance Type.

From the Change Instance Type dialog box, select the t1.micro option and click  on Apply. Voila! Your NAT instance is now officially a part of your free tier as well!  Simple, isn't it! Let's have a quick look at what actually happens behind the scenes when you create a VPC using the VPC Wizard. First up, let's look at the VPC itself.

Viewing VPCs Once the VPC is created and ready, you can view it from the VPC Management  Dashboard as well. Simply select the Your VPCs option from  the Navigation pane provided on the left­hand side of the dashboard. Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 33 *

You should see your newly created VPC, as shown in the following screenshot. You  can use the search bar to filter out results as well. Select the particular VPC to view its details.

Use the Summary tab to view the description of your selected VPC. Here, you can view the  VPC's default DHCP options set as well as the Tenancy and DNS hostnames and DNS  resolution options. You can optionally change these values by selecting the particular VPC  and from the Actions tab, selecting either Edit DHCP Options Set or Edit DNS Hostnames. You can create additional VPCs as well using the Create VPC option; however, as a good  practice, always keep things simple and minimal. Don't go over creating VPCs. Rather use  and create as many subnets as you require. Speaking of subnets, let's have a look at newly  created VPC's two subnets!

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 34 *

Listing out subnets You can view, add, and modify existing subnets using the VPC Management  Dashboard as well. Simply select the Subnetsoption from the Navigation Pane. This will list out all the subnets present in your account, so use the search bar to filter out  the new ones that we just created. Type in the name of the subnet in the Search bar  until the particular subnet gets listed out, as shown in the following screenshot:

You can view additional information associated with your subnet by simply selecting it and  viewing the Summary tab. The Summary tab will list out the particular subnet's  associated Route table, Network ACL, CIDR, and State as well. Besides these values, you  can also configure your subnet's ability to auto assign public IPs to its instances. By default,  this feature is disabled in your subnet, but you can always enable it as per your requirements.  To do so, simply select your Public Subnet, and from the Subnet Actions tab, select the  option, as shown in the following screenshot:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 35 *

In the Modify Auto­Assign Public IP dialog box, simply select the Enable auto­ assign public IP option and click on Save to complete the change setting. Do note  that you can always override this behavior for each individual instance at its launch  time. Besides the Summary tab, the Subnet Dashboard option also provides additional  tabs such as Route Table, Network ACL, and so on. You can use these tabs to view  the subnet's associated route table as well the network ACL; however, you cannot add or modify the individual rules from here. To add or modify rules, you need to go to  the Network ACL option or the Route Tablesoption from the navigation pane. Let's  have a quick look at the route tables created for our VPC.

Working with route tables As discussed earlier in this chapter, VPCs come with a default route table (Main  Route Table) associated with a subnet. So, since we have two subnets created in this VPC, we get two route tables as well, out of which one is the main route table. How do you tell whether a route table is the main one? Quite simple, actually! Just look for  the Main Column in the Route Table Dashboard, as shown in the following  screenshot. If the value in that column is Yes, then that particular route table is your  VPC's main route table. Now, here's a catch! If you do not explicitly associate a route  Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 36 *

table with a subnet, then the subnet ends up using the main route table. In our case,  both the route tables created do not have any subnets associated with them by  default, so let's first get that done.

To associate a route table with a subnet explicitly, first you need to select the particular route  table from the dashboard. In this case, I have selected the public subnet's router (US­WEST­ PROD­WEB­RT). Next, from the Subnet Associations tab, click on the Edit button, as  shown in the following screenshot. From here, you can select either of the two subnets that  are listed down; however, since this is a public subnet's route table, let's go ahead and select  the listed public subnet (US­WEST­PROD­WEB) as shown. Click on Save to save the  configuration changes.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 37 *

With this step completed, your subnet is now explicitly attached with a particular route  table. But what about the individual route rules? How do we list and modify them?  That's simple as well. Simply select the Routes tab to list your route table's existing  rule set. Here, you should see at least three route rules, as shown in the following  screenshot. The first rule is created by VPC for each and every route table, and it  basically allows communication within the VPC itself. You cannot delete this rule, so  don't even try it! The next rule is basically a VPC endpoint route rule. Remember the S3 endpoint that  we configured earlier with the public subnet? Well this rule will basically allow  communication to occur between the instances belonging to this subnet and Amazon  S3, and the best part is that this rule is auto­populated when you create a VPC  endpoint!

The final rule in the list basically allows for the instances to communicate over the Internet  using the Internet Gateway as the target. You can optionally choose to edit these rules by  selecting the Edit option. Once you have made your required changes, be sure to Save the  configuration changes before you proceed with the next steps. Don't forget to associate the 

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 38 *

private subnet (US­PROD­WEST­DB) with the remaining route table (US­WEST­PROD­DB­ RT) as well.

Listing Internet Gateways As discussed earlier in this chapter, Internet Gateways are scalable and redundant  virtual devices that provide Internet connectivity for your instances present in the VPC. You can list currently available Internet Gateways within your VPC by selecting  the Internet Gateways option from the VPC's navigation pane. The VPC wizard will create and attach one Internet Gateway to your VPC  automatically; however, you can create and attach an Internet Gateway to a VPC at  any time using the Internet Gateway. Simply click on the Create Internet  Gateway option and provide the VPC to which this Internet Gateway has to be  attached, that's it! You can list down available Internet Gateways and filter the results  using the search bar provided as well. To view your Internet Gateway's details, simply  select the particular Internet Gateway and click on the Summary tab. You should see  your Internet Gateway's ID, State (attached or detached), as well as  the Attachment state (available or not available), as shown in  the following screenshot:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 39 *

Also remember that to really use the Internet Gateway, your public subnet's route table must  contain a route rule that directs all Internet­bound traffic from the subnet to the Internet  Gateway

Working with security groups and Network ACLs The VPC is all about providing your applications a much more secure environment  than what your traditional EC2 service can offer. This security is provided in two layers in the form of security groups and Network ACLs. The security groups can be used to  set rules that can control both inbound and outbound traffic flow from the instance and hence work more at the instance level. The Network ACLs on the other hand operate  at the subnet level, either allowing or disallowing certain type of traffic to flow in and  out of your subnet. Important thing to remember here is that the Network ACLs are  actually optional and can be avoided altogether if your security requirements are at a  minimal. However, I would strongly recommend that you use both the security groups  and Network ACLs for your VPC environments. As the saying goes ­ better safe, than sorry! Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 40 *

Coming back to your newly created VPC, the VPC wizard creates and populates  a default Security Group and a default Network ACL option for you to use in an as­ is condition. The default Security Group has a single inbound and outbound rule, as  explained in the following: Default inbound security rule Source

Protocol

Port 

Remarks

Range Security_Group_ID

All

All

Permits inbound traffic from instances belonging  to the same security group

Default Outbound Security Rule Destination

Protocol

Port Range

Remarks

0.0.0.0/0

All

All

Permits all outbound traffic from the instances

You can add, edit, and modify the rules in the default Security Group; however, you  cannot delete it. As a good practice, it is always recommended that you do not use  this default Security Group but rather create your own. So, let's go ahead and create  three security groups: one for the web servers in the public subnet, one for the  database servers in the private subnet, and one for the specially created NAT  Instance. To create a new Security Group using the VPC Dashboard, select the Security  Groups option from the navigation pane. Next, from the Security Groups  dashboard, select Create Security Group, as shown in the following screenshot: Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 41 *

Using the Create Security Group wizard, fill in the required information as described in the  following: 

Name tag: A unique tag name for your Security Group.



Group name: A suitable name for your Security Group. In this case, I have provided it  as US­WEST­PROD­WEB­SG.



Description: An optional description for your security group.



VPC: Select the newly created VPC from the dropdown list, as shown in the following  screenshot. Click on Yes, Create once done.

Once your Security Group has been created, select it from the Security Groups  dashboard and click on the Inbound Rules tab. Click on the Edit option to add the following  rule sets:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 42 *

Web server inbound security rule Source

Protocol

Port Range

Remarks

0.0.0.0/0

TCP

22

Permit inbound SSH access to web server instance

0.0.0.0/0

TCP

80

Permit inbound HTTP access to web server instance

0.0.0.0/0

TCP

443

Permit inbound HTTPS access to web server instance

Similarly, click on the Outbound Rules tab and fill out the Security Group's outbound rules  as described in the following:

Web server outbound security rule Destination

Protocol

Port 

Remarks

Range DB_SECURITY_GROUP

TCP

1433

Permits outbound Microsoft SQL Server traffic  to the database servers

DB_SECURITY_GROUP

TCP

3306

Permits outbound MySQL traffic to the  database servers

Replace DB_SECURITY_GROUP with the Security Group ID of your database server's  Security Group. Remember to save the rules by selecting the Save option, as shown in the  following screenshot:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 43 *

Similarly, let's go ahead and create a Security Group for our database servers as well.  Populate the inbound rules as described in the following:

Database server inbound security rule Source

Protocol

Port 

Remarks

Range WEB_SECURITY_GROUP

TCP

1433

Permits Web Server instances  to access the Microsoft SQL Server

WEB_SECURITY_GROUP

TCP

3306

Permits Web Server instances  to access the MySQL Server

Replace WEB_SECURITY_GROUP with the Security Group ID of your web server's Security  Group ID and save the rules before you continue with the outbound rules additions:

Database server outbound security rule Source

Protocol

Port Range

Remarks

0.0.0.0/0

TCP

80

Permit outbound HTTP access to database server instance

0.0.0.0/0

TCP

443

Permit outbound HTTPS access to database server instance

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 44 *

Note that here we are permitting only the outbound Internet access to the database  servers so that they can receive important patches and updates from the net. In  reality, the Internet bound traffic from these servers will be routed through the NAT  instance, which will forward the traffic to the Internet via your Internet Gateway. Finally, go ahead and create the NAT instance's Security Group. Populate the inbound security rules as mentioned in the following: NAT instance inbound security rule Source

Protocol

Port Range

Remarks

0.0.0.0/0

TCP

22

Permits inbound SSH access to  the NAT Instance

192.168.1.0/24

TCP

80

Permit inbound HTTP access to the NAT  instance

192.168.1.0/24

TCP

443

Permit inbound HTTPS access to NAT  instance

NAT instance outbound security rule Source

Protocol

Port Range

Remarks

0.0.0.0/0

TCP

80

Permit outbound HTTP access to NAT  instance

0.0.0.0/0

TCP

443

Permit outbound HTTPS access to NAT  instance

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 45 *

With the security groups created, you are now ready to launch your instances into the VPC.  Let's have a quick look at the steps required to do so!

Launching instances in your VPC Once your VPC is ready and the security groups and Network ACLs have been modified as per  requirement, you are now ready to launch instances within your VPC. You can either launch  instances directly from the VPC Management Dashboard or from the EC2 Management  Console as well. In this case, let's go ahead and use the EC2 Management Console.

Creating the web servers From the EC2 Management Console, select the Launch Instance option. This will bring up  the Instance Wizard, using which you can create and launch your web server instances

From the next page, select any instance type for the new web server instances. In my  case, I went ahead and used the default t1.micro instance type. Next, from the Configure Instance Details page, select the newly created VPC (US­ WEST­PROD­1) from the Networkdropdown list and provide the web server's public  subnet (US­WEST­PROD­WEB), as shown in the following screenshot. You can  optionally choose to change the Auto­assign Public IP setting; however, in this case, make sure that this setting is set to Enable otherwise your web server instances will  not receive their public IPs.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 46 *

Add the required Storage, Tag the instance, and provide it with the web  server Security Group that we created a while back. Once you have completed the  formalities, review your instance's settings and finally launch it in your VPC! Once the instance starts, verify whether it received both the private and the public IP  or not. Log in to your web server instance and check whether it can reach the Internet  or not by simply pinging to one of Google's DNS servers like 8.8.8.8. If all goes well,  then your web server instances are all ready for production use!

Creating the database servers The same process applies for the database servers as well. Simply remember to  select the correct subnet (US­WEST­PROD­DB) for the database servers, as shown  in the following screenshot:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 47 *

Also note the Auto­assign Public IP setting for the database server's private subnet. By  default, this should be disabled for the private subnet as we don't want our database  instances to communicate with the Internet directly. All Internet­bound traffic from the  database servers will pass via the NAT instance only. But how do you test whether your  database servers are working correctly? By design, you cannot SSH into the database servers directly from your local desktops as the private subnet is isolated from the Internet. So, an  alternative would be to set up something called as a Bastion Host. A Bastion Host is a  special instance that acts as a proxy using which you can SSH into your database instances.  This Bastion Host will be deployed in your public subnet and will basically only route SSH  traffic from your local network over to the database server instances. But remember, this  feature comes with its own set of security risks! Running a weak or poorly configured Bastion  Host can prove to be harmful in production environments, so use them with care!

Planning next steps Well we have covered a lot in this chapter, but there are a few things still that you can  try out on your own with regards to VPCs. First up, is cleaning up a VPC! Creating a  VPC is easy enough and so is its deletion. You can delete an unused VPC from  the VPC Management dashboard by simply selecting the VPC, clicking on  the Actions tab, and selecting the Delete VPCoption. This will bring up the Delete  VPC dialog as shown in the following screenshot:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 48 *

As you can see, the delete VPC option will delete all aspects of your VPC, including  subnets, Network ACLs, Internet Gateways, and so on. You can optionally even delete any VPN connections as well by selecting the Delete VPN Connections when  deleting the VPC checkbox. Remember that once you delete a VPC, you can't  recover it back, so make sure that you don't have any active instances running on it  before you go ahead and delete it. Also remember to clean up on the instances as  well, especially the NAT Instance and the Bastion Host if you have created them. The second thing that I would recommend trying out is called as VPC peering. VPC  peering is nothing more than network connections between two different VPCs.  Instances in one VPC communicate with instances present in another VPC using their  private IP addresses alone, so there is no need to route the traffic over the Internet as  well. You can connect your VPC with a different VPC that is either owned by you or by someone else's Bastion Host. All it needs is a request to be generated from the  source VPC and sent to the destination VPC, along with a few route rules that will  allow the traffic to flow from one point to the other. The following is the image  describing the VPC peering: Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 49 *

You can read more about VPC peering  at http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc­peering.html  . The third thing that really is worth testing out is the hardware VPN connectivity with  your VPC. I know you are probably thinking that since it's a hardware VPN  connectivity, it means that I need some special hardware equipment like a router and  so on. Well that's not quite true! You can set up an easy VPN connection using  software as well, for example, OpenVPN. OpenVPN basically allows you to create a  secure network connection from your local network to Amazon VPC using a VPN  connection. All you need to do is deploy an OpenVPN server in your VPC and configure that to  accept incoming traffic from your private network. Then, install an OpenVPN client on  your remote desktop and try connecting to the OpenVPN server placed in the VPC. If  all goes well, you should have access to your VPC instances from your local desktop!  Do note that you will have to open up additional security rules and network ACLs to  allow this type of traffic to flow through your VPC subnet.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 50 *

Last but not least, I would also recommend for you to have a look at VPC's Flow Logs. This is a simple logging feature provided in VPC to capture traffic information and  store it using Amazon CloudWatch Logs. Flow Logs can help you analyze your  network traffic flow for bottlenecks, observe certain traffic trends, as well as monitor  traffic that reaches your instances. You can read more about Flow Logs  at http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/flow­logs.html  .

Best practices and recommendations The following are some key best practices and recommendations to keep in mind  when using VPCs: 

Plan and design your VPC before actually implementing one. Determine the right choice of subnet that your application will need and build your VPC around it.



Choose your VPC's network block allocation wisely. A /16 subnet can provide you with a  potential 65,534 IP addresses that rarely will get utilized. So ideally, go for a /18 (16,382 IP addresses) or a /20 (4094 IP addresses) as your VPC network choice.



Always plan and have a set of spare IP address capacity for your VPC. For example,  consider the network block for my VPC as 192.168.0.0/18.



In this case, we design the subnet IP addressing as follows: o

192.168.32.0/19 Public Subnet

o

192.168.64.0/19 Public Subnet spares

o

192.168.128.0/20  Private Subnet

o

192.168.192.0/20  Private Subnet spares

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 51 *



Remember that you cannot edit a network block's size once it is created for a VPC. The only way to change the network block is by deleting this VPC and creating a  new one in its place.



Use different security groups to secure and manage traffic flows from your  instances. For example, a separate Security Group for web servers and a different  one for your database servers. Avoid using the default security groups at all times.



Leverage multiple AZs to distribute your subnets across geographies. For example, the US­WEST region has three AZs, namely us­west­2a, us­west­2b, and us­west­ 2c. So an ideal situation would have you divide your VPC's network block and  create subnets in each of these AZs evenly. The more AZs, the better the fault  tolerance for your VPC.



Leverage IAM to secure your VPC at the user level as well. Create dedicated users with restricted access to your VPC and its resources.



Create and stick with a standard naming convention so that your VPC's  resources can be easily identified and tagged. For example, in our scenarios, we  named the VPC as US­WEST­PROD­1, which clearly identifies this particular VPC  to be hosted in the US­WEST region and to be a PRODUCTION environment.

VPC Cloud Formation Recipes Networking is a foundational component of using other AWS services such as EC2, RDS, and  others. Using constructs such as VPCs and NAT gateways gives you the capability and  confidence to secure your resources at a networking level. At a DNS level, Route 53 provides 

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 52 *

connectivity to your users in a responsive and fault­tolerant way that ensures the best  performance in a variety of scenarios.

Building a secure network In this recipe, we're going to build a secure network (VPC) in AWS. This network will consist  of two public and private subnets split across two Availability Zones. It will also allow inbound  connections to the public subnets for the following: 

SSH (port 22)



HTTP (port 80)



HTTPS (port 443)

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 53 *

Building a secure network

Getting ready Before we proceed, you're going to need to know the names of at least two Availability Zones in the region we're deploying to. The recipes in this book will typically deploy  to us-east-, so to get things moving you can just use the following:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 54 *



us-east-1a



us-east-1b

Note: When you create an AWS account, your zones are randomly allocated. This means that us-east1a in your account isn't necessarily the same data center as us-east-1a in my account.

How to do it... Go ahead and create a new CloudFormation template for our VPC. Just a heads­up: this will  be one of the larger templates that we'll create in this book: 1. The first two Parameters correspond to the Availability Zones we discussed previously. We don't provide any default values for these parameters, to maintain region portability:

Parameters: AvailabilityZone1: Description: Availability zone 1 name (e.g. us-east-1a) Type: AWS::EC2::AvailabilityZone::Name AvailabilityZone2: Description: Availability zone 2 name (e.g. us-east-1b) Type: AWS::EC2::AvailabilityZone::Name

2. The shell of our VPC has now been created. At this point, it's not connected to the Internet, so it's not entirely useful to us. We need to add an Internet gateway and attach it to our  VPC. Go ahead and do that, as follows:

Resources: # VPC & subnets ExampleVPC: Type: AWS::EC2::VPC Properties: CidrBlock: !Ref VPCCIDR

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 55 *

EnableDnsSupport: true EnableDnsHostnames: true Tags: - { Key: Name, Value: Example VPC } PublicSubnetA: Type: AWS::EC2::Subnet Properties: AvailabilityZone: !Ref AvailabilityZone1 CidrBlock: !Ref PublicSubnetACIDR MapPublicIpOnLaunch: true VpcId: !Ref ExampleVPC Tags: - { Key: Name, Value: Public Subnet A } PublicSubnetB: Type: AWS::EC2::Subnet Properties: AvailabilityZone: !Ref AvailabilityZone2 CidrBlock: !Ref PublicSubnetBCIDR MapPublicIpOnLaunch: true VpcId: !Ref ExampleVPC Tags: - { Key: Name, Value: Public Subnet B } PrivateSubnetA: Type: AWS::EC2::Subnet Properties: AvailabilityZone: !Ref AvailabilityZone1 CidrBlock: !Ref PrivateSubnetACIDR VpcId: !Ref ExampleVPC Tags: - { Key: Name, Value: Private Subnet A } PrivateSubnetB: Type: AWS::EC2::Subnet Properties: AvailabilityZone: !Ref AvailabilityZone2 CidrBlock: !Ref PrivateSubnetBCIDR VpcId: !Ref ExampleVPC Tags: - { Key: Name, Value: Private Subnet B }

Note: AWS reserves a small number of IP addresses in your IP space for AWS­specific services.  The VPC DNS server is one such example of this. It's usually located at the second ( *.2) IP  address in the block allocated to your VPC.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 56 *

3. The default values we provide for the subnets will allocate 512 IP addresses to each  subnet:

VPCCIDR: Description: CIDR block for VPC Type: String Default: "172.31.0.0/21" # 2048 IP addresses PublicSubnetACIDR: Description: CIDR block for public subnet A Type: String Default: "172.31.0.0/23" # 512 IP address PublicSubnetBCIDR: Description: CIDR block for public subnet B Type: String Default: "172.31.2.0/23" # 512 IP address PrivateSubnetACIDR: Description: CIDR block for private subnet A Type: String Default: "172.31.4.0/23" # 512 IP address PrivateSubnetBCIDR: Description: CIDR block for private subnet B Type: String Default: "172.31.6.0/23" # 512 IP address

4.    Now we can start to define Resources. We'll start by defining the VPC itself, as well as  the two public and two private subnets inside it:

# Internet Gateway ExampleIGW: Type: AWS::EC2::InternetGateway Properties: Tags: - { Key: Name, Value: Example Internet Gateway } IGWAttachment: Type: AWS::EC2::VPCGatewayAttachment DependsOn: ExampleIGW Properties: VpcId: !Ref ExampleVPC InternetGatewayId: !Ref ExampleIGW

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 57 *

5. We need to create a couple of route tables. The first one we'll focus on is the public route  table. We'll assign this route table to the two public subnets we've created. This route table will have just one route in it, which will direct all Internet­bound traffic to the Internet  gateway we created in the previous step:

# Public Route Table # Add a route for Internet bound traffic pointing to our IGW # A route for VPC bound traffic will automatically be added PublicRouteTable: Type: AWS::EC2::RouteTable Properties: VpcId: !Ref ExampleVPC Tags: - { Key: Name, Value: Public Route Table } PublicInternetRoute: Type: AWS::EC2::Route DependsOn: IGWAttachment Properties: RouteTableId: !Ref PublicRouteTable GatewayId: !Ref ExampleIGW DestinationCidrBlock: "0.0.0.0/0" RouteAssociationPublicA: Type: AWS::EC2::SubnetRouteTableAssociation Properties: RouteTableId: !Ref PublicRouteTable SubnetId: !Ref PublicSubnetA RouteAssociationPublicB: Type: AWS::EC2::SubnetRouteTableAssociation Properties: RouteTableId: !Ref PublicRouteTable SubnetId: !Ref PublicSubnetB

6.  We'll create the private route table in a similar fashion. Since the private subnet is isolated from the Internet

# Private Route Table # We don't add any entries to this route table because there is no NAT gateway # However a route for VPC bound traffic will automatically be added PrivateRouteTable:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 58 *

Type: AWS::EC2::RouteTable Properties: VpcId: !Ref ExampleVPC Tags: - { Key: Name, Value: Private Route Table } PrivateSubnetAssociationA: Type: AWS::EC2::SubnetRouteTableAssociation Properties: RouteTableId: !Ref PrivateRouteTable SubnetId: !Ref PrivateSubnetA PrivateSubnetAssociationB: Type: AWS::EC2::SubnetRouteTableAssociation Properties: RouteTableId: !Ref PrivateRouteTable SubnetId: !Ref PrivateSubnetB

7.   We can now focus on the security aspects of our network. Let's focus on the public  subnets. These are the subnets you'll add your load balancers to; you'll also add things  such as bastion boxes and NAT gateways. So we need to add a Network ACL (NACL) with several entries: 

Allow outbound traffic to all ports. Outbound access is unrestricted from hosts in  our public subnets.



Allow inbound traffic to ephemeral ports (above 1024). This ensures that packets  returned to us from our outbound connections are not dropped.



Allow inbound access to low port numbers for SSH, HTTP, and HTTPS ( 22, 80,  and 443):

# Public NACL PublicNACL: Type: AWS::EC2::NetworkAcl Properties: VpcId: !Ref ExampleVPC Tags: - { Key: Name, Value: Example Public NACL } # Allow outbound to everywhere NACLRulePublicEgressAllowAll: Type: AWS::EC2::NetworkAclEntry

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 59 *

Properties: CidrBlock: "0.0.0.0/0" Egress: true Protocol: 6 PortRange: { From: 1, To: 65535 } RuleAction: allow RuleNumber: 100 NetworkAclId: !Ref PublicNACL # Allow outbound to VPC on all protocols NACLRulePublicEgressAllowAllToVPC: Type: AWS::EC2::NetworkAclEntry Properties: CidrBlock: !Ref VPCCIDR Egress: true Protocol: -1 RuleAction: allow RuleNumber: 200 NetworkAclId: !Ref PublicNACL # Allow inbound from everywhere to ephemeral ports (above 1024) NACLRulePublicIngressAllowEphemeral: Type: AWS::EC2::NetworkAclEntry Properties: CidrBlock: "0.0.0.0/0" Protocol: 6 PortRange: { From: 1024, To: 65535 } RuleAction: allow RuleNumber: 100 NetworkAclId: !Ref PublicNACL # Allow inbound from everywhere on port 22 for SSH NACLRulePublicIngressAllowSSH: Type: AWS::EC2::NetworkAclEntry Properties: CidrBlock: "0.0.0.0/0" Protocol: 6 PortRange: { From: 22, To: 22 } RuleAction: allow RuleNumber: 200 NetworkAclId: !Ref PublicNACL # Allow inbound from everywhere on port 443 for HTTPS NACLRulePublicIngressAllowHTTPS: Type: AWS::EC2::NetworkAclEntry Properties: CidrBlock: "0.0.0.0/0" Protocol: 6 PortRange: { From: 443, To: 443 } RuleAction: allow RuleNumber: 300

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 60 *

NetworkAclId: !Ref PublicNACL # Allow inbound from everywhere on port 80 for HTTP NACLRulePublicIngressAllowHTTP: Type: AWS::EC2::NetworkAclEntry Properties: CidrBlock: "0.0.0.0/0" Protocol: 6 PortRange: { From: 80, To: 80 } RuleAction: allow RuleNumber: 400 NetworkAclId: !Ref PublicNACL # Allow inbound from VPC on all protocols NACLRulePublicIngressAllowFromVPC: Type: AWS::EC2::NetworkAclEntry Properties: CidrBlock: !Ref VPCCIDR Protocol: -1 RuleAction: allow RuleNumber: 500 NetworkAclId: !Ref PublicNACL NACLAssociationPublicSubnetA: Type: AWS::EC2::SubnetNetworkAclAssociation Properties: NetworkAclId: !Ref PublicNACL SubnetId: !Ref PublicSubnetA NACLAssociationPublicSubnetB: Type: AWS::EC2::SubnetNetworkAclAssociation Properties: NetworkAclId: !Ref PublicNACL SubnetId: !Ref PublicSubnetB

8. We need to do the same for our private subnets. These subnets are somewhat easier to  deal with. They should only be allowed to talk to hosts within our VPC, so we just need to  add some NACLs allowing inbound and outbound traffic to our VPCs IP range:

# Private NACL PrivateNACL: Type: AWS::EC2::NetworkAcl Properties: VpcId: !Ref ExampleVPC Tags:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 61 *

- { Key: Name, Value: Example Private NACL } # Allow all protocols from VPC range NACLRulePrivateIngressAllowVPC: Type: AWS::EC2::NetworkAclEntry Properties: CidrBlock: !Ref VPCCIDR Protocol: -1 RuleAction: allow RuleNumber: 100 NetworkAclId: !Ref PrivateNACL # Allow TCP responses from everywhere NACLRulePrivateIngressAllowEphemeral: Type: AWS::EC2::NetworkAclEntry Properties: CidrBlock: "0.0.0.0/0" Protocol: 6 PortRange: { From: 1024, To: 65535 } RuleAction: allow RuleNumber: 200 NetworkAclId: !Ref PrivateNACL # Allow outbound traffic to everywhere, all protocols NACLRulePrivateEgressAllowVPC: Type: AWS::EC2::NetworkAclEntry Properties: CidrBlock: "0.0.0.0/0" Egress: true Protocol: -1 RuleAction: allow RuleNumber: 100 NetworkAclId: !Ref PrivateNACL NACLAssociationPrivateSubnetA: Type: AWS::EC2::SubnetNetworkAclAssociation Properties: NetworkAclId: !Ref PrivateNACL SubnetId: !Ref PrivateSubnetA NACLAssociationPrivateSubnetB: Type: AWS::EC2::SubnetNetworkAclAssociation Properties: NetworkAclId: !Ref PrivateNACL SubnetId: !Ref PrivateSubnetB

9. Finally, we'll add some Outputs to our template. These outputs are usually candidates for  feeding into other templates or components of automation:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 62 *

Outputs: ExampleVPC: Value: !Ref ExampleVPC PublicSubnetA: Value: !Ref PublicSubnetA PublicSubnetB: Value: !Ref PublicSubnetB PrivateRouteTable: Value: !Ref PrivateRouteTable PublicRouteTable: Value: !Ref PublicRouteTable PrivateSubnetA: Value: !Ref PrivateSubnetA PrivateSubnetB: Value: !Ref PrivateSubnetB

10. You can go ahead and create your VPC in the web console or via the CLI using the  following command:

aws cloudformation create-stack \ --stack-name secure-vpc \ --template-body file://07-building-a-secure-network.yaml \ --parameters \ ParameterKey=AvailabilityZone1,ParameterValue= \ ParameterKey=AvailabilityZone2,ParameterValue=

How it works... When you run this template, AWS will go ahead and create an isolated, secure network just for  you. While it contains a number of resources and concepts which will be familiar to network  administrators, it's essentially an empty shell, which you can now go ahead and populate.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 63 *

For example, each VPC contains a virtual router. You can't see it and you can't log into it to  perform any special configuration, but you can customize its behavior by modifying the route  tables in this template. The NACLs we've deployed are not stateful and should not be considered a substitution for  security groups. NACLs are complementary to security groups, which are stateful and frankly  much easier to change and manage than NACLs. While the NACLs in our recipe allow everywhere (0.0.0.0/0) to make inbound connections to port 22, for example, you'll want to use security  groups to lock this down to a specific IP range (your corporate data center, for example).

There's more... Actually, there's a lot more. Despite the amount of code in this recipe, we've really only covered  the basics of what's possible with VPCs and networking in AWS. Here are some of the main VPC  topics you'll encounter as you progress with your VPC usage: 

Direct Connect: This is a method of connecting your DC to your VPC using a private,  dedicated pipe. Doing this often provides better network performance, and may also be  cheaper than a VPN connection over the Internet.



Virtual Private Gateway (VPN): You can configure your VPC to connect to your corporate  DC over the Internet via VPN. This requires that you run supported VPN hardware in your  DC.



IPv6 support was added recently. We've left it out to keep things simple.



VPC endpoints: This feature exposes AWS endpoints inside your VPC so that you don't  have to route traffic over public Internet to consume them. Only S3 is supported at the time of writing.



VPC peering: You can peer a VPC to one or more VPCs so that (unencrypted) traffic can  flow between them. The IP ranges must not clash and, while the peering is free, you will  still need to pay for traffic between VPCs. Transitive peering isn't supported, so if you need

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 64 *

traffic to traverse VPCs you'll require a VPN/routing appliance of some kind. Cross­account VPC peering is supported (we use this feature quite often), but cross­region peering isn't  yet available. 

VPC sizing: o

IPv4: You can deploy networks between sizes /28 and /16.

o

IPv6: Your VPCs will be fixed in size at /56.

o

Once your VPC has been deployed you can't change its size. If you run out of IP  space, your only option is to deploy a larger VPC and migrate everything (ouch!), or you can perhaps mitigate your problem with VPC peering. 



VPC flow­logs: You will want to enable VPC flow­logs in order to monitor traffic and do  any kind of network debugging.



Multicast traffic isn't supported.



Subnets must reside in a single availability zone; they can't span Availability Zones.



Elastic Load Balancers (ELBs) can scale out to use a lot of private IP addresses if you  are sending a large amount of traffic through them. Keep this in mind when you're sizing  your subnets.



The number of VPCs you can deploy is limited to five per region, per account. You can  request to increase this limit if necessary. Internet gateways have the same limit, and  increasing one limit increases the other.



The default VPC: o

First and foremost, the default VPC is created automatically for you when you  create your account. It has some different properties and behaviors to the VPCs  you create for yourself.

o

If you try to launch an EC2 instance without specifying a subnet ID, AWS will  attempt to launch it in your default VPC.

o

It consists of only public subnets. These subnets are configured to provide a public  IP address to all instances by default.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 65 *

o

It's possible to delete the default VPC in a region. If you do this by mistake, or have simply decided that you'd like to undo this action, you'll need to log a support ticket  with AWS to have them create a new one for you.

Creating a NAT gateway Unless required, your instances should not be publicly exposed to the Internet. When  your instances are on the Internet, you have to assume they will be attacked at some  stage. This means most of your workloads should run on instances in private subnets.  Private subnets are those that are not connected directly to the Internet. In order to give your private instances access to the Internet, you use network  address translation (NAT). A NAT gateway allows your instances to initiate a  connection to the Internet, without allowing connections from the Internet.

Getting ready For this recipe, you must have the following existing resources: 

A VPC with an Internet gateway (IGW)



A public subnet



A private subnet route table

You will need the IDs for the public subnet and private subnet route table. Both of these  resources should be in the same AZ.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 66 *

How to do it... 1. Start with the usual CloudFormation template version and description:

AWSTemplateFormatVersion: "2010-09-09" Description: Create NAT Gateway and associated route.

2. The template must take the following required parameters:

Parameters: PublicSubnetId: Description: Public Subnet ID to add the NAT Gateway to Type: AWS::EC2::Subnet::Id RouteTableId: Description: The private subnet route table to add the NAT Gateway route to Type: String

3. In the Resources section, define an Elastic IP (EIP) that will be assigned to the NAT  gateway:

Resources: EIP: Type: AWS::EC2::EIP Properties: Domain: vpc

4. Create the NAT gateway resource, assigning it the EIP you just defined in the public  subnet:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 67 *

NatGateway: Type: AWS::EC2::NatGateway Properties: AllocationId: !GetAtt EIP.AllocationId SubnetId: !Ref PublicSubnetId

5. Finally, define the route to the NAT gateway and associate it with the private subnet's route table:

Route: Type: AWS::EC2::Route Properties: RouteTableId: !Ref RouteTableId DestinationCidrBlock: 0.0.0.0/0 NatGatewayId: !Ref NatGateway

6. Save the template with a known filename; for example, 07-nat-gateway.yaml . 7. Launch the template with the following CLI command:

aws cloudformation create-stack \ --stack-name nat-gateway \ --template-body file://07-nat-gateway.yaml \ --parameters \ ParameterKey=RouteTableId,ParameterValue= \ ParameterKey=PublicSubnetId,ParameterValue=

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 68 *

How it works... The parameters required for this recipe are as follows: 

A public subnet ID



A private subnet route table ID

The public subnet ID is needed to host the NAT gateway, which must have Internet  access. The private subnet route table will be updated with a route to the NAT  gateway. Using the AWS NAT gateway service means that AWS takes care of hosting and  securing the service for you. The service will be hosted redundantly in a single AZ. n the unlikely (but possible) event of an AZ outage, you should deploy a NAT gateway  per subnet. This means that if one NAT gateway goes offline, instances in the other  AZ can continue to access the Internet as normal. You are deploying your application  in multiple subnets, aren't you? This recipe will only work if you have created your own private subnets, as the default  subnets in a new AWS account are all public. Instances in a public subnet have direct access to the Internet (via an IGW), so they do not need a NAT gateway.

Network logging and troubleshooting One of the benefits of using virtualized infrastructure is that you can get a level of  introspection that is difficult or costly with physical hardware. Being able to quickly  Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 69 *

switch on logging at a network­device level is an extremely useful feature, especially  when getting used to the interactions between VPCs, subnets, NACLs, routing, and  security groups. In this recipe, we will turn on logging for our network resources. You could do this all  the time, to give yourself another layer for monitoring and auditing, or you could  selectively enable it during troubleshooting, saving yourself any additional datastorage charges.

Getting ready For this recipe, you must have a VPC to log activity on.

How to do it... 1. Start by defining the template version and description:

AWSTemplateFormatVersion: "2010-09-09" Description: Flow logs for networking resources

2. Define the Parameters for the template. In this case, it is just the VpcId to turn logging on  for:

Parameters: VpcId: Type: String Description: The VPC to create flow logs for

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 70 *

3. Create the Resources section of the template and define the log group to use to send our  flow­logs to:

Resources: LogGroup: Type: AWS::Logs::LogGroup DeletionPolicy: Delete Properties: LogGroupName: LogGroup

4. Next we define the IAM role that will give the flow­logs service permission to write the logs:

IamRole: Type: AWS::IAM::Role Properties: AssumeRolePolicyDocument: Version: "2012-10-17" Statement: Effect: Allow Principal: Service: vpc-flow-logs.amazonaws.com Action: sts:AssumeRole Policies: PolicyName: CloudWatchLogsAccess PolicyDocument: Version: "2012-10-17" Statement: Action: - logs:CreateLogGroup - logs:CreateLogStream - logs:PutLogEvents - logs:DescribeLogGroups - logs:DescribeLogStreams Effect: Allow Resource: "*"

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 71 *

5. Finally, we define the flow­log itself:

FlowLog: Type: AWS::EC2::FlowLog DependsOn: LogGroup Properties: DeliverLogsPermissionArn: !GetAtt IamRole.Arn LogGroupName: LogGroup ResourceId: !Ref VpcId ResourceType: VPC TrafficType: ALL

6. Save the template, and give it a known filename such as 07-flow-logs.yaml. 7. Create the flow­logs and associated resources by creating the template with the following  command:

aws cloudformation create-stack \ --stack-name VpcFlowLogs \ --template-body file://07-flow-logs.yml \ --capabilities CAPABILITY_IAM \ --parameters ParameterKey=VpcId,ParameterValue=

8. Once launched (and assuming you have network activity), you will be able to see your  flow­log in the CloudWatch logs console.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 72 *

How it works... The only parameter required for this template is the VPC ID to target. We specifically  target a VPC to turn on flow­logging for, because it gives us the most bang for buck.  While you can enable flow­logs for subnets and Elastic Network Interfaces (ENIs)  individually, if you enable them on a VPC you get flow­logs for all the networking  resources contained in that VPC—which includes subnets and ENIs. In the resources section, we start by explicitly defining the log group to hold the flow­ logs. If you don't create the log group yourself (and specify it in your flow­log resource  configuration), a log group will be created for you. This means that you will still be able to use flow­logs, but the log group won't be managed by CloudFormation and will have to be maintained (for example, deleted) manually. We have also set a deletion  policy of delete for our log group. This means it will be deleted if the CloudFormation  stack is deleted, which is fine for a demonstration such as this. If using in  a real environment (such as production), remove the DeletionPolicy property and its  value. Next we define the IAM role to use. Via the AssumeRolePropertyDocument value, we  give the AWS flow­logs service permission to assume this role. Without this access,  the flow­logs service cannot access the account. In the Policiesproperty, we give the  role permission to create and update log groups and streams. Finally, now that we have created the dependent resources, we define the flow­log  resource itself. You don't need to define the resources in order of dependencies, but it  is usually easier to read if you do. In the resource, we also define 

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 73 *

a DependsOnrelationship to the log group we defined earlier, so that the log group is  ready to receive the flow­logs when it is created. The final step is to launch the template you have created, passing the VPC ID as  parameter. As this template creates an IAM role to allow the VPC service to send logs  to CloudWatch logs, the command to create the stack must be given  the CAPABILITY_IAM flag to signify that you are aware of the potential impact of  launching this template.

There's more... Turning on logging is just the start of the troubleshooting process. There are a few  other things you should be aware of when using flow­logs.

Log format Once logging is enabled, you can view the logs in the CloudWatch logs console. Here is a  summary of the type of information you will see in the flow­log (in order): 

The VPC flow­logs version



The AWS account ID



The ID of the network interface



The source IPv4 or IPv6 address



The destination IPv4 or IPv6 address



The source port of the traffic



The destination port of the traffic

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 74 *



The IANA protocol number of the traffic



The number of packets transferred



The number of bytes transferred



The start time of the capture window (in Unix seconds)



The end time of the capture window (in Unix seconds)



The action associated with the traffic; for example, ACCEPT or REJECT



The logging status of the flow­log; for example, OK, NODATA, or SKIPDATA

Updates You cannot update the configuration of an existing flow­log; you must delete it and recreate it if  you want to change any settings associated. This is another reason why it is good to explicitly  create and manage the associated log group.

Omissions Some traffic is not captured by the flow­logs service, as follows: 

Traffic to the Amazon DNS server ( x.x.x.2 in your allocated range)



Traffic for Amazon Windows license activation (obviously only applicable to Windows  instances)



Traffic to and from the instance metadata service (that is, IP address 169.254.169.254)



DHCP traffic



Traffic to the reserved VPC IP address for the default VPC router ( x.x.x.1 in your allocated  range)

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 75 *

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 76 *

AWS Compute with CLI Introduction Elastic Cloud Compute (EC2) is by far the most utilized and complex service in the AWS  catalogue. More than just virtual machines, EC2 provides a framework of sub­services to  help you secure and manage your instances elastically.

Creating a key pair A key pair is used to access your instances via SSH. This is the quickest and easiest way to  access your instances.

Getting ready To perform this, you must have your AWS CLI tool configured correctly.

How to do it... 1. Create the key pair, and save it to disk: aws ec2 create-key-pair \ --key-name MyEC2KeyPair \ --query 'KeyMaterial' \ --output text > ec2keypair.pem

2. Change the permissions on the created file:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] *

chmod 600 ec2keypair.pem

How it works... This call requests a new private key from EC2. The response is then parsed using a  JMESPath query, and the private key (in the KeyMaterial property) is saved to a new key file  with the .pem extension. Finally, we change the permissions on the key file so that it cannot be read by other users— this is required before SSH will allow you to use it.

Launching an instance There will be scenarios—usually when testing and developing your infrastructure code—when you need quick access to an instance. Creating it via the AWS CLI is the quickest and most  consistent way to create one­off instances.

Getting ready We are launching an instance of AWS Linux using an AMI ID in the us­east­1 region. f you are working in a different region, you will need to update your image­id parameter. You must have configured your AWS CLI tool with working credentials.

How to do it... Run the following AWS CLI command, using your own key­pair name:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] *

aws ec2 run-instances \ --image-id ami-9be6f38c \ --instance-type t2.micro \ --key-name

How it works... While you can create an instance via the AWS web console, it involves many distracting  options. When developing and testing, the CLI tool is the best way to provision instances. While the key­name argument is optional, you will not be able to connect to your instance  unless you have pre­configured some other way of logging in.

Note The t2.micro instance type used in this recipe is included in the AWS free tier. You can run one  micro instance per month for free during the first 12 months of your usage.  See https://aws.amazon.com/free  for more information.

As no VPC or security groups are specified, the instance will be launched in your account's  default VPC and security group. The default security group allows access from anywhere, on  all ports, and so is not suitable for long­lived instances. You can modify an instance's security  groups after it is launched, without stopping it.

There's more... If you have created your own AMI, then you can change the image­id argument to quickly  launch your specific AMI. You may also want to take note of the InstanceId value in the response from the API, as you  may need it for future commands.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] *

Attaching storage Ideally, you will have defined all your storage requirements up­front as code using a  service such as CloudFormation. However, sometimes that is not possible due to  application restrictions or changing requirements. You can easily add additional storage to your instances while they are running by  attaching a new volume.

Getting ready 

A running instance's ID. It will start with i- followed by alphanumeric characters.



The AZ the instance is running in. This looks like the region name with a letter after it; for  example, us-east-1a.

In this section, we are using an AWS Linux instance. If you are using a different  operating system, the steps to mount the volume will be different. We will be running  an instance in the AZ us-east-1a. You must have configured your AWS CLI tool with working credentials.

How to do it... 1. Create a volume:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] *

aws ec2 create-volume --availability-zone us-east-1a

Note Take note of the returned VolumeId in the response. It will start with vol- followed by alphanumeric characters. 2. Attach the volume to the instance, using the volume ID noted in the last step and the  instance ID you started with: aws ec2 attach-volume \ --volume-id \ --instance-id \ --device /dev/sdf

3. On the instance itself, mount the volume device: mount /dev/xvdf /mnt/volume

How it works...  we start by creating a volume. Volumes are created from snapshots. If you do not  specify a snapshot ID it uses a blank snapshot, and you get a blank volume. While volumes are hosted redundantly, they are only hosted in a single AZ, so must  be provisioned in the same AZ the instance is running in. The create-volume command returns a response that includes the newly created  volume's VolumeId. We then use this ID in the next step.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] *

Note It can sometimes take a few seconds for a volume to become available. If you are scripting these  commands, use the aws ec2 wait command to wait for the volume to become available.

In step 3, we attach a volume to the instance. When attaching to an instance, you must  specify the name of the device that it will be presented to the operating system as.  Unfortunately, this does not guarantee what the device will appear as. In the case of AWS  Linux, /dev/sdf becomes /dev/xvdf.

Note Device naming is kernel­specific, so if you are using something other than AWS Linux, the device  name may be different.  See http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/device_naming.html  for full details.

Securely accessing private instances Any instance or resource living in a private subnet in your VPC will be inaccessible  from the Internet. This makes good sense from a security perspective because it gives your instances a higher level of protection. Of course, if they can't be accessed from the Internet, then they're not going to be  easy to administer.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] *

One common pattern is to use a VPN server as a single, highly controlled, entry point  to your private network. This is what we're going to show you in this, as pictured in the following diagram:

Accessing private instances securely

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] *

Getting ready We're going to use OpenVPN for this example. They provide a free (for up to two users) AMI  in the AWS marketplace, which has OpenVPN already installed and configured. You'll need to accept the terms and conditions for using this AMI. You can do so by visiting the AMI's  marketplace page at https://aws.amazon.com/marketplace/pp/B00MI40CAE/  .

You need to decide on a password, which will be your temporary admin password. We'll feed  this password into a CloudFormation template and then change it after we create our stack.

Note You can use the default VPC for this example.

How to do it... 1. Create a new CloudFormation template and add the following Mappings. This is a list of all  the latest OpenVPN AMIs in each region. We're adding these to maximize region portability for our template—you can omit the regions you have no intention of using: Mappings: AWSRegion2AMI: # Latest OpenVPN AMI at time of publishing: 2.1.4 us-east-1: AMI: ami-bc3566ab us-east-2: AMI: ami-10306a75 us-west-2: AMI: ami-d3e743b3 us-west-1: AMI: ami-4a02492a eu-west-1: AMI: ami-f53d7386 eu-central-1: AMI: ami-ad1fe6c2 ap-southeast-1: AMI: ami-a859ffcb

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] *

ap-northeast-1: AMI: ami-e9da7c88 ap-southeast-2: AMI: ami-89477aea sa-east-1: AMI: ami-0c069b60

2. We now need to define some Parameters. Firstly we'll need to know which VPC and  subnet to deploy our VPN instance to. Note that you need to specify a public subnet here,  otherwise you won't be able to access your OpenVPN server: VpcId: Type: AWS::EC2::VPC::Id Description: VPC where load balancer and instance will launch SubnetId: Type: List Description: Subnet where OpenVPN server will launch (pick at least 1)

3. We also need to define InstanceType and KeyName. These are the EC2 instance class  and SSH key pair to use to launch our OpenVPN server: InstanceType: Type: String Description: OpenVPN server instance type Default: m3.medium KeyName: Type: AWS::EC2::KeyPair::KeyName Description: EC2 KeyPair for SSH access

4. We need a parameter for AdminPassword. This is the temporary password which will be  given to the openvpn user (administrator) when the server starts up: AdminPassword: Type: String Description: Password for 'openvpn' user Default: openvpn NoEcho: true

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] *

5. The last parameter is the CIDR block, which we wish to allow to connect to our VPN  server. You may wish to lock this down to the public IP range of your corporate network,  for example: AllowAccessFromCIDR: Type: String Description: IP range/address to allow VPN connections from Default: "0.0.0.0/0"

6. The first Resource we need to define is the security group our OpenVPN server will live in.  You'll also use this security group to allow access to other resources in your network. Add  it to your template as follows: VPNSecurityGroup: Type: AWS::EC2::SecurityGroup Properties: GroupDescription: Inbound access to OpenVPN server VpcId: !Ref VpcId SecurityGroupIngress: - CidrIp: !Ref AllowAccessFromCIDR FromPort: 443 IpProtocol: tcp ToPort: 443 - CidrIp: !Ref AllowAccessFromCIDR FromPort: 22 IpProtocol: tcp ToPort: 22 - CidrIp: !Ref AllowAccessFromCIDR FromPort: 1194 IpProtocol: udp ToPort: 1194

7. We can now define the actual OpenVPN instance itself. You'll notice that we are explicitly  configuring the network interface. This is required, because we want to declare that this  instance must get a public IP address (otherwise you won't be able to access it). In  the UserData, we declare some variables that the OpenVPN software will pick up when it  starts so that it can configure itself: OpenVPNInstance: Type: AWS::EC2::Instance Properties: ImageId: !FindInMap [ AWSRegion2AMI, !Ref "AWS::Region", AMI ] InstanceType: !Ref InstanceType

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] *

KeyName: !Ref KeyName NetworkInterfaces: - AssociatePublicIpAddress: true DeviceIndex: "0" GroupSet: - !Ref VPNSecurityGroup SubnetId: !Select [ 0, Ref: SubnetId ] Tags: - Key: Name Value: example-openvpn-server UserData: Fn::Base64: !Sub -| public_hostname=openvpn admin_user=openvpn admin_pw=${admin_pw} reroute_gw=1 reroute_dns=1 - admin_pw: !Ref AdminPassword

8. Finally, we add some helpful Outputs: Outputs: OpenVPNAdministration: Value: Fn::Join: - "" - - https:// - !GetAtt OpenVPNInstance.PublicIp - /admin/ Description: Admin URL for OpenVPN server OpenVPNClientLogin: Value: Fn::Join: - "" - - https:// - !GetAtt OpenVPNInstance.PublicIp -/ Description: Client login URL for OpenVPN server OpenVPNServerIPAddress: Value: !GetAtt OpenVPNInstance.PublicIp Description: IP address for OpenVPN server

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] *

9. Go ahead and launch this stack in the CloudFormation web console, or via the CLI, with  the following command: aws cloudformation create-stack \ --template-body file://04-securely-access-private-instances.yaml \ --stack-name example-vpn \ --parameters \ ParameterKey=KeyName,ParameterValue= \ ParameterKey=VpcId,ParameterValue= \ ParameterKey=SubnetId,ParameterValue=

Configuration 1. Once your stack is created, you'll want to change the password for the openvpn user  (administrator). Go to the admin control panel and do this now: https:///admin. If your VPN server is operating as expected you'll be greeted with a status page after logging in, as pictured in the following screenshot:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] *

While you're there, you should create a non­administrator user account. This will be the  account you'll use to connect to the VPN. Add this account on the User Permissions page  as pictured in the following screenshot:

2. Under Server Network Settings , in the Hostname or IP address  field, enter the  hostname or IP address of the server. This step is important, and when you download your OpenVPN config file from the server (next step), it will make your life much easier if it has  the correct hostname or IP address in it. The next screenshot shows what you can expect  to see on the Server Network Settings  page:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] *

How it works... You should now be able to connect to your VPN server. Go to the user login page and  log in with the credentials you gave to the previously mentioned non­administrator  user: https:///

After logging in, you will have the option to download the OpenVPN client with  configuration which is specific to your account. Alternatively, if you already have a  VPN client installed, you can just download the configuration on its own.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] *

There's more... There are a couple of important points you'll need to keep in mind now that you are up and running with an OpenVPN server: 

If you need to SSH to the instance, you must connect with the username openvpnas



To access your other instances, you'll need to allow connections from the VPN security  group 

Auto scaling an application server Auto scaling is a fundamental component of compute in the cloud. It provides not  only the ability to scale up and down in response to application load, but also  redundancy, by ensuring that capacity is always available. Even in the unlikely event  of an AZ outage, the auto scaling group will ensure that instances are available to run  your application. Auto scaling also allows you to pay for only the EC2 capacity you need, because  underutilized servers can be automatically de­provisioned.

Getting ready You must supply two or more subnet IDs for this recipe to work. The following example uses an AWS Linux AMI in the us-east-1 region. Update the  parameters as required if you are working in a different region.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] *

How to do it... 1. Start by defining the template version and description: AWSTemplateFormatVersion: "2010-09-09" Description: Create an Auto Scaling Group

2. Add a Parameters section with the required parameters that will be used later in the  template: Parameters: SubnetIds: Description: Subnet IDs where instances can be launched Type: List

3. Still under the Parameters section, add the optional instance configuration parameters: AmiId: Description: The application server's AMI ID Type: AWS::EC2::Image::Id Default: ami-9be6f38c # AWS Linux in us-east-1 InstanceType: Description: The type of instance to launch Type: String Default: t2.micro

4. Still under the Parameters section, add the optional auto scaling group­configuration  parameters: MinSize: Description: Minimum number of instances in the group Type: Number Default: 1 MaxSize: Description: Maximum number of instances in the group Type: Number Default: 4 ThresholdCPUHigh: Description: Launch new instances when CPU utilization is over this threshold Type: Number

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] *

Default: 60 ThresholdCPULow: Description: Remove instances when CPU utilization is under this threshold Type: Number Default: 40 ThresholdMinutes: Description: Launch new instances when over the CPU threshold for this many minutes Type: Number Default: 5

5. Add a Resources section, and define the auto scaling group resource: Resources: AutoScalingGroup: Type: AWS::AutoScaling::AutoScalingGroup Properties: MinSize: !Ref MinSize MaxSize: !Ref MaxSize LaunchConfigurationName: !Ref LaunchConfiguration Tags: - Key: Name Value: !Sub "${AWS::StackName} server" PropagateAtLaunch: true VPCZoneIdentifier: !Ref SubnetIds

6. Still under the Resources section, define the launch configuration used by the auto scaling  group: LaunchConfiguration: Type: AWS::AutoScaling::LaunchConfiguration Properties: ImageId: !Ref AmiId InstanceType: !Ref InstanceType UserData: Fn::Base64: !Sub | #!/bin/bash -xe # This will be run on startup, launch your application here

7. Next, define two scaling policy resources—one to scale up and the other to scale down: ScaleUpPolicy:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] *

Type: AWS::AutoScaling::ScalingPolicy Properties: AdjustmentType: ChangeInCapacity AutoScalingGroupName: !Ref AutoScalingGroup Cooldown: 60 ScalingAdjustment: 1 ScaleDownPolicy: Type: AWS::AutoScaling::ScalingPolicy Properties: AdjustmentType: ChangeInCapacity AutoScalingGroupName: !Ref AutoScalingGroup Cooldown: 60 ScalingAdjustment: -1

8. Define an alarm that will alert when the CPU goes over the ThresholdCPUHigh parameter:

CPULowAlarm: Type: AWS::CloudWatch::Alarm Properties: ActionsEnabled: true AlarmActions: - !Ref ScaleDownPolicy AlarmDescription: Scale down on CPU load ComparisonOperator: LessThanThreshold Dimensions: - Name: AutoScalingGroupName Value: !Ref AutoScalingGroup EvaluationPeriods: !Ref ThresholdMinutes Namespace: AWS/EC2 Period: 60 Statistic: Average Threshold: !Ref ThresholdCPULow MetricName: CPUUtilization

10. Save the template with the filename 04-auto-scaling-an-application-server.yaml . 11. Launch the template with the following AWS CLI command, supplying your subnet IDs: aws cloudformation create-stack \ --stack-name asg \ --template-body file://04-auto-scaling-an-application-server.yaml \ --parameters \

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] *

ParameterKey=SubnetIds,ParameterValue='<subnet-id-1>\, \ <subnet-id-2>'

How it works... This example defines an auto scaling group and the dependent resources. These include the  following: 

A launch configuration to use when launching new instances



Two scaling policies, one to scale the number of instances up, and an inverse policy to  scale back down



An alarm to alert when the CPU crosses a certain threshold, for a certain number of  minutes

The auto scaling group and launch­configuration resource objects in this  example use mostly default values. You will need to specify your  own SecurityGroups and a KeyName parameter in  the LaunchConfiguration resource configuration if you want to be able to  connect to the instances (for example, via SSH). AWS will automatically take care of spreading your instances evenly over the  subnets you have configured, so make sure they are in different AZs! When  scaling down, the oldest instances will be removed before the newer ones.

Scaling policies The scaling policies detail how many instances to create or delete when they are triggered. It  also defines a Cooldownvalue, which helps prevent flapping servers—when servers are  created and deleted before they have finished starting and are useful.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] *

Note While the scaling policies in this example use equal values, you might want to change that so your application can scale up quickly, and scale down slowly for the best user experience

The minimum charge for an instance is one hour, so creating and destroying them multiple  times in one hour may result in higher than expected charges.

Creating machine images Creating or baking your own Amazon Machine Images (AMIs) is a key part of  systems administration in AWS. Having a pre­baked image helps you provision your  servers faster, easier, and more consistently than configuring it by hand. Packer is the de facto standard tool that helps you make your own AMIs. By  automating the launch, configuration, and clean­up of your instances, it makes sure  you get a repeatable image every time. In this recipe, we will create an image with the Apache web server pre­installed and  configured. While this is a simple example, it is also a very common use­case. By baking­in your web server, you can scale up your web serving layer to dynamically  match the demands on your websites. Having the software already installed and  configured means you get the fastest and most reliable start­up possible.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] *

Getting ready For this recipe, you must have the Packer tool available on your system. Download and install  Packer from the project's website https://www.packer.io/downloads.html  .

How to do it... 1. Create a new Packer template file, and start by defining an amazon-ebs builder in  the builders section: "builders": [ { "type": "amazon-ebs", "instance_type": "t2.micro", "region": "us-east-1", "source_ami": "ami-9be6f38c", "ssh_username": "ec2-user", "ami_name": "aws-linux-apache {{timestamp}}" } ],

Note The entire template file must be a valid JSON object. Remember to enclose the sections in curly  braces: { ... }. 2. Create a provisioners section, and include the following snippet to install and activate  Apache: "provisioners": [ { "type": "shell", "inline": [ "sudo yum install -y httpd", "sudo chkconfig httpd on" ] }

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] *

]

3. Save the file with a specific name, such as 04-creating-machine-images.json . 4. Validate the configuration file you've created with the following packer validate command: packer validate 04-creating-machine-images.json

6. When valid, build the AMI with the following command: packer build 04-creating-machine-images.json   Wait until the process is complete. While it is running, you will see an output similar to the  following:

7. Take note of the AMI ID returned by Packer so that you can use it when launching  instances in the future:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] *

How it works... While this is  very simple, there is a lot going on behind the scenes. This is why we  recommend you use Packer to create your machine images.

Template In the builders section of the template, we define our build details. We are using the most common type of AMI builder: amazon-ebs. There are other  types of AWS builders, for instance, storage­backed instance types. Next, we define the type of instance to use when baking.

Note Make sure that you can often decrease the time it takes to bake your instance by using a larger  instance size. Remember that the minimum price paid for an instance is one hour of billable time.

The source_ami property in this recipe is an AWS Linux AMI ID in the region we have  specified. The ssh_username allows you to set the username used to connect and  run provisioners on the instance. This will be determined by your operating system,  which in our case is ec2-user. Finally, the ami_name field includes the built­in Packer variable {{timestamp}}. This  ensures the AMI you create will always have a unique name.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] *

Validate the template The packer validate command is a quick way to ensure your template is free of syntax errors before you launch any instances.

Build the AMI Once you have created and validated your template, the packer build command does  the following for you: 

Creates a one­time key pair for SSH access to the instance



Creates a dedicated security group to control access to the instance



Launches an instance



Waits until SSH is ready to receive connections



Runs the provisioner steps on the instance



Stops the instance



Generates an AMI from the stopped instance



Terminates the instance

There's more... While Packer makes the administration of images much easier on AWS, there are still  a few things to watch out for.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] *

Debugging Obviously, with so many steps being automated for you, there are many things that can potentially  go wrong. Packer gives you a few different ways to debug issues with your builds. One of the most useful arguments to use with Packer is the -debug flag. This will force you to  manually confirm each step before it takes place. Doing this makes it easy to work out exactly  which step in the command is failing, which in turn usually makes it obvious what needs to be  changed. Another useful thing to do is to raise the level of logging output during a Packer command. You  can do this by setting the PACKER_LOG variable to true. The easiest way to do this is  with PACKER_LOG=1 at the beginning of your Packer command line. This will mean you get a lot more information printed to the console (for example, SSH logs, AWS API calls, and so on) during  the command. You may even want to run with this level of logging normally in your builds, for  auditing purposes.

Orphaned resources Packer does a great job of managing and cleaning up the resource it uses, but it can only do that  while it is running. If your Packer job aborts for any reason (most likely network issues) then there may be some  resources left orphaned, or unmanaged. It is good practice to check for any Packer instances  (they will have Packer in their name), and stop them if there are no active Packer jobs running. You may also need to clean up any leftover key pairs and security groups, but this is less of an  issue as there is no cost associated with them (unlike instances).

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] *

Deregistering AMIs As it becomes easier to create AMIs, you may find you end up with more than you need! AMIs are made up of EC2 snapshots, which are stored in S3. There is a cost associated with  storing snapshots, so you will want to clean them up periodically. Given the size of most AMIs  (usually a few GBs), it is unlikely to be one of your major costs. An even greater cost is the administrative overhead of managing too many AMIs. As your images  improve and fixes are applied (especially security fixes), you may want to prevent people from  using them. To remove an AMI, you must first deregister it, and then remove the underlying snapshots.

Note Make sure you do not deregister AMIs that are currently in use. For example, an auto scaling  group that references a deregistered AMI will fail to launch new instances!

You can easily deregister snapshots through the web console or using the AWS CLI  tool. Once an AMI is no longer registered, you can remove the associated snapshots.  Packer automatically adds the AMI ID to the snapshots' description. By searching your snapshots for the deregistered AMI ID, you can find which ones need to be deleted. You will not be able to delete snapshots if the AMI has not been deregistered, or if the  deregistration is still taking place (it can take a few minutes).

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] *

Creating security groups AWS describes security groups as virtual firewalls. While this analogy helps  newcomers to the EC2 platform understand their purpose and function, it's probably  more accurate to describe them as a firewall­like method of authorizing traffic. They  don't offer all the functionality you'd find in a traditional firewall, but this simplification  also makes them extremely powerful, particularly when combined with Infrastructure  as Code and modern SDLC practices. We're going to go through a basic scenario involving a web server and load balancer.  We want the load balancer to respond to HTTP requests from everywhere, and we  want to isolate the web server from everything except the load balancer.

Getting ready Before we get started there's a small list of things you'll need to have ready: 

AmiId This is the ID of an AMI in your region. For this recipe, we'd recommend using an  AWS Linux AMI because our instance will attempt to run some yum commands on startup.



VPCID: This is the ID of the VPC you wish to launch the EC2 server into.



SubnetIDs: These are the subnets which our EC2 instance can launch in.

How to do it... 1. Open up your text editor and create a new CloudFormation template. We're going to start  by adding a few Parameters as follows:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] *

AWSTemplateFormatVersion: '2010-09-09' Parameters: AmiId: Type: AWS::EC2::AMI::Id Description: AMI ID to launch instances from VPCID: Type: AWS::EC2::VPC::Id Description: VPC where load balancer and instance will launch SubnetIDs: Type: List Description: Subnets where load balancer and instance will launch (pick at least 2)

2. Let's take a look at a security group we'll apply to a public load balancer: ExampleELBSecurityGroup: Type: AWS::EC2::SecurityGroup Properties: GroupDescription: Security Group for example ELB SecurityGroupIngress: - IpProtocol: tcp CidrIp: 0.0.0.0/0 FromPort: 80 ToPort: 80

Anything which resides in this security group will allow inbound TCP connections on  port 80 from anywhere (0.0.0.0/0). Note that a security group can contain more than  one rule; we'd almost certainly want to also allow HTTPS ( 443), but we've left it out to  simplify this recipe. 3. Now let's look at a security group for a web server sitting behind our load balancer: ExampleEC2InstanceSecurityGroup: Type: AWS::EC2::SecurityGroup Properties: GroupDescription: Security Group for example Instance SecurityGroupIngress: - IpProtocol: tcp SourceSecurityGroupName: Ref: ExampleELBSecurityGroup FromPort: 80 ToPort: 80

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] *

Here you can see we are not specifying a source IP range. Instead, we're specifying a source  security group, which we will accept connections from. In this case, we're saying that we want  to allow anything from our ELB security group to connect to anything in our EC2  instance security group on port 80. Since this is the only rule we're specifying, our web server  will not accept connections from anywhere except our load balancer, to port 80 or otherwise.  Our web server isn't wide open to the Internet, and it is even isolated from other instances in  our VPC

Note Remember that multiple instances can reside in a security group. In a scenario where you have  multiple web servers attached to this load balancer it would be unnecessary, inefficient, and  somewhat of an anti­pattern to create a new security group for each web server. Given that all web servers attached to this load balancer would be serving the same role or function, it makes sense  to apply the same security group to them.

This is where the power of security groups really comes in. If an EC2 instance is  serving multiple roles—let's say you have an outbound HTTP proxy server in your VPC which  you also want to act as an SMTP relay—then you can simply apply multiple security groups to it. 4. Next, we need to add our load balancer. This is probably the most basic load balancer  configuration you'll come across. The following code will give you a load balancer, a  listener and a target group containing our EC2 instance. ExampleLoadBalancer: Type: AWS::ElasticLoadBalancingV2::LoadBalancer Properties: Subnets: - Fn::Select: [ 0, Ref: SubnetIDs ] - Fn::Select: [ 1, Ref: SubnetIDs ] SecurityGroups: - Fn::GetAtt: ExampleELBSecurityGroup.GroupId ExampleListener: Type: AWS::ElasticLoadBalancingV2::Listener Properties: LoadBalancerArn: Ref: ExampleLoadBalancer DefaultActions:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] *

- Type: forward TargetGroupArn: Ref: ExampleTargetGroup Port: 80 Protocol: HTTP ExampleTargetGroup: Type: AWS::ElasticLoadBalancingV2::TargetGroup Properties: Port: 80 Protocol: HTTP VpcId: Ref: VPCID Targets: - Id: Ref: ExampleEC2Instance

5. The last resource we'll add to our template is an EC2 server. This server will install and  start nginx when it boots. ExampleEC2Instance: Type: AWS::EC2::Instance Properties: InstanceType: t2.nano UserData: Fn::Base64: Fn::Sub: | #!/bin/bash -ex yum install -y nginx service nginx start exit 0 ImageId: Ref: AmiId SecurityGroupIds: - Fn::GetAtt: ExampleEC2InstanceSecurityGroup.GroupId SubnetId: Fn::Select: [ 0, Ref: SubnetIDs ]

6. Lastly, we're going to add some Outputs to the template to make it a little more convenient  to use our ELB and EC2 instance after the stack is created. Outputs: ExampleEC2InstanceHostname: Value: Fn::GetAtt: [ ExampleEC2Instance, PublicDnsName ] ExampleELBURL:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] *

Value: Fn::Join: - '' - [ 'http://', { 'Fn::GetAtt': [ ExampleLoadBalancer, DNSName ] }, '/' ]

7. Go ahead and launch this template using the CloudFormation web console or the AWS  CLI.

There's more... You'll eventually run into circular dependency issues when configuring security groups using  CloudFormation. Let's say you want all servers in  our ExampleEC2InstanceSecurityGroup to be able to access each other on  port 22 (SSH). In order to achieve this, you would need to add this rule as the separate  resource type AWS::EC2::SecurityGroupIngress. This is because a security group can't  refer to itself in CloudFormation when it is yet to be created. This is what the extra resource  type looks like: ExampleEC2InstanceIngress: Type: AWS::EC2::SecurityGroupIngress Properties: IpProtocol: tcp SourceSecurityGroupName: Ref: ExampleEC2InstanceSecurityGroup GroupName: Ref: ExampleEC2InstanceSecurityGroup FromPort: 22 ToPort: 22

Differences from traditional firewalls 

Security groups can't be used to explicitly block traffic. Only rules of a permissive kind can  be added; deny style rules are not supported. Essentially, all inbound traffic is denied  unless you explicitly allow it.



Your rules also may not refer to source ports; only destination ports are supported.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] *



When security groups are created, they will contain a rule which allows all outbound  connections. If you remove this rule, new outbound connections will be dropped. It's a  common pattern to leave this rule in place and filter all your traffic using inbound rules only.



If you do replace the default outbound rule, it's important to note that only new outbound  connections will be filtered. Any outbound traffic being sent in response to an inbound  connection will still be allowed. This is because security groups are stateful.



Unlike security groups, network ACLs are not stateful and do support DENY rules. You can  use them as a complementary layer of security inside your VPC, especially if you need to  control traffic flow between subnets.

Creating a load balancer AWS offers two kinds of load balancers: 

Classic load balancer



Application load balancer

We're going to focus on the application load balancer. It's effectively an upgraded, second  generation of the ELB service, and it offers a lot more functionality than the classic load  balancer. HTTP/2 and WebSockets are supported natively, for example. The hourly rate also  happens to be cheaper.

Note Application load balancers do not support layer­4 load balancing. For this kind of functionality,  you'll need to use a classic load balancer.

How to do it... 1. Open up your text editor and create a new CloudFormation template. We're going to  require a VPC ID and some subnet IDs as Parameters. Add them to your template like this:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] *

AWSTemplateFormatVersion: '2010-09-09' Parameters: VPCID: Type: AWS::EC2::VPC::Id Description: VPC where load balancer and instance will launch SubnetIDs: Type: List Description: Subnets where load balancer and instance will launch (pick at least 2

2. Next we need to add some Mappings of ELB account IDs. These will make it easier for us  to give the load balancer permission to write logs to an S3 bucket. Your mappings should  look like this:

Note You can find the complete list of ELB account IDs  here http://docs.aws.amazon.com/elasticloadbalancing/latest/classic/enable­access­ logs.html#attach­bucket­policy  . Mappings: ELBAccountMap: us-east-1: ELBAccountID: 127311923021 ap-southeast-2: ELBAccountID: 783225319266

3. We can now start adding Resources to our template. First we're going to create an S3  bucket and bucket policy for storing our load balancer logs. In order to make this template  portable, we'll omit a bucket name, but for convenience we'll include the bucket name in  our outputs so that CloudFormation will echo the name back to us. Resources: ExampleLogBucket: Type: AWS::S3::Bucket ExampleBucketPolicy: Type: AWS::S3::BucketPolicy Properties: Bucket:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] *

Ref: ExampleLogBucket PolicyDocument: Statement: Action: - "s3:PutObject" Effect: "Allow" Resource: Fn::Join: - "" - "arn:aws:s3:::" - Ref: ExampleLogBucket - "/*" Principal: AWS: Fn::FindInMap: [ ELBAccountMap, Ref: "AWS::Region", ELBAccountID ]

4. Next, we need to create a security group for our load balancer to reside in. This security  group will allow inbound connections to port 80 (HTTP). To simplify this recipe, we'll leave  out port 443 (HTTPS), but we'll briefly cover how to add this functionality later in this  section. Since we're adding a public load balancer, we want to allow connections to it from  everywhere (0.0.0.0/0). This is what our security group looks like: ExampleELBSecurityGroup: Type: AWS::EC2::SecurityGroup Properties: GroupDescription: Security Group for example ELB SecurityGroupIngress: IpProtocol: tcp CidrIp: 0.0.0.0/0 FromPort: 80 ToPort: 80

5. We now need to define a target group. Upon completion of this recipe, you can go ahead  and register your instances in this group so that HTTP requests will be forwarded to it.  Alternatively, you can attach the target group to an auto scaling group and AWS will take  care of the instance registration and de­registration for you.

6. The target group is where we specify the health checks our load balancer should perform  against the target instances. This health check is necessary to determine if a registered 

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] *

instance should receive traffic. The example provided with this recipe includes these  health­check parameters with the values all set to their defaults. Go ahead and tweak  these to suit your needs, or, optionally, remove them if the defaults work for you.

ExampleTargetGroup: Type: AWS::ElasticLoadBalancingV2::TargetGroup Properties: Port: 80 Protocol: HTTP HealthCheckIntervalSeconds: 30 HealthCheckProtocol: HTTP HealthCheckPort: 80 HealthCheckPath: / HealthCheckTimeoutSeconds: 5 HealthyThresholdCount: 5 UnhealthyThresholdCount: 2 Matcher: HttpCode: '200' VpcId: Ref: VPCID

7. We need to define at least one listener to be added to our load balancer. A listener  will listen for incoming requests to the load balancer on the port and protocol we configure  for it. Requests matching the port and protocol will be forwarded through to our target  group.

The configuration of our listener is going to be reasonably simple. We're listening for  HTTP requests on port 80. We're also setting up a default action for this listener,  which will forward our requests to the target group we've defined before. There is a  limit of 10 listeners per load balancer.

Note Currently, AWS only supports one action: forward.

ExampleListener: Type: AWS::ElasticLoadBalancingV2::Listener Properties:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] *

LoadBalancerArn: Ref: ExampleLoadBalancer DefaultActions: - Type: forward TargetGroupArn: Ref: ExampleTargetGroup Port: 80 Protocol: HTTP

8. Finally, now that we have all Resources we need, we can go ahead and set up our load  balancer. We'll need to define at least two subnets for it to live in—these are included  as Parameters in our example template: ExampleLoadBalancer: Type: AWS::ElasticLoadBalancingV2::LoadBalancer Properties: LoadBalancerAttributes: - Key: access_logs.s3.enabled Value: true - Key: access_logs.s3.bucket Value: Ref: ExampleLogBucket - Key: idle_timeout.timeout_seconds Value: 60 Scheme: internet-facing Subnets: - Fn::Select: [ 0, Ref: SubnetIDs ] - Fn::Select: [ 1, Ref: SubnetIDs ] SecurityGroups: - Fn::GetAtt: ExampleELBSecurityGroup.GroupId

9. Lastly, we're going to add some Outputs to our template for convenience. We're  particularly interested in the name of the S3 bucket we created and the URL of the load  balancer. Outputs: ExampleELBURL: Value: Fn::Join: - '' - [ 'http://', { 'Fn::GetAtt': [ ExampleLoadBalancer, DNSName ] }, '/' ] ExampleLogBucket: Value: Ref: ExampleLogBucket

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] *

How it works... As you can see, we're applying a logging configuration which points to the S3 bucket  we've created. We're configuring this load balancer to be Internet­facing, with an idle  timeout of 60 seconds (the default). All load balancers are Internet­facing by default, so it's not strictly necessary to define  a Scheme in our example; however, it can be handy to include this anyway. This is  especially the case if your CloudFormation template contains a mix of public and  private load balancers.

Note If you specify a logging configuration but the load balancer can't access the S3 bucket, your  CloudFormation stack will fail to complete.

Private ELBs are not Internet­facing and are available only to resources which live  inside your VPC. That's it! You now have a working application load balancer configured to ship logs to  an S3 bucket.

There's more... Load balancers on AWS are highly configurable and there are many options available  to you. Here are some of the more frequent ELB options you'll encounter:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] *

HTTPS/SSL If you wish to accept HTTPS requests, you'll need to configure an additional listener. It will look  something like the following: ExampleHTTPSListener: Type: AWS::ElasticLoadBalancingV2::Listener Properties: Certificates: - CertificateArn: arn:aws:acm:ap-southeast-2:123456789012: certificate/12345678-1234-1234-1234-123456789012 LoadBalancerArn: Ref: ExampleLoadBalancer DefaultActions: - Type: forward TargetGroupArn: Ref: ExampleTargetGroup Port: 443 Protocol: HTTPS

The listener will need to reference a valid Amazon Resource Name (ARN) for the  certificate you wish to use. It's really easy to have AWS Certificate Manager create a  certificate for you, but it does require validation of the domain name you're generating  the certificate for. You can, of course, bring your own certificate if you wish. You'll  need to import it in to AWS Certificate Manager before you can use it with your ELB  (or CloudFront distribution). Unless you have specific requirements around ciphers, a good starting approach is to  not define an SSL Policy and let AWS choose what is currently best of breed.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] *

Path­based routing Once you are comfortable with ELB configuration, you can start to experiment with  path­based routing. In a nutshell, it provides a way to inspect a request and proxy it to  different targets based on the path requested. One common scenario you might encounter is needing to route requests for /blog to a  different set of servers running WordPress, instead of to your main server pool, which  is running your Ruby on Rails application.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] *

AZURE STORAGE AND DATABASES Contents An introduction to Microsoft Azure Storage..........................................................................................5 Why Azure Storage?...............................................................................................................................6 Terminologies.........................................................................................................................................6 Azure Storage types................................................................................................................................8 Durability...........................................................................................................................................9 Performance....................................................................................................................................12 Persistency.......................................................................................................................................12 Azure Storage accounts.........................................................................................................................13 General-purpose storage accounts................................................................................................13 Azure Storage Account tips............................................................................................................14 Creating an Azure Storage account...............................................................................................14 Automating your tasks..........................................................................................................................15 Azure PowerShell............................................................................................................................15 Azure command-line interface.......................................................................................................17 Delving into Azure Storage...................................................................................................................20 Azure Storage services..........................................................................................................................20 Blob storage.....................................................................................................................................20 Table storage....................................................................................................................................22 Queue storage..................................................................................................................................23 File storage.......................................................................................................................................24 Mount the Azure File share with File Explorer......................................................................................25 Mount the Azure File share with PowerShell.........................................................................................27 Use Azure Files with Linux.....................................................................................................................27 Prerequisites for mounting an Azure file share with Linux and the cifs-utils package...........................27 Mount the Azure file share on-demand with..........................................................................................28 Create a persistent mount point for the Azure file share with /etc/fstab ...............................................29

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 1 *

Prerequisites for mounting an Azure File share on macOS....................................................................29 Mount an Azure File share via Finder....................................................................................................30 Mount an Azure File share via Terminal................................................................................................31 Storage design for highly available applications..................................................................................32 RA-GRS...........................................................................................................................................33 Azure Backup..................................................................................................................................33 Azure Site Recovery........................................................................................................................34 Premium Storage.............................................................................................................................34 Automating tasks..................................................................................................................................34 Creating Blob storage using PowerShell.......................................................................................34 Creating Blob storage using the Azure CLI 2.0...........................................................................35 Creating Table storage using PowerShell.....................................................................................36 Creating Table storage using the Azure CLI 2.0..........................................................................36 Creating Queue storage using PowerShell....................................................................................37 Creating Queue storage using the Azure CLI 2.0........................................................................37 Azure Storage for VMs.........................................................................................................................37 An introduction to Azure VMs.............................................................................................................38 Azure VMs series.............................................................................................................................39 Storage considerations for Azure VMs.................................................................................................40 Managed versus unmanaged disks................................................................................................40 VM disks..........................................................................................................................................41 Capturing VMs......................................................................................................................................43 Sysprepping the VM.......................................................................................................................43 Capturing the VM with managed storage....................................................................................44 Capturing the VM with unmanaged storage................................................................................47 Automating the tasks.............................................................................................................................49 Creating an Azure VM using the Azure CLI 2.0..........................................................................50 Adding data disks to an Azure VM using the Azure CLI 2.0......................................................50 Resizing Azure VM disks using the Azure CLI 2.0......................................................................50 Capturing the VM using the Azure CLI 2.0.................................................................................51 Further information...............................................................................................................................51

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 2 *

Implementing Azure SQL Databases....................................................................................................51 An introduction to Azure SQL Database..............................................................................................52 Why Azure SQL Database?..................................................................................................................53 Service tiers...........................................................................................................................................54 Elastic database pools.....................................................................................................................54 Single databases..............................................................................................................................54 Service tier types.............................................................................................................................54 Azure SQL Database business continuity.............................................................................................55 How business continuity works for Azure SQL Database...........................................................58 Automating the tasks.............................................................................................................................63 Creating an Azure SQL Database using the Azure CLI 2.0........................................................64 Creating an SQL Server-level firewall rule using Azure CLI 2.0...............................................64 SQL Database (IaaS/PaaS)....................................................................................................................64 Azure SQL Database (PaaS)..........................................................................................................65 SQL on Azure VMs (IaaS)..............................................................................................................66 Azure SQL elastic database pools.........................................................................................................66 Creating an elastic database pool..................................................................................................66 Adding a database to the elastic database pool............................................................................73 Active geo-replication...........................................................................................................................74 Implementing active geo-replication.............................................................................................75 Automating the tasks.............................................................................................................................75 Creating an elastic database pool using Azure CLI 2.0...............................................................75 Adding an additional database to the elastic database pool using Azure CLI 2.0....................76

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 3 *

An introduction to Microsoft Azure Storage Storage has always been one of the most important cornerstones of every system. You cannot imagine a virtual machine (VM), web application, or mobile application running without any sort of dependency on storage, and that is what we will cover throughout this book, but from the perspective of the cloud generally, and Azure specifically. Microsoft Azure Storage is the bedrock of Microsoft's core storage solution offering in Azure. No matter what solution you are building for the cloud, you'll find a compelling use for Azure Storage. Microsoft Azure Storage is not just a traditional storage system; it's scalable and can store up to hundreds of terabytes of data, meaning that it fits almost every scenario you can ever imagine in many fields, such as IT, science, medical fields, and so on. At the time of writing, Microsoft Azure is generally available in 36 regions, with plans announced for six additional regions, as shown in the following figure: Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 4 *

This global presence means you can host your storage in the nearest region and access it from anywhere in the world. Considering that Microsoft continues to build new data centers in new regions, the latency between you and your services in Azure will decrease.

Why Azure Storage? There are many reasons for using Azure Storage which will be covered throughout this book. Below is a sneak peak of a couple of them: 

Global presence: You can host your storage wherever you want in the available Azure regions, allowing you to provide applications close to your user base.



Redundancy and recovery: As mentioned earlier in this chapter, Azure has a global presence which can be leveraged to maintain storage availability using data replication even if a disaster occurs in a region, which will be covered later in this chapter.



Many flavors: Azure Storage has many flavors, based on resiliency, durability, connectivity, performance, and so on, which can be used according to your needs in different scenarios. This will be covered later in this chapter.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 5 *



Pay as you go: Pay as you go has always been one of the distinguished reasons for using the cloud generally. It is no surprise that Azure Storage supports this model as well.

Terminologies Due to an overlap of terms and some misperceptions about the ways that Azure Services are delivered, terminology is a sticking point even for people who have been working with the technology for some time. The following table provides accurate but short definitions for terms related to Azure services. These definitions will be expanded upon in detail throughout the book, so don't worry if they are confused at first:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 6 *

Term

Definition

On-premises

Means that your data center is hosted and managed from within your company.

Off-premises

Azure VM

Blade

Means that your data center is hosted and managed in a remote place (for example, hosted and managed outside your company). The feature of providing VMs to Azure subscribers. The window that pops up when you click on one of the Azure services in the Azure portal, such as Virtual machines. A set of blades or chain of selections. For instance, when you select Virtual

Journey

Machines inside the Azure Portal, click on an existing virtual machine and then select its settings.

Resource group

Images

Disks

Provides a logical container for Azure resources (to help manage resources that are often used together). The VMs you've created in Azure and then captured to be used as templates for later use, or the VMs you've imported to Azure. Virtual Hard Disks (VHDs) that you attach to the VMs you create in Azure. Allows VMs and services that are part of the same virtual network to access

Virtual network

each other. However, services outside the virtual network have no way of connecting to services hosted within virtual networks unless you decide to do so.

Fault domain

A group of resources that could fail at the same time. For example, they are in the same rack.

Upgrade/update

A group of resources that can be updated simultaneously during system

domain

upgrades.

Storage container

The place where storage Blobs are stored, it is also used to assign security policies to the Blobs stored inside it.

Network Thought Security Quality www.facebook.com/qthought Determines the protocols, ports, who and what can access Azure VMs remotely. www.qualitythought.in Group(NSG) PH NO: 9963799240, 040-48526948 Email Id: [email protected] VM agent /extensions

Scale set

Software components that extend the VM functionality and simplify various management operations.

7 *

A set of identical VMs, that auto scale without pre-provisioning the VMs based

Azure Storage types Azure Storage has many types and even subtypes of those types in order to satisfy Azure services' consumer needs and to fit most scenarios. The most common types can be classified based on the following factors: 

Durability (replication)



Performance (Standard versus Premium)



Persistency (persistent versus non-persistent)

Durability One of the most buzzing questions about the cloud generally is: What if, for some reason, the SAN/servers that store my data are completely damaged? How can I restore my data? The answer is very simple because Microsoft Azure Storage is durable and supports data replication, therefore you can make sure your storage is highly available. Replication ensures that your data is copied somewhere else, whether in the same data center, another data center, or even another region. For more info about the SLA of Azure Storage, you can access it via the following link: https://azure.microsoft.com/en-us/support/legal/sla/storage/v1_2/

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 8 *

Replication types Microsoft Azure supports multiple options for data replication. You can use whatever you feel suits your business, especially as every type has its own price. In order to calculate your solution's cost, you can use the Azure Pricing Calculator, which can be reached via the following URL: https://azure.microsoft.com/en-us/pricing/calculator/ .

Locally redundant storage Locally redundant storage (LRS) replicates three copies of your data within the same data center you have your data in. The write requests you do with your storage are not committed until they are replicated to all three copies, which means it replicates synchronously. Not only this, it also makes sure that these three copies exist in different update domains and fault domains. You can revise the terms guide at the beginning of the chapter to understand what the update domain and the fault domain are. Drawbacks: 

The least durable option, as it replicates only within the same data center



Your data will be lost if a catastrophic event, such as a volcanic eruption or flood, affects the data center

Advantages: 

It is the cheapest type compared to the other types



It is the fastest type of data replication, offering the highest throughput since it replicates within the same data center, mitigating the risk of data loss that would occur during data replication caused by a failure having occurred on the original data host



It is the only available replication type that can be used with Premium Storage at the time of writing

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 9 *

Zone Redundant Storage Zone Redundant Storage (ZRS) replicates three copies of data across two or three data centers within one of two regions asynchronously, plus the three copies of data stored within the same data center of the original source of the data. Drawbacks: 

This type can only be used for Block Blobs (one of the Azure services covered in the next chapter), and a Standard Storage account (general purpose Standard Storage accounts will be covered later in this chapter)



Does not support metrics or logging



Does not support conversion for other replication types, such as LRS, GRS, and vice versa



If a disaster occurs, some data might be lost, because the data replicates to the other data center asynchronously



If a disaster occurs, there will be some delay in accessing your data until Microsoft failover to the secondary zone

Advantage: It provides higher durability and availability for data than LRS, as it not only replicates in the same data center but also in other data centers.

Geo-redundant storage Geo-redundant storage (GRS) replicates data not only within the same region but also to another region. Firstly, it replicates three copies of data within the same region synchronously, then it replicates another three copies of data to other regions asynchronously. Drawbacks: 

If a disaster occurs, some data might be lost, because the data replicates to the other regions asynchronously

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 10 *



If a disaster occurs, there will be some delay in accessing your data until Microsoft initiates failover to the secondary region

Advantages: 

It provides the highest durability and availability, even if a disaster occurs in an entire region



Unlike ZRS, if the original source of data faces an outage, there will be no possibility of data loss if the other three copies that exist within the same region don't face an outage too, as it replicates synchronously within the same region.

Read-access geo-redundant storage Read-access geo-redundant storage (RA-GRS) follows the same replication mechanism of GRS, in addition, to read access on your replicated data in the other regions. Drawback: If a disaster occurs, some data might be lost, because the data replicates to the other region asynchronously. Advantages: 

It provides the highest durability and availability, even if a disaster occurs in a whole region



If a disaster occurs, you still only have read access to the storage, but no write access until Microsoft initiates failover to the secondary region



The region with the read access can be used for data retrieval by the nearest offices to it without the need to go to another region to access the data; as a result, the data latency will decrease

Performance As mentioned earlier, Azure provides services for all business types and needs. There are two types based on Azure Storage performance--Standard and Premium.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 11 *

Standard Storage Standard Storage is the most common type for all the VMs sizes available in Azure. The Standard Storage type stores its data on non-SSD disks. It is commonly used with workloads within which the latency is not critical. Plus, it is low cost and has support for all Azure Storage services (which will be covered in the next chapter). It is also available in all regions.

Premium Storage Premium Storage is designed for low latency applications, such as SQL server, which needs intensive IOPs. Premium Storage is stored on SSD disks, that is why it costs more than Standard Storage. Microsoft recommends using Premium Storage for better performance and higher availability.

Persistency Another type of Azure Storage depends on data persistence, which means whether data will be there or not after stopping and starting the VM within which your data exists.

Persistent storage Persistent storage means that the data will be there after stopping and restarting the VM within which your data exists.

Non-persistent storage Non-persistent storage means that the data will be gone after restarting the VM within which your data exists.

Azure Storage accounts Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 12 *

An Azure Storage account is a secure account that provides access to Azure Storage services (which will be covered in the next chapter), and a unique namespace for storage resources. During the creation of a new Azure Storage account, you will have the option to choose one of two kinds of storage accounts: 

General-purpose storage account



Blob storage account

General-purpose storage accounts A general-purpose storage account gives you access to all Azure Storage services, such as Blobs, Tables, Files, and Queues (these will be covered in the next chapter), and has two performance tiers: 

Standard Storage tier



Premium Storage tier

Both were covered within the Performance type topic earlier in this chapter.

Azure Storage Account tips The following tips will increase your knowledge about Azure Storage, and will definitely help you when you want to design a storage solution on Azure:



You cannot switch between an Azure general-purpose storage account and an Azure Blob storage account

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 13 *



You can switch between access tiers with a Blob storage account, but with the possibility of additional charges being incurred



A Blob storage account does not support ZRS replication type at the time of writing



Premium Storage only supports Locally Redundant Storage as a replication type at the time of writing



Premium Storage is not supported for a Blob storage account at the time of writing



Azure supports up to 200 storage accounts per subscription by default



A storage account can store data up to 500 TB



Azure Storage supports encryption for only two storage services at the time of writing (Blobs and Files), and you can enable it during the storage account creation



If you are using REST APIs to connect to Azure Storage, you can secure the transfer by enabling that option during the creation of a storage account



Only lowercase letters and numbers are supported for the name of the storage account

Creating an Azure Storage account Let's get our hands dirty with creating a storage account: https://docs.microsoft.com/en-us/azure/storage/common/storage-create-storage-account

Note Something to keep in mind:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 14 *



When using Storage service encryption (blobs and files), your data is encrypted once it is written in Azure and gets decrypted once you try to access it.



When you enable Secure transfer required, the storage account will only be accessed using HTTPS if you are using REST APIs, and since Azure file service uses Server Message Block (SMB), the connection will fail if you are using SMB 2.1 and SMB 3.0 without encryption, and the same goes for the Linux SMB client in some flavors.



When you enable Secure transfer required, you will not be able to use a custom domain, because Azure Storage does not currently support that. As a result, you can only use the default .core.windows.net domain.

Automating your tasks It is no surprise that we commonly face repetitive and time-consuming tasks. For example, you might want to create multiple storage accounts. You would have to follow the previous guide multiple times to get your job done. This is why Microsoft supports its Azure services with multiple ways of automating most of the tasks that can be implemented in Azure. Throughout this book, two of the automation methods that Azure supports will be used.

Azure PowerShell PowerShell is commonly used with most Microsoft products, and Azure is no less important than these products.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 15 *

Mainly, you can use Azure PowerShell cmdlets to manage your Azure Storage, however, you should be aware that Microsoft Azure has two types of cmdlets: one for the classic portal, and another for the portal we are using. The main difference between the cmdlets of the classic portal and the current portal is there will be an RM added to the cmdlet of the current portal. For example, if you want to create a storage account in the classic portal, you would use the following cmdlet:

New-AzureStorageAccount But for the current portal, you would use: New-AzureRMStorageAccount

By default, you can use Azure PowerShell cmdlets in Windows PowerShell; you will have to install its module first.

Installing the Azure PowerShell module There are two ways to install the Azure PowerShell module: 

Download and install the module from the following link: https://www.microsoft.com/web/downloads/platform.aspx



Install the module from the PowerShell Gallery

Installing the Azure PowerShell module from the PowerShell Gallery 1. Open PowerShell in elevated mode. 2. To install the Azure PowerShell module for the current portal, run the Install-Module AzureRM cmdlet. If your PowerShell requires the NuGet provider, you will be asked to agree to install it and you will have to agree to the installation policy modification, as the repository is not available on your environment, as shown in the following screenshot:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 16 *

3. Log in to your Azure account using the Login-AzureRmAccount cmdlet. You will be prompted to enter your account credentials:

4. Create another storage account with the same properties as we used for the portal, but with a different name:

Azure command-line interface The Azure command-line interface (CLI) is open source, cross-platform, and supports implementing all the tasks you can do in the Azure portal with commands. Azure CLI comes in two flavors: 

Azure CLI 2.0, which only supports the current Azure portal



Azure CLI 1.0, which supports both portals

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 17 *

Throughout the book, we will be using Azure CLI 2.0. So, let’s get started with its installation.

Installing the Azure CLI 2.0 To understand what the Azure CLI 2.0 is capable of, we need to install it. Let's do so by following these steps:

1. Download Azure CLI 2.0 from the following link: https://azurecliprod.blob.core.windows.net/msi/azure-cli-2.0.12.msi . 2. Once downloaded, you can start the installation by following the screenshots shown here: a.

Run the executable files as administrator, and once the wizard opens, click on Install:

3. Once done, you can open the cmd and type az to access Azure CLI commands, as shown in the following diagram:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 18 *

Creating a Storage account using the Azure CLI 2.0 Let's get our hands dirty with the Azure CLI 2.0 to create an Azure Storage account: 1. Log in to your Azure account using the az login command. You have to open the URL that pops up on the CLI and enter the code, as shown in the following screenshot:

2. Create another storage account with the same properties as we used for the portal, but with a different name, as shown in the following screenshot:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 19 *

Delving into Azure Storage This chapter covers Microsoft Azure Storage services and how to work with them. For a better understanding of what is going on behind the scenes, the Azure Storage architecture and how to secure your Azure Storage will be covered too. The best practices that need to be followed to have a highly available application are also covered 

Azure Storage services



Understanding the Azure Storage architecture



Securing Azure Storage



Storage design for highly available applications

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 20 *



Automating Tasks

Azure Storage services Azure Storage has multiple services that would fit most scenarios. At the moment, there are four types of Azure Storage services, which are as follows: 

Blob storage



Table storage



Queue storage



File storage



Each of these services can be used for different scenarios, which we will cover in detail shortly.

Blob storage 

Blob stands for binary large object. This type of service can store almost everything since it stores unstructured data, such as documents, files, images, VHDs, and so on.



Using the Azure Blob storage service makes you capable of storing everything we have just mentioned, and able to access them from anywhere using different access methods, such as URLs, REST APIs, or even one of the Azure SDK Storage Client Libraries



There are three types of Blob storage:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 21 *

o

Block blobs: They are an excellent choice to store media files, documents, backups, and so on. They are good for files that are read from A-Z (sequential reading).

o

Page blobs: They support random access for files stored on them, that is why, they are commonly used for storing VHDs of Azure Virtual Machines.

o

Append blobs: They are similar to block blobs, but are commonly used for append operations. For example, each time a new block is created, it will be added to the end of the blob. One of the most common use cases within which append blobs can be used is logging, where you have multiple threads that need to be written to the same blob. This is an excellent solution that would help you to dump the logs in a fast and safe way.

Creating blob Storage : Refer https://docs.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-portal

Blob storage key points The following tips should be considered, as they will help you when designing your storage solution using blob services: 

Blob storage supports both standards, but only page blobs support Premium Storage.



Block blobs are named as such because files larger than 64 MB are uploaded as smaller blocks, then get combined into one final blob.



You cannot change the type of blob once it has been created.



Block blobs are named as such because they provide random read/write access to 512-byte pages.



Page blobs can store up to 8 TB.



Storage containers built for Blob storage may only contain lowercase letters, hyphens, and numbers, and must begin with a letter or a number, however, the name cannot contain two consecutive hyphens. The name length can vary between 3 to 63 characters.



The maximum number of blocks in a block blob or append blob is 50,000.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 22 *



The maximum size of the block in a block blob is 100 MB. As a result, a block blob can store data of up to 4.75 TB.



The maximum size of the block in an append blob is 4 MB. As a result, an append blob can store data of up to 195 TB.

Table storage High availability and scalability are key factors when you want to work with your storage, and that is exactly what is offered by Table storage. Table storage is a Microsoft NoSQL data store that can be used for the massive amount of semi-structured, non-relational data. Data is stored in tables as a collection of entities, where entities are like rows, and each entity has a primary key and a set of properties, considering that a property is like a column. The Table storage service is schema-less, therefore multiple entities in the same table may have different properties. An entity has three properties: 

PartitionKey



RowKey



Timestamp

PartitionKey The PartitionKey is a sequential range of entities that have the same key value. The way that tables are partitioned is to support load balancing across storage nodes, where tables entities are organized by partition. It is considered the first part of an entity's primary key.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 23 *

It may be a string value with a size of up to 1 KB, and every insert, update, or delete operation must be included in the partition key property.

RowKey RowKey is the second part of the entity's primary key. It is a unique identifier for the entity, and every entity within the table is uniquely identified by the combination of PartitionKey and RowKey. Like PartitionKey, it is a string value that may be up to 1 KB, and every insert, update, or delete operation must be included in the RowKey property.

Timestamp Timestamp is a datetime value, and it is kept on the server side to record when the last modification of the entity occurred. Every time there is a modification for the entity, the Timestamp value is increased. Considering that this value should not be set on insert or update operations.

Queue storage Queue storage is a storage service that is used to provide persistent and reliable messaging for application components. Generally, it creates a list of messages that process asynchronously, following the First-In, First-Out (FIFO) model. Not only this, asynchronous tasks and building process workflows can be managed with Queue storage too.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 24 *

One of the most common scenarios is passing messages from an Azure Web Role to an Azure Worker Role. Queue storage is not the only messaging solution available at Azure; there are also Service Bus queues, which can be used for more advanced scenarios.

Queue storage key points The following tips should be considered, as they will help you when designing your storage solution using the Queues service: 

Queue messages can be up to 64 KB in size, however, a Queue can contain messages up to the limiting size of the storage account.



The maximum lifetime of a message in a queue is 7 days.



As mentioned previously, messages follow the FIFO order, however, they can be out of order if an application crash occurs, which is why it would be better to use Azure Service Bus queues for a scenario where the FIFO order is highly important.



Messages can be scheduled for delivery later.



A Queue name may only contain lowercase letters, hyphens, and numbers, and must begin with a letter or number. It cannot contain two consecutive hyphens. Name length varies from between 3 and 63 characters.

File storage The File storage service is the easiest and coolest service to work with. You can use it to create network file shares on Azure, and access them from anywhere in the world. Server Message Block (SMB) and Common Internet File System (CIFS) are the only protocols that can be used to access these file shares. Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 25 *

As a result, multiple Azure VMs and on-premises machines can access the same file share and have read and write privileges on it. Azure File shares can be mounted to different operating systems, such as, Windows, Linux, and even macOS concurrently.

File storage advantages The File storage service has some good reasons to use it, which are: 

Software as a service (SaaS) service: The Azure File storage service is considered a SaaS service because you do not have to manage the hardware, operating system, patches, and so on. It is simply fully managed.



Shareability: It can be shared across multiple machines providing read and write privileges for all of those machines.



Automation: It supports working with PowerShell and the Azure CLI, which can be used to create scripts to automate repetitive tasks with minimal administration.



Flexibility and high availability: Using the Azure File storage service eliminates concerns regarding administration and outage issues that you face with traditional file servers.

Creating File storage Let's see how we can create File storage in the storage account we created in the last chapter. Refer : https://docs.microsoft.com/en-us/azure/storage/files/storage-how-to-create-file-share

Mount the Azure File share with File Explorer 1.

Open File Explorer: This can be done by opening from the Start Menu, or by pressing Win+E shortcut.

2.

Navigate to the "This PC" item on the left-hand side of the window. This will change the menus available in the ribbon. Under the Computer menu, select "Map Network Drive".

3.

Copy the UNC path from the "Connect" pane in the Azure portal

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 26 *

4.

Select the Drive letter and enter the UNC path.

5.

Use the Storage Account Name prepended with Azure\ as the username and a Storage Account Key as the password.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 27 *

6.

Use Azure File share as desired.

7.

When you are ready to dismount (or disconnect) the Azure File share, you can do so by right clicking on the entry for the share under the "Network locations" in File Explorer and selecting "Disconnect".

Mount the Azure File share with PowerShell Use the following command to mount the Azure File share: Remember to replace <storage-accountname> , <share-name> , <storage-account-key> , <desired-drive-letter> with the proper information.

$acctKey = ConvertTo-SecureString -String "<storage-account-key>" -AsPlainText -Force $credential = New-Object System.Management.Automation.PSCredential -ArgumentList "Azure\<storageaccount-name>", $acctKey New-PSDrive -Name <desired-drive-letter> -PSProvider FileSystem -Root "\\<storage-accountname>.file.core.windows.net\<share-name>" -Credential $credential

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 28 *

Use the Azure File share as desired. When you are finished, dismount the Azure File share using the following command. Remove-PSDrive -Name <desired-drive-letter>

Use Azure Files with Linux

Prerequisites for mounting an Azure file share with Linux and the cifs-utils package 



Pick a Linux distribution that can have the cifs-utils package installed. The following Linux distributions are available for use in the Azure gallery: o

Ubuntu Server 14.04+

o

RHEL 7+

o

CentOS 7+

o

Debian 8+

o

openSUSE 13.2+

o

SUSE Linux Enterprise Server 12

The cifs-utils package is installed. The cifs-utils package can be installed using the package manager on the Linux distribution of your choice. On Ubuntu and Debian-based distributions, use the apt-get package manager: sudo apt-get update sudo apt-get install cifs-utils

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 29 *

On RHEL and CentOS, use the yum package manager: sudo yum install samba-client samba-common cifs-utils

Mount the Azure file share on-demand with Create a folder for the mount point: A folder for a mount point can be created anywhere on the file system, but it's common convention to create this under the /mnt folder. For example: mkdir /mnt/MyAzureFileShare Use the mount command to mount the Azure file share: Remember to replace <storageaccount-name> , <share-name> , <smb-version> , <storage-account-key> , and <mount-point> with the appropriate information for your environment. If your Linux distribution supports SMB 3.0 with encryption (see Understand SMB client requirements for more information), use 3.0 for <smbversion> . For Linux distributions that do not support SMB 3.0 with encryption, use 2.1 for <smb-version> . Note that an Azure file share can only be mounted outside of an Azure region (including on-premises or in a different Azure region) with SMB 3.0. sudo mount -t cifs //<storage-account-name>.file.core.windows.net/<share-name> <mountpoint> -o vers=<smb-version>,username=<storage-account-name>,password=<storage-accountkey>,dir_mode=0777,file_mode=0777,serverino

Create a persistent mount point for the Azure file share with /etc/fstab Create a folder for the mount point: A folder for a mount point can be created anywhere on the file system, but it's common convention to create this under the /mnt folder. Wherever you create this, note the absolute path of the folder. For example, the following command creates a new folder under /mnt (the path is an absolute path). sudo mkdir /mnt/MyAzureFileShare Use the following command to append the following line to /etc/fstab : Remember to replace <storage-account-name> , <share-name> , <smb-version> , <storage-account-key> , and <mountpoint> with the appropriate information for your environment. If your Linux distribution supports SMB 3.0 with encryption (see Understand SMB client requirements for more information), use 3.0 for <smb-version> . For Linux distributions that do not support SMB 3.0 Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 30 *

with encryption, use 2.1 for <smb-version> . Note that an Azure file share can only be mounted outside of an Azure region (including on-premises or in a different Azure region) with SMB 3.0. sudo bash -c 'echo "//<storage-account-name>.file.core.windows.net/<share-name> <mountpoint> cifs nofail,vers=<smb-version>,username=<storage-account-name>,password=<storageaccount-key>,dir_mode=0777,file_mode=0777,serverino" >> /etc/fstab'

Prerequisites for mounting an Azure File share on macOS Storage account name: To mount an Azure File share, you will need the name of the storage account. Storage account key: To mount an Azure File share, you will need the primary (or secondary) storage key. SAS keys are not currently supported for mounting. Ensure port 445 is open: SMB communicates over TCP port 445. On your client machine (the Mac), check to make sure your firewall is not blocking TCP port 445.

  

Mount an Azure File share via Finder 1.

Open Finder: Finder is open on macOS by default, but you can ensure it is the currently selected application by clicking the "macOS face icon" on the dock: 2. Select "Connect to Server" from the "Go" Menu: Using the UNC path from the prerequisites, convert the beginning double backslash ( \\ ) to smb:// and all other backslashes ( \ ) to forwards slashes ( / ). Your link should look like the following:

3. Use the share name and storage account key when prompted for a username and password: When you click "Connect" on the "Connect to Server" dialog, you will be prompted for the username and password (This will be autopopulated with your macOS Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 31 *

username). You have the option of placing the share name/storage account key in your macOS Keychain. 4. Use the Azure File share as desired: After substituting the share name and storage account key in for the username and password, the share will be mounted. You may use this as you would normally use a local folder/file share, including dragging and dropping files into the file share:

Mount an Azure File share via Terminal 1.

Replace <storage-account-name> with the name of your storage account. Provide Storage Account Key as password when prompted.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 32 *

mount_smbfs //<storage-account-name>@<storage-accountname>.file.core.windows.net/<share-name> <desired-mount-point> 2. Use the Azure File share as desired: The Azure File share will be mounted at the mount point specified by the previous command.

Storage design for highly available applications What we have covered so far shows the importance of Azure Storage as a cornerstone for building whatever you want to build on Azure. Therefore, in this topic, we will cover some of the most important features you have to implement in order to be able to design highly available storage for your applications. The following points should be our main focus when designing a highly available storage solution:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 33 *



RA-GRS



Azure Backup



Azure Site Recovery



Premium Storage

RA-GRS RA-GRS is the highest durable replication type, as was covered in the last chapter. Not only this, but it also gives you read access to your storage in another region, which may reduce latency if you have to query your storage from a place that is nearer to the that region than the primary region, considering that you will access the Last Sync Time value, as Azure uses asynchronous replication between regions. But, you might wonder, what if the primary region was completely destroyed, and your application needed to make write operations to the storage? The answer is simple, Microsoft will failover the other region, but not immediately, it might take some time. Therefore, for a complete guide about what to do if a storage outage occurs, how to prepare for it, and even how to detect it, you can check out the following link: https://docs.microsoft.com/enus/azure/storage/storage-disaster-recovery-guidance .

Azure Backup Lately, many enterprises have been exposed to piracy, especially ransomware. Without having a backup of your storage, your data might be encrypted, and you will have to either pay to get Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 34 *

it decrypted, or you will never see it again. That is why considering Azure Backup as a part of you design will be very beneficial to keeping your application highly available.

Azure Site Recovery If you have your application built on Azure VMs, or even on-premises VMs, you have to make sure that there's a disaster recovery site for your environment, and to do so you can user Azure Site Recovery. Azure Site Recovery will replicate changes to the VMs' virtual hard disks, so, whatever happens, you will be good to go with your disaster recovery site, keeping your application highly available.

Premium Storage Using Premium Storage will increase the performance of your application; as a result, it will be highly available. Azure Premium Storage has a throughput rate of up to 50 GBps and 80,000 IOPs. Using storage with such specifications can make not only a highly available application, but also an application with super-fast performance.

Automating tasks Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 35 *

As usual when we reach the end of a chapter, we will work on automating the tasks that we have done manually. So, let's get started.

Creating Blob storage using PowerShell In this topic, we will cover how to create Blob storage that everyone has read/write access to in the storage account we created in the last chapter: $ContainerName = packtpubbs $SAK = Get-AzureRmStorageAccountKey -StorageAccountName "packtpubsaps" -ResourceGroupName packtpub $SAK = $SAK | Where {$_.KeyName -like “key1”} $SC = New-AzureStorageContext -StorageAccountName packtpubsaps -StorageAccountKey $SAK.Value New-AzureStorageContainer -Name $ContainerName -Permission Container -Context $SC Set-AzureStorageBlobContent -Container $ContainerName -File C:\test.txt -Context $SC

Steps in detail: 1. Create a variable for the container name, so you do not have to write it repeatedly. 2. Create a variable for the storage account key named $SAK, within which the access keys will be stored. 3. Since you need the primary key only, you have to select it and put it into the same variable. 4. Then, you have to create a storage context which encapsulates the storage account credentials, named $SC. 5. As the blob needs a container to be created within, you have to create a new one, considering that the Permission parameter refers to Access type in the portal, in addition to specifying the Context to know in which storage account this container will be created. 6. Once done, you can start to upload your storage to the blob.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 36 *

In the beginning, we have got to create a storage account key variable, $SAK, within which there will be the access key values. But, since we only need the primary key, we have got to select that key and put it into the $SAK variable

Creating Blob storage using the Azure CLI 2.0 Let's repeat the previous task, but this time, we are going to use the Azure CLI 2.0: az storage container create --name packtpubbs --public-access container --account-name packtpubsacli az storage blob upload --file C:\test.txt --container-name packtpubbs --name blobcli --account-name packtpubsacli

In the preceding command, we have only changed the storage account, which we have created to be used in tasks implemented by the Azure CLI 2.0, and we named the blob blobcli

Creating Table storage using PowerShell Using the same session opened in PowerShell in the previous task, let's create a table. New-AzureStorageTable -Name packtpubtable -Context $SC

Creating Table storage using the Azure CLI 2.0 Creating a table using the Azure CLI 2.0 is very straightforward; you only have to specify the name of the table and the storage account within which the table will be created: Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 37 *

az storage table create --name packtpubtable --account-name packtpubsacli

Creating Queue storage using PowerShell Using the same PowerShell session, let's create a queue: New-AzureStorageQueue -Name packtpubqueue -Context $SC

Creating Queue storage using the Azure CLI 2.0 Again, we will just follow the same format as the storage services we created previously: az storage queue create --name packtpubqueue --account-name packtpubsacli

Azure Storage for VMs In this chapter, we will go through the relationship between Azure Virtual Machines (VMs) and Azure Storage. The chapter will kick off by introducing Azure VMs, moving forward to how to create these VMs, then you will learn about the storage considerations you need to take

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 38 *

care of to get a better design for your environment, and even how to capture images from these VMs. Finally, you will learn how to automate these tasks. The following topics will be covered: 

An introduction to Azure VMs



Creating an Azure VM



Storage considerations for Azure VMs



Capturing VMs



Automating your common tasks with Azure VM storage

An introduction to Azure VMs Azure VMs is the most well-known, usable, and oldest service available in Azure. Azure VMs provides the deployment of different flavors of Windows and Linux VMs. Using Azure VMs provides you with full control over the configuration and management of a VM. Management refers to installing software, patching, and even maintaining a VM. It is no surprise that, when you create a normal VM, whether it is on-premises or off-premises, you need storage for the VM's virtual hard disk. That leads us to understanding that Azure VMs use Azure Storage as a storage provider, and that is what I am going to cover in detail throughout this chapter. But before getting started and getting our hands dirty with playing with Azure VMs, I'm going to illustrate some confusing points regarding them.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 39 *

Fortunately, Microsoft bills VMs per minute, not per hour, therefore, when you use a VM for 30 minutes, you will only be charged for 30 minutes. Also, when a VM is not running, you will not be charged for the computing resources (CPU and Memory), however, you will be charged for the VM storage, so let’s discuss VM states in more detail: State

Description

Running

The VM is running and you get charged for usage as usual

Stopped

Stopped (deallocated)

The VM is shut down by Windows/Linux, but you still get charged for the VM, as it still deployed to the same physical host and resources are still reserved for it

The VM is stopped by the stop button on the VM blade via the Azure portal

At the time of writing, Microsoft has two Service Level Agreements (SLAs) for Azure VMs: 

Two or more VMs within the same availability set have 99.95% availability to one VM guaranteed



Using a single VM that uses Premium Storage will provide at least 99.9% availability

Azure VMs series Azure VMs have multiple series to fit different cases and scenarios: 

A Series: This series is most commonly used in development and test scenarios



D Series: This series has a fast CPU and solid-state drives (SSD) disks, and is most commonly used for general-purpose computing, such as relational databases, and every application that requires high IOPs



F Series: This series targets applications that require intensive compute power, such as web servers

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 40 *



G Series: This series targets applications that require high memory and fast storage, such as ERP, and SAP solutions



H Series: This series has very high compute capabilities, and might fit in scenarios that require high performance, such as analytics



L Series: This series is dedicated to applications that require low latency, high throughput, high IOPs, and large size disks, such as NoSQL databases



N Series: This series has high GPU capabilities, and fits scenarios such as video editing, graphics rendering, and so on

Storage considerations for Azure VMs As mentioned earlier, Azure VMs depend on Azure Storage to function properly. Therefore, let's go through some storage considerations for Azure VMs for a better design for your environment.

Managed versus unmanaged disks As mentioned earlier, managed disks save lots of effort and even support Premium Storage and Standard Storage.

Managed disks key points Let's cover some of the managed disks key points that may influence your design and even your decision when it comes to selecting whether to use managed disks or not:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 41 *



Simplicity: Using managed disks will eliminate lots of concerns when it comes to storage design because you will not have to consider the limitations of the storage account, such as IOPs. For example, if you have multiple VMs running and their disks are assigned to the same storage account, you might face some performance issues because the VMs' disks have exceeded the IOPs limitations.



Managed disks limits: Managed disks support up to 10,000 disks per subscription, which means a massive number of VMs can be created within the same subscription using managed disks.



Reliability: Using managed disks ensure that the disks of VM in an availability set are completely isolated by assigning disks to different storage scale units, so whenever a failure occurs in one of the VM disks, other VM disks are working properly.



Azure Storage Replication Support: At the moment, managed disks support only the Locally Redundant Storage (LRS) replication type.



Azure Backup support: Managed disks support Azure Backup and it is a very important thing to consider because, if the data center in which your storage exists get damaged, you must have a backup of your storage in another region.



Snapshot support: Managed disks support snapshot, which is a read-only copy of a managed disk, therefore, can be used as a backup method. Consider that snapshotted storage gets charged independently based on its size.



Images support: When you want to create an image from a Sysprepped VM, you can capture an image of all its managed disks so it can be used later as a template to create other VMs.



When using Azure, you have to consider that everything comes with a price. Most of the time, it is not expensive; however, when using managed disks, you must consider storage type, disk size, and so on, because that adds more credits.

VM disks Whenever you create a VM, there will be only one disk attached to it, but there are some other disks that can be added to Azure VMs, so let's figure out the types of VM disks: 

OS disk: The disk on which the operating system files exist, which is the C drive by default for Windows, and dev/sda for Linux. The OS disk size can be up to 2 TB.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 42 *



Temporary disk: This disk exists by default in any VM, but as its name suggests, it is temporary, which means it is non-persistent. In other words, whenever your VM is turned off, the data will be lost. This disk provides temporary storage and uses the drive letter D by default in Windows, and /dev/sdb1 in Linux. Temporary storage exists in the physical host of the VM, but since the VM could be moved to any other host at any time due to a failure in that physical host, your data will be lost. Also, the temporary disk size varies based on the VM size. In addition to this, when using temporary storage, you will not incur any charges. If you restart the VM via Windows, you will not lose the data that exists in the temporary storage disk; otherwise, you will lose it if any downtime occurs for whatever reason.



Data disk: This disk is not added to the VM by default, you will have to add it yourself, as you will see shortly. Data disks are used to store permanent data, which means it is persistent. For example, you can save your SQL database or any other application data. At the moment, the data disk maximum size is almost 4 TB, but you can add more than one data disk to a VM, depending on the VM size.

Microsoft has made a good comparison of Azure Premium Disks versus Azure Standard Disks, as shown in the following table:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 43 *

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 44 *

Reference: https://docs.microsoft.com/en-us/azure/storage/storage-about-disks-and-vhdswindows#types-of-disks .

Capturing VMs Templates … Templates … Templates… That is what we always seek when we need to create a machine with the same specifications regularly, especially in dev/test environments. So, what would you do if you wanted to have an image of a VM that you could use later to recreate other VMs without having to do all the steps you did to get this VM up and running? The answer is very easy to say and easily implemented. You only need to capture the VM, considering that the image will include all the disks added to that VM. There are two ways to capture the VM from the Azure portal. The first is, if you use managed storage, you will directly capture the image from the VM blade. But if you use unmanaged storage, you will navigate to Images and start capturing the VM. But before doing any of that, you have to sysprep the VM first.

Sysprepping the VM Before you proceed, connect to the VM first and follow these steps:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 45 *

1. Run sysprep, tick on the Generalize box, and select Shutdown in Shutdown Options, as shown in the following screenshot:

2. Once you click on OK, sysprepping starts, as shown in the following screenshot:

Now that you are done with sysprepping, the next step is to start capturing.

Capturing the VM with managed storage This method is easier and very straightforward, as you will see in the following steps: 1. Navigate to the VM you want to capture, as shown in the following screenshot:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 46 *

Once you click on Capture, you will be navigated to another blade, asking you to specify the following parameters: 

Name: The image name



Resource group: In which resource group put this image



Whether you are willing to remove the VM after creating the image or not

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 47 *

3. Once you click on Create, the process of capturing an image will start, and once this is done, you can find the image in Images, as shown in the following screenshot:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 48 *

Capturing the VM with unmanaged storage This method is straightforward too, but you have to get the VM sysprepped first, and then follow these steps: 

Navigate to Images and click on Add, and a new blade will open, as shown in the following screenshot:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 49 *



All the fields to be filled in are pretty straightforward:



Name: The name of the image.



Subscription: The subscription that will be charged for storing the image.



Resource group: The resource group that the image will be assigned to.



Location: Select the nearest location to you.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 50 *



OS type: Select the OS type of the VM, whether it is Windows or Linux.



Storage blob: You will browse for the storage account in which the VM is stored, which will open a

new blade for containers. Select the container in which the VM VHD is stored, then select the VM disk. 

Account type: Select the type of account, whether it is Standard (HDD) or Premium.



Host caching: It is preferable to leave the host caching for the OS disk Read/write.



Data disks: If you have any data disks that you want to attach to the VM image, you can click on Add data disk and add the disks you wish.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 51 *



Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 52 *

Automating the tasks As usual at the end of each chapter, we get our hands dirty with automation.

Creating an Azure VM using the Azure CLI 2.0 Creating an Azure VM using the Azure CLI 2.0 is pretty easy, and is only one command, shown as follows: az vm create --resource-group PacktPub --name PacktPubVMCLI --location westeurope --size Standard_DS2 --image win2016datacenter --storage-account packtpubsacli --use-unmanaged-disk --vnetname PacktPubvNet --vnet-address-prefix 10.0.0.0/8 --subnet PacktPubSubnet --subnet-address-prefix 10.0.1.0/24 --admin-username pbuser --admin-password P@cktPub@2017

Adding data disks to an Azure VM using the Azure CLI 2.0 The process of adding data disks using the Azure CLI 2.0 is pretty easy and can be done using only one command, shown as follows. The disk I am going to create is a managed disk: az vm disk attach --vm-name PacktPubVMCLI --resource-group PacktPub --disk PacktPub-DataDiskCLI --size-gb 4095 –new

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 53 *

Resizing Azure VM disks using the Azure CLI 2.0 As we did when resizing using PowerShell, you have to first create a new disk with a small size, then run the following command: az disk update --name PacktPub-DataDiskCLI1 --resource-group packtpub --size-gb 4095

Capturing the VM using the Azure CLI 2.0 Again, you have to sysprep the VM by running the following commands: az vm deallocate --resource-group PacktPub --name PacktPubVMCLI az vm generalize --resource-group PacktPub --name PacktPubVMCLI az image create --resource-group PacktPub --name PacktPubCLImage --source PacktPubVMCLI

Further information Azure VMs and Azure Storage could not be covered entirely in this chapter, however, it has covered most of the common topics you will deal with. For further information about Azure Storage for VMs, check out the following URLs 

Migrating Azure VMs with unmanaged disks to managed disks: https://docs.microsoft.com/enus/azure/virtual-machines/windows/migrate-to-managed-disks



Disk snapshots: https://docs.microsoft.com/en-us/azure/virtual-machines/windows/snapshot-copymanaged-disk

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 54 *



Backup Azure unmanaged VM disks with incremental snapshots: https://docs.microsoft.com/enus/azure/virtual-machines/windows/incremental-snapshots



Convert Azure managed disks storage from Standard to Premium and vice versa:https://docs.microsoft.com/en-us/azure/virtual-machines/windows/convert-disk-storage



Migrate from Amazon Web Services (AWS) and other platforms to managed disks in Azure: https://docs.microsoft.com/en-us/azure/virtual-machines/windows/on-prem-to-azure

Implementing Azure SQL Databases In this chapter, we will go through one of the hottest topics, especially for database administrators (DBAs): Azure SQL Database. This chapter will be kicked off by an introduction to Azure SQL Database and why you should use this service, then the service tiers and performance level, which will be followed by some demonstrations of the Azure portal regarding how to create and restore Azure SQL Database. Finally, all the manual tasks we carry out in this chapter will be automated. The following topics will be covered: 

An introduction to Azure SQL Database



Why Azure SQL Database?



Service tiers



Creating an Azure SQL Database



Connecting to Azure SQL Database



Azure SQL Database business continuity

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 55 *



Automating your common tasks with Azure SQL Database

An introduction to Azure SQL Database A database is the most important component of most modern applications. Therefore, it is no surprise that we have two chapters of which I will cover most of the important key points and best practices for using Azure SQL Database. Azure SQL Database is a relational database as a service, which means it follows the Platform as a service (PaaS) cloud service model, wherein you do not have to manage the underlying infrastructure, including networks, storage, servers, the virtualization layer, the operating system, middleware, or runtime. You only have to manage your databases and do not even have to think about patching and updating your servers.

Why Azure SQL Database? Besides the reasons I've covered in the previous chapters as to why the cloud is always better than a traditional infrastructure, there are lots of other reasons for using Azure SQL Database, especially: 

Scalability: Azure SQL Database can be scaled according to your needs and usage, and more information about that topic will be covered later in the chapter.



Online scaling: No downtime is needed to scale your database size. For example, you can start your application with a size that fits it in the beginning, and Azure SQL Database can respond to the database's requirements by scaling whenever necessary without causing any downtime.



Hardcore monitoring: Azure SQL Database provides built-in monitoring and alerting tools that can be used to identify potential problems and even recommend actions to be taken in order to fix an issue. Alerts can also be generated based on the monitoring metrics, so you can receive an alert that something has gone wrong according to your baseline.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 56 *



Built-in intelligence: One of the coolest features of Azure SQL Database is built-in intelligence. It helps to reduce the costs involved in running databases and increases the performance of the application using Azure SQL Database as a backend.



Intelligent Threat Detection: This feature utilizes SQL Database auditing in order to detect any harmful attempts to access data. It simply provides alerts for any abnormal behaviors.



High availability: Microsoft provides many ways to ensure that Azure SQL Database is highly available: 

Automatic backup: To avoid any issues that might cause data loss, automatic backups are performed on SQL Databases, (these include full, differential, and transaction log backups).



Point-in-time restores: Azure SQL Database can be recovered to any point-in-time within the automatic backup retention period.



Active geo-replication: If you have an application that needs to be accessed from across the globe, you can use active geo-replication to avoid facing a high load on the original SQL Database. Azure geo-replication will create four secondary databases for the original database, with reading access.



Failover groups: This feature is designed to help customers to recover from databases in secondary regions if a disaster occurs in the region that the original database is stored in.

Service tiers Azure SQL Database is available in two flavors: 

Elastic database pools



Single databases

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 57 *

Elastic database pools Elastic database pools are a great solution for managing multiple databases, scaling their performance according to the databases' needs, which means it is a good fit for databases with unpredictable usage demands, and that leads to a saving on credits. Elastic database pools share performance across many databases since all of these databases are built on a single Azure SQL Database server.

Single databases Single databases are a good fit for a set of databases with predictable performance, where the required resources for the databases are predetermined.

Service tier types At the time of writing, there are four service tiers for Azure SQL Database: Basic, Standard, Premium, and Premium RS (in preview). All of these offer support for elastic database pools and single databases. The performance of these tiers is expressed in Database Transaction Units (DTUs) for single databases, and elastic Database Transaction Units(eDTUs) for elastic database pools. DTUs specify the performance for single databases, as they provide a specific amount of resources to that database. On the other hand, eDTUs do not provide a dedicated set of resources for a database, as they share resources within a specific Azure SQL Server to all the databases which run that server.

The following is a table from Microsoft which illustrates the different tiers' performance levels for elastic databases pools: Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 58 *

The following is a table that illustrates the different tiers' performance levels for single databases:

Creating and Connecting to Azure DB: https://docs.microsoft.com/en-us/azure/sql-database/sql-database-get-started-portal

Azure SQL Database business continuity So far, you have your database up and running on the cloud, and you can even connect to it, create, delete, and update the tables as you wish. I think that should satisfy most of your needs, but since a database is something critical and you need to make sure that it will not be lost if corruption occurs, you will need to take some backups.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 59 *

Unfortunately, when you check the database blade, you will notice that backup is not mentioned in the blade, as shown in the following screenshot:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 60 *

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 61 *

How business continuity works for Azure SQL Database Microsoft does its best to address any issues that might occur with Azure SQL Database, and it provides the following solutions for this purpose.

Hardware failure Hardware failure is something that is expected to happen, but it will not be the reason you lose your databases.

If the hardware failure occurs, there are three copies of your database separated across three physical nodes. The three copies consist of one primary replica and two secondary replicas, and in order to avoid any data loss, write operations are not committed in the primary replica until they have been committed in one of the secondary replicas. Therefore, whenever a hardware failure occurs, it will failover to the secondary replica.

Point-in-time restore As mentioned earlier, point-in-time restores can recover Azure SQL Database at any point in time within the automatic backup retention period. The retention period varies from one tier to another: 7 days for the Basic tier, 35 days for the Standard tier, 35 days for the Premium tier, and 35 days for the Premium RS tier. This solution would suit a scenario where your database has been corrupted and you want to restore it to the last healthy point. To restore your database to the last healthy point, you have to follow these steps:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 62 *



Navigate to the database you want to restore to the last point, as shown in the following screenshot:



Once you have clicked on Restore, a new blade will pop up, where you can give the restored database a new database name, determine the time that you want to restore to, and change the pricing tier for the restored database, as shown in the following screenshot:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 63 *



Once you click on OK, it starts restoring the database, as shown in the following screenshot:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 64 *

Restoring Azure SQL Database key points For a better understanding of the process, you should consider the following key points during implementation: 

When you restore a database, a new database will be created, which means you will have to pay for the new database too



You cannot name the new database with the same name as the original database because the original still exists; to do so you would have to remove the original one



You can choose a restore point between the earliest point and the latest backup time, which is six minutes before the current time



Database recovery time varies from one database to another according to many factors; here are some of them: o

The database size

o

The number of transaction logs involved in the operations

o

The database performance level

o

If you are restoring the database from a different region, the network bandwidth might cause a delay

Restoring a deleted database You can accidentally remove a database, or you might have removed a database and figured out later that you need it. This can be a tough situation. However, Microsoft Azure supports database recovery even in the case of deletion, but the SQL Server on which the database was built cannot have been deleted because, at the time of writing, there is no support for the recovery of deleted SQL Servers. To restore a deleted database, follow these steps:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 65 *

1. Navigate to SQL Servers and select the server on which the deleted database was built. 2. Scroll down to Deleted databases in the SQL Server blade, as shown in the following screenshot:

3. Select the database you want to restore and name it as you wish, considering that you cannot give the name of an existing database that is already running on the same SQL Server, but you can give it its old name, as shown in the following screenshot:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 66 *

4. Once you are done, you can click on OK, and it will start the restoration process.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 67 *

Automating the tasks As usual, at the end of each chapter, we get our hands dirty with automation.

Creating an Azure SQL Database using the Azure CLI 2.0 To create an Azure SQL Database using the Azure CLI 2.0, we have to create an Azure SQL Server first, which can be done by running the following command: az sql server create --name packtpubsqlcli --resource-group PacktPub --location westeurope --admin-user “SQL Admin User” --admin-password “SQL Admin Password”

Then, the following command will be run to create the database: az sql db create --resource-group PacktPub --server packtpubsqlcli --name PacktPubDBCLI --service-objective S3 --collation SQL_Latin1_General_CP1_CI_AS

Creating an SQL Server-level firewall rule using Azure CLI 2.0 Creating an SQL Server-level firewall rule using the Azure CLI 2.0 is pretty straightforward, as we did in PowerShell. To do so, you have to run the following command: az sql server firewall-rule create --resource-group PacktPub --server packtpubsqlcli --name PPRule --startip-address XXX.XXX.XXX.XXX --end-ip-address XXX.XXX.XXX.XXX

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 68 *

SQL Database (IaaS/PaaS) An SQL Database can be implemented in Azure in two ways: 

Using Azure SQL Database: It follows the PaaS model, and we have been using it so far



Using Azure VMs and building SQL on them: This follows an IaaS model, and will be covered in more detail shortly

Azure SQL Database (PaaS) As mentioned earlier, Azure SQL Database is a relational database as a service, built and hosted on Azure. Azure SQL Database minimizes the costs of managing and provisioning databases. Using this model will reduce the responsibility of managing the virtual machines that host the SQL server, the operating system, and even the SQL Server software. This model eliminates concerns regarding upgrades, backups, and even the high availability of databases, because they are not your responsibility anymore, in addition to being able to add databases as you wish, whenever you want. Taking this into account, you will pay less credits because, in this scenario, you will not pay for a VM with SQL installed on it, plus the license credits; you will only pay for the database you are using.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 69 *

Scenarios that would fit in Azure SQL Database Azure SQL Database would be a best fit for the following scenarios: 

Cloud applications that need to be developed quickly



Building a highly-available and auto-upgradable database that is recoverable in the event of disasters



A database with less management needed for its OS and configuration



Building a Software as a service (SaaS) application



If you want complete management of your SQL installation, but no worries about hardware

SQL on Azure VMs (IaaS) This type of deployment of an SQL Server is much more complicated than using Azure SQL Database, as a VM built on Azure and an SQL Server built upon it requires more administration. Also, you can use whichever versions you want to use (2008R2, 2012, 2014, 2016, 2017), and whichever edition you need (Developer, Express, Web, Standard, or Enterprise).

Scenarios that would suit SQL on Azure VMs The following scenarios would be the best fit for building SQL on Azure VMs: 

Migrating existing apps on-premises to Azure with minimal changes



Having a SQL environment wherein you have full access to it



Needing databases of up to 64 TB storage, since Azure SQL Database can support only up to 4 TB



Building hybrid applications with SQL Database as a backend

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 70 *

Azure SQL elastic database pools In the previous chapter, some interesting and important topics were covered regarding elastic database pools. In this section, we will be working on creating and managing elastic database pools.

Creating an elastic database pool To get your elastic database pool up and running, you have to follow these steps: 

Navigate to the SQL databases blade, and click on Add or Create SQL databases , as shown in the following screenshot:



Once you have clicked on Add or Create SQL databases, a new blade will pop up, as shown in the following screenshot:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 71 *



Since most of the required fields were covered in the previous chapter, only changed options will be covered: a) Want to use SQL elastic pool?: This will be Yes this time.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 72 *

b) Once Yes is selected, Pricing tier option, which is shown in the preceding screenshot, will be changed to Elastic database pool, where it will ask you to configure the pool setting. Once you have clicked on it, a new blade will pop up, as shown in the following screenshot:



Once you have clicked on Elastic database pool , a new blade will pop up, as shown in the following screenshot:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 73 *



You can name the elastic pool as you wish. To determine the pricing tier you are going to select, click on Pricing tier and a new blade will pop up displaying the different tiers, as shown in the following screenshot:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 74 *



Once you have selected your desired tier, you can open the Configure pool blade, where you can specify Pool eDTU, Pool GB (which are the reserved elastic Database Transaction Units (eDTUs)), and the maximum storage capacity of GB for the pool. Take into consideration that the ratio of eDTU to GB is determined by the pricing tier, as shown in the following screenshot:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 75 *



Once you are done, click on Select for the pool, and in the Elastic database pool blade too.



Finally, click on Create.



Once it is done with deployment, you will find the newly created database in the SQL databases blade, as shown in the following screenshot:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 76 *

Adding a database to the elastic database pool Once you get your elastic database pool up and running, you can add databases to it, and to do so, you need to follow these steps: 1. Navigate to the SQL servers blade and click on the SQL Server created earlier to be able to add new databases to it, as shown in the following screenshot:

There are two ways to add databases to the pool according to the preceding screenshot: 2. New database: In this case, you will go through the same steps we did earlier 3. Import database: In this case, you will import an existing database to be added to the pool 4. In this demonstration, adding a new database will be covered. Therefore, we will click on New database.

5. Once you have clicked on it, a new blade will pop up and you will have to determine whether you want to use an elastic database pool or not. If you do, you will have to specify which pool it should be added to until you have filled in all the fields, as shown in the following screenshot:

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 77 *

6. Once you are done, click OK, and the database will be created in seconds.

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 78 *

Active geo-replication Active geo-replication is one of the most important business continuity methodologies. When using active geo-replication, you can configure up to four secondary databases within the same region or in different regions with reading access. This will help to reduce latency for users or applications that need to query the database from a different region. If a catastrophic disaster occurs, you can failover to the other region using a failover group. Failover groups are mainly designed to manage every aspect of geo-replication automatically, such as connectivity, relationships, and failover. Considering that it is enabled across Azure SQL Database Basic and Standard service tiers.

Implementing active geo-replication https://docs.microsoft.com/en-us/azure/sql-database/sql-database-geo-replication-portal

Automating the tasks Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 79 *

As usual, at the end of each chapter, we get our hands dirty with automation.

Creating an elastic database pool using Azure CLI 2.0 To create an elastic database pool, you must have an Azure SQL Server. You can use the one created in the previous chapter using the Azure CLI 2.0 or create a new one yourself: az sql elastic-pool create --resource-group PacktPub --server packtpubsqlcli --name EDPCLI --dtu 400 --db-dtu-min 10 --db-dtu-max 100

Adding an additional database to the elastic database pool using Azure CLI 2.0 To add a database to the created elastic pool, run the following command: az sql db create --resource-group PacktPub --server packtpubsqlcli --name additionaldbcli --elastic-pool edpcli

Quality Thought www.facebook.com/qthought www.qualitythought.in PH NO: 9963799240, 040-48526948 Email Id: [email protected] 80 *