ClouDump: Upcoming Blogs Unformatted
AWS:
1. AWS service domains:
a. Compute
b. networking
c. storage
d. databases
2. Virtualization in AWS:
Basically there are two types of hypervisors:
Type 1: Native/Bare metal Hypervisor
It has 3 layers: Hardware===>>Hypervisor===>>OS
Type 2: Hosted hypervisor
It has 4 layers: Hardware===>>OS===>>Hypervisor===>>OS
AWS or any cloud provider uses Type1 Hypervisor.
To be specific AWS uses xen,kvm,nitro type1 hypervisors.
Microsoft Azure has its own hypervisor known as Microsoft HyperV.
3. EC2 instance types:
General purpose: Balanced(t2.micro etc, m5, m4, m3)
Storage Optimised: Lots of storage( h1, i3, D2)
Memory optimised: Lots of RAM(R3,R4, x1e, x1)
Compute Optimised: Lot of CPU( c5, c4, c3)
Accelerated Computing : Lot of CPU plus RAM(p3, p2, f1, g3)
4. EBS: It is raw unformatted block level storage. Default ec2 has formatted ebs but if we connect manually an EBS volume, it has to be formatted first.
Throughput: Data Transfer rate in Mbps. If you want to transfer one big amount of data you will see for better throughput.
IOPS: Number of read and write operations every second that a volume can go through. If you want to transfer large data but is in small chunks, then you will look for better iops.
EBS volume types:
1. General Purpose SSD or Gp2: 100-10000 IOPS & 160 MB/s Throughput.
2. IO1 Provisioned SSD: 100-32000 IOPS, 500 MB/s Thorughput.
3. ST1 Throughput Optimised HDD: 500 iops, 500 MB/s thoughput,
4. SC1 cold storage ssd:250 iops, 192 MB/s thorughput
5. EFS: It is file based storage.
Its advantages are:
a. Elastic
b. Sharable
c. Highly available
d. High performance
6. Instance Tenancy And Reserved And Spot Instances:
a. Shared/Default instance: ec2 launched on any hardware from aws. If we reboot the instance ip address will change completely because after reboot same instance might be launched on different hardware from aws.
b. Dedicated instance: It is like leasing a hardware from aws. SO, everytime you launch a instance it will be launched in dedicated hardware.
c. reserved instance: instance kept reserved for you
d. Spot instance
Pricing in AWS:
1. Varies from region to region.
2. Data transfer out from ec2 to s3, glacier, dynamodb, ses and sqs in same region is free. In different AZ is cost.
Data transfer out from ec2 to ec2, rds, redshift, elastic cache, ELB and ENI is free in same AZ with private ip, if with public ip it costs. Also in different az either public or private ip it will cost.
7. Load Balancers:
Classic load balancer: (functions at layer 7 of OSI model)
- uses round-robin format for load balancing.
- Routes traffic to healthy instance only.
Network load balancer: (functions at layer 4)
- Ideal for balancing the TCP traffic.
- Supports elastic and static ip.
- Handles millions of requests per sec.
Application loadbalancer:( functions at layer 7)
- Identifies incoming traffic and routes them to appropriate server.
2 load balancers connection to one istance is possible in any type.
In network load balancer and application load balancer, cross-zone load balancing is enabled by default. In Classic Loadbalancer we have to enable cross-zone loadbalancing manually.
8. Auto-Scaling:
If you dont want to use autoscaling for some time, rather than terminating instances or deleting Autoscaling Group, you can put Autoscaling in Standby mode. After yuo exit standby mode, instances will again be sent to pending for health check and after health check instances and autoscaling groups will be up and running.
Components of AutoScaling:
a. Launch Configuration ( cannot be modified once created)
b. Autoscaling Groups
c. Scaling options
- Manual Scaling
- Dynamic scaling (Like scaling according to cpu utilization)
- Scaling based on demand or schedule.
9. Scaling Policy and Alarm:
- Scaling policy tells how much to scale and alarm tells when to scale.
- LIke scaling policy of launching 2 instances when alarm of cpu utilization above 80percent is triggered.
- Minimum, desired and maximum number of instances are to be specified.
Types of scaling policy:
1. Simple scaling: only one trigger is provided.
2. Step scaling: LIke at 60 percent cpu add 2 instance, at 80 percent add 3 instances, at 85percent add 4.
Termination policy for instance:
These instances will be terminted first:
1. oldest instance
2. Old configuration instance
3. cloosest to next instance hour
If an instance is under instance protection, it will not get terminated no matter what.
Never connect autosaling group directly to a load balancer which has instances running. Doing so may terminate the instances.
10. Route 53: It is highly available and scalable Domain NAme System provided by AWS.
ISP DNS resolver connected to root server, TLD server and DNS server toghether work to display you the website page.
TLD is Top Level Domain which is nothing but .com, .edu, .gov ectcetra.
Authoritative name server: This is the component that holds actual DNS records such as A Name, C name, Alias records etcetra.
A name is mainly used to connect ip to your baught DNS .
Alias is used to connect AWS services DNS name to the Actual DNS name.
CNAME or canonical name is used to connect one dns name to another.
NS record set are to be provided from route 53 to your DNS provider for connection between your DNS and route 53.
11. Routing Policy:
- Simple routing policy.
- Failover routing policy: Active-passive routing
- Weighted routing policy
- Latency based routing
- GeoLocation based routing
#selfpaced
10. VPC
- VPC is virtual Private Cloud, a logically isolated section in AWS region where we can launch our services.
- Flow of connection of VPC to Branch Office:
VPC==>Internet Gateway==>Public Subnet==>Customer Gateway==>Branch Office
What is IP? Unique string of numbers assigned to a computer using internet protocol to communicate over a network.
CIDR: (Classless Inter Domain Routing)
Class A:
x.0.0.0/8 => First 8 bits are networking bits which cant be changed, rest can be changed as they are host bits.
Number of host ips possible are=(2^24)-1 i.e last ip is X.255.255.255
Class B:
X.X.0.0/16 => First 16 bits are networking bits, rest are host ip bits that can be changed.
Possible number of host ips are (2^16)-1 i.e 65535 possible ips. Last ip= X.X.255.255
Class C:
X.X.X.0/24 => (2^8)-1 i.e 255 ips can be made. LASt ip is X.X.X.255
127.x.x.x and 169.x.x.x cant be used to create vpc or for private ips.
Firewall: It is a system to prevent unauthorized traffic to and from your private network/computer/server.
Types:
Stateful: It can only allow traffic. No control over response traffic.
Stateless: It can allow and deny traffic. Control over both request and response traffic.
11. Components of VPC:
a. Network Interface: Network IO happens here. It is entrypoint for internet to communicate with your system. It contains MAC Adress, Public IP, Private IP, Elastic IP and Security Groups.
b. Route Tables: It tells machine where traffic is directed. Like to make subnet to contact with internet, in route table you have to route internet gateway to that subnet.
c. NAT Gateway (Network Address Translation)
d. Internet Gateway: It is horizontally scalable, highly available VPC component which allows communication via internet.
e. Security groups and NACL
Advantages of VPC: security, flexibilty,
Types of VPC and subnet: Default Vpc & Non-default VPC
Subnet: It is dividing large network into smaller logical networks.
- Ec2 in one subnet cant communicate with EC2 in another subnet. IT has to go through main router i.e using private ip.
Security:
Security Groups: (Stateful)
Inbound rules: Rules that control who can access your resources from the outside. Outbound rules: They manage which external destinations your resources can communicate with.
NACL:(Stateless)
- It is attached to subnets.
- Both allow and Deny traffic rules can be assigned.
12. VPC Peering:
- Network connection between 2 VPC so that services in two different VPC in same or different regions or accounts can comunicate with eachother using private ips.
- Also after VPC peering route tables has to be configured asell.
13. VPC Endpoints:
- If you dont want to connect two VPC's completely, you can use VPC endpoint to connect your VPC to that specific service only.
- If Gateway Endpoint is used it need to be routed in route table.
- If Interface Endpoint is used no need to route as it uses private link.
- It doesnt need IGW, NAT, VPN or AWS DIrect Connect for communication.
In summary, VPC endpoints provide private access to AWS services from your VPC, while VPC peering establishes private connectivity between VPCs for inter-VPC communication.
14. S3 (Simple Storage Service)
It is a cloud-based storage service. It offers scalable and secure object storage for storing and retrieving data. It allows you to store and retrieve any amount of data from anywhere on the web. With S3, you can organize your data into buckets, which are containers for your files, and easily manage access permissions for each bucket. S3 is highly durable and available, making it suitable for a wide range of storage use cases, such as backup and restore, data archiving, static website hosting, content distribution, and data lakes.
It can easily be integrated with other AWS services.
- Name of s3 bucket should be globally unique.
- Replication is done automatically in another AZ.
- Read after write consistency, i.e as soon as data is uploaded it can be used.
15. S3 data consistency model:
- Eventual consistency, i.e until data is completely deleted it can be accessed.
Consistent read vs Eventual consistent read:
- Consistent Read: No stale or incorrect read information, Higher read latency, Read throughput is lower.
- Eventual Consistent read: Possible stale or incorrect read, lower read latency, Read throughput is high.
So, to summarize:
- Uploading new objects and automatic replication in S3 follow read after write consistency.
- Deletion of objects and overwriting data in existing objects in S3 follow eventual consistency, with a short delay before the changes are fully visible.
16. S3 storage hierarchy:
a. Buckets
b. Objects
c. Folders (Prefixes): While S3 does not have an explicit folder structure, you can mimic folders by including a forward slash ("/") in the object key. Folders created are only logical folders.
It's important to note that S3 is an object storage system and not a traditional file system. Although the folder-like hierarchy is emulated through the object keys, there are no actual directories or subdirectories in S3. The object keys provide a logical way to organize and retrieve objects within a bucket.
17. Buckets:
- 100 is max number of buckets can be created in one account.
- While using SDK,for default n.verginia region endpoint is in:s3.amazon.com format but if bucket is in another region endpoint is:
s3<region name>.amazon.com.
- You can also access a specific object by using URL with that objects path.
- Name of s3 bucket should be globally unique. Also all letters should be in lower case.
18. S3 object metadata:
- Date of upload and time
- Size of object
- Last modified
- Server side encryption enabled or not
- version id
- delete marker
- storage classes
Versioning:
- First you need to enable versioning explicitly.
- Now if same objects are uploaded they will not be overwritten, but will be stored in different versions form.
- If you download object latest version will be downloaded.
- When you delete object, latest version will be deleted only from front page and can be still accessed from versioning page.
- So to completely delete versioned object, it has to be deleted twice from front page and then from versioning page.
19. Life cycle management:
- Standard to any storage class is possible.
- Any storage class to glacier or deep archive.
- From glacier only to deep archive.
- We cant go "back" in changing storage class. Like syandard_iA to standard or Deep_archive to glacier etcetra.
- Object has to be stored atleast for 30 datys before moving to another class.
- Objects can be expired or deleted.
- A particular folers lifecycle can also be done.
20. Storage Classes in S3:
a. Standard: 11 9s durability, 4 9s availability
b. Standard_IA: 4 9s durability anda availability
c. onezone_IA
d. Intelligent_Tiering
e. Glacier: Backup data
f. Deep_Archive
- In all storage classes it is per GB data fee, in Intelligent_tiering it is per object fee.
Storage analysis:
- 100 analysis can be done.
- Whole bucket analysis or analysis of specific objects or prefix with that tags can be done.
- Following factors are considered:
Data retrieved out
Percent of storage retrieved
Percent of storage infrequently accessedw
Inventory in s3: The inventory report provides a comprehensive list of object metadata, such as key (name), size, storage class, last modified date, and other relevant information.
The inventory report can be scheduled to run at regular intervals, enabling you to automate the process of generating inventory reports for your S3 buckets. The report can be stored in another S3 bucket or delivered to a specified destination, such as Amazon S3, Amazon Glacier, or Amazon Redshift for further analysis or archiving.
Cross-region replication:
- Automatic asynchronous replication of objects to different region.
- Versioning has to enabled for this.
- Replicated object cant be replicated again in another region.
Encryption in s3:
- Server Side Encryption: This is encryption done on the object stored in s3 is encrypted.
- SSE can be done using special SSE-S3 feature from aws or using AWS KMS or Even you can use your own key i.e custom ARN key.
-Client side encryption: Encryption done in client side when data is moved to s3. You can use AWS kms or your own key.
Server Access Logging:
- You can manually turn on access logs.
- It will provide data about access logs i.e data about who accessed data, when, time taked, his/her ip etcetra.
S3 Access point & VPC Endpoint:
- If an ec2 is created in private subnet with no access to internet through NAT, it cant communicate with s3.
- But if you connect S3 access point with VPC endpoint, ec2 in private instance with no internet access can communicate with s3.
S3 Access Points is creating unique entry points with specific permissions and configurations.
Here are some key points about S3 Access Points:
- Separate Entry Points: With S3 Access Points, you can create separate entry points (URLs) for your S3 buckets. Each access point has its own name, which can be associated with a specific application, user, or purpose.
- Simplified Permissions: Access points allow you to apply a separate access policy to control who can access the associated bucket and what actions they can perform. This simplifies permission management and reduces the risk of granting unintended access.
- Fine-Grained Access Control: You can define access policies for each access point to specify allowed actions (e.g., read, write, delete) and conditions based on factors like IP address, time of day, or encryption requirements. This enables you to apply more granular access control at the access point level.
- Security and Monitoring: S3 Access Points support AWS Identity and Access Management (IAM) policies, making it easier to integrate with existing identity management systems. You can also enable Amazon CloudWatch metrics for monitoring access point activity and logging to Amazon CloudTrail for auditing and compliance.
- Data Transfer Optimization: S3 Access Points provide a way to configure network settings, such as VPC endpoints or custom endpoints, to improve data transfer performance and security.
By using S3 Access Points, you can enhance the security, manageability, and scalability of your S3 bucket access control, allowing you to grant specific permissions to different applications or users while keeping your data protected.
#selfpaced
21. Databases:
Database is collection of organized information which can be easily accessed, updated and maintained. The data can be added, updated, expanded and deleted.
Types:
a. RDBS: Relational database. tabular database. SQL queries are ran to update, delete or insert data.
b. No-SQL: Non-relational database. Data stored in key/value format. Data stored in JSON (Java Script Object Notation) format.
Databases on AWS:
a. Relational Databases:
- Amazon RDS
- Amazon Aurora
- Amazon RedShift
b. No-Sql Database/KeyValue: DynamoDB
c. In-Memory: ElasticCache
22. RDS:
- Fully managed RDBS.
- RDS takes care of patching, update, failure detection etc.
- It has these database engines: My-sql, Postgre-sql, MariaDB, OracleDB, Amazon Aurora, Sql Server.
Multi-AZ Deployment:
- Database is replicated in another AZ.
- Main DB is Primary DB and replicated DB in another AZ is StandBy replica.
- Snapshot of Primary DB is taken and using that snapshot StandBy replica is made.
- If synchronous replication, data is backed very fast.
- Reduced I/O freeze in failover i.e fast switching from primary db to standby db.
- Backups are taken from Standby replica thus reducing load from primaryDB for better functioning.
- Amazon KMS or custom keys can be used to encrypt DB. Backup data will also be in encrypted form.
- Manual Database snapshots can also be taken.
Read Replica:
- It can be taken in same region or different region.
- It copies data from primarydb to its read replica asynchronously.
- Automatic Backup has to be enabled to use this function.
-Replication varies for different database engines.
- Like replica of read replica can be created for mysql or postgre sql engines but same can't be done with oracle DB.
- Mysql, Oracle DB: Physical replication. PostgreSQL: Logical replication.
- Oracle DB is difficult to backup.
Reserved DB instance:
- 1 or 3 year reserved instance for discounts on avg of 69%.
23. Amazon Aurora:
- It supports MySql and PostgreSQL.
- It provides high availability by making data copies.
- Useing the data copies we can create replicas easily.
- Read replica or read and write replica can be created.
How is it different from other rdbms?
- Aurora provide 5x throughput as that of other MySql engine and 3x throughput as that of other PostgreSQL engine.
- You can backtrack to a previous point in time.
- You can use keys to encrypt data.
- Fully managed, does data setup, patching, updates etc.
24. DynamoDB:
- Nosql data base by AWS.
- It can store and retrieve any amount of data.
- In mysql database we have to have same number of columns for each row and cant be empty.
- Nosql is schemaless, thus we can have any number of key, attributes and we can have empty keys aswell.
Tables, Items and Attributes:
- Table is collection of items.
- Items is like row in RDBMS.
- Each Item has attributes. Attribute is like columns in RDBMS.
Primary Keys:
-The primary key in DynamoDB consists of two components: the partition key and, optionally, the sort key.
- Partition key divides set of items. If two sets have same partition key, you can use sort key for dividing the data.
Secondary Indexes:
- secondary indexes in DynamoDB enable efficient and flexible querying.
- Global Secondary Indexes (GSI): A global secondary index allows you to define a new partition key and sort key for a table. This enables querying the table using different attributes than the primary key. With a global secondary index, you can perform fast and efficient queries that are not limited to the primary key attributes. A table can have multiple global secondary indexes.
- Local Secondary Indexes (LSI): A local secondary index uses the same partition key as the table's primary key but allows you to define a different sort key. This means that the query must specify the partition key value, but it can perform range queries or lookups based on the sort key.
- A table can have up to five local secondary indexes.
DynamoDB Streams:
- DynamoDB Streams is a feature of Amazon DynamoDB that provides a time-ordered sequence of item-level modifications made to a DynamoDB table. It captures a log of changes, including both the "before" and "after" images of the modified items, and stores them in a stream.
- To process the stream, you can use AWS Lambda, which can be configured as a stream consumer. Lambda functions can be triggered by changes in the stream and perform custom logic based on those changes.
DynamoDB API's:
- CreateTable
- DescribeTable
- ListTables
- UpdateTable
- DeleteItem
DynamoDB Datatypes:
a. Scalar Types: One item. number, string, binary, boolean (true or false), null.
b. Document Types: Multiple complex nested attributes. Like JSON.
c. Set Types: Multiple sets. Can be string set, binary set or number set.
Read consistency:
a. Eventual consistent: quick result. Data might not be latest or updated.
b. Strong Consistent: High latency. But Provides latest updated data.
Read/Write capacity modes:
a. On-demand mode:
- Used when we create table with unknown workload.
- Used if unpredictable traffic.
- It has On-demand pricing
b. Provisioned mode: fixed price. Not scalable. Fixed capacity.
Partitions and data distributions:
- Partition is allocated storage for a table.
- Also used for replication in another AZ.
- Partition is managed by AWS.
- Partition is increaced if more throughput is required or need more storage.
25. Amazon RedShift:
- Fully managed AWS datawarehouse.
- Multiple nodes i.e RedShift Cluster has to be launched first to create it.
- Database used for data storage and retreival i,e transactional queries.
- Datawarehouse like redshift used to run complex and larger queries i.e analytical queries.
- All data is stored in columnar format.
- It uses ML to predict incoming querytime.
- If repeated queries, it uses result caching for faster response.
- It creates backup to s3 automatically.
- It is very costeffective, you pay only for resources used.
- Provides end-to-end encrption key managed by AWS.
26. ElastiCache:
- Uses in-memory-system.
- By caching data in-memory, ElastiCache reduces the load on your primary data sources, such as databases. This can help to alleviate the pressure on your backend systems, leading to improved overall system performance and lower costs.
ElastiCache supports two popular open-source caching engines:
- Redis
- MemCache
- Automatically scales in or out.
- Aws maanged.
- You can get on demand nodes or provisioned nodes.
27. CloudFormation:
- Structure and create entire infrastructure from one place.
- Also easy to delete entire ingrastructure using one command.
- Template can be reused.
- You can track changes by comparing your current remplate with previous template.
Features:
- JSON/YAML is used.
- You can use common programming languges too after used cloudformatio in conjunction with SDK's.
- Delete protection to template or services.
- If stack creation fails, all stack is rolled back gracefully.
Components of CloudFormation:
a. Template: Can be saved in json, yaml, text or .template file format.
b. Stack: After launching template u get stack, delete stack you can delete all infra.
c. Change Sets: Change set shows how our infrastructure will react to changes if changes are made.
How does CF works?
Code in yaml/json==>Upload to s3==>Use CF to deploy stack==>Infra is done
How change set works?
Update file=>Upload to s3(make sure to enable versioning)==>use CF to generate Change set==>View change set(actions taken to change by CF)==>Apply changes.
CF Templates:
- You can use AWS CF Designer if you want to create and save template.
- Comments will be deleted if you convert file from json to yaml in aws cf designer.
- CF supports ECMA-404 JSON standard.
- CF supports YAML version 1.1.
- AWS CloudFormation doesn't support the following features in YAML 1.1:
The binary, omap, pairs, set, and timestamp tags.
Aliases.
CF Stack:
- It contains all the services created by using that template.
- You can delete stack to delete all services of that template.
- AWs CLI can be used to create stack.
- Nested Stack: WE can create a stack within a stack.
- Like if you want some configuration in current stack from another stack, you can use another stack in current stack.
StackSets:
- StackSets in AWS CloudFormation is a feature that enables you to create, update, or delete stacks across multiple AWS accounts and regions with a single CloudFormation template. It allows you to manage infrastructure deployments consistently and efficiently across your organization.
- Stack Instance: refers to an individual deployment of a CloudFormation stack within a specific AWS account and region.
Mapping:
- It allows you to define a set of key-value pairs. It provides a way to associate values based on a specific key, and you can use mappings to customize the configuration of your CloudFormation stack based on different conditions or input parameters.
Mappings are typically used when you need to select different values based on a specific input or condition, such as region-specific values or environment-specific configurations.
- In code, next you can use function to find in map.
- IN code, you can use "parameters" to return the data related to it.
- Example: if we use"AWS::Stack Id"it will return current stack id.
Template(json or yaml any) Anatomy:
a. Format Version: "AWSTemplateFormatVersion" : "2010-09-09"
b. Description: One line description
c. Metadata: Data like name, more description basically.
d. Parameters: To pass user values.
e. Mapping: above given
f. Conditions: conditions under which resource should be created.
g. Transform: json to yaml or yaml to json.
h. Resources: services to launch
i. output: display output calues like public ip od instance after launch.
28. CloudFormation Best Practices:
- Use life cycle etc to know what services u want ins tack.
- Use IAM to give specific permissions to specific user to perform actions on stack.
- Use nested stack to reuse the common template patterns.
- Use dynamic reference to enter sensitive information.
- Use change sets before making changes.
- Update ec2 regularly.
29. SNS: Simple Notification Service
- Provides Push-based messaging: to one person
- PRovides many-to-many messaging: to multiple people.
- SNS Topics is where message to be forwarded to subscribers is given to sns.
- Then message filtering is done and respective subscriber gets meesage or notification.
- It is auto-scalable.
- SNS can be used to send notifications to mobile devices, email addresses, SQS queues, and other AWS services. For example, you could use SNS to send push notifications to mobile devices when a new order is placed in your e-commerce store.
30. SES:Simple EMAIL Service
- Used to send mail to multiple people.
- Used to help digital marketers.
- SES is used in conjunction with SNS topic.
- You can setup your own SMTP server.
- SES can be used to send marketing emails, transactional emails, and other types of email. For example, you could use SES to send order confirmation emails to customers who purchase products from your online store.
31. SQS: Simple Queue Service:
- It enables us to decouple and scale microservices, distributed systems and serverless applications.
- If multiple messages are to be sent to multiple consumer, they are kept in queue and all messages are sent.
- No overhead or any installing required to use this.
- Scales elastically.
- SQS can be used to store messages that need to be processed by other applications. For example, you could use SQS to store messages that contain new order information that needs to be processed by your order processing application.
#selfpaced
32. IAM
- Used to control access to AWS Resources using poilicies.
ARN (Amazon Resource Name):
- Used to distinguish and identify AWS services using ARN.
- ARN can be used to like if you want to allow access to certain resource to certain user you can use ARN in policies.
- No permissions are given by default in IAM.
- Access key and secret access key is given to access.
- Maximum 2 keys can be active under one user.
- Group is collection of users.
- Permission tot he group are applied to users under it.
- One user can be in multiple groups.
- Using credential Report you can get all details about users.
MFA:
- Virtual MFA Device: GoogleAuth. Authy etcetra.
- U2F security key
- Other MFA Device
JSON: (Java Script Object Notation)
- Always starts and ends with curly or flower bracket.
- Key value pair.
- EAR i.e Effect, Action and Resources are three must have components in JSON policy.
- Effect: allow/deny
Action: Open, create, delete etc.
Resource, s3, ec2 etc.
Principal: Entity like group can control resource
- Polict starts with version 2012-10-17.
- Sid is statement id in IAM policies.
- Condition can also be provided under which the policy will be applied.
Types of policies:
a. Aws managed
b. Custom Managed
c. Inline policies: These can be attached only to certain user, group etc.
IAM Policy Evaluation Logic:
Step 1. Deny all
Step 2. Check if any policies attached
Step 3. Check for explicit allow and deny permissions.
Step 4. Deny and Allow respective requests.
- IAM policy can also be attached to external user with complete different 12 digit account number.
- If account is created under organization, each member under that organization will be like root account with 12 digit account number.
- You can attach permissions to each member.
IAM Roles:
- It can be used to give temporary access to users or groups.
- It is given to resource to access other resources.
Cross-Account IAM roles:
- By using role, resource in account 1 can be accessed from account 2.
- AWS services can be accesses by third party Identity Providers (IDPs)
- Like login using Amazon, google, facebook etc.
- You have to assign principal in IAM policy with token and IDP domain name.
IAM Access Analyzer:
- It provides findings using mathematical logic and interference to determine possible access paths.
- I continuously checks policies.
- It is global service. Thus there is one Access analyzer for one root account and all users or groups under it.
Root User:
- should never be used for creating infra.
- Use MFA always.
- Root user has access key and secret access key aswell.
33. IAM Federations:
- Sharing access key and secret access key is not safe always.
- IAM federations allows to grant access to users who are not part of your AWS account.
- It activates trust between your account and external identity systems such as Active Directory, LDAP (Lightweight Directory Access Protocol) or
SAML (Security Assertion Markup Language)
1. Web identity federation: Login using role arn + auth token
2. SAML: Login using Role ARN + SAML Provider ARN
- Register AWS with corporate IDP (LDAP)
- This will generate metadata XML
- Create SAML identity provider
- Create roles
- SAML assertions should be mapped in roles.
"principal": {"arn": "ARN of SAML provide"r}
"action": {"sts": "assumerolewithsaml"}
STS: (Security Token Service)
- It gives temporary credentials.
34. Cloudwatch:
- It is a monitoring tool which can monitor all aws services and provide notification if anything goes wrong.
Dimensions and statistics:
- Various properties along with its vlaue that we compare are called as dimensions like cpu utilisation:40% etc.
- Statistics is data aggregation of those dimensions over a period of time.
Metrics and Namespaces:
- Metrics are different properties that we compare like cpu utilisation etcetra.
- Namespaces are namespace for set of metrics.
- Like in EC2 name space all ec2 metrics will be there, on EBS namespace EBS metrics so and so.
Example: Resource Metrics of Ec2:
- CPU Utilization first metric
- Under Network INterface:
NetworkIc metric
NetworkOut metric
NetworkPacketsIn metric
NetworkPacketsOut metric
- Under Attached disks:
DiskReadOps metric
DistWriteOps metric
DiskReadBytes metric
DiskWriteByte metric
-CPU Credits Metrics:(imp)
CPU credits are a measurement of the performance of burstable performance instances.
You will get two main metrics under this:
a. CPU Credit Usage
b. CPU Credit Balance
In burstable instances instances if are used above baseline capacity CPU credit can be used for some time.
If CPU credits balance is zero, your CPU will now run to baseline performance only, and may throttle if tried to pushed further.
We can get minimum, max and average values of these metrics.
35. CloudWatch Alarms:
- Alarms watchover metrics and metrics only. You cant group metrics and set alarm or anything like that. One metric one alarm thats it.
- Example: Restart ec2 when alarm for cpu utilization goes above 98%.
- Alarm states: Ok, Alarm, Insufficient_data.
36.
37. AWS Lambda: (PaaS)
- It is serverless.
- It triggers and runs code on in response to event occurunce.
- Only few programming languages can be used in code.
- You can only choose from few given OS and language Runtime.
- You can only run backend code, cant show frontend website using Lambda.
- It is a stateless system. (No need of custom storage)
Limitations of AWS Lambda:
- 512MB of diskspace provided only.
- RAM varies only from 128MB to around 3GB.
- Function timeout is 3sec by default and 900 sec i.e 15 minutes at max. That is your code should run in max 15 mins else it will abriptly start.
- Only given set of languages can be used. Python, Java, Go, Ruby etc.
Lambda Concepts
- Function: Lambda passes invocation to the function provided in the code.
The function processes the code and returns response then.
Different function settings:
a. Runtime: It executes our code. It sits between LAmbda sevice and our Code.
b. Handler: It is starting point to code and entrypoint to function. It is in this format: filename.functionName. Example: node.js.function1
When a trigger event occurs, Lambda uses the specified handler to start the execution of your function. It loads the code from the specified file and invokes the specified function.
c. Code: actual code
d, e , f: Tags, Description and Timeout.
- Lambda layers: Layers are especially useful when you have common code or libraries shared across multiple Lambda functions or when your function's deployment package becomes large due to the inclusion of dependencies. They provide a modular and efficient way to manage and reuse code in AWS Lambda.
Key features: Code Separation, Reusability, Versioning.
-5000 layers per region allowed and each function can have 75 layers max.
- Log Streams: Log Streams in AWS Lambda are primarily designed for capturing and storing log events generated by your Lambda functions. Log events contain information about the execution, errors, and other relevant details of your functions. You can search, analyze, and gain insights from these log events using CloudWatch Logs.
38. Common Use Cases of AWS Lambda:
- Serverless Websites: You can host your website on s3. You can use cloudfront for triggers. Cloudfront will trigger specific lambda function and serves the respective files from s3.
- Automated Backups: You can take backups according to our use case on triggers.
- Filter ,Transform and load Data: Event trigger source can be S3, Kinesis, Redshift, DynamoDB or other database services endpoints.
39. AWS Elastic Beanstalk:
- It is paas.
- Using this we can deploy our code without worrying about the underlying infrastructure with a URL.
- It will manage EC2, database to application monitoring load balances and autoscaling aswell.
- We upload coad file, libraries, environment and configurations related to code in zipped file.
- Application versioning backup is available.
- But we can only run 1 version at a time.
- While launching beanstalk we have to choose one of the environment from these:
a. Web Server: Directly uploading code for web server.
b. Worker Environment: Tasks like video processing, creating zip file.
- Using SQS service web server environment and worker environments are coupled.
- Say we have a website in which it compresses videos.
- Frontend part will be on web server.
- The processing part will be done in worker environment.
- After video for compressing is uploaded it will be in sqs queue and then to Sqsd daemon(ec2) and will be compressed one by one in worker environment.
- WE can change runtime anytime say from node.js to java we can do it.
- It can do in same envt or delete and deploy new envt.
- Saved configuration is like Template for Beanstalk.
40. AWS Opswork:
- It is configuration management service.
- We can choose between chef and puppet for tool.
- Can only use it on EC2.
- AWS Opswork for Puppet Enterprise:
It will create puppet master.It will serve as a central repository for puppet modules.
You have to create and connect ec2 nodes. Once connected, AWS looks after lifecycle of these nodes i.e provisioning, monitoring and termination.
- AWS Opswork for Puppet Enterprise:
It will create chef master. It will manage nodes and serve as a central repository for chef cookbooks.
- AWS Opswork Stack: ec2+RDS+ Load Balancer+Storage
Stack represents ec2 instances with say same AMI. That is they should be similar for same purpose.
- AWS Opswork Apps:
Application Definition: OpsWorks applications are defined within an
OpsWorks stack. Each application is associated with a specific repository (
Git, Subversion, or AWS CodeCommit) that contains the application code.
Deployment: OpsWorks supports multiple deployment strategies, including
time-based deployments, load-based deployments, or manual deployments
triggered by a user. When a deployment is triggered, OpsWorks retrieves the
latest code from the associated repository and deploys it to the specified
instances.
#selfpaced
41. Amazon FSx:
- Using FSx we can launch very high performance file system and avoiding tasks like managing hardware, configuring software and taking backups.
a. Amazon FSx for Windows file server:
- Shared file system compatible with windows devices.
- Only this file system can be used with Windows as EFS is compatible only with linux.
- FSx uses SSD for fast performance.
Features of FSx for windows:
- Windows compatibility for all os for and above windows 7.
- Broadly accessible: Can connect to local workspaces, EC2 and even VMware.
- Fully managed
- And SSD use.
Usecases:
- Lift and shift application workloads.
- Used in software development environment for multiple developers to share storage and work on same project.
How to access? Using DNS or Distributed File System NameSpaces.
Failover backup. If main goes down secondary is launched in another AZ.
b. Amazon Fsx for Lustre.
- Lustre is an open source, parallel file system which is mainly used for
HPC( High Performance Computing) environments.
- Amazon FSx for lustre makes it easy to launch this lustre file system used for HPC.
Features:
- Worlds most fast fs.
- Compatible with windows and linux.
- Easy integration with s3.
- WHen integrated with s3, all objects in s3 will be shown as files in lustre and changes made in s3 will reflect in lustre aswell.
Use case: Machine learning, Media Processing and Transcoding.
42. AWS Global Accelerator:
- If we have say multiple beanstalk environments in multiple regions, multiple load balancers in multiple regions, multiple ec2 in multiple regions or even all of them together.
- Global accelerator will provide static ip, fixed entrypoint to these multiple endpoints of services.
- It improves avalibility and performance of your system.
- It uses Edge Locations for this process to happen.
- If global accelerator is not used, to access something, it takes time to send request from local isp, then it has to go through multiple networks in between then to your server.
- With global accelerator, it uses edge location and your request is directly mapped to respective AWS network.
Benefits:
- High Availibility.
- Regional failover capability.
- Improved performance.
- Static ip address, so no variability when client cached your ip.
- Supports BYOIP(Bring your Own IP facility). This helps to use your on premices apps which has hard coded ip.
Use case:
- When app usage go up and down, ips will also go up and down. Global accelerator takes care of this scaling.
- You can use internal ALB or private ip as endpoints so improved protection.
Global Accelerator Components:
a. Static ip: 2 static ip provided by GA or you can go with BYOIP aswell.
b. Accelerator: Directs traffic to healthy endpoints only.
c. Network zone: Multiple networks interconnected.
d. Listener: Listens to port number. Like apache2 then give 80 port i.e listener.
e. Endpoint or Endpoint Group: Say multiple beanstalk environments in multiple regions, multiple load balancers in multiple regions, multiple ec2 in multiple regions or even all of them together.
- You will be taken from nearest edge location to nearest server. More like geolocation based routing.
- It is not a part of free tier.
- 20 Accelerators allowed in each account.
- 10 listener,10 endpoint in enpoint group, 10 ports allowed.
- 50 tags allowed
43. AWS Organizations:
- It is an tool used to manage multiple AWS accounts.
Features of AWS Organizations:
- Centralized Management.
- Consolidated billing.
- Control over members service usage and API using SCP.
- Integration with IAM. Member can access those service only which are allowed by both SCP and IAM policy.
- No chages for using Organizations.
Ways to access organization:
- Using console, CLI, SDK and also HTTPS query API.
quotas:
- 1 root only.
- 1000 OU's and 1000 SCPs allowed.
AWS Organization Concepts:
a. Root: under this it will have multiple OU's.
b. Organizational Unit (OU): Under this it will have multiple members account.Under OU we can have more OU aswell i.e nested OU.
c. Accounts: MAster and member account
d. Invitation: AWS account mail id notification is given
e. Handshakes: Accepting notification to join.
44. Elastic Netwok Interface:
- It is a logical networking component in VPC that represents a network interface card.
- When you move this ENI from one instance to another new instance, traffic will move to new instance.
- Usually when you create ec2 instance, one ENI comes attached to it.
When will you need manual ENI?
a. Multi-NIC Instances: ENIs allow EC2 instances to have multiple network interfaces attached to them. This is useful in scenarios where an instance needs to communicate with multiple subnets or security groups simultaneously.
b. High Availability and Failover:By attaching an ENI to a secondary EC2 instance, traffic can be seamlessly redirected to the standby instance in case of a failure or during maintenance activities, minimizing downtime.
c. Enhanced Networking Features: ENIs support enhanced networking features, such as Elastic Network Adapter (ENA) , which provide higher network performance and lower latency. These features are beneficial for data-intensive workloads or network-intensive applications.
45. Elastic Network Adapter:
- It provides enhanced networking capabilities.
- Like 100 Gbps speed.
46. Elastic Fabric Adapter:
- It gives reduced latency and HPC for say MAchine Learning Applications.
- You will need your own ENI first to atttch EFA.
- EFA is just ENA with added functionalities.
- It uses OS Bypass functionallity, This will allow ML apps to directly communicate with network interface and thus reduced latency.
Limitations:
- Only one EFA can be connected to one instance.
- Cannot forward EFA traffic to another subnet. EFA is effective when used in one subnet only.
- EFS OS-Bypass traffic cant be rounted, only IP traffic from EFA is routed.
- SG should have both Inbound and outbound rules allowing EFA.
In summary, ENA focuses on enhancing general networking capabilities for a wide range of workloads, while EFA is specialized for high-performance computing workloads that demand low-latency and high-bandwidth communication between instances.
Comments