Let’s Config
- Aug 7, 2022
- 4 min read
Keeping track of resource configuration is essential when it comes to managing them in Cloud infrastructure. AWS Config is a service offered by AWS through which we can keep track of all the configurations of resources. Config enables us to examine changes in settings and interactions between AWS resources, review in-depth resource configuration histories, and evaluate our overall compliance with the configurations described in our internal rules.
The benefits of Config includes
Continuous monitoring
Continuous Assessment
Change management
Operational Troubleshooting
Enterprise-wide compliance monitoring
Recently, I came across a task to get the details of all the instances in a specific region along with its details. If we are familiar with AWS CLI, we know that describe-instances may serve this purpose. We know describe-instances is a service-specific API call.
aws ec2 describe-instances --instance-ids i-0d5e515bub841c9e2 This will be the output
{
"Reservations": [
{
"Groups": [],
"Instances": [
{
"AmiLaunchIndex": 0,
"ImageId": "ami-052cef05020f6d",
"InstanceId": "i-0d5e615bab341c9e2",
"InstanceType": "t3.nano",
"LaunchTime": "2022-08-01T05:02:05+00:00",
"Monitoring": {
"State": "disabled"
},
"Placement": {
"AvailabilityZone": "ap-south-1a",
"GroupName": "",
"Tenancy": "default"
},
"PrivateDnsName": "ip-10-1-1-85.ap-south-1.compute.internal",
"PrivateIpAddress": "10.1.0.65",
"ProductCodes": [],
"PublicDnsName": "",
"State": {
"Code": 80,
"Name": "stopped"
},
"StateTransitionReason": "User initiated (2022-08-03 06:55:11 GMT)",
"SubnetId": "subnet-045a7b3eb59f4e2e4",
"VpcId": "vpc-0b949f0d2afc1c626",
"Architecture": "x86_64",
"BlockDeviceMappings": [
{
"DeviceName": "/dev/xvda",
"Ebs": {
"AttachTime": "2022-01-09T17:21:40+00:00",
"DeleteOnTermination": true,
"Status": "attached",
"VolumeId": "vol-0ad615127c4552a43"
}
}
],
"ClientToken": "VFT-R-NatIn-12XPK2PWQWEVT",
"EbsOptimized": false,
"EnaSupport": true,
"Hypervisor": "xen",
"IamInstanceProfile": {
"Arn": "arn:aws:iam::585716541037:instance-profile/VInstanceProfile-3SO1512FME38",
"Id": "AIPAYQX3NEZW3SHXqwePNN"
},
"NetworkInterfaces": [
{
"Attachment": {
"AttachTime": "2022-01-09T17:21:40+00:00",
"AttachmentId": "eni-attach-00935245ud64e6bef",
"DeleteOnTermination": true,
"DeviceIndex": 0,
"Status": "attached",
"NetworkCardIndex": 0
},
"Description": "",
"Groups": [
{
"GroupName": "InstanceSecurityGroup",
"GroupId": "sg-0ty9efd3a6141coff5"
}
],
"Ipv6Addresses": [],
"MacAddress": "11:45:98:8f:4b:ca",
"NetworkInterfaceId": "eni-04w4bad14dcd79240",
"OwnerId": "58571613037",
"PrivateDnsName": "ip-30-1-0-65.ap-south-1.compute.internal",
"PrivateIpAddress": "11.1.0.65",
"PrivateIpAddresses": [
{
"Primary": true,
"PrivateDnsName": "ip-11-1-0-65.ap-south-1.compute.internal",
"PrivateIpAddress": "10.1.0.85"
}
],
"SourceDestCheck": false,
"Status": "in-use",
"SubnetId": "subnet-0dea7b3eb59f452e4",
"VpcId": "vpc-0b949f0juifc1c626",
"InterfaceType": "interface"
}
],
"RootDeviceName": "/dev/xvda",
"RootDeviceType": "ebs",
"SecurityGroups": [
{
"GroupName": "InstanceSecurityGroup",
"GroupId": "sg-0ac9efd3a6141cff5"
}
],
"SourceDestCheck": false,
"StateReason": {
"Code": "Client.UserInitiatedShutdown",
"Message": "Client.UserInitiatedShutdown: User initiated shutdown"
},
"Tags": [
{
"Key": "aws:cloudformation:logical-id",
"Value": "Instance1"
},
{
"Key": "Name",
"Value": "Instance1"
}
],
"VirtualizationType": "hvm",
"CpuOptions": {
"CoreCount": 1,
"ThreadsPerCore": 2
},
"CapacityReservationSpecification": {
"CapacityReservationPreference": "open"
},
"HibernationOptions": {
"Configured": false
},
"MetadataOptions": {
"State": "applied",
"HttpTokens": "required",
"HttpPutResponseHopLimit": 1,
"HttpEndpoint": "enabled",
"HttpProtocolIpv6": "disabled",
"InstanceMetadataTags": "disabled"
},
"EnclaveOptions": {
"Enabled": false
},
"PlatformDetails": "Linux/UNIX",
"UsageOperation": "RunInstances",
"UsageOperationUpdateTime": "2022-01-09T17:21:40+00:00",
"PrivateDnsNameOptions": {
"HostnameType": "ip-name",
"EnableResourceNameDnsARecord": false,
"EnableResourceNameDnsAAAARecord": false
},
"MaintenanceOptions": {
"AutoRecovery": "default"
}
}
],
"OwnerId": "5857141037",
"RequesterId": "746678304693",
"ReservationId": "r-02a41acrtyuh2b6"
}
]
}I'm not going to use a service-specific API call. What can be my other options?
What if we can create our own customized version of describe-instances? It's not exactly the use case.
AWS Config has a way of leveraging the concept of querying. This means we can leverage the SQL query for querying the current configuration of a resource.
Advanced Query in Config
AWS Config uses SQL) SELECT syntax to perform property-based queries and aggregations on the current configuration item (CI) data.
Without specifying service-specific API calls we can use Advanced Query to query the current metadata state of the resources. We can also use the same queries in cross regions managed from a central account.
The complexity of the queries ranges from simple matches against tag and/or resource identifiers to more intricate ones, such as displaying all S3 buckets with versioning turned off.
A sample query components
SELECT property [, ...]
[ WHERE condition ]
[ GROUP BY property ]
[ ORDER BY property [ ASC | DESC ] [, property [ ASC | DESC ] ...] ]Consider an example
aws configservice select-resource-config \
--expression "SELECT * WHERE relationships.resourceId= 'i-0d5e615bab1c9e2" \
--output yaml \
--profile coderman
This gives the relationship of the specified resource. With other services/resources.
Since the query specifies * (as in SQL) the query will return all resources that are related to the above EC2 Instance.
Such as the
AWS::EC2::NetworkInterface: which the above instance is connected with.
AWS::CloudFormation::Stack : The stack which creates the Instance.
AWS::EC2::RouteTable : the attached route table (if any)
AWS::EC2::SecurityGroup :attached security group.
AWS::EC2::Subnet : the subnet in which it is launched
AWS::EC2::Volume : the volume its attached with.
AWS::EC2::VPC : the VPC in which the EC2 resides.
And more if exists..
Likewise we can also specify the field that we want to retrieve
aws configservice select-resource-config \
--expression "SELECT resourceId,resourceName,resourceType,configuration.instanceType,configuration.SubnetId,Tags,availabilityZone WHERE relationships.resourceId = 'i-0d5e615bab84e2'"This query will give
resourceId
resourceName
resourceType
configuration.instanceType
configuration.SubnetId
Tags
availabilityZone
Of the given instance.
Now Lets find out how I came across a solution.
Q. We need to retrieve all the instance which is launched in a specific region. In this case ap-south-1. But we need the (vpc id, subnet id, instancetype and image id)
This query can serve the purpose
aws configservice select-resource-config \
--expression "SELECT
configuration.imageId,
configuration.instanceId,
configuration.subnetId,
configuration.vpcId
WHERE
resourceType IN ('AWS::EC2::Instance') AND awsRegion='ap-south-1'" \
--output yamlPretty cool Right!!
This can be the configservice version of describe-instances.
aws configservice select-resource-config \
--expression "SELECT
resourceId,
awsRegion,
availabilityZone,
arn,
configuration.amiLaunchIndex,
configuration.imageId,
configuration.instanceId,
configuration.instanceType,
configuration.keyName,
configuration.launchTime,
configuration.monitoring.state,
configuration.placement,
configuration.publicDnsName,
configuration.privateDnsName,
configuration.publicIpAddress,
configuration.privateIpAddress,
configuration.state,
configuration.productCodes,
configuration.stateTransitionReason,
configuration.subnetId,
configuration.vpcId,
configuration.architecture,
configuration.blockDeviceMappings,
configuration.clientToken,
configuration.ebsOptimized,
configuration.enaSupport,
configuration.hypervisor,
configuration.networkInterfaces,
configuration.rootDeviceName,
configuration.rootDeviceType,
configuration.securityGroups,
configuration.sourceDestCheck,
configuration.stateReason,
configuration.virtualizationType,
configuration.cpuOptions,
configuration.capacityReservationSpecification,
configuration.hibernationOptions,
configuration.metadataOptions,
configuration.enclaveOptions,
configuration.platformDetails,
configuration.usageOperation,
configuration.usageOperationUpdateTime,
configuration.privateDnsNameOptions,
configuration.maintenanceOptions,
tags
WHERE
resourceType = 'AWS::EC2::Instance'"
Feel free to test by adding or removing fields..
Limitations
Even though config is so much helpful in leveraging SQL concepts. It has limitations of its own
Advanced Query does not support ALL, AS, DISTINCT, FROM, HAVING, JOIN, and UNION keywords in a query.
You can refer to more limitations and information at
Query in Azure
Using the concept of querying made me wonder does Azure have the same servic.
Azure has a service called Azure Resource Graph to leverage querying to analyse our cloud environment in azure platform.
We can leverage Advanced Resource Graph query to perform action according to our use cases.
az graph query -q "Resources | where type =~ 'Microsoft.Compute/virtualMachines'"Returns the resources of type virtual machines (same as EC2).
On a final note
An advanced query is just a part of Config. We cant define AWS Config solely based on advanced query alone. We can do continuous monitoring and record configuration changes of resources, we can enforce compliance by creating Config rules, We will be able to track the relationships with other resources before making any changes that will affect those resources, and so on. As I always mention, it's just the tip of the iceberg. You can do wonders if you deep dive into this concept.
Comments