AWS

This config type is used to scrape information about your AWS infrastructure.

Registry

The Mission Control Registry includes an AWS Helm chart that provides a pre-configured Scraper with common defaults

aws-scraper.yaml
apiVersion: configs.flanksource.com/v1
kind: ScrapeConfig
metadata:
  name: aws-scraper
spec:
  aws:
    - region:
        - eu-west-2
        - us-east-1
        - af-south-1
        - ap-south-1
        - eu-central-1
      properties:
        - name: AWS Link
          filter: 'config_type == AWS::IAM::Role'
          icon: aws-iam
          links:
            - text: AWS Link
              url: https://us-east-1.console.aws.amazon.com/iamv2/home#/roles/details/{{.name}}?section=permissions
      compliance: true
      patch_states: false
      trusted_advisor_check: false
      patch_details: false
      costReporting:
        s3BucketPath: s3://flanksource-cost-reports/query-results
        database: athenacurcfn_flanksource_report
        table: flanksource_report
        region: af-south-1
      inventory: true
      exclude:
        - Amazon EC2 Reserved Instances Optimization
        - Savings Plan
        # - trusted_advisor
        # - cloudtrail
      # include:
      #   - vpc
      #   # - subnet
      # - vpc
      # - SecurityGroup
      transform:
        relationship:
          # EKS Cluster to Kubernetes Cluster & Kubernetes Node
          - filter: config_type == 'AWS::EKS::Cluster'
            expr: |
              [
                {"type": "Kubernetes::Cluster","tags": {"account": tags['account'],"cluster": labels["alpha.eksctl.io/cluster-name"]}},
                {"type": "Kubernetes::Node","tags": {"account": tags['account'],"cluster": labels["alpha.eksctl.io/cluster-name"]}}
              ].toJSON()
          # EC2 Instance to kubernetes node
          - filter: config_type == 'AWS::EC2:Instance'
            expr: |
              [{"type": "Kubernetes::Node", "labels": {"alpha.eksctl.io/instance-id": config["instance_id"]}}].toJSON()
          # IAM Role to Kubernetes Node
          - filter: config_type == 'AWS::IAM::Role'
            expr: |
              [{"type": "Kubernetes::Node", "labels": {"aws/iam-role": config["Arn"]}}].toJSON()
          # AvailabilityZone to Zone ID & Kubernetes Node
          - filter: config_type == 'AWS::AvailabilityZone'
            expr: |
              [
                {"type": "Kubernetes::Node", "tags": {"account": labels['account'], "topology.kubernetes.io/zone": name}}
              ].toJSON()
          # Region to ZoneID
          - filter: config_type == 'AWS::Region'
            expr: |
              [{"type": "AWS::AvailabilityZoneID", "tags": {"region": name}}].toJSON()
        exclude:
          - jsonpath: $.tags
          - jsonpath: $.privateDnsNameOptionsOnLaunch
          # - jsonpath: availableIpAddressCount
          - jsonpath: outpostArn
          - jsonpath: mapCustomerOwnedIpOnLaunch
          - jsonpath: subnetArn
          # - jsonpath: usageOperationUpdateTime
          # - jsonpath: $..privateIPAddresses

Field	Description	Scheme
`schedule`	Specify the interval to scrape in cron format. Defaults to every 60 minutes.	Cron
`retention`	Settings for retaining changes, analysis and scraped items	`Retention`
`aws`	Specifies the list of AWS configurations to scrape.	`[]AWS`

AWS

Field	Description	Scheme
`accessKey`	Access Key ID	EnvVar
`cloudtrail`	Ingest cloudtrail events	`CloudTrail`
`compliance`	Toggle scraping of compliance metadata	`boolean`
`connection`	The connection url to use, mutually exclusive with `accessKey` and `secretKey`	Connection
`costReporting`	Enable cost and usage reporting	`CostReporting`
`endpoint`	Custom AWS Endpoint to use	`string`
`exclude`	AWS resources to exclude from scraping	`[]string`
`include`	AWS resources to include for scraping	`[]string`
`region`	The AWS region	`string`
`secretKey`	Secret Access Key	EnvVar
`skipTLSVerify`	Skip TLS verify when connecting to AWS	`boolean`
`labels`	Labels for each config item.	`map[string]string`
`properties`	Custom templatable properties for the scraped config items.	`[]ConfigProperty`
`tags`	Tags for each config item. Max allowed: 5	`[]ConfigTag`
`transform`	Transform configs after they've been scraped	`Transform`

CloudTrail

Field	Description	Scheme
`exclude`	Set events to be excluded from scraping	`[]string`
`maxAge`	Set maximum age of events for scraping, Defaults to `7d`	Duration

Cost Reporting

Field	Description	Scheme
`s3BucketPath`	Set path for S3 bucket to scrape published AWS billing reports	`string`
`table`	Specify table containing cost and usage data	`string`
`database`	Specify database containing cost and usage data	`string`
`region`	Specify region for S3 bucket	`string`

Trusted Advisor

The scraper integrates with AWS Trusted Advisor to collect real-time guidance across cost optimization, performance, fault tolerance, security, and service limits to help optimize your AWS infrastructure.

Rate Limitations

Trusted Advisor checks run with a minimum interval of 16 hours (due to API rate limitations). This can be overridden by setting the property scraper.aws.trusted_advisor.minInterval.

To disable Trusted Advisor altogether, you can add an exclusion rule:

exclude:
  - trusted_advisor

Supported Resources

Resource Type	AWS Type	Config Class	Description
Account	AWS::IAM::Account	Account	AWS Account information
CloudFormationStack	AWS::CloudFormation::Stack	Stack	CloudFormation stacks
DHCPOptions	AWS::EC2::DHCPOptions	DHCP	DHCP Options Sets
DNSZone	AWS::Route53::HostedZone	DNSZone	Route53 Hosted Zones
EBSVolume	AWS::EBS::Volume	DiskStorage	Elastic Block Store Volumes
EC2Instance	AWS::EC2::Instance	VirtualMachine	EC2 Instances
ECRRepository	AWS::ECR::Repository	ContainerRegistry	Elastic Container Registry Repositories
ECSCluster	AWS::ECS::Cluster	ECSCluster	ECS Clusters
ECSService	AWS::ECS::Service	ECSService	ECS Services
ECSTask	AWS::ECS::Task	ECSTask	ECS Tasks
ECSTaskDefinition	AWS::ECS::TaskDefinition	ECSTaskDefinition	ECS Task Definitions
EFSFileSystem	AWS::EFS::FileSystem	FileSystem	Elastic File System
EKSCluster	AWS::EKS::Cluster	KubernetesCluster	Elastic Kubernetes Service Clusters
ElastiCache	AWS::ElastiCache::CacheCluster	Cache	ElastiCache Clusters
FargateProfile	AWS::EKS::FargateProfile	FargateProfile	EKS Fargate Profiles
IAMInstanceProfile	AWS::IAM::InstanceProfile	Profile	IAM Instance Profiles
IAMRole	AWS::IAM::Role	Role	IAM Roles
IAMUser	AWS::IAM::User	User	IAM Users
LambdaFunction	AWS::Lambda::Function	Lambda	Lambda Functions
LoadBalancer	AWS::ElasticLoadBalancing::LoadBalancer	LoadBalancer	Classic Load Balancers
LoadBalancerV2	AWS::ElasticLoadBalancingV2::LoadBalancer	LoadBalancer	Application/Network Load Balancers
RDSInstance	AWS::RDS::DBInstance	RelationalDatabase	RDS Database Instances
RouteTable	AWS::EC2::RouteTable	Route	VPC Route Tables
S3Bucket	AWS::S3::Bucket	ObjectStorage	S3 Buckets
SecurityGroup	AWS::EC2::SecurityGroup	SecurityGroup	Security Groups
SNSTopic	AWS::SNS::Topic	Topic	Simple Notification Service Topics
SQSQueue	AWS::SQS::Queue	Queue	Simple Queue Service Queues
Subnet	AWS::EC2::Subnet	Subnet	VPC Subnets
VPC	AWS::EC2::VPC	VPC	Virtual Private Clouds

AWS​

CloudTrail​

Cost Reporting​

Trusted Advisor​

Supported Resources​

AWS

CloudTrail

Cost Reporting

Trusted Advisor

Supported Resources