Troubleshooting Monitoring and Alerting

  • Scenario: IAM user cannot access CloudWatch Dashboard
    • Does the IAM user have the correct permrissions to allow them to read the CloudWatch dashboard?
  • Scenario: EC2 instance is not sending logs to CloudWatch
    • Is the CloudWatch agent installed?
    • Is the CloudWatch agent actually running?
    • Does the instance role have permission to write to CloudWatch?
  • Scenario: A user is crreating unauthorized EC2 instances. The Lambda function that is triggered by CloudWatch alarms to terminate the instance isn’t working
    • Does CloudWatch Events have permission to trigger the Lambda?
    • Does the execution role of the Lambda have access to terminate EC2 instances?

Troubleshooting Logging

  • Scenario: A user is generating activity via the APIs, but no logs are appearing in CloudTrail for these interactions
    • Check that CloudTrail is actually enabled
    • Have you provided the correct S3 bucket name for the trail logs?
    • Is the BucketPolicy correct?
    • S3 and Lambda Data Events are high volume so they are not enabled by default. They also incur added costs. KNOW THIS!!
  • Scenario: You need to provide access to your CloudTrail logs (like an auditor) but they aren’t able to read the logs
    • Does the auditor’s account have read access to the logs?
    • AWSCloudTrailReadOnlyAccess policy provides this

Troubleshooting Secure Network Infrastructure

  • Scenario: You have 2 subnets, one public and one private. You have an Internet Gateway and a Virtual Private Gateway for VPN Access. Traffic is not flowing correctly.
    • Check routing tables, security groups and NACLs
    • Internet Access - NAT Gateways and Internet Gateways
    • Check VPC Flow Logs to allow/deny messages
  • Remember NACLs are stateless, you need to configure both incoming and outgoing rules
  • Security Groups deny by default, use NACLs for explicit denys
  • If you are peering two VPCs, remember to configure this on the routing tables in both VPCs

Troubleshooting Authentication & Authorization

  • Common issues with Conflicting Policies
    • Least Privilege - all actions are denied by default
    • Explicit deny will always override an allow
    • With multiple policies in play, an action will only be allowed if no method explicitely denies and there is at least one method that explicitly allows access
    • If you are using AWS Organizations, check if there is a permission boundary preventing the action
  • Troubleshooting Identity Federation
    • Use the correct API for the Job
      • Authicate by a web identity provider like Facebook: Use sts:AssumeRoleWithWebIdentity call
      • Authenticate via SAML: use sts:AssumeRoleWithSAML
      • Authenticated by AWS: use sts:AssumeRole

Troubleshooting Cross Account Access

  • Scenario: A production account has an S3 account with a number of objects in it. We want to give a developmentt account read only access to data in production to test against.
    • Check that the dev account has permission to call sts:AssumeRole in the prod account.
    • Check that the dev account is trusted and has permission to perform the action you are attempting
  • Scenario: A production account has a KMS key that we want to share with another account.
    • The Key Policy needs to trust the external account
    • The external account needs an IAM Policy allowing users to run specific API calls for the resource (in this case the CMK)

Troubleshooting Lambda Access

  • Scenario: A Lambda function needs to be able to push logs to CloudWatch logs
    • Lambda will require permission to write to CloudWatch in its execution role
    • Do not confuse with the Function Policy, which identifies which services can execute the function
  • Scenario: A Lambda needs to connect to an RDS, whose secrets are in Secrets Manager
    • Lambda needs to be able to query the secrets from SecretsManager
  • Scenario: A Lambda needs to perform an action based upon a CloudWatch Event
    • CloudWatch needs to be configured as allowed to trigger the function using the Function Policy

Troubleshooting Access to KMS/CMKs

  • Scenario: You’re having trouble accessing a CMK
    • Check that the IAM user, group or role has permissions for the action they are attempting in KMS
    • There is also a Key Policy in place for CMKs that identifies who can actually use/manage the key itself