As we mentioned earlier in this chapter, CloudTrail doesn't log everything, including many services that are completely unsupported. Again, that list of unsupported services can be found here: https://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-unsupported-aws-services.html. What this means is that our API calls to these services will not get logged anywhere by CloudTrail (including Event history!). Some of these services can prove to be very lucrative for us as attackers, so if you compromise a user and find that they have access to any of those services, they are worth checking out because you can stay under the radar and still benefit greatly. Another big point about unsupported CloudTrail services is that that means you can't create CloudWatch Events rules for those API actions, which means you can't instantly respond to events happening in those services.
As an attacker, if we are looking for compute resources, we can abuse a few different unlogged services. At the time of writing, AppStream 2.0, Amplify, and Cloud9 all provide us with access to managed EC2 servers in one way or another. This means we can spin up servers and interact with them without ever getting logged.
As a defender, it is important to ensure that no users have access to these services unless necessary. If it is required to provide access to any of the unlogged services, then utilize any built-in logging the service may provide and make use of some of the other features that IAM provides to monitor this access. If you download an IAM credential report, you can see if a service was accessed recently by looking in the access_key_1_last_used_service and access_key_2_last_used_service columns, where those unlogged services will still show up. To get an IAM credential report, you can run the following command:
aws iam get-credential-report
Another option is to use the IAM GenerateServiceLastAccessedDetails and GetServiceLastAccessDetails APIs to determine when/if a user accessed a certain service, including the services that aren't logged by CloudTrail. To do this, we can first run the generate command to generate the report:
aws iam generate-service-last-accessed-details --arn arn:aws:iam::000000000000:user/TestUser
The value for the ARN argument must be the ARN of an IAM resource, including users, groups, roles, and managed policies. This API command should return a JobId back to you. Then we can get the report by using that ID:
aws iam get-service-last-accessed-details --job-id frt7ll81-9002-4371-0829-35t1927k30w2
The response from that command will include information about whether a resource has authenticated to a certain service and when that last authentication took place. These APIs won't tell you exactly what kind of activity is going on, but you can at least check to see who is trying to access those services.
These APIs also help detect the use of unlogged CloudTrail services for account enumeration. The Wired company released an article on research from Rhino Security Labs that entailed a method that essentially allows an attacker with keys to gather a small amount of AWS account information without getting logged by CloudTrail (https://www.wired.com/story/aws-honeytoken-hackers-avoid/). The reason this research is so important is because there are many canary token services that rely on CloudTrail to alert when keys have been compromised. Canary tokens are typically placed somewhere in an environment and are rigged to set off an alarm when used, which would indicate an attacker is in the environment and found those tokens. For AWS, canary token providers typically rely on CloudTrail for these alarms, but Rhino Security Labs showed that it was possible to bypass these alarms and determine whether AWS keys were canary tokens or not while staying under the radar.
At the time, it was found that some of the most popular canary token providers for AWS used a single account to generate these keys or would include identifying information in the user that indicated they were being used as a canary token. This information could then be exposed through verbose error messages returned from unsupported CloudTrail services, thus allowing the attacker to identify if AWS keys are canary tokens based on the account ID or user name/path without ever triggering the alarm the keys were meant to trigger. One project that was vulnerable to this attack was SpaceCrab by Atlassian.
Originally, the default SpaceCrab setting would set a path for the IAM user it created with /SpaceCrab/ as the value. An attacker could then run an AWS CLI command against an unsupported CloudTrail service, where the user's ARN would get disclosed in an error message. The ARN includes the user's path, so it was clear that the keys were canary tokens created by SpaceCrab. The following is an example error message returned when running the AppStream DescribeFleets command:
The issue was reported to Atlassian and the vulnerability was fixed. The issue was also reported to AWS themselves, but it was rejected because they don't consider an ARN to be sensitive information. This is correct, but a user should not be able to fetch that information without generating any logs.
AWS Amplify is another newer service that is unsupported in CloudTrail and it outputs similar verbose error messages. The following message was returned when trying to run the ListApps command without the right permissions:
An error occurred (AccessDeniedException) when calling the ListApps operation: User: arn:aws:iam::000000000000:user/TestUser is not authorized to perform: amplify:ListApps on resource: arn:aws:amplify:us-west-2:000000000000:apps/*
This small attack is essentially timeless if the AWS service output error messages like that and if there are services that CloudTrail doesn't support. This same attack will likely work for any new service that gets released and isn't logged.
Even this small amount of information can be helpful to an attacker, because they can then use other non-logged attack vectors, such as cross-account IAM user/role enumeration, to gather more information (https://rhinosecuritylabs.com/aws/aws-iam-user-enumeration/).