AWS in China

AWS has a ‘partition’ in Mainland China.
It’s not like building in normal AWS, some services/features are missing or work in odd ways.

Partitions

AWS has a concept of ‘partitions’, these are basically separate instances of the AWS Cloud.
You may have heard this term before when it comes to the different parts of the ARN e.g.“arn:partition:service:region:account-id:resource-id”
There are currently 3 partitions:

aws The normal aws cloud
aws-us-gov AKA GovCloud
aws-cn Mainland China

These different partitions have limited overlap and connectivity.

3 examples of this are Console access, cross-account roles, and S3 bucket names:

When you sign in to the AWS console, you might note that Beijing and Ningxia (the current China regions) are not accessible.
This is because you need to sign in to a separate China version of AWS (signin.amazonaws.cn) which can access only those regions and none of the normal regions.
You can’t grant access from an IAM principle (e.g. user) from one partition to assume a role in another partition. When you create the role’s trust policy, the principle will be invalid and you will get an error.
This means that for SaaS tools which you grant access to your account via a role will also need a China version. If their AWS account is only from a global partition then you will not be able to grant them any access.
You might have been told that S3 bucket names come from a global address space, this is not entirely true.
In fact, S3 bucket names need only be unique within the AWS partition.
This can be understood that the S3 buckets ARN needs to be unique, not the actual name.
Given different AWS partitions give different ARNs (as the partition name is a part of the ARN) thus you can have the exact same bucket name in China and in a non-china region.

When using IAC tools this can be annoying as hardcoded ARNs would typically use the global partition and you would either need to maintain a China version of the template / module / etc. or (a better way) use some dynamic variable and use string interpolation.
Cloudformation has the variable AWS::Partition
Terraform has the data source aws_partition

The AWS China regions are not wholly operated and run by AWS like AWS’ other regions (eg ap-southeast-2). Beijing Sinnet Technology Co., Ltd.(“Sinnet”) operates and provides services from the Amazon Web Services China (Beijing) Region, and Ningxia Western Cloud Data Technology Co., Ltd. (“NWCD”) operates and provides services from the Amazon Web Services China (Ningxia) Region.

Account Creation Process

The process for creating AWS accounts is not the same as for the global partitions.
You need to go through a registration process in which your details are provided to the Chinese government.
This process can take some time and upon success, you are NOT given details of the root user.
In fact, the AWS documentation claims there are no root users in the China partition.
(This does not seem entirely true though. Running IAM commands can show that there does seem to be a root user which has Access Keys, but you are not provided those keys…)

The lack of a root user means that some processes for managing the account need different methods.
For example, to change the root email address you need to go to the “My Security Credentials” page (see the role switcher dialogue). The link to which is only visible when you are logged in as a user, if you have switched roles the link disappears and the page redirects making it inaccessible.

Thankfully further accounts are easy to create when using AWS Organizations.
Although an email is sent to the email used to create this new account claiming that that additional information is required to complete set up (“business registration certificate and Information Security Administrator’s personal ID”). It seems that this email can be ignored however as everything seems to work regardless.
Again, you are not given root credentials, thus to access the new account, you need to assume the role OrganizationAccountAccessRole from the Organization’s root account.

Initially, any VPC in the account can only be accessed from the internet on ports 22 (ssh) and 3389 (rdp).
You are able to have AWS open ports 80, 8080, and 443 by filing for an ICP Recordal in which you register details with the Chinese government. AWS will first verify the given information and then pass it on to the Chinese government for approval; the government takes between 3 and 20 working days.

Missing Services or Functionality

It is typical for some regions to not support a new AWS service right away.
The China regions are similar, with some services missing, only working in one of the China regions, or are missing functionality that is taken for granted in other regions.
As a result, it can feel like working on AWS from a few years ago, requiring the use of older services or self-managed solutions. Even the console is from the previous generation.
An official list of supported services can be found on this page: AWS China Service availability matrix

One annoying aspect of this is that the documentation does not mention that something is not supported, instead, it just won’t mention that service or functionality, like it doesn’t exist at all.

Here is a list of some troubles faced when I was developing in AWS China (obviously not comprehensive):

AWS SSO is not supported.
- We considered AWS AD Connector, which is supported, but not for access to the console.
- The old method of using a SAML idP to a dedicated authentication account and role switching from there does work though
CloudTrail is supported but not Organization Trails
Config works but not Region or Account Aggregation
SCPs are not supported
- My theory is that because the root account is owned by someone else (government?) and SCPs can restrict the root user’s permissions, ’they’ don’t want the access of this other party to be restricted thus SCPs are not supported.
Route 53 hosted zones are only supported in Ningxia and not Beijing.
destination options for VPC Flow Logs are not supported
KMS has no multi-region keys

Great Firewall

I won’t go deep into how the Chinese Firewall functions, but I will say that it undoubtedly cause some issues for us.

Latency is high when connecting through it, for example when we were running kubectl commands from our local machines (we are in Australia) it would take many seconds for a response.

Direct Connect and VPNs

We set up a Direct Connect between 2 AWS accounts using a Zenlayer POP (point of presence) which resells Direct Connects on either side of the POP.
We then ran a VPN over it.

Getting this set up took ages, we initially got a trial, but that expired and when we went to set it up again, apparently a new policy was in place that meant no more trials, and getting a new and proper link was a nightmare.

Something to note is that VPNs going through the Chinese Firewall are not banned outright, they just need to be by a Chinese Government-approved vendor.
The reason for this is clear when you see a small note in the Microsoft Azure docs which states that when using a VPN, you should not assume that the Chinese Government is not able to intercept the traffic.

Publicly Accessible Endpoints (e.g. ALB)

I previously mentioned that ports 80 and 443 are not open for your VPC unless you have an approved ICP license.
AWS Support and documentation will claim that this is a blanket statement but after some experimentation, we found that an ALB could be accessible for a short time after creation.
I can’t comment on how it’s actually enforced: if there is some random amount of time for it to propagate to the firewall to block the new IP, after a certain number of requests, or a certain rate is reached.
Also ICP licenses do not seem to automatically propagate from an AWS Organization root account to the member accounts. We needed AWS Support intervention for that, thankfully we had a TAM.

Terraform Cloud - Timeouts

Deployments via TFC will occasionally timeout or sit indefinitely, this appears to be due to the initialisation of the Terraform AWS provider and the regional China API endpoint. In this circumstance, you should cancel the run and wait approx. 5 minutes, if it does not work you can force abort the run.

------------ Terraform Cloud System Message ------------

Cancel signal received. Attempting graceful shutdown.

--------------------------------------------------------
{"@level":"info","@message":"\nInterrupt received.\nPlease wait for Terraform to exit or data loss may occur.\nGracefully shutting down...\n","@module":"terraform.ui","@timestamp":"2021-11-05T01:38:32.657788Z","type":"log"}
Stopping operation...

------------ Terraform Cloud System Message ------------

Force cancel signal received. Terraform will now abort.

--------------------------------------------------------

Then trigger a new run from the UI (you don’t need to push a change to the git repo)

DNS Timeouts or missing results

When administrating an EKS cluster from my local machine sometimes the DNS of my ISP would timeout or fail to return an A record for the EKS admin endpoint, but other DNS providers might work fine.

Other quirks

Clearing findings in Security Hub from the delegated admin account does not seem to propagate the changes to the member accounts. The findings need to be closed from the same account the finding is for.
Most of the documentation is in Chinese.
- Thankfully it seems that it is just the English docs that have been updated for the China partition and then run through translation software, thus Google Chrome’s ’translate page’ feature does a really good job of translating it back.
There was no cost calculator until halfway through the build.
There is no free tier usage in Beijing region

Terraform Cloud - Helm Provider

For helm to work with our EKS cluster we need to provide it the ability to authenticate against it. This is done via variables that are passed to the helm provider (How to do that is out of scope for this article).

The official documentation recommends that to ensure the Kubernetes provider is receiving valid credentials, an exec-based plugin can be used to fetch a new token before initialising the provider. In China we found that the region parameter and relevant argument is also required to be specified (though this is apparently not required for the global AWS partition) - below is the helm provider configuration that we used to deploy helm charts via tf cloud:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
provider "helm" {
  debug = true
  kubernetes {
    host                   = data.aws_eks_cluster.eks.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks.certificate_authority[0].data)
    exec {
      api_version = "client.authentication.k8s.io/v1alpha1"
      args        = ["eks", "get-token", "--cluster-name", local.eks_cluster_name, "--region", local.aws_region]
      command     = "aws"
    }
  }
}

Additional Reading

AWS China Service availability matrix
AWS China Services List
AWS China FAQs
AWS China developer documentation

Searching for documentation

If you are unsure where or if documentation exists specifically to china you can use the following search modifiers in google (dropping the cn suffix will return aws global documentation):
inurl:docs.amazonaws.cn <your-search-terms>