v8.5.7: update BR gcp_v2 storage (#23122)

br/backup-and-restore-storages.md

@@ -138,11 +138,33 @@ It is recommended that you configure access to S3 using either of the following
 </div>
 <div label="GCS" value="gcs">
 
-You can configure the account used to access GCS by specifying the access key. If you specify the `credentials-file` parameter, the authentication is performed using the specified `credentials-file`. Besides specifying the key in the URI, the following methods are also supported:
+You can configure the credentials used to access GCS in the following ways:
 
-- BR reads the file in the path specified by the environment variable `$GOOGLE_APPLICATION_CREDENTIALS`
-- BR reads the file `~/.config/gcloud/application_default_credentials.json`.
-- BR obtains the credentials from the metadata server when the cluster is running in GCE or GAE.
+- Method 1: Explicitly specify `credentials-file`
+
+    - If `credentials-file` points to a Service Account JSON file, BR and TiKV access GCS using this credential.
+    - If TiKV uses the `gcp_v2` external storage backend, `credentials-file` can also point to the `external_account` JSON used by Google Cloud WIF.
+
+- Method 2: Use Application Default Credentials (ADC)
+
+    - BR reads the file in the path specified by the environment variable `$GOOGLE_APPLICATION_CREDENTIALS`.
+    - BR reads the file `~/.config/gcloud/application_default_credentials.json`.
+    - When BR runs in GCE or GAE, it uses the credentials obtained from the metadata server.
+
+If you want TiKV to use GCS WIF or ADC, you need to enable the `gcp_v2` external storage backend. **Starting from v8.5.7, TiKV enables the `gcp_v2` external storage backend by default.** You can configure `gcp_v2` in the following ways:
+
+- full backup and restore: set `[backup].gcp-v2-enable` to `true` in [TiKV Configuration File Descriptions](/tikv-configuration-file.md)
+- log backup: set `[log-backup].gcp-v2-enable` to `true` in [TiKV Configuration File Descriptions](/tikv-configuration-file.md)
+
+The default values of the preceding two configuration items are both `true`. If you disable `gcp_v2`, TiKV continues to use the legacy GCS implementation. This implementation supports only Service Account JSON and does not support using WIF directly.
+
+> **Note:**
+>
+> The GCS JSON credentials explicitly passed to `gcp_v2` support only the `service_account` and `external_account` types. If you are using the `authorized_user` JSON generated by ADC and need TiKV to access GCS directly, it is recommended to set `--send-credentials-to-tikv=false` and configure ADC on each TiKV node. Otherwise, BR might send the `authorized_user` JSON to TiKV as an explicit credential, but `gcp_v2` does not accept this type of explicit JSON.
+
+> **Tip:**
+>
+> After `gcp_v2` is enabled, if GCS JSON credentials are not explicitly provided on the TiKV side, TiKV uses the Google Default Credentials flow. Therefore, when using ADC and setting `--send-credentials-to-tikv=false`, make sure that each TiKV node itself has a usable Google credentials environment.
 
 </div>
 <div label="Azure Blob Storage" value="azure">

encryption-at-rest.md

@@ -171,7 +171,23 @@ credential-file-path = "/path/to/credential.json"
 ```
 
 - `key-id` specifies the key ID of the KMS CMK.
-- `credential-file-path` specifies the path of the authentication credentials file, which currently supports two types of credentials: Service Account and Authentication User. If the TiKV environment is already configured with [application default credentials](https://cloud.google.com/docs/authentication/application-default-credentials), there is no need to configure `credential-file-path`.
+- When `vendor = "gcp"`, `credential-file-path` specifies the path of the verification credentials file, which currently supports two types of credentials: Service Account and Authentication User. If the TiKV operating environment is already configured with [application default credentials](https://cloud.google.com/docs/authentication/application-default-credentials), there is no need to configure `credential-file-path`.
+
+If you need to use Workload Identity Federation (WIF) in the Google Cloud KMS scenario, use `gcp_v2` instead:
+
+```toml
+[security.encryption.master-key]
+type = "kms"
+key-id = "projects/project-name/locations/global/keyRings/key-ring-name/cryptoKeys/key-name"
+vendor = "gcp_v2"
+
+[security.encryption.master-key.gcp]
+credential-file-path = "/path/to/external-account.json"
+```
+
+- When `vendor = "gcp_v2"`, explicit credentials support only Service Account and `external_account`.
+- If you are using an `authorized_user` JSON generated by ADC, you cannot directly configure that JSON as `credential-file-path`. In this case, omit `credential-file-path` and let TiKV obtain authentication information through [application default credentials](https://cloud.google.com/docs/authentication/application-default-credentials) in the operating environment.
+- The old `vendor = "gcp"` does not support using `external_account` as explicit credentials, so WIF cannot be used in this way.
 
 </div>
 <div label="Azure KMS">

tikv-configuration-file.md

@@ -2399,6 +2399,14 @@ Configuration items related to BR backup.
 + If data is backed up to S3 and the backup file is larger than the value of this configuration item, [multipart upload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_UploadPart.html) is automatically enabled. Based on the compression ratio, the backup file generated by a 96-MiB Region is approximately 10 MiB to 30 MiB.
 + Default value: 5MiB
 
+### `gcp-v2-enable` <span class="version-mark">New in v8.5.7</span>
+
++ Whether to enable the `gcp_v2` external storage backend when using Google Cloud Storage (GCS) to execute full backup or restore.
++ Default value: `true`
++ When this configuration item is `true`, TiKV uses the `gcp_v2` implementation to access GCS; when this configuration item is `false`, TiKV continues to use the old GCS implementation.
++ If you need to use Google Cloud Workload Identity Federation (WIF) in full backup or restore scenarios, keep this configuration item set to `true`.
++ For information about GCS authentication methods and how to use WIF/ADC, see [Backup storage](/br/backup-and-restore-storages.md).
+
 ## backup.hadoop
 
 ### `home`
@@ -2454,6 +2462,14 @@ Configuration items related to log backup.
 + The temporary path to which log files are written before being flushed to external storage.
 + Default value: `${deploy-dir}/data/log-backup-temp`
 
+### `gcp-v2-enable` <span class="version-mark">New in v8.5.7</span>
+
++ Whether to enable the `gcp_v2` external storage backend when using Google Cloud Storage (GCS) for log backup.
++ Default value: `true`
++ When this configuration item is `true`, TiKV uses the `gcp_v2` implementation to access GCS; when this configuration item is `false`, TiKV continues to use the old GCS implementation.
++ If you need to use Google Cloud Workload Identity Federation (WIF) in log backup scenarios, keep this configuration item set to `true`.
++ For information about GCS authentication methods and how to use WIF/ADC, see [Backup storage](/br/backup-and-restore-storages.md).
+
 ## cdc
 
 Configuration items related to TiCDC.