Quickstart: Google GKE

Applies toSelf-Managed v2

2 min read

On this page

Quickstart
Building Apache Flink® Images with Support for GCS

Google Kubernetes Engine (GKE) is Google Cloud Platform's hosted Kubernetes service.

A Google Kubernetes Engine cluster created using the official guide is suitable to run Ververica Platform.

Quickstart

Create a Google Kubernetes Engine cluster using the official guide.
Follow the Helm installation guide to finish installing Ververica Platform.
Follow the steps below to add support for GCS as checkpoint and savepoint storage.

For an in-depth, step-by-step guide on setting up Ververica Platform on GKE, take a look at our knowledge base article: How to set up Ververica Platform with Google Cloud Storage (GCS).

Building Apache Flink® Images with Support for GCS

Apache Flink® currently does not bundle libraries for working with GCS. There is, however, a connector from Google based on Apache Hadoop®. In order to use the gs:// scheme for Universal Blob Storage, it is required to bundle this library along with the shaded Hadoop JAR.

Download GCS dependencies

Please download the following GCS dependencies. Click each link to start the download:

Move the downloaded files so you can access them to build the Dockerfile.

Create Dockerfile

YAML

1FROM registry.ververica.com/v{shortVersion}/flink:{ververicaFlinkRelease}-scala_2.12-java11
2
3COPY gcs-connector-hadoop2-latest.jar ../lib/
4COPY flink-shaded-hadoop-2-uber-2.8.3-10.0.jar ../lib/

To build a Dockerfile for Apache Flink® 1.16.1 run

BASH

1docker build -t {ververicaFlinkRelease}-scala_2.12-java11-gcs .
2docker tag {ververicaFlinkRelease}-scala_2.12-java11-gcs <YOUR_DOCKER_REGISTRY>/flink:{ververicaFlinkRelease}-scala_2.12-java11-gcs
3docker push <YOUR_DOCKER_REGISTRY>/flink:{ververicaFlinkRelease}-scala_2.12-java11-gcs

For other versions simply adjust the build arguments accordingly.

Please check Ververica Platform Docker Images for all available Apache Flink® images and additional tags.

Prepare Ververica Platform

Amend your values.yaml with the following parameters:

YAML

1vvp:
2  blobStorage:
3    baseUri: gs://<YOUR_GCS_BUCKET>
4
5  flinkVersionMetadata:
6    - flinkImageRegistry: <YOUR_DOCKER_REGISTRY>
7      flinkImageRepository: flink
8      flinkVersion: "{flinkRelease}"
9      imageTag: "{ververicaFlinkRelease}-scala_2.12-java11-gcs"
10      useForSqlDeployments: true
11      defaultFor:
12        - "{flinkVersion}"
13
14  globalDeploymentDefaults: |
15    spec:
16      template:
17        spec:
18          artifact:
19            kind: jar
20            flinkImageRegistry: <YOUR_DOCKER_REGISTRY>
21            flinkImageRepository: flink
22            flinkImageTag: {ververicaFlinkRelease}-scala_2.12-java11-gcs
23            flinkVersion: '{flinkVersion}'
24          kubernetes:
25            pods:
26              volumeMounts:
27              - name: gcs-key
28                volume:
29                  name: gcs-key
30                  secret:
31                    secretName: gcs-key
32                volumeMount:
33                  mountPath: /flink/gcs-creds/
34                  name: gcs-key
35              envVars:
36                - name: GOOGLE_APPLICATION_CREDENTIALS
37                  value: /flink/gcs-creds/gcs-key.json
38
39  globalSessionClusterDefaults: |
40    spec:
41      kubernetes:
42        pods:
43          volumeMounts:
44            - name: gcs-key
45              volume:
46                name: gcs-key
47                secret:
48                  secretName: gcs-key.json
49              volumeMount:
50                mountPath: /flink/gcs-creds/
51                name: gcs-key
52          envVars:
53            - name: GOOGLE_APPLICATION_CREDENTIALS
54              value: /flink/gcs-creds/gcs-key.json
55
56  volumes:
57    - name: "gcs-key"
58      secret:
59        secretName: "gcs-key"
60
61  volumeMounts:
62    - name: "gcs-key"
63      mountPath: "/vvp/gcs-creds"
64
65  env:
66    - name: "GOOGLE_APPLICATION_CREDENTIALS"
67      value: "/vvp/gcs-creds/gcs-key.json"

Permissions for GCS

Note

This step is optional if you already have service account credentials with the Storage Admin role permissions for your designated GCS bucket.

If you do not yet have a service account, create one:

BASH

1gcloud iam service-accounts create gcs-vvp-service-acc \
2--description="Service account for VVP GCS" \
3--display-name="gcs-vvp-service-acc"

Assuming that you use the "uniform" bucket access control, navigate to Storage Browser, click three dots next to your bucket -> Edit bucket permissions -> + Add member -> and paste your full service account ID (gcs-vvp-service-acc@<YOUR_PROJECT>.iam.gserviceaccount.com).
In "Select a role", pick "Storage Admin". Save.
Navigate to Service Accounts, click three dots next to the gcs-vvp-service-acc service account -> Create key -> JSON -> Store as gcs-key.json.
Create a Kubernetes secret in the namespace where Ververica Platform is installed (vvp in this example):

BASH

1kubectl create secret generic
2gcs-key --from-file=./gcs-key.json -n vvp

Repeat for each namespace that is used as a Deployment Target (vvp-jobs in this example):

BASH

1kubectl create secret generic gcs-key --from-file=./gcs-key.json -n vvp-jobs

Was this helpful?

Yes No