Logging & Metrics

Ververica Platform makes it easy to integrate your third-party logging and metrics services with Apache Flink® and establish a consistent monitoring infrastructure across your streaming application landscape.

As a platform administrator you can do this by pre-configuring logging profiles and setting default configurations for Apache Flink® metrics reporters.

Logging Profiles

Logging profiles are named Apache Log4j 2 configuration Twig templates that are available for usage in the logging section of Deployment resources.

Templates are rendered as a log4j2.xml as part of Flink jobs that use the respective profile.

Configuration

Ververica Platform ships a default logging profile named default (see Default Logging Profile (#default-logging-profile) in the appendix).

Administrators can overwrite the default profile or add additional profiles in the Ververica Platform configuration as shown in the example below.

1vvp:
2  flinkLoggingProfiles:
3  - name: default
4    template: |
5      <?xml version="1.0" encoding="UTF-8" standalone="no"?>
6      <Configuration xmlns="http://logging.apache.org/log4j/2.0/config" strict="true">
7      ...
8      </Configuration>
9  - name: custom
10    template: |
11      <?xml version="1.0" encoding="UTF-8" standalone="no"?>
12      <Configuration xmlns="http://logging.apache.org/log4j/2.0/config" strict="true">
13      ...
14      </Configuration>

Note that you are required to manually provide any additional code dependencies for your logging configuration to work as part of the Flink Docker image.

Placeholders

When configuring a template, you have the following Twig variables available:

• {{ namespace }}: Namespace of the Deployment resource.
• {{ jobId }}: ID of the Job resource.
• {{ rootLoggerLogLevel }}: Log level of the root logger.
• {{ userConfiguredLoggers }}: A key value map of user configured log levels (key: logger, value: log level).

Depending on the Deployment Mode additional Twig variables are available:

• Application mode:
  • {{ deploymentId }}: ID of Deployment resource.
  • {{ deploymentName }}: Name of Deployment resource.
• Session mode:
  • {{ sessionClusterID }}: ID of Session Cluster resource.
  • {{ sessionClusterName }}: Name of Session Cluster resource.

You can use these variables to customize your configuration template as needed. Please check the Default Logging Profile (#default-logging-profile) in the appendix for a full example.

Using the rootLoggerLogLevel and userConfiguredLoggers Twig variables guarantees that the logging section of Deployment resources is reflected in the rendered logging configuration.

Example

In the following example, we add a logging profile called kafkathat logs to the console and a Apache Kafka® cluster using the KafkaAppender.

1vvp:
2  flinkLoggingProfiles:
3  - name: kafka
4    template: |
5      <?xml version="1.0" encoding="UTF-8" standalone="no"?>
6      <Configuration xmlns="http://logging.apache.org/log4j/2.0/config" strict="true">
7        <Appenders>
8          <Appender name="StdOut" type="Console">
9            <Layout pattern="%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n" type="PatternLayout"/>
10          </Appender>
11          <Kafka name="Kafka" topic="log-example">
12            <PatternLayout pattern="%date %message"/>
13            <Property name="bootstrap.servers">localhost:9092</Property>
14          </Kafka>
15        </Appenders>
16        <Loggers>
17          {%- for name, level in userConfiguredLoggers -%}
18          <Logger level="{{ level }}" name="{{ name }}"/>
19          {%- endfor -%}
20          <!-- Avoid recursive logging -->
21          <Logger name="org.apache.kafka" level="INFO" />
22          <Root level="{{ rootLoggerLogLevel }}">
23            <AppenderRef ref="StdOut"/>
24            <AppenderRef ref="Kafka"/>
25          </Root>
26        </Loggers>
27      </Configuration>

Note that you are required to manually provide any additional code dependencies for your logging configuration to work as part of the Apache Flink® Docker image.

Metrics

Apache Flink® comes with a comprehensive and flexible metrics system, which covers system metrics provided by the framework itself as well as user-defined metrics.

These metrics can be exposed to an external system using so called metrics reporters. These reporters will be instantiated on each job- and taskmanager during startup.

Out of the box, Ververica Platform bundles metrics reporters for

• Prometheus
• InfluxDB
• Datadog
• Prometheus Pushgateway
• Graphite
• StatsD
• SLF4J

Metrics reports are configured via the Flink configuration. See the Apache Flink® documentation for the specific configuration options of each of the reporters.

Example

The following snippet configures a Deployment to expose metrics to Prometheus.

1spec:
2  template:
3    spec:
4      flinkConfiguration:
5        metrics.reporter.prom.factory.class: org.apache.flink.metrics.prometheus.PrometheusReporterFactory
6        metrics.reporter.prom.port: 9249

Integration with Kubernetes Pod Templates (for Prometheus Scraping):

To ensure Prometheus can scrape metrics from Flink jobs running on Kubernetes, you need to configure the following annotations for both the JobManager and TaskManager pod templates:

1spec:
2  template:
3    spec:
4      kubernetes:
5        taskManagerPodTemplate:
6          metadata:
7            annotations:
8              prometheus.io/port: '9249'
9              prometheus.io/scrape: 'true'
10        jobManagerPodTemplate:
11          metadata:
12            annotations:
13              prometheus.io/port: '9249'
14              prometheus.io/scrape: 'true'

We recommend using global or namespaced Deployment Defaults to configure metrics reporters for all your Deployments in a single place.

Web User Interface

For each Deployment, the web user interface links to a metrics and logging dashboard as well as the Flink UI. These links are customizable in the configuration.

1vvp:
2  ui:
3    linkTemplates:
4      flinkUi: <link_template>
5      jobLogs: <link_template>
6      metrics: <link_template>
7      deploymentLogs: <link_template>

The link templates can contain placeholders such as <%= jobId %>. The following placeholders are available:

jobStartDate and jobEndDate can be used for building flink job metrics in Grafana:

1vvp:
2  ui:
3    linkTemplates:
4      metrics: http://localhost:3000/path/to/your/dashboard?from=<%= jobStartDate %>&to=<%= jobEndDate %>

Appendix

Default Logging Profile

Ververica Platform ships a default logging profile named default. The default configuration logs to the console and to a local rolling file whose location is expected to be specified via the system property log.file (for display in the Flink UI).

1<?xml version="1.0" encoding="UTF-8" standalone="no"?>
2<Configuration xmlns="http://logging.apache.org/log4j/2.0/config" strict="true">
3  <Appenders>
4    <Appender name="StdOut" type="Console">
5      <Layout pattern="%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n" type="PatternLayout"/>
6    </Appender>
7    <Appender name="RollingFile" type="RollingFile" fileName="${sys:log.file}" filePattern="${sys:log.file}.%i">
8      <Layout pattern="%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n" type="PatternLayout"/>
9      <Policies>
10        <SizeBasedTriggeringPolicy size="5 MB"/>
11      </Policies>
12      <DefaultRolloverStrategy max="1"/>
13    </Appender>
14  </Appenders>
15  <Loggers>
16    <Logger level="INFO" name="org.apache.hadoop"/>
17    <Logger level="INFO" name="org.apache.kafka"/>
18    <Logger level="INFO" name="org.apache.zookeeper"/>
19    <Logger level="INFO" name="akka"/>
20    <Logger level="ERROR" name="org.jboss.netty.channel.DefaultChannelPipeline"/>
21    <Logger level="OFF" name="org.apache.flink.runtime.rest.handler.job.JobDetailsHandler"/>
22    {%- for name, level in userConfiguredLoggers -%}
23    <Logger level="{{ level }}" name="{{ name }}"/>
24    {%- endfor -%}
25    <Root level="{{ rootLoggerLogLevel }}">
26      <AppenderRef ref="StdOut"/>
27      <AppenderRef ref="RollingFile"/>
28    </Root>
29  </Loggers>
30</Configuration>