Ververica DocsVerverica Docs
Docs Home
Viewing docs for
BYOCSelf-Managed

JSON

Applies toBYOCSelf-Managed v3
1 min read
On this page

This article introduces you to the usage and type mapping of the JSON format.

Background Information

JSON format can read and write JSON data based on the JSON structure. Currently, the JSON structure is automatically deduced from the table structure.

Example of Use

An example of constructing a table using Kafka and JSON format is as follows.

SQL
1    CREATE TABLE user_behavior (
2        user_id BIGINT,
3        item_id BIGINT,
4        category_id BIGINT,
5        behavior STRING,
6    ts TIMESTAMP(3)
7    ) WITH (
8       'connector' = 'kafka',
9       'topic' = 'user_behavior',
10       'properties.bootstrap.servers' = 'localhost :9092',
11       'properties.group.id' = 'testGroup',
12       'format' = 'json',
13       'json.fail-on-missing-field' = 'false',
14       'json.ignore-parse-errors' = 'true'
15    );

Cofiguration Option

ParameterRequiredDefaultType of dataDescription
formatyes(none)StringThe format to use for the declaration. When using the JSON format, the parameter value is json.
json.fail-on-missing-fieldnofalseBooleanThe parameter values are as follows: true: Skip current field or row when parsing field is missing. false (default): When parsing fields are missing, an error is thrown and the job fails to start.
json.ignore-parse-errorsnofalseBooleanThe parameter values are as follows: true: When parsing exceptions, skip the current field or row. false (default): When parsing an exception, an error is thrown and the job fails to start.
json.timestamp-format.standardnoSQLStringSpecifies the input and output timestamp format. The parameter values are as follows: SQL: Parse input timestamps in yyyy-MM-dd HH:mm:ss.s{precision} format, such as 2020-12-30 12:13:14.123, and output timestamps in the same format. ISO-8601: Parse input timestamps in yyyy-MM-ddTHH:mm:ss.s{precision} format, such as 2020-12-30T12:13:14.123, and output timestamps in the same format.
json.map-null-key.modenofailStringSpecifies the method to handle empty key values in the Map. The parameter values are as follows: FAIL: An exception is thrown when the key value in the Map is empty. DROP: Discard the data whose key value in the Map is empty. LITERAL: Use string constants to replace empty key values in the Map. The value of the string constant is defined by canal-json.map-null-key.literal.
json.map-null-key.literalnonullStringWhen the parameter of json.map-null-key.mode is LITERAL, specify a string constant to replace the empty key value in the Map.
json.encode.decimal-as-plain-numbernofalseBooleanThe parameter values are as follows: true: All data of DECIMAL type remain as they are, and are not expressed in scientific notation, for example: 0.000000027 is expressed as 0.000000027. false: All data of DECIMAL type are expressed in scientific notation, for example: 0.000000027 is expressed as 2.7E-8.

Type Mapping

Currently, the JSON structure will be automatically deduced from the table structure. In Flink, the JSON format uses the jackson databind API (jackson databind API) to parse and generate JSON. The mapping relationship between Flink and JSON data types is as follows.

Flink SQL typeJSON type
CHAR/VARCHAR/STRINGstring
BOOLEANboolean
BINARY / VARBINARYstring with encoding: base64
DECIMALnumber
TINYINTnumber
SMALLINTnumber
INTnumber
BIGINTnumber
FLOATnumber
DOUBLEnumber
DATEstring with format: date
timestring with format: time
TIMESTAMPstring with format: date-time
TIMESTAMP_WITH_LOCAL_TIME_ZONEstring with format: date-time (with UTC time zone)
INTERVALnumber
ARRAYarray
MAP / MULTISETobject
ROWobject

Other Instructions for Use

For writing to object storage S3, currently it does not support writing files in JSON format. For specific reasons, see FLINK-30635 .

Was this helpful?