Summary
The user is facing a runtime exception related to string length exceeding the maximum allowed length in a Postgresql + Clickhouse replication setup.
Question
I have a Postgresql + Clickhouse replication set up and the following problem occurs:
java.lang.RuntimeException: com.fasterxml.jackson.core.exc.StreamConstraintsException: String length (20054016) exceeds the maximum length (20000000)
3957
How to fix it ?
This topic has been created from a Slack thread to give it more visibility.
It will be on Read-Only mode here. Click here if you want
to access the original thread.
Join the conversation on Slack
["postgresql", "clickhouse", "replication", "runtime-exception", "string-length", "maximum-length"]
<@U077KVDG7GA> thanks! In which file to do this? I clarify that I am using Docker
Here’s how you can do it:
- Spring Boot Application: If you’re using Spring Boot, you likely have an
application.properties or application.yml file where you can configure Jackson’s properties. Add the following property to increase the maximum string length:
- properties
- spring.jackson.max-string-length=50000000
- Replace
50000000 with your desired maximum string length.
- Standalone Jackson Configuration: If you’re configuring Jackson directly, you can set the maximum string length on the
ObjectMapper instance:
- java
- ObjectMapper objectMapper = new ObjectMapper();
- objectMapper.getFactory().setMaxLength(50000000);
- Add this configuration to wherever you configure your
ObjectMapper.
- Docker Environment: If you’re deploying your application within a Docker container, you can pass the Jackson configuration via environment variables or Docker secrets. Modify your Dockerfile or Docker Compose configuration to include environment variables like so:
- Dockerfile
- ENV SPRING_JACKSON_MAX_STRING_LENGTH=50000000
- Ensure that your application reads this environment variable and applies it to the Jackson configuration.
By increasing the maximum string length, you allow Jackson to handle larger strings without encountering the StreamConstraintsException. Adjust the maximum length value according to your application’s requirements and the size of the strings you expect to handle.
How do I do this in the <https://github.com/airbytehq/airbyte-platform/blob/main/docker-compose.yaml|official Docker Compose Airbyte> implementation? It’s not clear which service to pass the variable to.
Increase Jackson’s Maximum String Length: You can increase Jackson’s maximum string length property to accommodate larger strings. This can be done by configuring the ObjectMapper with a custom JsonFactory that has a larger maximum string length. Here’s an example:
ObjectMapper mapper = new ObjectMapper();
mapper.getFactory().setMaxLength(50000000); // Set maximum length to 50 MB```
<@U077KVDG7GA> If we consider the option with Docker Compose, which service should we pass the variable to?
The exception you’re encountering is a StreamConstraintsException from the Jackson JSON library, indicating that a string in the JSON data being processed exceeds the maximum allowed length. To resolve this issue, you need to increase the maximum allowed string length for Jackson. When using Docker Compose, you need to identify which service is using Jackson for JSON processing and configure it accordingly.
Here’s how you can handle it:
- Identify the Service:
◦ Determine which service is using Jackson for JSON processing and is encountering the error. This is typically the application service that is handling the replication between PostgreSQL and ClickHouse.
- Set Jackson Configuration:
◦ You need to configure the Jackson StreamReadConstraints to allow a longer maximum string length. This configuration can usually be done programmatically in your application’s configuration files or by passing environment variables if your application supports it.
Example: Spring Boot Application
If your service is a Spring Boot application, you can configure Jackson in the application.properties or application.yml file or programmatically in a configuration class.
Using application.properties or application.yml:
Unfortunately, setting StreamReadConstraints directly via application.properties is not supported. You need to configure it programmatically:
Programmatically:
Create a configuration class to set up the Jackson ObjectMapper with the desired constraints.
import com.fasterxml.jackson.databind.ObjectMapper;
import com.fasterxml.jackson.core.StreamReadConstraints;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
@Configuration
public class JacksonConfig {
@Bean
public ObjectMapper objectMapper() {
ObjectMapper mapper = new ObjectMapper();
mapper.getFactory().setStreamReadConstraints(
StreamReadConstraints.builder().maxStringLength(30000000).build());
return mapper;
}
}```
Docker Compose Configuration
In your `docker-compose.yml` file, ensure you pass any necessary environment variables or configuration files to the service if needed. Here is an example:
```yaml
version: '3.8'
services:
app-service:
image: your-app-image
environment:
- SPRING_PROFILES_ACTIVE=prod
volumes:
- ./config:/config
depends_on:
- postgres
- clickhouse
postgres:
image: postgres:latest
environment:
POSTGRES_DB: your_db
POSTGRES_USER: your_user
POSTGRES_PASSWORD: your_password
clickhouse:
image: yandex/clickhouse-server:latest
environment:
CLICKHOUSE_DB: your_db```
Summary
To fix the `StreamConstraintsException`:
1. *Identify the service using Jackson* and encountering the issue.
2. *Configure Jackson's `StreamReadConstraints` programmatically* to increase the maximum allowed string length.
3. *Ensure the Docker Compose configuration* correctly sets up the service with the required configuration.
By setting up the Jackson configuration correctly in your application, you should be able to handle larger strings without encountering the `StreamConstraintsException`.
can I do this on the cloud version? Same problem here