Log Insights: Kubernetes with OpenTelemetry

If you are running the database in a Kubernetes cluster, you can use the db_log_otel_server setting to have the collector listen for OpenTelemetry log data model messages. When this is specified, the collector will run an HTTP server behind the scenes on the specified port to receive logs at the /v1/logs endpoint. Currently, this endpoint expects OpenTelemetry log messages sent as protobuf.

The configuration setting should be local address like 0.0.0.0:4318 and it's recommended to use an unprivileged port number (1024 and up) to avoid running the HTTP server as root.

[pganalyze]
api_key = your_pga_organization_api_key
 
[server1]
db_host = your_db_host
...
db_log_otel_server = 0.0.0.0:4318

Multiple OpenTelemetry endpoints

If you have multiple Postgres clusters that you want to monitor through OpenTelemetry, you can configure one pganalyze collector to start multiple OpenTelemetry HTTP servers on different ports, each for a specific monitored cluster.

For each monitored server in the pganalyze-collector.conf file, you can specify the db_log_otel_server setting with a different port number, like this:

[pganalyze]
api_key = your_pga_organization_api_key
 
[server1]
db_host = your_db_host
...
db_log_otel_server = 0.0.0.0:4318
 
[server2]
db_host = your_db_host2
...
db_log_otel_server = 0.0.0.0:4319

This would start two OpenTelemetry HTTP servers, one on port 4318 and another on port 4319 when the collector starts. To verify, you can start the collector with verbose logging to verify output similar to the example below.

$ sudo pganalyze-collector --verbose
...
...
2025/07/29 11:20:53 I Setting up OTLP HTTP server receiving logs with 0.0.0.0:4318
2025/07/29 11:20:53 I Setting up OTLP HTTP server receiving logs with 0.0.0.0:4319
...

Using with a pganalyze collector Helm chart

If you are deploying the pganalyze collector using a Helm chart, you can update the values.yaml file to create a service for this HTTP server. In values.yaml file, update the section of service:

service:
  create: true
  name: pganalyze-collector-otel-service
  type: ClusterIP
  port: 4318
  targetPort: 4318

This will create a service with the name pganalyze-collector-otel-service.

To create multiple OpenTelemetry endpoints, you can add multiple port entries under the ports section:

service:
  create: true
  name: pganalyze-collector-otel-service
  type: ClusterIP
  ports:
    - name: pganalyze-collector-otel1
      port: 4318
      targetPort: 4318
    - name: pganalyze-collector-otel2
      port: 4319
      targetPort: 4319

When there is a ports section, the Helm chart will ignore the settings for any individual port and targetPort settings.

Log forwarding with Fluent Bit or Vector

We recommend using either Fluent Bit or Vector for processing logs from your Kubernetes cluster and forwarding them to the collector's OTLP logs endpoint:

• Setup instructions for Fluent Bit
• Setup instructions for Vector

Troubleshooting Fluent Bit, Vector and other OTLP payloads

One of the common issues when using Fluent Bit with OpenTelemetry is that the payloads are sent in Protobuf format (either binary or JSON encoded) which is a compact, efficient transport format, yet difficult to debug on the receiving end when it doesn't match the expected OpenTelemetry log data model.

To assist with troubleshooting, you can set the --very-verbose runtime parameter for the pganalyze collector. This will log the raw Protobuf payloads received by the OpenTelemetry HTTP server and output them as JSON, which can help you identify if the payloads are correctly formatted and match the expected OpenTelemetry log data model.

Note: Before starting another collector to test for OpenTelemetry input, you will have to stop the collector background process to avoid port conflicts. For package based installs you can do this through systemctl stop pganalyze-collector.

$ sudo systemctl stop pganalyze-collector
 
$ sudo pganalyze-collector --very-verbose

This will start the collector with very verbose logging enabled, and you should see output like this when it receives a Protobuf payload.

Warning: this output is very verbose and could quickly become overwhelming, so it's recommended to use it only for troubleshooting purposes for a short period of time.

2025/07/21 22:25:21 I {
  "resource_logs": [
    {
      "resource": {},
      "scope_logs": [
        {
          "scope": {},
          "log_records": [
            {
              "time_unix_nano": 1753151120429549420,
              "body": {
                "Value": {
                  "KvlistValue": {
                    "values": [
                      {
                        "key": "stream",
                        "value": {
                          "Value": {
                            "StringValue": "stderr"
                          }
                        }
                      },
...