Enterprise Server: Object Storage Setup (Local Storage)

Prerequisites

  • Follow the initial setup steps until you have at least have a /etc/pganalyze.env file created
  • Provision a local storage directory (e.g. at /var/pganalyze-data) with sufficient disk space
    • Having at least 10 GB per monitored database server is a good starting point
    • Actual disk usage will be dependent on the amount of Postgres error log text in a 7 day period, plus a small amount of additional space for temporary snapshot data

Step 1: Installation of the MinIO storage server

Since pganalyze requires an Amazon S3-compatible object storage to store log data, we support the open-source storage server MinIO for installations outside of AWS. The open-source MinIO storage server can be run in a simple Docker container, pointing to a local directory for storage.

Start by generating a unique pair of access and secret keys:

export MINIO_ACCESS_KEY=$(xxd -u -l 16 -p /dev/urandom)
export MINIO_SECRET_KEY=$(xxd -u -l 16 -p /dev/urandom)

Now start the MinIO container, pointing it to the storage directory (e.g. "/var/pganalyze-data"):

# Replace "/var/pganalyze-data" with your preferred location, e.g. "/mnt/mystorage:/data" (the ":/data" suffix indicates the mount point inside the container and should not be changed)
docker run --name minio -v /var/pganalyze-data:/data -e MINIO_ACCESS_KEY=$MINIO_ACCESS_KEY -e MINIO_SECRET_KEY=$MINIO_SECRET_KEY -d -p 9000:9000 minio/minio server /data

Note that whilst we do not require the MinIO container and storage to be on the same server as the pganalyze container, it is typically easiest to get started this way.

Step 2: Create buckets and configure retention policies

By default, MinIO will keep objects forever, which can cause issues with exhausting disk storage after a while. We recommend setting a 35 day expiry policy for MinIO using the mc client:

# Install minio "mc" client
curl https://dl.min.io/client/mc/release/linux-amd64/mc --create-dirs -o /usr/local/bin/mc
chmod +x /usr/local/bin/mc

# Replace "MINIO_ACCESS_KEY" and "MINIO_SECRET_KEY" with correct values set earlier
mc alias set pganalyze-minio http://localhost:9000 $MINIO_ACCESS_KEY $MINIO_SECRET_KEY

mc mb pganalyze-minio/pganalyze-logs
mc mb pganalyze-minio/pganalyze-snapshots

mc ilm add pganalyze-minio/pganalyze-logs --expiry-days 7
mc ilm add pganalyze-minio/pganalyze-snapshots --expiry-days 35

Step 3: Configuration of the pganalyze container

With the MinIO server working, you can now configure the pganalyze container.

First of all, we also need to generate an encryption key, which will be used to encrypt all log data in transit as well as at rest. This way you can be sure that only those that have access to the encryption key can read sensitive log information.

You can generate a key like this:

export LOG_ENCRYPTION_KEY=$(echo '{"1": "'$(dd if=/dev/urandom bs=32 count=1 2>/dev/null | openssl base64)'"}')

And then add the environment variables to your pganalyze.env file:

echo LOG_ENCRYPTION_KEY=$LOG_ENCRYPTION_KEY >> /etc/pganalyze.env
echo MINIO_ACCESS_KEY=$MINIO_ACCESS_KEY >> /etc/pganalyze.env
echo MINIO_SECRET_KEY=$MINIO_SECRET_KEY >> /etc/pganalyze.env
echo MINIO_ENDPOINT=http://172.17.0.1:9000 >> /etc/pganalyze.env

Make sure that the IP address specified in MINIO_ENDPOINT is either that of the Docker bridge (default 172.17.0.1), or of the MinIO container itself.

If you've set up MinIO as part of the initial setup, you can now continue with the setup.

If you have previously completed setup, make sure to restart the container. You can now use Log Insights and Automated EXPLAIN, as well as install the collector separately (if desired).


Couldn't find what you were looking for or want to talk about something specific?
Start a conversation with us →