Wednesday, June 11 2025

MinIO & Elasticsearch Integration

An Interactive Guide to Snapshot Repository Setup on Raspberry Pi

Interactive Report: MinIO & Elasticsearch Integration

MinIO & Elasticsearch Integration

An Interactive Guide to Snapshot Repository Setup on Raspberry Pi

Project Summary

This application details the successful setup of a robust data backup solution using MinIO as an S3-compatible snapshot repository for Elasticsearch. The entire system was deployed on a Raspberry Pi 2, demonstrating a cost-effective and powerful method for local data management and disaster recovery. This guide breaks down the process into clear, actionable sections, from initial server configuration to final integration verification.

Primary Objective

To enable reliable, automated backups of Elasticsearch data indices to a self-hosted, S3-compatible object storage server (MinIO).

Final Status

Integration complete and fully operational. Elasticsearch can successfully register the MinIO repository and perform snapshot and restore operations.

Core System Configuration

The foundation of this project lies in the correct setup of both the MinIO object storage server and the Elasticsearch instance. This section outlines the key hardware and software configurations required for each component to function correctly before integration.

🗄️ MinIO Server

  • Hardware: Raspberry Pi 2 Model B (ARMv7 32-bit)
  • Operating System: Raspberry Pi OS
  • Binary: 32-bit `linux-arm` version is essential.
  • Data Storage: External drive mounted at `/path/to/minio/data`
  • Permissions: Directory owned by the `minio-user` user.
  • Startup: Managed by a `systemd` service for persistence.
  • API Endpoint: `http://YOUR_MINIO_IP:9000`

🔍 Elasticsearch

  • Plugin: `repository-s3` plugin required (if version < 8.x).
  • Configuration: `elasticsearch.yml` updated to point to MinIO’s API endpoint.
  • Security: Credentials stored securely in the Elasticsearch Keystore.
  • Ownership: Keystore commands must be run as the `elasticsearch` user.
  • Key Config: `path_style_access` must be set to `true`.

Integration Process Flow

Connecting Elasticsearch to MinIO involves a precise sequence of steps. This flow visualizes the critical path from creating a storage location in MinIO to verifying the final connection in Kibana. Following this process ensures all dependencies are met in the correct order.

1

Create MinIO Bucket & User

Log in to the MinIO console, create a dedicated bucket (e.g., `elastic`), and generate a new Access Key with a `readwrite` policy for that bucket.

2

Configure elasticsearch.yml

Add the MinIO S3 client settings (`endpoint`, `protocol`, `path_style_access`) to the configuration file on all Elasticsearch nodes.

3

Add Credentials to Keystore

Securely add the Access Key and Secret Key you generated in Step 1 to the Elasticsearch Keystore using `elasticsearch-keystore add`.

4

Restart & Register

Restart the Elasticsearch service. Then, use the Kibana UI to register a new S3 repository, specifying the correct client name (`default`) and bucket name (`elastic`).

Troubleshooting Guide

Technical setups often encounter hurdles. This section compiles the key issues faced during this integration and their resolutions. Click on each problem to reveal the cause and the specific solution that was applied.

“Exec format error” when running MinIO

+

Cause: The MinIO binary was compiled for the wrong CPU architecture (e.g., 64-bit ARM or x86) and cannot run on the Raspberry Pi 2’s 32-bit ARMv7 processor.

Solution: Download the correct binary specifically for `linux-arm` from the official MinIO download site.

YAML Parsing Error in elasticsearch.yml

+

Cause: Incorrect indentation, stray characters, or uncommented example/separator lines in the `elasticsearch.yml` file. YAML is extremely sensitive to formatting.

Solution: Carefully review the file, remove or comment out any non-YAML lines (like `—-`), and ensure all settings use consistent spacing for indentation.

“Will not overwrite keystore” Error

+

Cause: Running `elasticsearch-keystore add` with `sudo` (as root). This would change the keystore file’s ownership, which Elasticsearch’s security model prevents.

Solution: Execute the command as the `elasticsearch` user to maintain correct file ownership: `sudo -u elasticsearch bin/elasticsearch-keystore add …`

“Unknown s3 client name” Error in Kibana

+

Cause: Entering the MinIO endpoint URL (e.g., `http://YOUR_MINIO_IP:9000`) into the “Client” field in the Kibana repository setup form.

Solution: The “Client” field must contain the name of the client defined in `elasticsearch.yml`, which is `default` in this setup.

Configuration Cheatsheet

This section provides a quick reference for the essential configuration files and commands used in this project. Use the copy buttons for easy, error-free implementation.

MinIO Systemd Service File

Location: `/etc/systemd/system/minio.service`

[Unit]
Description=MinIO Object Storage Server
Wants=network-online.target
After=network-online.target

[Service]
User=minio-user
Group=minio-user
EnvironmentFile=/etc/default/minio
ExecStart=/usr/local/bin/minio server /path/to/minio/data --console-address ":9001"
Restart=always

[Install]
WantedBy=multi-user.target

Elasticsearch S3 Config

Location: `elasticsearch.yml`

s3.client.default.endpoint: "http://YOUR_MINIO_IP:9000"
s3.client.default.protocol: http
s3.client.default.path_style_access: true

Key Keystore Commands

Run from Elasticsearch `bin` directory.

# Add Access Key (as elasticsearch user)
sudo -u elasticsearch ./elasticsearch-keystore add s3.client.default.access_key

# Add Secret Key (as elasticsearch user)
sudo -u elasticsearch ./elasticsearch-keystore add s3.client.default.secret_key
# List keys to verify (as elasticsearch user)
sudo -u elasticsearch ./elasticsearch-keystore list

Complete Command-Line Guide

This section provides all command-line instructions in a single, comprehensive guide, replicating the full setup process for MinIO and its integration with Elasticsearch.

1. MinIO Server Setup (on Raspberry Pi)

These steps prepare your Raspberry Pi to run the MinIO object storage server.

1.1. Download MinIO Binary

# Remove any old, incorrect binary if it exists
rm minio

# Download the correct 32-bit ARM binary (linux-arm)
wget https://dl.min.io/server/minio/release/linux-arm/minio

1.2. Make MinIO Executable and Move to System Path

# Make the binary executable
chmod +x minio

# Move the binary to /usr/local/bin
sudo mv minio /usr/local/bin/

1.3. Prepare MinIO Data Directory

Replace `/path/to/minio/data` with your actual data directory.

# Create the data directory if it doesn't exist
sudo mkdir -p /path/to/minio/data

# Change ownership of the data directory to the user MinIO will run as (e.g., minio-user).
# Replace 'minio-user' with the actual user account.
sudo chown -R minio-user:minio-user /path/to/minio/data

# Grant read, write, and execute permissions to the owner
sudo chmod u+rxw /path/to/minio/data

1.4. Configure MinIO Root Credentials (Environment File)

These are the credentials for initial MinIO console access. Replace with your chosen strong values.

# Create the environment file
sudo nano /etc/default/minio

Paste the following into `nano`, then save and exit:

MINIO_ROOT_USER='YOUR_ROOT_ACCESS_KEY'
MINIO_ROOT_PASSWORD='YOUR_ROOT_SECRET_KEY'

1.5. Create MinIO Systemd Service File

This defines how systemd manages MinIO. Replace `/path/to/minio/data` and `minio-user` with your specific details.

# Create the service file
sudo nano /etc/systemd/system/minio.service

Paste the following into `nano`, then save and exit:

[Unit]
Description=MinIO Object Storage Server
Documentation=https://docs.min.io
Wants=network-online.target
After=network-online.target

[Service]
User=minio-user
Group=minio-user
EnvironmentFile=/etc/default/minio
ExecStart=/usr/local/bin/minio server /path/to/minio/data --console-address ":9001"
Restart=always
RestartSec=5s

[Install]
WantedBy=multi-user.target

1.6. Enable and Start MinIO Service

# Reload systemd daemon to read the new service file
sudo systemctl daemon-reload

# Enable MinIO to start on boot
sudo systemctl enable minio

# Start the MinIO service immediately
sudo systemctl start minio

# Check the status of the MinIO service
sudo systemctl status minio

1.7. Access MinIO Console and Create Dedicated User for Elasticsearch

Open a web browser and navigate to `http://YOUR_MINIO_IP:9001`. Log in with your `YOUR_ROOT_ACCESS_KEY` and `YOUR_ROOT_SECRET_KEY`.

In the MinIO WebUI:

  • Go to Buckets and create a new bucket (e.g., `elastic`).
  • Go to Identity or Users (often labeled “Access Keys”).
  • Click “Create access key” or “Add user”.
  • Provide a unique Access Key (e.g., `es_app_key`) and let MinIO generate a Secret Key. Copy these NEW keys down securely.
  • Assign the `readwrite` policy to this new user/key, targeting your `elastic` bucket.

2. Elasticsearch Setup and Integration

These steps configure Elasticsearch to use your MinIO server for snapshots.

2.1. Install S3 Repository Plugin (If Elasticsearch < 8.x)

If your Elasticsearch version is 7.x or older, install this plugin. For 8.x+, it’s usually built-in.

# Navigate to Elasticsearch installation directory
cd /usr/share/elasticsearch # Adjust path if different

# Install the S3 repository plugin
sudo bin/elasticsearch-plugin install repository-s3
# Type 'y' and press Enter to accept.

2.2. Configure `elasticsearch.yml`

This file defines how Elasticsearch’s S3 client connects to MinIO. Replace `YOUR_MINIO_IP` with your MinIO server’s IP.

# Open elasticsearch.yml for editing
sudo nano /etc/elasticsearch/elasticsearch.yml

Add these lines to the end of the file. Ensure correct YAML indentation. Save and exit:

s3.client.default.endpoint: "http://YOUR_MINIO_IP:9000"
s3.client.default.protocol: http
s3.client.default.path_style_access: true

2.3. Add MinIO Credentials to Elasticsearch Keystore

Execute these commands as the `elasticsearch` user.

# Navigate to Elasticsearch bin directory
cd /usr/share/elasticsearch/bin # Adjust path if different

# Add Access Key
sudo -u elasticsearch ./elasticsearch-keystore add s3.client.default.access_key
# When prompted, paste YOUR_ACCESS_KEY (the new key from MinIO console)

# Add Secret Key
sudo -u elasticsearch ./elasticsearch-keystore add s3.client.default.secret_key
# When prompted, paste YOUR_SECRET_KEY (the new key from MinIO console)

2.4. Verify Keystore Contents

# List keystore contents as the elasticsearch user
sudo -u elasticsearch ./elasticsearch-keystore list

2.5. Restart Elasticsearch Service

# Restart Elasticsearch
sudo systemctl restart elasticsearch

# Check Elasticsearch status (ensure it's active/running)
sudo systemctl status elasticsearch

2.6. Register S3 Repository in Kibana

Open your web browser and go to your Kibana UI. Navigate to Stack Management -> Snapshot and Restore -> Repositories.

  • Click “Register a repository”.
  • Select Type: S3.
  • Fill in the fields:
    • Client: `default`
    • Bucket: `elastic` (or the name of your MinIO bucket)
    • Base path: (Optional, leave blank if no subdirectory needed)
    • Compress snapshots: Check this.
    • Adjust other optional settings (like `Max snapshot bytes per second`) as desired.
  • Click “Save” or “Create”.

3. Basic Troubleshooting Commands

Use these commands to diagnose common issues.

3.1. MinIO Binary Execution Errors (`Exec format error`)

# Re-download the correct binary (MinIO on Raspberry Pi 2 is linux-arm)
wget https://dl.min.io/server/minio/release/linux-arm/minio
chmod +x minio
sudo mv minio /usr/local/bin/

3.2. MinIO Data Directory Permissions

# Change ownership to the user MinIO runs as (e.g., minio-user)
sudo chown -R minio-user:minio-user /path/to/minio/data

# Grant read/write/execute permissions to the owner
sudo chmod u+rxw /path/to/minio/data

3.3. Elasticsearch Keystore Permissions Issues

# If you get "will not overwrite keystore" or similar
sudo systemctl stop elasticsearch
sudo -u elasticsearch bin/elasticsearch-keystore add s3.client.default.access_key # Re-add keys
sudo -u elasticsearch bin/elasticsearch-keystore add s3.client.default.secret_key
sudo systemctl start elasticsearch

3.4. Network Connectivity Check (from Elasticsearch host to MinIO)

# Check if MinIO health endpoint is reachable
curl http://YOUR_MINIO_IP:9000/minio/health/live

# Check direct TCP connection to MinIO port
nc -vz YOUR_MINIO_IP 9000

3.5. Viewing Logs for Diagnostics

# View MinIO service logs
sudo journalctl -u minio -f

# View Elasticsearch service logs
sudo journalctl -u elasticsearch -f

Future Recommendations

While the current setup is fully functional, several enhancements can improve its security, performance, and robustness for more demanding environments.

  • Enhanced Security (TLS/SSL)

    Secure the MinIO server with HTTPS to encrypt data in transit. This involves generating TLS certificates and updating the Elasticsearch configuration to use the `https` protocol.

  • Fine-Grained MinIO Policies

    Replace the broad `readwrite` policy with a custom, least-privilege policy that only grants the specific S3 permissions required for snapshot operations.

  • Automated Snapshot Management

    Use Kibana’s Snapshot Lifecycle Management (SLM) policies to automate the creation, retention, and deletion of snapshots on a regular schedule.

  • Persistent External Storage

    Ensure the external drive for MinIO is mounted persistently via `/etc/fstab` to guarantee it’s available with the correct permissions after every system reboot.