For a while now I’ve been looking to implement some kind of observability platform. I’ve been using a combination of Loki and Grafana for a while now, so utilising Grafana is preferred.

Some of the goals for this project were:

• ingest data from proxmox
• get data from guest VMs
• create a dashboard that shows overall performance that can be drilled down

After some investigation into some tools, I’ve landed on using:

• Grafana for dashboard, because it’s already in my stack
• InfluxDB2 for storage, because Proxmox supports feeding data to it natively
• Sensu for sending metrics, because I already use it for monitoring

Setting up InfluxDB and Proxmox Metrics

First thing was to spin up InfluxDB. This was easily done by using the influxdb container on dockerhub. I created a deployment for my k8s cluster, I have /var/lib/influxdb2 mapped to my TrueNAS box via NFS. I created an nginx ingress and it was available at influxdb.lab.home in minutes. I didn’t bother configuring the container with environment variables, given the config is persistent I’d rather avoid storing credentials in any config files.

Once InfluxDB was installed I headed over to my Proxmox box. Going to Data center > Metrics Server > Add > InfluxDB. I populated the information. Port needed to be 443 and protocol HTTPS. You can get the token from the API tokens menu in the interface. I opted to create a custom token that can only read and write to the Proxmox bucket, rather than the whole organisation. To check Proxmox is sending data, in the InfluxDB interface go to the data explorer, you should see some items appear in the proxmox bucket.

Setting up Sensu to send metrics to InfluxDB

It looks like data is flowing through from Proxmox, let’s get some data from Sensu into InfluxDB for each guest VM.

There is a handler that you can use from Bonsai that will be able to send metrics to InfluxDB. I created a handler and a metrics check. I’ll admit, this took me ages to figure this out and get it working. In case someone reads this and has the same problem, take the org name and bucket name out of the quotes. For reference, this is what the handler looks like:

api_version: core/v2
type: Handler
metadata:
  namespace: default
  name: influxdb
spec:
  type: pipe
  command: sensu-influxdb-handler --insecure-skip-verify
  timeout: 10
  env_vars:
  - INFLUXDB_ADDR=https://influxdb.lab.home:443
  - INFLUXDB_TOKEN=TOKEN HERE
  - INFLUXDB_BUCKET=sensu-metrics
  - INFLUXDB_ORG=homelab
  filters:
  #- has_metrics
  runtime_assets:
  - sensu/sensu-influxdb-handler

Having the token inside the YAML is not ideal, so I’d like to figure this out later.

Another annoying thing is the formats need to be adjusted in every check. My memory check worked ok but because the CPU check output in a different format it required some reading of the documentation to find the correct one.

Building some fancy graphs with Grafana

Now that all the data I want is going to InfluxDB, lets visualise it using Grafana.

Once logged into Grafana I created a new InfluxDB data source. It appears that Flux will be decommissioned in the near future so I setup using FluxQL. Set the HTTPS URL and added a custom HTTP Header with header as Authorization and value as Token MY_TOKEN. I did get a cert not trusted error so I mapped my CA configmap to /etc/ssl/certs/homelab.crt. Once this was done Grafana was able to connect to InfluxDB!

There are so many useful dashboards on the Grafana website, I used this one which has everything I need.

In the future I want to add more data into InfluxDB including pfSense, TrueNAS and Cisco IOS data.