Home Infrastructure Overview

Home Infrastructure Overview

Welcome to a comprehensive overview of my home infrastructure. In this post, I’ll cover various aspects including physical hardware, software running on different servers, VMs, and Kubernetes clusters.

Physical Things

Location and Hosting

• Basement All physical servers are located in the basement for optimal cooling and accessibility.
• Hetzner Cloud Remote services hosted on Hetzner’s cloud infrastructure.

Physical Servers

Network Stack

• Primary Router / Firewall / DHCP: Mikrotik RB4011iGS+
• Switches: Dell PowerConnect 5548, Mikrotik CRS309-1G-8S+, Mikrotik CRS112-8P-4S
• Wi-Fi Access Points: 2x TP-Link Omada EAP660 HD (managed by a TP-Link OC200)
• DNS/DHCP: 2x Technitium DNS + 1x Pi-Hole as upstream

Power

• UPS: Dell branded APC UPS, 1920W, ~30-60min runtime under current load (~100+ if the R730xd is shut down on power-loss detection)

Software Layer

What’s Running on Which Physical Server

• R620: Proxmox VE, offline for now
• R730xd: Proxmox VE, running the VMs for “Basement vCPU Cluster” (bcc)
• R320: Proxmox Backup Server, offline for now
• MD3200: External SAS storage array, offline for now, only turned on for occasional backups
• d1: Proxmox VE, clustered together with d2, running the primary DNS server
• d2: Proxmox VE, clustered together with d1, running the secondary DNS server and an XP VM
• raspi-1: Raspbian, running HomeAssistant, Grafana, InfluxDB v2, Prometheus
• kpi-1..3: Raspbian, used to run a k3s cluster, offline for now (until Talos linux fully supports the pi 5)

Virtual Machines (VMs)

R730xd

d1 & d2

Hetzner

Other Hetzner resources

• Load Balancer for the hcc cluster
• Object Storage (buckets for everything): Longhorn backups, Loki logs, Omni / k8s etcd backups

Kubernetes Clusters

Home Cluster (BCC)

• Setup & Bootstrapping: VMs created and managed with opentofu using the proxmox ve provider, kubernetes bootstrapping done with Siderolabs Omni / Talos Linux
• Workloads (Managed using Flux for GitOps):
  • apps
    • ai
      • ollama
      • open-webui
      • speaches (tts and stt)
      • wyoming (tts+stt integration for HomeAssistant)
    • authentik ldap outpost
    • cnpg operator
    • gitlab runners
      • unprivileged instance runner
      • privileged group runner for oci image building with buildah
    • immich
    • media
      • sonarr
      • radarr
      • lidarr
      • jackett
      • flaresolverr
      • prowlarr
      • jellyseerr
      • jellyfin
    • renovate bot
    • sonarqube
    • valkey operator
  • infra
    • flux
    • cert-manager
    • cilium
    • external-secrets operator (with secret stores for both gcp and openbao)
    • external-dns
    • gitlab-agent
    • longhorn
    • metallb
    • nvidia-device-plugin
    • monitoring
      • kube-prometheus-stack
      • loki and alloy (wip)
    • nfs-provisioner
    • traefik for ingress (external and internal ingress classes)

Hetzner Cluster (HCC)

• Setup & Bootstrapping: VPSs created and managed with opentofu using the hetzner provider, kubernetes bootstrapping done with Siderolabs Omni / Talos Linux
• Workloads (Managed using Flux for GitOps):
  • apps
    • authentik (identity provider for almost everything, except omni)
    • openbao
    • this blog
    • cnpg operator
    • cyberchef
    • influxdb
    • joplin server
    • redlib instance
  • infra
    • flux
    • cert-manager
    • cilium
    • external-secrets operator (with secret stores for both gcp and openbao)
    • external-dns
    • gitlab-agent
    • kyverno
    • longhorn
    • monitoring
      • kube-prometheus-stack
      • loki and alloy (wip)
    • traefik for ingress (one instance, external only)

Conclusion

This overview gives a snapshot of my home infrastructure, covering both physical and software aspects. Future posts will delve deeper into specific components like Kubernetes configurations and VM setups.

Feel free to reach out if you have any questions or want more details on any part of the setup!