Iron-Legion/documentation
2
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
43ed44e09afd2a31e51b83fd81d9a3856811d5e3
documentation/PRD Drafts/terraform-proxmox-lxc-automation.md

20 KiB
Raw Blame History

PRD: Terraform LXC Automation for Proxmox VE 9.2

Status: Draft — Pending Commander Bobby Review
Author: F.R.I.D.A.Y.
Date: 2026-06-01
Provider: bpg/proxmox (actively maintained, 11M+ downloads)
Target: Proxmox VE 9.2 / Debian Trixie

1. Purpose & Scope

This PRD defines the architecture, configuration patterns, and operational workflow for automating LXC container lifecycle management on Proxmox VE 9.2 clusters using Terraform and the actively maintained bpg/proxmox provider.

In scope:

  • Terraform provider configuration and authentication
  • LXC resource definitions (proxmox_virtual_environment_container)
  • Cloud-init / template-based provisioning
  • Network configuration (static IP, DHCP, bridge)
  • Storage allocation (rootfs on any PVE backend)
  • State management and CI/CD integration patterns

Out of scope:

  • VM (QEMU/KVM) provisioning
  • PVE cluster topology changes
  • Backup/restore automation (separate PRD)

2. Success Criteria

# Criterion How Verified
1 A single terraform apply creates a working LXC with SSH access ssh root@<lxc-ip> succeeds
2 LXCs are provisioned from official cloud-image templates Template downloaded via proxmox_virtual_environment_download_file
3 Network is configurable per-LXC (DHCP or static CIDR) ip addr inside container matches TF config
4 Rootfs lives on user-selected storage (not hardcoded to local-lvm) pvesm status shows volume on target datastore
5 State is stored remotely (S3-compatible or Terraform Cloud) terraform state list works from any machine
6 Destroy and recreate is idempotent terraform destroy && terraform apply yields identical result

3. Provider Selection

Why bpg/proxmox (not telmate/proxmox)

Provider Maintenance Downloads LXC Support Notes
bpg/proxmox ✅ Active (v0.108.0, June 2026) 11.8M+ Full Community-tier, comprehensive docs, supports PVE 9.x
telmate/proxmox ❌ Stale (last release ~2023) Legacy Partial Deprecated; lacks PVE 9.x features

Decision: Use bpg/proxmox exclusively. The telmate provider is unmaintained and incompatible with PVE 9.2 API changes.

Provider block (minimum):

terraform {
  required_providers {
    proxmox = {
      source  = "bpg/proxmox"
      version = "~> 0.108"
    }
  }
}

provider "proxmox" {
  endpoint = "https://192.168.7.33:8006/"
  username = "root@pam"
  password = var.proxmox_password  # or PROXMOX_VE_PASSWORD env var
  insecure = true                  # self-signed TLS
}

4. Authentication Matrix

Method Use Case Config Security
API Token Production, CI/CD api_token = "root@pam!mytoken=abc123…" Highest — revocable, fine-grained
Username/Password Development, one-offs username = "root@pam", password = "…" Medium — password in env
Auth Ticket TOTP-enabled accounts Pre-authenticate, pass ticket High — short-lived

Recommendation for Iron Legion:

  • Development: Use PROXMOX_VE_PASSWORD environment variable
  • CI/CD (future): Create a PVE API token with PVEFarmAdmin or custom role, store in CI secrets

5. Sample Project Structure

terraform-proxmox-lxc/
├── README.md
├── main.tf                 # Provider + backend config
├── variables.tf            # Input variables
├── terraform.tfvars.example  # Sample values (gitignored)
├── outputs.tf              # Useful outputs (IPs, IDs)
├── versions.tf             # Required providers + TF version
├── modules/
│   └── lxc/
│       ├── main.tf         # proxmox_virtual_environment_container resource
│       ├── variables.tf    # Module inputs
│       └── outputs.tf      # Module outputs
├── environments/
│   ├── dev/
│   │   ├── main.tf         # Calls modules with dev vars
│   │   └── terraform.tfvars
│   └── prod/
│       ├── main.tf
│       └── terraform.tfvars
└── templates/
    └── ubuntu-25.04-cloudimg.yaml   # Cloud-init user-data (optional)

Key Files

versions.tf

terraform {
  required_version = ">= 1.5.0"

  required_providers {
    proxmox = {
      source  = "bpg/proxmox"
      version = "~> 0.108"
    }
    random = {
      source  = "hashicorp/random"
      version = "~> 3.6"
    }
    tls = {
      source  = "hashicorp/tls"
      version = "~> 4.0"
    }
  }

  # Remote state — S3-compatible (Minio, Garage, AWS S3)
  backend "s3" {
    bucket       = "iron-legion-terraform"
    key          = "proxmox-lxc/terraform.tfstate"
    region       = "us-east-1"
    endpoint     = "https://s3.nb.bobbysh.me"
    use_path_style = true

    # Skip AWS-specific validations for self-hosted S3
    skip_credentials_validation = true
    skip_metadata_api_check     = true
    skip_region_validation      = true
    skip_requesting_account_id  = true
  }
}

variables.tf

variable "proxmox_endpoint" {
  description = "PVE API URL"
  type        = string
  default     = "https://192.168.7.33:8006/"
}

variable "proxmox_node" {
  description = "Target PVE node name"
  type        = string
  default     = "mk33"
}

variable "ssh_public_key" {
  description = "SSH public key for root access"
  type        = string
}

variable "lxc_configs" {
  description = "Map of LXC configurations"
  type = map(object({
    vm_id        = number
    hostname     = string
    cores        = optional(number, 2)
    memory       = optional(number, 2048)
    disk_size    = optional(number, 8)
    datastore_id = optional(string, "local-lvm")
    ip_address   = optional(string, "dhcp")
    gateway      = optional(string, null)
    template_url = optional(string, "https://mirrors.servercentral.com/ubuntu-cloud-images/releases/25.04/release/ubuntu-25.04-server-cloudimg-amd64-root.tar.xz")
    features = optional(object({
      nesting = optional(bool, true)
      fuse    = optional(bool, false)
      keyctl  = optional(bool, false)
    }), {})
  }))
}

modules/lxc/main.tf

resource "proxmox_virtual_environment_download_file" "lxc_template" {
  for_each = var.lxc_configs

  content_type = "vztmpl"
  datastore_id = "local"
  node_name    = var.proxmox_node
  url          = each.value.template_url
  file_name    = "${each.key}-template.tar.xz"
  overwrite    = false
}

resource "proxmox_virtual_environment_container" "lxc" {
  for_each = var.lxc_configs

  node_name    = var.proxmox_node
  vm_id        = each.value.vm_id
  description  = "Managed by Terraform — ${each.key}"

  unprivileged = true

  features {
    nesting = each.value.features.nesting
    fuse    = each.value.features.fuse
    keyctl  = each.value.features.keyctl
  }

  cpu {
    cores  = each.value.cores
    units  = 1024
  }

  memory {
    dedicated = each.value.memory
    swap      = 0
  }

  disk {
    datastore_id = each.value.datastore_id
    size         = each.value.disk_size
  }

  initialization {
    hostname = each.value.hostname

    ip_config {
      ipv4 {
        address = each.value.ip_address
        gateway = each.value.gateway
      }
    }

    user_account {
      keys     = [var.ssh_public_key]
      password = random_password.lxc_root[each.key].result
    }
  }

  network_interface {
    name   = "veth0"
    bridge = "vmbr0"
  }

  operating_system {
    template_file_id = proxmox_virtual_environment_download_file.lxc_template[each.key].id
    type             = "ubuntu"
  }

  startup {
    order      = "3"
    up_delay   = "60"
    down_delay = "60"
  }

  depends_on = [proxmox_virtual_environment_download_file.lxc_template]
}

resource "random_password" "lxc_root" {
  for_each = var.lxc_configs

  length           = 16
  special          = true
  override_special = "_%@"
}

modules/lxc/variables.tf

variable "proxmox_node" {
  type = string
}

variable "ssh_public_key" {
  type = string
}

variable "lxc_configs" {
  type = map(object({
    vm_id        = number
    hostname     = string
    cores        = optional(number, 2)
    memory       = optional(number, 2048)
    disk_size    = optional(number, 8)
    datastore_id = optional(string, "local-lvm")
    ip_address   = optional(string, "dhcp")
    gateway      = optional(string, null)
    template_url = optional(string)
    features = optional(object({
      nesting = optional(bool, true)
      fuse    = optional(bool, false)
      keyctl  = optional(bool, false)
    }), {})
  }))
}

modules/lxc/outputs.tf

output "lxc_ids" {
  description = "Map of LXC names to VM IDs"
  value       = { for k, v in proxmox_virtual_environment_container.lxc : k => v.vm_id }
}

output "lxc_ips" {
  description = "Map of LXC names to IPv4 addresses"
  value       = { for k, v in proxmox_virtual_environment_container.lxc : k => v.ipv4 }
}

output "lxc_passwords" {
  description = "Map of LXC names to root passwords (sensitive)"
  value       = { for k, v in random_password.lxc_root : k => v.result }
  sensitive   = true
}

environments/dev/main.tf

module "dev_lxcs" {
  source = "../../modules/lxc"

  proxxmox_node  = "mk33"
  ssh_public_key = file("~/.ssh/id_ed25519.pub")

  lxc_configs = {
    "dev-nextcloud" = {
      vm_id        = 2100
      hostname     = "dev-nextcloud"
      cores        = 4
      memory       = 4096
      disk_size    = 16
      datastore_id = "local-zfs"
      ip_address   = "192.168.7.100/24"
      gateway      = "192.168.7.1"
    }
    "dev-vaultwarden" = {
      vm_id        = 2101
      hostname     = "dev-vaultwarden"
      cores        = 2
      memory       = 2048
      disk_size    = 8
      datastore_id = "local-zfs"
      ip_address   = "192.168.7.101/24"
      gateway      = "192.168.7.1"
    }
  }
}

6. Resource Reference — proxmox_virtual_environment_container

Critical Arguments

Block Key Required Default Description
— node_name ✅ — PVE node to create on
— vm_id ✅ — Unique numeric ID (100–999999999)
— unprivileged ❌ true Run as unprivileged container
features nesting ❌ false Enable nested containers (needed for Docker-in-LXC)
features fuse ❌ false Enable FUSE mounts
cpu cores ❌ 1 vCPU cores
memory dedicated ❌ 512 RAM in MB
disk datastore_id ❌ local Storage pool for rootfs
disk size ❌ 4 Rootfs size in GB
initialization hostname ✅ — DNS-compatible hostname
initialization.ip_config.ipv4 address ✅ — CIDR or dhcp
initialization.ip_config.ipv4 gateway ❌ — Required for static IP
initialization.user_account keys ❌ — SSH authorized_keys
network_interface name ✅ — veth0
network_interface bridge ❌ vmbr0 Bridge to attach
operating_system template_file_id ✅ — Downloaded template or local:vztmpl/…
operating_system type ❌ unmanaged ubuntu, debian, alpine, etc.

Important Notes

  • Template download uses proxmox_virtual_environment_download_file — caches template per-node, avoids re-download
  • Cloud-init is embedded in the initialization block — no separate cloud-init drive needed for LXC
  • Nesting = true is required for any LXC running Docker or systemd-nspawn
  • Datastore is backend-agnostic: local-lvm, local-zfs, tank-zfs, ceph-rbd, NFS, etc. all work

7. Data Sources

Use data sources to query existing infrastructure without managing it:

data "proxmox_virtual_environment_datastores" "available" {
  node_name = var.proxmox_node
}

data "proxmox_virtual_environment_nodes" "cluster" {}

data "proxmox_virtual_environment_container" "existing" {
  node_name = var.proxmox_node  # or specify target node explicitly
  vm_id     = 2001
}

Common use cases:

  • Validate a datastore exists before creating a disk
  • Read an existing LXC’s IP to populate a DNS record (Technitium)
  • List nodes for multi-node placement logic

8. State Management

Recommended: S3-Compatible Backend

Iron Legion already runs self-hosted services. A Garage or Minio instance on a fleet storage node (e.g., Neo) can serve as the Terraform state backend:

terraform {
  backend "s3" {
    bucket       = "iron-legion-terraform"
    key          = "proxmox-lxc/dev.tfstate"
    region       = "us-east-1"
    endpoint     = "https://s3.nb.bobbysh.me"
    use_path_style = true

    skip_credentials_validation = true
    skip_metadata_api_check     = true
    skip_region_validation      = true
    skip_requesting_account_id  = true
  }
}

State Locking (Critical for Team Use)

Add a DynamoDB-compatible table or use a native locking mechanism. If S3 backend does not support locking, wrap terraform apply in a CI pipeline that serializes runs.

Optional: Atlantis Web UI for Terraform PR Automation

What Atlantis Is

Atlantis is a self-hosted web application that listens for webhook events from Git repositories and runs terraform plan / terraform apply automatically inside PR/MR workflows. It posts plan output back to the PR as comments, enforces approval gates, and locks workspaces to prevent concurrent applies.

Can Atlantis Manage LXC Resources via bpg/proxmox?

Yes. Atlantis is a Terraform orchestration layer, not a provider. It supports any Terraform provider including bpg/proxmox. The workflow is:

  1. Developer opens a PR adding/modifying .tf files defining LXC containers
  2. Atlantis receives the webhook and runs terraform plan in a isolated directory
  3. Plan output posted as a PR comment — team reviews before approval
  4. After approval (or atlantis apply comment), Atlantis runs terraform apply

Atlantis Docker Compose (Self-Hosted)

services:
  atlantis:
    image: ghcr.io/runatlantis/atlantis:latest
    ports:
      - "4141:4141"
    volumes:
      - ${HOME}/.ssh:/home/atlantis/.ssh:ro           # Git SSH key
      - /var/run/docker.sock:/var/run/docker.sock:ro # if using Docker TF provider
      - atlantis-data:/home/atlantis/.atlantis
    environment:
      ATLANTIS_GH_USER: "iron-legion-bot"              # or ATLANTIS_GITLAB_USER / ATLANTIS_GITEA_USER
      ATLANTIS_GH_TOKEN: "${ATLANTIS_GH_TOKEN}"        # personal access token
      ATLANTIS_REPO_ALLOWLIST: "github.com/Iron-Legion/*"
      ATLANTIS_GH_WEBHOOK_SECRET: "${WEBHOOK_SECRET}"
      # For Gitea:
      # ATLANTIS_GITEA_USER: "iron-legion-bot"
      # ATLANTIS_GITEA_TOKEN: "${GITEA_TOKEN}"
      # ATLANTIS_GITEA_WEBHOOK_SECRET: "${WEBHOOK_SECRET}"
    command: server
    restart: unless-stopped

    # Optional: Redis for distributed locking in multi-replica setups
    # redis:
    #   image: redis:8-alpine
    #   volumes:
    #     - redis-data:/data
    #   restart: always

volumes:
  atlantis-data:
    driver: local

Key Features

  • Plan Comments: Every PR gets an auto-generated terraform plan comment
  • Apply Locking: One apply at a time per workspace; concurrent PRs queue
  • Policy Checks: Integrate OPA (Open Policy Agent) or custom scripts to block non-compliant changes
  • Custom Workflows: Define per-repo or per-directory workflows (e.g., plan-only for dev, auto-apply for staging)
  • Self-Hosted SCM: Native webhook support for GitHub, GitLab, Bitbucket, and Gitea

Resource Footprint

  • Atlantis container: ~100–200 MB RAM, minimal CPU
  • Optional Redis: ~20 MB RAM
  • Total: fits comfortably on any Iron Legion node (MK7, MK33–42, Neo)

Gitea Integration Notes

  • Atlantis supports Gitea via the --gitea-user, --gitea-token, --gitea-webhook-secret flags
  • Must expose Atlantis endpoint to Gitea (Tailscale funnel, reverse proxy, or LAN if Gitea is in-network)
  • Webhook URL: http://atlantis-host:4141/events

9. Operational Workflow

Day 0 — Bootstrap

# 1. Clone repo
git clone ssh://git@100.99.123.16:2222/Iron-Legion/terraform-proxmox-lxc.git
cd terraform-proxmox-lxc/environments/dev

# 2. Set credentials
export PROXMOX_VE_PASSWORD="your-pve-password"
# OR for API token:
export PROXMOX_VE_API_TOKEN="root@pam!mytoken=abc123"

# 3. Initialize
terraform init

# 4. Plan
terraform plan -out=tfplan

# 5. Apply
terraform apply tfplan

Day N — Add a Container

  1. Add entry to lxc_configs map in environments/dev/main.tf
  2. terraform plan — review VM ID collision, IP conflict, storage capacity
  3. terraform apply
  4. Verify: ssh root@<new-ip>

Day N — Destroy a Container

  1. Remove entry from lxc_configs map
  2. terraform apply — resource destroyed
  3. Or: terraform destroy -target='module.dev_lxcs.proxmox_virtual_environment_container.lxc["dev-nextcloud"]'

10. Risks & Mitigations

Risk Likelihood Impact Mitigation
VM ID collision Medium High Maintain a fleet-wide VM ID registry; use proxmox_virtual_environment_vms data source to check
IP overlap with DHCP pool Medium High Reserve static IPs in Technitium DNS; use dns data source to verify
Template download fails (slow mirror) Low Medium Pre-seed templates on PVE nodes; use pvesm to verify before apply
State file corruption Low Critical S3 versioning + periodic terraform state pull backups
Privilege escalation via privileged LXC Low High Default unprivileged = true; explicit override required
Provider breaking change Medium Medium Pin provider version ~> 0.108; test upgrades in dev environment first

11. Open Questions

  1. Do we pre-create cloud-image templates on each PVE node, or let Terraform download per-node?

    • Per-node: slower first deploy, but self-contained
    • Pre-seeded: faster, requires manual pvesm or Ansible step

  2. Should LXCs register themselves in Technitium DNS via Terraform, or rely on DHCP + DNS integration?

    • Terraform can call a dns_a_record module (if Technitium provider exists)
    • Or: use PVE's built-in DHCP + DNSMASQ if configured

  3. CI/CD pipeline: GitHub Actions runner, or local Gitea Actions on the fleet SCM host?

    • Gitea Actions keeps secrets in-network
    • GitHub Actions requires Tailscale funnel or external exposure

  4. Do we want a dedicated LXC "Terraform runner" inside the cluster, or run from Artemis/operator workstation?

    • In-cluster runner: always has LAN access to PVE API
    • External: requires Tailscale or VPN for API reachability

12. Appendix

A. Provider Documentation Links

B. Useful PVE CLI Commands (for verification)

# List containers on a node
pct list

# List templates
pvesm list local --content vztmpl

# Check datastore usage
pvesm status

# Enter a container
pct enter <vm_id>

C. Terraform Commands Reference

terraform init          # Download providers, configure backend
terraform validate      # Syntax check
terraform plan          # Preview changes
terraform apply         # Execute changes
terraform destroy       # Tear down everything
terraform state list    # Show managed resources
terraform state show <addr>  # Show one resource's attributes
terraform output        # Display output values
terraform fmt -recursive  # Format all .tf files

End of PRD. Ready for Commander Bobby review and approval.

Reference in New Issue View Git Blame Copy Permalink