diff --git a/docs/cf-tunnel-mapping.md b/docs/cf-tunnel-mapping.md index f31928b..6a791d0 100644 --- a/docs/cf-tunnel-mapping.md +++ b/docs/cf-tunnel-mapping.md @@ -1,5 +1,13 @@ # Cloudflare-Tunnel — Mapping & #1033-Diagnose (Single Source of Truth) +> 🆕 **SSoT-Wechsel im Aufbau (TP44, seit 2026-09-02):** Die CF-Estate (Tunnel, WARP-Connector, +> VPC-Services, Hyperdrive, Workers, Access-Apps) wird als **native `nautobot.cloud`-Objekte** in +> Nautobot (`10.40.20.30`) geführt — Etappe 1 (manuelle Erstbefüllung) ✅. Seeder: +> `nautobot/tp36-cf-estate-seed.py`, Detail: `docs/handoff-20260902-cf-estate-nautobot-ssot.md`. +> **CF bleibt operativer Master, Nautobot = Dokumentations-SSoT.** Dieses File bleibt gültige Historie +> und As-Built-Log; die *strukturierte* Estate liest man ab jetzt in Nautobot. Etappe 2 (CF-API→Nautobot-Sync) +> ist offen. + > **Aktueller Stand: → §10 (2026-07-10, TP16 UMGESETZT ✅).** §9 = Analyse/Zielbild, §10 = As-Built. > Abschnitte 1–8 sind Historie (Track A, 2026-06-30); die „3 Connectors gut/Zielbild"-Aussagen dort sind > durch §9/§10 **überholt** (Round-Robin = macvlan-Host-Isolation → 1/N-Ausfälle). Track A aus `docs/handoff-cf-tunnel-1033.md`. diff --git a/docs/handoff-20260902-cf-estate-nautobot-ssot.md b/docs/handoff-20260902-cf-estate-nautobot-ssot.md new file mode 100644 index 0000000..82fc249 --- /dev/null +++ b/docs/handoff-20260902-cf-estate-nautobot-ssot.md @@ -0,0 +1,70 @@ +# Handoff 2026-09-02 — CF-Estate als SSoT in Nautobot (TP44, Etappe 1) + +> Scope: die Cloudflare-Infrastruktur (Tunnel + App-Ebene) als strukturierte, abfragbare SSoT in +> Nautobot abbilden. Etappe 1 = **manuelle Erstbefüllung** über die nativen `nautobot.cloud`-Modelle. +> Ausgelöst aus der DB-Pfad-Session (siehe `cf-tunnel-mapping.md` §17). + +## Entscheidung (User, 2026-09-02) + +- **SSoT-Modus:** Etappen — *manuell jetzt, Sync später*. `cf-tunnel-mapping.md` wird zum Snapshot/Zeiger. +- **Scope Etappe 1:** *volle CF-Estate* (Tunnel + WARP-Connector + VPC-Service + Hyperdrive + Workers + + alle Access-Apps). +- **Grundsatz:** Nautobot kann CF **nicht steuern** → „SSoT" = **Dokumentations**-SSoT. CF bleibt + operativer Master. + +## Warum ein eigenes TP (TP44), nicht TP36 + +TP36 ist in `teilprojekte.md` eng auf *`teilprojekte.md`-Live-Daten + Datei-Aufteilung* geschnitten. +Dieselbe Idee (Nautobot als Doku-SSoT), aber andere Dimension → **TP44** (nächste freie Nummer nach TP43). +Quick-Index-Zeile + TP-Abschnitt in `teilprojekte.md` sind **noch offen** (die Datei trug bei dieser +Session fremde, ungestagte Änderungen — bewusst nicht mit angefasst, um den Commit sauber zu halten). + +## Modell (Nautobot 3.1.7, App `nautobot.cloud`) + +``` +Manufacturer "Cloudflare" + └─ CloudAccount "Cloudflare" (account_number = 393c051f3accf51881d6c98fc5b66c36) + ├─ CloudResourceType ×6 + │ cloudflared Tunnel · WARP Connector → cloud.cloudnetwork + │ VPC Service · Hyperdrive · Worker · Access Application → cloud.cloudservice + ├─ CloudNetwork ×4 bf-public · fm.emsenhuber.net · terpene-db · bf-core-mesh + └─ CloudService ×14 VPC terpene-db · Hyperdrive terpene(ro)+terpene-rw · + Worker terpene+blaueblueten-n8n · 9× Access-App +``` + +- **IDs/Ports/Hostnames/Origins** liegen je Objekt in `extra_config` (JSON). +- **Service↔Tunnel-Verknüpfung** über das Through-Modell `cloud-service-network-assignments` (3 Rows): + `terpene-db↔terpene-db`, `*.emsenhuber.net↔bf-public`, `hass, vw, gitea↔bf-public`. +- **Prefix-Routen** (`10.40.0.0/16` bf-core-mesh, `10.101.0.0/24` fm) sind **nicht** als + `cloud-network-prefix-assignments` gesetzt, weil die Prefixe nicht in IPAM existieren → in + `extra_config.route` belassen (best-effort im Seeder, greift automatisch, sobald die Prefixe in IPAM sind). + +## Werkzeug + +`nautobot/tp36-cf-estate-seed.py` — **idempotenter** Seeder (get-or-create), Datenstand als Inline-Struktur. +Token aus `docs/nas-vlan20/stacks/210-nautobot.yml` (gitignored). Aufruf: Dry-Run default, +`--commit` schreibt. Re-Run = 26/26 „existiert". +*(Dateiname trägt noch `tp36`-Präfix aus dem ersten Arbeitsstand; inhaltlich TP44. Umbenennen bei Gelegenheit.)* + +## Gotchas (für Etappe 2 / Folgesessions) + +- 🔴 **`CloudService.cloud_networks` ist im Serializer `read_only` und bleibt bei GET leer** — die + Verknüpfung wird **ausschließlich** über `cloud-service-network-assignments` geschrieben UND gelesen. + Nicht am leeren Convenience-Feld verzweifeln; die Through-Rows sind die Wahrheit. +- VPC-Service-API liegt unter `connectivity/directory/services` (Update = **PUT**, voller Body), nicht `/vpc/`. +- Der Repo-`CF_API_TOKEN` darf `cfd_tunnel`, aber **nicht** Workers-VPC (`connectivity/directory`) — dort + über den `cloudflare`-MCP-Credential gehen. + +## Offen (Etappe 2, eigene Session) + +1. **CF-API→Nautobot-Sync** (Python/n8n): Inline-Daten im Seeder durch Live-Pull ersetzen, periodisch + (analog golden-config-bot). Dann ist Nautobot driftfrei. +2. **TP44 in `teilprojekte.md`** eintragen (Quick-Index + Abschnitt + Ist-Stand), sobald die dortigen + fremden Änderungen committet/geklärt sind. +3. IPAM-Prefixe `10.40.0.0/16` + `10.101.0.0/24` anlegen → dann greifen die Prefix-Assignments automatisch. +4. Seeder-Datei nach `tp44-…` umbenennen. + +## Verifikation (2026-09-02) + +Counts: CloudAccount 1 · ResourceType 6 · CloudNetwork 4 · CloudService 14 · Service-Network-Assignments 3. +Idempotenz-Gegenprobe grün. diff --git a/nautobot/tp36-cf-estate-seed.py b/nautobot/tp36-cf-estate-seed.py new file mode 100644 index 0000000..33f8236 --- /dev/null +++ b/nautobot/tp36-cf-estate-seed.py @@ -0,0 +1,294 @@ +#!/usr/bin/env python3 +""" +TP36 — Cloudflare-Estate als SSoT in Nautobot (Etappe 1: manuelle Erstbefuellung). + +Idempotenter Seeder (get-or-create) fuer die native nautobot.cloud-Modellwelt: + Manufacturer "Cloudflare" + -> CloudAccount "Cloudflare" (CF-Account-ID) + -> CloudResourceType (6x: cloudflared-Tunnel, WARP-Connector, VPC-Service, + Hyperdrive, Worker, Access-Application) + -> CloudNetwork (Tunnel + Mesh) + -> CloudService (VPC-Service, Hyperdrive, Worker, Access-App) + +Datenstand: Live-CF-Inventur 2026-09-02 (siehe docs/cf-tunnel-mapping.md). +CF bleibt operativer Master; Nautobot ist Dokumentations-SSoT. Etappe 2 (eigene +Session) ersetzt die Inline-Daten durch einen CF-API->Nautobot-Sync. + +Token-Quelle: docs/nas-vlan20/stacks/210-nautobot.yml (gitignored), +Schluessel NAUTOBOT_SUPERUSER_API_TOKEN. + +Aufruf: + python3 nautobot/tp36-cf-estate-seed.py # Dry-Run (default) + python3 nautobot/tp36-cf-estate-seed.py --commit # legt an +""" +import json +import os +import sys +import urllib.request +import urllib.parse +import urllib.error +from pathlib import Path + +NB = os.environ.get("NAUTOBOT_URL", "http://10.40.20.30:8080") +REPO = Path(__file__).resolve().parent.parent +YML = REPO / "docs" / "nas-vlan20" / "stacks" / "210-nautobot.yml" +CF_ACCOUNT_ID = "393c051f3accf51881d6c98fc5b66c36" +COMMIT = "--commit" in sys.argv + + +def load_token() -> str: + import re + for line in YML.read_text().splitlines(): + m = re.search(r"NAUTOBOT_SUPERUSER_API_TOKEN[=:]\s*([^\s\"']+)", line) + if m: + return m.group(1) + sys.exit(f"NAUTOBOT_SUPERUSER_API_TOKEN nicht in {YML} gefunden") + + +TOKEN = load_token() +HDRS = {"Authorization": f"Token {TOKEN}", "Accept": "application/json", + "Content-Type": "application/json"} + + +def req(method: str, path: str, body=None): + url = f"{NB}{path}" + data = json.dumps(body).encode() if body is not None else None + r = urllib.request.Request(url, data=data, headers=HDRS, method=method) + try: + with urllib.request.urlopen(r, timeout=20) as resp: + return resp.status, json.loads(resp.read().decode() or "{}") + except urllib.error.HTTPError as e: + return e.code, json.loads(e.read().decode() or "{}") + + +def find(endpoint: str, name: str): + q = urllib.parse.urlencode({"name": name, "depth": 0}) + st, d = req("GET", f"/api/{endpoint}/?{q}") + for row in d.get("results", []): + if row.get("name") == name: + return row + return None + + +def get_or_create(endpoint: str, name: str, payload: dict, label: str): + existing = find(endpoint, name) + if existing: + print(f" = {label}: '{name}' existiert ({existing['id']})") + return existing + if not COMMIT: + print(f" + {label}: '{name}' [DRY-RUN]") + return {"id": f""} + st, d = req("POST", f"/api/{endpoint}/", payload) + if st in (200, 201): + print(f" + {label}: '{name}' angelegt ({d.get('id')})") + return d + print(f" ! {label}: '{name}' FEHLER {st}: {json.dumps(d)[:300]}") + return None + + +def existing_pairs(endpoint: str, a: str, b: str): + """Menge der (a_id, b_id)-Paare eines Through-Modells (depth=0).""" + st, d = req("GET", f"/api/{endpoint}/?limit=0&depth=0") + out = set() + for row in d.get("results", []): + av = row.get(a) + bv = row.get(b) + av = av.get("id") if isinstance(av, dict) else av + bv = bv.get("id") if isinstance(bv, dict) else bv + if av and bv: + out.add((av, bv)) + return out + + +def link(endpoint: str, a: str, av, b: str, bv, seen: set, label: str): + if not av or not bv: + return + if (av, bv) in seen: + print(f" = {label} existiert") + return + if not COMMIT: + print(f" + {label} [DRY-RUN]") + seen.add((av, bv)) + return + st, d = req("POST", f"/api/{endpoint}/", {a: av, b: bv}) + if st in (200, 201): + print(f" + {label} angelegt") + seen.add((av, bv)) + else: + print(f" ! {label} FEHLER {st}: {json.dumps(d)[:200]}") + + +def find_prefix(prefix: str): + q = urllib.parse.urlencode({"prefix": prefix}) + st, d = req("GET", f"/api/ipam/prefixes/?{q}") + for row in d.get("results", []): + if row.get("prefix") == prefix: + return row["id"] + return None + + +# ---------------------------------------------------------------- Inventar +NETWORKS = [ + ("bf-public", "Cloudflare cloudflared Tunnel", + "Public-Ingress (Caddy 10.40.20.40:443), 3 Connectoren auf ap1/ap2/ap3 (vlan99)", + {"tunnel_id": "48d3b0b6-7233-4781-a76a-1de4a5e957ce", "kind": "cloudflared", + "connectors": "3x cloudflared @ ap1/ap2/ap3 (vlan99)", + "ingress_hostnames": ["vw.emsenhuber.net", "hass.emsenhuber.net", + "n8n.emsenhuber.net", "gitea.emsenhuber.net"]}), + ("fm.emsenhuber.net", "Cloudflare cloudflared Tunnel", + "Standort FM, Route 10.101.0.0/24", + {"tunnel_id": "b697d124-53bf-4a33-8fcf-384cced4db1a", "kind": "cloudflared", + "route": "10.101.0.0/24", "site": "FM"}), + ("terpene-db", "Cloudflare cloudflared Tunnel", + "DB-Pfad Terpenfinder; Connector terpene-cloudflared @ LXC400 (vlan20)", + {"tunnel_id": "f2d2fc8a-54ae-41df-b09a-97312cdb9fe2", "kind": "cloudflared", + "connector": "terpene-cloudflared @ LXC400 (vlan20)", + "note": "2026-09-02 neu aufgebaut (TP16.3-§6-Regression), Alttunnel b83b68f0 soft-deleted"}), + ("bf-core-mesh", "Cloudflare WARP Connector", + "Native WireGuard-Mesh auf crs326-core (TP16.1), Route 10.40.0.0/16", + {"tunnel_id": "d1da7c8a-2736-4cbb-8af1-32db5393fb9c", "kind": "warp_connector", + "route": "10.40.0.0/16", "host": "crs326-core"}), +] + +# service: (name, resource_type, description, cloud_network_names[], extra_config) +SERVICES = [ + ("terpene-db", "Cloudflare VPC Service", + "TCP-VPC-Service auf platform-pg (10.40.20.20:5432)", ["terpene-db"], + {"service_id": "019fa3d5-a11f-7591-8660-55cec192ca96", "type": "tcp", + "tcp_port": 5432, "ipv4": "10.40.20.20", "app_protocol": "postgresql", + "tls": "disabled"}), + ("terpene (ro)", "Cloudflare Hyperdrive", + "Hyperdrive read-only ueber VPC-Service terpene-db", [], + {"id": "e2023581159e44a4a0cdebb5ddfbea57", "database": "terpene", + "user": "terpene_ro", "mode": "ro", "caching": True, + "vpc_service_id": "019fa3d5-a11f-7591-8660-55cec192ca96"}), + ("terpene-rw", "Cloudflare Hyperdrive", + "Hyperdrive read-write ueber VPC-Service terpene-db", [], + {"id": "0b57bc4c958b49319e8af5045eba1ccf", "database": "terpene", + "user": "terpene_rw", "mode": "rw", "caching": True, + "vpc_service_id": "019fa3d5-a11f-7591-8660-55cec192ca96"}), + ("terpene", "Cloudflare Worker", + "Terpenfinder PWA + /api, Custom Domain 420.teilzeitladen.xyz", [], + {"script": "terpene", "custom_domain": "420.teilzeitladen.xyz", + "hyperdrive_bindings": ["terpene (ro)", "terpene-rw"], + "vpc_service": "terpene-db"}), + ("blaueblueten-n8n", "Cloudflare Worker", + "Worker blaueblueten-n8n", [], + {"script": "blaueblueten-n8n"}), + # ---- Access Applications (9) ---- + ("App Launcher", "Cloudflare Access Application", "Access App Launcher", [], + {"id": "6df5c431-adcf-4dde-805d-4eb22d7c19b4", "type": "app_launcher", + "domain": "emsenhuber.cloudflareaccess.com"}), + ("Terpenfinder v3", "Cloudflare Access Application", + "Self-hosted Access-App vor 420.teilzeitladen.xyz", [], + {"id": "e6a7991f-466c-4d1d-bac6-4b30fafccce4", "type": "self_hosted", + "domain": "420.teilzeitladen.xyz"}), + ("*.emsenhuber.net", "Cloudflare Access Application", + "Wildcard self-hosted Access-App", ["bf-public"], + {"id": "16aaec78-5951-4cf3-aad5-79f63163b5ea", "type": "self_hosted", + "domain": "*.emsenhuber.net"}), + ("hass, vw, gitea", "Cloudflare Access Application", + "Self-hosted Access-App (vw/hass/gitea/n8n via bf-public)", ["bf-public"], + {"id": "1dbc45ca-1376-4a0d-b608-69db58f1875c", "type": "self_hosted", + "domain": "vw.emsenhuber.net"}), + ("Terpenfinder (bookmark)", "Cloudflare Access Application", + "Bookmark-Kachel im App Launcher", [], + {"id": "6ce96376-cb0f-4b84-9c49-00d4ef2d1e80", "type": "bookmark", + "domain": "https://420.teilzeitladen.xyz"}), + ("Home Assistant (bookmark)", "Cloudflare Access Application", + "Bookmark-Kachel im App Launcher", [], + {"id": "a39dae4b-4fa7-4050-ac2e-07ecab2c0101", "type": "bookmark", + "domain": "https://hass.emsenhuber.net"}), + ("Vaultwarden (bookmark)", "Cloudflare Access Application", + "Bookmark-Kachel im App Launcher", [], + {"id": "6ce34ce2-53e6-409d-aa0b-a660b8107d64", "type": "bookmark", + "domain": "https://vw.emsenhuber.net"}), + ("Warp Login App", "Cloudflare Access Application", "WARP-Login-App", [], + {"id": "6c32fe23-24d8-433f-b00b-0a3d1558bb30", "type": "warp", + "domain": "emsenhuber.cloudflareaccess.com/warp"}), + ("Terpenfinder v3 assetlinks", "Cloudflare Access Application", + "assetlinks.json bypass fuer Android TWA", [], + {"id": "86287661-7b23-4f38-874c-926cab1c0535", "type": "self_hosted", + "domain": "420.teilzeitladen.xyz/.well-known/assetlinks.json"}), +] + +RESOURCE_TYPES = [ + ("Cloudflare cloudflared Tunnel", "cloud.cloudnetwork"), + ("Cloudflare WARP Connector", "cloud.cloudnetwork"), + ("Cloudflare VPC Service", "cloud.cloudservice"), + ("Cloudflare Hyperdrive", "cloud.cloudservice"), + ("Cloudflare Worker", "cloud.cloudservice"), + ("Cloudflare Access Application", "cloud.cloudservice"), +] + + +def main(): + print(f"== TP36 CF-Estate-Seed ({'COMMIT' if COMMIT else 'DRY-RUN'}) @ {NB} ==") + + print("\n[1] Manufacturer") + mfr = get_or_create("dcim/manufacturers", "Cloudflare", {"name": "Cloudflare"}, "Manufacturer") + mfr_id = mfr["id"] if mfr else None + + print("\n[2] CloudAccount") + acct = get_or_create("cloud/cloud-accounts", "Cloudflare", + {"name": "Cloudflare", "account_number": CF_ACCOUNT_ID, + "provider": mfr_id}, "CloudAccount") + acct_id = acct["id"] if acct else None + + print("\n[3] CloudResourceTypes") + rt_ids = {} + for name, ct in RESOURCE_TYPES: + rt = get_or_create("cloud/cloud-resource-types", name, + {"name": name, "provider": mfr_id, "content_types": [ct]}, + "ResourceType") + rt_ids[name] = rt["id"] if rt else None + + print("\n[4] CloudNetworks") + net_ids = {} + for name, rt_name, desc, extra in NETWORKS: + net = get_or_create("cloud/cloud-networks", name, + {"name": name, "cloud_resource_type": rt_ids.get(rt_name), + "cloud_account": acct_id, "description": desc, + "extra_config": extra}, "CloudNetwork") + net_ids[name] = net["id"] if net else None + + print("\n[5] CloudServices") + svc_ids = {} + svc_nets = {} + for name, rt_name, desc, net_names, extra in SERVICES: + svc = get_or_create("cloud/cloud-services", name, + {"name": name, "cloud_resource_type": rt_ids.get(rt_name), + "cloud_account": acct_id, "description": desc, + "extra_config": extra}, "CloudService") + svc_ids[name] = svc["id"] if svc else None + svc_nets[name] = net_names + + print("\n[6] Service<->Network-Assignments") + seen_sn = existing_pairs("cloud/cloud-service-network-assignments", + "cloud_network", "cloud_service") if COMMIT else set() + for name, net_names in svc_nets.items(): + for n in net_names: + link("cloud/cloud-service-network-assignments", + "cloud_network", net_ids.get(n), "cloud_service", svc_ids.get(name), + seen_sn, f"{name} <-> {n}") + + print("\n[7] Network<->Prefix-Assignments (best effort, nur wenn IPAM-Prefix existiert)") + seen_np = existing_pairs("cloud/cloud-network-prefix-assignments", + "cloud_network", "prefix") if COMMIT else set() + for name, rt_name, desc, extra in NETWORKS: + route = extra.get("route") + if not route: + continue + pid = find_prefix(route) + if not pid: + print(f" ~ {name}: Prefix {route} nicht in IPAM -> in extra_config belassen") + continue + link("cloud/cloud-network-prefix-assignments", + "cloud_network", net_ids.get(name), "prefix", pid, + seen_np, f"{name} <-> {route}") + + print(f"\n== fertig ({'COMMIT' if COMMIT else 'DRY-RUN — nichts geschrieben'}) ==") + + +if __name__ == "__main__": + main()