Add helper to install ceph cluster

README.md

@@ -53,6 +53,54 @@ Template folder for creating new E2E tests. Contains a complete framework with:
 
 Use `./tests/create-test.sh <your-test-name>` to create a new test from this template.
 
+### Ceph testkit
+
+Reusable testkit that provisions a Rook-managed Ceph cluster through
+`sds-elastic` and, when needed, wires a csi-ceph-backed `StorageClass` on top.
+It is meant for downstream module e2e suites that need a Ceph backend without
+copying the cluster bootstrap code.
+
+Built around `testkit.EnsureCephStorageClass` (see
+[docs/FUNCTIONS_GLOSSARY.md](docs/FUNCTIONS_GLOSSARY.md#ceph-storageclass-testkit)),
+which handles: enabling `sds-node-configurator` + `sds-elastic` + `csi-ceph`
+modules, optionally provisioning a `sds-local-volume` Thick `StorageClass`
+for OSD backing, seeding `rook-config-override` (for things like
+`ms_crc_data=false`), creating Rook `CephCluster` + `CephBlockPool`, and
+wiring `CephClusterConnection` / `CephClusterAuthentication` /
+`CephStorageClass` csi-ceph CRs.
+
+`EnsureCephCluster` stops before the csi-ceph wiring and only brings up the
+Rook/Ceph side. Downstream repos (for example `csi-ceph`) can import
+`github.com/deckhouse/storage-e2e/pkg/testkit` and reuse these helpers inside
+their own Ginkgo specs.
+
+Testkit-specific env variables:
+
+- `CSI_CEPH_OSD_STORAGE_CLASS` — pre-existing block-mode StorageClass used to
+  back Rook OSD PVCs. When empty, a `sds-local-volume` Thick SC is
+  auto-provisioned via `EnsureDefaultStorageClass`.
+- `CSI_CEPH_MODULE_PULL_OVERRIDE` — image tag for `csi-ceph`'s
+  ModulePullOverride (dev registries only, e.g. when testing a PR build).
+
+#### `modulePullOverride` env templating
+
+Any module entry in `cluster_config.yml` may reference an env var with the
+`${VAR}` form in `modulePullOverride`. `storage-e2e` resolves those at config
+load time, so CI can point a module at a per-PR/MR image without editing the
+YAML between runs:
+
+```yaml
+dkpParameters:
+  modules:
+    - name: csi-ceph
+      modulePullOverride: "${MODULE_IMAGE_TAG}"  # CI must set MODULE_IMAGE_TAG, e.g. "pr131" on GitHub or "mr131" on GitLab
+```
+
+If a referenced env var is unset, `LoadClusterConfig` fails fast with
+`module "<name>" references env var ${VAR} in modulePullOverride but it is not set`
+instead of silently falling back to `main` — so a missing variable in CI is
+caught before bootstrap, not after a 30-minute wrong-image install.
+
 ### csi-all-stress-tests
 
 Stress tests for all CSI storage drivers. This test suite:

docs/ARCHITECTURE.md

@@ -23,6 +23,7 @@ storage-e2e/
 │   ├── config/                    # Configuration management
 │   │   ├── config.go             # Main configuration struct
 │   │   ├── env.go                # Environment variable parsing
+│   │   ├── overrides.go          # ${VAR} expansion in modulePullOverride at config load time
 │   │   ├── types.go              # Configuration type definitions
 │   │   └── images.go             # OS image definitions
 │   │
@@ -75,6 +76,12 @@ storage-e2e/
 │   ├── kubernetes/               # Public Kubernetes utilities
 │   │   ├── apply.go              # YAML manifest application
 │   │   ├── blockdevice.go        # BlockDevice operations
+│   │   ├── cephblockpool.go      # Rook CephBlockPool operations
+│   │   ├── cephcluster.go        # Rook CephCluster operations
+│   │   ├── cephfilesystem.go     # Rook CephFilesystem operations
+│   │   ├── cephclusterconnection.go # csi-ceph connection/auth CRs
+│   │   ├── cephcredentials.go    # Rook Ceph credential discovery
+│   │   ├── cephstorageclass.go   # csi-ceph CephStorageClass CR
 │   │   ├── client.go             # Clientset/dynamic client with retry
 │   │   ├── localstorageclass.go  # LocalStorageClass CR operations
 │   │   ├── lvmvolumegroup.go     # LVMVolumeGroup operations
@@ -83,17 +90,25 @@ storage-e2e/
 │   │   ├── nodegroup.go          # NodeGroup operations
 │   │   ├── nodes.go              # Node listing, taints, labels
 │   │   ├── pod.go                # Pod operations
+│   │   ├── pod_exec.go           # Pods/exec helpers + DistrolessReader for distroless containers
+│   │   ├── poll.go               # Generic readiness poller (per-call timeout, WARN on net errors)
 │   │   ├── pvc.go                # PVC operations
+│   │   ├── rookconfigoverride.go # Rook ceph.conf override ConfigMap
 │   │   ├── secrets.go            # Secret operations
 │   │   ├── storageclass.go       # StorageClass get/wait/default
+│   │   ├── storageclass_manage.go # Global default StorageClass management
 │   │   ├── virtualdisk.go        # VirtualDisk attach/detach
-│   │   └── vmpod.go              # VM pod lookup
+│   │   ├── vmpod.go              # VM pod lookup
+│   │   └── volumesnapshotclass.go # VolumeSnapshotClass helpers
 │   │
 │   ├── retry/                    # Generic retry with exponential backoff
 │   │   └── retry.go
 │   │
 │   └── testkit/                  # Test framework utilities
-│       ├── storageclass.go       # Default StorageClass provisioning
+│       ├── ceph.go               # EnsureCephStorageClass (Rook + csi-ceph)
+│       ├── ceph_cluster.go       # EnsureCephCluster (Rook only, no csi-ceph)
+│       ├── ceph_crc.go           # Ceph CRC tuning helpers
+│       ├── storageclass.go       # EnsureDefaultStorageClass (sds-local-volume)
 │       └── stress-tests.go       # Stress test runner
 │
 ├── tests/                         # Test suites
@@ -326,6 +341,7 @@ Tests use Ginkgo's lifecycle hooks:
 config/
 ├── config.go           # Main configuration operations
 ├── env.go              # Environment variable definitions and validation
+├── overrides.go        # ${VAR} expansion in modulePullOverride at config load time
 ├── types.go            # Configuration type definitions
 └── images.go           # OS image URL definitions
 ```
@@ -430,7 +446,7 @@ internal/kubernetes/               # Internal Kubernetes clients
 
 ```
 infrastructure/ssh/
-├── client.go           # SSH client implementation
+├── client.go           # SSH client implementation (Exec, ExecCapture, tunnels)
 ├── interface.go        # SSH client interface
 ├── tunnel.go           # Port forwarding and tunneling
 └── types.go            # SSH-related types
@@ -441,12 +457,14 @@ infrastructure/ssh/
 - SSH key handling
 - Port forwarding (e.g., for Kubernetes API access)
 - Remote command execution
+- Remote command execution with separated stdout/stderr capture for diagnostics
 - File transfer operations (including UploadPrivate: chmod-before-data for sensitive payloads)
 
 **Key Features**:
 - Support for password and key-based authentication
 - SSH tunneling for accessing remote Kubernetes clusters
 - Connection pooling and reuse
+- `ExecCapture` keeps stdout and stderr separate while preserving retry/reconnect behavior
 - Proper resource cleanup
 
 ### 3.5 Logger Module (`internal/logger/`)
@@ -486,6 +504,12 @@ pkg/
 ├── kubernetes/
 │   ├── apply.go        # YAML manifest application
 │   ├── blockdevice.go  # BlockDevice operations
+│   ├── cephblockpool.go         # Rook CephBlockPool CRUD + wait
+│   ├── cephcluster.go           # Rook CephCluster CRUD + wait
+│   ├── cephfilesystem.go        # Rook CephFilesystem CRUD + wait
+│   ├── cephclusterconnection.go # csi-ceph CephClusterConnection/Auth CRs
+│   ├── cephcredentials.go       # Read fsid/mons/admin-key from Rook secrets
+│   ├── cephstorageclass.go      # csi-ceph CephStorageClass CR
 │   ├── client.go       # Clientset/dynamic client with retry
 │   ├── localstorageclass.go  # LocalStorageClass CR operations
 │   ├── lvmvolumegroup.go     # LVMVolumeGroup operations
@@ -494,15 +518,23 @@ pkg/
 │   ├── nodegroup.go    # NodeGroup operations
 │   ├── nodes.go        # Node listing, taints, labels
 │   ├── pod.go          # Pod operations
+│   ├── pod_exec.go     # Exec helpers + DistrolessReader (ephemeral-container session)
+│   ├── poll.go         # pollResourceUntilReady helper for Wait*Ready callers
 │   ├── pvc.go          # PVC operations
+│   ├── rookconfigoverride.go    # Rook global ceph.conf override
 │   ├── secrets.go      # Secret operations
 │   ├── storageclass.go # StorageClass get/wait/default
+│   ├── storageclass_manage.go   # Global default-SC management
 │   ├── virtualdisk.go  # VirtualDisk attach/detach
-│   └── vmpod.go        # VM pod lookup
+│   ├── vmpod.go        # VM pod lookup
+│   └── volumesnapshotclass.go   # VolumeSnapshotClass helpers
 ├── retry/
 │   └── retry.go        # Generic retry with exponential backoff
 └── testkit/
-    ├── storageclass.go  # Default StorageClass provisioning
+    ├── ceph.go          # EnsureCephStorageClass / EnsureDefaultCephStorageClass
+    ├── ceph_cluster.go  # EnsureCephCluster (Rook-only, no csi-ceph)
+    ├── ceph_crc.go      # Ceph CRC tuning helpers
+    ├── storageclass.go  # EnsureDefaultStorageClass (sds-local-volume)
     └── stress-tests.go  # Stress test runner
 ```
 
@@ -730,7 +762,8 @@ logger.Error("Failed to create resource: %v", err)
 | `TEST_CLUSTER_VIRTUAL_MACHINE_CLASS_NAME` | `generic` | VM class for VMs on the base cluster in `alwaysCreateNew`. If set to another name (DNS-1123 subdomain) and the class does not exist, it is created from `generic` with `spec.cpu.type: Host`, **`spec.nodeSelector` / `spec.tolerations` cleared**, sizing policies retained from template, labeled `storage-e2e.deckhouse.io/auto-created=true`, and left after cleanup |
 | `TEST_CLUSTER_CLEANUP` | `false` | Cleanup cluster after tests |
 | `LOG_LEVEL` | `debug` | Log level (debug/info/warn/error) |
-| `KUBE_CONFIG_PATH` | - | Fallback kubeconfig path |
+| `KUBE_CONFIG_PATH` | - | Explicit kubeconfig path. Used when SSH retrieval of `/etc/kubernetes/{super-admin,admin}.conf` from the master fails. If unset and SSH also fails, `GetKubeconfig` returns an error (no silent fallback to `~/.kube/config`). |
+| `MODULE_IMAGE_TAG` (and any other custom name) | - | Any `${VAR}` placeholder used inside `modulePullOverride:` in `cluster_config.yml` is expanded at config load time by `internal/config/overrides.ExpandEnvInModulePullOverride`. Missing/empty placeholders fail fast with an explicit error so CI can point modules at `pr<N>` / `mr<N>` images via a single env var without editing the YAML between runs. |
 
 ### Commander Variables (only when `TEST_CLUSTER_CREATE_MODE=commander`)