KopiaMaintenance CRD Reference

Overview and Introduction

What is KopiaMaintenance?

The KopiaMaintenance Custom Resource Definition (CRD) is a cluster-scoped resource that provides centralized, flexible management of Kopia repository maintenance operations in VolSync.

Key Benefits:

  • Centralized Management: Single point of control for maintenance across all namespaces

  • Advanced Matching: Sophisticated repository and namespace selection capabilities

  • Priority-Based Resolution: Automatic conflict resolution when multiple configurations target the same repository

  • Cross-Namespace Operations: Seamless secret management across namespace boundaries

  • Enhanced Flexibility: Support for both repository selector and direct repository modes

  • Better Resource Management: Optimized CronJob creation and lifecycle management

Why KopiaMaintenance was Created

Traditional maintenance configuration had several limitations:

  1. Limited Flexibility: Each ReplicationSource required its own maintenance configuration

  2. No Cross-Namespace Management: Maintenance was confined to individual namespaces

  3. Conflict Resolution: No systematic way to handle conflicting maintenance schedules

  4. Resource Duplication: Multiple ReplicationSources using the same repository created duplicate maintenance CronJobs

  5. Complex Administration: Difficult to manage maintenance policies at scale

The KopiaMaintenance CRD addresses these limitations by providing:

  • Cluster-wide maintenance policy management

  • Intelligent repository matching and deduplication

  • Priority-based conflict resolution

  • Cross-namespace secret management

  • Centralized monitoring and observability

When to Use KopiaMaintenance

Use KopiaMaintenance when:

  • Managing maintenance for multiple repositories across namespaces

  • Implementing organization-wide maintenance policies

  • Requiring advanced matching criteria (wildcards, labels, namespace selectors)

  • Setting up maintenance for repositories without ReplicationSources

  • Need priority-based conflict resolution for shared repositories

  • Centralizing maintenance operations for better observability

CRD Specification

Complete Field Reference

KopiaMaintenanceSpec Fields

repositorySelector (KopiaRepositorySelector, optional)

Repository matching configuration for finding existing ReplicationSources. This approach matches ReplicationSources by their repository configuration. Either repositorySelector OR repository must be specified, but not both.

repository (KopiaRepositorySpec, optional)

Repository defines a direct repository configuration for maintenance. This approach allows KopiaMaintenance to work independently of ReplicationSources. Either repositorySelector OR repository must be specified, but not both.

schedule (string, optional, default: “0 2 * * *”)

Cron schedule for when maintenance should run. The schedule is interpreted in the controller’s timezone. Must match the pattern for valid cron expressions.

enabled (bool, optional, default: true)

Determines if maintenance should be performed. When false, no maintenance will be scheduled.

priority (int32, optional, default: 0)

Priority of this maintenance configuration. When multiple KopiaMaintenance resources match the same repository, the one with the highest priority wins. Range: -100 to 100.

suspend (bool, optional)

Can be used to temporarily stop maintenance. When true, the CronJob will not create new Jobs, but existing Jobs will be allowed to complete.

successfulJobsHistoryLimit (int32, optional, default: 3)

Specifies how many successful maintenance Jobs should be kept. Minimum: 0.

failedJobsHistoryLimit (int32, optional, default: 1)

Specifies how many failed maintenance Jobs should be kept. Minimum: 0.

resources (ResourceRequirements, optional)

Compute resources required by the maintenance container. If not specified, defaults to 256Mi memory request and 1Gi memory limit.

podSecurityContext (PodSecurityContext, optional)

Security context for the entire maintenance pod.

securityContext (SecurityContext, optional)

Security context for the maintenance container.

nodeSelector (map[string]string, optional)

NodeSelector for scheduling maintenance pods on specific nodes.

tolerations ([]Toleration, optional)

Tolerations for scheduling maintenance pods.

affinity (Affinity, optional)

Affinity settings for pod scheduling.

KopiaRepositorySelector Fields

repository (string, required)

Repository name or pattern to match. Supports wildcards (* and ?). Use “*” to match all repositories.

namespaceSelector (NamespaceSelector, optional)

Criteria for selecting namespaces. If empty, all namespaces are considered.

NamespaceSelector Fields

matchNames ([]string, optional)

List of namespace names to include. Empty means all namespaces.

matchLabels (map[string]string, optional)

Label selector for namespaces. Only namespaces matching all labels will be considered.

KopiaRepositorySpec Fields

repository (string, required)

The name of a Secret containing repository configuration.

namespace (string, required)

The namespace containing the repository Secret.

customCA (CustomCASpec, optional)

Custom CA configuration for secure connections.

Operating Modes

Repository Selector Mode

This mode matches existing ReplicationSources by repository configuration patterns.

Key Features:

  • Pattern matching with wildcards (* and ?)

  • Namespace filtering via names or labels

  • Automatic discovery of ReplicationSources

  • Dynamic adaptation to changes

Example:

apiVersion: volsync.backube/v1alpha1
kind: KopiaMaintenance
metadata:
  name: all-production-repos
spec:
  repositorySelector:
    repository: "prod-*"  # Match all repositories starting with "prod-"
    namespaceSelector:
      matchLabels:
        environment: production
  schedule: "0 2 * * *"
  priority: 10

Direct Repository Mode

This mode allows direct repository configuration without requiring ReplicationSources.

Key Features:

  • Independent of ReplicationSource resources

  • Direct secret reference

  • Useful for maintenance-only scenarios

  • Simpler configuration for known repositories

Example:

apiVersion: volsync.backube/v1alpha1
kind: KopiaMaintenance
metadata:
  name: central-backup-maintenance
spec:
  repository:
    name: central-backup-config
    namespace: backup-system
  schedule: "0 3 * * 0"  # Weekly on Sunday at 3 AM
  resources:
    requests:
      memory: "512Mi"
    limits:
      memory: "2Gi"

Advanced Matching Criteria

Wildcard Patterns

Supported wildcard patterns for repository matching:

Pattern

Description

Example Matches

*

Match all repositories

Any repository

prod-*

Match prefix

prod-db, prod-app, prod-cache

*-backup

Match suffix

mysql-backup, redis-backup

app-?-data

Single character wildcard

app-1-data, app-2-data, app-a-data

*-prod-*

Multiple wildcards

us-prod-db, eu-prod-app

Namespace Selection

By Name:

namespaceSelector:
  matchNames:
    - production
    - staging
    - qa

By Labels:

namespaceSelector:
  matchLabels:
    environment: production
    team: platform

Combined Selection:

namespaceSelector:
  matchNames:
    - critical-apps
  matchLabels:
    backup-required: "true"

Priority-Based Conflict Resolution

When multiple KopiaMaintenance resources match the same repository, priority determines which one is used:

  1. Higher priority values win (range: -100 to 100)

  2. If priorities are equal, the first created resource wins

  3. Default priority is 0

Example Priority Hierarchy:

# Highest priority - specific override
apiVersion: volsync.backube/v1alpha1
kind: KopiaMaintenance
metadata:
  name: critical-app-override
spec:
  repositorySelector:
    repository: "critical-app-backup"
  priority: 50  # Highest priority
  schedule: "*/30 * * * *"  # Every 30 minutes

---
# Medium priority - department policy
apiVersion: volsync.backube/v1alpha1
kind: KopiaMaintenance
metadata:
  name: finance-dept-maintenance
spec:
  repositorySelector:
    repository: "finance-*"
  priority: 10
  schedule: "0 1 * * *"  # Daily at 1 AM

---
# Default priority - organization-wide
apiVersion: volsync.backube/v1alpha1
kind: KopiaMaintenance
metadata:
  name: global-maintenance
spec:
  repositorySelector:
    repository: "*"
  priority: 0  # Default
  schedule: "0 2 * * *"  # Daily at 2 AM

Best Practices

Organization Strategy

1. Hierarchical Configuration

Start with broad, low-priority configurations and add specific overrides:

  • Global default (priority: 0)

  • Environment-specific (priority: 10)

  • Department/team-specific (priority: 20)

  • Application-specific overrides (priority: 30+)

2. Naming Conventions

Use descriptive names that indicate scope and purpose:

  • global-maintenance-default

  • prod-env-maintenance

  • team-finance-maintenance

  • app-critical-db-override

Resource Management

Memory Configuration:

resources:
  requests:
    memory: "256Mi"  # Minimum for small repos
  limits:
    memory: "2Gi"    # Cap for large operations

Scheduling Best Practices:

  • Schedule maintenance during low-traffic periods

  • Stagger schedules for different repositories

  • Consider timezone implications

  • Use suspend feature for maintenance windows

Monitoring and Observability

Status Monitoring:

# Watch all KopiaMaintenance resources
kubectl get kopiamaintenance -w

# Check specific maintenance status
kubectl describe kopiamaintenance production-maintenance

CronJob Monitoring:

# View all maintenance CronJobs
kubectl get cronjobs -n volsync-system \
  -l volsync.backube/kopia-maintenance=true

# Check recent job executions
kubectl get jobs -n volsync-system \
  -l volsync.backube/kopia-maintenance=true \
  --sort-by=.metadata.creationTimestamp

Security Considerations

Cross-Namespace Secret Access

KopiaMaintenance can access secrets across namespaces. This requires proper RBAC configuration:

Required Permissions:

  • Read access to repository secrets in specified namespaces

  • Create/update permissions for CronJobs in volsync-system

  • List/watch permissions for ReplicationSources

Security Best Practices:

  1. Limit secret access to necessary namespaces only

  2. Use namespace selectors to restrict scope

  3. Regularly audit KopiaMaintenance resources

  4. Monitor cross-namespace access patterns

Pod Security

Recommended Security Context:

podSecurityContext:
  runAsNonRoot: true
  runAsUser: 1000
  fsGroup: 1000
  seccompProfile:
    type: RuntimeDefault

securityContext:
  allowPrivilegeEscalation: false
  capabilities:
    drop:
      - ALL
  readOnlyRootFilesystem: true

Troubleshooting

Common Issues

No Matching ReplicationSources

Symptoms: status.matchedSources is empty

Checks:

  1. Verify repository pattern matches actual names

  2. Check namespace selector criteria

  3. Ensure ReplicationSources exist in selected namespaces

  4. Review controller logs for matching details

CronJob Not Created

Symptoms: No CronJob in volsync-system namespace

Checks:

  1. Verify enabled: true in spec

  2. Check if higher priority maintenance exists

  3. Ensure repository secret exists and is accessible

  4. Review controller logs for errors

Maintenance Jobs Failing

Symptoms: Jobs in Failed state

Checks:

  1. Check job logs: kubectl logs -n volsync-system job/<job-name>

  2. Verify repository credentials are valid

  3. Check resource limits aren’t too restrictive

  4. Ensure network connectivity to repository

Debugging Commands

# List all maintenances with their matched sources
kubectl get kopiamaintenance -o custom-columns=\
  NAME:.metadata.name,\
  PRIORITY:.spec.priority,\
  MATCHED:.status.matchedSources[*].name

# Check controller logs for maintenance processing
kubectl logs -n volsync-system deployment/volsync | \
  grep -i kopiamaintenance

# View maintenance job history
kubectl get jobs -n volsync-system \
  -l volsync.backube/kopia-maintenance=true \
  -o custom-columns=\
  NAME:.metadata.name,\
  STATUS:.status.conditions[0].type,\
  START:.status.startTime,\
  COMPLETE:.status.completionTime

Special Cases

RepositoryPVC Not Supported

ReplicationSources using repositoryPVC cannot use KopiaMaintenance and must handle maintenance separately.

Multiple Repositories Same Namespace

When multiple repositories exist in the same namespace, use specific repository patterns:

# Target specific repository
repositorySelector:
  repository: "app-backup-config"  # Exact match
  namespaceSelector:
    matchNames: ["production"]

Complete Examples

Example 1: Global Default with Overrides

# Global default maintenance
apiVersion: volsync.backube/v1alpha1
kind: KopiaMaintenance
metadata:
  name: global-default
spec:
  repositorySelector:
    repository: "*"
  schedule: "0 2 * * *"
  priority: 0
  resources:
    requests:
      memory: "256Mi"
    limits:
      memory: "1Gi"

---
# Production override with higher frequency
apiVersion: volsync.backube/v1alpha1
kind: KopiaMaintenance
metadata:
  name: production-frequent
spec:
  repositorySelector:
    repository: "*"
    namespaceSelector:
      matchLabels:
        environment: production
  schedule: "0 */6 * * *"  # Every 6 hours
  priority: 20
  resources:
    requests:
      memory: "512Mi"
    limits:
      memory: "2Gi"

Example 2: Team-Based Maintenance

# Platform team repositories
apiVersion: volsync.backube/v1alpha1
kind: KopiaMaintenance
metadata:
  name: platform-team-maintenance
spec:
  repositorySelector:
    repository: "platform-*"
    namespaceSelector:
      matchLabels:
        team: platform
  schedule: "0 3 * * *"
  priority: 15
  nodeSelector:
    workload: maintenance
  tolerations:
    - key: maintenance
      operator: Equal
      value: "true"
      effect: NoSchedule

Example 3: Direct Repository Configuration

# Direct maintenance for specific repository
apiVersion: volsync.backube/v1alpha1
kind: KopiaMaintenance
metadata:
  name: central-backup-maintenance
spec:
  repository:
    name: central-backup-secret
    namespace: backup-system
    customCA:
      configMapName: custom-ca-bundle
      key: ca.crt
  schedule: "0 4 * * 0"  # Weekly on Sunday
  resources:
    requests:
      memory: "1Gi"
      cpu: "500m"
    limits:
      memory: "4Gi"
      cpu: "2"
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          preference:
            matchExpressions:
              - key: node-type
                operator: In
                values:
                  - maintenance