# Telemetry Generation for UCA Demos

## Overview
Generate realistic telemetry CSVs for CloudZero usage-based cost allocation demos. Compatible with [CloudZero UCA Tools](https://github.com/Cloudzero/cloudzero-uca-tools/blob/develop/docs/quick_starts/quick_start_allocation_telemetry/explanation_of_allocation_data.md).

## Multi-Pass Generation

**Pass 1: Even Distribution** → All elements start equal
**Pass 2: Dominant Element** → One element gets 60% share per hour
**Pass 3: Variance** → Apply ±50% noise for realism
**Pass 4: Anomalies** → Calculate target shares for cost isolation, then apply spikes/steps

## Configuration

```yaml
telemetry_generation:
  start_date: "2025-01-01"
  end_date: "2025-03-31"

  # Optional: Control volume scale (default: 1000.0)
  typical_hourly_volume: 10000.0

  # Optional: Control variance (default: 50.0)
  variance_pct: 30.0

  elements:
    api-auth: null
    api-search:
      is_dominant: true
      anomaly_type: "hourly_spike"
      cost_multiplier: 6.0
    api-analytics: null
    api-reports: null
    api-notifications: null
```

**Requirements:**
- Minimum 5 elements
- At most one dominant element
- At most one element with anomaly (clear causality)
- Anomaly types: `daily_spike`, `hourly_spike`, `daily_step_up`, `daily_step_down`, `hourly_step_up`, `hourly_step_down`

## Cost Filtering

Define which costs to allocate using shared_spend (allocation telemetry) or associated_cost (unit metrics).

### Allocation Telemetry (shared_spend)

Specify subset of shared costs to distribute across elements:

```yaml
telemetry_generation:
  shared_spend:
    - "Azure-SQL-Production"
    - "Subscription-A"
    - "prod-resource-group"
```

**CSV Output:**
```csv
timestamp,element_id,unit_value,shared_spend_1,shared_spend_2,shared_spend_3
2025-01-01T00:00:00Z,api-auth,26.7,Azure-SQL-Production,Subscription-A,prod-resource-group
```

**CloudZero Mapping**: Each column becomes a filter defining the cost subset:
- `shared_spend_1` → Cost pool or dimension group
- `shared_spend_2` → Additional filter (subscription, account, etc.)
- `shared_spend_3` → Further refinement (resource group, region, etc.)

**Common Patterns:**
- Single pool: `["Lambda-Data-Processing"]` (default if not specified)
- Pool + filter: `["Azure-SQL-Production", "Subscription-A"]`
- Precise scope: `["Azure-Compute", "prod-subscription", "us-east-1", "prod-rg"]`

### Unit Metric Telemetry (associated_cost)

Specify costs associated with unit metrics for cost-per-unit analysis:

```yaml
telemetry_generation:
  associated_cost:
    - "Lambda-Function-Group-A"
    - "us-east-1"
```

**CSV Output:**
```csv
timestamp,element_id,unit_value,associated_cost_1,associated_cost_2
2025-01-01T00:00:00Z,transaction-123,450,Lambda-Function-Group-A,us-east-1
```

**Use Case**: Track cost per transaction, cost per API call, etc.

**Note**: Use either `shared_spend` OR `associated_cost`, not both in one config.

## Parameters

### typical_hourly_volume
Controls order of magnitude for telemetry values. Choose based on business context:
- `100` - Small SaaS (hundreds of units/hour)
- `1000` - Medium platform (thousands) **[default]**
- `10000` - High-volume service (tens of thousands)
- `1000000` - Enterprise scale (millions)

**Example**: If dominant element should show ~30k API calls/hour, set `typical_hourly_volume: 10000`

### variance_pct
Noise applied to all elements (default: 50.0 = ±50%)
- Higher = more dramatic variation
- Lower = smoother curves
- Set to 0 to test anomalies without noise

### is_dominant
Mark one element to receive 60% share of each hour's total
- If not specified, all elements get varied shares (10-30%)
- Multiple dominant elements → validation error

### anomaly_type
Per-element anomaly pattern (only one element can have an anomaly):
- `daily_spike` - 3-day spike, mid-month
- `hourly_spike` - 4-hour spike, mid-month 2pm

**Note**: Step anomalies (`daily_step_up`, `daily_step_down`, `hourly_step_up`, `hourly_step_down`) require cumulative cost multipliers and will be supported in a future release.

### cost_multiplier
**Required for spike anomalies**

Specifies how much the shared resource's total cost increases during the anomaly window. This drives the cost isolation calculation.

**How it works:**
1. System captures baseline telemetry shares from Pass 3 (e.g., api-search: 60%, others: 40%)
2. You specify the cost increase (e.g., `cost_multiplier: 6.0` means cost goes from $100 → $600)
3. System calculates target telemetry shares to isolate cost to anomaly element
4. System derives the telemetry multiplier automatically

**Cost Isolation Formula:**
```
# Target: non-anomaly elements keep same dollar cost during spike
# Example: baseline $100 → spike $600 (6x)
#   api-search: $60 → $560 (absorbs $500 increase)
#   others: $40 → $40 (unchanged)

target_share_non_anomaly = baseline_share / cost_multiplier
target_share_anomaly = 1 - sum(target_shares_non_anomaly)
telemetry_multiplier = target_share_anomaly / baseline_share_anomaly
```

**Example:**
```yaml
elements:
  api-search:
    anomaly_type: "hourly_spike"
    cost_multiplier: 6.0  # Cost increases 6x during spike
    # System calculates: telemetry_multiplier = 9.33x
```

**Result**: Non-anomaly elements maintain baseline costs, anomaly element absorbs entire cost increase

## CSV Output Format

```csv
timestamp,element_id,unit_value,shared_spend_1
2025-01-01T00:00:00Z,api-auth,26.7,Lambda-Data-Processing
2025-01-01T00:00:00Z,api-search,5600.0,Lambda-Data-Processing
2025-01-01T00:00:00Z,api-analytics,26.7,Lambda-Data-Processing
2025-01-01T00:00:00Z,api-reports,26.7,Lambda-Data-Processing
2025-01-01T00:00:00Z,api-notifications,26.7,Lambda-Data-Processing
```

**Note**: Default `shared_spend_1` = "Lambda-Data-Processing" if not specified in config.

**During anomaly hour with cost_multiplier: 6.0**

**CloudZero proportional allocation** (assuming $600/hour resource cost):
- api-search: 5600 / 6000 × $600 = **$560** (93.3% of cost)
- api-auth: 26.7 / 6000 × $600 = **$2.67** (stays at baseline)
- api-analytics: 26.7 / 6000 × $600 = **$2.67** (stays at baseline)
- api-reports: 26.7 / 6000 × $600 = **$2.67** (stays at baseline)
- api-notifications: 26.7 / 6000 × $600 = **$2.67** (stays at baseline)

**Cost Isolation**: Only api-search's cost increases during the spike. Other elements maintain their baseline costs, creating clear causality in CloudZero dashboards.

## Output Structure

```
outputs/telemetry/2025-01-15_14-30-45/
  ├── telemetry.csv    # Three-column CSV
  └── config.yaml      # Copy of input for reproducibility
```

## Commands

```bash
# Generate telemetry
czbg telemetry generate --config configs/telemetry/my-config.yaml

# Validate config
czbg telemetry validate configs/telemetry/my-config.yaml

# Specify output directory
czbg telemetry generate --config my-config.yaml --output outputs/custom/
```

## Demo Workflow

1. Choose story from `configs/telemetry/examples/`
2. Work with Claude Code to create custom config
3. Generate: `czbg telemetry generate --config <config>.yaml`
4. Load to CloudZero via UCA Tools
5. Map telemetry to cloud resources and dimensions
6. Demonstrate unit economics in CloudZero dashboards

## Key Concepts

**Business Activity**: Telemetry measures business metrics (customers, transactions, users), not technical metrics (GB, CPU hours)

**Proportional Allocation**: CloudZero divides cost by telemetry ratio per timestamp

**Cost Isolation**: During anomalies, non-anomaly elements maintain baseline costs while anomaly element absorbs the entire cost increase. Achieved by calculating target telemetry proportions based on cost multiplier.

**Anomaly Timing**: Anomalies occur mid-month (day 16 of 30-day periods) calculated backward from job end. This ensures telemetry spikes align exactly with cost spikes from the generate stage, creating clear cause-and-effect relationships in CloudZero dashboards (e.g., "Lambda costs spike because invocations spiked").

**Deterministic**: Same config produces identical output every time

**Shared Implementation**: Telemetry and generate stages share anomaly timing logic (`src/shared/anomaly/`) to guarantee alignment

## See Also

- Sample configs: `configs/telemetry/samples/`
- Story examples: `configs/telemetry/examples/`
- Pattern reference: `configs/telemetry/PATTERN_REFERENCE.md`
- Timing details: `configs/telemetry/TIMING.md`
