Metadata-Version: 2.4
Name: cz-billing-generator
Version: 0.1.0
Summary: CLI tool for generating and uploading synthetic billing data to CloudZero
Author-email: CloudZero <support@cloudzero.com>
License-Expression: LicenseRef-Proprietary
Project-URL: Homepage, https://github.com/cloudzero/czbillgen
Keywords: CloudZero,czbg,billing,generator,cli
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Natural Language :: English
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Operating System :: MacOS
Classifier: Operating System :: Unix
Requires-Python: >=3.10
Description-Content-Type: text/markdown
Requires-Dist: typer>=0.9.0
Requires-Dist: polars>=0.20.0
Requires-Dist: pyyaml>=6.0.0
Requires-Dist: rich>=14.0.0
Requires-Dist: python-dotenv>=1.1.0
Requires-Dist: boto3>=1.26.0
Requires-Dist: requests>=2.32.0
Requires-Dist: tenacity>=9.0.0
Requires-Dist: psutil>=6.1.0
Requires-Dist: numpy>=1.24.0
Requires-Dist: python-dateutil>=2.8.0

# CloudZero Billing Generator

A CLI tool for generating synthetic cloud billing data across multiple stages - from basic cost generation through Kubernetes allocation, metadata injection, usage calculations, discounts, taxes, and upload to CloudZero for demos, training, and trials.

## Quick Start

Essential workflow to generate your first complete dataset:

```bash
# 1. Setup AWS credentials
aws sso login

# 2. Initialize workspace with templates and example configs
czbillgen init

# 3. Check reference data availability
czbillgen data status overview

# 4. Generate base billing data
czbillgen generate run --config-path configs/generate/standard/base/aws.pipeline.yaml

# 5. Process through additional stages
czbillgen k8s run --config configs/k8s/config.k8s.yaml
czbillgen metadata run --config configs/metadata/config.metadata.yaml
czbillgen usage run --config configs/usage/config.usage.yaml
czbillgen discount run --config configs/discount/config.discount.yaml
czbillgen tax run --config configs/tax/config.tax.yaml

# 6. Upload to CloudZero
czbillgen upload generated --config configs/upload/config.upload.yaml

# 7. Explore your datasets
czbillgen list-datasets
```

## Installation

### Prerequisites
- Python 3.10+
- AWS credentials (for S3 access)
- CloudZero API access (for upload)

### Install via pipx (recommended)
```bash
pipx install czbillgen
```

### Environment Setup
Create `~/.czbillgen/.env` with your S3 configuration:

```bash
CZBILLGEN_S3_BUCKET=your-bucket-name
CZBILLGEN_S3_REFERENCE_PATH=czbillgen/reference_data
AWS_DEFAULT_REGION=us-east-1
AWS_PROFILE=your-profile-name  # Optional
```

## Architecture Overview

CloudZero Billing Generator uses a **multi-stage pipeline architecture** where each stage processes data sequentially:

```
Reference Data → Generate → K8s → Metadata → Usage → Discount → Tax → Upload
```

### Stages Explained
- **Generate**: Creates base synthetic billing data from config templates
- **K8s**: Allocates compute resources to Kubernetes workloads and namespaces  
- **Metadata**: Adds tags, resource metadata, and regional information
- **Usage**: Adds usage amounts, rates, and usage-based pricing
- **Discount**: Applies discount calculations and savings plans
- **Tax**: Adds tax calculations based on regional rules
- **Upload**: Uploads processed data to CloudZero API

### Data Storage
All stages store data in S3 with this structure:
```
s3://bucket/czbillgen/datasets/{dataset-name}/{stage}/{provider}/{month}.parquet
```

Example:
```
s3://bucket/czbillgen/datasets/2025-07-22_11-01-07/
├── generate/AWS/2025-02.parquet
├── k8s/AWS/2025-02.parquet  
├── metadata/AWS/2025-02.parquet
└── upload/AWS/2025-02.parquet
```

## Main Commands

### Data Generation
```bash
# Generate base billing data
czbillgen generate run --config-path CONFIG_FILE [--dataset-name NAME] [--dry-run] [--force]
czbillgen generate validate --config-path CONFIG_FILE
czbillgen generate list-configs --config-path CONFIG_PATH
czbillgen generate delete-dataset DATASET_NAME

# Process through pipeline stages  
czbillgen k8s run --config CONFIG_FILE [--month YYYY-MM] [--provider PROVIDER] [--force]
czbillgen metadata run --config CONFIG_FILE [--month YYYY-MM] [--provider PROVIDER] [--force]
czbillgen usage run --config CONFIG_FILE [--month YYYY-MM] [--provider PROVIDER] [--force]
czbillgen discount run --config CONFIG_FILE [--month YYYY-MM] [--provider PROVIDER] [--force] 
czbillgen tax run --config CONFIG_FILE [--month YYYY-MM] [--provider PROVIDER] [--force]
```

### Data Management
```bash
# Reference data management
czbillgen data status overview [--verbose]
czbillgen data explore [--provider PROVIDER] [--service SERVICE]
czbillgen data download [--force]
czbillgen data upload [--force]

# Dataset exploration
czbillgen list-datasets [--dataset NAME] [--stage STAGE] [--provider PROVIDER] [--month YYYY-MM]
czbillgen list-upload-datasets [--verbose]
```

### Upload to CloudZero
```bash
# Upload generated data
czbillgen upload generated --config CONFIG_FILE [--dry-run] [--force]
czbillgen upload connections --config CONFIG_FILE [--dry-run] [--force]
```

### System Management
```bash
# Initialize workspace
czbillgen init [--template TEMPLATE]

# System utilities
czbillgen version
czbillgen status [--verbose]
```

## Configuration Files

### Generate Stage Config
Generate stage uses YAML configs with this structure:

```yaml
generation_jobs:
  - name: "AWS EC2 Production"
    start_date: "2025-02-01"
    end_date: "2025-07-31"
    total_cost: 5000000.00    # $50,000 over 6 months
    resource_count: 20        # 20 EC2 instances
    filters:
      - cloud_provider: AWS
        variance_profile: "linear_growth"
        services:
          - name: AmazonEC2
            usage_families: []  # Wildcard: all families
```

### Other Stage Configs
Other stages use simpler YAML configs that specify the dataset to process:

```yaml
# k8s/config.k8s.yaml
dataset_name: "2025-07-22_11-01-07"

# upload/config.upload.yaml  
dataset_name: "2025-07-22_11-01-07"
stage_name: "tax"  # Which stage to upload
```

## Documentation

### User Guides
- **[Generate Stage Guide](docs/generate-guide.md)** - Complete guide to configuration and data generation
- **[Multi-Stage Workflow Guide](docs/workflow-guide.md)** - End-to-end pipeline execution
- **[Variance & Anomaly Profiles](docs/variance-anomaly-profiles.md)** - Behavioral patterns for realistic data

### Configuration References  
- **[Config Templates](docs/config-templates.md)** - Pre-built configurations for common scenarios
- **[Filter Hierarchy Guide](docs/filter-guide.md)** - Understanding service/resource filtering
- **[Cost Distribution Guide](docs/cost-guide.md)** - How costs are calculated and distributed

### Technical Documentation
- **[Architecture Overview](docs/architecture.md)** - System design and data flow
- **[S3 Data Layout](docs/s3-layout.md)** - Storage structure and organization  
- **[API Reference](docs/api-reference.md)** - Complete command reference
- **[Development Guide](DEVELOPMENT.md)** - Contributing and development workflow

## Example Workflows

### Basic AWS Generation
```bash
# Generate 3 months of AWS EC2 + RDS data
czbillgen generate run --config-path configs/generate/standard/base/aws.pipeline.yaml

# Explore what was generated
czbillgen list-datasets --dataset "2025-07-22_11-01-07" --stage generate

# Process through K8s allocation
czbillgen k8s run --config configs/k8s/config.k8s.yaml
```

### Multi-Cloud AI/ML Scenario
```bash
# Generate AI/ML services across all clouds
czbillgen generate run --config-path configs/generate/standard/ai/

# Process through full pipeline
czbillgen k8s run --config configs/k8s/config.k8s.yaml
czbillgen metadata run --config configs/metadata/config.metadata.yaml
czbillgen usage run --config configs/usage/config.usage.yaml
czbillgen discount run --config configs/discount/config.discount.yaml
czbillgen tax run --config configs/tax/config.tax.yaml

# Upload to CloudZero
czbillgen upload generated --config configs/upload/config.upload.yaml
```

### Custom Configuration Development
```bash
# Start with template
cp configs/generate/standard/base/aws.pipeline.yaml my-custom-config.yaml

# Validate configuration
czbillgen generate validate --config-path my-custom-config.yaml

# Test with dry run
czbillgen generate run --config-path my-custom-config.yaml --dry-run

# Generate actual data
czbillgen generate run --config-path my-custom-config.yaml
```

## Features

### Multi-Stage Pipeline
- **6-Stage Processing**: Generate → K8s → Metadata → Usage → Discount → Tax → Upload
- **Stage Independence**: Run individual stages or complete pipelines
- **Data Consistency**: Cross-month consistency maintained throughout pipeline
- **Fail-Fast Processing**: Stages stop immediately on validation failures

### Realistic Data Generation  
- **Behavioral Patterns**: Linear growth, burst cycles, seasonal variations
- **Anomaly Injection**: Configurable cost spikes and optimization events
- **Multi-Cloud Support**: AWS, Azure, GCP, and 10+ SaaS providers
- **Service Hierarchy**: Realistic cloud service/resource/usage relationships

### Advanced Configuration
- **Wildcard Filtering**: Use empty arrays for automatic service discovery
- **Precise Control**: Specify exact usage types and operations
- **Cost Distribution**: Intelligent cost allocation across time and resources
- **Template System**: Pre-built configs for common scenarios

### Enterprise Integration
- **S3-Based Storage**: Scalable data storage with organized structure
- **CloudZero Integration**: Direct upload to CloudZero API
- **AWS SSO Support**: Enterprise authentication integration
- **Multi-Dataset Management**: Organize and track multiple synthetic datasets

## Requirements

- **Python**: 3.10 or higher
- **AWS Access**: Valid credentials for S3 operations
- **CloudZero API**: Access token for upload operations
- **Dependencies**: Managed automatically via pipx installation

## Support

For questions, issues, or feature requests:
- **Issues**: [GitHub Issues](https://github.com/cloudzero/cz-billing-generator/issues)
- **Documentation**: Check the [docs/](docs/) directory for detailed guides
- **Internal Support**: CloudZero Slack #billing-generator channel
