Metadata-Version: 2.1
Name: feature_costbot_iapi
Version: 1.1.2
Summary: IAPI Interface Files
Home-page: https://github.com/Cloudzero/feature-costbot
Author: CloudZero
Author-email: support@cloudzero.com
License: UNLICENSED
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Natural Language :: English
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Description-Content-Type: text/markdown
Requires-Dist: cz-common-python>=1.0.15
Requires-Dist: feature-billing-iapi>=1.2.7


# CostBot IAPI

This package contains the interface for the CostBot iAPI. 

## Installing and Use
Add `feature_costbot_iapi` to your requirements file or run `pip install feature_costbot_iapi`. To use, add `from feature_costbot_iapi import costbot_client`.

## Dataset Configurations

The dataset configuration APIs are used to manage the configuration of datasets. Datasets are a mechanism for different features to define data projections
that can be reused across requests to provide consumable answers to questions, or build data tables usable by the calling feature.

There are three types of datasets:

- **Billing**: These datasets are backed by the billing data and are defined using filters and group bys similar to the explorer.
- **SQL**: These datasets are defined by a SQL query which can be executed against a specified database or against a another dataset.
- **View**: These datasets are defined by a view ID and use the view configuration to determine the filter and group bys.

### Configuration Parameters vs. Runtime Parameters

Dataset configurations have two sets of parameters:

- **Configuration Parameters**: These parameters are used to define the dataset configuration. They are specified when the dataset is created and may be 
    used when resolving the dataset. They can be used to resolve the runtime parameters for dependencies or internally within the dataset.
- **Runtime Parameters**: Runtime parameters are parameters that can be specified by the caller at runtime when the dataset request is created or through
    the configuration parameters of a dataset as the dependency chain is resolved.

### Dataset Dependencies

Datasets can have dependencies on other datasets. This is useful when you want to build up a dataset from other datasets. For example, you may
want to query some special conditions against resources of a specific service in a specific cloud provider. A billing projection can be used to provide
a base dataset filtering to the cloud provider and service. However, the special conditions require SQL that cannot be expressed in our filtering model.
Therefore, a SQL dataset can be used to define the SQL query and specify a dependency on the billing dataset. These dependencies are resolved
at runtime when the dataset is being processed.

Dependencies are specified as a mapping of logical IDs to a dictionary of parameters defining the dataset and parameters to be used when resolving the dependency.
For example, the following dependency will use the dataset `base-billing-projection` to resolve the dependency and use the `time_span`, `dimensions`, and `filters`
parameters when calling the billing data IAPI.

```python
{
    'bp': {
        'name': 'base-billing-projection',
        'runtime_parameters': {
            'time_span': 'last_30_days',
            'dimensions': {'costcontext:Resource Summary': 'resource_summary'},
            'filters': ['service_alias: "Amazon Elastic Compute Cloud"'],
        }
    }
}
```

The key value `bp` can then be used in the dataset SQL as `${bp}` to use the resolved dataset. For example:

```python
'sql': 'SELECT * FROM ${bp} WHERE resource_summary: "Amazon EC2" AND resource_summary: "EBS Snapshots"'
```

Datasets can have multiple dependencies which enables building more complex datasets that need to join data from multiple projections. Datasets can also be chained so that
one dataset can depend another dataset which in turn can have its own dependencies. The resolution of the dataset will be done in the appropriate order.

### Dataset Configuration IAPIs

The dataset configuration APIs are used to create, update, and delete datasets. The dataset configuration is defined by the `DatasetConfiguration` model.

#### Default Dataset Configurations

Dataset configurations are typically added to the system and managed by specifying the default organization ID (`cz_organization_id=DEFAULT`) in the IAPIs.
Default dataset configurations are available to all organizations and are useful for providing common across all organizations. However, organization specific configurations
can also be added to the system. When a dataset is requested, either directly or indirectly through a dependency, the system will use the organization specific configuration
if it exists. When an organization specific configuration exist for any dataset, the default configuration will be used.

## Dataset Requests

Callers can create dataset requests which will resolve the dataset based on the configuration and runtime parameters. A single request can request multiple datasets using logical IDs to 
identify the specific dataset requested. When specifying the datasets to be requested, the caller can specify runtime parameters for each dataset which will be used to resolve the dataset.

For example:

```python
    request_id = costbot_client.create_dataset_request(
        cz_organization_id='DEFAULT',
        datasets={
            'last_30_days_billing': {'name': 'base-billing-projection', 'runtime_parameters': {'time_span': 'last_30_days'}},
            'ec2_ebs_snapshot_costs': {'name': 'sql-dataset'}}
        }
    )
```
The above request requests two separate datasets. The first dataset will resolve the `base-billing-projection` dataset using the `last_30_days` runtime parameter. The second dataset
will resolve the `sql-dataset` dataset without providing any runtime parameters.

The same logical IDs are used when receiving results for a request. When receiving results for a request, the results will be in a dictionary using the same logical IDs as keys and
the results as the values. The use of logical IDs enables the caller to potentially request the same dataset multiple times with different runtime parameters.

### Request Completion Events

Dataset requests are resolved asynchronously and an EventBridge event will be published when the request is complete. The call to create a request will return a request ID which can be used
to identify the request and check the status of the request. If multiple datasets are requested, the event will contain a mapping of logical IDs to the results for each dataset.

The event will look like the following:

```json
{
    "detail": {
        "event_name": "feature-costbot-datapipeline-request-succeeded",
        "cz_organization_id": "<organization_id>",
        "data": {
            "request_id": "<the request ID that was returned when the request was created>",
            "request_tags": {"<tag_key>": "<tag_value>"},  // The tags that were specified when the request was created
            "request_results": {}  // A dictionary of logical IDs to the results for each dataset requested
        },
        "__test_key__": "<the value of the __test_key__ input if one was provided>"
    },
    "detail-type": "Feature Costbot DataPipeline Request Succeeded",
}
```

### Retrieving Request Results

For datasets that are `inline` (the configuration parameter `inline_results=True`), the results will be returned directly in the event. For datasets that are not marked as `inline`, the results
will be stored in the S3 bucket and must be downloaded. When the completion event is received, the caller can call the IAPI `get_results` to retrieve the results for a request. The response from
this call will provide a download link for any non-inline datasets (inline dataset results will be directly in the response).

Call made to `get_results` for a request that is still running or one that failed will throw a BadRequestError exception.

### Request Completion Tokens

If a request is being created as a step in an external state machine, the call to create the request can include a `task_token` parameter. If provided, the state machine
will use the token to complete the task with success or failure when the request is complete. This enables the calling state machine to wait for the request to complete
before proceeding. 

For example, the following step definition would provide the task token to the request creation call:

```json
    ...
    "RequestDataset": {
        "Type": "Task",
        "Resource": "arn:aws:states:::lambda:invoke.waitForTaskToken",
        "Parameters": {
            "Payload": {
                "cz_organization_id.$": "$.input.cz_organization_id",
                "datasets.$": "$.some_previous_step.datasets",
                "tags.$": "$.input.tags",
                "task_token": "$$.Task.Token"
            },
            "FunctionName": "$CreateRequestFunctionArn}"
        },
    ...
```
