Metadata-Version: 2.1
Name: cz-testing-python
Version: 0.6.1
Summary: Useful testing functions; remove all the boilerplate
Home-page: https://github.com/Cloudzero/cz-testing-python
Author: CloudZero
Author-email: support@cloudzero.com
License: UNLICENSED
Classifier: Development Status :: 4 - Beta
Classifier: Intended Audience :: Developers
Classifier: Natural Language :: English
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: boto3>=1.14.62
Requires-Dist: boto3-stubs[stepfunctions]>=1.14.62
Requires-Dist: botocore>=1.17.62
Requires-Dist: cachetools>=4.1.1
Requires-Dist: typing-extensions>=3.7.4.3
Requires-Dist: toolz>=0.10.0
Requires-Dist: voluptuous>=0.12.0
Requires-Dist: pyfaaster>=0.2.0
Requires-Dist: aws-lambda-powertools>=1.22.0
Requires-Dist: cz-common-python>=0.10.53
Requires-Dist: faker>=13.3.4
Requires-Dist: docker>=6.0.0
Requires-Dist: deprecated>=1.2.10


[![Build Status](https://cloudzero.semaphoreci.com/badges/cz-testing-python/branches/main.svg)](https://cloudzero.semaphoreci.com/projects/cz-testing-python)

# Overview

> 🛑: This library requires Python 3.9+

`cz-testing-python` provides the `czt` python package that contains useful feature modules like:

- `czt.datagenerator`: generates fake data
- `czt.smoke`:
  - `api_test_runner`: low level smoke test runners
  - `api_test_client`: high level smoke test client APIs
- `czt.unittest`:
  - `Context` context manager for use in context fixtures
  - `MockEx` class decorates mocks, adding assertable chainable methods, e.g. `MockEx(mock).was_called_once_with(*args, **kwargs)` does all of the nasty argument unpacking and comparison and compares call count.


# Scopes Testing

In order to easily add basic role security testing to for your Web API routes, you can use the ScopeTest fixture. You can create one test that will dynamically generate
tests for each role against each scope in your web API (a scope is the combination of web method (i.e. GET, POST, etc.) and route. By using this mechanism, you will not
need to keep adding new smoke tests to test out role access. Your smoke tests can focus on functionality.

This fixture works by discovering all routes/methods from the deployed APIGateway and compares them to the `setup-template.json` file. Additionally you can define test configurations
that detail what roles should be allowed to call the API (by default it assumes only organizers and super-users can call it).

These API calls for these test do not need to necessarily return a 2xx. If a role is allowed to call an API, then any 2xx status and some 4xx status codes (400, 404, 406, 409, 410, 411, 412, 415, 416, 417)
will count as a success (i.e. if they are expected to be allowed to call and a 401 or 403 is returned, then the scope is not setup correctly). Any 5xx codes would be considered a failure as those return codes
indicate internal failures.

To add these tests you must do the following:

1. Add the `PublicApiId` to the `Outputs` section of your CloudFormation deployment file (`template.yaml`) file:

```
Outputs:

  #### NEW OUTPUT ####
  PublicApiId:
    Description: Public Feature REST API ID
    Value: !Sub ${PublicApi}
  ####################
```

2. In your `tests/smoke/web` folder, add the `test_scopes.py` file. It should look something like the following:

```
import pytest
from czt.smoke.scopes import ALL_ROLES, get_scope_tests, ScopeTest
from tests.smoke.common import CONFIGURATION

# This configuration allows the configuration of what roles can call what scopes.

SCOPE_TEST_CONFIGURATIONS = {
    "insights:get-insight": {'allowed_roles': ALL_ROLES},
    "insights:get-all-insights": {'allowed_roles': ALL_ROLES},
    "insights:export-insights": {'allowed_roles': ALL_ROLES},
    "insights:get-all-comments": {'allowed_roles': ALL_ROLES},
    "insights:create-comment": {'allowed_roles': ALL_ROLES},
    "insights:update-comment": {'allowed_roles': ALL_ROLES},
    "insights:get-all-resources": {'allowed_roles': ALL_ROLES},
    "insights:export-resources": {'allowed_roles': ALL_ROLES},
    "insights:get-summary": {'allowed_roles': ALL_ROLES},
    "insights:update-insight": {'allowed_roles': ALL_ROLES},

    #  You can specify to skip a scope if the tests don't apply of don't work out well for that scope
    "insights:create-insight": {'skip': True},

    # If now 'allowed_roles' are specified, then it is assumed that only the organizer and super-user can call the route
    "insights:delete-insight": {}

    # Any method/routes not specified, but found in the APIGateway will be tested with default settings.
    # If a method/route is found in the gateway, but not in the `setup-template.json` that test will fail.
}

# You must add this function (named exactly as is) and it must call `get_scope_tests`
def pytest_generate_tests(metafunc):
    get_scope_tests(metafunc, SCOPE_TEST_CONFIGURATIONS, CONFIGURATION)


@pytest.mark.smoke
def test_scopes(namespace, credentials, public_api_url, scope_test: ScopeTest):
    scope_test.run(namespace, credentials, public_api_url)

```

If a valid AWS console token is not available in the environment, the tests will not be enumerated successfully. This is because calls must be made to AWS APIGateway to get the routes.
By default only routes that begin with `/organizations/` will be used. This can be overridden by adding the `route_prefix` parameter when calling `get_scope_tests`.

# IAPI Event Body Testing

IAPIs need to be tested that they can handle extra event fields that are appended via extensions like OpenTelemetry (OTEL). To add basic testing, do the following:

1. In `tests/smoke/iapi` add a new file `test_event.py`

    ```python
    # Copyright (c) 2024 CloudZero - ALL RIGHTS RESERVED - PROPRIETARY AND CONFIDENTIAL
    # Unauthorized copying of this file and/or project, via any medium is strictly prohibited.
    # Direct all questions to legal@cloudzero.com

    import os

    import pytest

    from czt.smoke.iapi import get_iapi_event_tests, IapiEventTest, event_headers_validator

    def custom_validator(lambda_invoke_response):
      assert lambda_invoke_response['StatusCode'] == 403

    EVENT_TEST_CONFIGURATIONS = {
        'data-access-controls:IAPIGetUserFilters': {
            'event': {
                'user_id': 'user_id',
                'cz_organization_id': os.environ['SMOKE_TEST_ORG_ID'],
            },
        },
        'data-access-controls:IAPISyncUserGroups':  {
            'event': {
                'user_id': 'user_id',
                'cz_organization_id': os.environ['SMOKE_TEST_ORG_ID'],
            },
            'response_validator': custom_validator,
        },
        'data-access-controls:IAPIMaterializeFilter': {'skip': True},
        'data-access-controls:IAPIDeleteMaterialFilter': {'skip': True},
    }


    def pytest_generate_tests(metafunc):
        get_iapi_event_tests(metafunc, EVENT_TEST_CONFIGURATIONS, use_cz_test_key=True)


    @pytest.mark.smoke
    def test_iapi_event(iapi_event_test: IapiEventTest):
        iapi_event_test.run()
    ```

2. Ensure that all the keys match `feature:LogicResourceId`. The LogicalResourceId is the field name of the YAML block encapsulating `AWS::Serverless::Function` in the `template.yaml`.

    ```yaml
      IAPIGetUserFilters:
        Type: AWS::Serverless::Function
        ...
    ```

3. All deployed IAPI lambdas *must* have a configuration set for it. If this is forgotten, an error will raise with a reminder to add the missing lambda.
4. By default, `response_validator` is set to `event_headers_validator` will handle most use cases for validation. Custom validators can be

# Data Access Testing
To add basic data access testing to your Web API routes, you can use the DataAccessTest fixture.
This fixture will dynamically generate tests for each data access filter type (NO_ACCESS, LIMITED_ACCESS,
FULL_ACCESS) against each scope in your web API route.

To add these tests you must do the following:

1. Add the `PublicApiId` to the `Outputs` section of your CloudFormation deployment file (`template.yaml`) file:

```
Outputs:

  #### NEW OUTPUT ####
  PublicApiId:
    Description: Public Feature REST API ID
    Value: !Sub ${PublicApi}
  ####################
```

2. In your `tests/smoke/web` folder, add the `test_data_access.py` file. It should look something like the following:

```
import pytest

from czt.smoke.data_access import get_data_access_tests, FilterType, DataAccessTest

from tests.smoke.common import CONFIGURATION


DATA_ACCESS_TEST_CONFIGURATIONS = {
    # example config will ensure only users with filter_type=FULL_ACCESS get statusCode=200 for given scope
    # expect users with LIMITED_ACCESS and NO_ACESS to get statuscode=403
    "data-access-controls:get-all-groups": {'allowed_filter_types': [FilterType.FULL_ACCESS]},
    # example config will skip the given scope
    "data-access-controls:get-group": {'skip': True},
    # scopes in setup-template.json not listed here default to expect statusCode=200 for each filter_type
}


def pytest_generate_tests(metafunc):
    get_data_access_tests(metafunc, DATA_ACCESS_TEST_CONFIGURATIONS, CONFIGURATION)


@pytest.mark.smoke
def test_data_access(namespace, user_credentials, public_api_url, data_access_test: DataAccessTest):
    data_access_test.run(namespace, user_credentials, public_api_url)
```

By default, only routes that begin with `/organizations/` will be used.

# Infrastructure Testing

## Testing Lambda Function Versions

To test that the correct version of a lambda function is being used, you can use the `LambdaVersionTest` fixture. This fixture will dynamically generate tests for each lambda function in your stack that
has a published version. The test will call the lambda function and check that the version of the function that is being used is the same as the version that was published.

To add these tests you must do the following:

1. In your `tests/smoke/infrastructure` folder, add a `test_lambda_versions.py` file. It should look something like the following:

```python
import pytest

from czt.smoke.infrastructure.lambda_versions import (
    Configuration,
    FunctionConfiguration,
    LambdaFunctionVersionTest,
    get_lambda_function_version_tests,
)

configuration: Configuration = {'iapi-get-organization': FunctionConfiguration(skip=True)}


def pytest_generate_tests(metafunc):
    get_lambda_function_version_tests(metafunc)


@pytest.mark.smoke
def test_lambda_functions_have_latest_version(lambda_function_version_test: LambdaFunctionVersionTest):
    lambda_function_version_test.run(configuration=configuration)
```

2. Update the `Configuration` dictionary with the lambda functions that you want to skip or test. By default, all lambda functions will be tested.
