Metadata-Version: 2.4
Name: LbAPCommon
Version: 0.16.4
Summary: Common utilities used by LHCb DPA WP2 related software
Author: LHCb
Project-URL: Homepage, https://gitlab.cern.ch/lhcb-dpa/analysis-productions/LbAPCommon
Project-URL: Bug Reports, https://gitlab.cern.ch/lhcb-dpa/analysis-productions/LbAPCommon/issues
Project-URL: Source, https://gitlab.cern.ch/lhcb-dpa/analysis-productions/LbAPCommon
Keywords: LHCb HEP CERN
Classifier: Development Status :: 3 - Alpha
Classifier: License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
Classifier: Programming Language :: Python :: 3
Requires-Python: >=3.12
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: awkward>=2.1.1
Requires-Dist: boost-histogram
Requires-Dist: hist
Requires-Dist: jinja2
Requires-Dist: lxml
Requires-Dist: matplotlib
Requires-Dist: numpy
Requires-Dist: pandas
Requires-Dist: pyyaml
Requires-Dist: requests
Requires-Dist: setuptools
Requires-Dist: uproot>=5.0.0
Requires-Dist: networkx
Requires-Dist: pydantic>2
Requires-Dist: lbdiracwrappers
Requires-Dist: cwl-utils
Requires-Dist: cwltool
Provides-Extra: testing
Requires-Dist: pytest; extra == "testing"
Requires-Dist: pytest-cov; extra == "testing"
Requires-Dist: pytest-timeout; extra == "testing"
Requires-Dist: flaky; extra == "testing"
Dynamic: license-file

# LbAPCommon

[![pipeline status](https://gitlab.cern.ch/lhcb-dpa/analysis-productions/lbapcommon/badges/master/pipeline.svg)](https://gitlab.cern.ch/lhcb-dpa/analysis-productions/lbapcommon/-/commits/master)
[![coverage report](https://gitlab.cern.ch/lhcb-dpa/analysis-productions/lbapcommon/badges/master/coverage.svg)](https://gitlab.cern.ch/lhcb-dpa/analysis-productions/lbapcommon/-/commits/master)

Common utilities for LHCb DPA WP2 related software, including Analysis Productions workflow parsing, validation, and conversion tools.

## Features

- **Workflow Parsing**: Parse and render Analysis Productions YAML configuration files
- **DIRAC Conversion**: Convert workflow definitions to DIRAC production requests
- **CWL Conversion**: Convert production requests to Common Workflow Language (CWL) format
- **Validation & Linting**: Validate workflow configurations and check for common issues
- **Data Models**: Pydantic v2 models for workflow and production data structures

## Installation

```bash
pip install LbAPCommon
```

For development:

```bash
# Using pixi (recommended)
pixi install
pixi run test

# Using conda/mamba
mamba env create --file environment.yaml
conda activate lbaplocal-dev
pip install -e '.[testing]'
```

## Usage

### DIRAC Production Request Generation

Convert Analysis Productions `info.yaml` files to DIRAC production requests. This is the primary CLI used by the Analysis Productions system:

```bash
python -m LbAPCommon <production_name> --input info.yaml --output output.json

# With specific AP package version
python -m LbAPCommon my_production --input info.yaml --output result.json --ap-pkg-version v1r0

# Process only a specific job from the workflow
python -m LbAPCommon my_production --input info.yaml --output result.json --only-include job_name

# Dump individual YAML request files for each production
python -m LbAPCommon my_production --input info.yaml --output result.json --dump-requests

# With DIRAC server credentials
python -m LbAPCommon my_production --input info.yaml --output result.json --server-credentials user password
```

**Arguments:**
- `production_name` - Name of the production (alphanumeric + underscore, 2-200 chars)
- `--input` - Path to the `info.yaml` workflow definition file
- `--output` - Path for the output JSON file containing production requests
- `--ap-pkg-version` - AnalysisProductions package version (default: `v999999999999`)
- `--only-include` - Only process the workflow chain containing this job name
- `--dump-requests` - Also write individual `.yaml` files for each production request
- `--server-credentials` - DIRAC server credentials (username and password)

The output JSON contains:
- `rendered_yaml` - The rendered workflow YAML after Jinja2 processing
- `productions` - Dictionary of production requests, each containing:
  - `request` - The DIRAC production request in LbAPI format
  - `input-dataset` - Input dataset specification from Bookkeeping query
  - `dynamic_files` - Auto-generated configuration files
  - `raw-yaml` - Original YAML for the jobs in this production

### CWL Conversion CLI

Convert DIRAC production request YAML files to CWL workflows:

```bash
# List productions in a file
lhcb-production-yaml-to-cwl list_productions production.yaml

# Generate CWL workflow files
lhcb-production-yaml-to-cwl generate production.yaml --output-dir ./cwl-output

# Convert a specific production
lhcb-production-yaml-to-cwl generate production.yaml --production "MyProduction"
```

### Python API

```python
from LbAPCommon import parse_yaml, render_yaml, validate_yaml
from LbAPCommon.dirac_conversion import group_in_to_requests, step_to_production_request
from LbAPCommon.prod_request_to_cwl import fromProductionRequestYAMLToCWL

# Parse and validate workflow YAML
rendered = render_yaml(yaml_content)
jobs_data = parse_yaml(rendered, production_name, wg)
validate_yaml(jobs_data, wg, production_name)

# Group jobs into production requests
for job_names in group_in_to_requests(jobs_data):
    request = step_to_production_request(
        production_name, jobs_data, job_names, input_spec, ap_pkg_version
    )

# Convert DIRAC output to CWL
workflow, inputs, metadata = fromProductionRequestYAMLToCWL(yaml_path)
```

## Development

### Running Tests

```bash
pixi run test              # Run all tests
pixi run test-dirac        # Run DIRAC conversion tests
pixi run test-cov          # Run with coverage report
```

### Updating Test Fixtures

When CWL output changes, update reference fixtures:

```bash
pixi run update-fixtures   # Regenerate reference .cwl files
```

### Code Quality

```bash
pixi run pre-commit        # Run all pre-commit hooks (linting, formatting, etc.)
```

## Documentation

Documentation goes in the [LbAPDoc](https://gitlab.cern.ch/lhcb-dpa/analysis-productions/LbAPDoc/) project.

## Related Projects

- [LbAPI](https://gitlab.cern.ch/lhcb-dpa/analysis-productions/LbAPI/) - REST API server for Analysis Productions
- [LbAPLocal](https://gitlab.cern.ch/lhcb-dpa/analysis-productions/LbAPLocal/) - Local testing tools
- [LbAPWeb](https://gitlab.cern.ch/lhcb-dpa/analysis-productions/LbAPWeb/) - Web interface for Analysis Productions

## License

GPL-3.0 - See [COPYING](COPYING) for details.
