Cookiecutter FAIR RDM Template

This plan outlines the development and execution of a Cookiecutter template for Python research data management projects, emphasizing FAIR principles, automation, and user-friendly naming convention enforcement.

---

1. Package Structure Overview

a. Cookiecutter Template Structure

cookiecutter-rdm-template/
│
├── cookiecutter.json
├── hooks/
│   └── post_gen_project.py
└── {{cookiecutter.project_slug}}/
    ├── README.md
    ├── LICENSE
    ├── requirements.txt
    ├── pyproject.toml
    ├── .gitignore
    ├── .zenodo.json
    ├── {{cookiecutter.project_config_file_name}}
    ├── data/
    │   ├── raw/
    │   ├── processed/
    │   ├── metadata/
    │   └── README.md
    ├── src/
    │   ├── __init__.py
    │   ├── checks/
    │   │   ├── check_naming_convention.py
    │   │   ├── generate_data_dictionary.py
    │   │   └── quality_check.py
    │   └── utils/
    │       ├── regex_builder.py
    │       ├── generate_zenodo_json.py
    │       └── csv_to_ge_suite.py
    ├── docs/
    │   ├── README.md
    │   ├── data_dictionaries/
    │   ├── metadata_schema.md
    │   └── naming_conventions.md
    └── .github/
        └── workflows/
            ├── data_checks.yml
            └── zenodo_release.yml

---

2. Main Functions and Responsibilities

a. hooks/post_gen_project.py

• Reads user choices from cookiecutter.json.
• Constructs initial regex for file naming.
• Writes config file (e.g., .project_config.json).
• Prints user guidance for next steps.

b. src/utils/regex_builder.py

• Loads regex and settings from config.
• Provides CLI for interactive regex building/refinement.
• Explains regex in human-readable form.
• Allows live testing of regex against filenames.
• Saves updated regex to config.

c. src/checks/check_naming_convention.py

• Loads regex from config.
• Scans data directory for files.
• Validates filenames against regex.
• Outputs report (pass/fail, non-compliant files).

d. src/checks/generate_data_dictionary.py

• Reads data directory and metadata.
• Generates a data dictionary (structure, fields, types).
• Optionally uses config for paths/settings.

e. src/checks/quality_check.py

• Loads data dictionary and data files.
• Performs automated quality checks (type, range, allowed values, uniqueness, missing values) using pandas.
• Outputs a detailed report of issues.
• Note: All core quality checks are independent of Great Expectations for reliability and testability.

f. src/utils/csv_to_ge_suite.py

• (Advanced/Optional) Converts a data dictionary CSV to a Great Expectations suite for advanced validation workflows.
• Not required for core FAIRy quality checks or automation.
• Logic for parsing the data dictionary and building expectations is tested independently using pandas.

g. .github/workflows/data_checks.yml

• Runs check_naming_convention.py and quality_check.py on push/PR.
• Fails workflow if non-compliant files or data issues are found.

h. .github/workflows/zenodo_release.yml

• Handles Zenodo integration for releases (no direct regex or data validation impact).

i. Documentation

• README.md: Project overview, quick start, how to use regex builder, quality checks, and advanced GE suite generation.
• docs/naming_conventions.md: Naming convention rationale, regex usage, examples.
• docs/README.md: Links to naming conventions and regex builder.

j. .zenodo.json

• Pre-filled metadata for Zenodo, with dynamic fields.

---

3. Execution Roadmap

Phase 1: Template and Structure

• Scaffold the Cookiecutter template as above.
• Implement cookiecutter.json with all variables, including naming convention options.
• Write hooks/post_gen_project.py to generate config and initial regex.
• Create templates for all static files (README, LICENSE, requirements, etc.).

Phase 2: Core Functionality

• Develop Python scripts:
  • regex_builder.py (interactive CLI)
  • check_naming_convention.py (validation)
  • generate_data_dictionary.py (data dictionary)
  • quality_check.py (data quality checks, pandas-based)
  • csv_to_ge_suite.py (optional, for GE suite generation)
• Write documentation:
  • Main README
  • Naming conventions guide
  • Data dictionary guide (if needed)
• Set up GitHub Actions workflows for data checks and Zenodo release.

Phase 3: Testing and Refinement

• Test template generation with various options.
• Test scripts and workflows with compliant and non-compliant data.
• Ensure all core logic is covered by unit tests in tests/, using pandas for expectation logic validation.
• Tests for CSV-to-GE suite logic are decoupled from Great Expectations and use pandas for robust, fast validation.
• Iterate and refine based on feedback and test results.

---

4. Checklist for Each Script

• regex_builder.py: Loads/saves regex, interactive CLI, human-readable explanations, live testing.
• check_naming_convention.py: Loads regex, validates files, outputs report.
• generate_data_dictionary.py: Scans data, generates dictionary, uses config.
• quality_check.py: Loads data, performs quality checks, outputs report.
• post_gen_project.py: Reads variables, generates config, prints guidance.

---

5. Next Steps

• Start by creating the template structure and cookiecutter.json.
• Implement the post-generation hook and config logic.
• Develop and test the core Python scripts.
• Write and refine documentation.
• Test the full workflow and iterate.

---

This plan is now ready for execution. Each function and file has a clear responsibility, and the structure supports maintainability and FAIR compliance.