| # CodeMode Core Scripts 🚀 | |
| This directory contains the **modular core logic** for the CodeMode pipeline. It is designed to be cleaner and more production-ready than the experimental notebooks. | |
| ## Structure | |
| ### 1. Ingestion (`scripts/core/ingestion`) | |
| Handles data collection and processing. | |
| - `ingest.py`: The Git Crawler (formerly `git_crawler.py`). | |
| - `chunk.py`: The Universal Chunker (formerly `repo_chunker.py`). | |
| - `generate_data.py`: Creates training triplets (formerly `pairs_triplets_generator.py`). | |
| **Usage:** | |
| ```bash | |
| # Example: Ingest a repo | |
| python -m scripts.core.ingestion.ingest --url https://github.com/crewAIInc/crewAI | |
| # Example: Generate Triplets | |
| python -m scripts.core.ingestion.generate_data --chunks data/processed/chunks.jsonl --output data/training | |
| ``` | |
| ### 2. Training (`scripts/core/training`) | |
| Handles model training and embedding generation. | |
| - `train.py`: Main training loop. | |
| - `model.py`: The CodeEmbedder model architecture. | |
| - `trainer.py`: The training loop logic. | |
| **Usage:** | |
| ```bash | |
| # Example: Train the model | |
| python -m scripts.core.training.train --data_path data/training/triplets.jsonl --epochs 3 | |
| ``` | |
| ## Why this structure? | |
| - **Separation of Concerns:** Training logic doesn't depend on web scraping libraries. | |
| - **Reusability:** You can import `CodeEmbedder` or `RepoChunker` in other projects easily. | |
| - **Production Ready:** Direct python scripts instead of notebooks. | |