| # BitTransformerLM Scripts |
|
|
| This directory contains organized scripts for BitTransformerLM development, training, and evaluation. |
|
|
| ## Directory Structure |
|
|
| ``` |
| scripts/ |
| ├── training/ # Training scripts and experiments |
| ├── examples/ # Example usage and demonstrations |
| ├── testing/ # Test scripts and validation |
| ├── benchmarks/ # Performance benchmarks |
| └── tools/ # Utility scripts and data processing |
| ``` |
|
|
| ## Training Scripts (`training/`) |
|
|
| - **basic_training.py** - Simple training setup for small models |
| - **breakthrough_training.py** - Advanced training with breakthrough techniques |
| - **cpu_edge_training.py** - CPU-optimized training for edge deployment |
| - **final_breakthrough_training.py** - Production training pipeline |
| - **full_attention_training.py** - Full attention mechanism training |
| - **full_bits_train.py** - Complete bit-level training |
| - **production_training.py** - Production-ready training script |
| - **progressive_scaleup.py** - Progressive model scaling |
| - **quick_training_run.py** - Fast training for development |
|
|
| ## Example Scripts (`examples/`) |
|
|
| - **example.py** - Basic usage example |
| - **better_sampling.py** - Advanced sampling techniques |
| - **debug_generation.py** - Generation debugging utilities |
| - **raw_generation.py** - Low-level generation examples |
| - **simple_test.py** - Simple model testing |
|
|
| ## Testing Scripts (`testing/`) |
|
|
| - **code_test.py** - Code functionality testing |
| - **diffusion_tests.py** - Diffusion mode testing |
| - **enhanced_generation_test.py** - Advanced generation testing |
| - **full_attention_inference_test.py** - Attention mechanism tests |
| - **test_conversation.py** - Conversational AI testing |
|
|
| ## Benchmark Scripts (`benchmarks/`) |
|
|
| - **wikitext_benchmark.py** - WikiText dataset benchmarking |
| - **wikitext_schedule.py** - WikiText training schedule |
|
|
| ## Utility Tools (`tools/`) |
|
|
| - **build_full_bits.py** - Bit sequence construction |
| - **create_dataset.py** - Dataset creation utilities |
| - **enhanced_checkpoint_system.py** - Advanced checkpointing |
| - **integration_flow.py** - Integration workflow |
| - **integration_schedule.py** - Integration scheduling |
| - **sync_to_hf.py** - HuggingFace synchronization |
| - **unified_workflow.py** - Unified training workflow |
| - **watcher.py** - File system monitoring |
|
|
| ## Usage |
|
|
| All scripts support the standardized CLI interface provided by `bit_transformer.cli_standards`. Use `--help` with any script to see available options. |
|
|
| ### Quick Start |
|
|
| ```bash |
| # Train a small model |
| python scripts/training/basic_training.py --model-size small --epochs 5 |
| |
| # Run a simple test |
| python scripts/examples/simple_test.py --d-model 64 |
| |
| # Benchmark on WikiText |
| python scripts/benchmarks/wikitext_benchmark.py --dataset-name wikitext-2 |
| ``` |
|
|
| ### Environment Variables |
|
|
| Scripts support configuration via environment variables with `BT_` prefix: |
|
|
| ```bash |
| export BT_D_MODEL=128 |
| export BT_NUM_LAYERS=4 |
| export BT_BATCH_SIZE=16 |
| python scripts/training/basic_training.py |
| ``` |
|
|
| ## Development Guidelines |
|
|
| - All scripts should use `bit_transformer.cli_standards` for argument parsing |
| - Include proper logging and error handling |
| - Support both CPU and GPU execution |
| - Follow the naming conventions established in existing scripts |
| - Add documentation for any new hyperparameters or features |
