Development notes

# Development notes 

## First model and inital exploration
- Replicated benchmark LigthGBM classifier model
- Dropped SIGNED_VOLUME_1 (73% missing values)
- No feature engineering yet
- 5-fold cross-validation

#### Data Understanding
- **Training data:** 527,073 rows × 44 columns
- **Test data:** 31,870 rows × 44 columns
- **Target:** Continuous returns, converted to binary (1 = positive, 0 = negative)
- **Class balance:** 50.72% positive, 49.28% negative (well balanced!)
- **Structure:** Each row = one allocation on one date with 20 days of history

#### Key Discoveries

**Missing Values Problem:**
- SIGNED_VOLUME_1: 73.5% missing (387,506 rows!)
- Other SIGNED_VOLUME columns: <2% missing
- Pattern: Most recent day (day 1) systematically unavailable for most allocations
- **Decision:** Dropped SIGNED_VOLUME_1 entirely rather than filling 73% with fake zeros

**Other Missing Values:**
- Filled remaining missing values (<2%) with 0
- Final clean dataset: 41 features, 0 missing values

#### Models Trained

**Logistic Regression (Baseline):**
- Single train/val split (80/20)
- Accuracy: 50.97%
- Beat benchmark with simple model

**LightGBM:**
- 5-fold Cross-Validation
- Hyperparameters: n_estimators=100, learning_rate=0.05
- **CV Accuracy: 53.59%** ± 0.17% std dev
- **Leaderboard: 50.90%** (rank#210 (on 15.02.2026), dropped to rank#453 (on 27.02.2026))
- Benchmark: 50.79%

#### Key Learnings

**Overfitting:**
- CV score (53.59%) vs Leaderboard (50.90%) = 2.69% gap
- Classic overfitting to validation set
- Lesson: CV is optimistic; real test set is harder

**Why dates being shuffled matters:**
- Can use random K-fold validation (not time-based splits)
- But still need to be careful about generalisation

**Understanding the challenge:**
- Predicting direction of small returns is hard
- Benchmark at 50.79% barely beats random (50%)
- Even 51-52% would be valuable in real trading

#### What Worked
- Dropping SIGNED_VOLUME_1 (cleaner than filling 73% with zeros)
- Simple data cleaning approach
- K-fold validation for robust estimates
- LightGBM handles remaining missing values naturally

#### What Didn't Work
- No feature engineering = minimal improvement over benchmark
- Default hyperparameters might not be optimal
- CV score was misleadingly optimistic

#### Technical Details
- Features used: RET_1 to RET_20, SIGNED_VOLUME_2 to SIGNED_VOLUME_20, MEDIAN_DAILY_TURNOVER, GROUP
- Total: 41 features
- No feature engineering yet
- No hyperparameter tuning

### Questions to Explore
- Why is SIGNED_VOLUME_1 73% missing? Data collection issue?
- What patterns do the 4 different GROUPs represent?
- Are certain allocations more predictable than others?
- Does temporal information matter even though dates are shuffled?

## Second (V2) and V2.5 optimised but worst performing models

- Feature engineering and hyperparameter tunig
- Created 18 new engineered features
- Tested different hyperparameter configurations

#### Feature Engineering
**New features created:**
- Moving averages (RET_AVG_3, 5, 10, 15, 20)
- Volatility (RET_STD_5, 10, 20)
- Momentum (MOMENTUM_SHORT, MOMENTUM_LONG)
- Cross-allocation comparisons (RELATIVE_PERF_5, 10, 20)
- Trend direction (POSITIVE_DAYS_5, 10)

**Results:**
- Baseline CV: 53.65%
- With features CV: 54.61%
- Improvement: +0.96%

#### Hyperparameter Tuning
Tested 11 different configurations:
- Best CV: 55.92% (n_estimators=200, lr=0.07)
- Original: 54.61% (n_estimators=100, lr=0.05)

#### Leaderboard Submissions
1. Baseline (no features): 0.5090 (rank ~450)
2. With features: 0.5118 (rank#391) **BEST**
3. Optimised hyperparams: 0.5047 (rank would be ~500)  **OVERFITTING**

#### Key Learnings

**The Overfitting Problem:**
- CV scores kept improving (53.65% → 54.61% → 55.92%)
- Leaderboard scores got WORSE (50.90% → 51.18% → 50.47%)
- Gap between CV and leaderboard grew (2.7% → 3.4% → 5.5%)

**Critical lesson:** Optimising for CV score can hurt real performance

**What went wrong:**
- More trees + higher learning rate = memorised training patterns
- Features that work on training data don't generalise to test
- Need to be more conservative with complexity

#### What Worked
- Feature engineering (modest improvement: +0.28% on leaderboard)
- Keeping it simple (baseline features > over-optimised)

#### What Didn't Work
- Aggressive hyperparameter tuning (overfitted severely)
- Trusting CV scores blindly
- Adding complexity without validation on hold-out test

### Questions to Explore
- Why is CV-to-leaderboard gap so large?
- Are we validating on the wrong distribution?
- Would simpler features work better?
- Should we use fewer features, not more?

### Important Reminder
**Best model so far: Feature engineering with default hyperparameters**
- Don't always chase higher CV scores
- Simplicity often beats complexity
- Real test performance > validation performance

## Third ensemble and only Random Forest model(s)

#### What I Did
- Tested 4 different models individually
- Tried ensemble averaging
- Learned that simpler ≠ always worse

#### Models Tested (5-fold CV scores)
- Random Forest: 52.60%
- XGBoost: 52.43%
- LightGBM Simple: 52.40%
- Logistic Regression: 50.77%

#### Ensemble Results
- All 4 models: 52.45% (worse than RF alone)
- Tree models only: 52.51% (still worse)
- **Conclusion:** Models too similar, no diversity benefit

#### Leaderboard Submissions
- Random Forest: 0.5101 (rank ~430)
- **Still best: LightGBM + Features: 0.5118** 

#### Key Learnings
- Ensemble only helps if models are diverse (make different mistakes)
- The tree models all learned same patterns → no benefit from averaging
- Simpler models = better generalisation (smaller CV-to-leaderboard gap)
- Gap analysis:
  - Optimized model: 5.5% gap (severe overfitting)
  - Random Forest: 1.6% gap (good generalisation)
  - LightGBM + features: 3.4% gap (moderate overfitting)

#### What Worked
- Testing multiple models to understand their behavior
- Conservative hyperparameters (prevent overfitting)

#### What Didn't Work
- Ensemble (models not diverse enough)
- Including weak model (LogReg) in ensemble