Discussion Report: Flat Band Evaluation Methodology & Strategic Direction

Date: 2026-02-11 Participants: Ryotaro, Masaki Adachi Topic: Refinement of Pairwise Evaluation Metrics, Discrepancy Analysis, and Roadmap Definition

---

1. Executive Summary

The meeting focused on reviewing the results of the newly implemented Pairwise Evaluation Framework (both Physics-based and Vision-based). While the framework successfully ranks materials, a discrepancy between the two methods (mp-29950 vs mp-19961) sparked a deeper discussion on the definition of "flatness" and the subjectivity of expert evaluation.

Key Decision: instead of manually tuning weights or assuming a linear objective function, we will shift to an Inverse Estimation approach. We will create a small "Golden Dataset" of expert-labeled pairs to learn the optimal weights for the scoring function.

References:

• Vision Evaluation Report
• Pairwise Evaluation Report
• Vision Documentation

---

2. Review of Current Status

Ryotaro presented the progress on two parallel evaluation tracks:

A. Physics-Based Evaluation (Action 1)

• Method: Weighted sum of 4 metrics (Flatness, Extent, Near-Fermi, Isolation) with Monte Carlo weight perturbation.
• Result: mp-29950 was the undisputed winner (100% Win Rate).
• Observation: This material is numerically very flat within the defined energy window but may have dispersive features outside it.

B. Vision-Based Evaluation (Action 2)

• Method: GPT-4o Vision "Zero-Shot" pairwise comparison.
• Result: mp-19961 was the winner (100% Win Rate). mp-29950 dropped to 4th place.
• Observation: The LLM penalized mp-29950 for "messiness" or "dispersion," favoring the cleaner, straighter lines of mp-19961.

The Discrepancy

The divergence between the "Numerical Winner" and the "Visual Winner" highlights that our current physics formula might over-prioritize local numerical bandwidth ($W$) while the vision model (and likely human experts) implicitly value global "cleanliness" and linearity.

---

3. Technical Discussion & Critique

3.1. Physical Definition of Flatness

Adachi-san emphasized that "Flatness" ($W \to 0$) is a proxy, not the ultimate physical goal.

• U/W Ratio: The true driver of interesting physics (correlation effects, superconductivity) is the ratio of Coulomb repulsion ($U$) to Bandwidth ($W$).
• Partial Flatness: A band does not need to be flat across the entire Brillouin Zone. A Van Hove Singularity (inflection point where $\nabla E = 0$) leads to a DOS peak and is physically sufficient.
• Constraint: The current strict "global flatness" requirement might be excluding interesting candidates that are only flat in specific k-path segments.

3.2. Evaluation Philosophy: "Inverse Estimation"

A core realization was that we do not know the "Ground Truth" function. Manually debating whether "Isolation" should be weighted 0.1 or 0.2 is inefficient.

• Proposal: Treat the expert's subjective ranking (e.g., Prof. Okabe's intuition) as the target $y$.
• Action: Instead of "Forward Optimization" (picking weights $\to$ finding materials), we will use "Inverse Optimization":
  1. Select a small, diverse set of materials ($N \approx 20$).
  2. Get expert labels (Human Preference $A \succ B$).
  3. Learn the weights ($w_{flat}, w_{ext}, \dots$) that best reproduce these labels.

3.3. Vision Model Skepticism

Adachi-san noted that the Vision model returned "Confidence: High" for almost all decisions.

• Critique: In valid scientific assessment, comparing two very similar flat bands should yield low confidence.
• Hypothesis: The model might be "hallucinating certainty."
• Correction: Future prompts should encourage "Tie" declarations or lower confidence scores for close calls.

---

4. Strategic Pivot: The "Human-in-the-Loop" Pipeline

The team agreed to separate the workflow into two distinct phases:

Phase A: Calibrating the "Ground Truth" (Immediate)

1. Dataset Construction: Select ~20 materials.
   • Include: "Obvious Flat", "Obvious Non-Flat", and crucially "Ambiguous/Intermediate" cases.
   • Note: The current dataset is too binary (Best vs Worst). We need "Middle" examples to train the discriminator.
2. Manual Labeling: Ryotaro (and potentially other experts) will manually rank these pairs to establish a Golden Set.
3. Weight Regression: Use this set to calculate the optimal weights for the Physics Function.

Phase B: Generative Exploration (Medium Term)

Once the Evaluation Function is calibrated (Phase A):

1. Generation: Use Diffusion Models (crystallographic generation) or Graph Transformers to generate new candidates.
2. Goal: Demonstrate that the "Hit Rate" (probability of generating a flat band) increases when using the calibrated evaluator and agentic loop compared to random generation.
3. Pareto Frontier: If a single "Best" weighting is impossible, plot the Pareto Frontier of the 4 metrics to show the trade-off surface.

---

5. New Idea: "Unsolved Problems" Benchmark

Adachi-san proposed a broader initiative inspired by the "Erdős Problems" or "LDS Problems" in mathematics.

• Concept: Create a curated list of "Unsolved Problems in Materials Science" (e.g., specific contradictions in theory, un-synthesized but predicted structures).
• Usage: Use this as a high-level benchmark for AI Agents. If an agent can propose a valid solution path to an open problem, it demonstrates true reasoning capability beyond simple "property optimization."
• Action: Start compiling a list of such problems by consulting with theoretical physicists.

---

6. Action Items

Owner	Task	Deadline
Ryotaro	Dataset Expansion: Select ~20 materials including "intermediate" flatness cases (random selection from MP).	Next Meeting
Ryotaro	Manual Evaluation: Perform pairwise comparison (Human Eye) on this set to create the "Golden Labels."	Next Meeting
Ryotaro	Inverse Weighting: Attempt to fit the Physics Function weights to match the Manual Labels.	Next Meeting
Ryotaro	Pareto Plot: Visualize the trade-off between the 4 current metrics (Flatness vs Isolation etc.).	Next Meeting
Team	Schedule: Next regular meeting set for Wed 17:00 PST / Thu 10:00 JST.	-

---

7. Links & Resources

• Recording Transcript: Dropbox Link
• Zoom Summary: Zoom Doc