results-analysis

Results Analysis for ML/AI Research

A systematic experimental results analysis workflow connecting experimental data to paper writing.

Core Features

This skill provides three core capabilities:

1. Experimental Data Analysis - Read and analyze experimental data in various formats
2. Statistical Validation - Perform statistical significance tests and performance comparisons
3. Paper Content Generation - Generate text and visualizations for the Results section

When to Use

Use this skill when you need to:

• Analyze experimental results (CSV, JSON, TensorBoard logs)
• Generate the Results section of a paper
• Compare performance across multiple models
• Perform statistical significance tests
• Create publication-quality visualizations
• Validate the reliability of experimental results

Workflow

Standard Analysis Pipeline

Data Loading → Data Validation → Statistical Analysis → Visualization → Writing → Quality Check

Step 1: Data Loading and Validation

Supported Data Formats:

• CSV files - Tabular data
• JSON files - Structured results
• TensorBoard logs - Training curves
• Python pickle - Complex objects

Data Validation Checks:

• Completeness check - Missing values, outliers
• Consistency check - Data format, units
• Reproducibility check - Random seeds, version info

Select appropriate tools for data loading and preliminary validation based on data format.

Step 2: Statistical Analysis

Basic Statistics:

• Mean
• Standard Deviation
• Standard Error
• Confidence Interval

Significance Tests:

• t-test - Two-group comparison
• ANOVA - Multi-group comparison
• Wilcoxon test - Non-parametric test
• Bonferroni correction - Multiple comparison correction

Select appropriate statistical tests based on data characteristics.

Key Principles:

• Report complete statistical information (mean ± std/SE)
• Specify the test method and significance level used
• Report p-values and effect sizes
• Consider multiple comparison issues
• Recompute examples and report values from the same unrounded inputs; mean, sample SD, n, test statistic, P and effect size must describe the same comparison and analysis unit.
• For a mean across datasets, first form each run's mean, then estimate variation across runs; averaging dataset SDs does not recover this uncertainty. Report only diagnostics actually run.

See references/statistical-methods.md for the complete statistical methods guide.

Step 3: Model Performance Comparison

Comparison Dimensions:

• Accuracy/Performance metrics
• Training time/Inference speed
• Model complexity/Parameter count
• Robustness/Generalization ability

Comparison Methods:

• Baseline comparison - Compare with existing methods
• Ablation study - Validate component contributions
• Cross-dataset validation - Test generalization

Systematically compare performance across different methods, ensuring fair comparison.

Step 4: Visualization

Publication-Quality Visualization Requirements:

• Vector format (PDF/EPS)
• Colorblind-friendly palette
• Clear labels and legends
• Appropriate error bars
• Readable in black-and-white print

Common Chart Types:

• Line chart - Training curves, trend analysis
• Bar chart - Performance comparison
• Box plot - Distribution display
• Heatmap - Correlation analysis
• Scatter plot - Relationship display

Use appropriate visualization tools to generate publication-quality figures.

See references/visualization-best-practices.md for the visualization guide.

Step 5: Writing the Results Section

Results Section Structure:

## Results
 
### Overview of Main Findings
[1-2 paragraphs summarizing core results]
 
### Experimental Setup

Writing Principles:

• Clearly state the hypothesis each experiment validates
• Guide readers to observe key phenomena: "Figure X shows..."
• Report complete statistical information
• Report null and negative results plainly, as findings; state the scope of the work once, in the Discussion

See references/results-writing-guide.md for the complete writing guide.

Step 6: Quality Check

Checklist:

• All values include error bars/confidence intervals
• Statistical test methods are specified
• Figures are clear and readable (including black-and-white print)
• Hyperparameter search ranges are reported
• Computational resources are specified (GPU type, time)
• Random seed settings are specified
• Results are reproducible (code/data available)

Common Mistakes and Pitfalls

Statistical Errors

❌ Wrong approach:

• Reporting only the best results (cherry-picking)
• Confusing standard deviation and standard error
• Claiming a difference without its effect size, interval and n
• Reporting several inferential tests without a multiplicity policy

✅ Correct approach:

• Report all experimental results
• Clearly specify whether standard deviation or standard error is used
• Perform statistical tests where the design supports them, with the unit of replication stated
• State one multiplicity policy (for example Holm or Benjamini-Hochberg) when several tests are reported

Visualization Errors

❌ Wrong approach:

• Using non-colorblind-friendly palettes
• Y-axis not starting from 0 (exaggerating differences)
• Missing error bars
• Overly complex figures

✅ Correct approach:

• Use Okabe-Ito or Paul Tol palettes
• Set reasonable axis ranges
• Include error bars and confidence intervals
• Keep figures clean and clear

Writing Errors

❌ Wrong approach:

• Over-interpreting results
• Not describing experimental setup
• Hiding negative results
• Missing statistical information

✅ Correct approach:

• Objectively describe observed phenomena
• Provide sufficient experimental details
• Honestly report all results
• Report complete statistical information

See references/common-pitfalls.md for the complete error patterns and fixes.

Integration with Paper Writing

Collaboration with Writing Skills

This skill focuses on experimental results analysis and works in tandem with the writing skills in this repository:

results-analysis handles:

• Data analysis and statistical tests
• Visualization generation
• Results interpretation

scientific-writing or conference-paper-writing handle:

• Complete paper structure
• Citation integration
• Venue-specific framing and formatting

Workflow Integration:

Output Format

After analysis, the following are generated:

1. Analysis Report (analysis-report.md)

   • Statistical summary
   • Key findings
   • Suggested figures

2. Visualization Files (figures/)

   • PDF format figures
   • Standalone figure captions

3. Results Draft (results-draft.md)

   • Text ready for direct use in the paper
   • Includes figure references

Examples and Templates

Example Files

The synthetic walkthrough in USAGE.md includes complete CSV runs and scripts/example_statistics.py (requires SciPy). Run it with --example usage or --example benchmark to recompute the appropriate example; it prints JSON without writing files.

Refer to the examples/ directory for complete examples:

• example-analysis-report.md - Complete analysis report example
• example-results-section.md - Paper Results section example

Workflow Overview

The complete analysis pipeline includes:

1. Data Loading - Read results from experiment output files
2. Statistical Analysis - Compute basic statistics and perform significance tests
3. Visualization - Create publication-quality figures
4. Report Generation - Integrate analysis results and visualizations

See the guides in the references/ directory for detailed methods and best practices.

Reference Resources

Detailed Guides

• references/statistical-methods.md - Complete statistical methods guide
• references/results-writing-guide.md - Results section writing standards
• references/visualization-best-practices.md - Visualization best practices
• references/common-pitfalls.md - Common errors and fixes

External Resources

• Nature Statistics Checklist
• Science Reproducibility Guidelines
• NeurIPS Paper Checklist

Best Practices Summary

Data Analysis

✅ Recommended:

• Run experiments multiple times (at least 3-5 runs)
• Report complete statistical information
• Use appropriate statistical tests
• Check data completeness

❌ Prohibited:

• Cherry-picking best results
• Claiming a difference the effect size and interval do not support
• Hiding negative results
• Not reporting experimental setup

Visualization

✅ Recommended:

• Use vector format
• Colorblind-friendly palettes
• Include error bars
• Clear labels

❌ Prohibited:

• Raster formats (PNG/JPG)
• Misleading axis scales
• Overly complex figures
• Missing legends

Writing

✅ Recommended:

• Objectively describe results
• Provide sufficient detail
• Report null and negative results plainly, as findings
• Guide reader attention

❌ Prohibited:

• Over-interpretation
• Hiding details
• Exaggerating effects
• Vague descriptions

Summary

This skill provides a systematic experimental results analysis workflow:

1. Data Loading and Validation - Ensure data quality
2. Statistical Analysis - Perform appropriate statistical tests
3. Model Comparison - Systematic performance comparison
4. Visualization - Publication-quality figures
5. Writing - Results section content
6. Quality Check - Ensure reproducibility

Following these principles produces high-quality, reproducible experimental results analysis that meets top conference standards.