OverviewHistoryStatsSecurity
npx skills add ...
Documentation
SKILL.md
npx skills add davila7/claude-code-templates --skill pufferlib
This skill should be used when working with reinforcement learning tasks including high-performance RL training, custom environment development, vectorized parallel simulation, multi-agent systems, or integration with existing RL environments (Gymnasium, PettingZoo, Atari, Procgen, etc.). Use this skill for implementing PPO training, creating PufferEnv environments, optimizing RL performance, or developing policies with CNNs/LSTMs.
npx skills add davila7/claude-code-templates --skill pufferlib
PufferLib is a high-performance reinforcement learning library designed for fast parallel environment simulation and training. It achieves training at millions of steps per second through optimized vectorization, native multi-agent support, and efficient PPO implementation (PuffeRL). The library provides the Ocean suite of 20+ environments and seamless integration with Gymnasium, PettingZoo, and specialized RL frameworks.
Use this skill when:
PuffeRL is PufferLib's optimized PPO+LSTM training algorithm achieving 1M-4M steps/second.
Quick start training:
Python training loop:
For comprehensive training guidance, read references/training.md for:
Create custom high-performance environments with the PufferEnv API.
Basic environment structure:
Use the template script: scripts/env_template.py provides complete single-agent and multi-agent environment templates with examples of:
For complete environment development, read references/environments.md for:
Achieve maximum throughput with optimized parallel simulation.
Vectorization setup:
Key optimizations:
For vectorization optimization, read references/vectorization.md for:
Build policies as standard PyTorch modules with optional utilities.
Basic policy structure:
For complete policy development, read references/policies.md for:
Seamlessly integrate environments from popular RL frameworks.
Gymnasium integration:
PettingZoo multi-agent:
Supported frameworks:
For integration details, read references/integration.md for:
scripts/train_template.py as starting pointreferences/training.md for optimizationscripts/env_template.pyreset() and step() methodspufferlib.emulate() or make()references/environments.md for advanced patternsreferences/vectorization.md if neededlayer_init for proper weight initializationreferences/policies.mdreferences/vectorization.md for systematic optimizationtrain_template.py - Complete training script template with:
env_template.py - Environment implementation templates:
training.md - Comprehensive training guide:
environments.md - Environment development guide:
vectorization.md - Vectorization optimization:
policies.md - Policy architecture guide:
integration.md - Framework integration guide:
Start simple: Begin with Ocean environments or Gymnasium integration before creating custom environments
Profile early: Measure steps per second from the start to identify bottlenecks
Use templates: scripts/train_template.py and scripts/env_template.py provide solid starting points
Read references as needed: Each reference file is self-contained and focused on a specific capability
Optimize progressively: Start with Python, profile, then optimize critical paths with C if needed
Leverage vectorization: PufferLib's vectorization is key to achieving high throughput
Monitor training: Use WandB or Neptune to track experiments and identify issues early
Test environments: Validate environment logic before scaling up training
Check existing environments: Ocean suite provides 20+ pre-built environments
Use proper initialization: Always use layer_init from pufferlib.pytorch for policies
import pufferlib
from pufferlib import PuffeRL
# Create vectorized environment
env = pufferlib.make('procgen-coinrun', num_envs=256)
# Create trainer
trainer = PuffeRL(
env=env,
policy=my_policy,
device='cuda',
learning_rate=3e-4,
batch_size=32768
)
# Training loop
for iteration in range(num_iterations):
trainer.evaluate() # Collect rollouts
trainer.train() # Train on batch
trainer.mean_and_log() # Log resultsimport numpy as np
from pufferlib import PufferEnv
class MyEnvironment(PufferEnv):
def __init__(self, buf=None):
super().__init__(buf)
# Define spaces
self.observation_space = self.make_space((4,))
self.action_space = self.make_discrete(4)
self.reset()
def reset(self):
# Reset state and return initial observation
return np.zeros(4, dtype=np.float32)
def step(self, action):
# Execute action, compute reward, check done
obs = self._get_observation()
reward = self._compute_reward()
done = self._is_done()
info = {}
return obs, reward, done, infoimport pufferlib
# Automatic vectorization
env = pufferlib.make('environment_name', num_envs=256, num_workers=8)
# Performance benchmarks:
# - Pure Python envs: 100k-500k SPS
# - C-based envs: 100M+ SPS
# - With training: 400k-4M total SPSimport torch.nn as nn
from pufferlib.pytorch import layer_init
class Policy(nn.Module):
def __init__(self, observation_space, action_space):
super().__init__()
# Encoder
self.encoder = nn.Sequential(
layer_init(nn.Linear(obs_dim, 256)),
nn.ReLU(),
layer_init(nn.Linear(256, 256)),
nn.ReLU()
)
# Actor and critic heads
self.actor = layer_init(nn.Linear(256, num_actions), std=0.01)
self.critic = layer_init(nn.Linear(256, 1), std=1.0)
def forward(self, observations):
features = self.encoder(observations)
return self.actor(features), self.critic(features)import gymnasium as gym
import pufferlib
# Wrap Gymnasium environment
gym_env = gym.make('CartPole-v1')
env = pufferlib.emulate(gym_env, num_envs=256)
# Or use make directly
env = pufferlib.make('gym-CartPole-v1', num_envs=256)# Multi-agent environment
env = pufferlib.make('pettingzoo-knights-archers-zombies', num_envs=128)# Atari
env = pufferlib.make('atari-pong', num_envs=256)
# Procgen
env = pufferlib.make('procgen-coinrun', num_envs=256)
# Minigrid
env = pufferlib.make('minigrid-empty-8x8', num_envs=256)# PettingZoo
env = pufferlib.make('pettingzoo-pistonball', num_envs=128)
# Shared policy for all agents
policy = create_policy(env.observation_space, env.action_space)
trainer = PuffeRL(env=env, policy=policy)# Create custom environment
class MyTask(PufferEnv):
# ... implement environment ...
# Vectorize and train
env = pufferlib.emulate(MyTask, num_envs=256)
trainer = PuffeRL(env=env, policy=my_policy)# Maximize throughput
env = pufferlib.make(
'my-env',
num_envs=1024, # Large batch
num_workers=16, # Many workers
envs_per_worker=64 # Optimize per worker
)uv pip install pufferlib