nvflare-convert-lightning

NVFLARE Convert PyTorch Lightning

Use When

Use only when the user asks to convert PyTorch Lightning code into an NVFLARE federated training job; require both federation intent and Lightning ownership. Treat requests for multiple sites or institutions to train collaboratively while each site's data remains local as federation intent, even when the request does not say "federated" or "NVFLARE." Lightning source evidence alone is not sufficient. Relevant source may contain a LightningModule, LightningDataModule, a Trainer fit/validate/test loop, Lightning callbacks, checkpointing, or loggers. Supported: the PyTorch recipe family with flare.patch(trainer) as the model exchange integration, Lightning-native evaluation, custom aggregation through the same recipe aggregator= hook, and local validation and export.

Standard Path

Always read this converter SKILL.md with ../nvflare-shared/references/conversion-common.md. For an explicit-FedAvg conversion, load only these references, in workflow order:

1. During inspection, references/lightning-detection.md.
2. For generated splits, relative paths, or per-site data locations, ../nvflare-shared/references/site-data-and-paths.md.
3. After nvflare recipe show fedavg-pt --format json, ../nvflare-shared/references/pytorch-family-recipe-construction.md.
4. During conversion, references/lightning-conversion.md, then ../nvflare-shared/references/pytorch-model-exchange.md.
5. Only after generated files exist, ../nvflare-shared/references/validation-evidence.md, then references/lightning-validation.md.

Complete each workflow phase before loading the next phase's reference. Do not enumerate reference directories or preload validation, DDP/tracking, broad workflow, dependency, runtime-output, or reporting references. Do not depend on NVFLARE repository examples.

Do Not Use When

Do not use for non-federated Lightning changes such as DDP-only configuration, profiling, inference serving, callbacks, early stopping, or schedulers; or for plain torch.nn.Module manual training loops without Lightning (route to nvflare-convert-pytorch), Hugging Face Trainer (route to nvflare-convert-huggingface), TensorFlow, XGBoost, scikit-learn, a failed job (route to nvflare-diagnose-job), federated statistics without training (route to nvflare-fed-stats), or generic Lightning debugging without FLARE intent; when the inspected project actively contains both Lightning and Hugging Face Trainer entrypoints, route to nvflare-orient. Out of conversion scope: production deployment, Kubernetes, POC lifecycle, deployment privacy/security policy design, custom distributed launch policies not expressible by product APIs, experiment tracking redesign, and experiment search across recipes. Privacy-protection requests — homomorphic encryption (HE) / encrypted aggregation, differential privacy, and privacy filters — are not supported: they require provisioning or deployment policy beyond conversion scope, so report such a request as unsupported and route it to provisioning/deployment, never substituting an unprotected recipe or disclaimer. If a request combines federated statistics and model-training conversion, treat it as two independent jobs and workflows: do not merge or automatically chain them, do not route the combination to nvflare-orient, and ask which workflow to run first before generating or running either job. Recommend nvflare-fed-stats first only when the user's purpose is to understand data distribution; handle conversion later as a separate request.

Workflow

1. Apply ../nvflare-shared/references/conversion-common.md for the whole conversion; this SKILL.md states only the framework-specific deltas.
2. Inspect before editing with nvflare agent inspect source <path> --format json plus direct reading; fact extraction is static. Confirm Lightning versus plain PyTorch and hand off to nvflare-convert-pytorch when no Lightning evidence exists. If inspection recommends nvflare-orient for active Lightning and Hugging Face Trainer owners, stop and hand off before editing.
3. Apply the dependency-install ordering rule in ../nvflare-shared/references/conversion-common.md before any Python command imports user, Lightning, NVFLARE, or declared dependency modules. Determine applicable dependencies from the selected execution path first. If required data artifacts already exist and static inspection shows that the selected path will not reach a download helper or its imports, treat its download-only requirements as inapplicable: do not install or import-probe them. Probe only modules the generated conversion and selected validation path will execute, and keep an optional probe separate and exit-zero when unavailable.
4. Identify the existing LightningModule, LightningDataModule, trainer construction, callbacks, checkpointing, validation_step/test_step and dataloaders, metrics, logger usage, source partition evidence, distributed process-spawning evidence, custom aggregation intent, and the concrete model constructor values that server and clients must share.
5. Reuse the PyTorch recipe family; Lightning is not a separate recipe family. For the standard case — the user explicitly requests FedAvg and inspection identifies Lightning — run nvflare recipe show fedavg-pt --format json directly and construct it. For fedavg-pt, import FedAvgRecipe only from nvflare.app_opt.pt.recipes.fedavg, never from nvflare.recipe. Use FedEval for evaluation-only. After every recipe show, derive construction capabilities from the construction reference. Then use that documented path: do not run exploratory NVFLARE imports or use inspect, hasattr, constant discovery, SDK source/docstring reads, or lifecycle probes. If a required detail is absent, report a skill gap or fail closed instead of guessing. Call recipe.execute(SimEnv(...)).
6. Convert the training entry point to the Lightning Client API: build the Trainer, call flare.patch(trainer), and let the patched trainer own model exchange. Keep evaluation inside Lightning per references/lightning-conversion.md and use self.log. Derive evaluate_only=True only for FedEval; omit it for training recipes so its default stays False. Derive evaluate_before_train = recipe_algorithm != "cyclic": Cyclic persists only its final sequential model; every other algorithm uses explicit validation for server metrics and, for training, best-model selection. Verify the key in server evidence or fail closed.
7. Add or update job.py under the shared constructor-serialization rule. Use the recipe's class_path or path key plus complete args when values are needed; a permitted zero-argument instance is the complete module. Add requested aggregator= wiring and the metric, tensor-transport, server offload, and execution settings derived from the shared PyTorch-family construction profile. If sites need distinct train_args, make every site override the complete argument string; never split shared arguments and a site-specific data path across recipe-level and per-site values expecting a merge.
8. Immediately after generated files exist and before any preflight, smoke test, cleanup, validation, or execution command, load the two validation references in Standard Path order. Before executing a full run, select and record exactly one final validation target:
   • for a requested local or first-run simulation without an export claim, run python job.py and do not export or run the exported simulator afterward;
   • for a requested exported/deployable artifact, export first and run only the exported folder with the simulator CLI; do not first run python job.py. If the selected full-run target fails, diagnose it, apply a scoped fix, and rerun that same target. Change targets only when evidence shows the original target does not represent the requested artifact, and record that reason. Export inspection belongs only to the exported path. Keep cleanup, export, and simulation as separate tool calls; never combine recursive cleanup with execution. Stop at the first failed validation rung before diagnosing it; do not add speculative recovery probes. Use the environment and permission mechanisms supplied by the agent host; do not inspect or enforce its security boundary.
9. Report the recipe, changed files, selected validation target, validation status, metrics, and exact artifact paths.

Non-standard Cases

Load only the reference matching an encountered case:

• ../nvflare-shared/references/conversion-workflow.md for an unresolved non-standard rerun, authorization, or missing-semantics case; it no longer holds the data-location or partitioning contracts.
• ../nvflare-shared/references/pytorch-family-recipe-selection.md for an ambiguous or non-FedAvg algorithm; use its catalog for FedAvg, FedOpt, FedProx, SCAFFOLD, Cyclic, Swarm, or FedEval, and reserve nvflare recipe list for these cases.
• ../nvflare-shared/references/dependency-install.md when an applicable dependency is missing.
• ../nvflare-shared/references/runtime-output-guidance.md for a read-only source root or user-chosen output destination.
• references/lightning-ddp-and-tracking.md when inspection finds its trigger.
• ../nvflare-shared/references/metrics-and-artifact-reporting.md when normal metric artifacts are absent or inconsistent.

Requirements

• Must integrate through flare.patch(trainer) and let the patched trainer own model exchange. Must not generate a manual FLModel send/receive path as the default Lightning exchange, and must not pass the received input_model into the Trainer.
• Must treat flare.receive() inside the patched loop as optional metadata or task-progression access only, not as a second model-load path.
• Must keep evaluation inside Lightning (trainer.validate/trainer.test, validation_step, self.log); must not generate a raw PyTorch model.eval() loop for ordinary Lightning conversion.
• Except for Cyclic, must run an explicit standalone trainer.validate(...) before trainer.fit(...) and rely on the patched callback to attach its finite scalar metrics; never populate model.__fl_meta__[MetaKey.INITIAL_METRICS]. Validation inside trainer.fit(...) is not a received-global-model metric. Cyclic must skip the pre-fit call and report its persisted final model, not a best model.
• Must audit model constructor arguments before writing job.py by reading the LightningModule.__init__ signature and the selected recipe's model parameter from nvflare recipe show <recipe-name> --format json, not by reading NVFLARE library source. Emit the recipe-documented class_path or path key plus complete args for every required or overridden value. Direct LightningModule use is allowed only when unchanged zero-argument defaults reconstruct it. Values must be clear from source, configuration, or supplied metadata. Otherwise ask one semantic question when an answer channel exists or fail closed.
• Must use the PyTorch recipe family; must not invent a Lightning-only recipe. Apply the construction reference after recipe show; it is canonical for optional recipe parameters, model selection, tensor transport, server disk offload, and execution mode.
• Must preserve local-only callbacks and logger behavior where safe. Existing network-connected tracking, upload callbacks, and custom/unknown loggers are evidence, not a user request: keep them disabled during validation unless explicitly requested, and do not ask solely to enable them. This narrows references/lightning-conversion.md.
• Must not make non-PyTorch-family skills load ../nvflare-shared/references/pytorch-model-exchange.md.
• Site partitioning, custom aggregation, the Source Of Truth Boundary, and user input/authorization follow ../nvflare-shared/references/conversion-common.md.