Attention

June 2024 Status Update: Removing DataPipes and DataLoader V2

We are re-focusing the torchdata repo to be an iterative enhancement of torch.utils.data.DataLoader. We do not plan on continuing development or maintaining the [DataPipes] and [DataLoaderV2] solutions, and they will be removed from the torchdata repo. We’ll also be revisiting the DataPipes references in pytorch/pytorch. In release torchdata==0.8.0 (July 2024) they will be marked as deprecated, and in 0.10.0 (Late 2024) they will be deleted. Existing users are advised to pin to torchdata<=0.9.0 or an older version until they are able to migrate away. Subsequent releases will not include DataPipes or DataLoaderV2. Please reach out if you suggestions or comments (please use this issue for feedback)

TorchData

This library is part of the PyTorch project. PyTorch is an open source machine learning framework.

torchdata is a Beta library of common modular data loading primitives for easily constructing flexible and performant data pipelines. And, there are a few features still in prototype stage.

Features described in this documentation are classified by release status:

Stable: These features will be maintained long-term and there should generally be no major performance limitations or gaps in documentation. We also expect to maintain backwards compatibility (although breaking changes can happen and notice will be given one release ahead of time).

Beta: Features are tagged as Beta because the API may change based on user feedback, because the performance needs to improve, or because coverage across operators is not yet complete. For Beta features, we are committing to seeing the feature through to the Stable classification. We are not, however, committing to backwards compatibility.

Prototype: These features are typically not available as part of binary distributions like PyPI or Conda, except sometimes behind run-time flags, and are at an early stage for feedback and testing.

Developer Notes:

• What is torchdata.nodes (beta)?
  • Why torchdata.nodes?
  • How does torchdata.nodes perform?
  • Design choices

API Reference:

• torchdata.nodes (beta)
• Stateful DataLoader

Tutorial and Examples:

• Getting Started With torchdata.nodes (beta)
  • Generator Example
  • Sampler Example
• Migrating to torchdata.nodes from torch.utils.data
  • Map-Style Datasets
  • IterableDatasets
• Stateful DataLoader Tutorial
  • Saving and loading state
  • Saving Custom State with Map-Style Datasets
  • Saving Custom State with Iterable-Style Datasets

PyTorch Libraries

• PyTorch
• torchtune
• torchaudio
• torchvision
• TorchElastic
• TorchServe
• PyTorch on XLA Devices

Indices

• Index