MinariExperienceReplay¶
- class torchrl.data.datasets.MinariExperienceReplay(dataset_id, batch_size: int, *, root: str | Path | None = None, download: bool | str = True, sampler: Sampler | None = None, writer: Writer | None =None, collate_fn: Callable | None = None, pin_memory: bool = False, prefetch: int | None = None, transform: 'torchrl.envs.Transform' | None = None, split_trajs: bool = False)[source]¶
Minari 回放經驗資料集。
在 Minari 官方網站了解更多資訊：https://minari.farama.org/
資料格式遵循 TED 約定。
- 引數：
dataset_id (str) – 要下載的資料集。必須是 MinariExperienceReplay.available_datasets 的一部分。
batch_size (int) – 取樣時使用的批次大小。如有必要，可以透過 data.sample(batch_size) 覆蓋。
- 關鍵字引數：
root (Path 或 str, 可選) – Minari 資料集根目錄。實際的資料集記憶體對映檔案將儲存在 <root>/<dataset_id> 下。如果未提供，則預設為 ~/.cache/torchrl/atari.minari`。
download (bool 或 str, 可選) – 如果未找到資料集，是否應下載。預設為
True。也可以將 download 設定為"force"，在這種情況下將覆蓋已下載的資料。sampler (Sampler, 可選) – 要使用的取樣器。如果未提供，將使用預設的 RandomSampler()。
writer (Writer, 可選) – 要使用的寫入器。如果未提供，將使用預設的
ImmutableDatasetWriter。collate_fn (callable, 可選) – 合併樣本列表以形成 Tensor(s)/輸出的小批次。在使用對映風格資料集進行批次載入時使用。
pin_memory (bool) – 是否應在 rb 樣本上呼叫 pin_memory()。
prefetch (int, 可選) – 使用多執行緒預取的下一批次的數量。
transform (Transform, 可選) – 呼叫 sample() 時要執行的 Transform。要連結 transforms，請使用
Compose類。split_trajs (bool, 可選) – 如果為
True，軌跡將沿第一維拆分並進行填充以使其形狀匹配。要拆分軌跡，將使用 "done" 訊號，該訊號透過done = truncated | terminated恢復。換句話說，假定任何truncated或terminated訊號都等同於軌跡的結束。預設為False。
- 變數：
available_datasets – 可下載的可用資料集列表。
注意
文字資料目前從包裝的資料集中丟棄，因為 PyTorch 沒有原生方式表示文字資料。如果需要此功能，請在 TorchRL 的 GitHub 倉庫上提交一個 issue。
示例
>>> from torchrl.data.datasets.minari_data import MinariExperienceReplay >>> data = MinariExperienceReplay("door-human-v1", batch_size=32, download="force") >>> for sample in data: ... torchrl_logger.info(sample) ... break TensorDict( fields={ action: Tensor(shape=torch.Size([32, 28]), device=cpu, dtype=torch.float32, is_shared=False), index: Tensor(shape=torch.Size([32]), device=cpu, dtype=torch.int64, is_shared=False), info: TensorDict( fields={ success: Tensor(shape=torch.Size([32]), device=cpu, dtype=torch.bool, is_shared=False)}, batch_size=torch.Size([32]), device=cpu, is_shared=False), next: TensorDict( fields={ done: Tensor(shape=torch.Size([32, 1]), device=cpu, dtype=torch.bool, is_shared=False), info: TensorDict( fields={ success: Tensor(shape=torch.Size([32]), device=cpu, dtype=torch.bool, is_shared=False)}, batch_size=torch.Size([32]), device=cpu, is_shared=False), observation: Tensor(shape=torch.Size([32, 39]), device=cpu, dtype=torch.float64, is_shared=False), reward: Tensor(shape=torch.Size([32, 1]), device=cpu, dtype=torch.float64, is_shared=False), state: TensorDict( fields={ door_body_pos: Tensor(shape=torch.Size([32, 3]), device=cpu, dtype=torch.float64, is_shared=False), qpos: Tensor(shape=torch.Size([32, 30]), device=cpu, dtype=torch.float64, is_shared=False), qvel: Tensor(shape=torch.Size([32, 30]), device=cpu, dtype=torch.float64, is_shared=False)}, batch_size=torch.Size([32]), device=cpu, is_shared=False), terminated: Tensor(shape=torch.Size([32, 1]), device=cpu, dtype=torch.bool, is_shared=False), truncated: Tensor(shape=torch.Size([32, 1]), device=cpu, dtype=torch.bool, is_shared=False)}, batch_size=torch.Size([32]), device=cpu, is_shared=False), observation: Tensor(shape=torch.Size([32, 39]), device=cpu, dtype=torch.float64, is_shared=False), state: TensorDict( fields={ door_body_pos: Tensor(shape=torch.Size([32, 3]), device=cpu, dtype=torch.float64, is_shared=False), qpos: Tensor(shape=torch.Size([32, 30]), device=cpu, dtype=torch.float64, is_shared=False), qvel: Tensor(shape=torch.Size([32, 30]), device=cpu, dtype=torch.float64, is_shared=False)}, batch_size=torch.Size([32]), device=cpu, is_shared=False)}, batch_size=torch.Size([32]), device=cpu, is_shared=False)
- add(data: TensorDictBase) int¶
向回放緩衝區新增單個元素。
- 引數：
data (Any) – 要新增到回放緩衝區的資料
- 返回：
資料在回放緩衝區中的索引。
- append_transform(transform: Transform, *, invert: bool = False) ReplayBuffer¶
在末尾新增 transform。
呼叫 sample 時，transform 將按順序應用。
- 引數：
transform (Transform) – 要新增的 transform
- 關鍵字引數：
invert (bool, 可選) – 如果為
True，transform 將被反轉（寫入時呼叫 forward，讀取時呼叫 inverse）。預設為False。
示例
>>> rb = ReplayBuffer(storage=LazyMemmapStorage(10), batch_size=4) >>> data = TensorDict({"a": torch.zeros(10)}, [10]) >>> def t(data): ... data += 1 ... return data >>> rb.append_transform(t, invert=True) >>> rb.extend(data) >>> assert (data == 1).all()
- property data_path: Path¶
資料集的路徑，包含 split。
- property data_path_root: Path¶
資料集根目錄的路徑。
- delete()¶
從磁碟刪除資料集儲存。
- dump(*args, **kwargs)¶
dumps() 的別名。
- dumps(path)¶
將回放緩衝區儲存到磁碟上的指定路徑。
- 引數：
path (Path 或 str) – 儲存回放緩衝區的路徑。
示例
>>> import tempfile >>> import tqdm >>> from torchrl.data import LazyMemmapStorage, TensorDictReplayBuffer >>> from torchrl.data.replay_buffers.samplers import PrioritizedSampler, RandomSampler >>> import torch >>> from tensordict import TensorDict >>> # Build and populate the replay buffer >>> S = 1_000_000 >>> sampler = PrioritizedSampler(S, 1.1, 1.0) >>> # sampler = RandomSampler() >>> storage = LazyMemmapStorage(S) >>> rb = TensorDictReplayBuffer(storage=storage, sampler=sampler) >>> >>> for _ in tqdm.tqdm(range(100)): ... td = TensorDict({"obs": torch.randn(100, 3, 4), "next": {"obs": torch.randn(100, 3, 4)}, "td_error": torch.rand(100)}, [100]) ... rb.extend(td) ... sample = rb.sample(32) ... rb.update_tensordict_priority(sample) >>> # save and load the buffer >>> with tempfile.TemporaryDirectory() as tmpdir: ... rb.dumps(tmpdir) ... ... sampler = PrioritizedSampler(S, 1.1, 1.0) ... # sampler = RandomSampler() ... storage = LazyMemmapStorage(S) ... rb_load = TensorDictReplayBuffer(storage=storage, sampler=sampler) ... rb_load.loads(tmpdir) ... assert len(rb) == len(rb_load)
- empty()¶
清空回放緩衝區並將游標重置為 0。
- extend(tensordicts: TensorDictBase) Tensor¶
使用 iterable 中包含的一個或多個元素擴充套件回放緩衝區。
如果存在，將呼叫 inverse transforms。
- 引數：
data (iterable) – 要新增到回放緩衝區的資料集合。
- 返回：
新增到回放緩衝區的資料的索引。
警告
處理值列表時，
extend()的簽名可能含糊不清，它們應被解釋為 PyTree（在這種情況下，列表中的所有元素將被放入儲存中儲存的 PyTree 的一個切片中），或者是一個要逐個新增的值列表。為了解決這個問題，TorchRL 對 list 和 tuple 進行了明確區分：tuple 將被視為 PyTree，list（在根級別）將被解釋為要逐個新增到緩衝區的值棧。對於ListStorage例項，只能提供未繫結的元素（不能是 PyTree）。
- insert_transform(index: int, transform: Transform, *, invert: bool = False) ReplayBuffer¶
插入 transform。
呼叫 sample 時，transform 按順序執行。
- 引數：
index (int) – 插入 transform 的位置。
transform (Transform) – 要新增的 transform
- 關鍵字引數：
invert (bool, 可選) – 如果為
True，transform 將被反轉（寫入時呼叫 forward，讀取時呼叫 inverse）。預設為False。
- load(*args, **kwargs)¶
loads() 的別名。
- loads(path)¶
在給定路徑載入回放緩衝區狀態。
緩衝區應具有匹配的元件，並且使用
dumps()儲存。- 引數：
path (Path 或 str) – 回放緩衝區儲存的路徑。
有關更多資訊，請參見
dumps()。
- preprocess(fn: Callable[[TensorDictBase], TensorDictBase], dim: int = 0, num_workers: int | None = None, *, chunksize: int | None = None, num_chunks: int | None = None, pool: mp.Pool | None = None, generator: torch.Generator | None = None, max_tasks_per_child: int | None = None, worker_threads: int = 1, index_with_generator: bool = False, pbar: bool = False, mp_start_method: str | None = None, num_frames: int | None = None, dest: str | Path) TensorStorage¶
預處理資料集並返回包含格式化資料的新儲存。
資料 transform 必須是單一的（作用於資料集的單個樣本）。
引數和關鍵字引數被轉發到
map()。資料集隨後可以使用
delete()刪除。- 關鍵字引數：
dest (path 或 等效) – 新資料集的位置路徑。
num_frames (int, 可選) – 如果提供，將只轉換前 num_frames 個數據。這對於首先除錯 transform 很有用。
返回：一個用於
ReplayBuffer例項的新儲存。示例
>>> from torchrl.data.datasets import MinariExperienceReplay >>> >>> data = MinariExperienceReplay( ... list(MinariExperienceReplay.available_datasets)[0], ... batch_size=32 ... ) >>> print(data) MinariExperienceReplay( storages=TensorStorage(TensorDict( fields={ action: MemoryMappedTensor(shape=torch.Size([1000000, 8]), device=cpu, dtype=torch.float32, is_shared=True), episode: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.int64, is_shared=True), info: TensorDict( fields={ distance_from_origin: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), forward_reward: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), goal: MemoryMappedTensor(shape=torch.Size([1000000, 2]), device=cpu, dtype=torch.float64, is_shared=True), qpos: MemoryMappedTensor(shape=torch.Size([1000000, 15]), device=cpu, dtype=torch.float64, is_shared=True), qvel: MemoryMappedTensor(shape=torch.Size([1000000, 14]), device=cpu, dtype=torch.float64, is_shared=True), reward_ctrl: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), reward_forward: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), reward_survive: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), success: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.bool, is_shared=True), x_position: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), x_velocity: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), y_position: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), y_velocity: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True)}, batch_size=torch.Size([1000000]), device=cpu, is_shared=False), next: TensorDict( fields={ done: MemoryMappedTensor(shape=torch.Size([1000000, 1]), device=cpu, dtype=torch.bool, is_shared=True), info: TensorDict( fields={ distance_from_origin: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), forward_reward: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), goal: MemoryMappedTensor(shape=torch.Size([1000000, 2]), device=cpu, dtype=torch.float64, is_shared=True), qpos: MemoryMappedTensor(shape=torch.Size([1000000, 15]), device=cpu, dtype=torch.float64, is_shared=True), qvel: MemoryMappedTensor(shape=torch.Size([1000000, 14]), device=cpu, dtype=torch.float64, is_shared=True), reward_ctrl: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), reward_forward: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), reward_survive: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), success: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.bool, is_shared=True), x_position: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), x_velocity: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), y_position: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), y_velocity: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True)}, batch_size=torch.Size([1000000]), device=cpu, is_shared=False), observation: TensorDict( fields={ achieved_goal: MemoryMappedTensor(shape=torch.Size([1000000, 2]), device=cpu, dtype=torch.float64, is_shared=True), desired_goal: MemoryMappedTensor(shape=torch.Size([1000000, 2]), device=cpu, dtype=torch.float64, is_shared=True), observation: MemoryMappedTensor(shape=torch.Size([1000000, 27]), device=cpu, dtype=torch.float64, is_shared=True)}, batch_size=torch.Size([1000000]), device=cpu, is_shared=False), reward: MemoryMappedTensor(shape=torch.Size([1000000, 1]), device=cpu, dtype=torch.float64, is_shared=True), terminated: MemoryMappedTensor(shape=torch.Size([1000000, 1]), device=cpu, dtype=torch.bool, is_shared=True), truncated: MemoryMappedTensor(shape=torch.Size([1000000, 1]), device=cpu, dtype=torch.bool, is_shared=True)}, batch_size=torch.Size([1000000]), device=cpu, is_shared=False), observation: TensorDict( fields={ achieved_goal: MemoryMappedTensor(shape=torch.Size([1000000, 2]), device=cpu, dtype=torch.float64, is_shared=True), desired_goal: MemoryMappedTensor(shape=torch.Size([1000000, 2]), device=cpu, dtype=torch.float64, is_shared=True), observation: MemoryMappedTensor(shape=torch.Size([1000000, 27]), device=cpu, dtype=torch.float64, is_shared=True)}, batch_size=torch.Size([1000000]), device=cpu, is_shared=False)}, batch_size=torch.Size([1000000]), device=cpu, is_shared=False)), samplers=RandomSampler, writers=ImmutableDatasetWriter(), batch_size=32, transform=Compose( ), collate_fn=<function _collate_id at 0x120e21dc0>) >>> from torchrl.envs import CatTensors, Compose >>> from tempfile import TemporaryDirectory >>> >>> cat_tensors = CatTensors( ... in_keys=[("observation", "observation"), ("observation", "achieved_goal"), ... ("observation", "desired_goal")], ... out_key="obs" ... ) >>> cat_next_tensors = CatTensors( ... in_keys=[("next", "observation", "observation"), ... ("next", "observation", "achieved_goal"), ... ("next", "observation", "desired_goal")], ... out_key=("next", "obs") ... ) >>> t = Compose(cat_tensors, cat_next_tensors) >>> >>> def func(td): ... td = td.select( ... "action", ... "episode", ... ("next", "done"), ... ("next", "observation"), ... ("next", "reward"), ... ("next", "terminated"), ... ("next", "truncated"), ... "observation" ... ) ... td = t(td) ... return td >>> with TemporaryDirectory() as tmpdir: ... new_storage = data.preprocess(func, num_workers=4, pbar=True, mp_start_method="fork", dest=tmpdir) ... rb = ReplayBuffer(storage=new_storage) ... print(rb) ReplayBuffer( storage=TensorStorage( data=TensorDict( fields={ action: MemoryMappedTensor(shape=torch.Size([1000000, 8]), device=cpu, dtype=torch.float32, is_shared=True), episode: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.int64, is_shared=True), next: TensorDict( fields={ done: MemoryMappedTensor(shape=torch.Size([1000000, 1]), device=cpu, dtype=torch.bool, is_shared=True), obs: MemoryMappedTensor(shape=torch.Size([1000000, 31]), device=cpu, dtype=torch.float64, is_shared=True), observation: TensorDict( fields={ }, batch_size=torch.Size([1000000]), device=cpu, is_shared=False), reward: MemoryMappedTensor(shape=torch.Size([1000000, 1]), device=cpu, dtype=torch.float64, is_shared=True), terminated: MemoryMappedTensor(shape=torch.Size([1000000, 1]), device=cpu, dtype=torch.bool, is_shared=True), truncated: MemoryMappedTensor(shape=torch.Size([1000000, 1]), device=cpu, dtype=torch.bool, is_shared=True)}, batch_size=torch.Size([1000000]), device=cpu, is_shared=False), obs: MemoryMappedTensor(shape=torch.Size([1000000, 31]), device=cpu, dtype=torch.float64, is_shared=True), observation: TensorDict( fields={ }, batch_size=torch.Size([1000000]), device=cpu, is_shared=False)}, batch_size=torch.Size([1000000]), device=cpu, is_shared=False), shape=torch.Size([1000000]), len=1000000, max_size=1000000), sampler=RandomSampler(), writer=RoundRobinWriter(cursor=0, full_storage=True), batch_size=None, collate_fn=<function _collate_id at 0x168406fc0>)
- register_load_hook(hook: Callable[[Any], Any])¶
為儲存註冊一個載入鉤子。
注意
鉤子當前在儲存回放緩衝區時不會被序列化：每次建立緩衝區時必須手動重新初始化它們。
- register_save_hook(hook: Callable[[Any], Any])¶
為儲存註冊一個儲存鉤子。
注意
鉤子當前在儲存回放緩衝區時不會被序列化：每次建立緩衝區時必須手動重新初始化它們。
- sample(batch_size: int | None = None, return_info: bool = False, include_info: bool = None) TensorDictBase¶
從回放緩衝區取樣一批資料。
使用 Sampler 取樣索引，並從 Storage 中檢索它們。
- 引數：
batch_size (int, 可選) – 要收集的資料大小。如果未提供，此方法將按照取樣器指示的批次大小進行取樣。
return_info (bool) – 是否返回資訊。如果為 True，結果是元組 (data, info)。如果為 False，結果是資料。
- 返回：
一個包含在回放緩衝區中選擇的一批資料的 tensordict。如果設定了 return_info 標誌為 True，則返回一個包含此 tensordict 和 info 的元組。
- save(*args, **kwargs)¶
dumps() 的別名。
- set_storage(storage: Storage, collate_fn: Callable | None = None)¶
在回放緩衝區設定一個新的儲存並返回之前的儲存。
- 引數：
storage (Storage) – 緩衝區的新儲存。
collate_fn (callable, 可選) – 如果提供，collate_fn 將設定為此值。否則，它將重置為預設值。
- property write_count¶
透過 add 和 extend 方法寫入緩衝區中的專案總數。
