OpenMLExperienceReplay¶
- 類 torchrl.data.datasets.OpenMLExperienceReplay(name: str, batch_size: int, root: Path | None = None, sampler: Sampler | None = None, writer: Writer | None = None, collate_fn: Callable | None = None, pin_memory: bool = False, prefetch: int | None = None, transform: 'Transform' | None = None)[source]¶
用於 OpenML 資料的經驗回放緩衝區。
此類為公共資料集提供了便捷的入口。請參見“Dua, D. and Graff, C. (2017) UCI Machine Learning Repository. http://archive.ics.uci.edu/ml”
資料格式遵循 TED 約定。
透過 scikit-learn 訪問資料。請確保在檢索資料之前已安裝 sklearn 和 pandas
$ pip install scikit-learn pandas -U
- 引數：
name (str) – 支援以下資料集：
"adult_num","adult_onehot","mushroom_num","mushroom_onehot","covertype","shuttle"和"magic"。batch_size (int) – 取樣時使用的批次大小。
sampler (Sampler, optional) – 使用的取樣器。如果未提供，將使用預設的 RandomSampler()。
writer (Writer, optional) – 使用的寫入器。如果未提供，將使用預設的
ImmutableDatasetWriter。collate_fn (callable, optional) – 將樣本列表合併以形成 Tensor(s)/輸出的 mini-batch。在從 map-style 資料集批次載入時使用。
pin_memory (bool) – 是否應在 rb 樣本上呼叫 pin_memory()。
prefetch (int, optional) – 使用多執行緒預取的下一個批次的數量。
transform (Transform, optional) – 呼叫 sample() 時執行的轉換。要鏈式呼叫轉換，請使用
Compose類。
- add(data: TensorDictBase) int¶
向經驗回放緩衝區新增單個元素。
- 引數：
data (Any) – 要新增到經驗回放緩衝區的資料
- 返回：
資料在經驗回放緩衝區中的索引。
- append_transform(transform: Transform, *, invert: bool = False) ReplayBuffer¶
將轉換追加到末尾。
呼叫 sample 時，轉換按順序應用。
- 引數：
transform (Transform) – 要追加的轉換
- 關鍵字引數：
invert (bool, optional) – 如果為
True，則轉換將被反轉（在寫入時呼叫正向方法，在讀取時呼叫反向方法）。預設為False。
示例
>>> rb = ReplayBuffer(storage=LazyMemmapStorage(10), batch_size=4) >>> data = TensorDict({"a": torch.zeros(10)}, [10]) >>> def t(data): ... data += 1 ... return data >>> rb.append_transform(t, invert=True) >>> rb.extend(data) >>> assert (data == 1).all()
- 抽象 屬性 data_path: Path¶
資料集的路徑，包括分割。
- 抽象 屬性 data_path_root: Path¶
資料集根目錄的路徑。
- delete()¶
從磁碟刪除資料集儲存。
- dump(*args, **kwargs)¶
dumps() 的別名。
- dumps(path)¶
將經驗回放緩衝區儲存到磁碟上的指定路徑。
- 引數：
path (Path 或 str) – 儲存經驗回放緩衝區的路徑。
示例
>>> import tempfile >>> import tqdm >>> from torchrl.data import LazyMemmapStorage, TensorDictReplayBuffer >>> from torchrl.data.replay_buffers.samplers import PrioritizedSampler, RandomSampler >>> import torch >>> from tensordict import TensorDict >>> # Build and populate the replay buffer >>> S = 1_000_000 >>> sampler = PrioritizedSampler(S, 1.1, 1.0) >>> # sampler = RandomSampler() >>> storage = LazyMemmapStorage(S) >>> rb = TensorDictReplayBuffer(storage=storage, sampler=sampler) >>> >>> for _ in tqdm.tqdm(range(100)): ... td = TensorDict({"obs": torch.randn(100, 3, 4), "next": {"obs": torch.randn(100, 3, 4)}, "td_error": torch.rand(100)}, [100]) ... rb.extend(td) ... sample = rb.sample(32) ... rb.update_tensordict_priority(sample) >>> # save and load the buffer >>> with tempfile.TemporaryDirectory() as tmpdir: ... rb.dumps(tmpdir) ... ... sampler = PrioritizedSampler(S, 1.1, 1.0) ... # sampler = RandomSampler() ... storage = LazyMemmapStorage(S) ... rb_load = TensorDictReplayBuffer(storage=storage, sampler=sampler) ... rb_load.loads(tmpdir) ... assert len(rb) == len(rb_load)
- empty()¶
清空經驗回放緩衝區並將遊標重置為 0。
- extend(tensordicts: TensorDictBase) Tensor¶
用可迭代物件中包含的一個或多個元素擴充套件經驗回放緩衝區。
如果存在，將呼叫反向轉換。`
- 引數：
data (iterable) – 要新增到經驗回放緩衝區的資料集合。
- 返回：
新增到經驗回放緩衝區的資料的索引。
警告
extend()在處理值列表時可能具有模糊的簽名，這些列表應被解釋為 PyTree（在這種情況下，列表中的所有元素將被放入儲存的 PyTree 中的一個切片中）或要逐個新增的值列表。為了解決這個問題，TorchRL 在列表和元組之間做了明確區分：元組將被視為 PyTree，列表（在根級別）將被解釋為要逐個新增到緩衝區的值棧。對於ListStorage例項，只能提供未繫結的元素（非 PyTrees）。
- insert_transform(index: int, transform: Transform, *, invert: bool = False) ReplayBuffer¶
插入轉換。
呼叫 sample 時，轉換按順序執行。
- 引數：
index (int) – 插入轉換的位置。
transform (Transform) – 要追加的轉換
- 關鍵字引數：
invert (bool, optional) – 如果為
True，則轉換將被反轉（在寫入時呼叫正向方法，在讀取時呼叫反向方法）。預設為False。
- load(*args, **kwargs)¶
loads() 的別名。
- loads(path)¶
在給定路徑載入經驗回放緩衝區狀態。
緩衝區應具有匹配的元件，並且應使用
dumps()儲存。- 引數：
path (Path 或 str) – 經驗回放緩衝區儲存的路徑。
更多資訊請參見
dumps()。
- preprocess(fn: Callable[[TensorDictBase], TensorDictBase], dim: int = 0, num_workers: int | None = None, *, chunksize: int | None = None, num_chunks: int | None = None, pool: mp.Pool | None = None, generator: torch.Generator | None = None, max_tasks_per_child: int | None = None, worker_threads: int = 1, index_with_generator: bool = False, pbar: bool = False, mp_start_method: str | None = None, num_frames: int | None = None, dest: str | Path) TensorStorage¶
預處理資料集並返回包含格式化資料的新儲存。
資料轉換必須是單一的（作用於資料集的單個樣本）。
Args 和 Keyword Args 將轉發給
map()。資料集隨後可以使用
delete()刪除。- 關鍵字引數：
dest (路徑 或 等效值) – 新資料集的位置路徑。
num_frames (int, optional) – 如果提供，將僅轉換前 num_frames 個幀。這對於初次除錯轉換非常有用。
返回：可在
ReplayBuffer例項中使用的新儲存。示例
>>> from torchrl.data.datasets import MinariExperienceReplay >>> >>> data = MinariExperienceReplay( ... list(MinariExperienceReplay.available_datasets)[0], ... batch_size=32 ... ) >>> print(data) MinariExperienceReplay( storages=TensorStorage(TensorDict( fields={ action: MemoryMappedTensor(shape=torch.Size([1000000, 8]), device=cpu, dtype=torch.float32, is_shared=True), episode: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.int64, is_shared=True), info: TensorDict( fields={ distance_from_origin: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), forward_reward: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), goal: MemoryMappedTensor(shape=torch.Size([1000000, 2]), device=cpu, dtype=torch.float64, is_shared=True), qpos: MemoryMappedTensor(shape=torch.Size([1000000, 15]), device=cpu, dtype=torch.float64, is_shared=True), qvel: MemoryMappedTensor(shape=torch.Size([1000000, 14]), device=cpu, dtype=torch.float64, is_shared=True), reward_ctrl: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), reward_forward: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), reward_survive: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), success: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.bool, is_shared=True), x_position: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), x_velocity: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), y_position: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), y_velocity: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True)}, batch_size=torch.Size([1000000]), device=cpu, is_shared=False), next: TensorDict( fields={ done: MemoryMappedTensor(shape=torch.Size([1000000, 1]), device=cpu, dtype=torch.bool, is_shared=True), info: TensorDict( fields={ distance_from_origin: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), forward_reward: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), goal: MemoryMappedTensor(shape=torch.Size([1000000, 2]), device=cpu, dtype=torch.float64, is_shared=True), qpos: MemoryMappedTensor(shape=torch.Size([1000000, 15]), device=cpu, dtype=torch.float64, is_shared=True), qvel: MemoryMappedTensor(shape=torch.Size([1000000, 14]), device=cpu, dtype=torch.float64, is_shared=True), reward_ctrl: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), reward_forward: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), reward_survive: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), success: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.bool, is_shared=True), x_position: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), x_velocity: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), y_position: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True), y_velocity: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.float64, is_shared=True)}, batch_size=torch.Size([1000000]), device=cpu, is_shared=False), observation: TensorDict( fields={ achieved_goal: MemoryMappedTensor(shape=torch.Size([1000000, 2]), device=cpu, dtype=torch.float64, is_shared=True), desired_goal: MemoryMappedTensor(shape=torch.Size([1000000, 2]), device=cpu, dtype=torch.float64, is_shared=True), observation: MemoryMappedTensor(shape=torch.Size([1000000, 27]), device=cpu, dtype=torch.float64, is_shared=True)}, batch_size=torch.Size([1000000]), device=cpu, is_shared=False), reward: MemoryMappedTensor(shape=torch.Size([1000000, 1]), device=cpu, dtype=torch.float64, is_shared=True), terminated: MemoryMappedTensor(shape=torch.Size([1000000, 1]), device=cpu, dtype=torch.bool, is_shared=True), truncated: MemoryMappedTensor(shape=torch.Size([1000000, 1]), device=cpu, dtype=torch.bool, is_shared=True)}, batch_size=torch.Size([1000000]), device=cpu, is_shared=False), observation: TensorDict( fields={ achieved_goal: MemoryMappedTensor(shape=torch.Size([1000000, 2]), device=cpu, dtype=torch.float64, is_shared=True), desired_goal: MemoryMappedTensor(shape=torch.Size([1000000, 2]), device=cpu, dtype=torch.float64, is_shared=True), observation: MemoryMappedTensor(shape=torch.Size([1000000, 27]), device=cpu, dtype=torch.float64, is_shared=True)}, batch_size=torch.Size([1000000]), device=cpu, is_shared=False)}, batch_size=torch.Size([1000000]), device=cpu, is_shared=False)), samplers=RandomSampler, writers=ImmutableDatasetWriter(), batch_size=32, transform=Compose( ), collate_fn=<function _collate_id at 0x120e21dc0>) >>> from torchrl.envs import CatTensors, Compose >>> from tempfile import TemporaryDirectory >>> >>> cat_tensors = CatTensors( ... in_keys=[("observation", "observation"), ("observation", "achieved_goal"), ... ("observation", "desired_goal")], ... out_key="obs" ... ) >>> cat_next_tensors = CatTensors( ... in_keys=[("next", "observation", "observation"), ... ("next", "observation", "achieved_goal"), ... ("next", "observation", "desired_goal")], ... out_key=("next", "obs") ... ) >>> t = Compose(cat_tensors, cat_next_tensors) >>> >>> def func(td): ... td = td.select( ... "action", ... "episode", ... ("next", "done"), ... ("next", "observation"), ... ("next", "reward"), ... ("next", "terminated"), ... ("next", "truncated"), ... "observation" ... ) ... td = t(td) ... return td >>> with TemporaryDirectory() as tmpdir: ... new_storage = data.preprocess(func, num_workers=4, pbar=True, mp_start_method="fork", dest=tmpdir) ... rb = ReplayBuffer(storage=new_storage) ... print(rb) ReplayBuffer( storage=TensorStorage( data=TensorDict( fields={ action: MemoryMappedTensor(shape=torch.Size([1000000, 8]), device=cpu, dtype=torch.float32, is_shared=True), episode: MemoryMappedTensor(shape=torch.Size([1000000]), device=cpu, dtype=torch.int64, is_shared=True), next: TensorDict( fields={ done: MemoryMappedTensor(shape=torch.Size([1000000, 1]), device=cpu, dtype=torch.bool, is_shared=True), obs: MemoryMappedTensor(shape=torch.Size([1000000, 31]), device=cpu, dtype=torch.float64, is_shared=True), observation: TensorDict( fields={ }, batch_size=torch.Size([1000000]), device=cpu, is_shared=False), reward: MemoryMappedTensor(shape=torch.Size([1000000, 1]), device=cpu, dtype=torch.float64, is_shared=True), terminated: MemoryMappedTensor(shape=torch.Size([1000000, 1]), device=cpu, dtype=torch.bool, is_shared=True), truncated: MemoryMappedTensor(shape=torch.Size([1000000, 1]), device=cpu, dtype=torch.bool, is_shared=True)}, batch_size=torch.Size([1000000]), device=cpu, is_shared=False), obs: MemoryMappedTensor(shape=torch.Size([1000000, 31]), device=cpu, dtype=torch.float64, is_shared=True), observation: TensorDict( fields={ }, batch_size=torch.Size([1000000]), device=cpu, is_shared=False)}, batch_size=torch.Size([1000000]), device=cpu, is_shared=False), shape=torch.Size([1000000]), len=1000000, max_size=1000000), sampler=RandomSampler(), writer=RoundRobinWriter(cursor=0, full_storage=True), batch_size=None, collate_fn=<function _collate_id at 0x168406fc0>)
- register_load_hook(hook: Callable[[Any], Any])¶
為儲存註冊一個載入鉤子。
注意
當前，儲存經驗回放緩衝區時不會序列化鉤子：每次建立緩衝區時必須手動重新初始化它們。
- register_save_hook(hook: Callable[[Any], Any])¶
為儲存註冊一個儲存鉤子。
注意
當前，儲存經驗回放緩衝區時不會序列化鉤子：每次建立緩衝區時必須手動重新初始化它們。
- sample(batch_size: int | None = None, return_info: bool = False, include_info: bool = None) TensorDictBase¶
從經驗回放緩衝區取樣一批資料。
使用 Sampler 取樣索引，並從 Storage 中檢索它們。
- 引數：
batch_size (int, optional) – 要收集的資料大小。如果未提供，此方法將按照取樣器指示的批次大小進行取樣。
return_info (bool) – 是否返回資訊。如果為 True，結果將是一個元組 (data, info)。如果為 False，結果是資料。
- 返回：
一個 tensordict，包含在經驗回放緩衝區中選擇的一批資料。如果設定了 return_info 標誌為 True，則返回一個包含此 tensordict 和 info 的元組。
- save(*args, **kwargs)¶
dumps() 的別名。
- set_storage(storage: Storage, collate_fn: Callable | None = None)¶
在經驗回放緩衝區中設定新的儲存並返回之前的儲存。
- 引數：
storage (Storage) – 緩衝區的新儲存物件。
collate_fn (callable, optional) – 如果提供，則將 collate_fn 設定為此值。否則，將重置為預設值。
- property write_count¶
迄今為止透過 add 和 extend 方法寫入緩衝區的專案總數。
