目錄
  • 教程 >
  • 透過示例學習 PyTorch
快捷方式
beginner/pytorch_with_examples

透過示例學習 PyTorch

創建於: Mar 24, 2017 | 最後更新於: Jan 21, 2025 | 最後驗證於: Nov 05, 2024

作者: Justin Johnson

注意

這是我們較舊的 PyTorch 教程之一。你可以在 學習基礎知識 中檢視我們最新的入門內容。

本教程透過自包含的示例介紹了 PyTorch 的基本概念。

PyTorch 的核心功能主要有兩個

  • 一個 n 維張量 (Tensor)，類似於 numpy 陣列，但可以在 GPU 上執行

  • 用於構建和訓練神經網路的自動微分

我們將使用一個用三階多項式擬合 \(y=\sin(x)\) 的問題作為貫穿本教程的示例。網路將有四個引數，並透過梯度下降法進行訓練，透過最小化網路輸出和真實輸出之間的歐幾里得距離來擬合隨機資料。

注意

你可以在本頁的 頁面末尾 處瀏覽單獨的示例。

張量

熱身：numpy

在介紹 PyTorch 之前，我們首先使用 numpy 來實現這個網路。

Numpy 提供了一個 n 維陣列物件，以及許多用於操作這些陣列的函式。Numpy 是一個用於科學計算的通用框架；它不瞭解計算圖、深度學習或梯度。然而，我們可以透過手動實現網路的前向和後向傳播，並使用 numpy 操作，輕鬆地使用 numpy 將三階多項式擬合到正弦函式

# -*- coding: utf-8 -*-
import numpy as np
import math

# Create random input and output data
x = np.linspace(-math.pi, math.pi, 2000)
y = np.sin(x)

# Randomly initialize weights
a = np.random.randn()
b = np.random.randn()
c = np.random.randn()
d = np.random.randn()

learning_rate = 1e-6
for t in range(2000):
    # Forward pass: compute predicted y
    # y = a + b x + c x^2 + d x^3
    y_pred = a + b * x + c * x ** 2 + d * x ** 3

    # Compute and print loss
    loss = np.square(y_pred - y).sum()
    if t % 100 == 99:
        print(t, loss)

    # Backprop to compute gradients of a, b, c, d with respect to loss
    grad_y_pred = 2.0 * (y_pred - y)
    grad_a = grad_y_pred.sum()
    grad_b = (grad_y_pred * x).sum()
    grad_c = (grad_y_pred * x ** 2).sum()
    grad_d = (grad_y_pred * x ** 3).sum()

    # Update weights
    a -= learning_rate * grad_a
    b -= learning_rate * grad_b
    c -= learning_rate * grad_c
    d -= learning_rate * grad_d

print(f'Result: y = {a} + {b} x + {c} x^2 + {d} x^3')

PyTorch：張量

Numpy 是一個很棒的框架，但它無法利用 GPU 來加速其數值計算。對於現代深度神經網路，GPU 通常能提供 50 倍或更高 的加速，因此遺憾的是 numpy 不足以滿足現代深度學習的需求。

在這裡，我們介紹 PyTorch 最基本的一個概念：**張量** (Tensor)。一個 PyTorch 張量在概念上與 numpy 陣列相同：張量是一個 n 維陣列，PyTorch 提供了許多用於操作這些張量的函式。在底層，張量可以跟蹤計算圖和梯度，但它們也是用於科學計算的通用工具。

此外，與 numpy 不同，PyTorch 張量可以利用 GPU 來加速其數值計算。要在 GPU 上執行 PyTorch 張量，你只需指定正確的裝置。

在這裡，我們使用 PyTorch 張量來擬合三階多項式到正弦函式。就像上面的 numpy 示例一樣，我們需要手動實現網路的前向和後向傳播。

# -*- coding: utf-8 -*-

import torch
import math


dtype = torch.float
device = torch.device("cpu")
# device = torch.device("cuda:0") # Uncomment this to run on GPU

# Create random input and output data
x = torch.linspace(-math.pi, math.pi, 2000, device=device, dtype=dtype)
y = torch.sin(x)

# Randomly initialize weights
a = torch.randn((), device=device, dtype=dtype)
b = torch.randn((), device=device, dtype=dtype)
c = torch.randn((), device=device, dtype=dtype)
d = torch.randn((), device=device, dtype=dtype)

learning_rate = 1e-6
for t in range(2000):
    # Forward pass: compute predicted y
    y_pred = a + b * x + c * x ** 2 + d * x ** 3

    # Compute and print loss
    loss = (y_pred - y).pow(2).sum().item()
    if t % 100 == 99:
        print(t, loss)

    # Backprop to compute gradients of a, b, c, d with respect to loss
    grad_y_pred = 2.0 * (y_pred - y)
    grad_a = grad_y_pred.sum()
    grad_b = (grad_y_pred * x).sum()
    grad_c = (grad_y_pred * x ** 2).sum()
    grad_d = (grad_y_pred * x ** 3).sum()

    # Update weights using gradient descent
    a -= learning_rate * grad_a
    b -= learning_rate * grad_b
    c -= learning_rate * grad_c
    d -= learning_rate * grad_d


print(f'Result: y = {a.item()} + {b.item()} x + {c.item()} x^2 + {d.item()} x^3')

Autograd

PyTorch：張量和 Autograd

在上面的示例中，我們必須手動實現神經網路的前向和後向傳播。對於一個小的兩層網路來說，手動實現後向傳播並不是什麼大問題，但對於大型複雜網路來說，這很快就會變得非常麻煩。

值得慶幸的是，我們可以使用 自動微分 來自動化神經網路中後向傳播的計算。PyTorch 中的 **autograd** 包正是提供了這個功能。使用 autograd 時，網路的前向傳播將定義一個 **計算圖**；圖中的節點是張量，邊是根據輸入張量生成輸出張量的函式。然後，透過這個圖進行反向傳播，你可以輕鬆地計算梯度。

這聽起來很複雜，但在實踐中使用起來相當簡單。每個張量都代表計算圖中的一個節點。如果 x 是一個張量且 x.requires_grad=True，那麼 x.grad 是另一個張量，其中包含 x 關於某個標量值的梯度。

在這裡，我們使用 PyTorch 張量和 autograd 來實現我們用三階多項式擬合正弦波的示例；現在我們不再需要手動實現網路的後向傳播了。

# -*- coding: utf-8 -*-
import torch
import math

# We want to be able to train our model on an `accelerator <https://pytorch.com.tw/docs/stable/torch.html#accelerators>`__
# such as CUDA, MPS, MTIA, or XPU. If the current accelerator is available, we will use it. Otherwise, we use the CPU.

dtype = torch.float
device = torch.accelerator.current_accelerator().type if torch.accelerator.is_available() else "cpu"
print(f"Using {device} device")
torch.set_default_device(device)

# Create Tensors to hold input and outputs.
# By default, requires_grad=False, which indicates that we do not need to
# compute gradients with respect to these Tensors during the backward pass.
x = torch.linspace(-math.pi, math.pi, 2000, dtype=dtype)
y = torch.sin(x)

# Create random Tensors for weights. For a third order polynomial, we need
# 4 weights: y = a + b x + c x^2 + d x^3
# Setting requires_grad=True indicates that we want to compute gradients with
# respect to these Tensors during the backward pass.
a = torch.randn((), dtype=dtype, requires_grad=True)
b = torch.randn((), dtype=dtype, requires_grad=True)
c = torch.randn((), dtype=dtype, requires_grad=True)
d = torch.randn((), dtype=dtype, requires_grad=True)

learning_rate = 1e-6
for t in range(2000):
    # Forward pass: compute predicted y using operations on Tensors.
    y_pred = a + b * x + c * x ** 2 + d * x ** 3

    # Compute and print loss using operations on Tensors.
    # Now loss is a Tensor of shape (1,)
    # loss.item() gets the scalar value held in the loss.
    loss = (y_pred - y).pow(2).sum()
    if t % 100 == 99:
        print(t, loss.item())

    # Use autograd to compute the backward pass. This call will compute the
    # gradient of loss with respect to all Tensors with requires_grad=True.
    # After this call a.grad, b.grad. c.grad and d.grad will be Tensors holding
    # the gradient of the loss with respect to a, b, c, d respectively.
    loss.backward()

    # Manually update weights using gradient descent. Wrap in torch.no_grad()
    # because weights have requires_grad=True, but we don't need to track this
    # in autograd.
    with torch.no_grad():
        a -= learning_rate * a.grad
        b -= learning_rate * b.grad
        c -= learning_rate * c.grad
        d -= learning_rate * d.grad

        # Manually zero the gradients after updating weights
        a.grad = None
        b.grad = None
        c.grad = None
        d.grad = None

print(f'Result: y = {a.item()} + {b.item()} x + {c.item()} x^2 + {d.item()} x^3')

PyTorch：定義新的 autograd 函式

在底層，每個原始的 autograd 運算元實際上是作用於張量的兩個函式。**forward** 函式根據輸入張量計算輸出張量。**backward** 函式接收輸出張量關於某個標量值的梯度，並計算輸入張量關於同一標量值的梯度。

在 PyTorch 中，我們可以透過定義一個繼承自 torch.autograd.Function 的子類，並實現 forward 和 backward 函式來輕鬆定義我們自己的 autograd 運算元。然後，我們可以透過構造一個例項並像呼叫函式一樣呼叫它來使用我們新的 autograd 運算元，傳入包含輸入資料的張量。

在此示例中，我們將模型定義為 \(y=a+b P_3(c+dx)\) 而不是 \(y=a+bx+cx^2+dx^3\)，其中 \(P_3(x)=\frac{1}{2}\left(5x^3-3x\right)\) 是三階 勒讓德多項式。我們編寫了自己的自定義 autograd 函式來計算 \(P_3\) 的前向和後向傳播，並使用它來實現我們的模型。

# -*- coding: utf-8 -*-
import torch
import math


class LegendrePolynomial3(torch.autograd.Function):
    """
    We can implement our own custom autograd Functions by subclassing
    torch.autograd.Function and implementing the forward and backward passes
    which operate on Tensors.
    """

    @staticmethod
    def forward(ctx, input):
        """
        In the forward pass we receive a Tensor containing the input and return
        a Tensor containing the output. ctx is a context object that can be used
        to stash information for backward computation. You can cache arbitrary
        objects for use in the backward pass using the ctx.save_for_backward method.
        """
        ctx.save_for_backward(input)
        return 0.5 * (5 * input ** 3 - 3 * input)

    @staticmethod
    def backward(ctx, grad_output):
        """
        In the backward pass we receive a Tensor containing the gradient of the loss
        with respect to the output, and we need to compute the gradient of the loss
        with respect to the input.
        """
        input, = ctx.saved_tensors
        return grad_output * 1.5 * (5 * input ** 2 - 1)


dtype = torch.float
device = torch.device("cpu")
# device = torch.device("cuda:0")  # Uncomment this to run on GPU

# Create Tensors to hold input and outputs.
# By default, requires_grad=False, which indicates that we do not need to
# compute gradients with respect to these Tensors during the backward pass.
x = torch.linspace(-math.pi, math.pi, 2000, device=device, dtype=dtype)
y = torch.sin(x)

# Create random Tensors for weights. For this example, we need
# 4 weights: y = a + b * P3(c + d * x), these weights need to be initialized
# not too far from the correct result to ensure convergence.
# Setting requires_grad=True indicates that we want to compute gradients with
# respect to these Tensors during the backward pass.
a = torch.full((), 0.0, device=device, dtype=dtype, requires_grad=True)
b = torch.full((), -1.0, device=device, dtype=dtype, requires_grad=True)
c = torch.full((), 0.0, device=device, dtype=dtype, requires_grad=True)
d = torch.full((), 0.3, device=device, dtype=dtype, requires_grad=True)

learning_rate = 5e-6
for t in range(2000):
    # To apply our Function, we use Function.apply method. We alias this as 'P3'.
    P3 = LegendrePolynomial3.apply

    # Forward pass: compute predicted y using operations; we compute
    # P3 using our custom autograd operation.
    y_pred = a + b * P3(c + d * x)

    # Compute and print loss
    loss = (y_pred - y).pow(2).sum()
    if t % 100 == 99:
        print(t, loss.item())

    # Use autograd to compute the backward pass.
    loss.backward()

    # Update weights using gradient descent
    with torch.no_grad():
        a -= learning_rate * a.grad
        b -= learning_rate * b.grad
        c -= learning_rate * c.grad
        d -= learning_rate * d.grad

        # Manually zero the gradients after updating weights
        a.grad = None
        b.grad = None
        c.grad = None
        d.grad = None

print(f'Result: y = {a.item()} + {b.item()} * P3({c.item()} + {d.item()} x)')

nn 模組

PyTorch：nn

計算圖和 autograd 是一個非常強大的正規化，用於定義複雜的運算元和自動求導；然而，對於大型神經網路來說，原始的 autograd 可能有點過於底層。

在構建神經網路時，我們經常考慮將計算組織成**層**，其中一些層具有在學習過程中將被最佳化的**可學習引數**。

在 TensorFlow 中，像 Keras、TensorFlow-Slim 和 TFLearn 這樣的包提供了對原始計算圖的更高階抽象，這對於構建神經網路很有用。

在 PyTorch 中，nn 包用於達到同樣的目的。nn 包定義了一系列**模組** (Modules)，它們大致相當於神經網路的層。一個模組接收輸入張量並計算輸出張量，但也可能持有內部狀態，例如包含可學習引數的張量。nn 包還定義了一系列在訓練神經網路時常用的有用損失函式。

在此示例中，我們使用 nn 包來實現我們的多項式模型網路。

# -*- coding: utf-8 -*-
import torch
import math


# Create Tensors to hold input and outputs.
x = torch.linspace(-math.pi, math.pi, 2000)
y = torch.sin(x)

# For this example, the output y is a linear function of (x, x^2, x^3), so
# we can consider it as a linear layer neural network. Let's prepare the
# tensor (x, x^2, x^3).
p = torch.tensor([1, 2, 3])
xx = x.unsqueeze(-1).pow(p)

# In the above code, x.unsqueeze(-1) has shape (2000, 1), and p has shape
# (3,), for this case, broadcasting semantics will apply to obtain a tensor
# of shape (2000, 3) 

# Use the nn package to define our model as a sequence of layers. nn.Sequential
# is a Module which contains other Modules, and applies them in sequence to
# produce its output. The Linear Module computes output from input using a
# linear function, and holds internal Tensors for its weight and bias.
# The Flatten layer flatens the output of the linear layer to a 1D tensor,
# to match the shape of `y`.
model = torch.nn.Sequential(
    torch.nn.Linear(3, 1),
    torch.nn.Flatten(0, 1)
)

# The nn package also contains definitions of popular loss functions; in this
# case we will use Mean Squared Error (MSE) as our loss function.
loss_fn = torch.nn.MSELoss(reduction='sum')

learning_rate = 1e-6
for t in range(2000):

    # Forward pass: compute predicted y by passing x to the model. Module objects
    # override the __call__ operator so you can call them like functions. When
    # doing so you pass a Tensor of input data to the Module and it produces
    # a Tensor of output data.
    y_pred = model(xx)

    # Compute and print loss. We pass Tensors containing the predicted and true
    # values of y, and the loss function returns a Tensor containing the
    # loss.
    loss = loss_fn(y_pred, y)
    if t % 100 == 99:
        print(t, loss.item())

    # Zero the gradients before running the backward pass.
    model.zero_grad()

    # Backward pass: compute gradient of the loss with respect to all the learnable
    # parameters of the model. Internally, the parameters of each Module are stored
    # in Tensors with requires_grad=True, so this call will compute gradients for
    # all learnable parameters in the model.
    loss.backward()

    # Update the weights using gradient descent. Each parameter is a Tensor, so
    # we can access its gradients like we did before.
    with torch.no_grad():
        for param in model.parameters():
            param -= learning_rate * param.grad

# You can access the first layer of `model` like accessing the first item of a list
linear_layer = model[0]

# For linear layer, its parameters are stored as `weight` and `bias`.
print(f'Result: y = {linear_layer.bias.item()} + {linear_layer.weight[:, 0].item()} x + {linear_layer.weight[:, 1].item()} x^2 + {linear_layer.weight[:, 2].item()} x^3')

PyTorch：optim

到目前為止，我們透過使用 torch.no_grad() 手動改變持有可學習引數的張量來更新模型的權重。對於隨機梯度下降等簡單的最佳化演算法來說，這不是一個很大的負擔，但在實踐中，我們經常使用更復雜的最佳化器來訓練神經網路，例如 AdaGrad、RMSProp、Adam 等。

PyTorch 中的 optim 包抽象了最佳化演算法的概念，並提供了常用最佳化演算法的實現。

在此示例中，我們將像之前一樣使用 nn 包定義模型，但我們將使用 optim 包提供的 RMSprop 演算法來最佳化模型。

# -*- coding: utf-8 -*-
import torch
import math


# Create Tensors to hold input and outputs.
x = torch.linspace(-math.pi, math.pi, 2000)
y = torch.sin(x)

# Prepare the input tensor (x, x^2, x^3).
p = torch.tensor([1, 2, 3])
xx = x.unsqueeze(-1).pow(p)

# Use the nn package to define our model and loss function.
model = torch.nn.Sequential(
    torch.nn.Linear(3, 1),
    torch.nn.Flatten(0, 1)
)
loss_fn = torch.nn.MSELoss(reduction='sum')

# Use the optim package to define an Optimizer that will update the weights of
# the model for us. Here we will use RMSprop; the optim package contains many other
# optimization algorithms. The first argument to the RMSprop constructor tells the
# optimizer which Tensors it should update.
learning_rate = 1e-3
optimizer = torch.optim.RMSprop(model.parameters(), lr=learning_rate)
for t in range(2000):
    # Forward pass: compute predicted y by passing x to the model.
    y_pred = model(xx)

    # Compute and print loss.
    loss = loss_fn(y_pred, y)
    if t % 100 == 99:
        print(t, loss.item())

    # Before the backward pass, use the optimizer object to zero all of the
    # gradients for the variables it will update (which are the learnable
    # weights of the model). This is because by default, gradients are
    # accumulated in buffers( i.e, not overwritten) whenever .backward()
    # is called. Checkout docs of torch.autograd.backward for more details.
    optimizer.zero_grad()

    # Backward pass: compute gradient of the loss with respect to model
    # parameters
    loss.backward()

    # Calling the step function on an Optimizer makes an update to its
    # parameters
    optimizer.step()


linear_layer = model[0]
print(f'Result: y = {linear_layer.bias.item()} + {linear_layer.weight[:, 0].item()} x + {linear_layer.weight[:, 1].item()} x^2 + {linear_layer.weight[:, 2].item()} x^3')

PyTorch：自定義 nn 模組

有時你會想指定比現有模組序列更復雜的模型；對於這些情況，你可以透過繼承 nn.Module 並定義一個 forward 方法來定義自己的模組，該方法接收輸入張量，並使用其他模組或張量上的其他 autograd 操作來生成輸出張量。

在此示例中，我們將三階多項式實現為一個自定義 Module 子類。

# -*- coding: utf-8 -*-
import torch
import math


class Polynomial3(torch.nn.Module):
    def __init__(self):
        """
        In the constructor we instantiate four parameters and assign them as
        member parameters.
        """
        super().__init__()
        self.a = torch.nn.Parameter(torch.randn(()))
        self.b = torch.nn.Parameter(torch.randn(()))
        self.c = torch.nn.Parameter(torch.randn(()))
        self.d = torch.nn.Parameter(torch.randn(()))

    def forward(self, x):
        """
        In the forward function we accept a Tensor of input data and we must return
        a Tensor of output data. We can use Modules defined in the constructor as
        well as arbitrary operators on Tensors.
        """
        return self.a + self.b * x + self.c * x ** 2 + self.d * x ** 3

    def string(self):
        """
        Just like any class in Python, you can also define custom method on PyTorch modules
        """
        return f'y = {self.a.item()} + {self.b.item()} x + {self.c.item()} x^2 + {self.d.item()} x^3'


# Create Tensors to hold input and outputs.
x = torch.linspace(-math.pi, math.pi, 2000)
y = torch.sin(x)

# Construct our model by instantiating the class defined above
model = Polynomial3()

# Construct our loss function and an Optimizer. The call to model.parameters()
# in the SGD constructor will contain the learnable parameters (defined 
# with torch.nn.Parameter) which are members of the model.
criterion = torch.nn.MSELoss(reduction='sum')
optimizer = torch.optim.SGD(model.parameters(), lr=1e-6)
for t in range(2000):
    # Forward pass: Compute predicted y by passing x to the model
    y_pred = model(x)

    # Compute and print loss
    loss = criterion(y_pred, y)
    if t % 100 == 99:
        print(t, loss.item())

    # Zero gradients, perform a backward pass, and update the weights.
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

print(f'Result: {model.string()}')

PyTorch：控制流 + 權重共享

作為動態圖和權重共享的一個示例，我們實現了一個非常奇特的模型：一個三到五階多項式，它在每次前向傳播時隨機選擇 3 到 5 之間的一個數作為階數，並多次重用相同的權重來計算四階和五階項。

對於這個模型，我們可以使用正常的 Python 控制流來實現迴圈，並且可以透過在定義前向傳播時簡單地多次重用相同的引數來實現權重共享。

我們可以輕鬆地將這個模型實現為一個 Module 子類。

# -*- coding: utf-8 -*-
import random
import torch
import math


class DynamicNet(torch.nn.Module):
    def __init__(self):
        """
        In the constructor we instantiate five parameters and assign them as members.
        """
        super().__init__()
        self.a = torch.nn.Parameter(torch.randn(()))
        self.b = torch.nn.Parameter(torch.randn(()))
        self.c = torch.nn.Parameter(torch.randn(()))
        self.d = torch.nn.Parameter(torch.randn(()))
        self.e = torch.nn.Parameter(torch.randn(()))

    def forward(self, x):
        """
        For the forward pass of the model, we randomly choose either 4, 5
        and reuse the e parameter to compute the contribution of these orders.

        Since each forward pass builds a dynamic computation graph, we can use normal
        Python control-flow operators like loops or conditional statements when
        defining the forward pass of the model.

        Here we also see that it is perfectly safe to reuse the same parameter many
        times when defining a computational graph.
        """
        y = self.a + self.b * x + self.c * x ** 2 + self.d * x ** 3
        for exp in range(4, random.randint(4, 6)):
            y = y + self.e * x ** exp
        return y

    def string(self):
        """
        Just like any class in Python, you can also define custom method on PyTorch modules
        """
        return f'y = {self.a.item()} + {self.b.item()} x + {self.c.item()} x^2 + {self.d.item()} x^3 + {self.e.item()} x^4 ? + {self.e.item()} x^5 ?'


# Create Tensors to hold input and outputs.
x = torch.linspace(-math.pi, math.pi, 2000)
y = torch.sin(x)

# Construct our model by instantiating the class defined above
model = DynamicNet()

# Construct our loss function and an Optimizer. Training this strange model with
# vanilla stochastic gradient descent is tough, so we use momentum
criterion = torch.nn.MSELoss(reduction='sum')
optimizer = torch.optim.SGD(model.parameters(), lr=1e-8, momentum=0.9)
for t in range(30000):
    # Forward pass: Compute predicted y by passing x to the model
    y_pred = model(x)

    # Compute and print loss
    loss = criterion(y_pred, y)
    if t % 2000 == 1999:
        print(t, loss.item())

    # Zero gradients, perform a backward pass, and update the weights.
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

print(f'Result: {model.string()}')

示例

你可以在此處瀏覽上述示例。

文件

訪問 PyTorch 的全面開發者文件

檢視文件

教程

獲取面向初學者和高階開發者的深度教程

檢視教程

資源

查詢開發資源並獲得問題解答

檢視資源

為了分析流量和最佳化你的體驗，我們在本網站上使用 Cookie。點選或導航即表示你同意我們使用 Cookie。作為本網站的當前維護者，Facebook 的 Cookie 政策適用。瞭解更多資訊，包括可用控制：Cookie 政策。