---

Úvod do strojového učení v DPZ¶

PyTorch - framework pro hluboké učení¶

Obsah: * instalace PyTorch

• tensory (vícedimensionální pole)
• konstruktory
• datové typy
• matematické oprace
• PyTorch proměnná typu Variables (s gradienty)
• NumPy <-> PyTorch
• tensory -> GPU -> CPU

Instalace¶

pip install torch

pip install torchvision

torchvision

• příprava dat * datasety * pomocné funkce Dataset a DataLoader pro trnasformaci data "loading" datasetů pro učení modelu.

PyTorch tensor¶

Tensory jsou základní datové struktury pro reprezentaci data v PyTorch. tensor .. vícedimensionální pole. V kontextu hlubokého uční tensory jsou vektroy, matice arbitrárního počtu dimenzí. PyTorch zahrnuje nejen definici tensorů, ale i mnoho definovaných matematických funkcí utilit.

PyTorch tensor vs. NumPy pole¶

• jsou podobné
• PyTorch umí využít GPU k akceleraci výpočtů
• v PyTorch je implementován autograd pro automatické derivace funkcí * PyTorch má velké množství základních bloků pro definici širokého množství architektur ANN.

PyTorch tensor API¶

Dokumentce: (https://pytorch.org/docs/stable/index.html).

Tensors: Multidimensional arrays¶

In [67]:

import torch 
import numpy as np

import torch import numpy as np

In [68]:

torch.__version__

torch.__version__

Out[68]:

'2.1.1'

PyTorch Tensors constructors¶

Comparing NumPy and PyTorch

In [3]:

a = np.ones(3)
print(a)

a = np.ones(3) print(a)

[1. 1. 1.]

Constructors from other containers¶

In [4]:

# passing Python list to te constructor, the same effect! 
points = torch.tensor([4.0, 1.0, 5.0, 3.0, 2.0, 1.0])
points

# passing Python list to te constructor, the same effect! points = torch.tensor([4.0, 1.0, 5.0, 3.0, 2.0, 1.0]) points

Out[4]:

tensor([4., 1., 5., 3., 2., 1.])

In [5]:

# 2D, (list of lists) passed to the constructor
points = torch.tensor([[4.0, 1.0], [5.0, 3.0], [2.0, 1.0]])
points

# 2D, (list of lists) passed to the constructor points = torch.tensor([[4.0, 1.0], [5.0, 3.0], [2.0, 1.0]]) points

Out[5]:

tensor([[4., 1.],
        [5., 3.],
        [2., 1.]])

In [103]:

# dimensionality 
points.shape

# dimensionality points.shape

Out[103]:

torch.Size([3, 2])

Indexing tensors¶

Tensors uses the same notion. We can use range indexing for each of the tensor's dimensions.

In [104]:

# tensor two indices to access 2D elements
points[0, 1]

# tensor two indices to access 2D elements points[0, 1]

Out[104]:

tensor(1.)

In [105]:

# the first row
points[0]

# the first row points[0]

Out[105]:

tensor([4., 1.])

Tensors storage¶

Values in tensors are allocated in contignuous chunks of memory managed by torch.Storage instances. A storage is a one-dimensional array of numerical data: that is, a contignuous block of memory containing numbers of a given type, such as float (32 bits representing floating-point number). A PyTorch Tensor instance is a view of such a Storage instance that is capable of indexing into that storage using an offset. Multiple tensors can index the same storage even if they index into the data differently.

Each element is a 32-bit (4-byte) float (in the above case). Storing a 1D tensor of 1.000.000 float numbers will require 4.000.000 contignuous bytes plus small overhead for the metadata.

In [106]:

# indexing into the storage 
points = torch.tensor([[4.0, 1.0], [5.0, 3.0], [2.0, 1.0]])

# indexing into the storage points = torch.tensor([[4.0, 1.0], [5.0, 3.0], [2.0, 1.0]])

In [107]:

points

points

Out[107]:

tensor([[4., 1.],
        [5., 3.],
        [2., 1.]])

In [108]:

# use method storage() to see the content
points.storage()

# use method storage() to see the content points.storage()

Out[108]:

 4.0
 1.0
 5.0
 3.0
 2.0
 1.0
[torch.storage.TypedStorage(dtype=torch.float32, device=cpu) of size 6]

In [19]:

# even though the tensor is 3x2, the storage is contignous array of size 6

# even though the tensor is 3x2, the storage is contignous array of size 6

In [24]:

# get second point in the tensor 
points = torch.tensor([[4.0, 1.0], [5.0, 3.0], [2.0, 1.0]])
second_point = points[1]
print(second_point)

# get second point in the tensor points = torch.tensor([[4.0, 1.0], [5.0, 3.0], [2.0, 1.0]]) second_point = points[1] print(second_point)

tensor([5., 3.])

In [6]:

# size
points.size()

# size points.size()

Out[6]:

torch.Size([3, 2])

In [7]:

# it is the same information contained in the shape
points.shape

# it is the same information contained in the shape points.shape

Out[7]:

torch.Size([3, 2])

Specifying the numeric type with dtype¶

The dtypeargument to tensor constructors specifies the numerical data (d) type, similar to NumPy.

• torch.float .. 32-bit floating point number

• torch.double .. 64-bit

• torch.float16 or torch.half .. 16-bit

• torch.int8 .. signed 8-bit integers

• torch.uint8 .. unsigned 8-bit

• torch.int16 or torch.short .. signed 16-bit

• torch.int32 or torch.int .. signed 32-bit

• torch.int64 or torch.long .. signed 64-bit

• torch.bool .. Boolean

Computations happening in neural networks are typically executed with 32-bit floating-point precision (torch.float or torch.float32).

Tensors can be used as indexes in other tensors, PyTorch expects indexing tensors to have 64-bit integer data type (torch.int64).

Predicates on tensors, such as points > 1.0, produce bool tensors (torch.bool).

Managing a tensor's dtype attribute¶

In [13]:

# float
double_points = torch.ones(10, 2, dtype=torch.double)
# integer
short_points = torch.tensor([[1, 2], [3, 4]], dtype=torch.short)

# float double_points = torch.ones(10, 2, dtype=torch.double) # integer short_points = torch.tensor([[1, 2], [3, 4]], dtype=torch.short)

In [14]:

double_points.dtype

double_points.dtype

Out[14]:

torch.float64

In [15]:

short_points.dtype

short_points.dtype

Out[15]:

torch.int16

In [16]:

# casting 
double_points = torch.zeros(10, 2).double()
short_points = torch.ones(10, 2).short()

# casting double_points = torch.zeros(10, 2).double() short_points = torch.ones(10, 2).short()

In [17]:

double_points.dtype

double_points.dtype

Out[17]:

torch.float64

In [18]:

short_points.dtype

short_points.dtype

Out[18]:

torch.int16

In [19]:

# or the more convenient and readble method .to()
double_points = torch.zeros(10, 2).to(torch.double)
short_points = torch.ones(10, 2).to(dtype=torch.short)

# or the more convenient and readble method .to() double_points = torch.zeros(10, 2).to(torch.double) short_points = torch.ones(10, 2).to(dtype=torch.short)

In [20]:

# mixing input data types in operations converts to the 'larger' type 
points_64 = torch.rand(5, dtype=torch.double)  # <1>
points_short = points_64.to(torch.short)
points_64 * points_short  # works from PyTorch 1.3 onwards

# mixing input data types in operations converts to the 'larger' type points_64 = torch.rand(5, dtype=torch.double) # <1> points_short = points_64.to(torch.short) points_64 * points_short # works from PyTorch 1.3 onwards

Out[20]:

tensor([0., 0., 0., 0., 0.], dtype=torch.float64)

Random values¶

In [21]:

# NumPy random 
np.random.rand(2,2)
# Torch random 
torch.rand(2,2)

# NumPy random np.random.rand(2,2) # Torch random torch.rand(2,2)

Out[21]:

tensor([[0.9967, 0.8005],
        [0.7617, 0.7876]])

Math operations¶

In [27]:

# Element wise addition
a = torch.ones(2,2)
b = torch.ones(2)

c = a + b
c

# Element wise addition a = torch.ones(2,2) b = torch.ones(2) c = a + b c

Out[27]:

tensor([[2., 2.],
        [2., 2.]])

In [28]:

c = torch.add(a, b)
c

c = torch.add(a, b) c

Out[28]:

tensor([[2., 2.],
        [2., 2.]])

In [29]:

# In-place addition
print(c)
c.add_(a)

# In-place addition print(c) c.add_(a)

tensor([[2., 2.],
        [2., 2.]])

Out[29]:

tensor([[3., 3.],
        [3., 3.]])

In [30]:

# Multiplication: torch.mul(a, b))
# ...

# Multiplication: torch.mul(a, b)) # ...

In [36]:

# Tensor Mean
a = torch.Tensor([[1, 2, 3, 4, 5, 6, 7, 8, 9], [10, 12, 13, 14, 15, 16, 17, 18, 19]])
print(a.size()) 

print(a.mean(dim=1))

# Tensor Mean a = torch.Tensor([[1, 2, 3, 4, 5, 6, 7, 8, 9], [10, 12, 13, 14, 15, 16, 17, 18, 19]]) print(a.size()) print(a.mean(dim=1))

torch.Size([2, 9])
tensor([ 5.0000, 14.8889])

Vast majority of operations on tensors are available in the torch module and can be called as methods of a tensor object.

Look in to the web documentation(https://pytorch.org/docs/stable/index.html) for:

• Math operations:

  • Pointwise ops:
  • Reduction ops:
  • Comparision ops:
  • Spectral ops:

• Random sampling

• Parallelism

PyTorch Abstraction¶

Tensor: Like array in Numpy, but runs on GPU

Variable: Stores data and gradient; Node in a computational graph;

PyTorch Variables¶

Variables allows us to accumulate gradients!

When using autograd, the forward pass of your network will define a computational graph; nodes in the graph will be Tensors, and edges will be functions that produce output Tensors from input Tensors. PyTorch Tensors can be created as variable objects where a variable represents a node in computational graph.

requires_grad = True

In [37]:

from torch.autograd import Variable

a = Variable(torch.ones(2,2), requires_grad = True)
a

from torch.autograd import Variable a = Variable(torch.ones(2,2), requires_grad = True) a

Out[37]:

tensor([[1., 1.],
        [1., 1.]], requires_grad=True)

In [38]:

# not a variable
torch.ones(2,2)

# not a variable torch.ones(2,2)

Out[38]:

tensor([[1., 1.],
        [1., 1.]])

What is requires_grad?¶

Allows calculation of gradients w.r.t. the variable!

NumPy interoperability¶

PyTorch tensors can be converted to NumPyarrays and vice versa very efficiently. This allows to take advantage of huge swath of functionality in the wider Python ecosystem that has built up around the NumPy array type. The zero-copy interoperabilty with NumPy arrays is due to the storage system working with the Python buffer protocol (https://docs.python.org/3/c-api/buffer.html).

.numpy()

In [43]:

points = torch.ones(3, 4)
points_np = points.numpy()
points_np

points = torch.ones(3, 4) points_np = points.numpy() points_np

Out[43]:

array([[1., 1., 1., 1.],
       [1., 1., 1., 1.],
       [1., 1., 1., 1.]], dtype=float32)

In [44]:

type(points_np)

type(points_np)

Out[44]:

numpy.ndarray

We can use such conversions at basically no cost, as long as the data sits in CPU RAM. However, if the tensor is allocated on the GPU, PyTorch will make a copy of the tensor into a NumPy array allocated on the CPU.

from_numpy()

In [45]:

# to torch Tensor 
points = torch.from_numpy(points_np)

# to torch Tensor points = torch.from_numpy(points_np)

In [47]:

points

points

Out[47]:

tensor([[1., 1., 1., 1.],
        [1., 1., 1., 1.],
        [1., 1., 1., 1.]])

In [48]:

points2 = torch.rand(3,4)

points2 = torch.rand(3,4)

In [49]:

points2

points2

Out[49]:

tensor([[0.3398, 0.5504, 0.2272, 0.4683],
        [0.5183, 0.6231, 0.3598, 0.4764],
        [0.3914, 0.7710, 0.4794, 0.6644]])

In [50]:

points2.numpy()

points2.numpy()

Out[50]:

array([[0.33983666, 0.5503959 , 0.22718483, 0.46827793],
       [0.5183251 , 0.6230857 , 0.35983914, 0.47641146],
       [0.39138806, 0.7709591 , 0.47940183, 0.66443247]], dtype=float32)

Serializing tensors¶

PyTorch uses pickle under the hood to serialize the tensor object, plus dedicated serialization code for the storage.

In [51]:

points

points

Out[51]:

tensor([[1., 1., 1., 1.],
        [1., 1., 1., 1.],
        [1., 1., 1., 1.]])

In [52]:

# save our points tensor to a file
torch.save(points, './ourpoints.t')

# save our points tensor to a file torch.save(points, './ourpoints.t')

In [53]:

# we can pass a file descriptor in lieu of the file name
with open('./ourpoints2.t','wb') as f:
   torch.save(points, f)

# we can pass a file descriptor in lieu of the file name with open('./ourpoints2.t','wb') as f: torch.save(points, f)

In [54]:

points_in = torch.load('./ourpoints.t')

points_in = torch.load('./ourpoints.t')

In [55]:

points_in

points_in

Out[55]:

tensor([[1., 1., 1., 1.],
        [1., 1., 1., 1.],
        [1., 1., 1., 1.]])

In [56]:

with open('./ourpoints.t','rb') as f:
   points = torch.load(f)

with open('./ourpoints.t','rb') as f: points = torch.load(f)

In [57]:

points

points

Out[57]:

tensor([[1., 1., 1., 1.],
        [1., 1., 1., 1.],
        [1., 1., 1., 1.]])

Moving tensors to the GPU¶

Every PyTorch tensor can be transferred to the GPU(s) in order to perform massively parrallel, fast computatios. In adition to dtype, a PyTorch Tensor also has the notion of device, which is where the computer the tensor data is placed.

In [58]:

torch.cuda.is_available()

torch.cuda.is_available()

Out[58]:

False

In [59]:

if torch.cuda.is_available(): 
    points_gpu = torch.tensor([[4.0, 1.0], [5.0, 3.0], [2.0, 1.0]], device='cuda')
else: 
    print('CUDA is not available')

if torch.cuda.is_available(): points_gpu = torch.tensor([[4.0, 1.0], [5.0, 3.0], [2.0, 1.0]], device='cuda') else: print('CUDA is not available')

CUDA is not available

Také metoda to.

In [60]:

if torch.cuda.is_available(): 
    points_gpu = points.to(device='cuda')
else: 
    print('CUDA is not available')

if torch.cuda.is_available(): points_gpu = points.to(device='cuda') else: print('CUDA is not available')

CUDA is not available

In [61]:

# specify the number of the GPU device 
if torch.cuda.is_available(): 
    points_gpu = points.to(device='cuda:0')
else: 
    print('CUDA is not available')

# specify the number of the GPU device if torch.cuda.is_available(): points_gpu = points.to(device='cuda:0') else: print('CUDA is not available')

CUDA is not available

In [62]:

# Some more GPU operations, if CUDA is installed

# Some more GPU operations, if CUDA is installed

In [63]:

if torch.cuda.is_available(): 
    points = 2 * points  # <1> on CPU
    points_gpu = 2 * points.to(device='cuda')  # <2> on GPU 
else: 
    print('CUDA is not available')

if torch.cuda.is_available(): points = 2 * points # <1> on CPU points_gpu = 2 * points.to(device='cuda') # <2> on GPU else: print('CUDA is not available')

CUDA is not available

In [64]:

if torch.cuda.is_available(): 
    points_gpu = points_gpu + 4

if torch.cuda.is_available(): points_gpu = points_gpu + 4

In [65]:

if torch.cuda.is_available(): 
    points_cpu = points_gpu.to(device='cpu')

if torch.cuda.is_available(): points_cpu = points_gpu.to(device='cpu')

We can also use the shorthand method cpuand cuda instead of the to method.

In [66]:

points_gpu = points.cuda()  # <1>
points_gpu = points.cuda(0)
points_cpu = points_gpu.cpu()

points_gpu = points.cuda() # <1> points_gpu = points.cuda(0) points_cpu = points_gpu.cpu()

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
Input In [66], in <module>
----> 1 points_gpu = points.cuda()  # <1>
      2 points_gpu = points.cuda(0)
      3 points_cpu = points_gpu.cpu()

File /Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/torch/cuda/__init__.py:289, in _lazy_init()
    284     raise RuntimeError(
    285         "Cannot re-initialize CUDA in forked subprocess. To use CUDA with "
    286         "multiprocessing, you must use the 'spawn' start method"
    287     )
    288 if not hasattr(torch._C, "_cuda_getDeviceCount"):
--> 289     raise AssertionError("Torch not compiled with CUDA enabled")
    290 if _cudart is None:
    291     raise AssertionError(
    292         "libcudart functions unavailable. It looks like you have a broken build?"
    293     )

AssertionError: Torch not compiled with CUDA enabled

In [ ]: