PyTorch 0.3 Is Out With Performance Improvements, ONNX/CUDA 9/CUDNN 7 Support

Dec. 5, 2017, 7:58 p.m. By: Kirti Bakshi

PyTorch 0.3

PyTorch, as we know, is a Python package that provides its users two high-level features:

Tensor computation just like that of NumPy with strong GPU acceleration.
Deep neural networks that are built on an autograd system that is tape-based.

You can also reuse all of your favorite Python packages such as SciPy, NumPy, and Cython in order to extend PyTorch when needed. PyTorch can be used by any user either as:

A replacement for NumPy in order to use the power of GPUs.
A deep learning research platform that results in the provision of maximum flexibility as well as speed.

And Now PyTorch 0.3 Is again Out With Improvements in Performance as well as ONNX/CUDA 9/CUDNN 7 Support. Among all, some of the New features that have been added to the new release are as follows:

1. Unreduced losses: Now, Some loss functions in a mini-batch can compute per-sample losses.

By default PyTorch now has overcome something that was limited to users. It now sums losses over the mini-batch and returns back as a single scalar loss.
Now, a subset of loss functions in order to return individual losses for each sample in the mini-batch allow specifying reduce=False
The list of all Currently supported losses are: MSELoss, NLLLoss, NLLLoss2d, KLDivLoss, CrossEntropyLoss, SmoothL1Loss, L1Loss
There are also More loss functions that are said to be covered in the next release.

2. An in-built Profiler in the autograd engine:

They have also now built a low-level profiler in order to help you identify all the bottlenecks in your models.

3. Higher-order gradients:

Gradients that support layers like ConvTranspose, AvgPool1d, AvgPool2d etc.

4. Optimizers:

The Optimizers now also have an add_param_group function that allows its users to add new parameter groups to an optimizer that is already constructed.
optim.SparseAdam now Implements a lazy version of Adam algorithm that is considered as suitable for sparse tensors.

5. New layers and nn functionality:

Added AdaptiveAvgPool3d as well as AdpativeMaxPool3d.
Addition of LPPool1d.
The DataParallel container on CPU instead of considering it as an error out is now a no-op.
F.pad now has support for 'reflection' and 'replication' padding on all 1d, 2d, 3d signals, and constant padding on n-d signals as well.

6. New Tensor functions and features:

The Introduction of torch.erfinv as well as torch.erf that computes the inverse error function and the error function of each element present in the Tensor.
The addition of a broadcasting support to all bitwise operators.
The addition of zeros_like as well as zeros for sparse Tensors.
1-element Tensors can now also be cast to Python scalars.

Some of the improvements in the performance are as follows:

The overhead of torch functions on Variables that was around 10 microseconds previously has been brought down to as less as 1.5 microseconds.
All pointwise ops now get multi-core CPU benefits with the use of OpenMP.
binaries.
The Reduction of overhead broadcasting if Tensors do not turn out to be broadcastable.
A performance improvement of 2.5x to 3x in the distributed AllReduce (gloo backend) by the enabling of GPUDirect.

In addition to all of the above the improvements in its usability are as follows:

Increased cogent error messages during the indexing of Tensors or Variables
Breaking changes.
Addition of proper error message for the specification of dimensions on a tensor without any dimensions.
The betterment of error messages for Conv*d input shape checking.
Increase in the user-friendly error messages for the indexing of LongTensor.
Better error messages and argument checking for Conv*d routines.
A try in the construction of a Tensor from a Variable fails more appropriately.
If a user is supposedly using an insufficient CUDA version, a PyTorch binary, then a warning is printed back to the user.
Incoherent error messages that are Fixed in the load_state_dict.
Fix error message along with sparse tensors for type mismatches.