當咱們在安裝tensorflow時，咱們在安裝什麼？- Intro to TF, Virtualenv, Docker, CUDA, cuDNN, NCCL, Bazel

(Mainly quoted from its official website)

Summary:

1. TensorFlow™ is an open source software library.

2. Virtualenv is a tool to create isolated Python environments.

3. Docker:

    An image is a executable package.

    A container is a runtime instance of an image.

4. CUDA® is a parallel computing platform and programming model.

    The CUDA Toolkit is used to develop GPU-accelerated applications.

5.  cuDNN is a GPU-accelerated library of primitives for deep neural networks.

6.  NCCL implements multi-GPU and multi-node collective communication primitives.

7.  Bazel is an open-source build and test tool.

------------------------------------------------------------------------------------

TensorFlow™

TensorFlow™ is an open source software library for high performance numerical computation.

Its flexible architecture allows easy deployment of computation across a variety of platforms (CPUs, GPUs, TPUs), and from desktops to clusters of servers to mobile and edge devices.

------------------------------

Virtualenv

Virtualenv is a tool to create isolated Python environments.

e.g. Some applications need python 2.x, while some need python 3.x, so that they cannot share the system default python.

One solution is to use virtualenv to create isolated python environments for different applications.

Usage:

1. Create an environment (e.g. venv)
2. Use source to enter this environment (e.g. $ source venv/bin/activate)
3. Inside the venv, the packages installed by pip will be installed in the venv and will not be influenced by system default python. (e.g. (venv) $ pip install ***)
4. Use deactivate to exit the current environment. (e.g. (venv) $ deactivate)

------------------------------

Docker

An image is a lightweight, stand-alone, executable package that includes everything needed to run a piece of software, including the code, a runtime, libraries, environment variables, and config files.

A container is a runtime instance of an image—what the image becomes in memory when actually executed. It runs completely isolated from the host environment by default, only accessing host files and ports if configured to do so.

 

Container vs. virtual machines

Containers run apps natively on the host machine’s kernel. They have better performance characteristics than virtual machines that only get virtual access to host resources through a hypervisor. Containers can get native access, each one running in a discrete process, taking no more memory than any other executable.

 

Virtual machines run guest operating systems—note the OS layer in each box. This is resource intensive, and the resulting disk image and application state is an entanglement of OS settings, system-installed dependencies, OS security patches, and other easy-to-lose, hard-to-replicate ephemera. 這是資源密集型的，所產生的磁盤映像和應用程序狀態，會致使衆多問題的糾纏，如操做系統設置、系統安裝依賴項、OS安全修補程序和其餘易丟失、因僅短時間有效而難以複製的生命期。

Containers can share a single kernel, and the only information that needs to be in a container image is the executable and its package dependencies, which never need to be installed on the host system. These processes run like native processes, and you can manage them individually by running commands like docker ps—just like you would run ps on Linux to see active processes. Finally, because they contain all their dependencies, there is no configuration entanglement; a containerized app 「runs anywhere.」 容器能夠共享單個內核，而且容器映像中惟一須要的信息是可執行文件及其包依賴項，它們永遠不須要安裝在主機系統上。 這些進程像本機進程同樣運行，您能夠經過運行docker ps之類的命令來單獨管理它們，就像在Linux上運行ps來查看活動進程同樣。 最後，由於它們包含全部依賴項，因此沒有配置糾纏; 容器化的應用程序「隨處運行」。

With containerization, developers don’t need to write application code into different VMs operating different app components to retrieve compute, storage and networking resources. A complete application component can be executed in its entirety within its isolated environment without affecting other app components or software. Conflicts within libraries or app components do not occur during execution and the application container can move between the cloud or data center instances efficiently.

(https://www.bmc.com/blogs/containers-vs-virtual-machines/)

------------------

Virtualenv vs. Container

Virtualenv is used to isolate different python versions.

Container is used to let different components of a complete application be executed in its entirety within its isolated resources. Each container contains everything needed to run a piece of software.

-----------------------------------

CUDA (https://developer.nvidia.com/cuda-zone)

CUDA® is a parallel computing platform and programming model developed by NVIDIA for general computing on graphical processing units (GPUs). With CUDA, developers are able to dramatically speed up computing applications by harnessing the power of GPUs.

In GPU-accelerated applications, the sequential part of the workload runs on the CPU – which is optimized for single-threaded performance – while the compute intensive portion of the application runs on thousands of GPU cores in parallel. When using CUDA, developers program in popular languages such as C, C++, Fortran, Python and MATLAB and express parallelism through extensions in the form of a few basic keywords.

The CUDA Toolkit from NVIDIA provides everything you need to develop GPU-accelerated applications. The CUDA Toolkit includes GPU-accelerated libraries, a compiler, development tools and the CUDA runtime.

-------------------------------

cuDNN (https://developer.nvidia.com/cudnn)

The NVIDIA CUDA® Deep Neural Network library (cuDNN) is a GPU-accelerated library of primitives for deep neural networks. cuDNN provides highly tuned implementations for standard routines such as forward and backward convolution, pooling, normalization, and activation layers. cuDNN is part of the NVIDIA Deep Learning SDK.

Deep learning researchers and framework developers worldwide rely on cuDNN for high-performance GPU acceleration. It allows them to focus on training neural networks and developing software applications rather than spending time on low-level GPU performance tuning. cuDNN accelerates widely used deep learning frameworks, including Caffe,Caffe2, Chainer, Keras,MATLAB, MxNet, TensorFlow, and PyTorch.

---------------------

NCCL

The NVIDIA Collective Communications Library (NCCL) implements multi-GPU and multi-node collective communication primitives that are performance optimized for NVIDIA GPUs. NCCL provides routines such as all-gather, all-reduce, broadcast, reduce, reduce-scatter, that are optimized to achieve high bandwidth over PCIe and NVLink high-speed interconnect.

Developers of deep learning frameworks can rely on NCCL’s highly optimized, MPI compatible and topology aware routines, to take full advantage of all available GPUs within and across multiple nodes. Leading deep learning frameworks such as Caffe,Caffe2, Chainer, MxNet, TensorFlow, and PyTorch have integrated NCCL to accelerate deep learning training on multi-GPU systems. 

 

---------------------

Bazel

Bazel is an open-source build and test tool similar to Make, Maven, and Gradle. It uses a human-readable, high-level build language. Bazel supports projects in multiple languages and builds outputs for multiple platforms. Bazel supports large codebases across multiple repositories, and large numbers of users.

(Can use it to build tensorflow from source.)