For any deep learning project, the end goal is to create a system which can be easily used for inference. A simple solution can be to create an application that takes an input and gives out the output without any further guidance from the user. The application could be a Python or C++ deployed application, or a web application hosted on a server. Web Deployments are computationally costly and require a constant connection to the internet for the application to work, making it unreliable and costly. Desktop applications like Python or C++ deployments are much more easier, and can be structured to run on any desktop machine. In this tutorial we shall focus upon Deploying PyTorch based applications using C++.

A major pre-requisite for the project is having OpenCV installed in the application’s root directory, this OpenCV installation library shall be required during the CMake installation of our application. To know more about how to install OpenCV C++ please follow the blog post given here. The next step is to get PyTorch’s library for C++, or LibTorch as it is called officially. You can download the library’s zip file from here. Once you have the zip file downloaded on your application’s root folder, the next step is to extract the folder. The path of these files shall be provided to the CMakeList.txt file we create at the end of the tutorial.

Once the pre-requisites are completed, the only remaining step is to create the main file that contains the source code for our application and the header files we will create to load the images into PyTorch format dataset, we shall call these file as main.cpp, dataset.h, and load_from_folder.h and they will be placed in the src folder. The directory structure of the applications root folder is shown in Fig. 1.


In the folder_reader.h file we will add a function that returns a vector of strings which contain the paths of the test files. We use the filesystem library to iterate over the files inside a directory. The function given blow reads the files in a directory and stores them in a vector of strings:


auto folder_to_file(std::string& location) -> std::vector<std::tuple<std::string>>{
std::vector<std::tuple<std::string>> file_location;
std::string path = location;
for (const auto & entry : fs::directory_iterator(path)){
file_location.push_back (std::make_tuple(entry.path()));
}
return file_location;
}

For a better understanding, all the source files will be uploaded in a GitHub repository to replicate the project easily, the repository can be found here. The next step is to use this iterator function to create a PyTorch style dataset, this dataset can be then fed into a Dataloader to return batches of image tensors we can use for inference. We can declare the dataset using the class definition given below:


class image_dataset : public torch::data::Dataset<image_dataset>
{
private:
std::vector<std::tuple<std::string>> path_to_files;
public:
explicit image_dataset(std::string& file_name) : path_to_files(folder_to_file(file_name)){};

torch::data::Example<> get(size_t index) override {
std::string file_location = std::get<0> (path_to_files[index]);
cv::Mat img = cv::imread(file_location,1);
cv::resize(img, img, cv::Size(224,224), 0, 0, cv::INTER_LINEAR);
cv::normalize(img, img, -1, 1, cv::NORM_MINMAX);
torch::Tensor img_tensor = torch::from_blob(img.data, {img.rows, img.cols, 3}, torch::kF32).clone();
img_tensor = img_tensor.permute({2,0,1});
return {img_tensor, img_tensor};
};
torch::optional<size_t> size() const override {
return path_to_files.size();
};
};


The above function creates a dataset class which loads inference images as OpenCV image files, resizes them into a size of 224 pixels by 224 pixels, and then performs the min max normalization. We then convert this image into PyTorch tensors and then rearrange the tensor array to channel first. Now that we have our dataset class ready we can start the inference using Libtorch library.


In the main.cpp file the first step is to import all the required libraries and functions we created in our header file. We then check if the machine has CUDA supported GPUs or not. This is done using:

auto cuda_available = torch::cuda::is_available();
torch::Device device(cuda_available ? torch::kCUDA : torch::kCPU);


If the machine has CUDA supported GPUs available the device is set to kCUDA, other wise the inference is performed on kCPU. Now that we have our inference device set, the next step is to import the dataset, and feed it to a Dataloader to get batches of images. The dataset class referenced above takes the absolute path of folder as an input, we would be feeding the path as a command line argument, this can be done using:

std::string path_to_folder = argv[1];
int batch_size = 32;
auto data_set = image_dataset(path_to_folder) .map(torch::data::transforms::Stack<>());
auto data_loader = torch ::data::make_data_loader <torch::data::samplers::SequentialSampler>(
data_set,
batch_size);


path_to_folder is the command line argument we will pass when running the deployment program, the batch_size is set to 32. We then load the dataset into stack of columns which is then fed to the dataloader which samples the dataset sequentially using the sequential sampler. Now that we have our dataloader set, we iterate over the dataset using this and perform inference sequentially. The outputs of these inferences are stored in a vector of tensors which can be then converted into a vector of floating point numbers. The complete process can be executed using:


std::vector<torch::jit::IValue> outputs_array;
for(auto& batch : *data_loader){
std::cout<<"Infering on a batch"<<std::endl;
auto images = batch.data.to(device);
std::vector<torch::jit::IValue> inputs;
inputs.push_back(images);
at::Tensor output = module.forward(inputs).toTensor();
std::cout << output << ' ';
outputs_array.push_back(output);
}


We now set up the CMakeLists.txt file and then proceed with the final steps of our deployment. The CMakeLists.txt file has the following content:

cmake_minimum_required(VERSION 3.0 FATAL_ERROR)
project(infer-app)
find_package(OpenCV REQUIRED)
find_package(Torch REQUIRED)
add_executable(infer-app main.cpp)
set_property(TARGET infer-app PROPERTY CXX_STANDARD 17)


The next step is to arrange the directory as shown in the diagram at the start. This will create an infer-app named executable in the build folder. The application can be run using:


mkdir build
cd build
cmake-DCMAKE_PREFIX_PATH =ABSOLUTE_PATH_TO_LIBTORCH_FOLDER .. && make -j32


This will create an infer-app named executable in the build folder. The application can be run using:


./infer-app PATH_TO_IMAGE_FOLDER