Prepare Models Overview

Model is the pipeline component used to process ingested unstructured data. VDP uses Triton Inference server for model serving. It supports multiple deep learning frameworks including TensorFlow, PyTorch, TensorRT, ONNX and OpenVINO. Besides, the Triton Python Backend enables Triton to support any model written in Python.

#VDP model layout

To deploy a model on VDP, we suggest you to prepare the model files following the layout:

├── <pre-model>
│   ├── 1
│   │   └──
│   └── config.pbtxt
├── <infer-model>
│ ├── 1
│ │ └── <model-file>
│ └── config.pbtxt
├── <post-model>
│   ├── 1
│   │   └── 
│   └── config.pbtxt
└── <ensemble-model>
   ├── 1
   │   └── .keep
   └── config.pbtxt

The above layout displays a typical VDP model consisting of

  • - model card to embed the metadata in front matter and descriptions in Markdown format
  • <pre-model> - Python model to pre-process input images
  • <infer-model> - Model to convert the unstructured data into structured data output, usually a Deep Learning (DL) / Machine Learning (ML) model
  • <post-model> - Python model to post-process the output of the infer-model into desired formats
  • <ensemble-model> - Triton ensemble model to connect the input and output tensors between the pre-processing, inference and post-processing models.
  • config.pbtxt - Model configuration for each sub model

You can name <pre-model>, <infer-model>, <post-model> and <ensemble-model> folders freely provided that the folder names are clear and semantic. All these models bundle into a deployable model for VDP.


As long as your model fulfils the required Triton model repository layout, it can be safely imported into VDP and deployed online.

#Serve models written in Python

To deploy your pre-processing and post-processing models with Python code, use Triton Python Backend that supports conda-pack to deploy Python models with dependencies. We have prepared a custom Conda environment with pre-installed libraries including scikit-learn, Pillow, PyTorch, torchvision, Transformers and triton_python_model. It is shipped with the NVIDIA GPU Cloud containers using Python 3.8.

If your model is not compatible with Python 3.8 or if it requires additional dependencies, you could create your own Conda environment and configure the config.pbtext to point to the custom conda-pack tar file accordingly.

#Prepare your model to be VDP compatible

🙌 After preparing your model to be VDP compatible, check out Import Models to learn about how to import the model into VDP from different sources.

Last updated: 5/29/2023, 12:50:07 AM