trt-cpp-runtime-quickstart

TensorRT C++ Runtime Deploy

Load a serialized TensorRT engine from disk and run inference from C++ using only the modern Runtime API. Produces a minimal, copy-pasteable deploy harness that drops next to any .plan / .engine file and extends to production.

Reference samples to open before writing new code:

• quickstart/SemanticSegmentation/tutorial-runtime.cpp — cleanest minimal load-and-run example. Mirrors Steps 1–7 below.
• samples/sampleOnnxMNIST/sampleOnnxMNIST.cpp — end-to-end sample that also builds the engine; the runtime portion shows realistic I/O wiring.
• Public headers: include/NvInferRuntime.h — read IRuntime, ICudaEngine, IExecutionContext, IStreamReaderV2.

When to Use

Prerequisites

1. TensorRT installed. Verify NvInferRuntime.h is on the include path and libnvinfer.so is on the link path. On a TRT dev container these are in /usr/include/x86_64-linux-gnu/ and /usr/lib/x86_64-linux-gnu/ (or /opt/tensorrt/... for tarball installs).
2. CUDA toolkit available. cuda_runtime_api.h and libcudart.so must be reachable; nvcc --version should match the CUDA version the engine was built against.
3. A serialized engine. A .plan/.engine file built on the same major TRT version and the same GPU architecture (compute capability) you will deploy on. Engines are not portable across major TRT versions or across SMs unless the builder was given --hardwareCompatibilityLevel.
4. The engine's I/O tensor names. Inspect with:

   trtexec --loadEngine=model.plan --verbose 2>&1 | grep -E 'Input|Output'
5. A C++17 compiler (g++ >= 9 or clang++ >= 10).

Step 1: Create the IRuntime

The runtime owns engine deserialization and must outlive every ICudaEngine it creates. Construct one per process for typical deployments.

class Logger : public nvinfer1::ILogger {
public:
    void log(Severity severity, char const* msg) noexcept override {
        if (severity <= Severity::kWARNING) {
            std::cerr << msg << std::endl;
        }
    }
};
 
Logger gLogger;
std::unique_ptr<nvinfer1::IRuntime> runtime{
    nvinfer1::createInferRuntime(gLogger)};
if (!runtime) throw std::runtime_error("createInferRuntime failed");

A custom logger is mandatory - TensorRT does not log internally. Keep it process-global so deserialization warnings (version skew, calibrator mismatch) are not lost.

Step 2: Read the Plan into Memory

For small/medium engines (< ~1 GiB) read the whole file into a std::vector<char> and hand the pointer to IRuntime::deserializeCudaEngine(blob, size). This is what the SemanticSegmentation tutorial does and the simplest correct path:

std::ifstream f(planPath, std::ios::binary);
if (!f) throw std::runtime_error("cannot open " + planPath);
f.seekg(0, std::ios::end);
auto size = static_cast<size_t>(f.tellg());
f.seekg(0, std::ios::beg);
std::vector<char> blob(size);
if (!f.read(blob.data(), size))
    throw std::runtime_error("short read on " + planPath);

For very large engines, or when the bytes live behind a stream (HTTP, mmap'd archive, encrypted store), implement an IStreamReaderV2 - see Step 3.

Step 3 (optional): Use IStreamReaderV2 for Streaming Loads

IStreamReader (v1) is deprecated in TensorRT 11.0. Always use IStreamReaderV2: it reads into both host and device memory and is the only stream-reader form guaranteed for new code. Subclass and implement read(...) and seek(...):

class FileStreamReader : public nvinfer1::IStreamReaderV2 {
public:
    explicit FileStreamReader(std::string const& path)
        : mFile(path, std::ios::binary) {
        if (!mFile) throw std::runtime_error("open failed: " + path);
    }
    int64_t read(void* dst, int64_t n,
                 cudaStream_t /*stream*/) noexcept override {
        mFile.read(static_cast<char*>(dst), n);
        return mFile.gcount();
    }
    bool seek(int64_t off, nvinfer1::SeekPosition where) noexcept override {
        auto dir = (where == nvinfer1::SeekPosition::kSET) ? std::ios::beg
                 : (where == nvinfer1::SeekPosition::kCUR) ? std::ios::cur
                 : std::ios::end;
        mFile.clear();
        mFile.seekg(off, dir);
        return static_cast<bool>(mFile);
    }
private:
    std::ifstream mFile;
};
 
FileStreamReader rd{planPath};
std::unique_ptr<nvinfer1::ICudaEngine> engine{
    runtime->deserializeCudaEngine(rd)};

Step 4: Deserialize and Create an Execution Context

ICudaEngine is thread-safe for read-only queries; IExecutionContext is not - allocate one per inference thread.

std::unique_ptr<nvinfer1::ICudaEngine> engine{
    runtime->deserializeCudaEngine(blob.data(), blob.size())};
if (!engine) throw std::runtime_error("deserializeCudaEngine failed");
 
std::unique_ptr<nvinfer1::IExecutionContext> ctx{
    engine->createExecutionContext()};
if (!ctx) throw std::runtime_error("createExecutionContext failed");

Step 5: Wire Tensors with setTensorAddress

Enumerate I/O tensors via getNbIOTensors() + getIOTensorName(i). Use getTensorIOMode, getTensorDataType, and getTensorShape to size and allocate buffers. Set every tensor address before enqueueV3 - the modern API has no implicit binding-index map.

for (int i = 0; i < engine->getNbIOTensors(); ++i) {
    char const* name = engine->getIOTensorName(i);
    auto mode  = engine->getTensorIOMode(name);
    auto shape = engine->getTensorShape(name);   // -1 = dynamic dim
    if (mode == nvinfer1::TensorIOMode::kINPUT && hasDynamic(shape)) {
        // Fill in concrete shape, e.g. batch=1
        shape.d[0] = 1;
        ctx->setInputShape(name, shape);
    }
}
// After setInputShape on all dynamic inputs, query output shapes.
for (int i = 0; i < engine->getNbIOTensors(); ++i) {
    char const* name = engine->getIOTensorName(i);
    auto bytes = elementCount(ctx->getTensorShape(name))
               * dtypeSize(engine->getTensorDataType(name));
    void* dev = nullptr;
    cudaMalloc(&dev, bytes);
    ctx->setTensorAddress(name, dev);
}

Always call setInputShape for dynamic inputs before querying output shapes - the latter depends on the former.

Step 6: Run enqueueV3

enqueueV3(stream) is the only non-deprecated enqueue API; enqueueV2/execute* are gone in modern flows.

cudaStream_t stream{};
cudaStreamCreate(&stream);
 
cudaMemcpyAsync(devInput, hostInput, inBytes,
                cudaMemcpyHostToDevice, stream);
if (!ctx->enqueueV3(stream))
    throw std::runtime_error("enqueueV3 failed");
cudaMemcpyAsync(hostOutput, devOutput, outBytes,
                cudaMemcpyDeviceToHost, stream);
cudaStreamSynchronize(stream);

If you reuse buffers across iterations, skip the per-call setTensorAddress - addresses persist on the context until overwritten.

Step 7: Shutdown Order

Destroy in reverse construction order: contexts -> engines -> runtime, then free CUDA memory and destroy the stream. With std::unique_ptr this is automatic as long as the context is declared after the engine, and the engine after the runtime. Free cudaMalloc allocations explicitly (RAII wrapper recommended).

Build

Wire the steps above into your application's build system. For a standalone smoke test, a minimal build is:

g++ -std=c++17 runtime.cpp -o run -lnvinfer -lcudart   # adjust CUDA/TRT include + lib paths
./run model.plan

Common Errors

Pitfalls

• Do not use IStreamReader v1. Deprecated in TRT 11.0. Use IStreamReaderV2 (note cudaStream_t parameter on read).
• Do not use enqueueV2 / execute / binding indices. These are legacy paths; the only stable modern path is name-based setTensorAddress + enqueueV3.
• One IExecutionContext per thread. Sharing contexts across threads is undefined behavior; sharing the engine is fine.
• Stream lifetime. The CUDA stream passed to enqueueV3 must outlive the inference. Destroying it while work is in flight crashes or corrupts output.
• Async vs sync copies. Mixing synchronous cudaMemcpy with enqueueV3 on a stream serializes the GPU; always pair enqueueV3 with cudaMemcpyAsync on the same stream.
• Engine portability. A .plan is tied to (TRT major version, GPU SM, CUDA major version). Never check engines into a repo without recording these three facts.
• Logger lifetime. The logger passed to createInferRuntime must outlive the runtime; a stack-local logger in main is fine, a function- scope local is a use-after-free.
• Refit / weight streaming. Engines built with refit or weight streaming enabled need extra setup calls (setWeightStreamingBudgetV2, IRefitter); out of scope here.*