TensorRT/PluginFAQ

Why we need clone() for Plugin interface?

TensorRT API documentation provides an explanation about this. Simply, it intends to share immutable resources, like input/output dimensions, workspace ptr or weights and etc, among different execution contexts and these immutable resources or parameters are created per each engine, and copied over the invocation of clone().

• The following shows the callback flow of all IPlugin key APIs from network parsing, to engine building to inferencing (take reference from TensorRT 5.1.5).
  NOTE: the digit ‘0x12458e00’ is the plugin object ptr and can be used to track when it gets destroyed.

parsing:
		CropAndResizePluginCreator::createPlugin() --> 0x12458e00
		CropAndResizePlugin::CropAndResizePlugin()
		CropAndResizePlugin::setPluginNamespace
		CropAndResizePlugin::getNbOutputs
	
		CropAndResizePlugin::clone() --> 0x12459300
		CropAndResizePlugin::CropAndResizePlugin()
		CropAndResizePlugin::getNbOutputs()
		CropAndResizePlugin::getOutputDataType()
		CropAndResizePlugin::isOutputBroadcastAcrossBatch()
		CropAndResizePlugin::getOutputDimensions()
	parsing done:

 	engine building:    --> graph optimization, tactic selection and decide format
		CropAndResizePlugin::clone() --> 0xf9e8e0
		CropAndResizePlugin::CropAndResizePlugin()
		CropAndResizePlugin::supportsFormat() 
		CropAndResizePlugin::getOutputDataType()	

		CropAndResizePlugin::clone() --> 0x6380f750
		CropAndResizePlugin::CropAndResizePlugin() 
		CropAndResizePlugin::supportsFormat()
		CropAndResizePlugin::configurePlugin()
		CropAndResizePlugin::getWorkspaceSize()
		CropAndResizePlugin::initialize()
		
		CropAndResizePlugin::getSerializationSize()
		CropAndResizePlugin::serialize()
		
		CropAndResizePlugin::destroy() --> 0xf9e8e0
		CropAndResizePlugin::~CropAndResizePlugin()
		
		CropAndResizePlugin::getSerializationSize()
		CropAndResizePlugin::serialize()
		
		CropAndResizePlugin::destroy() --> 0x12458e00
		CropAndResizePlugin::~CropAndResizePlugin()
		
		CropAndResizePlugin::destroy() --> 0x12459300
		CropAndResizePlugin::~CropAndResizePlugin()
	engine building done:

	createExecutionContext and infer()
		CropAndResizePlugin::clone() --> 0x166b2840
		CropAndResizePlugin::CropAndResizePlugin()
		CropAndResizePlugin::attachToContext()
		CropAndResizePlugin::enqueue()
		
		ropAndResizePlugin::detachFromContext
		CropAndResizePlugin::terminate
		CropAndResizePlugin::destroy() --> 0x166b2840
		CropAndResizePlugin::~CropAndResizePlugin()
		
		CropAndResizePlugin::terminate()
		CropAndResizePlugin::destroy() --> 0x6380f750
		CropAndResizePlugin::~CropAndResizePlugin()
	Finished	

• The following shows the callback flow from engine/plan deserializing to inferencing (take reference from TensorRT 5.1.5).

Deserializing:
		CropAndResizePluginCreator::deserializePlugin --> 0xd10c5b0
			CropAndResizePlugin::CropAndResizePlugin
		CropAndResizePlugin::initialize()	
	Deserializing done
	
	createExecutionContext and infer:
		CropAndResizePlugin::enqueue()
		CropAndResizePlugin::terminate()
		CropAndResizePlugin::destroy()
		CropAndResizePlugin::~CropAndResizePlugin()
	Finished

NOTE: TensorRT 5.1 has a known issue that initialize() and terminate() don’t appear in pairs (terminate() will be invoked twice by different objects). If you want to allocate immutable resource for your plugin in initialize(), you better make it as std::shared_ptr so that it won’t get released twice from terminate().