Demos

Demo for TensorFlow Quant_INT8-8 Quantization in Non-Training Mode

Preparing the Environment

For details about how to prepare the environment, refer to Preparing the Environment to install TensorFlow and its dependencies.

Configuring the Model

• Prepare the quantization model.

  Load the .pb file of the baseline model to dopt_tf_py3/demo/quant8-8/notrain/tensorflow_mnist/basemodel/. This path contains the MNIST baseline model file mnist.pb.

• Prepare the input data for quantization.

  Load the image or binary calibration dataset to dopt_tf_py3/demo/quant8-8/notrain/tensorflow_mnist/mnist_test/ by referring to Quantizing the Model. This path contains a preset image calibration dataset.

Quantizing the Model

Run run_release.sh in dopt_tf_py3/demo/quant8-8/notrain/tensorflow_mnist/.

The quantized PB model and quantization configuration file are stored in dopt_tf_py3/demo/quant8-8/notrain/tensorflow_mnist. Run the demo and the following files will be generated:

Demo for PyTorch Quant_INT8-8 Quantization in Non-Training Mode

Preparing the Environment

For details about how to prepare the environment, refer to Preparing the Environment to install PyTorch and its dependencies.

Configuring the Model

• Prepare the quantization model.

  Save the model definition file (.py) and model parameter file of the baseline model to dopt_pytorch_py3/demo/quant8-8/notrain/pytorch_mnist/.

  The mnist.py baseline model definition file and the model parameter file mnist.pkl have been stored in the path.

• Prepare the input data for quantization.

  Load the image or binary calibration dataset to dopt_pytorch_py3/demo/quant8-8/notrain/pytorch_mnist/ by referring to Quantizing the Model.

Quantizing the Model

Run run_release.sh in dopt_pytorch_py3/demo/quant8-8/notrain/pytorch_mnist/.

In dopt_pytorch_py3/demo/quant8-8/notrain/pytorch_mnist/, you can find a PyTorch non-training quantization demo file, as shown in the following figure:

Demo for ONNX Quant_INT8-8 Quantization in Non-Training Mode

Preparing the Environment

For details about how to prepare the environment, refer to Preparing the Environment to install ONNX and its dependencies.

Sample Code

Add the dopt_onnx_py3 directory to the system environment and execute it in the terminal environment.

python3 ./dopt_so.py \
    --framework 5 \
    --mode   0 \
    --model "./resnet18_matmul.onnx" \          ## ONNX model to be quantized.
    --cal_conf "./config.prototxt" \            ## Calibration set configuration file.
    --output  "./resnet18_matmul_quant.onnx" \  ## Quantized ONNX file.
    --input_shape   input:1,3,128,128 \         ## Shape of the floating-point model's input.
    --compress_conf  ./mnist_param              ## Quantization configuration file generated by the dopt tool.

For details about the configuration of ./config.prototxt, please refer to How to Use the Configuration File.

strategy: 'Quant_INT8-8'
device: USE_CPU
preprocess_parameter:
{
    input_type: BINARY
    input_file_path: './input1.bin'
}

Demo for TensorFlow Quant_INT8-8 Plugin-based Quantization

Preparing the Environment

Prepare the TensorFlow environment by referring to Preparing the TensorFlow Environment. Install TensorFlow-gpu 2.8.0 and its necessary dependencies.

Sample Code

import sys
sys.path.append(".../dopt_tf_py3") ## The path should be an absolute path.

def generate_config():
    with tf.Session(config=config) as sess:
        build_tf_model() ## Customize the graph of the TensorFlow model to be quantized. Only the topology will be built, and the weight will not be loaded.
        from dopt.dopt_tf.opt_main import generate_config_file
        generate_config_file(sess, dst_path="./config_gen.json")

def train_model():
    with tf.Session(config=config) as sess:
        build_tf_model() ## Customize the graph of the TensorFlow model to be quantized. Only the topology will be built, and the weight will not be loaded.
        from dopt.dopt_tf.opt_main import optimize_model
        quant_flag = tf.placeholder(tf.int32)
        is_train_flag = tf.placeholder(tf.bool, name='is_train')
        ## Quantize the model. The graph will automatically be modified in tf.get_default_graph().
        optimize_model( 
            sess,
            "./config_gen.json",
            is_train_flag,
            quant_flag
        )
        ## Load the model weight after calling optimize_model.
        saver = tf.Saver()
        saver.restore(ckpt)
        tf.global_variables_initializer().run()
        ## train model
        for i in range(...):
            optimizer = ...
            feed_dict[is_train_flag] = True
            feed_dict[quant_flag] = 1
            sess.run(train_op, feed_dict)
        ## eval model
        feed_dict[is_train_flag] = False
        feed_dict[quant_flag] = 1
        sess.run(output, feed_dict)
        evaluate_output(output)

def calibrate_model():
    with tf.Session(config=config) as sess:
        build_tf_model() ## Customize the graph of the TensorFlow model to be quantized. Only the topology will be built, and the weight will not be loaded.
        from dopt.dopt_tf.opt_main import optimize_model, set_calibrate_state 
        quant_flag = tf.placeholder(tf.int32)
        is_train_flag = tf.placeholder(tf.bool, name='is_train')
        ## Quantize the model. The graph will automatically be modified in tf.get_default_graph().
        optimize_model( 
            sess,
            "./config_gen.json",
            is_train_flag,
            quant_flag
        )
        ## Load the model weight after calling optimize_model.
        saver = tf.Saver()
        saver.restore(ckpt)

        calibration_mode = True
        set_calibrate_state(sess, calibration_mode ) 
        ## eval model
        feed_dict[is_train_flag] = False
        feed_dict[quant_flag] = 1
        sess.run(output, feed_dict)
        evaluate_output(output)

def generate_params():
    with tf.Session(config=config) as sess:
        build_tf_model()
        from dopt.dopt_tf.opt_main import generate_final_model
        generate_final_model(
            sess,
            config_file         = "./config_gen.json",
            output_name_list    = ["output"],
            ckpt_file           = "train_ckpt_path",
            output_dir          = "./output_dir"
        )
if __name__ == "__main__":
    ## step 1
    ## Modify the configuration.
    generate_config()
   
    ## step 2
    ## Train the model until it meets the requirements.
    train_model() ## Retrain the model.
    ## calibrate_model()  ## Calibrate the model.
    
    ## step 3
    ## Extract parameters for subsequent model deployment.
    generate_params()

Demo for PyTorch Quant_INT8-8 Plugin-based Quantization

Preparing the Environment

import sys
sys.path.append(".../dopt_tf_py3") ## The path should be an absolute path.

def generate_config():
    model = build_torch_model()  ## Floating-point model to be quantized.
    generate_config_file(model, input_shape, dst_path="./config_gen.json") # model:torch.nn.Module, input_shape : "input1:input1.shape;input2:input2.shape"
    return model

def train_model():
    model = build_torch_model()  ## Floating-point model to be quantized.
    from dopt.dopt_torch.opt_main import optimize_model
    model.load_state_dict(state)  ## Load parameters for the floating-point model.
    
    ## Call optimize_model to quantize the model.
    quanted_model = optimize_model(model, config_path)
    
    ## train model
    quant_loss = get_quant_loss(quant_model)
    optimizer = torch.optim.SGD(quanted_model.parameters(), lr=0.001, momentum=0.9) ## Assumes that the SGD optimizer is used.
    
    for input_data, label in range(...):
        optimizer.zero_grad()
        outputs = model(input_data)
        loss = loss_fn(outputs, label) ## loss_fn indicates the loss of the original floating-point network training.
        
        total_loss = loss + quant_weight * quant_loss ## quant_weight indicates the proportion of the quantization loss.
        loss.backward()
        
        optimizer.step()

def calibrate_model():
    model = build_torch_model()  ## Floating-point model to be quantized.
    from dopt.dopt_torch.opt_main import optimize_model, set_calibrate_state
    model.load_state_dict(state)  ## Load parameters for the floating-point model.
    
    ## Call optimize_model to quantize the model.
    quanted_model = optimize_model(model, config_path)
    
    calibrate_mode = True
    set_calibrate_state(model, calibrate_mode)
    
    for input_data, label in range(...):
        outputs = model(input_data)

def generate_params():
    model = build_torch_model()
    from dopt.dopt_torch.opt_main import generate_final_model
    generate_final_model(model, 
                        config_file,
                        pth_file="quant.pth",
                        output_dir="./results_dir")

if __name__ == "__main__":
    ## step 1
    ## Modify the configuration.
    generate_config()
   
    ## step 2
    ## Train the model until it meets the requirements.
    train_model()
    ## calibrate_model()  ## Non-training mode.

    ## step 3
    ## Extract parameters for subsequent model deployment.
    generate_params()

TensorFlow NASEA NAS Demo

NASEA Classification Network Demo

The classification network demo is stored in tools_dopt/dopt_tf_py3/demo/nas_ea/ea_cls_imagenet and contains the following files.

• blocks.so: A search space file.
• readme.md: A search training guide.
• run_release.sh: A script that starts the search.
• scen.yaml: A configuration item.
• user_module.py: Customized APIs of the tool.

The procedure is as follows:

1. Prepare an ImageNet dataset (in TFRecord format) and change the dataset directory in scen.yaml.
2. For details, see Preparing the Environment.
3. Load the dependent open-source code.
   1. Go to the demo directory of the classification network.
      Collapse

      Word wrap

      Dark theme

      Copy code

      cd tools_dopt/dopt_tf_py3/demo/nas_ea/ea_cls_imagenet

   2. Download the open-source code.
      Collapse

      Word wrap

      Dark theme

      Copy code

      git clone https://github.com/Tensorflow/models.git

   3. Go to the directory of the open-source code.
      Collapse

      Word wrap

      Dark theme

      Copy code

      cd models

   4. Switch to the specified version:
      • If TensorFlow 1.12.0 is used, run the following command:
      Collapse

      Word wrap

      Dark theme

      Copy code

      git checkout v1.12.0

      • If TensorFlow 2.1.0 is used, run the following command:
      Collapse

      Word wrap

      Dark theme

      Copy code

      git checkout v2.1.0

   5. Return to the demo directory of the classification network.
      Collapse

      Word wrap

      Dark theme

      Copy code

      cd ..

   6. Set the default path of PYTHONPATH.
      Collapse

      Word wrap

      Dark theme

      Copy code

      export PYTHONPATH=$PYTHONPATH:`pwd`/models/

   NOTICE

   Each time you open the terminal, you need to run the command again. Alternatively, add the command to the ~/.bashrc file and run source ~/.bashrc.

4. Configure the scen.yaml file in the demo. For details, see Configuring Search Parameters. The scen.yaml file provides recommended parameters. You can modify them as required.
5. Modify the user_module.py file in the demo. For details about the model API definition, see Custom TensorFlow APIs. The user_module.py file provides recommended configurations. You can modify them as required.
6. Execute the run_release.sh script. Multiple model_arch_result_*.py files will be generated in results. You can select a proper network architecture for training based on the information provided in log_classification. For details about subsequent training, see readme.md.

NASEA Detection Network Demo

The detection network demo is stored in tools_dopt/dopt_tf_py3/demo/nas_ea/ea_det_coco and contains the following files.

NASEA Segmentation Network Demo

The segmentation network demo is stored in tools_dopt/dopt_tf_py3/demo/nas_ea/ea_seg_voc and contains the following files.

PyTorch NASEA NAS Demo

NASEA Classification Network

The classification network demo is stored in tools_dopt/dopt_pytorch_py3/demo/nas_ea/ea_cls_imagenet_pytorch and contains the following files.

• blocks.so: A search space file.
• readme.md: A search training guide.
• run_release.sh: A script that starts the search.
• scen.yaml: A configuration item.
• user_module.py: Customized APIs of the tool.

The procedure is as follows:

1. Prepare an ImageNet dataset (in original format) and change the dataset directory in scen.yaml.
2. For details, see Preparing the Environment.
3. Configure the scen.yaml file in the demo. For details, see Configuring Search Parameters. The scen.yaml file provides recommended parameters. You can modify them as required.
4. Modify the user_module.py file in the demo. For details about the model API definition, see Custom PyTorch APIs. The user_module.py file provides recommended configurations. You can modify them as required.
5. Execute the run_release.sh script. Multiple model_arch_result_*.py files will be generated in results. You can select a proper network architecture for training based on the information provided in log_classification. For details about subsequent training, see readme.md.

NASEA Segmentation Network Demo

The segmentation network demo is stored in tools_dopt/dopt_pytorch_py3/demo/nas_ea/ea_seg_voc_pytorch and contains the following files.

• blocks.so: A search space file.
• pre_train.yaml: A configuration item for pre-training.
• readme.md: A search training guide.
• run_release.sh: A script that starts the search.
• scen.yaml: A configuration item.
• user_module.py: Customized APIs of the tool.

The procedure is as follows:

1. Prepare datasets, including the ImageNet dataset (in original format) used for pre-training and the VOC dataset (in original format) used for training. If a pre-trained CKPT file is available, you do not need to prepare the ImageNet dataset. Change the dataset directory in the scen.yaml file by referring to Configuring Search Parameters.
2. For details, see Preparing the Environment.
3. Load the dependent open source code. For details, please refer to tools_dopt/dopt_pytorch_py3/demo/nas_ea/ea_seg_voc_pytorch/readme.md.
4. Configure the scen.yaml and pre_train.yaml files in the demo directory. For details, see Configuring Search Parameters. The scen.yaml file provides recommended parameters. You can modify them as required.
5. Modify the user_module.py file in the demo. For details about the model API definition, see Custom PyTorch APIs. The user_module.py file provides recommended configurations. You can modify them as required.
6. Execute the run_release.sh script. Multiple model_arch_result_*.py files will be generated in results. You can select a proper network architecture for training based on the information provided in log_segmentation. For details about subsequent training, see readme.md.