Prepare the model to be quantized.

Save the .pb file of the baseline model to dopt_tf_py3/demo/quant8-8/notrain/tensorflow_mnist/basemodel/. In the demo, we have saved the MNIST baseline model file mnist.pb to the path.
Prepare the input data for quantization.

For details, see Model Quantization. Put the calibration set in image or binary format into dopt_tf_py3/demo/quant8-8/notrain/tensorflow_mnist/mnist_test/. In the demo, we have saved a preset image calibration set to the path.

[h2]Model Quantization

Run run_release.sh in dopt_tf_py3/demo/quant8-8/notrain/tensorflow_mnist/.

The quantized PB model and quantization configuration file are stored in dopt_tf_py3/demo/quant8-8/notrain/tensorflow_mnist. Run the demo and the following files will be generated:

Demo for PyTorch Quant_INT8-8 Quantization in Non-Training Mode

[h2]Environment Setup

Install PyTorch and its dependencies. For details, see Environment Setup.

[h2]Configuring the Model

Prepare the model to be quantized.

Save the baseline model definition file (.py) and the model parameter file to dopt_pytorch_py3/demo/quant8-8/notrain/pytorch_mnist/.

In the demo, we have saved the mnist.py baseline model definition file and the model parameter file mnist.pth in the path.
Prepare the input data for quantization.

For details, see Model Quantization. Put the calibration set in image or binary format into dopt_pytorch_py3/demo/quant8-8/notrain/pytorch_mnist/.

[h2]Model Quantization

Run run_release.sh in dopt_pytorch_py3/demo/quant8-8/notrain/pytorch_mnist/.

In dopt_pytorch_py3/demo/quant8-8/notrain/pytorch_mnist/, you can find a PyTorch non-training quantization demo file, as shown in the following figure:

Demo for ONNX Quant_INT8-8 Quantization in Non-Training Mode

[h2]Environment Setup

Install ONNX and its dependencies. For details, see Environment Setup.

[h2]Sample Code

Add the dopt_onnx_py3 directory to the system environment and execute it in the terminal environment.

python3 ./dopt_so.py \
    --framework 5 \
    --mode   0 \
    --model "./resnet18_matmul.onnx" \          ## ONNX model to be quantized.
    --cal_conf "./config.prototxt" \            ## Calibration set configuration file.
    --output  "./resnet18_matmul_quant.onnx" \  ## Quantized ONNX file.
    --input_shape   input:1,3,128,128 \         ## Shape of the floating-point model's input.
    --compress_conf  ./mnist_param              ## Quantization configuration file generated by the dopt tool

The ./config.prototxt configuration content is as follows (see How to Use the Configuration File):

strategy: 'Quant_INT8-8'
device: USE_CPU
preprocess_parameter:
{
    input_type: BINARY
    input_file_path: './input1.bin'
}

Demo for TensorFlow Quant_INT8-8 Plugin-based Quantization

[h2]Environment Setup

For details, see Preparing the TensorFlow Environment. Install TensorFlow-gpu 2.8.0 and its necessary dependencies.

[h2]Sample Code

import sys
sys.path.append(".../dopt_tf_py3") ## The path should be an absolute path.

def generate_config():
    with tf.Session(config=config) as sess:
        build_tf_model() ## Customize the graph of the TensorFlow model to be quantized. Only the topology will be built, and the weight will not be loaded.
        from dopt.dopt_tf.opt_main import generate_config_file
        generate_config_file(sess, dst_path="./config_gen.json")

def train_model():
    with tf.Session(config=config) as sess:
        build_tf_model() ## Customize the graph of the TensorFlow model to be quantized. Only the topology will be built, and the weight will not be loaded.
        from dopt.dopt_tf.opt_main import optimize_model
        quant_flag = tf.placeholder(tf.int32)
        is_train_flag = tf.placeholder(tf.bool, name='is_train')
        ## Quantize the model. The graph will automatically be modified in **tf.get_default_graph()**.
        optimize_model(
            sess,
            "./config_gen.json",
            is_train_flag,
            quant_flag
        )
        ## Load the model weight after calling **optimize_model**.
        saver = tf.Saver()
        saver.restore(ckpt)
        tf.global_variables_initializer().run()
        ## train model
        for i in range(...):
            optimizer = ...
            feed_dict[is_train_flag] = True
            feed_dict[quant_flag] = 1
            sess.run(train_op, feed_dict)
        ## eval model
        feed_dict[is_train_flag] = False
        feed_dict[quant_flag] = 1
        sess.run(output, feed_dict)
        evaluate_output(output)

def calibrate_model():
    with tf.Session(config=config) as sess:
        build_tf_model() ## Customize the graph of the TensorFlow model to be quantized. Only the topology will be built, and the weight will not be loaded.
        from dopt.dopt_tf.opt_main import optimize_model, set_calibrate_state
        quant_flag = tf.placeholder(tf.int32)
        is_train_flag = tf.placeholder(tf.bool, name='is_train')
        ## Quantize the model. The graph will automatically be modified in **tf.get_default_graph()**.
        optimize_model(
            sess,
            "./config_gen.json",
            is_train_flag,
            quant_flag
        )
        ## Load the model weight after calling **optimize_model**.
        saver = tf.Saver()
        saver.restore(ckpt)

        calibration_mode = True
        set_calibrate_state(sess, calibration_mode )
        ## eval model
        feed_dict[is_train_flag] = False
        feed_dict[quant_flag] = 1
        sess.run(output, feed_dict)
        evaluate_output(output)

def generate_params():
    with tf.Session(config=config) as sess:
        build_tf_model()
        from dopt.dopt_tf.opt_main import generate_final_model
        generate_final_model(
            sess,
            config_file         = "./config_gen.json",
            output_name_list    = ["output"],
            ckpt_file           = "train_ckpt_path",
            output_dir          = "./output_dir"
        )
if __name__ == "__main__":
    ## step 1
    ## Modify the configuration.
    generate_config()
    
    ## step 2
    ## Train the model until it meets the requirements.
    train_model() ## Retrain the model.
    ## calibrate_model()  ## Calibrate the model.
    
    ## step 3
    ## Extract parameters for subsequent model deployment.
    generate_params()

Demo for PyTorch Quant_INT8-8 Plugin-based Quantization

[h2]Environment Setup

For details, see Preparing the PyTorch Environment. Install PyTorch-gpu 1.11 and its necessary dependencies.

[h2]Sample Code

import sys
sys.path.append(".../dopt_tf_py3") ## The path should be an absolute path.

def generate_config():
    model = build_torch_model()  ## Floating-point model to be quantized.
    generate_config_file(model, input_shape, dst_path="./config_gen.json") # model: torch.nn.Module, input_shape : "input1:input1.shape;input2:input2.shape"
    return model

def train_model():
    model = build_torch_model() ## Floating-point model to be quantized.
    from dopt.dopt_torch.opt_main import optimize_model
    model.load_state_dict(state)  ## Load parameters for the floating-point model.
    
    ## Call **optimize_model** to quantize the model.
    quanted_model = optimize_model(model, config_path)
    
    ## train model
    quant_loss = get_quant_loss(quant_model)
    optimizer = torch.optim.SGD(quanted_model.parameters(), lr=0.001, momentum=0.9) ## Assumes that the SGD optimizer is used.
    
    for input_data, label in range(...):
        optimizer.zero_grad()
        outputs = model(input_data)
        loss = loss_fn(outputs, label) ## **loss_fn** indicates the training loss of the original floating-point network.
        
        total_loss = loss + quant_weight * quant_loss ## **quant_weight** indicates the proportion of the quantization loss.
        loss.backward()
        
        optimizer.step()

def calibrate_model():
    model = build_torch_model() ## Floating-point model to be quantized.
    from dopt.dopt_torch.opt_main import optimize_model, set_calibrate_state
    model.load_state_dict(state)  ## Load parameters for the floating-point model.
    
    ## Call **optimize_model** to quantize the model.
    quanted_model = optimize_model(model, config_path)
    
    calibrate_mode = True
    set_calibrate_state(model, calibrate_mode)
    
    for input_data, label in range(...):
        outputs = model(input_data)

def generate_params():
    model = build_torch_model()
    from dopt.dopt_torch.opt_main import generate_final_model
    generate_final_model(model,
                        config_file,
                        pth_file="quant.pth",
                        output_dir="./results_dir")

if __name__ == "__main__":
    ## step 1
    ## Modify the configuration.
    generate_config()
    
    ## step 2
    ## Train the model until it meets the requirements.
    train_model()
    ## Non-training mode
    ## calibrate_model()
 
    ## step 3
    ## Extract parameters for subsequent model deployment.
    generate_params()

TensorFlow NASEA NAS Demo

[h2]NASEA Classification Network Demo

The classification network demo is stored in tools_dopt/dopt_tf_py3/demo/nas_ea/ea_cls_imagenet and contains the following files.

blocks.so: search space file.
readme.md: search training guide.
run_release.sh: script that starts the search.
scen.yaml: configuration items.
user_module.py: customized APIs of the tool.

Procedure:

Prepare an ImageNet dataset (in TFRecord format) and change the dataset directory in scen.yaml.
For details about how to set up the environment, see Environment Setup.
Load the dependent open-source code.
1. Go to the demo directory of the classification network.
```
cd tools_dopt/dopt_tf_py3/demo/nas_ea/ea_cls_imagenet
```
2. Download the open-source code.
```
git clone https://github.com/Tensorflow/models.git
```
3. Go to the directory of the open-source code.
```
cd models
```
4. Switch to the specified version:
  - If TensorFlow 1.12.0 is used, run the following command:
```
git checkout v1.12.0
```
  - If TensorFlow 2.1.0 is used, run the following command:
```
git checkout v2.1.0
```
5. Return to the demo directory of the classification network.
```
cd ..
```
6. Set the default path of PYTHONPATH.
```
export PYTHONPATH=$PYTHONPATH:`pwd`/models/
```
Each time you open the terminal, you need to run the command again. Alternatively, add the command to the ~/.bashrc file and run source ~/.bashrc.
Configure the scen.yaml file in the demo. For details, see Search Parameter Configuration. The scen.yaml file provides recommended parameters. You can modify them as required.
Modify the user_module.py file in the demo. For details about the model port description, see TensorFlow User-Defined APIs. The user_module.py file provides recommended configurations. You can modify them as required.
Execute the run_release.sh script. Multiple model_arch_result_*.py files will be generated in results. You can select a proper network architecture for training based on the information provided in log_classification. For details about subsequent training, see readme.md.

[h2]NASEA Detection Network Demo

The detection network demo is stored in tools_dopt/dopt_tf_py3/demo/nas_ea/ea_det_coco and contains the following files.

blocks.so: search space file.
pre_train.yaml: configuration items for pre-training.
readme.md: search training guide.
run_release.sh: script that starts the search.
scen.yaml: configuration items.
user_module.py: customized APIs of the tool.

Procedure:

Prepare datasets, including the ImageNet dataset (in TFRecord format) used for pre-training and the COCO dataset (in original format) used for training. If a pre-trained CKPT file is available, you do not need to prepare the ImageNet dataset. Modify the dataset path in the scen.yaml file. For details, see Search Parameter Configuration.
For details about how to set up the environment, see Environment Setup.
Load the dependent open-source code.
1. Go to the demo directory of the detection network.
```
cd tools_dopt/dopt_tf_py3/demo/nas_ea/ea_det_coco
```
2. Download the open-source code.
```
git clone https://github.com/pierluigiferrari/ssd_keras.git
git clone https://github.com/Tensorflow/models.git
```
3. Go to the directory of the open-source code.
```
cd ssd_keras
```
4. Switch to the specified version.
```
git checkout -b v0.9.0
```
5. Return to the demo directory of the detection network.
```
cd ..
```
6. Go to the models open-source code directory.
```
cd models
```
7. Switch the models to the specified version:
  
  If TensorFlow 1.12.0 is used, run the following command:
```
git checkout v1.12.0
```
  If TensorFlow 2.1.0 is used, run the following command:
```
git checkout v2.1.0
```
8. Go to the models open-source code directory.
```
cd models
```
9. Set the default path of PYTHONPATH.
```
export PYTHONPATH=$PYTHONPATH:`pwd`/models/
```
10. Modify related open-source files by referring to Step 1 to 4 in readme.md.
Configure the scen.yaml and pre_train.yaml files in the demo. For details, see Search Parameter Configuration. The scen.yaml file provides recommended parameters. You can modify them as required.
Modify the user_module.py file in the demo. For details about the model port description, see TensorFlow User-Defined APIs. The user_module.py file provides recommended configurations. You can modify them as required.
Execute the run_release.sh script. Multiple model_arch_result_*.py files will be generated in results. You can select a proper network architecture for training based on the information provided in log_detection. For details about subsequent training, see readme.md.

[h2]NASEA Segmentation Network Demo

The segmentation network demo is stored in tools_dopt/dopt_tf_py3/demo/nas_ea/ea_seg_voc and contains the following files.

blocks.so: search space file.
pre_train.yaml: configuration items for pre-training.
readme.md: search training guide.
run_release.sh: script that starts the search.
scen.yaml: configuration items.
user_module.py: customized APIs of the tool.

Procedure:

Prepare datasets, including the ImageNet dataset (in TFRecord format) used for pre-training and the VOC dataset (in TFRecord format) used for training. If a pre-trained CKPT file is available, you do not need to prepare the ImageNet dataset. Modify the dataset path in the scen.yaml file. For details, see Search Parameter Configuration.
For details about how to set up the environment, see Environment Setup.
Load the dependent open-source code.
1. Go to the demo directory of the segmentation network.
```
cd tools_dopt/dopt_tf_py3/demo/nas_ea/ea_seg_voc
```
2. Download the open-source code.
```
git clone https://github.com/Tensorflow/models.git
```
3. Go to the directory of the open-source code.
```
cd models
```
4. Switch to the specified version.
```
git checkout v1.13.0
```
5. Return to the demo directory of the segmentation network.
```
cd ..
```
6. Set the default path of PYTHONPATH.
```
export PYTHONPATH=$PYTHONPATH:`pwd`/models/research:`pwd`/models/research/slim
```
7. If TensorFlow 2.1.0 is used, run the following command:
  1. Create models_tf2.1 and access the folder.
```
mkdir models_tf2.1
cd models_tf2.1
```
  2. Download the open-source implementation code.
```
git clone https://github.com/Tensorflow/models.git
```
  3. Go to the directory of the open-source code.
```
cd models
```
  4. Switch to the specified version.
```
git checkout v2.1.0
```
  5. Return to the models_tf2.1 folder.
```
cd ..
```
  6. Set the default path of PYTHONPATH.
```
export PYTHONPATH=$PYTHONPATH:`pwd`/models/
```
    Each time you open the terminal, you need to run the command again. Alternatively, add the command to the ~/.bashrc file and run source ~/.bashrc.
8. Modify the open-source implementation code according to the procedures specified in readme.md.
Configure the scen.yaml and pre_train.yaml files in the demo. For details, see Search Parameter Configuration. The scen.yaml file provides recommended parameters. You can modify them as required.
Modify the user_module.py file in the demo. For details about the model port description, see TensorFlow User-Defined APIs. The user_module.py file provides recommended configurations. You can modify them as required.
Execute the run_release.sh script. Multiple model_arch_result_*.py files will be generated in results. You can select a proper network architecture for training based on the information provided in log_segmentation. For details about subsequent training, see readme.md.

PyTorch NASEA NAS Demo

[h2]NASEA Classification Network Demo

The classification network demo is stored in tools_dopt/dopt_pytorch_py3/demo/nas_ea/ea_cls_imagenet_pytorch and contains the following files.

blocks.so: search space file.
readme.md: search training guide.
run_release.sh: script that starts the search.
scen.yaml: configuration items.
user_module.py: customized APIs of the tool.

Procedure:

Prepare an ImageNet dataset (in original format) and change the dataset directory in scen.yaml.
For details about how to set up the environment, see Environment Setup.
Configure the scen.yaml file in the demo. For details, see Search Parameter Configuration. The scen.yaml file provides recommended parameters. You can modify them as required.
Modify the user_module.py file in the demo. For details about the model port description, see Custom APIs for PyTorch Developers. The user_module.py file provides recommended configurations. You can modify them as required.
Execute the run_release.sh script. Multiple model_arch_result_*.py files will be generated in results. You can select a proper network architecture for training based on the information provided in log_classification. For details about subsequent training, see readme.md.

[h2]NASEA Segmentation Network Demo

The segmentation network demo is stored in tools_dopt/dopt_pytorch_py3/demo/nas_ea/ea_seg_voc_pytorch and contains the following files.

blocks.so: search space file.
pre_train.yaml: configuration items for pre-training.
readme.md: search training guide.
run_release.sh: script that starts the search.
scen.yaml: configuration items.
user_module.py: customized APIs of the tool.

Procedure:

Prepare datasets, including the ImageNet dataset (in original format) used for pre-training and the VOC dataset (in original format) used for training. If a pre-trained CKPT file is available, you do not need to prepare the ImageNet dataset. Modify the dataset path in the scen.yaml file. For details, see Search Parameter Configuration.
For details about how to set up the environment, see Environment Setup.
Load the dependent open-source code. For details, see tools_dopt/dopt_pytorch_py3/demo/nas_ea/ea_seg_voc_pytorch/readme.md.
Configure the scen.yaml and pre_train.yaml files in the demo. For details, see Search Parameter Configuration. The scen.yaml file provides recommended parameters. You can modify them as required.
Modify the user_module.py file in the demo. For details about the model port description, see Custom APIs for PyTorch Developers. The user_module.py file provides recommended configurations. You can modify them as required.
Execute the run_release.sh script. Multiple model_arch_result_*.py files will be generated in results. You can select a proper network architecture for training based on the information provided in log_segmentation. For details about subsequent training, see readme.md.

目录

Model Size Reduction Example

Demo for TensorFlow Quant_INT8-8 Quantization in Non-Training Mode

[h2]Environment Setup

[h2]Configuring the Model

[h2]Model Quantization

Demo for PyTorch Quant_INT8-8 Quantization in Non-Training Mode

[h2]Environment Setup

[h2]Configuring the Model

[h2]Model Quantization

Demo for ONNX Quant_INT8-8 Quantization in Non-Training Mode

[h2]Environment Setup

[h2]Sample Code

Demo for TensorFlow Quant_INT8-8 Plugin-based Quantization

[h2]Environment Setup

[h2]Sample Code

Demo for PyTorch Quant_INT8-8 Plugin-based Quantization

[h2]Environment Setup

[h2]Sample Code

TensorFlow NASEA NAS Demo

[h2]NASEA Classification Network Demo

[h2]NASEA Detection Network Demo

[h2]NASEA Segmentation Network Demo

PyTorch NASEA NAS Demo

[h2]NASEA Classification Network Demo

[h2]NASEA Segmentation Network Demo