\;juddlZddlZddlmZddlmZddlmZddlm Z m Z ddl m Z m Z mZmZdd lmZmZd Zd Zd ZGd dZGddZGddZy)N)IrGraph)core) quant_layers)QuantWeightPassReplaceFakeQuantDequantPass)_get_input_name_index_get_op_input_var_names_get_output_name_index$move_persistable_var_to_global_block) fuse_utilsutils.pdmodelz .pdiparamsc^ddlm}|jjjj |d<|jjjj |d<|j|jj |j|jj ||fS)Nr)fleetColumnParallelLinearRowParallelLinear)paddle.distributedr meta_parallelparallel_layers mp_layersrrappend)layer_name_mapfake_quant_input_layersrs kG:\00. PROJECTS\API\Inventory\templateJSON\kerjaOCR\Lib\site-packages\paddle/quantization/imperative/qat.pylazy_import_fleetr&s( ++55JJ ++55GG""5#6#6#H#HI""5#6#6#K#KL 2 22c NeZdZdZgddddddddddddf fd Zd Zd d ZxZS) ImperativeQuantAwarezI Applying quantization aware training (QAT) to the dgraph model. )Conv2DLinearConv2DTransposerrabs_maxmoving_average_abs_max?FNc t|||_|||||||| | | d } tdi| |_t ||| |_y)a The constructor for ImperativeQuantAware. Args: quantizable_layer_type(list[str | layer]): List the type of layers that will be quantized. Default is ['Conv2D', 'Linear']. weight_quantize_type(str): quantization type for weights, which supports 'abs_max' and 'channel_wise_abs_max'. activation_quantize_type(str): quantization type for activations, which supports 'abs_max' and 'moving_average_abs_max' now. If using 'abs_max' mode, the quantization scale will be calculated dynamically each step in both training and testing period. If using 'moving_average_abs_max', the static quantization scale will be calculated during training and used in inference. weight_bits(int): quantization bit number for weights, whereas the bias is not quantized. activation_bits(int): quantization bit number for activations. moving_rate(float): the parameter for 'moving_average_abs_max' quantization. fuse_conv_bn(bool): Whether to fuse conv and bn, default is False. weight_preprocess_layer(paddle.nn.Layer, optional): A paddle Layer that defines how to preprocess weight before quantization. Using this can quickly test if user's preprocess method works or not. The input is non-quantized weight and function returns processed weight to be quantized. If None, the weight will be quantized directly. Default is None. act_preprocess_layer(paddle.nn.Layer, optional): A paddle Layer that defines how to preprocess activation before quantization. Using this can quickly test if user's preprocess method works or not. The input is non-quantized activation and function returns processed activation to be quantized. If None, the activation will be quantized directly. Default is None. weight_quantize_layer(paddle.nn.Layer, optional): A paddle Layer that defines how to quantize weight. Using this can quickly test if user's quantization method works or not. In this layer, user should both define quantization method and dequantization method, that is, the function's input is non-quantized weight and returns dequantized weight. If None, will use uantization op defined by 'weight_quantize_type'. Default is None. act_quantize_layer(paddle.nn.Layer, optional): A paddle Layer that defines how to quantize activation. Using this can quickly test if user's quantization method works or not. In this layer, user should both define quantization method and dequantization method, that is, the function's input is non-quantized activation and returns dequantized activation. If None, will use quantization op defined by 'activation_quantize_type'. Default is None. onnx_format (bool, optional): Whether to export the quantized model with format of ONNX. Default is False. Note: If user sets attribute 'skip_quant' to a Layer that support dynamic quantization and sets it to true, the layer would not be quantized during training. If this attribute is not sets or the attribute is false, the Layer would be qunatized in training. Examples: .. code-block:: python >>> import paddle >>> from paddle.static.quantization import ( ... ImperativeQuantAware, ... ) >>> from paddle.vision.models import ( ... resnet, ... ) >>> model = resnet.resnet50(pretrained=True) >>> imperative_qat = ImperativeQuantAware( ... weight_quantize_type='abs_max', ... activation_quantize_type='moving_average_abs_max') >>> # Add the fake quant logical. >>> # The original model will be rewrite. >>> # The outscale of outputs in supportted layers would be calculated. >>> imperative_qat.quantize(model) >>> # Fine-tune the quantized model >>> # ... >>> # Save quant model for the inference. >>> imperative_qat.save_quantized_model( ... layer=model, ... model_path="./resnet50_qat", ... input_spec=[ ... paddle.static.InputSpec( ... shape=[None, 3, 224, 224], dtype='float32')]) .. code-block:: python >>> import paddle >>> from paddle.static.quantization import ( ... ImperativeQuantAware, ... ) >>> class ImperativeModel(paddle.nn.Layer): ... def __init__(self): ... super().__init__() ... # self.linear_0 would skip the quantization. ... self.linear_0 = paddle.nn.Linear(784, 400) ... self.linear_0.skip_quant = True ... # self.linear_1 would not skip the quantization. ... self.linear_1 = paddle.nn.Linear(400, 10) ... self.linear_1.skip_quant = False ... def forward(self, inputs): ... x = self.linear_0(inputs) ... x = self.linear_1(inputs) ... return x >>> model = ImperativeModel() >>> imperative_qat = ImperativeQuantAware( ... weight_quantize_type='abs_max', ... activation_quantize_type='moving_average_abs_max') >>> # Add the fake quant logical. >>> # The original model will be rewrite. >>> # >>> # There is only one Layer(self.linear1) would be added the >>> # fake quant logical. >>> imperative_qat.quantize(model) >>> # Fine-tune the quantized model >>> # ... >>> # Save quant model for the inference. >>> imperative_qat.save_quantized_model( ... layer=model, ... model_path="./imperative_model_qat") ) quantizable_layer_typeweight_quantize_typeactivation_quantize_type weight_bitsactivation_bits moving_rateweight_preprocess_layeract_preprocess_layerweight_quantize_layeract_quantize_layerN)super__init__ fuse_conv_bnImperativeQuantizeInputs_quantize_inputsImperativeQuantizeOutputs_quantize_outputs)selfr)r*r+r,r-r.r6r/r0r1r2 onnx_formatkwargs __class__s rr5zImperativeQuantAware.__init__9sfz ('=$8(@&.&'>$8%:"4  !9 B6 B!: +" rc t|tjjsJd|jrt j||j j||jj||S)a According to weights' and activations' quantization types, the model will be added some fake quant ops, such as fake_quantize_dequantize_moving_average_abs_max, fake_quantize_dequantize_abs_max and so on. At the same time, the out_scale value of outputs would be calculated. Args: model(paddle.nn.Layer): the model to be quantized. Returns: None Examples: .. code-block:: python >>> import paddle >>> from paddle.static.quantization import ( ... ImperativeQuantAware, ... ) >>> class ImperativeModel(paddle.nn.Layer): ... def __init__(self): ... super().__init__() ... # self.linear_0 would skip the quantization. ... self.linear_0 = paddle.nn.Linear(784, 400) ... self.linear_0.skip_quant = True ... # self.linear_1 would not skip the quantization. ... self.linear_1 = paddle.nn.Linear(400, 10) ... self.linear_1.skip_quant = False ... def forward(self, inputs): ... x = self.linear_0(inputs) ... x = self.linear_1(inputs) ... return x >>> model = ImperativeModel() >>> imperative_qat = ImperativeQuantAware( ... weight_quantize_type='abs_max', ... activation_quantize_type='moving_average_abs_max') >>> # Add the fake quant logical. >>> # The original model will be rewrite. >>> # >>> # There is only one Layer(self.linear1) would be added the >>> # fake quant logical. >>> imperative_qat.quantize(model) 2The model must be the instance of paddle.nn.Layer.) isinstancepaddlennLayerr6rr8applyr:)r;models rquantizezImperativeQuantAware.quantizestb 699??  @ ? @      # #E * ##E* $$U+ rc B|jj|||fi|yN)r:save_quantized_model)r;layerpath input_specconfigs rrJz)ImperativeQuantAware.save_quantized_model(s'333 4 '- rrI)__name__ __module__ __qualname____doc__r5rGrJ __classcell__r>s@rr r 4sB  '!9 $!"'q f:x rr c HeZdZdZgddddddddddf fd Zd Zd ZxZS) r7z Based on the input params, add the quant_dequant computational logic both for activation inputs and weight inputs. )r!r"r#r$r%r&r'Nc ftttjtj \__t fd|D_jD]*} t| ts| j vr"Jd| zhd} ddh} |dk7r|| vs Jd|z|| vs Jd|zd}||sJd ||sJd d }||sJd ||sJd || sJd|| sJd|||||||| | d _ y)zz The constructor for ImperativeQuantizeInputs. Please refer to the args of ImperativeQuantAware. c3\K|]#}|jvrj|n|%ywrI)r).0rKr;s r z4ImperativeQuantizeInputs.__init__..Ks@- 0+++    & 0s),z!%s is unspported to be quantized.>r$ lsq_weightchannel_wise_abs_maxr%channel_wise_lsq_weightr%lsq_actzUUnsupported weight_quantize_type: %s. It can only be abs_max or channel_wise_abs_max.z_Unsupported activation_quantize_type: %s. It can only be moving_average_abs_max or lsq_act now.c>t|txr |dk\xr|dkS)Nr)rAint)bitss rz3ImperativeQuantizeInputs.__init__..ns D#.K419KKrz%weight_bits should be 1, 2,... or 16.z)activation_bits should be 1, 2,... or 16.cV|duxs$t|tjjSrI) issubclassrBrCrD)methods rrbz3ImperativeQuantizeInputs.__init__..us&Vt^& z FIIOO8 & rz%weight_preprocess should be nn.Layer.z"act_preprocess should be nn.Layer.z#weight_quantize should be nn.Layer.z act_quantize should be nn.Layer.) r*r+r,r-r.weight_pre_layer act_pre_layerweight_quant_layeract_quant_layerN) r4r5rrrrtuple_quantizable_layer_typerAstr_kwargs)r;r)r*r+r,r-r.r/r0r1r2rK quantize_typeact_quantize_type bits_check layer_checkr>s` rr5z!ImperativeQuantizeInputs.__init__4s$  Nrr7cJeZdZdZd fd ZdZd dZdZdZdZ dZ xZ S) r9z8 Calculate the output scales for target layers. cLt|||_||_||_y)a4 The constructor for ImperativeQuantizeOutputs. Args: moving_rate(float): The decay coefficient of moving average. The default value is 0.9. activation_bits(int, optional): quantization bit number for activation. Default is 8. N)r4r5 _moving_rate_activation_bits _onnx_format)r;r.r-r<r>s rr5z"ImperativeQuantizeOutputs.__init__s( ' /'rct|tjjsJd|j D]\}}d|vr |j |st j||\}}d}t|tt jr#tj||j|}n"tj||j|}t|||y)aB Insert the `moving_average_abs_max_scale` layers to calculate the output scales for specific layers in the dygraph model. Args: model(paddle.nn.Layer): The target model which would be calculate the output quantization scale. Returns: None r@_act_preprocessN) reduce_type)rArBrCrDrt_is_target_layerrrvrjfake_quant_output_layersrFakeQuantMAOutputScaleLayerrMAOutputScaleLayerrx)r;rFcur_namerzr{r|rr}s rrEzImperativeQuantizeOutputs.applys 699??  @ ? @ $)#8#8#: Hi H,((3%*%I%Ix& "L(K)U5+I+I%JK"."J"Jt00k##/"A"At00k# L(O <+$;rc t|tjjsJd|r!tjj ||tjj d|||d|d}tjrd}tjtj}tjj}tjj|}tjj!|} tjj#|} | t$z} | t&z} tjj)| || | \} }}|j*s|j-| ||t/tj0| j2d}|j5D]L}|j7D]'}|j9dk(s|j;|)|j=N|j?} |jA| d}nt/tj0| j2d}tC|||jD }|j5D]}d|_#|jI|tK||}|j5D]}d|_#|jI||j?} d}tM| d }| d }n)| jOd r| jQd dd}n| }tjjS| |}|Dcgc]!}| jUjW|#}}tjjY||||| j[||rtj\y y cc}w)a Save the quantized model for the inference. Args: model (Layer): The model to be saved. path (str): The path prefix to save model. The format is ``dirname/file_prefix`` or ``file_prefix``. input_spec (list[InputSpec|Tensor], optional): Describes the input of the saved model's forward method, which can be described by InputSpec or example Tensor. If None, all input variables of the original Layer's forward method would be the inputs of the saved model. Default None. **config (dict, optional): Other save configuration options for compatibility. We do not recommend using these configurations, they may be removed in the future. If not necessary, DO NOT use them. Default None. The following options are currently supported: (1) output_spec (list[Tensor]): Selects the output targets of the saved model. By default, all return variables of original Layer's forward method are kept as the output of the saved model. If the provided ``output_spec`` list is not all output variables, the saved model will be pruned according to the given ``output_spec`` list. Returns: None r@)rM)rKrLrMFT)executormodel_filenameparams_filename)for_testmoving_average_abs_max_scale) quant_bitsNrFr.r r)rprogram clip_extrar3)/rArBrCrDjit to_staticsavein_dynamic_mode enable_staticrCPUPlacestatic global_scopeExecutorosrLdirnamebasenameINFER_MODEL_SUFFIXINFER_PARAMS_SUFFIXload_inference_modelr_gather_scalesrGraphdescall_sub_graphs all_op_nodesrysafe_remove_nodesresolve_hazard to_program_set_skip_quant_attrrr _for_testrErr endswithrsplitjoin global_blockvarsave_inference_modelclonedisable_static)r;rFrLrMrNis_dynamic_modeplacescopeexerrrr infer_programfeed_target_names fetch_targetsgraph sub_graph_oprtransform_passquant_weight_pass model_name path_prefixry feed_varss rrJz.ImperativeQuantizeOutputs.save_quantized_modelsU8 699??  @ ? @   JJ : > Pe$:PP  ! ! #"O  "  **,mm$$U+''//$'77##D)!$66"%88 MM . . )+ /         um DDJJ}'9'9:UKE"113 $113Cxxz%CC!33C84((* 4 ",,.M  % %m 4JDJJ}'9'9:UKE8u)>)>N#113 &* #$$Y/4!0u = "113 &* #!'' 24",,.MJ,];  ! J  $ $Z 0'..sA6q9J'Jggll7J7 ?P ?PtM & & ( , ,T 2?P    **   !'')! +    ! ! #  s &N>ct|tjjsy|jr't|t t jrdSdSd}t j|r%t|t t jsd}t|t t jrd}t|tjjjrd}|S)zE Whether the layer needs to calculate output scales. FT) rArBrCrDrrjrfake_quant_wrap_layers is_leaf_layerfake_quant_leaf_layersquantFloatFunctionalLayer)r;rKflags rrz*ImperativeQuantizeOutputs._is_target_layeros %1   eU5+G+G%HI      u %j 5556/ D eU5#?#?@ AD eVYY__AA BD rc@fd}fd}||y)z Get all scales from fake ops, save them into the corresponding ops and delete all moving_average_abs_max_scale ops. chg}tjdgz} jD]3}|jD]"}|j|vs|j |$5|D]}t |D]}tj|j|}|&d|jvs|jdk(sD|jdd}tj |}tj|}t||\}} |j|t| zdz||jddy)Nrquantize_dequantizeOutScaler _thresholdwith_quant_attrT)r!fake_quantize_dequantize_op_typesblocksopstyperr find_previous_opblockoutputload_variable_datafp_numpy_to_naiver _set_attrrl) target_opsskip_opsrop in_var_name previous_op scale_namein_scaleargnameindexrrs r_gather_input_scalezEImperativeQuantizeOutputs._gather_scales.._gather_input_scales$J>>.BH!))Bwwh."))"-$( !#:2#>K"'"8"8;"OK".-1A1AA&++/MM%0%7%7 %CA%F #(#;#;E:#N#(#:#:8#D)>r;)O #c%j0<? %6=$?!rc`g}jD]4}|jD]#}|jdk(s|j|%6|D]b}|j dd}|j dd}|j }tj||}tj||}|j dd}tj|}tj|}|jdk7rXt||} | J| \} } |j| t| zdz||jd||jd d |D]T} | j||t!t#D])} | j$|k(s|j'|| <+Vey) NrXrOutrfeedr out_thresholdrT)rrrrinputrrrr find_next_opsrrr rrl _rename_inputrangelenryr)rrrr out_var_namernext_opsout_scale_name out_scaleresrrnext_opirrrs r_gather_output_scalezFImperativeQuantizeOutputs._gather_scales.._gather_output_scalesJ ))Bww"@@"))"-$( ! hhsmA. !yy/2 #44UKH  ..ulC!#:!6q!9!44UNK !33I> ##v-0kJC),#--#c%j0<?$--oyI#--.?F'G)), D#3}#56(+00L@/4yy/EM!,7 (+!rNr3)r;rrrrrs ``` rrz(ImperativeQuantizeOutputs._gather_scaless  >6" FH rc|jD]J}|jD]9}|j||s|jdd|jdd;Ly)z/ Label the skip quantized ops. rsTrN)rr_is_skip_quant_opr)r;rrrs rrz.ImperativeQuantizeOutputs._set_skip_quant_attrsM^^Eii))%4LLt4LL!2D9 $rcgd}|j|vry|jDcgc]}tj||}}t d|DScc}w)z The input op should be skipped quantization. 1. the type of input op should be conv2d, depthwise_conv2d or matmul 2. the previous ops of the input op are not fake_quantize_dequantize ops )conv2ddepthwise_conv2dmatmulconv2d_transposeFc3`K|]&}|duxr|jtjv(ywrI)rrr)rXrs rrYz>ImperativeQuantizeOutputs._is_skip_quant_op..s< # dN GuFFF G"s,.)rinput_arg_namesrrany)r;rin_optarget_op_typesarg_name previous_opss rrz+ImperativeQuantizeOutputs._is_skip_quant_opsr   ::_ ,"11 1  " "5( 31   #    sA)r'r&FrI) rOrPrQrRr5rErJrrrrrSrTs@rr9r9s0 (%=Nu$n:FP: rr9)rrBpaddle.base.frameworkrpaddle.frameworkrpaddle.nn.quantr%static.quantization.quantization_passrrstatic.quantization.utilsr r r r rrrrrr r7r9r3rrrsc )!(  " 3w w tMNMN`w w r