\;jddlZddlmZddlmZddlZddlmZm Z ddl m Z ddl m Z ddlmZddlmZmZdd lmZd d lmZGd d eZdZGddZGddeZy)N) defaultdict)Enum)_C_ops _legacy_C_ops)core) check_type) to_variable)_dygraph_tracer dygraph_only)in_dynamic_mode)amp_global_stateceZdZdZdZdZy)OptimizerStaterr N)__name__ __module__ __qualname__INITUNSCALEDSTEPPED_G:\00. PROJECTS\API\Inventory\templateJSON\kerjaOCR\Lib\site-packages\paddle/amp/grad_scaler.pyrrs DHGrrc&dtjiS)Nstate)rrrrr_refresh_optimizer_stater%s ^(( ))rceZdZdZe ddZdZdZdZdZ dZ dZ d Z d Z d Zd Zd ZdZdZdZdZdZdZdZy) AmpScalera AmpScaler is used for Auto-Mixed-Precision training/inferring in imperative mode. It controls the scaling of loss, helps avoiding numerical overflow. The object of this class has seventeen methods `scale()`, `unscale_()`, `minimize()` and `get`/`set` api of parameters. `scale()` is used to multiply the loss by a scale ratio. `unscale_()` is used to unscale the gradients of parameters, multiplies the gradients of parameters by 1/(scale ratio) `minimize()` is similar as `optimizer.minimize()`, performs parameters updating, and it will update the loss_scaling. Commonly, it is used together with `amp_guard` to achieve Auto-Mixed-Precision in imperative mode. Args: enable(bool, optional): Enable loss scaling or not. Default is True. init_loss_scaling (float, optional): The initial loss scaling factor. Default is 2**15. incr_ratio(float, optional): The multiplier to use when increasing the loss scaling. Default is 2.0. decr_ratio(float, optional): The less-than-one-multiplier to use when decreasing the loss scaling. Default is 0.5. incr_every_n_steps(int, optional): Increases loss scaling every n consecutive steps with finite gradients. Default is 1000. decr_every_n_nan_or_inf(int, optional): Decreases loss scaling every n accumulated steps with nan or inf gradients. Default is 2. use_dynamic_loss_scaling(bool, optional): Whether to use dynamic loss scaling. If False, fixed loss_scaling is used. If True, the loss scaling is updated dynamicly. Default is True. Returns: An AmpScaler object. Examples: .. code-block:: python >>> import numpy as np >>> import paddle >>> data = np.random.uniform(-1, 1, [10, 3, 32, 32]).astype('float32') >>> model = paddle.nn.Conv2D(3, 2, 3) >>> optimizer = paddle.optimizer.SGD( ... learning_rate=0.01, parameters=model.parameters()) >>> scaler = paddle.amp.AmpScaler(init_loss_scaling=1024) >>> data = paddle.to_tensor(data) >>> with paddle.amp.amp_guard(): ... conv = model(data) ... loss = paddle.mean(conv) ... scaled = scaler.scale(loss) ... scaled.backward() ... scaler.minimize(optimizer, scaled) c:t}|s td|rr|jjsX|jj s>|jj s$t jd|jzd}||_|jr|dkDsJd|dksJd||_ ||_ ||_ ||_ ||_ d|_d|_||_t#t%j&dgj)t$j*|_t#t%j&dgj)t$j*|_t#t%j&dgj)t$j*|_t#t%j&dgj)t$j*|_t#t%j&dgj)t$j*|_t#t%j&|jgj)t$j6|_d|_t=t>|_ yy)Nz;current_tracer is None, maybe it is not in imperative mode.zqAmpScaler can only be enabled on CUDAPlace, XPUPlace and CustomPlace, current place is %s, so it makes no effect.F?zThe incr_ratio must be > 1.0.zThe decr_ratio must be < 1.0.r)!r ValueError_expected_place is_gpu_place is_xpu_placeis_custom_placewarningswarn_enable_init_loss_scaling _incr_ratio _decr_ratio_incr_every_n_steps_decr_every_n_nan_or_inf _incr_count _decr_count_use_dynamic_loss_scalingr nparrayastypebool_ _found_inf_temp_found_inf_value_false_temp_found_inf_fp16_temp_found_inf_bf16_temp_found_inf_fp32float32_scale_cache_founf_infrr_optimizer_states) selfenableinit_loss_scaling incr_ratio decr_ratioincr_every_n_stepsdecr_every_n_nan_or_infuse_dynamic_loss_scalingtracers r__init__zAmpScaler.__init__Zs!"M    " " / / 1%%224%%557 MMD(() F <<# D%D D## D%D D#&7D #)D )D '9D $,CD ) D  D -ED *)"((A3-*>*>rxx*HIDO/:! $$RXX.0D ,)4! $$RXX.)D %)4! $$RXX.)D %)4! $$RXX.)D %&$1123::2::FDK%)D !%01I%JD "= rcVt|dtjjd|jr[t j dk7rD|jr8d|_d|_tjdt j z|js|S||jzS)al Multiplies a Tensor by the scale factor and returns scaled outputs. If this instance of :class:`AmpScaler` is not enabled, output are returned unmodified. Args: var (Tensor): The Tensor to scale. Returns: The scaled Tensor or original Tensor. Examples: .. code-block:: python >>> import numpy as np >>> import paddle >>> data = np.random.uniform(-1, 1, [10, 3, 32, 32]).astype('float32') >>> model = paddle.nn.Conv2D(3, 2, 3) >>> optimizer = paddle.optimizer.SGD( ... learning_rate=0.01, parameters=model.parameters()) >>> scaler = paddle.amp.AmpScaler(init_loss_scaling=1024) >>> data = paddle.to_tensor(data) >>> with paddle.amp.amp_guard(): ... conv = model(data) ... loss = paddle.mean(conv) ... scaled = scaler.scale(loss) ... scaled.backward() ... scaler.minimize(optimizer, scaled) varzAmpScaler.scale()float16Fz^It is not recommended to use dynamic loss scaling for %s, so GradScaler is disable by default.) rreagerTensorr)r amp_dtyper1r'r(r<)r?rJs rscalezAmpScaler.scales< 3tzz002EF LL ",, 9.. DL-2D * MMp#%//1  ||JT[[  rc4|js|j|i|S|jt|}|dtj ur|j |d\}}t|drH|jd|j|j|i|\}}|jd|_ n0|jrd|_ n|j|i|\}}d|_ |jr|jtt|_||fS)a This function is similar as `Optimizer.minimize()`, which performs parameters updating. If the scaled gradients of parameters contains NAN or INF, the parameters updating is skipped. Otherwise, if `unscale_()` has not been called, it first unscales the scaled gradients of parameters, then updates the parameters. Finally, the loss scaling ratio is updated. Args: optimizer(Optimizer): The optimizer used to update parameters. args: Arguments, which will be forward to `optimizer.minimize()`. kwargs: Keyword arguments, which will be forward to `Optimizer.minimize()`. Examples: .. code-block:: python >>> import numpy as np >>> import paddle >>> data = np.random.uniform(-1, 1, [10, 3, 32, 32]).astype('float32') >>> model = paddle.nn.Conv2D(3, 2, 3) >>> optimizer = paddle.optimizer.SGD( ... learning_rate=0.01, ... parameters=model.parameters() ... ) >>> scaler = paddle.amp.AmpScaler(init_loss_scaling=1024) >>> data = paddle.to_tensor(data) >>> with paddle.amp.amp_guard(): ... conv = model(data) ... loss = paddle.mean(conv) ... scaled = scaler.scale(loss) ... scaled.backward() ... scaler.minimize(optimizer, scaled) r)NN_set_auxiliary_var found_infTF)r)minimizer>idrr_unscalehasattrrQr6_get_auxiliary_varr=r1_updaterr)r? optimizerargskwargsoptimizer_state optimize_ops params_gradss rrSzAmpScaler.minimizesH||%9%%t6v6 600I? 7 #~':': : MM) $%1" l 92 3  ( (doo F);););T)LV)L &L,$-$@$@$MD !(,%-?Y-?-?-P-P* l(-%  ) ) LLN!,-E!F\))rc|jsy|jt|}|dtjur t d|dtj ur t dt|ddrTt|jdtr6g}g}g}g}|jD]}|dD]}|j|j|j|jjtjj j"k(r |j|j|jjtjj j$k(r |j|j|j|jn/t'r.tj(j+|j,\}}}n|j,Dcgc]"}|j|j$}}|Dcgc]5}|jtjj j"k(r|7}}|Dcgc]5}|jtjj j$k(r|7}}|Dcgc]5}|jtjj j.k(r|7}}|j0|_t5|r[t7j8||j:||j<t?j@|j2|j<|_t5|r[t7j8||j:||jBt?j@|j2|jB|_t5|r[t7j8||j:||jDt?j@|j2|jD|_tj|d<ycc}wcc}wcc}wcc}w)a Unscale the gradients of parameters, multiplies the gradients of parameters by 1/(loss scaling ratio). If this instance of :class:`GradScaler` is not enabled, output are returned unmodified. Args: optimizer(Optimizer): The optimizer used to update parameters. Returns: The unscaled parameters or original parameters. NrzMunscale_() has already been called on this optimizer since the last update().z(unscale_() is being called after step(). _param_groupsrparams)#r)r>rTrr RuntimeErrorrgetattr isinstancer`dict _grad_ivarappenddtyperVarDescVarTypeFP16BF16r rLget_grads_lists_parameter_listFP32r7r6lenrcheck_finite_and_unscaler<r8r bitwise_orr9r:) r?rYr\ param_gradsparam_grads_fp16param_grads_bf16param_grads_fp32groupparams rrUzAmpScaler._unscale s|| 00I? 7 #~'>'> >_ W %)?)? ?IJ J 9ot 4  # #A &: K! ! ! "00"8_E'')5#**5+;+;+=>!,,.44#||33889-33E4D4D4FG!,,.44#||33889-33E4D4D4FG,33E4D4D4FG-1" JJ..y/H/HI $$$"+!:!:!:'')5$$&!:"-$!,{{dll&:&:&?&??!,!$"-$!,{{dll&:&:&?&??!,!$"-$!,{{dll&:&:&?&??!,!$ ::   2 2   ))   %//!:!:DO   2 2   ))   %//!:!:DO   2 2   ))   %//!:!:DO$2#:#: i $ $ $s'Q;:Q;:Q;:Qc F|jsy|jrd|_|jdz|_|j|jk(rzt dj t|jt|jt|j|j|jz|_d|_yd|_|jdz|_|j|jk(r%|j|jz|_d|_y)z+ Updates the loss_scaling. Nrr z:Found inf or nan, current scale is: {}, decrease to: {}*{}) r)r=r/r0r.printformatfloatr<r,r-r+r?s rrXzAmpScaler._updateys||    D #//!3D 4#@#@@PWWdkk*dkk*d../#kkD,<,<< #$   !D #//!3D 4#;#;;"kkD,<,<< #$ rc|jS)z Enable loss scaling or not. Returns: bool: enable loss scaling return True else return False. )r)r}s r is_enablezAmpScaler.is_enables||rc|jS)z Whether to use dynamic loss scaling. Returns: bool: if fixed loss_scaling is used return False, if the loss scaling is updated dynamicly return true. )r1r}s ris_use_dynamic_loss_scalingz%AmpScaler.is_use_dynamic_loss_scalings---rc|jS)z Return the initial loss scaling factor. Reurns: float: the initial loss scaling factor. )r*r}s rget_init_loss_scalingzAmpScaler.get_init_loss_scalings&&&rc||_ttj|jgj tj |_y)z Set the initial loss scaling factor by `new_init_loss_scaling`. Args: new_init_loss_scaling(int): The new_init_loss_scaling used to update initial loss scaling factor.s N)r*r r2r3r4r;r<)r?new_init_loss_scalings rset_init_loss_scalingzAmpScaler.set_init_loss_scalings<#8! HHd--. / 6 6rzz B  rc|jS)z Return the multiplier to use when increasing the loss scaling. Reurns: float: the multiplier to use when increasing the loss scaling. r+r}s rget_incr_ratiozAmpScaler.get_incr_ratiorc*|dkDsJd||_y)a  Set the multiplier to use when increasing the loss scaling by `new_incr_ratio`, `new_incr_ratio` should > 1.0. Args: new_incr_ratio(float): The new_incr_ratio used to update the multiplier to use when increasing the loss scaling. r!z!The new_incr_ratio must be > 1.0.Nr)r?new_incr_ratios rset_incr_ratiozAmpScaler.set_incr_ratio #H%HH#)rc|jS)z Get the less-than-one-multiplier to use when decreasing the loss scaling. Reurns: float: the less-than-one-multiplier to use when decreasing the loss scaling. r,r}s rget_decr_ratiozAmpScaler.get_decr_ratiorrc*|dksJd||_y)a) Set the less-than-one-multiplier to use when decreasing the loss scaling by `new_incr_ratio`, `new_decr_ratio` should < 1.0. Args: new_decr_ratio(float): The new_decr_ratio used to update the less-than-one-multiplier to use when decreasing the loss scaling. r!z!The new_decr_ratio must be < 1.0.Nr)r?new_decr_ratios rset_decr_ratiozAmpScaler.set_decr_ratiorrc|jS)a Return the num `n`, `n` represent increases loss scaling every `n` consecutive steps with finite gradients. Reurns: int: the num `n`, `n` represent increases loss scaling every `n` consecutive steps with finite gradients. r-r}s rget_incr_every_n_stepsz AmpScaler.get_incr_every_n_stepss'''rc||_y)a^ Set the num `n` by `new_incr_every_n_steps`, `n` represent increases loss scaling every `n` consecutive steps with finite gradients. Args: new_incr_every_n_steps(int): The new_incr_every_n_steps used to update the num `n`, `n` represent increases loss scaling every `n` consecutive steps with finite gradients. Nr)r?new_incr_every_n_stepss rset_incr_every_n_stepsz AmpScaler.set_incr_every_n_stepss $: rc|jS)a  Return the num `n`, `n` represent decreases loss scaling every `n` accumulated steps with nan or inf gradients. Reurns: int: the num `n`, `n` represent decreases loss scaling every `n` accumulated steps with nan or inf gradients. r.r}s rget_decr_every_n_nan_or_infz%AmpScaler.get_decr_every_n_nan_or_infs,,,rc||_y)au Set the num `n` by `new_decr_every_n_nan_or_inf`, `n` represent decreases loss scaling every `n` accumulated steps with nan or inf gradients. Args: new_decr_every_n_nan_or_inf(int): The new_decr_every_n_nan_or_inf used to update the num `n`, `n` represent decreases loss scaling every `n` accumulated steps with nan or inf gradients. Nr)r?new_decr_every_n_nan_or_infs rset_decr_every_n_nan_or_infz%AmpScaler.set_decr_every_n_nan_or_infs )D%rc |jri|jj|j|j|j |j |j|j|jdSiS)a Returns the state of the scaler as a `dict`, If this instance is not enabled, returns an empty dict. Reurns: A dict of scaler includes: scale (tensor): The loss scaling factor. incr_ratio(float): The multiplier to use when increasing the loss scaling. decr_ratio(float): The less-than-one-multiplier to use when decreasing the loss scaling. incr_every_n_steps(int): Increases loss scaling every n consecutive steps with finite gradients. decr_every_n_nan_or_inf(int): Decreases loss scaling every n accumulated steps with nan or inf gradients. incr_count(int): The number of recent consecutive unskipped steps. decr_count(int): The number of recent consecutive skipped steps. use_dynamic_loss_scaling(bool): Whether to use dynamic loss scaling. If False, fixed loss_scaling is used. If True, the loss scaling is updated dynamicly. Default is True. )rOrBrCrDrE incr_count decr_countrF) r)r<numpyr+r,r-r.r/r0r1r}s r state_dictzAmpScaler.state_dictsp4||**,".."..&*&>&>+/+H+H".."..,0,J,J   rc|jsyt|dk(r td|dd|_t t j |jgjt j|_ |d|_ |d|_ |d|_ |d|_ |d |_|d |_|d |_y) z Loads the scaler state. Args: state_dict(dict): scaler state. Should be an object returned from a call to `AmpScaler.state_dict()`. NrzdThe input state dict is empty, possibly because it was saved from a disabled instance of GradScaler.rOrBrCrDrErrrF)r)rprbr*r r2r3r4r;r<r+r,r-r.r/r0r1)r?rs rload_state_dictzAmpScaler.load_state_dict%s||  z?a :  #-W"5a"8! HHd--. / 6 6rzz B  &l3%l3#-.B#C (23L(M%%l3%l3)34N)O&rN)Tg@@?ir T)rrr__doc__r rHrOrSrUrXrrrrrrrrrrrrrrrrrrr)s.`! !!%;K;Kz/!b@*Dl;\:.'   * *(:-D <PrrceZdZdZ dfd ZfdZfdZdZdZfdZ fdZ fd Z fd Z fd Z fd Zfd ZfdZfdZfdZfdZfdZfdZfdZfdZxZS) GradScalerak GradScaler is used for Auto-Mixed-Precision training in dynamic graph mode. It controls the scaling of loss, helps avoiding numerical overflow. The object of this class has nineteen methods `scale()`, `unscale_()`, `minimize()`, `step()`, `update()` and `get`/`set` api of parameters. `scale()` is used to multiply the loss by a scale ratio. `unscale_()` is used to unscale the gradients of parameters, multiplies the gradients of parameters by 1/(scale ratio) `minimize()` is similar as `optimizer.minimize()`, performs parameters updating, and it will update the loss_scaling, it equal to `step()` + `update()`. `step()` is similar as `optimizer.step()`, which performs parameters updating. `update` is used to update the loss_scaling. Commonly, it is used together with `paddle.amp.auto_cast` to achieve Auto-Mixed-Precision in dynamic graph mode. Args: enable(bool, optional): Enable loss scaling or not. Default is True. init_loss_scaling (float, optional): The initial loss scaling factor. Default is 65536.0. incr_ratio(float, optional): The multiplier to use when increasing the loss scaling. Default is 2.0. decr_ratio(float, optional): The less-than-one-multiplier to use when decreasing the loss scaling. Default is 0.5. incr_every_n_steps(int, optional): Increases loss scaling every n consecutive steps with finite gradients. Default is 2000. decr_every_n_nan_or_inf(int, optional): Decreases loss scaling every n accumulated steps with nan or inf gradients. Default is 1. use_dynamic_loss_scaling(bool, optional): Whether to use dynamic loss scaling. If False, fixed loss_scaling is used. If True, the loss scaling is updated dynamicly. Default is True. Returns: An GradScaler object. Examples: .. code-block:: python >>> import paddle >>> model = paddle.nn.Conv2D(3, 2, 3, bias_attr=True) >>> optimizer = paddle.optimizer.SGD(learning_rate=0.01, parameters=model.parameters()) >>> scaler = paddle.amp.GradScaler(init_loss_scaling=1024) >>> data = paddle.rand([10, 3, 32, 32]) >>> with paddle.amp.auto_cast(): ... conv = model(data) ... loss = paddle.mean(conv) >>> scaled = scaler.scale(loss) # scale the loss >>> scaled.backward() # do backward >>> scaler.minimize(optimizer, scaled) # update parameters >>> optimizer.clear_grad() c 0t||||||||y)N)superrH) r?r@rArBrCrDrErF __class__s rrHzGradScaler.__init__vs'       # $ rc"t||S)aJ Multiplies a Tensor by the scale factor and returns scaled outputs. If this instance of :class:`GradScaler` is not enabled, output are returned unmodified. Args: var (Tensor): The tensor to scale. Returns: The scaled tensor or original tensor. Examples: .. code-block:: python >>> import paddle >>> model = paddle.nn.Conv2D(3, 2, 3, bias_attr=True) >>> optimizer = paddle.optimizer.SGD(learning_rate=0.01, parameters=model.parameters()) >>> scaler = paddle.amp.GradScaler(init_loss_scaling=1024) >>> data = paddle.rand([10, 3, 32, 32]) >>> with paddle.amp.auto_cast(): ... conv = model(data) ... loss = paddle.mean(conv) >>> scaled = scaler.scale(loss) # scale the loss >>> scaled.backward() # do backward >>> scaler.minimize(optimizer, scaled) # update parameters >>> optimizer.clear_grad() )rrO)r?rJrs rrOzGradScaler.scales<w}S!!rc*t||g|i|S)a This function is similar as `optimizer.minimize()`, which performs parameters updating. If the scaled gradients of parameters contains NAN or INF, the parameters updating is skipped. Otherwise, if `unscale_()` has not been called, it first unscales the scaled gradients of parameters, then updates the parameters. Finally, the loss scaling ratio is updated. Args: optimizer(Optimizer): The optimizer used to update parameters. args: Arguments, which will be forward to `optimizer.minimize()`. kwargs: Keyword arguments, which will be forward to `optimizer.minimize()`. Examples: .. code-block:: python >>> import paddle >>> model = paddle.nn.Conv2D(3, 2, 3, bias_attr=True) >>> optimizer = paddle.optimizer.SGD(learning_rate=0.01, parameters=model.parameters()) >>> scaler = paddle.amp.GradScaler(init_loss_scaling=1024) >>> data = paddle.rand([10, 3, 32, 32]) >>> with paddle.amp.auto_cast(): ... conv = model(data) ... loss = paddle.mean(conv) >>> scaled = scaler.scale(loss) # scale the loss >>> scaled.backward() # do backward >>> scaler.minimize(optimizer, scaled) # update parameters >>> optimizer.clear_grad() )rrS)r?rYrZr[rs rrSzGradScaler.minimizes Dw ;D;F;;rcT|js|jS|jt|}|dtj ur t d|dtjur|j|t|drC|jd|j|j|jd|_ n+|jrd|_ n|jd|_ tj |d<|jstt |_yy)at This function is similar as `optimizer.step()`, which performs parameters updating. If the scaled gradients of parameters contains NAN or INF, the parameters updating is skipped. Otherwise, if `unscale_()` has not been called, it first unscales the scaled gradients of parameters, then updates the parameters. Args: optimizer(Optimizer): The optimizer used to update parameters. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU) >>> import paddle >>> paddle.device.set_device('gpu') >>> model = paddle.nn.Conv2D(3, 2, 3, bias_attr=True) >>> optimizer = paddle.optimizer.SGD(learning_rate=0.01, parameters=model.parameters()) >>> scaler = paddle.amp.GradScaler(init_loss_scaling=1024) >>> data = paddle.rand([10, 3, 32, 32]) >>> with paddle.amp.auto_cast(): ... conv = model(data) ... loss = paddle.mean(conv) >>> scaled = scaler.scale(loss) # scale the loss >>> scaled.backward() # do backward >>> scaler.step(optimizer) # update parameters >>> scaler.update() # update the loss scaling ratio >>> optimizer.clear_grad() rz7step() has already been called since the last update().rQrRTFN)r)stepr>rTrrrbrrUrVrQr6rWr=r1rr)r?rYr\s rrzGradScaler.steps>||>># #00I? 7 #~'='= =I  7 #~':': : MM) $ 92 3  ( (doo F NN $-$@$@$MD !(,% (-%#1#9#9 --%01I%JD ".rc~|jsy|jr$|jtt|_y)a Updates the loss_scaling. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU) >>> import paddle >>> paddle.device.set_device('gpu') >>> model = paddle.nn.Conv2D(3, 2, 3, bias_attr=True) >>> optimizer = paddle.optimizer.SGD(learning_rate=0.01, parameters=model.parameters()) >>> scaler = paddle.amp.GradScaler(init_loss_scaling=1024) >>> data = paddle.rand([10, 3, 32, 32]) >>> with paddle.amp.auto_cast(): ... conv = model(data) ... loss = paddle.mean(conv) >>> scaled = scaler.scale(loss) # scale the loss >>> scaled.backward() # do backward >>> scaler.step(optimizer) # update parameters >>> scaler.update() # update the loss scaling ratio >>> optimizer.clear_grad() N)r)r1rXrrr>r}s rupdatezGradScaler.update s12||   ) ) LLN%01I%JD "rc"t||S)aE Unscale the gradients of parameters, multiplies the gradients of parameters by 1/(loss scaling ratio). If this instance of :class:`GradScaler` is not enabled, output are returned unmodified. Args: optimizer(Optimizer): The optimizer used to update parameters. Returns: The unscaled parameters or original parameters. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU) >>> import paddle >>> paddle.device.set_device('gpu') >>> model = paddle.nn.Conv2D(3, 2, 3, bias_attr=True) >>> optimizer = paddle.optimizer.SGD(learning_rate=0.01, parameters=model.parameters()) >>> scaler = paddle.amp.GradScaler(init_loss_scaling=1024) >>> data = paddle.rand([10, 3, 32, 32]) >>> with paddle.amp.auto_cast(): ... conv = model(data) ... loss = paddle.mean(conv) >>> scaled = scaler.scale(loss) # scale the loss >>> scaled.backward() # do backward >>> scaler.unscale_(optimizer) # unscale the parameter >>> scaler.step(optimizer) >>> scaler.update() >>> optimizer.clear_grad() )rrU)r?rYrs runscale_zGradScaler.unscale_*sBw **rc t|S)a Enable loss scaling or not. Returns: bool: enable loss scaling return True else return False. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> enable = scaler.is_enable() >>> print(enable) True )rrr?rs rrzGradScaler.is_enableMs2w ""rc t|S)av Whether to use dynamic loss scaling. Returns: bool: if fixed loss_scaling is used return False, if the loss scaling is updated dynamicly return true. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> use_dynamic_loss_scaling = scaler.is_use_dynamic_loss_scaling() >>> print(use_dynamic_loss_scaling) True )rrrs rrz&GradScaler.is_use_dynamic_loss_scalingh2w244rc t|S)a% Return the initial loss scaling factor. Reurns: float: the initial loss scaling factor. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> init_loss_scaling = scaler.get_init_loss_scaling() >>> print(init_loss_scaling) 1024 )rrrs rrz GradScaler.get_init_loss_scalings2w,..rc$t||y)a Set the initial loss scaling factor by `new_init_loss_scaling`. Args: new_init_loss_scaling(float): The new_init_loss_scaling used to update initial loss scaling factor. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> print(scaler.get_init_loss_scaling()) 1024 >>> new_init_loss_scaling = 1000 >>> scaler.set_init_loss_scaling(new_init_loss_scaling) >>> print(scaler.get_init_loss_scaling()) 1000 N)rr)r?rrs rrz GradScaler.set_init_loss_scalings8 %&;>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> incr_ratio = scaler.get_incr_ratio() >>> print(incr_ratio) 2.0 )rrrs rrzGradScaler.get_incr_ratio2w%''rc$t||y)a Set the multiplier to use when increasing the loss scaling by `new_incr_ratio`, `new_incr_ratio` should > 1.0. Args: new_incr_ratio(float): The new_incr_ratio used to update the multiplier to use when increasing the loss scaling. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> print(scaler.get_incr_ratio()) 2.0 >>> new_incr_ratio = 3.0 >>> scaler.set_incr_ratio(new_incr_ratio) >>> print(scaler.get_incr_ratio()) 3.0 N)rr)r?rrs rrzGradScaler.set_incr_ratio8 ~.rc t|S)aV Get the less-than-one-multiplier to use when decreasing the loss scaling. Reurns: float: the less-than-one-multiplier to use when decreasing the loss scaling. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> decr_ratio = scaler.get_decr_ratio() >>> print(decr_ratio) 0.5 )rrrs rrzGradScaler.get_decr_ratiorrc$t||y)a7 Set the less-than-one-multiplier to use when decreasing the loss scaling by `new_incr_ratio`, `new_decr_ratio` should < 1.0. Args: new_decr_ratio(float): The new_decr_ratio used to update the less-than-one-multiplier to use when decreasing the loss scaling. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> print(scaler.get_decr_ratio()) 0.5 >>> new_decr_ratio = 0.1 >>> scaler.set_decr_ratio(new_decr_ratio) >>> print(scaler.get_decr_ratio()) 0.1 N)rr)r?rrs rrzGradScaler.set_decr_ratiorrc t|S)a Return the num `n`, `n` represent increases loss scaling every `n` consecutive steps with finite gradients. Reurns: int: the num `n`, `n` represent increases loss scaling every `n` consecutive steps with finite gradients. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> incr_every_n_steps = scaler.get_incr_every_n_steps() >>> print(incr_every_n_steps) 1000 )rrrs rrz!GradScaler.get_incr_every_n_steps.s2w-//rc$t||y)a Set the num `n` by `new_incr_every_n_steps`, `n` represent increases loss scaling every `n` consecutive steps with finite gradients. Args: new_incr_every_n_steps(int): The new_incr_every_n_steps used to update the num `n`, `n` represent increases loss scaling every `n` consecutive steps with finite gradients. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> print(scaler.get_incr_every_n_steps()) 1000 >>> new_incr_every_n_steps = 2000 >>> scaler.set_incr_every_n_steps(new_incr_every_n_steps) >>> print(scaler.get_incr_every_n_steps()) 2000 N)rr)r?rrs rrz!GradScaler.set_incr_every_n_stepsIs8 &'=>rc t|S)a Return the num `n`, `n` represent decreases loss scaling every `n` accumulated steps with nan or inf gradients. Reurns: int: the num `n`, `n` represent decreases loss scaling every `n` accumulated steps with nan or inf gradients. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> decr_every_n_nan_or_inf = scaler.get_decr_every_n_nan_or_inf() >>> print(decr_every_n_nan_or_inf) 2 )rrrs rrz&GradScaler.get_decr_every_n_nan_or_infgrrc$t||y)a Set the num `n` by `new_decr_every_n_nan_or_inf`, `n` represent decreases loss scaling every `n` accumulated steps with nan or inf gradients. Args: new_decr_every_n_nan_or_inf(int): The new_decr_every_n_nan_or_inf used to update the num `n`, `n` represent decreases loss scaling every `n` accumulated steps with nan or inf gradients. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> print(scaler.get_decr_every_n_nan_or_inf()) 2 >>> new_decr_every_n_nan_or_inf = 3 >>> scaler.set_decr_every_n_nan_or_inf(new_decr_every_n_nan_or_inf) >>> print(scaler.get_decr_every_n_nan_or_inf()) 3 N)rr)r?rrs rrz&GradScaler.set_decr_every_n_nan_or_infs8 +,GHrc t|S)a. Returns the state of the scaler as a `dict`, If this instance is not enabled, returns an empty dict. Returns: A dict of scaler includes: scale (tensor): The loss scaling factor. incr_ratio(float): The multiplier to use when increasing the loss scaling. decr_ratio(float): The less-than-one-multiplier to use when decreasing the loss scaling. incr_every_n_steps(int): Increases loss scaling every n consecutive steps with finite gradients. decr_every_n_nan_or_inf(int): Decreases loss scaling every n accumulated steps with nan or inf gradients. incr_count(int): The number of recent consecutive unskipped steps. decr_count(int): The number of recent consecutive skipped steps. use_dynamic_loss_scaling(bool): Whether to use dynamic loss scaling. If False, fixed loss_scaling is used. If True, the loss scaling is updated dynamicly. Default is True. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> scaler_state = scaler.state_dict() )rrrs rrzGradScaler.state_dictsDw!##rc$t||y)a; Loads the scaler state. Args: state_dict(dict): scaler state. Should be an object returned from a call to `GradScaler.state_dict()`. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env:GPU, env:XPU) >>> import paddle >>> scaler = paddle.amp.GradScaler( ... enable=True, ... init_loss_scaling=1024, ... incr_ratio=2.0, ... decr_ratio=0.5, ... incr_every_n_steps=1000, ... decr_every_n_nan_or_inf=2, ... use_dynamic_loss_scaling=True ... ) >>> scaler_state = scaler.state_dict() >>> scaler.load_state_dict(scaler_state) N)rr)r?rrs rrzGradScaler.load_state_dicts4  +r)Tg@rrir T)rrrrrHrOrSrrrrrrrrrrrrrrrrr __classcell__)rs@rrrBs1j! !!% ("@"rsV#(.+?,'T *VPVPr\ ,\ ,r