\;j/ddlZddlZddlmZddlmZmZmZddlmZm Z m Z ddl m Z m Z gZGdd e ZGd d eZGd d eZy)N)_C_ops)core framework unique_name)_current_expected_placein_dygraph_mode in_pir_mode) Initializercalculate_gainc6eZdZdZ dfd ZddZxZS)MSRAInitializera]Implements the MSRA initializer a.k.a. Kaiming Initializer This class implements the weight initialization from the paper `Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification `_ by Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun. This is a robust initialization method that particularly considers the rectifier nonlinearities. In case of Uniform distribution, the range is [-x, x], where .. math:: x = gain \times \sqrt{\frac{3}{fan\_in}} In case of Normal distribution, the mean is 0 and the standard deviation is .. math:: \frac{gain}{\sqrt{{fan\_in}}} Args: uniform (bool, optional): whether to use uniform or normal distribution. Default is True. fan_in (float32|None, optional): fan_in (in_features) of trainable Tensor, If None, it will be infered automaticly. If you don't want to use in_features of the Tensor, you can set the value of 'fan_in' smartly by yourself. Default is None. seed (int32, optional): random seed. Default is 0. negative_slope (float, optional): negative_slope (only used with leaky_relu). Default is 0.0. nonlinearity(str, optional): the non-linear function. Default is relu. Note: It is recommended to set fan_in to None for most cases. cx|J|Jt|||_||_||_||_||_y)zConstructor for MSRAInitializerN)super__init___uniform_fan_in_seed_negative_slope _nonlinearity)selfuniformfan_inseednegative_slope nonlinearity __class__s fG:\00. PROJECTS\API\Inventory\templateJSON\kerjaOCR\Lib\site-packages\paddle/nn/initializer/kaiming.pyrzMSRAInitializer.__init__AsM"""    -)c |j|}t|tjtj j jfsJt|tjtj jfsJ|j|\}}|j|n |j}|jdk(r|jj|_ |jt jj j"k(s=|jt jj j$k(r|j&st jj j(}|j+t-j.dj1d|j2dg|j4|t jj j6d}nw|jt j8j:t j8j<fvr)|j&st j8j>}|}n|j}|}tAr|j&r|tC|jD|jF}|tIjJdtM|z z} tOjP|j4|| | |jtS}nytC|jD|jF}|tIjJtM|z } tS} tOjT|j4d | |j|| }|jt jj j"k(s=|jt jj j$k(r>|j&s2tOjV||j} | jY|y|jY|yt[ro|j&r|tC|jD|jF}|tIjJdtM|z z} tOjP|j4|| | |jtS}nytC|jD|jF}|tIjJtM|z } tS} tOjT|j4d | |j|| }|jt j8j:t j8j<fvr,|j&s tOjV||jS|S|j&rtC|jD|jF}|tIjJdtM|z z} |j]d id |i|j4t_|| | |jd d } n{tC|jD|jF}|tIjJtM|z } |j]dd |i|j4t_|d | |jdd } |jt jj j"k(s=|jt jj j$k(r<|j&s0|j]dd|id |i|j|jd| |_0| S)aWInitialize the input tensor with MSRA initialization. Args: var(Tensor): Tensor that needs to be initialized. block(Block, optional): The block in which initialization ops should be added. Used in static graph only, default None. Returns: The initialization op. Nr. masra_inittmpF)nameshapedtypetype persistableg@uniform_randomOut)r&r'minmaxrT)r(inputsoutputsattrs stop_gradientgaussian_random)r&r'meanstdr)r(r0r1r2castX)in_dtype out_dtype)r(r/r0r1)1 _check_block isinstancerVariablepaddlepirr ParameterMetaBlock _compute_fansrrprogram random_seedr'VarDescVarTypeFP16BF16rFP32 create_varrgeneratejoinr%r& LOD_TENSORDataTypeFLOAT16BFLOAT16FLOAT32r r rrmathsqrtfloatrrrgaussianr6_share_underline_tensor_tor append_opintop)rvarblockf_inf_outrr9out_vargainlimitr5placevar_tmprXs rforwardzMSRAInitializer.forwardSs|!!%( )$$fjjoo&C&CD   %)//6::3C3C!DEEE((- e-4<< ::?22DJ 99 ,,11 1 II--22 24== ,,11I&& ))HHlCHHe<=ii\\))44!'G II$--//1G1GH HMM --IG IG  }}%d&8&8$:N:NOtyyuV})<== ..IIFJJ+- &d&8&8$:N:NOTYYuV}55/1 //MM3TZZEyyDLL00555 T\\11666t}} ++gsyy922372237 ]}}%d&8&8$:N:NOtyyuV})<== ..IIFJJ+- &d&8&8$:N:NOTYYuV}55/1 //MM3TZZE  dmm33T]]5K5KLL {{7CII66N}}%d&8&8$:N:NOtyyuV})<==__)"G,!(!$Y %v$ $ #'% &d&8&8$:N:NOTYYuV}55__*"G,!(!$Y #" $ #'% yyDLL00555 T\\11666t}}>"CL'.}}399M  CFIr )TNrrrelu)N)__name__ __module__ __qualname____doc__rrb __classcell__rs@rrr s%D  *$Tr rc$eZdZdZdfd ZxZS) KaimingNormala_Implements the Kaiming Normal initializer This class implements the weight initialization from the paper `Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification `_ by Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun. This is a robust initialization method that particularly considers the rectifier nonlinearities. In case of Normal distribution, the mean is 0 and the standard deviation is .. math:: \frac{gain}{\sqrt{{fan\_in}}} Args: fan_in (float32|None, optional): fan_in (in_features) of trainable Tensor, If None, it will be infered automaticly. If you don't want to use in_features of the Tensor, you can set the value of 'fan_in' smartly by yourself. Default is None. negative_slope (float, optional): negative_slope (only used with leaky_relu). Default is 0.0. nonlinearity(str, optional): the non-linear function. Default is relu. Note: It is recommended to set fan_in to None for most cases. Examples: .. code-block:: python >>> import paddle >>> import paddle.nn as nn >>> linear = nn.Linear(2, 4, weight_attr=nn.initializer.KaimingNormal()) >>> data = paddle.rand([30, 10, 2], dtype='float32') >>> res = linear(data) c.t|d|d||y)NFrrrrrrrrrrrrrs rrzKaimingNormal.__init__s$ )%  r Nr*rcrdrerfrgrrhris@rrkrks"H  r rkc$eZdZdZdfd ZxZS)KaimingUniformaYImplements the Kaiming Uniform initializer This class implements the weight initialization from the paper `Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification `_ by Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun. This is a robust initialization method that particularly considers the rectifier nonlinearities. In case of Uniform distribution, the range is [-x, x], where .. math:: x = gain \times \sqrt{\frac{3}{fan\_in}} Args: fan_in (float32|None, optional): fan_in (in_features) of trainable Tensor, If None, it will be infered automaticly. If you don't want to use in_features of the Tensor, you can set the value of 'fan_in' smartly by yourself. Default is None. negative_slope (float, optional): negative_slope (only used with leaky_relu). Default is 0.0. nonlinearity(str, optional): the non-linear function. Default is relu. Note: It is recommended to set fan_in to None for most cases. Examples: .. code-block:: python >>> import paddle >>> import paddle.nn as nn >>> linear = nn.Linear(2, 4, weight_attr=nn.initializer.KaimingUniform()) >>> data = paddle.rand([30, 10, 2], dtype='float32') >>> res = linear(data) c.t|d|d||y)NTrrmrnros rrzKaimingUniform.__init__=s$ )%  r rprqris@rrsrss!F  r rs)rQr=rbaserrrbase.frameworkrr r initializerr r __all__rrkrsr rrzsR  00 5 GkGT, O, ^+ _+ r