\;jS)ddlZddlZddlmZddlmZGddZGddZdZdZ d Z d Z d Z d Z dd ZddZdZddZddZddZddZy)N) frameworkceZdZdZddZedZedZedZedZ edZ ed Z ed Z ed Z d Zd ZdZy)Groupz/ The abstract representation of group. Ncx||_|dk\r t|nd|_||_||_||_||_y)Nr)_rank_in_grouplen _world_size_id_ranks_pg_name)self rank_in_groupidrankspgnames oG:\00. PROJECTS\API\Inventory\templateJSON\kerjaOCR\Lib\site-packages\paddle/distributed/communication/group.py__init__zGroup.__init__s;+)6!);3u:  c|jSN)rrs rrankz Group.rank#s"""rc|jSr)r rs rrz Group.ranks's {{rc,t|jSr)r r rs rnranksz Group.nranks+s4;;rc|jSr)rrs rrz Group.name/s zzrc|jSr)r rs r process_groupzGroup.process_group3 xxrc|jSr)r rs r world_sizezGroup.world_size7src6|jjSr)r rrs rbackendz Group.backend;sxx}}rc|jSr)r rs rrzGroup.id?r"rcD|jdkry|jdkryy)NrFT)rrrs r is_memberzGroup.is_memberCs! 99q= ;;?rcZ|jr|jj|Sy)Nr)r*rindex)rrs rget_group_rankzGroup.get_group_rankJs$ >> ::##D) )rcd|jd|jd|jd}|djt t |j z }|dz }||jr|jz }|Sdz }|S)Nzrank: z , nranks: z, id: z , ranks: z, z; name: None)rrrjoinmapstrrr)r debug_strs r__repr__zGroup.__repr__PsTYYKz$++fTWWIY O  TYYs3 344 Z $))TYY7 287 r)NN)__name__ __module__ __qualname____doc__rpropertyrrrrr!r$r&rr*r-r4rrrrs##     rrceZdZdZiZy) _GroupManagerrN)r5r6r7global_group_idgroup_map_by_idr:rrr<r<Zs OOrr<ctjtjvr tdtjtjS)Nz$The global group is not initialized.)r<r=r> RuntimeErrorr:rr_get_global_grouprA_s9$$M,I,IIABB  ( ()F)F GGrc|jtjvrtd|jd|tj|j<y)NzThe group with id z already exist.)rr<r>r@groups r_add_new_grouprEesA xx=000/zIJJ.3M!!%((+rc<|jtjk(Sr)rr<r=rCs r_is_global_grouprGks 88}44 44rctj}|r6|js&tjd|d|j yy)NzCurrent global rank z is not in group TF)distget_rankr*warningswarnr)rD global_ranks r_warn_cur_rank_not_in_grouprNosB--/K U__& ";-/@ M  rc`|j|}|dk\sJd|d|j|S)NrzThe input rank z# can not be found inside the group )r-r)rMrD group_ranks r_get_or_throw_group_rankrQysC%%k2JaV  %H UV rcBtjtjvS)a@ Check whether the distributed environment has been initialized Returns: `True` if distributed environment has been initialized, otherwise `False`. Warning: This API only supports the dygraph mode. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env: DISTRIBUTED) >>> import paddle >>> print(paddle.distributed.is_initialized()) False >>> paddle.distributed.init_parallel_env() >>> print(paddle.distributed.is_initialized()) True )r<r=r>r:rris_initializedrSs2  ( (M,I,I IIrc | tn|}|jtjvsJd|jdt |rtjj ytj|j=y)a Destroy a given group for communication Args: group (Group, optional): The group to be destroyed. All of process groups, including the default group, will be destroyed and the distributed environment will be deinitialized. Returns : None Warning: This API only supports the dygraph mode. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env: DISTRIBUTED) >>> import paddle >>> import paddle.distributed as dist >>> dist.init_parallel_env() >>> group = dist.new_group([0, 1]) >>> dist.destroy_process_group(group) >>> print(dist.is_initialized()) True >>> dist.destroy_process_group() >>> print(dist.is_initialized()) False NzDestroy group with id z is invalid.)rArr<r>rGclearrCs rdestroy_process_grouprVso@$)=  eE M1117 z671%%++-  ) )%(( 3rc|tjvrtj|Stjd|dy)a Get group instance by group id. Args: id (int): the group id. Default value is 0. Returns: Group: the group instance. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env: DISTRIBUTED) >>> import paddle >>> import paddle.distributed as dist >>> dist.init_parallel_env() >>> gid = paddle.distributed.new_group([2,4,6]) >>> paddle.distributed.get_group(gid.id) zGroup z is not initialized.N)r<r>rKrL)rs r get_grouprXs<0 ] * **,,R00 MMF2$234 rctjr tjj ||Sd}tj |fit }|j|d|gid|giy)Nc_sync_calc_streamXOut)typeinputsoutputs)rin_dynamic_modepaddle _legacy_C_opsrZ LayerHelperlocals append_op)tensorop_typehelpers r_sync_calc_streamrisl  "##66vvFF&&&w;&(;&?VH%  rctjr$tjj |g|gd|Sd}tj |fit }|j|d|gid|gid|iy)Nring_idc_sync_comm_streamr[r\r]r^r_attrs)rr`rarbrlrcrdre)rfrkrgrhs r_sync_comm_streamros  "##66 Hvh 7  '&&w;&(;&?VH%g&  rc~||jsy|r t|y|dn |j}t||y)a wait to sync stream for group. Args: tensor (Tensor): The Tensor used before sync. group (Group): The Group instance to perform sync. use_calc_stream (bool): Wether to use calculation stream (True) or communication stream (False). Default to True. Returns: None. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env: DISTRIBUTED) >>> import paddle >>> paddle.distributed.init_parallel_env() >>> tindata = paddle.randn(shape=[2, 3]) >>> paddle.distributed.all_reduce(tindata, sync_op=True) >>> paddle.distributed.wait(tindata) Nr)r*rirro)rfrDuse_calc_streamrks rwaitrrs:4 !2&!}!%((&'*rc||jsytjr| tn|}tj}t |tj r|jj}n+|j}|jj|}|jy|dn |j}tjdgdd}tjr"tjj||d|Sd}t |ts t!dtj"|fit%}|j'|d |gid |gid|i y) a* Barrier among all participators in the group. Args: group (Group): The group instance return by new_group or None for global default group. Returns: None. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env: DISTRIBUTED) >>> import paddle >>> from paddle.distributed import init_parallel_env >>> paddle.set_device('gpu:%d'%paddle.distributed.ParallelEnv().dev_id) >>> init_parallel_env() >>> paddle.distributed.barrier() Nrint32)dtyperkbarrierz,The type of 'group' for barrier must be int.r[r\rm)r*rr`rA_current_expected_place isinstanceCPUPlacer!rw get_device_idrrrrafullrbint ValueErrorrcrdre)rDplacetask device_idrkbarrier_tensorrgrhs rrwrw&sL, !2  "',}!#%113 eY// 0&&..0D++-I&&..y9D =aehhG[[!aw7N  "##++ NIw  '3'KL L&&w;&(;.)*^,-g&  rczt|r td|t}|jS|}|jS)a Get the backend of given group. Args: group (Group): The group to work on. Use the global group as default. Returns: Returns the name of the given group backend. Examples: .. code-block:: python >>> # doctest: +REQUIRES(env: DISTRIBUTED) >>> import paddle >>> paddle.distributed.init_parallel_env() >>> paddle.distributed.get_backend() NCCL zInvalid group specified)rNr@rAr&rCs r get_backendr^s@(#5)455#(=  E ==7rsp !AAH H 4 5J8'4T<    !+H5 pr