From NVIDIA Megatron-LM for visibility by RaymondLi0 · Pull Request #18 · bigcode-project/Megatron-LM

RaymondLi0 · 2023-01-24T20:01:13Z

No description provided.

Signed-off-by: oliver könig <okoenig@nvidia.com>

Co-authored-by: root <root@gpu-h100-0348.cm.cluster> Co-authored-by: root <root@gpu-h100-0193.cm.cluster> Co-authored-by: root <root@gpu-h100-0082.cm.cluster> Co-authored-by: root <root@gpu-h100-0495.cm.cluster> Co-authored-by: William Dykas <wdykas@cw-pdx-cs-001-vscode-02.cm.cluster> Co-authored-by: root <root@gpu-h100-0213.cm.cluster> Co-authored-by: root <root@gpu-h100-0435.cm.cluster> Co-authored-by: root <root@gpu-h100-0188.cm.cluster> Co-authored-by: root <root@gpu-h100-0032.cm.cluster> Co-authored-by: root <root@gpu-h100-0023.cm.cluster> Co-authored-by: root <root@gpu-h100-0368.cm.cluster> Co-authored-by: root <root@gpu-h100-0203.cm.cluster> Co-authored-by: root <root@gpu-h100-0229.cm.cluster> Co-authored-by: root <root@gpu-h100-0123.cm.cluster> Co-authored-by: root <root@gpu-h100-0217.cm.cluster> Co-authored-by: root <root@gpu-h100-0496.cm.cluster> Co-authored-by: root <root@gpu-h100-0261.cm.cluster>

…ernorm. (#2434) Co-authored-by: Yuzhong Wang <yuzhongw@nvidia.com>

Signed-off-by: Hongbin Liu <hongbinl@nvidia.com>

Signed-off-by: Youngeun Kwon <youngeunk@nvidia.com> Co-authored-by: Youngeun Kwon <youngeunk@nvidia.com>

Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com>

This reverts commit ffbc43f.

Signed-off-by: oliver könig <okoenig@nvidia.com>

Signed-off-by: Jimmy Zhang <jiemingz@nvidia.com>

Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> Co-authored-by: Xin Yao <xiny@nvidia.com>

Co-authored-by: Rabeeh Mahabadi <rkarimimahab@nb-hel-cs-001-vscode-02.cm.cluster> Co-authored-by: Sanjeev Satheesh <sasatheesh@nvidia.com> Co-authored-by: Deepak Narayanan <dnarayanan@nvidia.com>

Signed-off-by: Santosh Bhavani <santosh.bhavani@live.com> Co-authored-by: Xin Yao <xiny@nvidia.com>

…offload` (#2874)

Co-authored-by: Maanu Grover <109391026+maanug-nv@users.noreply.github.com>

Signed-off-by: Deepak Narayanan <dnarayanan@nvidia.com>

Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>

…3194)" This reverts commit e836e62.

This reverts commit 300d1b6.

Co-authored-by: Antoni-Joan Solergibert <asolergibert@nvidia.com>

Signed-off-by: Hollow Man <hollowman@opensuse.org> Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> Co-authored-by: Xin Yao <xiny@nvidia.com>

Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>

Signed-off-by: Robin Zhang <robinz@nvidia.com>

…ocator (#2857)

Signed-off-by: oliver könig <okoenig@nvidia.com>

Co-authored-by: Xin Yao <xiny@nvidia.com>

Signed-off-by: oliver könig <okoenig@nvidia.com>

Signed-off-by: Maanu Grover <maanug@nvidia.com>

Co-authored-by: Xin Yao <xiny@nvidia.com>

Signed-off-by: Guyue Huang <guyueh@oci-hsg-cs-001-vscode-01.cm.cluster> Co-authored-by: Guyue Huang <guyueh@oci-hsg-cs-001-vscode-01.cm.cluster> Co-authored-by: Xin Yao <xiny@nvidia.com>

Signed-off-by: oliver könig <okoenig@nvidia.com>

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

…3591)

Co-authored-by: Xin Yao <xiny@nvidia.com>

…3300) Co-authored-by: Xin Yao <xiny@nvidia.com>

Signed-off-by: nvidia <nvidia@TRY-64956-gpu01.nvidialaunchpad.com>

#3630)

Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>

… Manager patch, docs (#3507) Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>

Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com>

Signed-off-by: dimapihtar <dpihtar@gmail.com>

Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>

…3633)

RaymondLi0 changed the base branch from multi-query-attention to before-merge June 20, 2023 20:12

RaymondLi0 changed the base branch from before-merge to multi-query-attention June 20, 2023 20:12

ko3n1g and others added 28 commits January 30, 2026 19:40

ci: Add DSv3 proxy (#3169)

a205538

Signed-off-by: oliver könig <okoenig@nvidia.com>

Update copy-pr-bot.yaml [skip ci]

fdc04f6

[Community][Main] fix(moe): Fix theoretical memory calculation of lay…

9ad5906

…ernorm. (#2434) Co-authored-by: Yuzhong Wang <yuzhongw@nvidia.com>

fix: Set --refit-method default to gloo (#3172)

5415e1d

[fix] Bug fix for offloading in evaluate() (#3043)

a976754

Signed-off-by: Hongbin Liu <hongbinl@nvidia.com>

Update copy-pr-bot.yaml [skip ci]

991c38f

cp: Fix: nccl-ub in ddp path (3181) into main (#3182)

5d0a7fd

Signed-off-by: Youngeun Kwon <youngeunk@nvidia.com> Co-authored-by: Youngeun Kwon <youngeunk@nvidia.com>

Miscellaneous inference cleanup (#2955)

ffbc43f

Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com>

Revert "Miscellaneous inference cleanup (#2955)"

0fe3232

This reverts commit ffbc43f.

ci: Fix DSv3 (#3188)

69a5c63

Signed-off-by: oliver könig <okoenig@nvidia.com>

Fix missing argument in MoELayer.forward() (#3133)

2fadde8

Signed-off-by: Jimmy Zhang <jiemingz@nvidia.com>

Fix H2D stream synchronization in optimizer offload (#3140)

ae67076

Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> Co-authored-by: Xin Yao <xiny@nvidia.com>

Add MTP support for hybrid models (#2363)

300d1b6

Co-authored-by: Rabeeh Mahabadi <rkarimimahab@nb-hel-cs-001-vscode-02.cm.cluster> Co-authored-by: Sanjeev Satheesh <sasatheesh@nvidia.com> Co-authored-by: Deepak Narayanan <dnarayanan@nvidia.com>

docs: improve Megatron-LM and Megatron Core descriptions (#3115)

dceb1fb

Signed-off-by: Santosh Bhavani <santosh.bhavani@live.com> Co-authored-by: Xin Yao <xiny@nvidia.com>

Handle step key correctly in checkpoint save with `--optimizer-cpu-…

f4502eb

…offload` (#2874)

mRoPE for MTP (#3114)

70719cd

Co-authored-by: Maanu Grover <109391026+maanug-nv@users.noreply.github.com>

Fix two minor bugs in MTP implementation for hybrid models (#3194)

e836e62

Signed-off-by: Deepak Narayanan <dnarayanan@nvidia.com>

Update README.md (#2111)

1362e4a

Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>

Revert "Fix two minor bugs in MTP implementation for hybrid models (#…

31d0c87

…3194)" This reverts commit e836e62.

Revert "Add MTP support for hybrid models (#2363)"

a0cc8ca

This reverts commit 300d1b6.

Fix bug in SFTDataset (#3185)

50546da

Co-authored-by: Antoni-Joan Solergibert <asolergibert@nvidia.com>

Fix several syntax error (#3004)

dff4189

Signed-off-by: Hollow Man <hollowman@opensuse.org> Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com> Co-authored-by: Xin Yao <xiny@nvidia.com>

Fix for RL Test (#3148)

c4bea0a

Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>

Fix latent moe flops and backward_dw (#2977)

a4008d0

Signed-off-by: Robin Zhang <robinz@nvidia.com>

Use global user buffer when the bucket size does not fit FixedPoolAll…

afe443b

…ocator (#2857)

ci: Checkpoint retention (#3205)

78475fe

Signed-off-by: oliver könig <okoenig@nvidia.com>

Add unit test for LatentMoE (#2892)

7080697

Co-authored-by: Xin Yao <xiny@nvidia.com>

ko3n1g and others added 30 commits February 26, 2026 03:12

ci: Define secrets (#3599)

19444b9

Signed-off-by: oliver könig <okoenig@nvidia.com>

ci: gh-release-from-tag (#3600)

a08ad25

Signed-off-by: oliver könig <okoenig@nvidia.com>

Ko3n1g/ci/remove twine username (#3601)

ac449c4

Signed-off-by: oliver könig <okoenig@nvidia.com>

Add training code to MCore wheel (#3573)

9088d4f

Signed-off-by: Maanu Grover <maanug@nvidia.com>

FP8 attention knob for nvFP4 recipe (#3363)

dd4c0b7

Co-authored-by: Xin Yao <xiny@nvidia.com>

Fix error with --load-main-params-from-ckpt (#3569)

c2782fd

Signed-off-by: Guyue Huang <guyueh@oci-hsg-cs-001-vscode-01.cm.cluster> Co-authored-by: Guyue Huang <guyueh@oci-hsg-cs-001-vscode-01.cm.cluster> Co-authored-by: Xin Yao <xiny@nvidia.com>

ci: Create comment (#3610)

c921891

Signed-off-by: oliver könig <okoenig@nvidia.com>

ci: Skip cleanup-taint-node jobs during deployments (#3612)

9100119

Signed-off-by: oliver könig <okoenig@nvidia.com>

ci: No comment for release workflow (#3615)

6161f7a

Signed-off-by: oliver könig <okoenig@nvidia.com>

ci: Re-add release tag prefix (#3619)

36d8a9d

Signed-off-by: oliver könig <okoenig@nvidia.com>

docs: Fix version picker urls (#3621)

1b1f5c4

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

ci: Increase changelog generation max PRs fetched (#3620)

027e0f3

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

Add debug info to an assert. (#3588)

18c94d2

fix: async_utils: explicit GC in persistent checkpoint worker loop (#…

e1a9ac9

…3591)

Fix: Perform sigmoid calculation in fp32 for aux loss stability (#2765)

5f668c1

Co-authored-by: Xin Yao <xiny@nvidia.com>

fix: forward use_te_activation_func flag in non-MoE GPT layer spec (#…

d3c10df

…3300) Co-authored-by: Xin Yao <xiny@nvidia.com>

Multimodal: Limit transformer version in Dockerfile (#3448)

7e3f670

Signed-off-by: nvidia <nvidia@TRY-64956-gpu01.nvidialaunchpad.com>

Track and plot per-token off-policy in RL (#3515)

6c0b9c6

Revert "Add single-process checkpoint save to avoid forked multiproce… (

a7c207f

#3630)

Multimodal: fix VQA dataset selection (#3464)

6503bf8

Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>

Multimodal: Fix multimodal training example - tokenizer, Triton Cache…

2f549e5

… Manager patch, docs (#3507) Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>

Support TP > GQA for inference (#3627)

afbce84

Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com>

μP: Maximal Update Parameterization (#3058)

310082a

Add flexible virtual pipeline parallel (fVPP) to hybrid model (#3377)

7418b1b

Explicitly close and join Pool in preprocess_data.py (#3592)

53c5973

remove indexer (#3416)

36a95ae

Signed-off-by: dimapihtar <dpihtar@gmail.com>

Multimodal: add load weights only (#3452)

f0519b7

Signed-off-by: Faradawn Yang <73060648+faradawn@users.noreply.github.com>

Add single-process checkpoint save to avoid forked multiprocessing (#…

61a293d

…3633)

Update oncall schedule (#3632)

6287e7f

Update copy-pr-bot.yaml [skip ci]

93d2739

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

From NVIDIA Megatron-LM for visibility#18

From NVIDIA Megatron-LM for visibility#18
RaymondLi0 wants to merge 6400 commits intobigcode-project:multi-query-attentionfrom
NVIDIA:main

RaymondLi0 commented Jan 24, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Conversation

RaymondLi0 commented Jan 24, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants