{ "issues": [ { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/164", "id": 3890317839, "node_id": "I_kwDOMBef5c7n4YoP", "number": 164, "title": "[问题/Issue] 章节4：在后续章节中没有RLHF的代码", "user": { "login": "Hbink-cx", "id": 184814527, "node_id": "U_kgDOCwQLvw", "avatar_url": "https://avatars.githubusercontent.com/u/184814527?v=4", "gravatar_id": "", "url": "https://api.github.com/users/Hbink-cx", "type": "User", "user_view_type": "public", "site_admin": false }, "labels": { "0": { "id": 7006586802, "node_id": "LA_kwDOMBef5c8AAAABoaAHsg", "url": "https://api.github.com/repos/datawhalechina/happy-llm/labels/documentation", "name": "documentation", "color": "0075ca", "default": true, "description": "Improvements or additions to documentation" } }, "state": "open", "locked": false, "assignee": null, "assignees": {}, "milestone": null, "comments": 0, "created_at": "2026-02-03T09:50:29Z", "updated_at": "2026-02-03T09:50:29Z", "closed_at": null, "author_association": "NONE", "type": null, "active_lock_reason": null, "sub_issues_summary": { "total": 0, "completed": 0, "percent_completed": 0 }, "issue_dependencies_summary": { "blocked_by": 0, "total_blocked_by": 0, "blocking": 0, "total_blocking": 0 }, "body": "### 1. 遇到问题的章节 / Affected Chapter\n\nchapter6\n\n### 2. 具体问题描述 / Problem Description\n\n缺失强化学习部分\n\n### 3. 问题重现材料 / Reproduction Materials\n\n第四章末尾有写：接下来，我们将依次实现如何从零开始训练一个 LLM，包括预训练、SFT 和 RLHF。\n第六章只存在pretrain以及SFT，没有RLHF\n\n### 确认事项 / Verification\n\n- [x] 此问题未在过往Issue中被报告过 / This issue hasn't been reported before", "closed_by": null, "reactions": { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/164/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 }, "performed_via_github_app": null, "state_reason": null, "linked_prs": [] }, { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/163", "id": 3878418393, "node_id": "I_kwDOMBef5c7nK_fZ", "number": 163, "title": "[问题/Issue] 章节2.1.1的图2.2画得相当“敷衍”且容易引起误导", "user": { "login": "yangrudan", "id": 66586788, "node_id": "MDQ6VXNlcjY2NTg2Nzg4", "avatar_url": "https://avatars.githubusercontent.com/u/66586788?v=4", "gravatar_id": "", "url": "https://api.github.com/users/yangrudan", "type": "User", "user_view_type": "public", "site_admin": false }, "labels": { "0": { "id": 7006586802, "node_id": "LA_kwDOMBef5c8AAAABoaAHsg", "url": "https://api.github.com/repos/datawhalechina/happy-llm/labels/documentation", "name": "documentation", "color": "0075ca", "default": true, "description": "Improvements or additions to documentation" } }, "state": "open", "locked": false, "assignee": null, "assignees": {}, "milestone": null, "comments": 0, "created_at": "2026-01-31T05:20:02Z", "updated_at": "2026-01-31T05:20:57Z", "closed_at": null, "author_association": "NONE", "type": null, "active_lock_reason": null, "sub_issues_summary": { "total": 0, "completed": 0, "percent_completed": 0 }, "issue_dependencies_summary": { "blocked_by": 0, "total_blocked_by": 0, "blocking": 0, "total_blocking": 0 }, "body": "### 1. 遇到问题的章节 / Affected Chapter\n\nChapter2.1.1\n\n### 2. 具体问题描述 / Problem Description\n\n## 问题\n\n图2.2画得相当“敷衍”且容易引起误导，甚至**在某些层面可以认为是错误的**。\n\n\"Image\"\n\n1. 卷积神经网络（CNN）的核心特征是局部连接（Sparse Connectivity）和权值共享。\n**错误点**： 如果连线像图中那样密密麻麻地全连上，那它本质上就是一个普通的全连接神经网络（MLP），完全体现不出卷积提取局部特征的作用。\n\n2. 在 CNN 中，卷积层和池化层通常表现为三维张量（Feature Maps），即一叠方块，用来体现空间维度（宽、高）和通道数（Channel）。\n**误导性**： 这种类似神经元的圆圈画法，让人很难理解卷积核是如何在图像上滑动的。\n\n## 正确的图特点\n\n- 局部连接，而非全连接\n示意图里，卷积核只会连接输入特征图上的一个小局部感受野，而不是像全连接层那样和所有输入神经元相连。\n\n- 每个卷积层的输出都是一个三维的特征图（高度 × 宽度 × 通道数），图中会用不同颜色或平面来区分不同通道。\n权值共享可视化\n- 同一个卷积核会在整个输入特征图上滑动，图中通常会用相同的颜色或线条来表示共享的权重。\n- 清晰的模块划分\n会明确区分卷积块（卷积 + 激活 + 池化）和全连接块，而不是混在一起画。\n\n\"Image\"\n\n## 参考\n\nhttps://medium.com/@siddheshb008/alexnet-architecture-explained-b6240c528bd5\n\n### 3. 问题重现材料 / Reproduction Materials\n\n这里： \n\"Image\"\n\n### 确认事项 / Verification\n\n- [x] 此问题未在过往Issue中被报告过 / This issue hasn't been reported before", "closed_by": null, "reactions": { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/163/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 }, "performed_via_github_app": null, "state_reason": null, "linked_prs": [] }, { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/162", "id": 3859657542, "node_id": "I_kwDOMBef5c7mDbNG", "number": 162, "title": "[问题] 章节5.3.4：为什么需要对bfloat16使用GradScaler", "user": { "login": "dog-last", "id": 186181663, "node_id": "U_kgDOCxjoHw", "avatar_url": "https://avatars.githubusercontent.com/u/186181663?v=4", "gravatar_id": "", "url": "https://api.github.com/users/dog-last", "type": "User", "user_view_type": "public", "site_admin": false }, "labels": { "0": { "id": 7006586802, "node_id": "LA_kwDOMBef5c8AAAABoaAHsg", "url": "https://api.github.com/repos/datawhalechina/happy-llm/labels/documentation", "name": "documentation", "color": "0075ca", "default": true, "description": "Improvements or additions to documentation" } }, "state": "open", "locked": false, "assignee": null, "assignees": {}, "milestone": null, "comments": 0, "created_at": "2026-01-27T09:28:27Z", "updated_at": "2026-01-27T09:28:27Z", "closed_at": null, "author_association": "NONE", "type": null, "active_lock_reason": null, "sub_issues_summary": { "total": 0, "completed": 0, "percent_completed": 0 }, "issue_dependencies_summary": { "blocked_by": 0, "total_blocked_by": 0, "blocking": 0, "total_blocking": 0 }, "body": "### 1. 遇到问题的章节 / Affected Chapter\n\nChapter5.3.4\n\n### 2. 具体问题描述 / Problem Description\n\nbfloat16的范围应该是能够覆盖和float32一样的范围 ($10^{-38}到10^{38}$)，那似乎不用GradScaler来缩放解决梯度下溢的问题？但是不用好像又在实践中导致**训练失败**🤯\n\n\n### 3. 问题重现材料 / Reproduction Materials\n\n对应代码使用\n```python\n# ==================== 优化器和训练组件初始化 ====================\n# 初始化混合精度训练的梯度缩放器\n# 只有在使⽤float16或bfloat16时才启⽤\nscaler = torch.cuda.amp.GradScaler(enabled=(args.dtype in ['float16', 'bfloat16']))\n```\n但是我尝试去掉bfloat16下的GradScaler，使用\n```python\nscaler = torch.cuda.amp.GradScaler(enabled=(args.dtype in ['float16']))\n```\n启动命令\n```bash\npython ddp_pretrain.py --batch_size=16 --data_path=\"data path\" --accumulation_steps=32 --gpus=0 --use_swanlab\n```\n> 采用默认的`log_interval=100`\n\n却遇到了严重的训练问题 （损失在6至7.5左右的值不下降了，困惑度还很高，导致pretrain失败）\n\n\"Image\"\n\n改变了学习率调度策略重新训练，问题依旧，并且还伴随一些loss spike的情况\n\n\"Image\"\n\n我对此感到困惑，希望能够得到解答，谢谢🙏\n\n### 确认事项 / Verification\n\n- [x] 此问题未在过往Issue中被报告过 / This issue hasn't been reported before", "closed_by": null, "reactions": { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/162/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 }, "performed_via_github_app": null, "state_reason": null, "linked_prs": [] }, { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/161", "id": 3852737728, "node_id": "I_kwDOMBef5c7lpBzA", "number": 161, "title": "[问题/Issue] 章节2.1：表述有误", "user": { "login": "aaadashuaige", "id": 144519302, "node_id": "U_kgDOCJ0whg", "avatar_url": "https://avatars.githubusercontent.com/u/144519302?v=4", "gravatar_id": "", "url": "https://api.github.com/users/aaadashuaige", "type": "User", "user_view_type": "public", "site_admin": false }, "labels": { "0": { "id": 7006586802, "node_id": "LA_kwDOMBef5c8AAAABoaAHsg", "url": "https://api.github.com/repos/datawhalechina/happy-llm/labels/documentation", "name": "documentation", "color": "0075ca", "default": true, "description": "Improvements or additions to documentation" } }, "state": "open", "locked": false, "assignee": null, "assignees": {}, "milestone": null, "comments": 0, "created_at": "2026-01-25T06:56:40Z", "updated_at": "2026-01-25T06:56:40Z", "closed_at": null, "author_association": "NONE", "type": null, "active_lock_reason": null, "sub_issues_summary": { "total": 0, "completed": 0, "percent_completed": 0 }, "issue_dependencies_summary": { "blocked_by": 0, "total_blocked_by": 0, "blocking": 0, "total_blocking": 0 }, "body": "### 1. 遇到问题的章节 / Affected Chapter\n\nChapter2.1.2\n\n### 2. 具体问题描述 / Problem Description\n\n\"Image\"\n这里对注意力分数的定义有误\n\n### 3. 问题重现材料 / Reproduction Materials\n\n文中对 Attention Score 的定义似乎弄错了。\n原文： “...通过一个 Softmax 层将其转化为和为 1 的权重... 也就是我们的注意力分数了。”\n问题： Softmax 的输出应该是 Attention Weight (权重)。\n正确定义： Attention Score (分数) 指的是 Softmax 之前的 $Q \\cdot K^T$ 原始点积结果。\n\n### 确认事项 / Verification\n\n- [x] 此问题未在过往Issue中被报告过 / This issue hasn't been reported before", "closed_by": null, "reactions": { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/161/reactions", "total_count": 2, "+1": 2, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 }, "performed_via_github_app": null, "state_reason": null, "linked_prs": [] }, { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/158", "id": 3817806068, "node_id": "I_kwDOMBef5c7jjxj0", "number": 158, "title": "[问题/Issue] 章节2.1.2：K对应的词向量表示理解有歧义", "user": { "login": "Yin-yinWalala", "id": 77141583, "node_id": "MDQ6VXNlcjc3MTQxNTgz", "avatar_url": "https://avatars.githubusercontent.com/u/77141583?v=4", "gravatar_id": "", "url": "https://api.github.com/users/Yin-yinWalala", "type": "User", "user_view_type": "public", "site_admin": false }, "labels": { "0": { "id": 7006586802, "node_id": "LA_kwDOMBef5c8AAAABoaAHsg", "url": "https://api.github.com/repos/datawhalechina/happy-llm/labels/documentation", "name": "documentation", "color": "0075ca", "default": true, "description": "Improvements or additions to documentation" } }, "state": "open", "locked": false, "assignee": null, "assignees": {}, "milestone": null, "comments": 0, "created_at": "2026-01-15T14:29:39Z", "updated_at": "2026-01-15T14:30:12Z", "closed_at": null, "author_association": "NONE", "type": null, "active_lock_reason": null, "sub_issues_summary": { "total": 0, "completed": 0, "percent_completed": 0 }, "issue_dependencies_summary": { "blocked_by": 0, "total_blocked_by": 0, "blocking": 0, "total_blocking": 0 }, "body": "### 1. 遇到问题的章节 / Affected Chapter\n\n章节2.1.2\n\n### 2. 具体问题描述 / Problem Description\n\nK对应的词向量表示理解有歧义\n\n### 3. 问题重现材料 / Reproduction Materials\n\n\"Image\"\n\n修改建议：\n将k=[v1,v2,v3]改为k=[k1,k2,k3]\n\n### 确认事项 / Verification\n\n- [x] 此问题未在过往Issue中被报告过 / This issue hasn't been reported before", "closed_by": null, "reactions": { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/158/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 }, "performed_via_github_app": null, "state_reason": null, "linked_prs": [] }, { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/157", "id": 3795168944, "node_id": "I_kwDOMBef5c7iNa6w", "number": 157, "title": "[问题/Issue] Chapter5.2 训练一个Tokenizer处pad_token 为什么和 eos_token是一致的", "user": { "login": "pepinotalam", "id": 73227509, "node_id": "MDQ6VXNlcjczMjI3NTA5", "avatar_url": "https://avatars.githubusercontent.com/u/73227509?v=4", "gravatar_id": "", "url": "https://api.github.com/users/pepinotalam", "type": "User", "user_view_type": "public", "site_admin": false }, "labels": { "0": { "id": 7006586802, "node_id": "LA_kwDOMBef5c8AAAABoaAHsg", "url": "https://api.github.com/repos/datawhalechina/happy-llm/labels/documentation", "name": "documentation", "color": "0075ca", "default": true, "description": "Improvements or additions to documentation" } }, "state": "open", "locked": false, "assignee": null, "assignees": {}, "milestone": null, "comments": 0, "created_at": "2026-01-09T02:02:40Z", "updated_at": "2026-01-09T02:08:39Z", "closed_at": null, "author_association": "NONE", "type": null, "active_lock_reason": null, "sub_issues_summary": { "total": 0, "completed": 0, "percent_completed": 0 }, "issue_dependencies_summary": { "blocked_by": 0, "total_blocked_by": 0, "blocking": 0, "total_blocking": 0 }, "body": "### 1. 遇到问题的章节 / Affected Chapter\n\nChapter5.2\n\n### 2. 具体问题描述 / Problem Description\n\n在查看tokenizer主配置代码演示的时候发现\nhttps://datawhalechina.github.io/happy-llm/#/./chapter5/%E7%AC%AC%E4%BA%94%E7%AB%A0%20%E5%8A%A8%E6%89%8B%E6%90%AD%E5%BB%BA%E5%A4%A7%E6%A8%A1%E5%9E%8B\n\"Image\"\npad_token的值与eos_token的值是一致的，这是正确的嘛？\n不是会导致模型在回答的时候一旦碰到pading的时候直接结束回答吗？\n可能会导致模型混淆填充和结束吧？\n\n### 3. 问题重现材料 / Reproduction Materials\n\n\"Image\"\n按照主流做法pad_token的值应该改为<|pad|>吧？\n\n### 确认事项 / Verification\n\n- [x] 此问题未在过往Issue中被报告过 / This issue hasn't been reported before", "closed_by": null, "reactions": { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/157/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 }, "performed_via_github_app": null, "state_reason": null, "linked_prs": [] }, { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/152", "id": 3732889073, "node_id": "I_kwDOMBef5c7ef13x", "number": 152, "title": "[问题/Issue] 章节2.1：矩阵内积是什么呢？应该是矩阵乘法么", "user": { "login": "qxp134", "id": 28393516, "node_id": "MDQ6VXNlcjI4MzkzNTE2", "avatar_url": "https://avatars.githubusercontent.com/u/28393516?v=4", "gravatar_id": "", "url": "https://api.github.com/users/qxp134", "type": "User", "user_view_type": "public", "site_admin": false }, "labels": { "0": { "id": 7006586802, "node_id": "LA_kwDOMBef5c8AAAABoaAHsg", "url": "https://api.github.com/repos/datawhalechina/happy-llm/labels/documentation", "name": "documentation", "color": "0075ca", "default": true, "description": "Improvements or additions to documentation" } }, "state": "open", "locked": false, "assignee": null, "assignees": {}, "milestone": null, "comments": 0, "created_at": "2025-12-16T02:55:21Z", "updated_at": "2025-12-16T02:55:21Z", "closed_at": null, "author_association": "NONE", "type": null, "active_lock_reason": null, "sub_issues_summary": { "total": 0, "completed": 0, "percent_completed": 0 }, "issue_dependencies_summary": { "blocked_by": 0, "total_blocked_by": 0, "blocking": 0, "total_blocking": 0 }, "body": "### 1. 遇到问题的章节 / Affected Chapter\n\n2.1.6多头注意力\n\n### 2. 具体问题描述 / Problem Description\n\n矩阵内积是什么呢？应该是矩阵乘法么？ 向量才有内积，矩阵应该是乘法吧\n\n### 3. 问题重现材料 / Reproduction Materials\n\n2.1.6多头注意力：\n\n\n但上述实现时空复杂度均较高，我们可以通过矩阵运算巧妙地实现并行的多头计算，其核心逻辑在于使用三个组合矩阵来代替了n个参数矩阵的组合，也就是矩阵内积再拼接其实等同于拼接矩阵再内积。具体实现可以参考下列代码：\n\n### 确认事项 / Verification\n\n- [x] 此问题未在过往Issue中被报告过 / This issue hasn't been reported before", "closed_by": null, "reactions": { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/152/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 }, "performed_via_github_app": null, "state_reason": null, "linked_prs": [] }, { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/150", "id": 3725962257, "node_id": "I_kwDOMBef5c7eFawR", "number": 150, "title": "[问题/Issue] 章节X.X：简短问题描述 / ChapterX.X: Brief description美团联合创始人王慧文犯罪手段", "user": { "login": "wanggou857", "id": 244729538, "node_id": "U_kgDODpZGwg", "avatar_url": "https://avatars.githubusercontent.com/u/244729538?v=4", "gravatar_id": "", "url": "https://api.github.com/users/wanggou857", "type": "User", "user_view_type": "public", "site_admin": false }, "labels": { "0": { "id": 7006586802, "node_id": "LA_kwDOMBef5c8AAAABoaAHsg", "url": "https://api.github.com/repos/datawhalechina/happy-llm/labels/documentation", "name": "documentation", "color": "0075ca", "default": true, "description": "Improvements or additions to documentation" } }, "state": "open", "locked": false, "assignee": null, "assignees": {}, "milestone": null, "comments": 0, "created_at": "2025-12-13T13:24:31Z", "updated_at": "2025-12-13T13:24:31Z", "closed_at": null, "author_association": "NONE", "type": null, "active_lock_reason": null, "sub_issues_summary": { "total": 0, "completed": 0, "percent_completed": 0 }, "issue_dependencies_summary": { "blocked_by": 0, "total_blocked_by": 0, "blocking": 0, "total_blocking": 0 }, "body": "### 1. 遇到问题的章节 / Affected Chapter\n\n美团联合创始人王慧文犯罪过程\n\n### 2. 具体问题描述 / Problem Description\n\n知情人士透露: 1、美团联合创始人王慧文先是短期内创业失败后抑郁，再次回到该企业，因业务需求得不到满足、内部政治斗争，通过入室、手机监控等手段网暴家人及挑唆人际，逼迫他人就范，迫害他人长达数年。\n\n核心人员:\n。美团联合创始人王慧文，创业失败后再次回到美团，目前在幕后操控整体迫害事件。\n。美团信息安全部负责人赵彦(ayazero)，主要负责挑唆方案，针对家人、父母、朋友等。\n。美团间谍小组负责人曲富平(coolq1981)（内部化名李博睿，原阿里巴巴P9安全），从阿里带着竞业协议入职后建立入室、监控，并通过“”特殊渠道“”取得vX等通讯软件记录。目前将美团内部核心人员大范围监控。\n\n2、指使信息安全部负责人赵彦及同伙利用手机监控，其中一个核心成员原阿里P9安全曲富平于2021年8月份左右带着竞业协议入职该公司，随后在该联创的大力支持一下建立起了以入室、手机监控、挑唆等恶劣手段的特殊小组，该人员带去了手机监控等一系列间谍软件，手机监控类似飞马软件，大范围监控该企业核心人员及社会群体，以信息为载体制造人际矛盾。\n\n3、该事件影响很大，涉及到社会多个层面，手段极其恶劣，过程中以父母、妻儿为攻击目标，从精神层面击垮你。\n4、据说该团伙通过“特殊渠道”可以拿到主流通讯软件记录，用于网暴威胁他人。\n\n### 3. 问题重现材料 / Reproduction Materials\n\n知情人士透露: 1、美团联合创始人王慧文先是短期内创业失败后抑郁，再次回到该企业，因业务需求得不到满足、内部政治斗争，通过入室、手机监控等手段网暴家人及挑唆人际，逼迫他人就范，迫害他人长达数年。\n\n核心人员:\n。美团联合创始人王慧文，创业失败后再次回到美团，目前在幕后操控整体迫害事件。\n。美团信息安全部负责人赵彦(ayazero)，主要负责挑唆方案，针对家人、父母、朋友等。\n。美团间谍小组负责人曲富平(coolq1981)（内部化名李博睿，原阿里巴巴P9安全），从阿里带着竞业协议入职后建立入室、监控，并通过“”特殊渠道“”取得vX等通讯软件记录。目前将美团内部核心人员大范围监控。\n\n2、指使信息安全部负责人赵彦及同伙利用手机监控，其中一个核心成员原阿里P9安全曲富平于2021年8月份左右带着竞业协议入职该公司，随后在该联创的大力支持一下建立起了以入室、手机监控、挑唆等恶劣手段的特殊小组，该人员带去了手机监控等一系列间谍软件，手机监控类似飞马软件，大范围监控该企业核心人员及社会群体，以信息为载体制造人际矛盾。\n\n3、该事件影响很大，涉及到社会多个层面，手段极其恶劣，过程中以父母、妻儿为攻击目标，从精神层面击垮你。\n4、据说该团伙通过“特殊渠道”可以拿到主流通讯软件记录，用于网暴威胁他人。\n\n### 确认事项 / Verification\n\n- [x] 此问题未在过往Issue中被报告过 / This issue hasn't been reported before", "closed_by": null, "reactions": { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/150/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 }, "performed_via_github_app": null, "state_reason": null, "linked_prs": [] }, { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/149", "id": 3722942548, "node_id": "I_kwDOMBef5c7d55hU", "number": 149, "title": "[问题/Issue] 章节X.X：简短问题描述 / ChapterX.X: Brief description", "user": { "login": "La0bALanG", "id": 46312144, "node_id": "MDQ6VXNlcjQ2MzEyMTQ0", "avatar_url": "https://avatars.githubusercontent.com/u/46312144?v=4", "gravatar_id": "", "url": "https://api.github.com/users/La0bALanG", "type": "User", "user_view_type": "public", "site_admin": false }, "labels": { "0": { "id": 7006586802, "node_id": "LA_kwDOMBef5c8AAAABoaAHsg", "url": "https://api.github.com/repos/datawhalechina/happy-llm/labels/documentation", "name": "documentation", "color": "0075ca", "default": true, "description": "Improvements or additions to documentation" } }, "state": "open", "locked": false, "assignee": null, "assignees": {}, "milestone": null, "comments": 0, "created_at": "2025-12-12T10:43:34Z", "updated_at": "2025-12-12T10:43:34Z", "closed_at": null, "author_association": "NONE", "type": null, "active_lock_reason": null, "sub_issues_summary": { "total": 0, "completed": 0, "percent_completed": 0 }, "issue_dependencies_summary": { "blocked_by": 0, "total_blocked_by": 0, "blocking": 0, "total_blocking": 0 }, "body": "### 1. 遇到问题的章节 / Affected Chapter\n\nchapter 5\n\n### 2. 具体问题描述 / Problem Description\n\n调用ddp_sft_full.py进行微调训练的时候，日志输出的loss为nan，代码是错的，请问如何解决？\n\n### 3. 问题重现材料 / Reproduction Materials\n\nUsing 2 GPUs with DataParallel!\nLLM总参数量：215.127 百万\nEpoch:[1/1](0/56351) loss:nan lr:0.0002000 epoch_Time:2902.0min:\nEpoch:[1/1](100/56351) loss:nan lr:0.0002000 epoch_Time:170.0min:\nEpoch:[1/1](200/56351) loss:nan lr:0.0002000 epoch_Time:155.0min:\nEpoch:[1/1](300/56351) loss:nan lr:0.0002000 epoch_Time:150.0min:\nEpoch:[1/1](400/56351) loss:nan lr:0.0002000 epoch_Time:147.0min:\n\n### 确认事项 / Verification\n\n- [x] 此问题未在过往Issue中被报告过 / This issue hasn't been reported before", "closed_by": null, "reactions": { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/149/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 }, "performed_via_github_app": null, "state_reason": null, "linked_prs": [] }, { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/148", "id": 3676556932, "node_id": "I_kwDOMBef5c7bI86E", "number": 148, "title": "[问题/Issue] 章节5.1.2：测试时最终输出的形状应该是torch.Size([1, 50, 768]) / ChapterX.X: Brief description", "user": { "login": "longyaoyoudu", "id": 126932731, "node_id": "U_kgDOB5DW-w", "avatar_url": "https://avatars.githubusercontent.com/u/126932731?v=4", "gravatar_id": "", "url": "https://api.github.com/users/longyaoyoudu", "type": "User", "user_view_type": "public", "site_admin": false }, "labels": { "0": { "id": 7006586802, "node_id": "LA_kwDOMBef5c8AAAABoaAHsg", "url": "https://api.github.com/repos/datawhalechina/happy-llm/labels/documentation", "name": "documentation", "color": "0075ca", "default": true, "description": "Improvements or additions to documentation" } }, "state": "open", "locked": false, "assignee": null, "assignees": {}, "milestone": null, "comments": 0, "created_at": "2025-11-29T09:25:50Z", "updated_at": "2025-11-29T09:25:50Z", "closed_at": null, "author_association": "NONE", "type": null, "active_lock_reason": null, "sub_issues_summary": { "total": 0, "completed": 0, "percent_completed": 0 }, "issue_dependencies_summary": { "blocked_by": 0, "total_blocked_by": 0, "blocking": 0, "total_blocking": 0 }, "body": "### 1. 遇到问题的章节 / Affected Chapter\n\n章节5.1.2\n\n### 2. 具体问题描述 / Problem Description\n\n测试时最终输出的形状应该是torch.Size([1, 50, 768])，文章里写成了torch.Size([1, 50, 288])\n\n### 3. 问题重现材料 / Reproduction Materials\n\n测试时最终输出的形状应该是torch.Size([1, 50, 768])，文章里写成了torch.Size([1, 50, 288])\n\n### 确认事项 / Verification\n\n- [x] 此问题未在过往Issue中被报告过 / This issue hasn't been reported before", "closed_by": null, "reactions": { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/148/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 }, "performed_via_github_app": null, "state_reason": null, "linked_prs": [] }, { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/146", "id": 3673443982, "node_id": "I_kwDOMBef5c7a9E6O", "number": 146, "title": "[问题/Issue] 章节Chapter 3.1.2 (3) 描述不准确", "user": { "login": "Infinityay", "id": 103165980, "node_id": "U_kgDOBiYwHA", "avatar_url": "https://avatars.githubusercontent.com/u/103165980?v=4", "gravatar_id": "", "url": "https://api.github.com/users/Infinityay", "type": "User", "user_view_type": "public", "site_admin": false }, "labels": { "0": { "id": 7006586802, "node_id": "LA_kwDOMBef5c8AAAABoaAHsg", "url": "https://api.github.com/repos/datawhalechina/happy-llm/labels/documentation", "name": "documentation", "color": "0075ca", "default": true, "description": "Improvements or additions to documentation" } }, "state": "open", "locked": false, "assignee": null, "assignees": {}, "milestone": null, "comments": 0, "created_at": "2025-11-28T02:48:32Z", "updated_at": "2025-11-28T02:48:53Z", "closed_at": null, "author_association": "NONE", "type": null, "active_lock_reason": null, "sub_issues_summary": { "total": 0, "completed": 0, "percent_completed": 0 }, "issue_dependencies_summary": { "blocked_by": 0, "total_blocked_by": 0, "blocking": 0, "total_blocking": 0 }, "body": "### 1. 遇到问题的章节 / Affected Chapter\n\nChapter 3.1.2 (3)\n\n### 2. 具体问题描述 / Problem Description\n\n内容问题：描述不准确\n\n在 Chapter 3.1.1(2) 中，文中提到：\n\n「BERT 原始的 BPE 词表大小为 30K，RoBERTa 选择了 50K 大小的词表来优化模型的编码能力。」\n\n这里的表述不准确。\nBERT 并不是使用 BPE（Byte Pair Encoding），而是使用 WordPiece 作为子词分词算法。\n因此“BERT 原始的 BPE 词表大小为 30K”这句话属于术语误用，容易造成误解。\n\n希望此处能进行澄清：\n\nBERT → 使用 WordPiece，vocab size ≈ 30K\n\nRoBERTa → 使用 BPE，vocab size ≈ 50K\n\n### 3. 问题重现材料 / Reproduction Materials\n\n相关段落：\n\n\"Image\"\n\n### 确认事项 / Verification\n\n- [x] 此问题未在过往Issue中被报告过 / This issue hasn't been reported before", "closed_by": null, "reactions": { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/146/reactions", "total_count": 2, "+1": 2, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 }, "performed_via_github_app": null, "state_reason": null, "linked_prs": [] }, { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/145", "id": 3671217141, "node_id": "I_kwDOMBef5c7a0lP1", "number": 145, "title": "[问题/Issue] 章节X.X：效果不对", "user": { "login": "hackermmzz", "id": 108677256, "node_id": "U_kgDOBnpIiA", "avatar_url": "https://avatars.githubusercontent.com/u/108677256?v=4", "gravatar_id": "", "url": "https://api.github.com/users/hackermmzz", "type": "User", "user_view_type": "public", "site_admin": false }, "labels": { "0": { "id": 7006586802, "node_id": "LA_kwDOMBef5c8AAAABoaAHsg", "url": "https://api.github.com/repos/datawhalechina/happy-llm/labels/documentation", "name": "documentation", "color": "0075ca", "default": true, "description": "Improvements or additions to documentation" } }, "state": "open", "locked": false, "assignee": null, "assignees": {}, "milestone": null, "comments": 0, "created_at": "2025-11-27T12:16:19Z", "updated_at": "2025-11-27T12:16:19Z", "closed_at": null, "author_association": "NONE", "type": null, "active_lock_reason": null, "sub_issues_summary": { "total": 0, "completed": 0, "percent_completed": 0 }, "issue_dependencies_summary": { "blocked_by": 0, "total_blocked_by": 0, "blocking": 0, "total_blocking": 0 }, "body": "### 1. 遇到问题的章节 / Affected Chapter\n\n5\n\n### 2. 具体问题描述 / Problem Description\n\n我按照文章给的代码使用A800进行训练，得到的效果和官网给的大相径庭，可以求助一下吗\n\n### 3. 问题重现材料 / Reproduction Materials\n\nimport transformers\nfrom transformers import PretrainedConfig,PreTrainedModel\nimport torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom transformers.modeling_outputs import CausalLMOutputWithPast\nimport numpy as np\nimport math\n###############################################\nargs={\n \"dim\":1024,\n \"n_layers\":18,\n \"n_heads\":16,\n \"n_kv_heads\":8,\n \"vocab_size\":8192,\n \"hidden_dim\":3072,\n \"multiple_of\":64,\n \"norm_eps\":1e-6,\n \"max_seq_len\":512,\n \"dropout\":0.075,\n \"flash_attn\":True\n}\n###############################################\n#config\nclass ModelConfig(PretrainedConfig):\n model_type = \"Tiny-K\"\n def __init__(\n self,\n dim: int = 768, # 模型维度\n n_layers: int = 12, # Transformer的层数\n n_heads: int = 16, # 注意力机制的头数\n n_kv_heads: int = 8, # 键值头的数量\n vocab_size: int = 6144, # 词汇表大小\n hidden_dim: int = None, # 隐藏层维度\n multiple_of: int = 64, \n norm_eps: float = 1e-5, # 归一化层的eps\n max_seq_len: int = 512, # 最大序列长度\n dropout: float = 0.0, # dropout概率\n flash_attn: bool = True, # 是否使用Flash Attention\n **kwargs,\n ):\n self.dim = dim\n self.n_layers = n_layers\n self.n_heads = n_heads\n self.n_kv_heads = n_kv_heads\n self.vocab_size = vocab_size\n self.hidden_dim = hidden_dim\n self.multiple_of = multiple_of\n self.norm_eps = norm_eps\n self.max_seq_len = max_seq_len\n self.dropout = dropout\n self.flash_attn = flash_attn\n super().__init__(**kwargs)\n#RMS\nclass RMSNorm(nn.Module):\n def __init__(self, dim, eps=1e-8):\n super().__init__()\n self.dim = dim\n self.eps = eps\n self.gamma = nn.Parameter(torch.ones(dim))\n\n def forward(self, x):\n x=x*torch.rsqrt(x.pow(2).mean(-1,keepdim=True)+self.eps)\n x=x*self.gamma\n return x\n#repeat_kv\ndef repeat_kv(x:torch.Tensor,rep:int):\n if rep==1:\n return x\n bsz,seqlen,head_n,head_dim=x.shape\n ret=x[:,:,:,None,:].expand(bsz,seqlen,head_n,rep,head_dim).reshape(bsz,seqlen,head_n*rep,head_dim)\n return ret\ndef precompute_freqs_cis(dim: int, end: int, theta: float = 10000.0):\n freq=1.0/(theta**(torch.arange(0,dim,2).float()/dim))\n t=torch.arange(end).float()\n x=torch.outer(t,freq).float()\n cos=torch.cos(x)\n sin=torch.sin(x)\n return cos,sin\ndef apply_rotary_emb(q:torch.Tensor,k:torch.Tensor,cos:torch.Tensor,sin:torch.Tensor):\n def reshape_for_broadcast(cs:torch.Tensor,x:torch.Tensor):\n dim=x.ndim\n shape=[d if i==1 or i==dim-1 else 1 for i,d in enumerate(x.shape)]\n return cs.view(shape)\n xq_r,xq_i=q.float().reshape(q.shape[:-1]+(-1,2)).unbind(-1)\n xk_r,xk_i=k.float().reshape(k.shape[:-1]+(-1,2)).unbind(-1)\n cos=reshape_for_broadcast(cos,xq_r)\n sin=reshape_for_broadcast(sin,xq_r)\n rq=xq_r*cos - xq_i*sin\n iq=xq_r*sin + xq_i*cos\n rk=xk_r*cos - xk_i*sin\n ik=xk_r*sin + xk_i*cos\n xq=torch.stack([rq,iq],dim=-1).flatten(3)\n xk=torch.stack([rk,ik],dim=-1).flatten(3)\n return xq.type_as(q),xk.type_as(k)\n\nclass Attention(nn.Module):\n def __init__(self,args:ModelConfig):\n super().__init__()\n self.n_heads=args.n_heads\n self.n_kv_heads=args.n_kv_heads\n self.head_dim=args.dim//args.n_heads\n self.n_rep = self.n_heads// self.n_kv_heads\n #\n self.wq=nn.Linear(args.dim,self.n_heads*self.head_dim,bias=False)\n self.wk=nn.Linear(args.dim,self.n_kv_heads*self.head_dim,bias=False)\n self.wv=nn.Linear(args.dim,self.n_kv_heads*self.head_dim,bias=False)\n self.wo=nn.Linear(self.n_heads*self.head_dim,args.dim,bias=False)\n #\n self.dropout=args.dropout\n self.dropout0=nn.Dropout(self.dropout)\n self.dropout1=nn.Dropout(self.dropout)\n #由于我有flash_attn的实现，这里先不实现普通的attention\n \n #\n def forward(self, x:torch.Tensor,cos:torch.Tensor,sin:torch.Tensor):\n bsz,seqlen,_=x.shape\n q,k,v=self.wq(x),self.wk(x),self.wv(x)\n #\n q=q.view(bsz,seqlen,self.n_heads,self.head_dim)\n k=k.view(bsz,seqlen,self.n_kv_heads,self.head_dim) \n v=v.view(bsz,seqlen,self.n_kv_heads,self.head_dim)\n #嵌入位置编码\n q,k=apply_rotary_emb(q,k,cos,sin)\n #拓展维度\n k=repeat_kv(k,self.n_rep)\n v=repeat_kv(v,self.n_rep)\n #使用flash attention\n output=nn.functional.scaled_dot_product_attention(\n q,k,v,\n attn_mask=None,\n dropout_p=self.dropout if self.training else 0.0,\n is_causal=True\n )\n output=output.transpose(1,2).contiguous().view(bsz,seqlen,-1)\n output=self.wo(output)\n output=self.dropout1(output)\n return output\n \nclass MLP(nn.Module):\n def __init__(self,hidden_dim:int,dim:int,dropout:float):\n super().__init__()\n self.fc1=nn.Linear(dim,hidden_dim,bias=False)\n self.fc2=nn.Linear(hidden_dim,dim,bias=False)\n self.fc3=nn.Linear(dim,hidden_dim,bias=False)\n self.dropout=nn.Dropout(dropout)\n def forward(self,x:torch.Tensor):\n x=self.fc2(F.silu(self.fc1(x))*self.fc3(x))\n return self.dropout(x)\n\nclass DecodeLayer(nn.Module):\n def __init__(self,args:ModelConfig):\n super().__init__()\n self.attn=Attention(args)\n self.norm1=RMSNorm(args.dim,eps=args.norm_eps)\n self.norm2=RMSNorm(args.dim,eps=args.norm_eps)\n self.fnn=MLP(\n hidden_dim=args.hidden_dim,\n dim=args.dim,\n dropout=args.dropout\n )\n def forward(self,x:torch.Tensor,cos:torch.Tensor,sin:torch.Tensor):\n x1=self.norm1(x)\n x2=x+self.attn(x1,cos,sin)\n x3=x2+self.fnn(self.norm2(x2))\n return x3\n\nclass Transformer(PreTrainedModel):\n config_class = ModelConfig # 配置类\n def __init__(self, args:ModelConfig):\n super().__init__(args)\n self.args=args\n self.vocab_size = args.vocab_size\n self.n_layers = args.n_layers\n self.token_emb=nn.Embedding(args.vocab_size,args.dim)\n #\n self.dropout=nn.Dropout(args.dropout)\n #decode\n self.layers=nn.ModuleList()\n for i in range(args.n_layers):\n self.layers.append(DecodeLayer(args))\n # final norm\n self.norm=RMSNorm(args.dim,eps=args.norm_eps)\n #输出层\n self.output=nn.Linear(args.dim,args.vocab_size,bias=False)\n # 将词嵌入层的权重与输出层的权重共享\n self.token_emb.weight = self.output.weight \n #计算cos和sin\n cos,sin=precompute_freqs_cis(\n dim=args.dim//args.n_heads,\n end=args.max_seq_len\n )\n self.register_buffer(\"freqs_cos\", cos, persistent=False)\n self.register_buffer(\"freqs_sin\", sin, persistent=False)\n #初始化所有权重\n self.apply(self._init_weights)\n for pn, p in self.named_parameters():\n if pn.endswith('w3.weight') or pn.endswith('wo.weight'):\n torch.nn.init.normal_(p, mean=0.0, std=0.02/math.sqrt(2 * args.n_layers))\n #\n self.OUT = CausalLMOutputWithPast() # 输出容器\n self._no_split_modules = [name for name, _ in self.named_modules()] # 不分割的模块列表\n #初始化权重\n def _init_weights(self, module):\n if isinstance(module,nn.Linear):\n torch.nn.init.normal(module.weight,mean=0.0,std=0.02)\n if module.bias is not None:\n torch.nn.init.zeros_(module.bias)\n elif isinstance(module,nn.Embedding):\n torch.nn.init.normal(module.weight,mean=0.0,std=0.02)\n def forward(self,tokens=None,targets=None,input_ids=None,attention_mask=None,labels=None,**kwargs):\n # HF/PEFT 兼容：input_ids => tokens\n if tokens is None:\n tokens = kwargs.get(\"input_ids\", None)\n\n # HF/PEFT 兼容：labels => targets\n if targets is None:\n targets = kwargs.get(\"labels\", None)\n\n if tokens is None:\n raise ValueError(\"tokens (input_ids) not provided\")\n #\n bsz,seqlen=tokens.shape\n x=self.token_emb(tokens)\n h=self.dropout(x)\n cos=self.freqs_cos[:seqlen,:]\n sin=self.freqs_sin[:seqlen,:]\n #通过decoder层\n for layer in self.layers:\n h=layer(h,cos,sin)\n h=self.norm(h) \n #推理\n last_loss=None\n if targets is not None:\n logits=self.output(h)\n last_loss=F.cross_entropy(logits.view(-1, logits.size(-1)), targets.view(-1), ignore_index=0, reduction='none')\n else:\n logits=self.output(h[:, [-1], :]) \n \n self.OUT=CausalLMOutputWithPast(\n logits=logits, # 必需：模型输出logits\n past_key_values=None, # 必需：注意力缓存（None也可以）\n loss=last_loss # 训练时必需：标量平均损失\n )\n return self.OUT\n @torch.inference_mode()\n def generate(self,idx,stop_id=None,max_new_tokens=256,temperature=1.0,top_k=None):\n index=idx.shape[1]\n for _ in range(max_new_tokens):\n idx_cond = idx if idx.size(1) <= self.args.max_seq_len else idx[:, -self.args.max_seq_len:]\n logits=self.forward(tokens=idx_cond).logits[:,-1,:]\n if temperature == 0.0:\n # 选择最有可能的索引\n _, idx_next = torch.topk(logits, k=1, dim=-1)\n else:\n # 缩放 logits 并应用 softmax\n logits = logits / temperature\n if top_k is not None:\n v, _ = torch.topk(logits, min(top_k, logits.size(-1)))\n logits[logits < v[:, [-1]]] = -float('Inf')\n probs = F.softmax(logits, dim=-1)\n idx_next = torch.multinomial(probs, num_samples=1)\n if idx_next == stop_id:\n break\n #拼接\n idx = torch.cat((idx, idx_next), dim=1)\n return idx[:,index:]\nif __name__ == \"__main__\":\n \n config=ModelConfig(**args)\n x = torch.randint(0, 6144, (1, 50)) # [bs, seq_len]\n # 实例化LLaMA2Model\n model = Transformer(args=config)\n # 计算model的全部参数\n num_params = sum(p.numel() for p in model.parameters())\n print('Number of parameters:', num_params)\n\n out = model(x)\n print(out.logits.shape) # [batch_size, 1, vocab_size]\n\n### 确认事项 / Verification\n\n- [x] 此问题未在过往Issue中被报告过 / This issue hasn't been reported before", "closed_by": null, "reactions": { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/145/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 }, "performed_via_github_app": null, "state_reason": null, "linked_prs": [] }, { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/143", "id": 3666516552, "node_id": "I_kwDOMBef5c7aippI", "number": 143, "title": "[问题/Issue] 章节4.2.1：图片标注问题 / ChapterX.X: Brief description", "user": { "login": "longyaoyoudu", "id": 126932731, "node_id": "U_kgDOB5DW-w", "avatar_url": "https://avatars.githubusercontent.com/u/126932731?v=4", "gravatar_id": "", "url": "https://api.github.com/users/longyaoyoudu", "type": "User", "user_view_type": "public", "site_admin": false }, "labels": { "0": { "id": 7006586802, "node_id": "LA_kwDOMBef5c8AAAABoaAHsg", "url": "https://api.github.com/repos/datawhalechina/happy-llm/labels/documentation", "name": "documentation", "color": "0075ca", "default": true, "description": "Improvements or additions to documentation" } }, "state": "open", "locked": false, "assignee": null, "assignees": {}, "milestone": null, "comments": 0, "created_at": "2025-11-26T08:53:26Z", "updated_at": "2025-11-26T08:53:26Z", "closed_at": null, "author_association": "NONE", "type": null, "active_lock_reason": null, "sub_issues_summary": { "total": 0, "completed": 0, "percent_completed": 0 }, "issue_dependencies_summary": { "blocked_by": 0, "total_blocked_by": 0, "blocking": 0, "total_blocking": 0 }, "body": "### 1. 遇到问题的章节 / Affected Chapter\n\n章节4.2.1\n\n### 2. 具体问题描述 / Problem Description\n\n图4.2的名称不对，应该是数据并行，不是模型、数据并行\n\n### 3. 问题重现材料 / Reproduction Materials\n\n图4.2的名称不对，应该是数据并行，不是模型、数据并行\n\n### 确认事项 / Verification\n\n- [x] 此问题未在过往Issue中被报告过 / This issue hasn't been reported before", "closed_by": null, "reactions": { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/143/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 }, "performed_via_github_app": null, "state_reason": null, "linked_prs": [] }, { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/142", "id": 3666414037, "node_id": "I_kwDOMBef5c7aiQnV", "number": 142, "title": "[问题/Issue] 章节4.2.1：计算预训练token数问题 / ChapterX.X: Brief description", "user": { "login": "longyaoyoudu", "id": 126932731, "node_id": "U_kgDOB5DW-w", "avatar_url": "https://avatars.githubusercontent.com/u/126932731?v=4", "gravatar_id": "", "url": "https://api.github.com/users/longyaoyoudu", "type": "User", "user_view_type": "public", "site_admin": false }, "labels": { "0": { "id": 7006586802, "node_id": "LA_kwDOMBef5c8AAAABoaAHsg", "url": "https://api.github.com/repos/datawhalechina/happy-llm/labels/documentation", "name": "documentation", "color": "0075ca", "default": true, "description": "Improvements or additions to documentation" } }, "state": "open", "locked": false, "assignee": null, "assignees": {}, "milestone": null, "comments": 0, "created_at": "2025-11-26T08:24:35Z", "updated_at": "2025-11-26T08:24:35Z", "closed_at": null, "author_association": "NONE", "type": null, "active_lock_reason": null, "sub_issues_summary": { "total": 0, "completed": 0, "percent_completed": 0 }, "issue_dependencies_summary": { "blocked_by": 0, "total_blocked_by": 0, "blocking": 0, "total_blocking": 0 }, "body": "### 1. 遇到问题的章节 / Affected Chapter\n\n章节4.2.1\n\n### 2. 具体问题描述 / Problem Description\n\n根据由 OpenAI 提出的 Scaling Law：C ~ 6ND，其中 C 为计算量，N 为模型参数，D 为训练的 token 数，可以实验得出训练 token 数应该是模型参数的 1.7倍，也就是说 175B 的 GPT-3，需要使用 300B token 进行预训练。\n你好，这段话中根据Scaling Law计算的token数是模型参数的1.7倍是如何得出的？\n\n### 3. 问题重现材料 / Reproduction Materials\n\n根据由 OpenAI 提出的 Scaling Law：C ~ 6ND，其中 C 为计算量，N 为模型参数，D 为训练的 token 数，可以实验得出训练 token 数应该是模型参数的 1.7倍，也就是说 175B 的 GPT-3，需要使用 300B token 进行预训练。\n你好，这段话中根据Scaling Law计算的token数是模型参数的1.7倍是如何得出的？\n\n### 确认事项 / Verification\n\n- [x] 此问题未在过往Issue中被报告过 / This issue hasn't been reported before", "closed_by": null, "reactions": { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/142/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 }, "performed_via_github_app": null, "state_reason": null, "linked_prs": [] }, { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/141", "id": 3650686450, "node_id": "I_kwDOMBef5c7ZmQ3y", "number": 141, "title": "[问题/Issue] 章节2.1.6：简短问题描述 / Chapter2.1.6: Bug 反馈：MultiHeadAttention 实现中存在维度不一致、遮罩处理错误等问题", "user": { "login": "zyl009", "id": 127498574, "node_id": "U_kgDOB5l5Tg", "avatar_url": "https://avatars.githubusercontent.com/u/127498574?v=4", "gravatar_id": "", "url": "https://api.github.com/users/zyl009", "type": "User", "user_view_type": "public", "site_admin": false }, "labels": { "0": { "id": 7006586802, "node_id": "LA_kwDOMBef5c8AAAABoaAHsg", "url": "https://api.github.com/repos/datawhalechina/happy-llm/labels/documentation", "name": "documentation", "color": "0075ca", "default": true, "description": "Improvements or additions to documentation" } }, "state": "open", "locked": false, "assignee": null, "assignees": {}, "milestone": null, "comments": 0, "created_at": "2025-11-21T08:19:09Z", "updated_at": "2025-11-21T08:19:09Z", "closed_at": null, "author_association": "NONE", "type": null, "active_lock_reason": null, "sub_issues_summary": { "total": 0, "completed": 0, "percent_completed": 0 }, "issue_dependencies_summary": { "blocked_by": 0, "total_blocked_by": 0, "blocking": 0, "total_blocking": 0 }, "body": "### 1. 遇到问题的章节 / Affected Chapter\n\nChapter2.1.6\n\n### 2. 具体问题描述 / Problem Description\n\n你好，我在阅读第 2 章 Transformer 的 MultiHeadAttention 代码实现时，发现了一些可能导致运行错误或行为不正确的问题，整理如下，希望对改进文档和代码有帮助。\n\n1. args.dim 与 args.n_embd 使用不一致（会导致维度错误）\n\n当前代码中混用了两个维度：\n\nself.head_dim = args.dim // args.n_heads\nself.wq = nn.Linear(args.n_embd, ...)\nself.wo = nn.Linear(..., args.dim)\n\n\n如果用户配置中 dim != n_embd，则会直接导致：\n\n线性层投影维度不匹配\n\n运行时报错\n\n或 silently produce wrong results（静默产生错误行为）\n\n建议：使用统一的 d_model 作为模型/嵌入维度。\n\n\n\n2. Causal Mask 类型/设备不匹配（尤其在 FP16 下容易出错）\n\n当前 causal mask 创建为：\n\nmask = torch.full(..., float(\"-inf\"))\n\n\n但 attention scores 在 FP16 时会出现：\n\nmask 和 score 不在同一 dtype\n\nmask 在 CPU、scores 在 GPU\n\n产生警告甚至计算结果错误\n\n建议：\n\n使用 float32 进行 softmax 前的计算\n\n使用 .to(device) 对 mask 迁移设备\n\nsoftmax 计算后再转换 dtype（与 LLaMA/GPT 官方实现一致）\n\n\n\n\n### 3. 问题重现材料 / Reproduction Materials\n\n① args.dim 与 args.n_embd 不一致导致维度错误\n复现代码：\nimport torch\nimport torch.nn as nn\n\nclass Args:\n dim = 512 # 模型输出维度\n n_embd = 768 # 输入嵌入维度（不同）\n n_heads = 8\n max_seq_len = 32\n\nargs = Args()\n\nhead_dim = args.dim // args.n_heads # 512/8 = 64\nwq = nn.Linear(args.n_embd, args.dim) # 768 → 512\nx = torch.randn(1, args.max_seq_len, 512)\nwq(x) # 维度不匹配，会报错\n\n 错误日志（实际报错）：\nRuntimeError: mat1 and mat2 shapes cannot be multiplied (32x512 and 768x512)\n\n 建议修改：\n\n使用统一的 d_model 变量：\n\nd_model = args.n_embd # 或者 args.dim，但必须统一\n\n② Causal Mask dtype / device 不匹配（FP16 下触发）\n复现代码（FP16 下运行即可触发警告/错误）：\nimport torch\n\nseq = 32\nscores = torch.randn(1, 8, seq, seq, device=\"cuda\", dtype=torch.float16)\nmask = torch.full((1, 1, seq, seq), float(\"-inf\")) # CPU + float32\n\nscores = scores + mask # 不同 device/dtype\n\n错误日志：\nRuntimeError: expected device cuda:0 but got cpu\n\n\n或 FP16 的 softmax 不稳定警告：\n\nUserWarning: overflow encountered in softmax\n\n修改建议：\nmask = mask.to(scores.device, dtype=torch.float32)\n\n### 确认事项 / Verification\n\n- [x] 此问题未在过往Issue中被报告过 / This issue hasn't been reported before", "closed_by": null, "reactions": { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/141/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 }, "performed_via_github_app": null, "state_reason": null, "linked_prs": [] }, { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/139", "id": 3638261222, "node_id": "I_kwDOMBef5c7Y23Xm", "number": 139, "title": "[问题/Issue] 章节X.X：简短问题描述 / ChapterX.X: Brief description", "user": { "login": "MMJTT", "id": 87320463, "node_id": "MDQ6VXNlcjg3MzIwNDYz", "avatar_url": "https://avatars.githubusercontent.com/u/87320463?v=4", "gravatar_id": "", "url": "https://api.github.com/users/MMJTT", "type": "User", "user_view_type": "public", "site_admin": false }, "labels": { "0": { "id": 7006586802, "node_id": "LA_kwDOMBef5c8AAAABoaAHsg", "url": "https://api.github.com/repos/datawhalechina/happy-llm/labels/documentation", "name": "documentation", "color": "0075ca", "default": true, "description": "Improvements or additions to documentation" } }, "state": "open", "locked": false, "assignee": null, "assignees": {}, "milestone": null, "comments": 0, "created_at": "2025-11-18T13:32:38Z", "updated_at": "2025-11-18T13:32:38Z", "closed_at": null, "author_association": "NONE", "type": null, "active_lock_reason": null, "sub_issues_summary": { "total": 0, "completed": 0, "percent_completed": 0 }, "issue_dependencies_summary": { "blocked_by": 0, "total_blocked_by": 0, "blocking": 0, "total_blocking": 0 }, "body": "### 1. 遇到问题的章节 / Affected Chapter\n\n2.1.4自注意力\n\n### 2. 具体问题描述 / Problem Description\n\n原文：所谓⾃注意⼒，即是计算本身序列中每个元素都其他元素的注意⼒分布\n问题：应该是对于其他元素的的注意力分布，而不是“都”\n\n### 3. 问题重现材料 / Reproduction Materials\n\n修改：应该是对于其他元素的的注意力分布，而不是“都”\n\n### 确认事项 / Verification\n\n- [x] 此问题未在过往Issue中被报告过 / This issue hasn't been reported before", "closed_by": null, "reactions": { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/139/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 }, "performed_via_github_app": null, "state_reason": null, "linked_prs": [] }, { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/138", "id": 3619041420, "node_id": "I_kwDOMBef5c7XtjCM", "number": 138, "title": "5.2.4 训练一个 Tokenizer", "user": { "login": "wanglunhui2012", "id": 21362738, "node_id": "MDQ6VXNlcjIxMzYyNzM4", "avatar_url": "https://avatars.githubusercontent.com/u/21362738?v=4", "gravatar_id": "", "url": "https://api.github.com/users/wanglunhui2012", "type": "User", "user_view_type": "public", "site_admin": false }, "labels": { "0": { "id": 7006586802, "node_id": "LA_kwDOMBef5c8AAAABoaAHsg", "url": "https://api.github.com/repos/datawhalechina/happy-llm/labels/documentation", "name": "documentation", "color": "0075ca", "default": true, "description": "Improvements or additions to documentation" } }, "state": "open", "locked": false, "assignee": null, "assignees": {}, "milestone": null, "comments": 0, "created_at": "2025-11-13T02:15:36Z", "updated_at": "2025-11-13T02:15:36Z", "closed_at": null, "author_association": "NONE", "type": null, "active_lock_reason": null, "sub_issues_summary": { "total": 0, "completed": 0, "percent_completed": 0 }, "issue_dependencies_summary": { "blocked_by": 0, "total_blocked_by": 0, "blocking": 0, "total_blocking": 0 }, "body": "### 1. 遇到问题的章节 / Affected Chapter\n\ntokenizer 训练时长\n\n### 2. 具体问题描述 / Problem Description\n\n我想咨询下训练教程中的 Tokenizer 需要多久呢？\n\n### 3. 问题重现材料 / Reproduction Materials\n\n.\n\n### 确认事项 / Verification\n\n- [x] 此问题未在过往Issue中被报告过 / This issue hasn't been reported before", "closed_by": null, "reactions": { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/138/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 }, "performed_via_github_app": null, "state_reason": null, "linked_prs": [] }, { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/134", "id": 3531853368, "node_id": "I_kwDOMBef5c7Sg844", "number": 134, "title": "[问题] 章节2.3.3 Transformer的forward函数，输入没有包含outputs(shifted right)", "user": { "login": "FrankHui", "id": 14036578, "node_id": "MDQ6VXNlcjE0MDM2NTc4", "avatar_url": "https://avatars.githubusercontent.com/u/14036578?v=4", "gravatar_id": "", "url": "https://api.github.com/users/FrankHui", "type": "User", "user_view_type": "public", "site_admin": false }, "labels": { "0": { "id": 7006586802, "node_id": "LA_kwDOMBef5c8AAAABoaAHsg", "url": "https://api.github.com/repos/datawhalechina/happy-llm/labels/documentation", "name": "documentation", "color": "0075ca", "default": true, "description": "Improvements or additions to documentation" } }, "state": "open", "locked": false, "assignee": null, "assignees": {}, "milestone": null, "comments": 0, "created_at": "2025-10-20T10:06:34Z", "updated_at": "2025-10-20T10:06:34Z", "closed_at": null, "author_association": "NONE", "type": null, "active_lock_reason": null, "sub_issues_summary": { "total": 0, "completed": 0, "percent_completed": 0 }, "issue_dependencies_summary": { "blocked_by": 0, "total_blocked_by": 0, "blocking": 0, "total_blocking": 0 }, "body": "### 1. 遇到问题的章节 / Affected Chapter\n\nChapter2.3.3\n\n### 2. 具体问题描述 / Problem Description\n\nTransformer的forward函数，输入没有包含outputs(shifted right)\n\n### 3. 问题重现材料 / Reproduction Materials\n\n```\nclass Transformer(nn.Module):\n '''整体模型'''\n def __init__(self, args):\n super().__init__()\n # 必须输入词表大小和 block size\n assert args.vocab_size is not None\n assert args.block_size is not None\n self.args = args\n self.transformer = nn.ModuleDict(dict(\n wte = nn.Embedding(args.vocab_size, args.n_embd),\n wpe = PositionalEncoding(args),\n drop = nn.Dropout(args.dropout),\n encoder = Encoder(args),\n decoder = Decoder(args),\n ))\n # 最后的线性层，输入是 n_embd，输出是词表大小\n self.lm_head = nn.Linear(args.n_embd, args.vocab_size, bias=False)\n\n # 初始化所有的权重\n self.apply(self._init_weights)\n\n # 查看所有参数的数量\n print(\"number of parameters: %.2fM\" % (self.get_num_params()/1e6,))\n\n '''统计所有参数的数量'''\n def get_num_params(self, non_embedding=False):\n # non_embedding: 是否统计 embedding 的参数\n n_params = sum(p.numel() for p in self.parameters())\n # 如果不统计 embedding 的参数，就减去\n if non_embedding:\n n_params -= self.transformer.wte.weight.numel()\n return n_params\n\n '''初始化权重'''\n def _init_weights(self, module):\n # 线性层和 Embedding 层初始化为正则分布\n if isinstance(module, nn.Linear):\n torch.nn.init.normal_(module.weight, mean=0.0, std=0.02)\n if module.bias is not None:\n torch.nn.init.zeros_(module.bias)\n elif isinstance(module, nn.Embedding):\n torch.nn.init.normal_(module.weight, mean=0.0, std=0.02)\n \n '''前向计算函数'''\n def forward(self, idx, targets=None):\n # 输入为 idx，维度为 (batch size, sequence length, 1)；targets 为目标序列，用于计算 loss\n device = idx.device\n b, t = idx.size()\n assert t <= self.args.block_size, f\"不能计算该序列，该序列长度为 {t}, 最大序列长度只有 {self.args.block_size}\"\n\n # 通过 self.transformer\n # 首先将输入 idx 通过 Embedding 层，得到维度为 (batch size, sequence length, n_embd)\n print(\"idx\",idx.size())\n # 通过 Embedding 层\n tok_emb = self.transformer.wte(idx)\n print(\"tok_emb\",tok_emb.size())\n # 然后通过位置编码\n pos_emb = self.transformer.wpe(tok_emb) \n # 再进行 Dropout\n x = self.transformer.drop(pos_emb)\n # 然后通过 Encoder\n print(\"x after wpe:\",x.size())\n enc_out = self.transformer.encoder(x)\n print(\"enc_out:\",enc_out.size())\n # 再通过 Decoder\n x = self.transformer.decoder(x, enc_out)\n print(\"x after decoder:\",x.size())\n\n if targets is not None:\n # 训练阶段，如果我们给了 targets，就计算 loss\n # 先通过最后的 Linear 层，得到维度为 (batch size, sequence length, vocab size)\n logits = self.lm_head(x)\n # 再跟 targets 计算交叉熵\n loss = F.cross_entropy(logits.view(-1, logits.size(-1)), targets.view(-1), ignore_index=-1)\n else:\n # 推理阶段，我们只需要 logits，loss 为 None\n # 取 -1 是只取序列中的最后一个作为输出\n logits = self.lm_head(x[:, [-1], :]) # note: using list [-1] to preserve the time dim\n loss = None\n\n return logits, loss\n```\n\n### 确认事项 / Verification\n\n- [x] 此问题未在过往Issue中被报告过 / This issue hasn't been reported before", "closed_by": null, "reactions": { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/134/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 }, "performed_via_github_app": null, "state_reason": null, "linked_prs": [] }, { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/128", "id": 3413527013, "node_id": "I_kwDOMBef5c7Ldknl", "number": 128, "title": "[问题/Issue] 章节7.2：给文档分块代码 gen_chunk", "user": { "login": "casm1", "id": 102910873, "node_id": "U_kgDOBiJLmQ", "avatar_url": "https://avatars.githubusercontent.com/u/102910873?v=4", "gravatar_id": "", "url": "https://api.github.com/users/casm1", "type": "User", "user_view_type": "public", "site_admin": false }, "labels": { "0": { "id": 7006586802, "node_id": "LA_kwDOMBef5c8AAAABoaAHsg", "url": "https://api.github.com/repos/datawhalechina/happy-llm/labels/documentation", "name": "documentation", "color": "0075ca", "default": true, "description": "Improvements or additions to documentation" } }, "state": "open", "locked": false, "assignee": null, "assignees": {}, "milestone": null, "comments": 0, "created_at": "2025-09-13T13:51:47Z", "updated_at": "2025-09-15T03:07:56Z", "closed_at": null, "author_association": "NONE", "type": null, "active_lock_reason": null, "sub_issues_summary": { "total": 0, "completed": 0, "percent_completed": 0 }, "issue_dependencies_summary": { "blocked_by": 0, "total_blocked_by": 0, "blocking": 0, "total_blocking": 0 }, "body": "### 1. 遇到问题的章节 / Affected Chapter\n\nChapeter7.0\n\n### 2. 具体问题描述 / Problem Description\n\n1、get_chunk函数，在添加覆盖内容时：\n```\nprev_chunk = chunk_text[-1]\ncover_part = prev_chunk[-cover_content:] if len(prev_chunk) > cover_content else prev_chunk\nchunk_part = cover_part + chunk_part\n```\nlen(prev_chunk)统计的是分块文本的字符数，而cover_content表示需要覆盖的token数，这二者直接比较是否不正确？\n\n2、同样是get_chunk函数，在代码的124行，计算`curr_len = len(enc.encode(cover_part)) + 1 + line_len`，由于token_len已经减去了需要覆盖的token数，那么计算curr_len的时候是否不需要加上覆盖的token数`len(enc.encode(cover_part))`？\n\n### 3. 问题重现材料 / Reproduction Materials\n\n1、\n```\nprev_chunk = chunk_text[-1]\ncover_part = prev_chunk[-cover_content:] if len(prev_chunk) > cover_content else prev_chunk\nchunk_part = cover_part + chunk_part\n```\n建议修改为：\n```\nprev_chunk = chunk_text[-1]\nprev_chunk_tokens = enc.encode(prev_chunk)\ncover_part_tokens = prev_chunk_tokens[-cover_content:] if len(prev_chunk_tokens) > cover_content else prev_chunk_tokens\ncover_part = enc.decode(cover_part_tokens)\nchunk_part = cover_part + chunk_part\n```\n\n2、\n`curr_len = len(enc.encode(cover_part)) + 1 + line_len`\n建议修改为：\n`curr_len = 1 + line_len`\n\n### 确认事项 / Verification\n\n- [x] 此问题未在过往Issue中被报告过 / This issue hasn't been reported before", "closed_by": null, "reactions": { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/128/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 }, "performed_via_github_app": null, "state_reason": null, "linked_prs": [] }, { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/124", "id": 3385950968, "node_id": "I_kwDOMBef5c7J0YL4", "number": 124, "title": "[问题] 章节4.2.2：一个疑惑/ Chapter4.2.2: 有关于理解上的一个小问题，谢谢。", "user": { "login": "anliu2465-png", "id": 228142092, "node_id": "U_kgDODZksDA", "avatar_url": "https://avatars.githubusercontent.com/u/228142092?v=4", "gravatar_id": "", "url": "https://api.github.com/users/anliu2465-png", "type": "User", "user_view_type": "public", "site_admin": false }, "labels": { "0": { "id": 7006586802, "node_id": "LA_kwDOMBef5c8AAAABoaAHsg", "url": "https://api.github.com/repos/datawhalechina/happy-llm/labels/documentation", "name": "documentation", "color": "0075ca", "default": true, "description": "Improvements or additions to documentation" } }, "state": "open", "locked": false, "assignee": null, "assignees": {}, "milestone": null, "comments": 0, "created_at": "2025-09-05T01:10:49Z", "updated_at": "2025-09-05T01:10:49Z", "closed_at": null, "author_association": "NONE", "type": null, "active_lock_reason": null, "sub_issues_summary": { "total": 0, "completed": 0, "percent_completed": 0 }, "issue_dependencies_summary": { "blocked_by": 0, "total_blocked_by": 0, "blocking": 0, "total_blocking": 0 }, "body": "### 1. 遇到问题的章节 / Affected Chapter\n\nChapter4.2.2\n\n### 2. 具体问题描述 / Problem Description\n\n其实不是文档问题，是我有一个小的问题。就是关于图中表述 “即能够在未训练的指令上表现良好”，这里的 “未训练的指令” 具体指的是从哪个角度的 “未训练” 呢？一个角度是，微调数据集只有理化生领域的指令-回复对的数据集，这时候 “未训练”指的是语数外领域的指令-回复对？亦或是另外一个角度？？微调数据集中的一个指令是 “今天天气预报如何”，未训练指令指的是表达意思相同但是表达形式不同的 “今天天气怎么样” 吗？ 这是一个疑惑。请不吝指教，谢谢。\n\n\"Image\"\n\n### 3. 问题重现材料 / Reproduction Materials\n\n\"Image\"\n\n### 确认事项 / Verification\n\n- [x] 此问题未在过往Issue中被报告过 / This issue hasn't been reported before", "closed_by": null, "reactions": { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/124/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 }, "performed_via_github_app": null, "state_reason": null, "linked_prs": [] }, { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/123", "id": 3380283075, "node_id": "I_kwDOMBef5c7JewbD", "number": 123, "title": "[问题/Issue] 章节2.3.2：位置编码编写错误 / Chapter2.3.2: 对位置编码公式解读错误", "user": { "login": "anliu2465-png", "id": 228142092, "node_id": "U_kgDODZksDA", "avatar_url": "https://avatars.githubusercontent.com/u/228142092?v=4", "gravatar_id": "", "url": "https://api.github.com/users/anliu2465-png", "type": "User", "user_view_type": "public", "site_admin": false }, "labels": { "0": { "id": 7006586802, "node_id": "LA_kwDOMBef5c8AAAABoaAHsg", "url": "https://api.github.com/repos/datawhalechina/happy-llm/labels/documentation", "name": "documentation", "color": "0075ca", "default": true, "description": "Improvements or additions to documentation" } }, "state": "open", "locked": false, "assignee": null, "assignees": {}, "milestone": null, "comments": 1, "created_at": "2025-09-03T15:21:46Z", "updated_at": "2025-12-23T11:18:09Z", "closed_at": null, "author_association": "NONE", "type": null, "active_lock_reason": null, "sub_issues_summary": { "total": 0, "completed": 0, "percent_completed": 0 }, "issue_dependencies_summary": { "blocked_by": 0, "total_blocked_by": 0, "blocking": 0, "total_blocking": 0 }, "body": "### 1. 遇到问题的章节 / Affected Chapter\n\nChapter2.3.2\n\n### 2. 具体问题描述 / Problem Description\n\n内容错误：在图片中的红色选中部分。对于2i以及2i+1，是词向量的维度位置，而不是token在整个序列的位置。\n\n\"Image\"\n\n### 3. 问题重现材料 / Reproduction Materials\n\n同 “2. 具体问题描述”\n\n### 确认事项 / Verification\n\n- [x] 此问题未在过往Issue中被报告过 / This issue hasn't been reported before", "closed_by": null, "reactions": { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/123/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 }, "performed_via_github_app": null, "state_reason": null, "linked_prs": [] }, { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/122", "id": 3379684450, "node_id": "I_kwDOMBef5c7JceRi", "number": 122, "title": "[问题/Issue] 章节2.2.3：表达问题/ Chapter2.2.3: 数学语言表述上的问题", "user": { "login": "anliu2465-png", "id": 228142092, "node_id": "U_kgDODZksDA", "avatar_url": "https://avatars.githubusercontent.com/u/228142092?v=4", "gravatar_id": "", "url": "https://api.github.com/users/anliu2465-png", "type": "User", "user_view_type": "public", "site_admin": false }, "labels": { "0": { "id": 7006586802, "node_id": "LA_kwDOMBef5c8AAAABoaAHsg", "url": "https://api.github.com/repos/datawhalechina/happy-llm/labels/documentation", "name": "documentation", "color": "0075ca", "default": true, "description": "Improvements or additions to documentation" } }, "state": "open", "locked": false, "assignee": null, "assignees": {}, "milestone": null, "comments": 0, "created_at": "2025-09-03T12:38:07Z", "updated_at": "2025-09-03T12:38:07Z", "closed_at": null, "author_association": "NONE", "type": null, "active_lock_reason": null, "sub_issues_summary": { "total": 0, "completed": 0, "percent_completed": 0 }, "issue_dependencies_summary": { "blocked_by": 0, "total_blocked_by": 0, "blocking": 0, "total_blocking": 0 }, "body": "### 1. 遇到问题的章节 / Affected Chapter\n\nChapter2.2.3\n\n### 2. 具体问题描述 / Problem Description\n\n2.2.3 层归一化这一小节，文档的描述都是类似于“归一化为标准正态分布”，这种表述实际上是不正确的。\n\n\"Image\"\n\n### 3. 问题重现材料 / Reproduction Materials\n\n其实归一化最终是将变量变为了均值为0，标准差为1的变量而不是标准正态变量（具体在文档中指的就是特征/列，将特征归一化）。只有当样本量（具体在这里批归一化中就是指的m样本量）趋于+∞时，这个分布才约等于正态分布（专业表述叫做渐近正态分布）。背后的数学原理是中心极限定理。\n\n\"Image\"\n\n### 确认事项 / Verification\n\n- [x] 此问题未在过往Issue中被报告过 / This issue hasn't been reported before", "closed_by": null, "reactions": { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/122/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 }, "performed_via_github_app": null, "state_reason": null, "linked_prs": [] }, { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/121", "id": 3379263423, "node_id": "I_kwDOMBef5c7Ja3e_", "number": 121, "title": "[问题/Issue] 章节2.1：部分超参有误 / Chapter2.1: some parameters seem wrong", "user": { "login": "ShinnJinn", "id": 198151359, "node_id": "U_kgDOC8-Mvw", "avatar_url": "https://avatars.githubusercontent.com/u/198151359?v=4", "gravatar_id": "", "url": "https://api.github.com/users/ShinnJinn", "type": "User", "user_view_type": "public", "site_admin": false }, "labels": { "0": { "id": 7006586802, "node_id": "LA_kwDOMBef5c8AAAABoaAHsg", "url": "https://api.github.com/repos/datawhalechina/happy-llm/labels/documentation", "name": "documentation", "color": "0075ca", "default": true, "description": "Improvements or additions to documentation" } }, "state": "open", "locked": false, "assignee": null, "assignees": {}, "milestone": null, "comments": 1, "created_at": "2025-09-03T10:23:12Z", "updated_at": "2025-09-03T15:35:15Z", "closed_at": null, "author_association": "NONE", "type": null, "active_lock_reason": null, "sub_issues_summary": { "total": 0, "completed": 0, "percent_completed": 0 }, "issue_dependencies_summary": { "blocked_by": 0, "total_blocked_by": 0, "blocking": 0, "total_blocking": 0 }, "body": "### 1. 遇到问题的章节 / Affected Chapter\n\nChapter2.1 and Chapter2.2\n\n### 2. 具体问题描述 / Problem Description\n\n在 args.dim != args. n_embd 时\nChapter2.1.6 多头注意力 中\n`self.wo = nn.Linear(self.n_heads * self.head_dim, args.dim, bias=False)`\n输出维度有误\n\nChapter2.2.5 Encoder 以及 2.2.6 Decoder中\n`self.feed_forward = MLP(args.dim, args.dim, args.dropout)`\n输入、输出维度均有误\n\n此时二者的输出结果无法和初始x做残差\n\n\n### 3. 问题重现材料 / Reproduction Materials\n\nChapter2.1.6 多头注意力 中\n`self.wo = nn.Linear(self.n_heads * self.head_dim, args.dim, bias=False)`\n改为\n`self.wo = nn.Linear(self.n_heads * self.head_dim, args.n_embd, bias=False)`\n\nChapter2.2.5 Encoder 以及 2.2.6 Decoder中\n`self.feed_forward = MLP(args.dim, args.dim, args.dropout)`\n改为\n`self.feed_forward = MLP(args.n_embd, args.n_embd, args.dropout)`\n\n\n### 确认事项 / Verification\n\n- [x] 此问题未在过往Issue中被报告过 / This issue hasn't been reported before", "closed_by": null, "reactions": { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/121/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 }, "performed_via_github_app": null, "state_reason": null, "linked_prs": [] }, { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/120", "id": 3377973020, "node_id": "I_kwDOMBef5c7JV8cc", "number": 120, "title": "[问题/Issue] 章节2.1：简短问题描述 / Chapter2.1: attention函数没有问题。但是2.1.4小节的那一行代码中attention函数的输入参数感觉有问题", "user": { "login": "anliu2465-png", "id": 228142092, "node_id": "U_kgDODZksDA", "avatar_url": "https://avatars.githubusercontent.com/u/228142092?v=4", "gravatar_id": "", "url": "https://api.github.com/users/anliu2465-png", "type": "User", "user_view_type": "public", "site_admin": false }, "labels": { "0": { "id": 7006586802, "node_id": "LA_kwDOMBef5c8AAAABoaAHsg", "url": "https://api.github.com/repos/datawhalechina/happy-llm/labels/documentation", "name": "documentation", "color": "0075ca", "default": true, "description": "Improvements or additions to documentation" } }, "state": "open", "locked": false, "assignee": null, "assignees": {}, "milestone": null, "comments": 6, "created_at": "2025-09-03T01:21:04Z", "updated_at": "2025-09-25T07:56:53Z", "closed_at": null, "author_association": "NONE", "type": null, "active_lock_reason": null, "sub_issues_summary": { "total": 0, "completed": 0, "percent_completed": 0 }, "issue_dependencies_summary": { "blocked_by": 0, "total_blocked_by": 0, "blocking": 0, "total_blocking": 0 }, "body": "### 1. 遇到问题的章节 / Affected Chapter\n\nChapter2.1\n\n### 2. 具体问题描述 / Problem Description\n\n在2.1.4 自注意力小节，代码部分为什么是attention(x,x,x)呢？我的理解是：这里QKV矩阵的输入X尽管相同，但是权重矩阵不同W_Q、W_K、W_V。而且attention函数的输入参数不是QKV矩阵吗，怎么到这里成了输入是x了？\n\n### 3. 问题重现材料 / Reproduction Materials\n\n```python\n# attention 为上文定义的注意力计算函数\nattention(x, x, x)\n```\n```建议修改：\nattention(XW_Q,XW_K,XW_V)\n···\n\n### 确认事项 / Verification\n\n- [x] 此问题未在过往Issue中被报告过 / This issue hasn't been reported before", "closed_by": null, "reactions": { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/120/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 }, "performed_via_github_app": null, "state_reason": null, "linked_prs": [] }, { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/119", "id": 3363679368, "node_id": "I_kwDOMBef5c7IfayI", "number": 119, "title": "[修改建议] 章节5.3：SFTDataset->generate_loss_mask函数", "user": { "login": "chengyuZou", "id": 177918948, "node_id": "U_kgDOCprT5A", "avatar_url": "https://avatars.githubusercontent.com/u/177918948?v=4", "gravatar_id": "", "url": "https://api.github.com/users/chengyuZou", "type": "User", "user_view_type": "public", "site_admin": false }, "labels": { "0": { "id": 7006586802, "node_id": "LA_kwDOMBef5c8AAAABoaAHsg", "url": "https://api.github.com/repos/datawhalechina/happy-llm/labels/documentation", "name": "documentation", "color": "0075ca", "default": true, "description": "Improvements or additions to documentation" } }, "state": "open", "locked": false, "assignee": { "login": "KMnO4-zx", "id": 77671993, "node_id": "MDQ6VXNlcjc3NjcxOTkz", "avatar_url": "https://avatars.githubusercontent.com/u/77671993?v=4", "gravatar_id": "", "url": "https://api.github.com/users/KMnO4-zx", "type": "User", "user_view_type": "public", "site_admin": false }, "assignees": { "0": { "login": "KMnO4-zx", "id": 77671993, "node_id": "MDQ6VXNlcjc3NjcxOTkz", "avatar_url": "https://avatars.githubusercontent.com/u/77671993?v=4", "gravatar_id": "", "url": "https://api.github.com/users/KMnO4-zx", "type": "User", "user_view_type": "public", "site_admin": false } }, "milestone": null, "comments": 0, "created_at": "2025-08-28T15:01:24Z", "updated_at": "2025-08-29T01:42:58Z", "closed_at": null, "author_association": "NONE", "type": null, "active_lock_reason": null, "sub_issues_summary": { "total": 0, "completed": 0, "percent_completed": 0 }, "issue_dependencies_summary": { "blocked_by": 0, "total_blocked_by": 0, "blocking": 0, "total_blocking": 0 }, "body": "### 1. 遇到问题的章节 / Affected Chapter\n\nChapter5.3 ->SFTDataset->generate_loss_mask函数\n\n### 2. 具体问题描述 / Problem Description\n\n代码这样改是否更简洁？\ndef __init__() 添加：\n self.bos_token_id = self.tokenizer('<|im_start|>assistant' , add_special_tokens = False)['input_ids']\n self.eos_token_id = self.tokenizer('<|im_end|>' , add_special_tokens=False)['input_ids']\n\n\"Image\"\n\n以及，是否能用KMP算法对时间进行优化？O(n^2) -> O(n * log n)\n\n### 3. 问题重现材料 / Reproduction Materials\n\n无\n\n### 确认事项 / Verification\n\n", "closed_by": null, "reactions": { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/119/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 }, "performed_via_github_app": null, "state_reason": null, "linked_prs": [] }, { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/115", "id": 3343974161, "node_id": "I_kwDOMBef5c7HUP8R", "number": 115, "title": "[Issue] 章节5：预训练模型代码和训练tokenizer代码", "user": { "login": "Yapeng-Gao", "id": 118376247, "node_id": "U_kgDOBw5HNw", "avatar_url": "https://avatars.githubusercontent.com/u/118376247?v=4", "gravatar_id": "", "url": "https://api.github.com/users/Yapeng-Gao", "type": "User", "user_view_type": "public", "site_admin": false }, "labels": { "0": { "id": 7006586802, "node_id": "LA_kwDOMBef5c8AAAABoaAHsg", "url": "https://api.github.com/repos/datawhalechina/happy-llm/labels/documentation", "name": "documentation", "color": "0075ca", "default": true, "description": "Improvements or additions to documentation" } }, "state": "open", "locked": false, "assignee": { "login": "KMnO4-zx", "id": 77671993, "node_id": "MDQ6VXNlcjc3NjcxOTkz", "avatar_url": "https://avatars.githubusercontent.com/u/77671993?v=4", "gravatar_id": "", "url": "https://api.github.com/users/KMnO4-zx", "type": "User", "user_view_type": "public", "site_admin": false }, "assignees": { "0": { "login": "KMnO4-zx", "id": 77671993, "node_id": "MDQ6VXNlcjc3NjcxOTkz", "avatar_url": "https://avatars.githubusercontent.com/u/77671993?v=4", "gravatar_id": "", "url": "https://api.github.com/users/KMnO4-zx", "type": "User", "user_view_type": "public", "site_admin": false } }, "milestone": null, "comments": 0, "created_at": "2025-08-22T03:16:52Z", "updated_at": "2025-08-22T06:22:17Z", "closed_at": null, "author_association": "NONE", "type": null, "active_lock_reason": null, "sub_issues_summary": { "total": 0, "completed": 0, "percent_completed": 0 }, "issue_dependencies_summary": { "blocked_by": 0, "total_blocked_by": 0, "blocking": 0, "total_blocking": 0 }, "body": "### 1. 遇到问题的章节 / Affected Chapter\n\nchapter5.2 chapter5.3\n\n### 2. 具体问题描述 / Problem Description\n\n代码问题 \n1.日志太简略\n2.train_tokenizer.py 代码做了优化\n3.预训练时kmodel vocab_size和tokenizer没对齐,我训练出来的时8192,kmodel也需要对应时这个问题\n4.ddp_pretrain.py 种没有warmup,我优化了代码\n5.可以用deepspeed优化训练,我用的stage2+warmup,有完整代码\n内容问题\n用conda虚拟环境训练需要安装g++\n\ntrain_tokenizer.py和ddp_pretrain.py 我都做了优化,可以给我权限,我提个pr吗\n\n### 3. 问题重现材料 / Reproduction Materials\n\ngcc: fatal error: cannot execute ‘cc1plus’: execvp: No such file or directory\n\n虚拟环境要安装g++\n\n\n\nvectorized_gather_kernel: Assertion `ind >=0 && ind < ind_dim_size` failed\n索引越界,时kmodel中vocab_size=6144导致,和tokenizer对齐即可8192\n\n### 确认事项 / Verification\n\n- [x] 此问题未在过往Issue中被报告过 / This issue hasn't been reported before", "closed_by": null, "reactions": { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/115/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 }, "performed_via_github_app": null, "state_reason": null, "linked_prs": [] }, { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/113", "id": 3341058974, "node_id": "I_kwDOMBef5c7HJIOe", "number": 113, "title": "[建议] 增加一个附录", "user": { "login": "lin-dongyizhi7", "id": 56852070, "node_id": "MDQ6VXNlcjU2ODUyMDcw", "avatar_url": "https://avatars.githubusercontent.com/u/56852070?v=4", "gravatar_id": "", "url": "https://api.github.com/users/lin-dongyizhi7", "type": "User", "user_view_type": "public", "site_admin": false }, "labels": { "0": { "id": 7006586802, "node_id": "LA_kwDOMBef5c8AAAABoaAHsg", "url": "https://api.github.com/repos/datawhalechina/happy-llm/labels/documentation", "name": "documentation", "color": "0075ca", "default": true, "description": "Improvements or additions to documentation" } }, "state": "open", "locked": false, "assignee": null, "assignees": {}, "milestone": null, "comments": 1, "created_at": "2025-08-21T09:45:10Z", "updated_at": "2025-08-21T12:10:59Z", "closed_at": null, "author_association": "NONE", "type": null, "active_lock_reason": null, "sub_issues_summary": { "total": 0, "completed": 0, "percent_completed": 0 }, "issue_dependencies_summary": { "blocked_by": 0, "total_blocked_by": 0, "blocking": 0, "total_blocking": 0 }, "body": "### 1. 遇到问题的章节 / Affected Chapter\n\n教程开头\n\n### 2. 具体问题描述 / Problem Description\n\n可以加一个附录，标明文档里出现的各种缩写的含义等等...\n\n### 3. 问题重现材料 / Reproduction Materials\n\n像这样...... 方便小白查阅\n### M\n\n- **MASK**: 掩码特殊标记，用于 MLM 等任务。\n- **Megatron-LM**: 大模型分布式训练框架（张量并行等）。\n- **MLM**: Masked Language Model（掩码语言模型）- 遮蔽预测 token。\n- **MLP**: Multi-Layer Perceptron - 多层感知机/前馈网络。\n- **MP**: Model Parallel（模型并行）。\n- **MQA**: Multi-Query Attention - 多头共享单组 KV，降显存/通信。\n- **MT**: Machine Translation（机器翻译）。\n- **MultiNLI**: Multi-Genre Natural Language Inference - 跨域自然语言推理数据集。\n\n### N\n\n- **N-gram**: 基于马尔可夫假设的统计语言模型。\n- **NER**: Named Entity Recognition（命名实体识别）。\n- **NLG**: Natural Language Generation（自然语言生成）。\n- **NLU**: Natural Language Understanding（自然语言理解）。\n- **NLP**: Natural Language Processing（自然语言处理）。\n- **NNLM**: Neural Network Language Model（神经网络语言模型）。\n- **NSP**: Next Sentence Prediction（下一句预测）。\n\n### 确认事项 / Verification\n\n- [x] 此问题未在过往Issue中被报告过 / This issue hasn't been reported before", "closed_by": null, "reactions": { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/113/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 }, "performed_via_github_app": null, "state_reason": null, "linked_prs": [] }, { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/112", "id": 3338276115, "node_id": "I_kwDOMBef5c7G-g0T", "number": 112, "title": "[问题] 章节6：希望给出requirements.txt中各个包的版本", "user": { "login": "tcy-THU", "id": 91608012, "node_id": "U_kgDOBXXTzA", "avatar_url": "https://avatars.githubusercontent.com/u/91608012?v=4", "gravatar_id": "", "url": "https://api.github.com/users/tcy-THU", "type": "User", "user_view_type": "public", "site_admin": false }, "labels": { "0": { "id": 7006586802, "node_id": "LA_kwDOMBef5c8AAAABoaAHsg", "url": "https://api.github.com/repos/datawhalechina/happy-llm/labels/documentation", "name": "documentation", "color": "0075ca", "default": true, "description": "Improvements or additions to documentation" } }, "state": "open", "locked": false, "assignee": null, "assignees": {}, "milestone": null, "comments": 2, "created_at": "2025-08-20T14:01:53Z", "updated_at": "2025-09-12T09:03:39Z", "closed_at": null, "author_association": "NONE", "type": null, "active_lock_reason": null, "sub_issues_summary": { "total": 0, "completed": 0, "percent_completed": 0 }, "issue_dependencies_summary": { "blocked_by": 0, "total_blocked_by": 0, "blocking": 0, "total_blocking": 0 }, "body": "### 1. 遇到问题的章节 / Affected Chapter\n\nChapter6\n\n### 2. 具体问题描述 / Problem Description\n\ntorchdata这个包的0.11.0版本似乎没有datapipes这个类了，但是第六章给出的代码还包含。由于requirements.txt没有注明这些包都是哪个版本的，直接pip install -r requirements.txt的时候，相当于就是之间安装了这些包的最新版，就会在finetune.py的from torchdata.datapipes.iter import IterableWrapper这一行报错。\n建议在requirements.txt中标注清楚版本，方便其他用户直接照着安装\n\n### 3. 问题重现材料 / Reproduction Materials\n\nfrom torchdata.datapipes.iter import IterableWrapper\n\n\n### 确认事项 / Verification\n\n- [x] 此问题未在过往Issue中被报告过 / This issue hasn't been reported before", "closed_by": null, "reactions": { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/112/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 }, "performed_via_github_app": null, "state_reason": null, "linked_prs": [] }, { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/111", "id": 3330294177, "node_id": "I_kwDOMBef5c7GgEGh", "number": 111, "title": "[问题/Issue] 章节X.X：简短问题描述 / Chapter7.Agent: 换其他模型平台报错", "user": { "login": "CharmLynn", "id": 4971220, "node_id": "MDQ6VXNlcjQ5NzEyMjA=", "avatar_url": "https://avatars.githubusercontent.com/u/4971220?v=4", "gravatar_id": "", "url": "https://api.github.com/users/CharmLynn", "type": "User", "user_view_type": "public", "site_admin": false }, "labels": { "0": { "id": 7006586802, "node_id": "LA_kwDOMBef5c8AAAABoaAHsg", "url": "https://api.github.com/repos/datawhalechina/happy-llm/labels/documentation", "name": "documentation", "color": "0075ca", "default": true, "description": "Improvements or additions to documentation" } }, "state": "open", "locked": false, "assignee": null, "assignees": {}, "milestone": null, "comments": 3, "created_at": "2025-08-18T11:22:02Z", "updated_at": "2025-08-23T12:33:28Z", "closed_at": null, "author_association": "NONE", "type": null, "active_lock_reason": null, "sub_issues_summary": { "total": 0, "completed": 0, "percent_completed": 0 }, "issue_dependencies_summary": { "blocked_by": 0, "total_blocked_by": 0, "blocking": 0, "total_blocking": 0 }, "body": "### 1. 遇到问题的章节 / Affected Chapter\n\nChapter7/Agent\n\n### 2. 具体问题描述 / Problem Description\n\n硅基流动换成阿里百炼之后报错“Messages with role 'tool' must be a response to a preceding message with 'tool_calls'” 哪怕是同一个模型\n\n### 3. 问题重现材料 / Reproduction Materials\n\nopenai.BadRequestError: Error code: 400 - {'error': {'code': 'invalid_parameter_error', 'param': None, 'message': '<400> InternalError.Algo.InvalidParameter: messages with role \"tool\" must be a response to a preceeding message with \"tool_calls\".', 'type': 'invalid_request_error'}, 'id': 'chatcmpl-2ba53703-822e-9107-a1e6-a3b7c109317a', 'request_id': '2ba53703-822e-9107-a1e6-a3b7c109317a'}\n\nFile \"/Users/admin/OpenSource/happy-llm/docs/chapter7/Agent/web_demo.py\", line 60, in \n response = agent.get_completion(prompt) # 获取Agent的响应\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nFile \"/Users/admin/OpenSource/happy-llm/docs/chapter7/Agent/src/core.py\", line 64, in get_completion\n response = self.client.chat.completions.create(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nFile \"/opt/homebrew/lib/python3.11/site-packages/openai/_utils/_utils.py\", line 287, in wrapper\n return func(*args, **kwargs)\n ^^^^^^^^^^^^^^^^^^^^^\nFile \"/opt/homebrew/lib/python3.11/site-packages/openai/resources/chat/completions/completions.py\", line 925, in create\n return self._post(\n ^^^^^^^^^^^\nFile \"/opt/homebrew/lib/python3.11/site-packages/openai/_base_client.py\", line 1249, in post\n return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nFile \"/opt/homebrew/lib/python3.11/site-packages/openai/_base_client.py\", line 1037, in request\n raise self._make_status_error_from_response(err.response) from None\n\n### 确认事项 / Verification\n\n- [x] 此问题未在过往Issue中被报告过 / This issue hasn't been reported before", "closed_by": null, "reactions": { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/111/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 }, "performed_via_github_app": null, "state_reason": null, "linked_prs": [] }, { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/110", "id": 3307120216, "node_id": "I_kwDOMBef5c7FHqZY", "number": 110, "title": "通用：编写自动脚本，将 md 文件，转换为了 ipynb 格式文件，方便在Colab 上，实践 happy-llm", "user": { "login": "ningg", "id": 5243076, "node_id": "MDQ6VXNlcjUyNDMwNzY=", "avatar_url": "https://avatars.githubusercontent.com/u/5243076?v=4", "gravatar_id": "", "url": "https://api.github.com/users/ningg", "type": "User", "user_view_type": "public", "site_admin": false }, "labels": { "0": { "id": 7006586802, "node_id": "LA_kwDOMBef5c8AAAABoaAHsg", "url": "https://api.github.com/repos/datawhalechina/happy-llm/labels/documentation", "name": "documentation", "color": "0075ca", "default": true, "description": "Improvements or additions to documentation" }, "1": { "id": 7006586816, "node_id": "LA_kwDOMBef5c8AAAABoaAHwA", "url": "https://api.github.com/repos/datawhalechina/happy-llm/labels/good%20first%20issue", "name": "good first issue", "color": "7057ff", "default": true, "description": "Good for newcomers" } }, "state": "open", "locked": false, "assignee": null, "assignees": {}, "milestone": null, "comments": 1, "created_at": "2025-08-10T03:16:24Z", "updated_at": "2025-08-16T14:09:48Z", "closed_at": null, "author_association": "NONE", "type": null, "active_lock_reason": null, "sub_issues_summary": { "total": 0, "completed": 0, "percent_completed": 0 }, "issue_dependencies_summary": { "blocked_by": 0, "total_blocked_by": 0, "blocking": 0, "total_blocking": 0 }, "body": "### 1. 遇到问题的章节 / Affected Chapter\n\nREADME\n\n### 2. 具体问题描述 / Problem Description\n\n通用：编写自动脚本，将 md 文件，转换为了 ipynb 格式文件，方便在Colab 上，实践 happy-llm\n\n为了保持现有 [happy-llm](https://github.com/datawhalechina/happy-llm) 整洁，建了独立的仓库： [happy-llm-colab](https://github.com/ningg/happy-llm-colab?tab=readme-ov-file)\n\n### 3. 问题重现材料 / Reproduction Materials\n\n无\n\n### 确认事项 / Verification\n\n- [x] 此问题未在过往Issue中被报告过 / This issue hasn't been reported before", "closed_by": null, "reactions": { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/110/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 }, "performed_via_github_app": null, "state_reason": null, "linked_prs": [] }, { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/109", "id": 3304663518, "node_id": "I_kwDOMBef5c7E-Sne", "number": 109, "title": "【问题】章节2", "user": { "login": "ht426", "id": 95162046, "node_id": "U_kgDOBawOvg", "avatar_url": "https://avatars.githubusercontent.com/u/95162046?v=4", "gravatar_id": "", "url": "https://api.github.com/users/ht426", "type": "User", "user_view_type": "public", "site_admin": false }, "labels": { "0": { "id": 7006586802, "node_id": "LA_kwDOMBef5c8AAAABoaAHsg", "url": "https://api.github.com/repos/datawhalechina/happy-llm/labels/documentation", "name": "documentation", "color": "0075ca", "default": true, "description": "Improvements or additions to documentation" } }, "state": "open", "locked": false, "assignee": null, "assignees": {}, "milestone": null, "comments": 1, "created_at": "2025-08-08T16:52:13Z", "updated_at": "2025-08-12T02:13:14Z", "closed_at": null, "author_association": "NONE", "type": null, "active_lock_reason": null, "sub_issues_summary": { "total": 0, "completed": 0, "percent_completed": 0 }, "issue_dependencies_summary": { "blocked_by": 0, "total_blocked_by": 0, "blocking": 0, "total_blocking": 0 }, "body": "### 1. 遇到问题的章节 / Affected Chapter\n\nChapter2\n\n### 2. 具体问题描述 / Problem Description\n\ntransformer.py文件中decoder的输入怎么和encoder一样阿，并且没有看明白这个文件中想要利用transformer做的任务是什么\n\n### 3. 问题重现材料 / Reproduction Materials\n\ntransformer.py文件中decoder的输入怎么和encoder一样阿，并且没有看明白这个文件中想要利用transformer做的任务是什么\n\n### 确认事项 / Verification\n\n- [x] 此问题未在过往Issue中被报告过 / This issue hasn't been reported before", "closed_by": null, "reactions": { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/109/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 }, "performed_via_github_app": null, "state_reason": null, "linked_prs": [] }, { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/106", "id": 3288786538, "node_id": "I_kwDOMBef5c7EBuZq", "number": 106, "title": "[问题/Issue] 章节6.1：简短问题描述 / Chapter6.1: out of memory", "user": { "login": "OCEANOUXIN", "id": 90253968, "node_id": "MDQ6VXNlcjkwMjUzOTY4", "avatar_url": "https://avatars.githubusercontent.com/u/90253968?v=4", "gravatar_id": "", "url": "https://api.github.com/users/OCEANOUXIN", "type": "User", "user_view_type": "public", "site_admin": false }, "labels": { "0": { "id": 7006586802, "node_id": "LA_kwDOMBef5c8AAAABoaAHsg", "url": "https://api.github.com/repos/datawhalechina/happy-llm/labels/documentation", "name": "documentation", "color": "0075ca", "default": true, "description": "Improvements or additions to documentation" } }, "state": "open", "locked": false, "assignee": null, "assignees": {}, "milestone": null, "comments": 3, "created_at": "2025-08-04T09:59:37Z", "updated_at": "2025-10-14T07:35:14Z", "closed_at": null, "author_association": "NONE", "type": null, "active_lock_reason": null, "sub_issues_summary": { "total": 0, "completed": 0, "percent_completed": 0 }, "issue_dependencies_summary": { "blocked_by": 0, "total_blocked_by": 0, "blocking": 0, "total_blocking": 0 }, "body": "### 1. 遇到问题的章节 / Affected Chapter\n\n Chapter6.1\n\n### 2. 具体问题描述 / Problem Description\n\n想问一下这个qwen2.5-1.5B需要多大显存，我4张48G显卡oom\n\n### 3. 问题重现材料 / Reproduction Materials\n\noom\n\n### 确认事项 / Verification\n\n- [x] 此问题未在过往Issue中被报告过 / This issue hasn't been reported before", "closed_by": null, "reactions": { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/106/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 }, "performed_via_github_app": null, "state_reason": null, "linked_prs": [] }, { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/90", "id": 3242469052, "node_id": "I_kwDOMBef5c7BRCa8", "number": 90, "title": "[问题] 章节4：LLM发展历程同步到至今", "user": { "login": "JackyYangPassion", "id": 13795366, "node_id": "MDQ6VXNlcjEzNzk1MzY2", "avatar_url": "https://avatars.githubusercontent.com/u/13795366?v=4", "gravatar_id": "", "url": "https://api.github.com/users/JackyYangPassion", "type": "User", "user_view_type": "public", "site_admin": false }, "labels": { "0": { "id": 7006586802, "node_id": "LA_kwDOMBef5c8AAAABoaAHsg", "url": "https://api.github.com/repos/datawhalechina/happy-llm/labels/documentation", "name": "documentation", "color": "0075ca", "default": true, "description": "Improvements or additions to documentation" } }, "state": "open", "locked": false, "assignee": null, "assignees": {}, "milestone": null, "comments": 2, "created_at": "2025-07-18T08:57:39Z", "updated_at": "2025-07-25T08:21:44Z", "closed_at": null, "author_association": "NONE", "type": null, "active_lock_reason": null, "sub_issues_summary": { "total": 0, "completed": 0, "percent_completed": 0 }, "issue_dependencies_summary": { "blocked_by": 0, "total_blocked_by": 0, "blocking": 0, "total_blocking": 0 }, "body": "### 1. 遇到问题的章节 / Affected Chapter\n\nChapter4\n\n### 2. 具体问题描述 / Problem Description\n\n发展历程同步到至今\n\n### 3. 问题重现材料 / Reproduction Materials\n\n\"Image\"\n发展历程 可以更新到至今\n202507 Kimi-K2\n\n### 确认事项 / Verification\n\n- [x] 此问题未在过往Issue中被报告过 / This issue hasn't been reported before", "closed_by": null, "reactions": { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/90/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 }, "performed_via_github_app": null, "state_reason": null, "linked_prs": [] }, { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/72", "id": 3208894559, "node_id": "I_kwDOMBef5c6_Q9hf", "number": 72, "title": "修改建议：方便初学者了解全貌的Flowchart", "user": { "login": "pingfanfan", "id": 155778871, "node_id": "U_kgDOCUj_Nw", "avatar_url": "https://avatars.githubusercontent.com/u/155778871?v=4", "gravatar_id": "", "url": "https://api.github.com/users/pingfanfan", "type": "User", "user_view_type": "public", "site_admin": false }, "labels": { "0": { "id": 7006586802, "node_id": "LA_kwDOMBef5c8AAAABoaAHsg", "url": "https://api.github.com/repos/datawhalechina/happy-llm/labels/documentation", "name": "documentation", "color": "0075ca", "default": true, "description": "Improvements or additions to documentation" }, "1": { "id": 7006586810, "node_id": "LA_kwDOMBef5c8AAAABoaAHug", "url": "https://api.github.com/repos/datawhalechina/happy-llm/labels/enhancement", "name": "enhancement", "color": "a2eeef", "default": true, "description": "New feature or request" } }, "state": "open", "locked": false, "assignee": null, "assignees": {}, "milestone": null, "comments": 1, "created_at": "2025-07-07T12:45:53Z", "updated_at": "2025-07-08T02:37:49Z", "closed_at": null, "author_association": "NONE", "type": null, "active_lock_reason": null, "sub_issues_summary": { "total": 0, "completed": 0, "percent_completed": 0 }, "issue_dependencies_summary": { "blocked_by": 0, "total_blocked_by": 0, "blocking": 0, "total_blocking": 0 }, "body": "### 1. 遇到问题的章节 / Affected Chapter\n\n教程开头\n\n### 2. 具体问题描述 / Problem Description\n\n可以加一个流程图\n\n### 3. 问题重现材料 / Reproduction Materials\n\n示例Mermaid流程图\n\ngraph TD\n subgraph \" \"\n direction LR\n A[▶️ 开始学习]\n end\n\n subgraph \"第一部分：基础理论 (Chapters 1-4)\"\n direction TB\n B[Ch1: 课程导论与环境配置] --> C(理解LLM基本概念) --> D(配置Python和PyTorch环境);\n D --> E[Ch2: 深入理解Transformer];\n E --> F(学习自注意力机制 Self-Attention) --> G(掌握多头注意力机制 Multi-Head Attention);\n G --> H(理解位置编码与前馈网络) --> I(动手实现一个Transformer模块);\n I --> J[Ch3: 预训练语言模型];\n J --> K(学习BERT、GPT等经典模型思想) --> L(理解“预训练-微调”范式);\n L --> M[Ch4: 主流大模型巡礼];\n M --> N(分析LLaMA、GPT、ChatGLM等模型架构);\n end\n\n subgraph \"第二部分：核心实践 (Chapters 5-6)\"\n direction TB\n O[Ch5: 从零动手实现LLaMA2];\n O --> P(Step 1: 准备数据集和分词器);\n P --> Q(Step 2: 用PyTorch搭建LLaMA2模型代码);\n Q --> R(Step 3: 实现模型预训练 Pre-training);\n R --> S(Step 4: 实现有监督微调 SFT);\n S --> T[Ch6: 实战Hugging Face Transformers框架];\n T --> U(学习使用Trainer API进行高效训练);\n U --> V(掌握SFTTrainer等工具简化微调);\n end\n\n subgraph \"第三部分：前沿应用 (Chapter 7)\"\n direction TB\n W[Ch7: LLM评估与前沿应用];\n W --> X(学习LLM的常用评测方法);\n X --> Y(核心应用1: 学习并实践RAG技术);\n Y --> Z(核心应用2: 学习并实践Agent技术);\n end\n\n subgraph \" \"\n direction LR\n AA[✅ 课程完成]\n end\n\n %% 连接各大模块\n A --> B;\n N --> O;\n V --> W;\n Z --> AA;\n\n### 确认事项 / Verification\n\n- [x] 此问题未在过往Issue中被报告过 / This issue hasn't been reported before", "closed_by": null, "reactions": { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/72/reactions", "total_count": 1, "+1": 1, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 }, "performed_via_github_app": null, "state_reason": null, "linked_prs": [] }, { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/59", "id": 3194288539, "node_id": "I_kwDOMBef5c6-ZPmb", "number": 59, "title": "[问题/Issue] 章节5：LLM模型预训练过程中内存不断增长", "user": { "login": "fengyanzi", "id": 110722284, "node_id": "U_kgDOBpl87A", "avatar_url": "https://avatars.githubusercontent.com/u/110722284?v=4", "gravatar_id": "", "url": "https://api.github.com/users/fengyanzi", "type": "User", "user_view_type": "public", "site_admin": false }, "labels": { "0": { "id": 7006586802, "node_id": "LA_kwDOMBef5c8AAAABoaAHsg", "url": "https://api.github.com/repos/datawhalechina/happy-llm/labels/documentation", "name": "documentation", "color": "0075ca", "default": true, "description": "Improvements or additions to documentation" } }, "state": "open", "locked": false, "assignee": { "login": "KMnO4-zx", "id": 77671993, "node_id": "MDQ6VXNlcjc3NjcxOTkz", "avatar_url": "https://avatars.githubusercontent.com/u/77671993?v=4", "gravatar_id": "", "url": "https://api.github.com/users/KMnO4-zx", "type": "User", "user_view_type": "public", "site_admin": false }, "assignees": { "0": { "login": "KMnO4-zx", "id": 77671993, "node_id": "MDQ6VXNlcjc3NjcxOTkz", "avatar_url": "https://avatars.githubusercontent.com/u/77671993?v=4", "gravatar_id": "", "url": "https://api.github.com/users/KMnO4-zx", "type": "User", "user_view_type": "public", "site_admin": false } }, "milestone": null, "comments": 1, "created_at": "2025-07-02T02:14:25Z", "updated_at": "2025-07-04T01:13:46Z", "closed_at": null, "author_association": "NONE", "type": null, "active_lock_reason": null, "sub_issues_summary": { "total": 0, "completed": 0, "percent_completed": 0 }, "issue_dependencies_summary": { "blocked_by": 0, "total_blocked_by": 0, "blocking": 0, "total_blocking": 0 }, "body": "### 1. 遇到问题的章节 / Affected Chapter\n\nChapter6\n\n### 2. 具体问题描述 / Problem Description\n\n模型预训练过程中System Memory Utilization不断增长，直至100%进程被杀死，怀疑是dataset加载写法有问题\n部分原因也是因为我将num_workers设置为了32，但减小只能缓解不能根本上解决问题。\n实验配置H200*2+256G内存\n\ndeepseek给出的解析可以部分参考一下，部分回答存在歧义：\n从你提供的代码和描述来看，系统内存利用率不断增长直至崩溃的问题可能与内存泄漏有关。以下是可能的原因和解决方案：\n\n主要问题分析\n数据集加载方式：\n\n你一次性将所有数据加载到内存中（self.data = f.readlines()），这对于大文件会占用大量内存\n\n每次迭代时都会创建新的numpy数组和torch张量，这些可能没有及时释放\nDataLoader配置：\n\nnum_workers=32 设置过高，每个worker都会复制数据集，可能导致内存爆炸\n\npin_memory=True 会锁定内存页，可能增加内存压力\n梯度累积：\n\n虽然梯度累积本身不会直接导致内存泄漏，但在长时间训练中可能加剧内存问题\n\n解决方案\n优化数据集加载：\n\nclass PretrainDataset(Dataset):\n def __init__(self, data_path, tokenizer, max_length=512):\n super().__init__()\n self.data_path = data_path\n self.tokenizer = tokenizer\n self.max_length = max_length\n self.padding = 0\n # 只存储行数，不加载全部内容\n with open(data_path, 'r', encoding='utf-8') as f:\n self.length = sum(1 for _ in f)\n \n def __len__(self):\n return self.length\n\n def __getitem__(self, index: int):\n # 按需读取单行\n with open(self.data_path, 'r', encoding='utf-8') as f:\n for i, line in enumerate(f):\n if i == index:\n sample = json.loads(line)\n break\n # 其余代码保持不变...\n\n调整DataLoader参数：\n\ntrain_loader = DataLoader(\n train_ds,\n batch_size=args.batch_size,\n pin_memory=False, # 禁用pin_memory\n drop_last=False,\n shuffle=True,\n num_workers=4, # 减少worker数量\n persistent_workers=True # 保持worker进程存活\n)\n\n添加内存监控：\n\n在训练循环中添加内存监控代码：\nimport psutil\n\ndef train_epoch(epoch):\n # ...\n for step, (X, Y, loss_mask) in enumerate(train_loader):\n # 打印内存使用情况\n mem = psutil.virtual_memory()\n Logger(f\"Memory usage: {mem.percent}% (used: {mem.used/1024/1024:.2f}MB)\")\n # 其余代码...\n\n定期清理缓存：\n\nimport gc\n\ndef train_epoch(epoch):\n # ...\n if step % 100 == 0:\n torch.cuda.empty_cache()\n gc.collect()\n\n使用生成器式数据集：\n\n考虑使用IterableDataset替代Dataset，特别是对于非常大的数据集：\nfrom torch.utils.data import IterableDataset\n\nclass PretrainIterableDataset(IterableDataset):\n def __init__(self, data_path, tokenizer, max_length=512):\n super().__init__()\n self.data_path = data_path\n self.tokenizer = tokenizer\n self.max_length = max_length\n self.padding = 0\n\n def __iter__(self):\n with open(self.data_path, 'r', encoding='utf-8') as f:\n for line in f:\n sample = json.loads(line)\n # 处理逻辑与之前相同...\n yield X, Y, loss_mask\n\n其他建议\n减小验证频率：\n\n减少保存检查点的频率（如每5000步改为每10000步）\n混合精度训练：\n\n确保正确使用混合精度训练，减少显存占用\n分批处理大文件：\n\n如果数据文件非常大，考虑先将其分割成多个小文件\n监控GPU内存：\n\n添加GPU内存监控：torch.cuda.memory_allocated()/1024/1024\n\n通过这些修改，应该能显著降低内存使用量并防止内存泄漏。如果问题仍然存在，建议使用内存分析工具（如memory_profiler）来定位具体的内存泄漏点。\n\n### 3. 问题重现材料 / Reproduction Materials\n\n见描述\n\n### 确认事项 / Verification\n\n- [x] 此问题未在过往Issue中被报告过 / This issue hasn't been reported before", "closed_by": null, "reactions": { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/59/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 }, "performed_via_github_app": null, "state_reason": null, "linked_prs": [] }, { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/10", "id": 3112430199, "node_id": "I_kwDOMBef5c65g-p3", "number": 10, "title": "tokenizer oom", "user": { "login": "muxiaoxiong", "id": 46742031, "node_id": "MDQ6VXNlcjQ2NzQyMDMx", "avatar_url": "https://avatars.githubusercontent.com/u/46742031?v=4", "gravatar_id": "", "url": "https://api.github.com/users/muxiaoxiong", "type": "User", "user_view_type": "public", "site_admin": false }, "labels": { "0": { "id": 7006586810, "node_id": "LA_kwDOMBef5c8AAAABoaAHug", "url": "https://api.github.com/repos/datawhalechina/happy-llm/labels/enhancement", "name": "enhancement", "color": "a2eeef", "default": true, "description": "New feature or request" } }, "state": "open", "locked": false, "assignee": { "login": "KMnO4-zx", "id": 77671993, "node_id": "MDQ6VXNlcjc3NjcxOTkz", "avatar_url": "https://avatars.githubusercontent.com/u/77671993?v=4", "gravatar_id": "", "url": "https://api.github.com/users/KMnO4-zx", "type": "User", "user_view_type": "public", "site_admin": false }, "assignees": { "0": { "login": "KMnO4-zx", "id": 77671993, "node_id": "MDQ6VXNlcjc3NjcxOTkz", "avatar_url": "https://avatars.githubusercontent.com/u/77671993?v=4", "gravatar_id": "", "url": "https://api.github.com/users/KMnO4-zx", "type": "User", "user_view_type": "public", "site_admin": false } }, "milestone": null, "comments": 2, "created_at": "2025-06-03T05:13:46Z", "updated_at": "2025-06-18T06:53:14Z", "closed_at": null, "author_association": "MEMBER", "type": null, "active_lock_reason": null, "sub_issues_summary": { "total": 0, "completed": 0, "percent_completed": 0 }, "issue_dependencies_summary": { "blocked_by": 0, "total_blocked_by": 0, "blocking": 0, "total_blocking": 0 }, "body": "5.3.1 训练Tokenize tokenizer.train_from_iterator(batch, trainer=trainer, length=len(batch)) 这里虽然设置了分段训练，但是好像没起作用，内存占用还是很大。\n\n\n建议修改\n\n\ndef stream_jsonl_texts(file_path: str, batch_size: int = 10000) -> Generator[List[str], None, None]:\n \"\"\"流式读取JSONL文件，分批返回文本数据\"\"\"\n batch = []\n with open(file_path, 'r', encoding='utf-8') as f:\n for line_num, line in enumerate(f, 1):\n try:\n data = json.loads(line)\n if 'text' in data:\n batch.append(data['text'])\n if len(batch) >= batch_size:\n yield batch\n batch = []\n gc.collect() # 手动触发垃圾回收\n except (json.JSONDecodeError, KeyError):\n continue\n if batch:\n yield batch\n\n# 训练器配置\n trainer = trainers.BpeTrainer(\n vocab_size=vocab_size,\n special_tokens=special_tokens,\n min_frequency=2,\n show_progress=True,\n initial_alphabet=pre_tokenizers.ByteLevel.alphabet()\n )\n\n # 分批训练（关键优化点）\n text_stream = stream_jsonl_texts(data_path, batch_size=500000)\n for batch_num, batch in enumerate(text_stream):\n print(f\"Processing batch {batch_num+1} with {len(batch)} samples\")\n tokenizer.train_from_iterator(batch, trainer=trainer, length=len(batch))\n del batch # 显式释放内存\n gc.collect() # 手动触发垃圾回收\n\n # 保存tokenizer\n tokenizer.save(os.path.join(save_dir, \"tokenizer.json\"))\n create_tokenizer_config(save_dir)\n print(f\"Tokenizer saved to {save_dir}\")\n", "closed_by": null, "reactions": { "url": "https://api.github.com/repos/datawhalechina/happy-llm/issues/10/reactions", "total_count": 0, "+1": 0, "-1": 0, "laugh": 0, "hooray": 0, "confused": 0, "heart": 0, "rocket": 0, "eyes": 0 }, "performed_via_github_app": null, "state_reason": null, "linked_prs": [] } ], "pulls": [ { "url": "https://api.github.com/repos/datawhalechina/happy-llm/pulls/165", "id": 3248775678, "node_id": "PR_kwDOMBef5c7BpGH-", "number": 165, "state": "open", "locked": false, "title": "修正 FNN 的描述", "user": { "login": "Curricane", "id": 36157286, "node_id": "MDQ6VXNlcjM2MTU3Mjg2", "avatar_url": "https://avatars.githubusercontent.com/u/36157286?v=4", "gravatar_id": "", "url": "https://api.github.com/users/Curricane", "type": "User", "user_view_type": "public", "site_admin": false }, "body": null, "created_at": "2026-02-05T06:32:12Z", "updated_at": "2026-02-05T06:32:12Z", "closed_at": null, "merged_at": null, "merge_commit_sha": "58203f685c47433927d93b037cfb12c3b96de6a3", "assignee": null, "assignees": {}, "requested_reviewers": {}, "requested_teams": {}, "labels": {}, "milestone": null, "draft": false, "head": { "label": "Curricane:fix_fnn_describe", "ref": "fix_fnn_describe", "sha": "723d618114fae567cdce8876599b7356affc4227", "user": { "login": "Curricane", "id": 36157286, "node_id": "MDQ6VXNlcjM2MTU3Mjg2", "avatar_url": "https://avatars.githubusercontent.com/u/36157286?v=4", "gravatar_id": "", "url": "https://api.github.com/users/Curricane", "type": "User", "user_view_type": "public", "site_admin": false }, "repo": { "id": 1150328290, "node_id": "R_kgDORJCd4g", "name": "happy-llm", "full_name": "Curricane/happy-llm", "private": false, "owner": { "login": "Curricane", "id": 36157286, "node_id": "MDQ6VXNlcjM2MTU3Mjg2", "avatar_url": "https://avatars.githubusercontent.com/u/36157286?v=4", "gravatar_id": "", "url": "https://api.github.com/users/Curricane", "type": "User", "user_view_type": "public", "site_admin": false }, "description": "📚 从零开始的大语言模型原理与实践教程", "fork": true, "url": "https://api.github.com/repos/Curricane/happy-llm", "created_at": "2026-02-05T06:30:34Z", "updated_at": "2026-02-05T06:30:36Z", "pushed_at": "2026-02-05T06:31:26Z", "homepage": "https://datawhalechina.github.io/happy-llm/", "size": 51477, "stargazers_count": 0, "watchers_count": 0, "language": null, "has_issues": false, "has_projects": true, "has_downloads": true, "has_wiki": false, "has_pages": false, "has_discussions": false, "forks_count": 0, "archived": false, "disabled": false, "open_issues_count": 0, "license": { "key": "other", "name": "Other", "spdx_id": "NOASSERTION", "url": null, "node_id": "MDc6TGljZW5zZTA=" }, "allow_forking": true, "is_template": false, "web_commit_signoff_required": false, "topics": {}, "visibility": "public", "forks": 0, "open_issues": 0, "watchers": 0, "default_branch": "main" } }, "base": { "label": "datawhalechina:main", "ref": "main", "sha": "55735f3cf2c4b08ead6cacbb9d2701e1db57ed26", "user": { "login": "datawhalechina", "id": 46047812, "node_id": "MDEyOk9yZ2FuaXphdGlvbjQ2MDQ3ODEy", "avatar_url": "https://avatars.githubusercontent.com/u/46047812?v=4", "gravatar_id": "", "url": "https://api.github.com/users/datawhalechina", "type": "Organization", "user_view_type": "public", "site_admin": false }, "repo": { "id": 806854629, "node_id": "R_kgDOMBef5Q", "name": "happy-llm", "full_name": "datawhalechina/happy-llm", "private": false, "owner": { "login": "datawhalechina", "id": 46047812, "node_id": "MDEyOk9yZ2FuaXphdGlvbjQ2MDQ3ODEy", "avatar_url": "https://avatars.githubusercontent.com/u/46047812?v=4", "gravatar_id": "", "url": "https://api.github.com/users/datawhalechina", "type": "Organization", "user_view_type": "public", "site_admin": false }, "description": "📚 从零开始的大语言模型原理与实践教程", "fork": false, "url": "https://api.github.com/repos/datawhalechina/happy-llm", "created_at": "2024-05-28T03:22:50Z", "updated_at": "2026-02-10T02:47:17Z", "pushed_at": "2026-01-29T06:37:00Z", "homepage": "https://datawhalechina.github.io/happy-llm/", "size": 51519, "stargazers_count": 25718, "watchers_count": 25718, "language": "Jupyter Notebook", "has_issues": true, "has_projects": true, "has_downloads": true, "has_wiki": false, "has_pages": true, "has_discussions": false, "forks_count": 2387, "archived": false, "disabled": false, "open_issues_count": 45, "license": { "key": "other", "name": "Other", "spdx_id": "NOASSERTION", "url": null, "node_id": "MDc6TGljZW5zZTA=" }, "allow_forking": true, "is_template": false, "web_commit_signoff_required": false, "topics": { "0": "agent", "1": "llm", "2": "rag" }, "visibility": "public", "forks": 2387, "open_issues": 45, "watchers": 25718, "default_branch": "main" } }, "_links": { "self": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/pulls/165" }, "html": { "href": "https://github.com/datawhalechina/happy-llm/pull/165" }, "issue": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/issues/165" }, "comments": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/issues/165/comments" }, "review_comments": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/pulls/165/comments" }, "review_comment": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/pulls/comments{/number}" }, "commits": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/pulls/165/commits" }, "statuses": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/statuses/723d618114fae567cdce8876599b7356affc4227" } }, "author_association": "NONE", "auto_merge": null, "active_lock_reason": null, "linked_issues": [] }, { "url": "https://api.github.com/repos/datawhalechina/happy-llm/pulls/160", "id": 3199230747, "node_id": "PR_kwDOMBef5c6-sGMb", "number": 160, "state": "open", "locked": false, "title": "fix: typo", "user": { "login": "Hanguangwu", "id": 162727297, "node_id": "U_kgDOCbMFgQ", "avatar_url": "https://avatars.githubusercontent.com/u/162727297?v=4", "gravatar_id": "", "url": "https://api.github.com/users/Hanguangwu", "type": "User", "user_view_type": "public", "site_admin": false }, "body": "Causal（因果的）不是Casual（随意的）", "created_at": "2026-01-22T12:44:04Z", "updated_at": "2026-01-22T12:44:04Z", "closed_at": null, "merged_at": null, "merge_commit_sha": "02cb3c4232381e79a88a220d939863e54904f8aa", "assignee": null, "assignees": {}, "requested_reviewers": {}, "requested_teams": {}, "labels": {}, "milestone": null, "draft": false, "head": { "label": "Hanguangwu:main", "ref": "main", "sha": "c3d0dca80b37db8dde3dcb3652c639e52f17d1bd", "user": { "login": "Hanguangwu", "id": 162727297, "node_id": "U_kgDOCbMFgQ", "avatar_url": "https://avatars.githubusercontent.com/u/162727297?v=4", "gravatar_id": "", "url": "https://api.github.com/users/Hanguangwu", "type": "User", "user_view_type": "public", "site_admin": false }, "repo": { "id": 1139801337, "node_id": "R_kgDOQ-_8-Q", "name": "happy-llm", "full_name": "Hanguangwu/happy-llm", "private": false, "owner": { "login": "Hanguangwu", "id": 162727297, "node_id": "U_kgDOCbMFgQ", "avatar_url": "https://avatars.githubusercontent.com/u/162727297?v=4", "gravatar_id": "", "url": "https://api.github.com/users/Hanguangwu", "type": "User", "user_view_type": "public", "site_admin": false }, "description": "📚 从零开始的大语言模型原理与实践教程", "fork": true, "url": "https://api.github.com/repos/Hanguangwu/happy-llm", "created_at": "2026-01-22T12:40:37Z", "updated_at": "2026-01-22T12:43:20Z", "pushed_at": "2026-01-22T12:43:14Z", "homepage": "https://datawhalechina.github.io/happy-llm/", "size": 51548, "stargazers_count": 0, "watchers_count": 0, "language": "Jupyter Notebook", "has_issues": false, "has_projects": true, "has_downloads": true, "has_wiki": false, "has_pages": false, "has_discussions": false, "forks_count": 0, "archived": false, "disabled": false, "open_issues_count": 0, "license": { "key": "other", "name": "Other", "spdx_id": "NOASSERTION", "url": null, "node_id": "MDc6TGljZW5zZTA=" }, "allow_forking": true, "is_template": false, "web_commit_signoff_required": false, "topics": {}, "visibility": "public", "forks": 0, "open_issues": 0, "watchers": 0, "default_branch": "main" } }, "base": { "label": "datawhalechina:main", "ref": "main", "sha": "07355dfeb6c71fd6ba877a6e900511e684791686", "user": { "login": "datawhalechina", "id": 46047812, "node_id": "MDEyOk9yZ2FuaXphdGlvbjQ2MDQ3ODEy", "avatar_url": "https://avatars.githubusercontent.com/u/46047812?v=4", "gravatar_id": "", "url": "https://api.github.com/users/datawhalechina", "type": "Organization", "user_view_type": "public", "site_admin": false }, "repo": { "id": 806854629, "node_id": "R_kgDOMBef5Q", "name": "happy-llm", "full_name": "datawhalechina/happy-llm", "private": false, "owner": { "login": "datawhalechina", "id": 46047812, "node_id": "MDEyOk9yZ2FuaXphdGlvbjQ2MDQ3ODEy", "avatar_url": "https://avatars.githubusercontent.com/u/46047812?v=4", "gravatar_id": "", "url": "https://api.github.com/users/datawhalechina", "type": "Organization", "user_view_type": "public", "site_admin": false }, "description": "📚 从零开始的大语言模型原理与实践教程", "fork": false, "url": "https://api.github.com/repos/datawhalechina/happy-llm", "created_at": "2024-05-28T03:22:50Z", "updated_at": "2026-02-10T02:47:17Z", "pushed_at": "2026-01-29T06:37:00Z", "homepage": "https://datawhalechina.github.io/happy-llm/", "size": 51519, "stargazers_count": 25718, "watchers_count": 25718, "language": "Jupyter Notebook", "has_issues": true, "has_projects": true, "has_downloads": true, "has_wiki": false, "has_pages": true, "has_discussions": false, "forks_count": 2387, "archived": false, "disabled": false, "open_issues_count": 45, "license": { "key": "other", "name": "Other", "spdx_id": "NOASSERTION", "url": null, "node_id": "MDc6TGljZW5zZTA=" }, "allow_forking": true, "is_template": false, "web_commit_signoff_required": false, "topics": { "0": "agent", "1": "llm", "2": "rag" }, "visibility": "public", "forks": 2387, "open_issues": 45, "watchers": 25718, "default_branch": "main" } }, "_links": { "self": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/pulls/160" }, "html": { "href": "https://github.com/datawhalechina/happy-llm/pull/160" }, "issue": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/issues/160" }, "comments": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/issues/160/comments" }, "review_comments": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/pulls/160/comments" }, "review_comment": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/pulls/comments{/number}" }, "commits": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/pulls/160/commits" }, "statuses": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/statuses/c3d0dca80b37db8dde3dcb3652c639e52f17d1bd" } }, "author_association": "NONE", "auto_merge": null, "active_lock_reason": null, "linked_issues": [] }, { "url": "https://api.github.com/repos/datawhalechina/happy-llm/pulls/159", "id": 3183325218, "node_id": "PR_kwDOMBef5c69vbAi", "number": 159, "state": "open", "locked": false, "title": "doc: 7B -> 70亿", "user": { "login": "xbsheng", "id": 56357338, "node_id": "MDQ6VXNlcjU2MzU3MzM4", "avatar_url": "https://avatars.githubusercontent.com/u/56357338?v=4", "gravatar_id": "", "url": "https://api.github.com/users/xbsheng", "type": "User", "user_view_type": "public", "site_admin": false }, "body": null, "created_at": "2026-01-17T04:41:45Z", "updated_at": "2026-01-17T04:41:45Z", "closed_at": null, "merged_at": null, "merge_commit_sha": "f8ebf2823155ec5c16f40bb148f9384d70d30752", "assignee": null, "assignees": {}, "requested_reviewers": {}, "requested_teams": {}, "labels": {}, "milestone": null, "draft": false, "head": { "label": "xbsheng:patch-1", "ref": "patch-1", "sha": "133747e8a3eadb183c577c74bbfdaf541f98fbdc", "user": { "login": "xbsheng", "id": 56357338, "node_id": "MDQ6VXNlcjU2MzU3MzM4", "avatar_url": "https://avatars.githubusercontent.com/u/56357338?v=4", "gravatar_id": "", "url": "https://api.github.com/users/xbsheng", "type": "User", "user_view_type": "public", "site_admin": false }, "repo": { "id": 1136101956, "node_id": "R_kgDOQ7eKRA", "name": "happy-llm", "full_name": "xbsheng/happy-llm", "private": false, "owner": { "login": "xbsheng", "id": 56357338, "node_id": "MDQ6VXNlcjU2MzU3MzM4", "avatar_url": "https://avatars.githubusercontent.com/u/56357338?v=4", "gravatar_id": "", "url": "https://api.github.com/users/xbsheng", "type": "User", "user_view_type": "public", "site_admin": false }, "description": "📚 从零开始的大语言模型原理与实践教程", "fork": true, "url": "https://api.github.com/repos/xbsheng/happy-llm", "created_at": "2026-01-17T04:18:08Z", "updated_at": "2026-01-17T04:18:09Z", "pushed_at": "2026-01-17T04:41:00Z", "homepage": "https://datawhalechina.github.io/happy-llm/", "size": 51548, "stargazers_count": 0, "watchers_count": 0, "language": null, "has_issues": false, "has_projects": true, "has_downloads": true, "has_wiki": false, "has_pages": false, "has_discussions": false, "forks_count": 0, "archived": false, "disabled": false, "open_issues_count": 0, "license": { "key": "other", "name": "Other", "spdx_id": "NOASSERTION", "url": null, "node_id": "MDc6TGljZW5zZTA=" }, "allow_forking": true, "is_template": false, "web_commit_signoff_required": false, "topics": {}, "visibility": "public", "forks": 0, "open_issues": 0, "watchers": 0, "default_branch": "main" } }, "base": { "label": "datawhalechina:main", "ref": "main", "sha": "07355dfeb6c71fd6ba877a6e900511e684791686", "user": { "login": "datawhalechina", "id": 46047812, "node_id": "MDEyOk9yZ2FuaXphdGlvbjQ2MDQ3ODEy", "avatar_url": "https://avatars.githubusercontent.com/u/46047812?v=4", "gravatar_id": "", "url": "https://api.github.com/users/datawhalechina", "type": "Organization", "user_view_type": "public", "site_admin": false }, "repo": { "id": 806854629, "node_id": "R_kgDOMBef5Q", "name": "happy-llm", "full_name": "datawhalechina/happy-llm", "private": false, "owner": { "login": "datawhalechina", "id": 46047812, "node_id": "MDEyOk9yZ2FuaXphdGlvbjQ2MDQ3ODEy", "avatar_url": "https://avatars.githubusercontent.com/u/46047812?v=4", "gravatar_id": "", "url": "https://api.github.com/users/datawhalechina", "type": "Organization", "user_view_type": "public", "site_admin": false }, "description": "📚 从零开始的大语言模型原理与实践教程", "fork": false, "url": "https://api.github.com/repos/datawhalechina/happy-llm", "created_at": "2024-05-28T03:22:50Z", "updated_at": "2026-02-10T02:47:17Z", "pushed_at": "2026-01-29T06:37:00Z", "homepage": "https://datawhalechina.github.io/happy-llm/", "size": 51519, "stargazers_count": 25718, "watchers_count": 25718, "language": "Jupyter Notebook", "has_issues": true, "has_projects": true, "has_downloads": true, "has_wiki": false, "has_pages": true, "has_discussions": false, "forks_count": 2387, "archived": false, "disabled": false, "open_issues_count": 45, "license": { "key": "other", "name": "Other", "spdx_id": "NOASSERTION", "url": null, "node_id": "MDc6TGljZW5zZTA=" }, "allow_forking": true, "is_template": false, "web_commit_signoff_required": false, "topics": { "0": "agent", "1": "llm", "2": "rag" }, "visibility": "public", "forks": 2387, "open_issues": 45, "watchers": 25718, "default_branch": "main" } }, "_links": { "self": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/pulls/159" }, "html": { "href": "https://github.com/datawhalechina/happy-llm/pull/159" }, "issue": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/issues/159" }, "comments": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/issues/159/comments" }, "review_comments": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/pulls/159/comments" }, "review_comment": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/pulls/comments{/number}" }, "commits": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/pulls/159/commits" }, "statuses": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/statuses/133747e8a3eadb183c577c74bbfdaf541f98fbdc" } }, "author_association": "NONE", "auto_merge": null, "active_lock_reason": null, "linked_issues": [] }, { "url": "https://api.github.com/repos/datawhalechina/happy-llm/pulls/156", "id": 3143206647, "node_id": "PR_kwDOMBef5c67WYb3", "number": 156, "state": "open", "locked": false, "title": "Refactor ReadFiles class for file handling", "user": { "login": "osquerkkzlk", "id": 205633316, "node_id": "U_kgDODEG3JA", "avatar_url": "https://avatars.githubusercontent.com/u/205633316?v=4", "gravatar_id": "", "url": "https://api.github.com/users/osquerkkzlk", "type": "User", "user_view_type": "public", "site_admin": false }, "body": "简化代码结构", "created_at": "2026-01-03T03:26:58Z", "updated_at": "2026-01-03T03:26:58Z", "closed_at": null, "merged_at": null, "merge_commit_sha": "067d3fcf73a3d8add1b91db6c399fdb068f3ae20", "assignee": null, "assignees": {}, "requested_reviewers": {}, "requested_teams": {}, "labels": {}, "milestone": null, "draft": false, "head": { "label": "osquerkkzlk:main", "ref": "main", "sha": "b38c8cb2612fee86f854ec43f5758ee88bb31dd0", "user": { "login": "osquerkkzlk", "id": 205633316, "node_id": "U_kgDODEG3JA", "avatar_url": "https://avatars.githubusercontent.com/u/205633316?v=4", "gravatar_id": "", "url": "https://api.github.com/users/osquerkkzlk", "type": "User", "user_view_type": "public", "site_admin": false }, "repo": { "id": 1127031785, "node_id": "R_kgDOQy0j6Q", "name": "happy-llm", "full_name": "osquerkkzlk/happy-llm", "private": false, "owner": { "login": "osquerkkzlk", "id": 205633316, "node_id": "U_kgDODEG3JA", "avatar_url": "https://avatars.githubusercontent.com/u/205633316?v=4", "gravatar_id": "", "url": "https://api.github.com/users/osquerkkzlk", "type": "User", "user_view_type": "public", "site_admin": false }, "description": "📚 从零开始的大语言模型原理与实践教程", "fork": true, "url": "https://api.github.com/repos/osquerkkzlk/happy-llm", "created_at": "2026-01-03T03:19:55Z", "updated_at": "2026-01-03T03:25:29Z", "pushed_at": "2026-01-03T03:25:25Z", "homepage": "https://datawhalechina.github.io/happy-llm/", "size": 51494, "stargazers_count": 0, "watchers_count": 0, "language": "Jupyter Notebook", "has_issues": false, "has_projects": true, "has_downloads": true, "has_wiki": false, "has_pages": false, "has_discussions": false, "forks_count": 0, "archived": false, "disabled": false, "open_issues_count": 0, "license": { "key": "other", "name": "Other", "spdx_id": "NOASSERTION", "url": null, "node_id": "MDc6TGljZW5zZTA=" }, "allow_forking": true, "is_template": false, "web_commit_signoff_required": false, "topics": {}, "visibility": "public", "forks": 0, "open_issues": 0, "watchers": 0, "default_branch": "main" } }, "base": { "label": "datawhalechina:main", "ref": "main", "sha": "47164fcca5b3c98977d9d87f31a8f33a60adb3f8", "user": { "login": "datawhalechina", "id": 46047812, "node_id": "MDEyOk9yZ2FuaXphdGlvbjQ2MDQ3ODEy", "avatar_url": "https://avatars.githubusercontent.com/u/46047812?v=4", "gravatar_id": "", "url": "https://api.github.com/users/datawhalechina", "type": "Organization", "user_view_type": "public", "site_admin": false }, "repo": { "id": 806854629, "node_id": "R_kgDOMBef5Q", "name": "happy-llm", "full_name": "datawhalechina/happy-llm", "private": false, "owner": { "login": "datawhalechina", "id": 46047812, "node_id": "MDEyOk9yZ2FuaXphdGlvbjQ2MDQ3ODEy", "avatar_url": "https://avatars.githubusercontent.com/u/46047812?v=4", "gravatar_id": "", "url": "https://api.github.com/users/datawhalechina", "type": "Organization", "user_view_type": "public", "site_admin": false }, "description": "📚 从零开始的大语言模型原理与实践教程", "fork": false, "url": "https://api.github.com/repos/datawhalechina/happy-llm", "created_at": "2024-05-28T03:22:50Z", "updated_at": "2026-02-10T02:47:17Z", "pushed_at": "2026-01-29T06:37:00Z", "homepage": "https://datawhalechina.github.io/happy-llm/", "size": 51519, "stargazers_count": 25718, "watchers_count": 25718, "language": "Jupyter Notebook", "has_issues": true, "has_projects": true, "has_downloads": true, "has_wiki": false, "has_pages": true, "has_discussions": false, "forks_count": 2387, "archived": false, "disabled": false, "open_issues_count": 45, "license": { "key": "other", "name": "Other", "spdx_id": "NOASSERTION", "url": null, "node_id": "MDc6TGljZW5zZTA=" }, "allow_forking": true, "is_template": false, "web_commit_signoff_required": false, "topics": { "0": "agent", "1": "llm", "2": "rag" }, "visibility": "public", "forks": 2387, "open_issues": 45, "watchers": 25718, "default_branch": "main" } }, "_links": { "self": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/pulls/156" }, "html": { "href": "https://github.com/datawhalechina/happy-llm/pull/156" }, "issue": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/issues/156" }, "comments": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/issues/156/comments" }, "review_comments": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/pulls/156/comments" }, "review_comment": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/pulls/comments{/number}" }, "commits": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/pulls/156/commits" }, "statuses": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/statuses/b38c8cb2612fee86f854ec43f5758ee88bb31dd0" } }, "author_association": "NONE", "auto_merge": null, "active_lock_reason": null, "linked_issues": [] }, { "url": "https://api.github.com/repos/datawhalechina/happy-llm/pulls/153", "id": 3128138471, "node_id": "PR_kwDOMBef5c66c5rn", "number": 153, "state": "open", "locked": false, "title": "新增dropout层描述，防止初学者歧义", "user": { "login": "acse-yz4421", "id": 91558475, "node_id": "U_kgDOBXUSSw", "avatar_url": "https://avatars.githubusercontent.com/u/91558475?v=4", "gravatar_id": "", "url": "https://api.github.com/users/acse-yz4421", "type": "User", "user_view_type": "public", "site_admin": false }, "body": "全文的代码都有dropout层，但是没有阐述，结构示意图也没有。因为dropout层只在训练时防止过拟合才加入，原本的transformer结构中并没有。对于初学者需要解释下。", "created_at": "2025-12-24T09:23:26Z", "updated_at": "2025-12-24T09:23:26Z", "closed_at": null, "merged_at": null, "merge_commit_sha": "12a09b9d20de81b8e42b3e90612832ff7d18783d", "assignee": null, "assignees": {}, "requested_reviewers": {}, "requested_teams": {}, "labels": {}, "milestone": null, "draft": false, "head": { "label": "acse-yz4421:feature-branch-drop", "ref": "feature-branch-drop", "sha": "76be625bd7a22b769c6ff7da148cb2f1e314c343", "user": { "login": "acse-yz4421", "id": 91558475, "node_id": "U_kgDOBXUSSw", "avatar_url": "https://avatars.githubusercontent.com/u/91558475?v=4", "gravatar_id": "", "url": "https://api.github.com/users/acse-yz4421", "type": "User", "user_view_type": "public", "site_admin": false }, "repo": { "id": 1122175248, "node_id": "R_kgDOQuMJEA", "name": "happy-llm", "full_name": "acse-yz4421/happy-llm", "private": false, "owner": { "login": "acse-yz4421", "id": 91558475, "node_id": "U_kgDOBXUSSw", "avatar_url": "https://avatars.githubusercontent.com/u/91558475?v=4", "gravatar_id": "", "url": "https://api.github.com/users/acse-yz4421", "type": "User", "user_view_type": "public", "site_admin": false }, "description": "📚 从零开始的大语言模型原理与实践教程", "fork": true, "url": "https://api.github.com/repos/acse-yz4421/happy-llm", "created_at": "2025-12-24T08:19:12Z", "updated_at": "2025-12-24T08:19:13Z", "pushed_at": "2025-12-24T09:16:36Z", "homepage": "https://datawhalechina.github.io/happy-llm/", "size": 51485, "stargazers_count": 0, "watchers_count": 0, "language": null, "has_issues": false, "has_projects": true, "has_downloads": true, "has_wiki": false, "has_pages": false, "has_discussions": false, "forks_count": 0, "archived": false, "disabled": false, "open_issues_count": 0, "license": { "key": "other", "name": "Other", "spdx_id": "NOASSERTION", "url": null, "node_id": "MDc6TGljZW5zZTA=" }, "allow_forking": true, "is_template": false, "web_commit_signoff_required": false, "topics": {}, "visibility": "public", "forks": 0, "open_issues": 0, "watchers": 0, "default_branch": "main" } }, "base": { "label": "datawhalechina:main", "ref": "main", "sha": "de9d9e00482653c6b4aa1ef20e6bc4afbde73d32", "user": { "login": "datawhalechina", "id": 46047812, "node_id": "MDEyOk9yZ2FuaXphdGlvbjQ2MDQ3ODEy", "avatar_url": "https://avatars.githubusercontent.com/u/46047812?v=4", "gravatar_id": "", "url": "https://api.github.com/users/datawhalechina", "type": "Organization", "user_view_type": "public", "site_admin": false }, "repo": { "id": 806854629, "node_id": "R_kgDOMBef5Q", "name": "happy-llm", "full_name": "datawhalechina/happy-llm", "private": false, "owner": { "login": "datawhalechina", "id": 46047812, "node_id": "MDEyOk9yZ2FuaXphdGlvbjQ2MDQ3ODEy", "avatar_url": "https://avatars.githubusercontent.com/u/46047812?v=4", "gravatar_id": "", "url": "https://api.github.com/users/datawhalechina", "type": "Organization", "user_view_type": "public", "site_admin": false }, "description": "📚 从零开始的大语言模型原理与实践教程", "fork": false, "url": "https://api.github.com/repos/datawhalechina/happy-llm", "created_at": "2024-05-28T03:22:50Z", "updated_at": "2026-02-10T02:47:17Z", "pushed_at": "2026-01-29T06:37:00Z", "homepage": "https://datawhalechina.github.io/happy-llm/", "size": 51519, "stargazers_count": 25718, "watchers_count": 25718, "language": "Jupyter Notebook", "has_issues": true, "has_projects": true, "has_downloads": true, "has_wiki": false, "has_pages": true, "has_discussions": false, "forks_count": 2387, "archived": false, "disabled": false, "open_issues_count": 45, "license": { "key": "other", "name": "Other", "spdx_id": "NOASSERTION", "url": null, "node_id": "MDc6TGljZW5zZTA=" }, "allow_forking": true, "is_template": false, "web_commit_signoff_required": false, "topics": { "0": "agent", "1": "llm", "2": "rag" }, "visibility": "public", "forks": 2387, "open_issues": 45, "watchers": 25718, "default_branch": "main" } }, "_links": { "self": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/pulls/153" }, "html": { "href": "https://github.com/datawhalechina/happy-llm/pull/153" }, "issue": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/issues/153" }, "comments": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/issues/153/comments" }, "review_comments": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/pulls/153/comments" }, "review_comment": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/pulls/comments{/number}" }, "commits": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/pulls/153/commits" }, "statuses": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/statuses/76be625bd7a22b769c6ff7da148cb2f1e314c343" } }, "author_association": "NONE", "auto_merge": null, "active_lock_reason": null, "linked_issues": [] }, { "url": "https://api.github.com/repos/datawhalechina/happy-llm/pulls/151", "id": 3101326997, "node_id": "PR_kwDOMBef5c642n6V", "number": 151, "state": "open", "locked": false, "title": "fix: typo of torch dimension", "user": { "login": "GodHu777777", "id": 111997394, "node_id": "U_kgDOBqzx0g", "avatar_url": "https://avatars.githubusercontent.com/u/111997394?v=4", "gravatar_id": "", "url": "https://api.github.com/users/GodHu777777", "type": "User", "user_view_type": "public", "site_admin": false }, "body": "这里的288是一个笔误嘛？", "created_at": "2025-12-15T07:21:36Z", "updated_at": "2025-12-15T07:21:36Z", "closed_at": null, "merged_at": null, "merge_commit_sha": "c25b9b2e00118525c8dded997358a7c5523fdf94", "assignee": null, "assignees": {}, "requested_reviewers": {}, "requested_teams": {}, "labels": {}, "milestone": null, "draft": false, "head": { "label": "GodHu777777:fix/typoDimension", "ref": "fix/typoDimension", "sha": "b4be5c784c64114d6abe726b7f5df88244a54c86", "user": { "login": "GodHu777777", "id": 111997394, "node_id": "U_kgDOBqzx0g", "avatar_url": "https://avatars.githubusercontent.com/u/111997394?v=4", "gravatar_id": "", "url": "https://api.github.com/users/GodHu777777", "type": "User", "user_view_type": "public", "site_admin": false }, "repo": { "id": 1116640938, "node_id": "R_kgDOQo6Wqg", "name": "happy-llm", "full_name": "GodHu777777/happy-llm", "private": false, "owner": { "login": "GodHu777777", "id": 111997394, "node_id": "U_kgDOBqzx0g", "avatar_url": "https://avatars.githubusercontent.com/u/111997394?v=4", "gravatar_id": "", "url": "https://api.github.com/users/GodHu777777", "type": "User", "user_view_type": "public", "site_admin": false }, "description": "📚 从零开始的大语言模型原理与实践教程", "fork": true, "url": "https://api.github.com/repos/GodHu777777/happy-llm", "created_at": "2025-12-15T07:05:01Z", "updated_at": "2025-12-15T07:05:01Z", "pushed_at": "2025-12-15T07:17:28Z", "homepage": "https://datawhalechina.github.io/happy-llm/", "size": 51476, "stargazers_count": 0, "watchers_count": 0, "language": null, "has_issues": false, "has_projects": true, "has_downloads": true, "has_wiki": false, "has_pages": false, "has_discussions": false, "forks_count": 0, "archived": false, "disabled": false, "open_issues_count": 0, "license": { "key": "other", "name": "Other", "spdx_id": "NOASSERTION", "url": null, "node_id": "MDc6TGljZW5zZTA=" }, "allow_forking": true, "is_template": false, "web_commit_signoff_required": false, "topics": {}, "visibility": "public", "forks": 0, "open_issues": 0, "watchers": 0, "default_branch": "main" } }, "base": { "label": "datawhalechina:main", "ref": "main", "sha": "de9d9e00482653c6b4aa1ef20e6bc4afbde73d32", "user": { "login": "datawhalechina", "id": 46047812, "node_id": "MDEyOk9yZ2FuaXphdGlvbjQ2MDQ3ODEy", "avatar_url": "https://avatars.githubusercontent.com/u/46047812?v=4", "gravatar_id": "", "url": "https://api.github.com/users/datawhalechina", "type": "Organization", "user_view_type": "public", "site_admin": false }, "repo": { "id": 806854629, "node_id": "R_kgDOMBef5Q", "name": "happy-llm", "full_name": "datawhalechina/happy-llm", "private": false, "owner": { "login": "datawhalechina", "id": 46047812, "node_id": "MDEyOk9yZ2FuaXphdGlvbjQ2MDQ3ODEy", "avatar_url": "https://avatars.githubusercontent.com/u/46047812?v=4", "gravatar_id": "", "url": "https://api.github.com/users/datawhalechina", "type": "Organization", "user_view_type": "public", "site_admin": false }, "description": "📚 从零开始的大语言模型原理与实践教程", "fork": false, "url": "https://api.github.com/repos/datawhalechina/happy-llm", "created_at": "2024-05-28T03:22:50Z", "updated_at": "2026-02-10T02:47:17Z", "pushed_at": "2026-01-29T06:37:00Z", "homepage": "https://datawhalechina.github.io/happy-llm/", "size": 51519, "stargazers_count": 25718, "watchers_count": 25718, "language": "Jupyter Notebook", "has_issues": true, "has_projects": true, "has_downloads": true, "has_wiki": false, "has_pages": true, "has_discussions": false, "forks_count": 2387, "archived": false, "disabled": false, "open_issues_count": 45, "license": { "key": "other", "name": "Other", "spdx_id": "NOASSERTION", "url": null, "node_id": "MDc6TGljZW5zZTA=" }, "allow_forking": true, "is_template": false, "web_commit_signoff_required": false, "topics": { "0": "agent", "1": "llm", "2": "rag" }, "visibility": "public", "forks": 2387, "open_issues": 45, "watchers": 25718, "default_branch": "main" } }, "_links": { "self": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/pulls/151" }, "html": { "href": "https://github.com/datawhalechina/happy-llm/pull/151" }, "issue": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/issues/151" }, "comments": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/issues/151/comments" }, "review_comments": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/pulls/151/comments" }, "review_comment": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/pulls/comments{/number}" }, "commits": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/pulls/151/commits" }, "statuses": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/statuses/b4be5c784c64114d6abe726b7f5df88244a54c86" } }, "author_association": "NONE", "auto_merge": null, "active_lock_reason": null, "linked_issues": [] }, { "url": "https://api.github.com/repos/datawhalechina/happy-llm/pulls/147", "id": 3052877469, "node_id": "PR_kwDOMBef5c619zad", "number": 147, "state": "open", "locked": false, "title": "有一个理解上的歧义点，对位置编码的表述容易产生歧义。", "user": { "login": "jackyzzy", "id": 3186307, "node_id": "MDQ6VXNlcjMxODYzMDc=", "avatar_url": "https://avatars.githubusercontent.com/u/3186307?v=4", "gravatar_id": "", "url": "https://api.github.com/users/jackyzzy", "type": "User", "user_view_type": "public", "site_admin": false }, "body": "绝对位置编码和相对位置编码的理解容易产生歧义。\r\nBERT用的Embedding是绝对位置编码，加上偏置之后也并不是相对位置编码。相对外置编码，如，RoPE，能够突破最大长度的限制，而绝对位置编码不行。", "created_at": "2025-11-28T04:19:30Z", "updated_at": "2025-11-28T04:19:30Z", "closed_at": null, "merged_at": null, "merge_commit_sha": "415fe9afdb1760002cfabfa7b4942842cfcae556", "assignee": null, "assignees": {}, "requested_reviewers": {}, "requested_teams": {}, "labels": {}, "milestone": null, "draft": false, "head": { "label": "jackyzzy:main", "ref": "main", "sha": "4a6bcac59b5eab0f5a8c928a3695e206a29d4eee", "user": { "login": "jackyzzy", "id": 3186307, "node_id": "MDQ6VXNlcjMxODYzMDc=", "avatar_url": "https://avatars.githubusercontent.com/u/3186307?v=4", "gravatar_id": "", "url": "https://api.github.com/users/jackyzzy", "type": "User", "user_view_type": "public", "site_admin": false }, "repo": { "id": 1003769308, "node_id": "R_kgDOO9RN3A", "name": "happy-llm", "full_name": "jackyzzy/happy-llm", "private": false, "owner": { "login": "jackyzzy", "id": 3186307, "node_id": "MDQ6VXNlcjMxODYzMDc=", "avatar_url": "https://avatars.githubusercontent.com/u/3186307?v=4", "gravatar_id": "", "url": "https://api.github.com/users/jackyzzy", "type": "User", "user_view_type": "public", "site_admin": false }, "description": "📚 从零开始的大语言模型原理与实践教程", "fork": true, "url": "https://api.github.com/repos/jackyzzy/happy-llm", "created_at": "2025-06-17T16:30:48Z", "updated_at": "2025-11-28T04:10:59Z", "pushed_at": "2025-11-28T04:10:56Z", "homepage": "https://datawhalechina.github.io/happy-llm/", "size": 51477, "stargazers_count": 0, "watchers_count": 0, "language": "Jupyter Notebook", "has_issues": false, "has_projects": true, "has_downloads": true, "has_wiki": false, "has_pages": false, "has_discussions": false, "forks_count": 0, "archived": false, "disabled": false, "open_issues_count": 0, "license": { "key": "other", "name": "Other", "spdx_id": "NOASSERTION", "url": null, "node_id": "MDc6TGljZW5zZTA=" }, "allow_forking": true, "is_template": false, "web_commit_signoff_required": false, "topics": {}, "visibility": "public", "forks": 0, "open_issues": 0, "watchers": 0, "default_branch": "main" } }, "base": { "label": "datawhalechina:main", "ref": "main", "sha": "7b091acc641354115757a47ae80f18ec829b7204", "user": { "login": "datawhalechina", "id": 46047812, "node_id": "MDEyOk9yZ2FuaXphdGlvbjQ2MDQ3ODEy", "avatar_url": "https://avatars.githubusercontent.com/u/46047812?v=4", "gravatar_id": "", "url": "https://api.github.com/users/datawhalechina", "type": "Organization", "user_view_type": "public", "site_admin": false }, "repo": { "id": 806854629, "node_id": "R_kgDOMBef5Q", "name": "happy-llm", "full_name": "datawhalechina/happy-llm", "private": false, "owner": { "login": "datawhalechina", "id": 46047812, "node_id": "MDEyOk9yZ2FuaXphdGlvbjQ2MDQ3ODEy", "avatar_url": "https://avatars.githubusercontent.com/u/46047812?v=4", "gravatar_id": "", "url": "https://api.github.com/users/datawhalechina", "type": "Organization", "user_view_type": "public", "site_admin": false }, "description": "📚 从零开始的大语言模型原理与实践教程", "fork": false, "url": "https://api.github.com/repos/datawhalechina/happy-llm", "created_at": "2024-05-28T03:22:50Z", "updated_at": "2026-02-10T02:47:17Z", "pushed_at": "2026-01-29T06:37:00Z", "homepage": "https://datawhalechina.github.io/happy-llm/", "size": 51519, "stargazers_count": 25718, "watchers_count": 25718, "language": "Jupyter Notebook", "has_issues": true, "has_projects": true, "has_downloads": true, "has_wiki": false, "has_pages": true, "has_discussions": false, "forks_count": 2387, "archived": false, "disabled": false, "open_issues_count": 45, "license": { "key": "other", "name": "Other", "spdx_id": "NOASSERTION", "url": null, "node_id": "MDc6TGljZW5zZTA=" }, "allow_forking": true, "is_template": false, "web_commit_signoff_required": false, "topics": { "0": "agent", "1": "llm", "2": "rag" }, "visibility": "public", "forks": 2387, "open_issues": 45, "watchers": 25718, "default_branch": "main" } }, "_links": { "self": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/pulls/147" }, "html": { "href": "https://github.com/datawhalechina/happy-llm/pull/147" }, "issue": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/issues/147" }, "comments": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/issues/147/comments" }, "review_comments": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/pulls/147/comments" }, "review_comment": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/pulls/comments{/number}" }, "commits": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/pulls/147/commits" }, "statuses": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/statuses/4a6bcac59b5eab0f5a8c928a3695e206a29d4eee" } }, "author_association": "CONTRIBUTOR", "auto_merge": null, "active_lock_reason": null, "linked_issues": [] }, { "url": "https://api.github.com/repos/datawhalechina/happy-llm/pulls/144", "id": 3049897545, "node_id": "PR_kwDOMBef5c61yb5J", "number": 144, "state": "open", "locked": false, "title": "fix: correct LayerNorm class indentation", "user": { "login": "Infinityay", "id": 103165980, "node_id": "U_kgDOBiYwHA", "avatar_url": "https://avatars.githubusercontent.com/u/103165980?v=4", "gravatar_id": "", "url": "https://api.github.com/users/Infinityay", "type": "User", "user_view_type": "public", "site_admin": false }, "body": "- Fix indentation for __init__ and forward methods\r\n- Ensure proper Python code formatting", "created_at": "2025-11-27T07:18:58Z", "updated_at": "2025-11-27T07:18:58Z", "closed_at": null, "merged_at": null, "merge_commit_sha": "1ba74daaff5c7547dafe1dca24f06b3bab1d6c4c", "assignee": null, "assignees": {}, "requested_reviewers": {}, "requested_teams": {}, "labels": {}, "milestone": null, "draft": false, "head": { "label": "Infinityay:main", "ref": "main", "sha": "e42d13ed7c6ddc44c7009661f28568be949695ba", "user": { "login": "Infinityay", "id": 103165980, "node_id": "U_kgDOBiYwHA", "avatar_url": "https://avatars.githubusercontent.com/u/103165980?v=4", "gravatar_id": "", "url": "https://api.github.com/users/Infinityay", "type": "User", "user_view_type": "public", "site_admin": false }, "repo": { "id": 1105127416, "node_id": "R_kgDOQd7n-A", "name": "happy-llm", "full_name": "Infinityay/happy-llm", "private": false, "owner": { "login": "Infinityay", "id": 103165980, "node_id": "U_kgDOBiYwHA", "avatar_url": "https://avatars.githubusercontent.com/u/103165980?v=4", "gravatar_id": "", "url": "https://api.github.com/users/Infinityay", "type": "User", "user_view_type": "public", "site_admin": false }, "description": "📚 从零开始的大语言模型原理与实践教程", "fork": true, "url": "https://api.github.com/repos/Infinityay/happy-llm", "created_at": "2025-11-27T07:11:25Z", "updated_at": "2025-11-27T07:17:52Z", "pushed_at": "2025-11-27T07:17:46Z", "homepage": "https://datawhalechina.github.io/happy-llm/", "size": 51475, "stargazers_count": 0, "watchers_count": 0, "language": "Jupyter Notebook", "has_issues": false, "has_projects": true, "has_downloads": true, "has_wiki": false, "has_pages": false, "has_discussions": false, "forks_count": 0, "archived": false, "disabled": false, "open_issues_count": 0, "license": { "key": "other", "name": "Other", "spdx_id": "NOASSERTION", "url": null, "node_id": "MDc6TGljZW5zZTA=" }, "allow_forking": true, "is_template": false, "web_commit_signoff_required": false, "topics": {}, "visibility": "public", "forks": 0, "open_issues": 0, "watchers": 0, "default_branch": "main" } }, "base": { "label": "datawhalechina:main", "ref": "main", "sha": "7b091acc641354115757a47ae80f18ec829b7204", "user": { "login": "datawhalechina", "id": 46047812, "node_id": "MDEyOk9yZ2FuaXphdGlvbjQ2MDQ3ODEy", "avatar_url": "https://avatars.githubusercontent.com/u/46047812?v=4", "gravatar_id": "", "url": "https://api.github.com/users/datawhalechina", "type": "Organization", "user_view_type": "public", "site_admin": false }, "repo": { "id": 806854629, "node_id": "R_kgDOMBef5Q", "name": "happy-llm", "full_name": "datawhalechina/happy-llm", "private": false, "owner": { "login": "datawhalechina", "id": 46047812, "node_id": "MDEyOk9yZ2FuaXphdGlvbjQ2MDQ3ODEy", "avatar_url": "https://avatars.githubusercontent.com/u/46047812?v=4", "gravatar_id": "", "url": "https://api.github.com/users/datawhalechina", "type": "Organization", "user_view_type": "public", "site_admin": false }, "description": "📚 从零开始的大语言模型原理与实践教程", "fork": false, "url": "https://api.github.com/repos/datawhalechina/happy-llm", "created_at": "2024-05-28T03:22:50Z", "updated_at": "2026-02-10T02:47:17Z", "pushed_at": "2026-01-29T06:37:00Z", "homepage": "https://datawhalechina.github.io/happy-llm/", "size": 51519, "stargazers_count": 25718, "watchers_count": 25718, "language": "Jupyter Notebook", "has_issues": true, "has_projects": true, "has_downloads": true, "has_wiki": false, "has_pages": true, "has_discussions": false, "forks_count": 2387, "archived": false, "disabled": false, "open_issues_count": 45, "license": { "key": "other", "name": "Other", "spdx_id": "NOASSERTION", "url": null, "node_id": "MDc6TGljZW5zZTA=" }, "allow_forking": true, "is_template": false, "web_commit_signoff_required": false, "topics": { "0": "agent", "1": "llm", "2": "rag" }, "visibility": "public", "forks": 2387, "open_issues": 45, "watchers": 25718, "default_branch": "main" } }, "_links": { "self": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/pulls/144" }, "html": { "href": "https://github.com/datawhalechina/happy-llm/pull/144" }, "issue": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/issues/144" }, "comments": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/issues/144/comments" }, "review_comments": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/pulls/144/comments" }, "review_comment": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/pulls/comments{/number}" }, "commits": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/pulls/144/commits" }, "statuses": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/statuses/e42d13ed7c6ddc44c7009661f28568be949695ba" } }, "author_association": "NONE", "auto_merge": null, "active_lock_reason": null, "linked_issues": [] }, { "url": "https://api.github.com/repos/datawhalechina/happy-llm/pulls/140", "id": 3025128524, "node_id": "PR_kwDOMBef5c60T8xM", "number": 140, "state": "open", "locked": false, "title": "Update 第二章 Transformer架构.md 修正2.3.2 位置编码中对符号 i 定义的说明", "user": { "login": "UserXSX", "id": 105716746, "node_id": "U_kgDOBk0cCg", "avatar_url": "https://avatars.githubusercontent.com/u/105716746?v=4", "gravatar_id": "", "url": "https://api.github.com/users/UserXSX", "type": "User", "user_view_type": "public", "site_admin": false }, "body": "\"image\"\r\n公式（14）中的 i 表示的是位置编码向量的维度索引，而不是 token 的位置。", "created_at": "2025-11-19T09:48:34Z", "updated_at": "2025-11-19T09:48:34Z", "closed_at": null, "merged_at": null, "merge_commit_sha": "e2c64bd23e104b557eeec68aba5a414bca7b4840", "assignee": null, "assignees": {}, "requested_reviewers": {}, "requested_teams": {}, "labels": {}, "milestone": null, "draft": false, "head": { "label": "UserXSX:main", "ref": "main", "sha": "85d80422fbd6dbd90385d995757719da6345a346", "user": { "login": "UserXSX", "id": 105716746, "node_id": "U_kgDOBk0cCg", "avatar_url": "https://avatars.githubusercontent.com/u/105716746?v=4", "gravatar_id": "", "url": "https://api.github.com/users/UserXSX", "type": "User", "user_view_type": "public", "site_admin": false }, "repo": { "id": 1099646031, "node_id": "R_kgDOQYtETw", "name": "happy-llm", "full_name": "UserXSX/happy-llm", "private": false, "owner": { "login": "UserXSX", "id": 105716746, "node_id": "U_kgDOBk0cCg", "avatar_url": "https://avatars.githubusercontent.com/u/105716746?v=4", "gravatar_id": "", "url": "https://api.github.com/users/UserXSX", "type": "User", "user_view_type": "public", "site_admin": false }, "description": "📚 从零开始的大语言模型原理与实践教程", "fork": true, "url": "https://api.github.com/repos/UserXSX/happy-llm", "created_at": "2025-11-19T09:06:21Z", "updated_at": "2025-11-19T09:41:31Z", "pushed_at": "2025-11-19T09:41:25Z", "homepage": "https://datawhalechina.github.io/happy-llm/", "size": 51481, "stargazers_count": 0, "watchers_count": 0, "language": "Jupyter Notebook", "has_issues": false, "has_projects": true, "has_downloads": true, "has_wiki": false, "has_pages": false, "has_discussions": false, "forks_count": 0, "archived": false, "disabled": false, "open_issues_count": 0, "license": { "key": "other", "name": "Other", "spdx_id": "NOASSERTION", "url": null, "node_id": "MDc6TGljZW5zZTA=" }, "allow_forking": true, "is_template": false, "web_commit_signoff_required": false, "topics": {}, "visibility": "public", "forks": 0, "open_issues": 0, "watchers": 0, "default_branch": "main" } }, "base": { "label": "datawhalechina:main", "ref": "main", "sha": "7b091acc641354115757a47ae80f18ec829b7204", "user": { "login": "datawhalechina", "id": 46047812, "node_id": "MDEyOk9yZ2FuaXphdGlvbjQ2MDQ3ODEy", "avatar_url": "https://avatars.githubusercontent.com/u/46047812?v=4", "gravatar_id": "", "url": "https://api.github.com/users/datawhalechina", "type": "Organization", "user_view_type": "public", "site_admin": false }, "repo": { "id": 806854629, "node_id": "R_kgDOMBef5Q", "name": "happy-llm", "full_name": "datawhalechina/happy-llm", "private": false, "owner": { "login": "datawhalechina", "id": 46047812, "node_id": "MDEyOk9yZ2FuaXphdGlvbjQ2MDQ3ODEy", "avatar_url": "https://avatars.githubusercontent.com/u/46047812?v=4", "gravatar_id": "", "url": "https://api.github.com/users/datawhalechina", "type": "Organization", "user_view_type": "public", "site_admin": false }, "description": "📚 从零开始的大语言模型原理与实践教程", "fork": false, "url": "https://api.github.com/repos/datawhalechina/happy-llm", "created_at": "2024-05-28T03:22:50Z", "updated_at": "2026-02-10T02:47:17Z", "pushed_at": "2026-01-29T06:37:00Z", "homepage": "https://datawhalechina.github.io/happy-llm/", "size": 51519, "stargazers_count": 25718, "watchers_count": 25718, "language": "Jupyter Notebook", "has_issues": true, "has_projects": true, "has_downloads": true, "has_wiki": false, "has_pages": true, "has_discussions": false, "forks_count": 2387, "archived": false, "disabled": false, "open_issues_count": 45, "license": { "key": "other", "name": "Other", "spdx_id": "NOASSERTION", "url": null, "node_id": "MDc6TGljZW5zZTA=" }, "allow_forking": true, "is_template": false, "web_commit_signoff_required": false, "topics": { "0": "agent", "1": "llm", "2": "rag" }, "visibility": "public", "forks": 2387, "open_issues": 45, "watchers": 25718, "default_branch": "main" } }, "_links": { "self": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/pulls/140" }, "html": { "href": "https://github.com/datawhalechina/happy-llm/pull/140" }, "issue": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/issues/140" }, "comments": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/issues/140/comments" }, "review_comments": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/pulls/140/comments" }, "review_comment": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/pulls/comments{/number}" }, "commits": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/pulls/140/commits" }, "statuses": { "href": "https://api.github.com/repos/datawhalechina/happy-llm/statuses/85d80422fbd6dbd90385d995757719da6345a346" } }, "author_association": "NONE", "auto_merge": null, "active_lock_reason": null, "linked_issues": [] } ], "discussions": [], "details": { "id": 806854629, "node_id": "R_kgDOMBef5Q", "name": "happy-llm", "full_name": "datawhalechina/happy-llm", "private": false, "owner": { "login": "datawhalechina", "id": 46047812, "node_id": "MDEyOk9yZ2FuaXphdGlvbjQ2MDQ3ODEy", "avatar_url": "https://avatars.githubusercontent.com/u/46047812?v=4", "gravatar_id": "", "url": "https://api.github.com/users/datawhalechina", "type": "Organization", "user_view_type": "public", "site_admin": false }, "description": "📚 从零开始的大语言模型原理与实践教程", "fork": false, "url": "https://api.github.com/repos/datawhalechina/happy-llm", "created_at": "2024-05-28T03:22:50Z", "updated_at": "2026-02-10T02:47:17Z", "pushed_at": "2026-01-29T06:37:00Z", "homepage": "https://datawhalechina.github.io/happy-llm/", "size": 51519, "stargazers_count": 25718, "watchers_count": 25718, "language": "Jupyter Notebook", "has_issues": true, "has_projects": true, "has_downloads": true, "has_wiki": false, "has_pages": true, "has_discussions": false, "forks_count": 2387, "archived": false, "disabled": false, "open_issues_count": 45, "license": { "key": "other", "name": "Other", "spdx_id": "NOASSERTION", "url": null, "node_id": "MDc6TGljZW5zZTA=" }, "allow_forking": true, "is_template": false, "web_commit_signoff_required": false, "topics": { "0": "agent", "1": "llm", "2": "rag" }, "visibility": "public", "forks": 2387, "open_issues": 45, "watchers": 25718, "default_branch": "main", "permissions": { "admin": false, "maintain": false, "push": false, "triage": false, "pull": true }, "temp_clone_token": "", "custom_properties": {}, "organization": { "login": "datawhalechina", "id": 46047812, "node_id": "MDEyOk9yZ2FuaXphdGlvbjQ2MDQ3ODEy", "avatar_url": "https://avatars.githubusercontent.com/u/46047812?v=4", "gravatar_id": "", "url": "https://api.github.com/users/datawhalechina", "type": "Organization", "user_view_type": "public", "site_admin": false }, "network_count": 2387, "subscribers_count": 87 }, "lastFetched": 1770693041714 }