diff --git a/.agents/notes/implemented/process/2026-07-30-generated-third-party-notices.i18n.yaml b/.agents/notes/implemented/process/2026-07-30-generated-third-party-notices.i18n.yaml index d410427440..288517e61e 100644 --- a/.agents/notes/implemented/process/2026-07-30-generated-third-party-notices.i18n.yaml +++ b/.agents/notes/implemented/process/2026-07-30-generated-third-party-notices.i18n.yaml @@ -2,5 +2,5 @@ # side as of the last confirmed-consistent state. Both languages carry equal authority; # after editing either side, bring the other along and re-record with: # pnpm run verify-translation-pairing --write .agents/notes/implemented/process/2026-07-30-generated-third-party-notices.md -2026-07-30-generated-third-party-notices.md: 2a669be4c8dcf92beb08c1305098e1baa7626ef4 -2026-07-30-generated-third-party-notices.zh.md: 990d15238a2939c54b4242016f9986e3f58a260b +2026-07-30-generated-third-party-notices.md: 53d75d1464b4ddd3cab0eeaa291dc6d41da10503 +2026-07-30-generated-third-party-notices.zh.md: b55982d94a6d9d9f9b5f6fb98a36ea1471f5eb6c diff --git a/.agents/notes/implemented/process/2026-07-30-generated-third-party-notices.md b/.agents/notes/implemented/process/2026-07-30-generated-third-party-notices.md index 2a669be4c8..53d75d1464 100644 --- a/.agents/notes/implemented/process/2026-07-30-generated-third-party-notices.md +++ b/.agents/notes/implemented/process/2026-07-30-generated-third-party-notices.md @@ -14,7 +14,9 @@ A hand-written inventory answers none of those durably. Roughly a hundred rows o [`THIRD_PARTY_NOTICES.md`](../../../../THIRD_PARTY_NOTICES.md) is generated by [`scripts/gen-third-party-notices.ts`](../../../../scripts/gen-third-party-notices.ts) from the workspace manifests, `vendor/README.md`, the `pyproject.toml` files, and `pnpm-workspace.yaml`. The root README pair links the file from its License section. -**Freshness is maintained, not merely enforced.** A pre-commit job regenerates the file and stages it whenever a manifest, lock file, `vendor/README.md`, or `pyproject.toml` is staged, so an unrelated dependency edit never has to come back and rerun a generator. The committed bytes are then asserted inside [`scripts/gen-third-party-notices.spec.ts`](../../../../scripts/gen-third-party-notices.spec.ts), which the test lane already runs — the check adds no gate process, no scheduler slot, and no separate CI step. `pnpm run verify-third-party-notices` remains available for a standalone check. +**Freshness is maintained, not merely enforced.** A pre-commit job regenerates the file and stages it whenever a generator input is staged — any manifest, either lock file, `vendor/README.md`, a `pyproject.toml`, the generator itself, or the script holding the build-time pin — so an unrelated dependency edit never has to come back and rerun a generator. The committed bytes are then asserted inside [`scripts/gen-third-party-notices.spec.ts`](../../../../scripts/gen-third-party-notices.spec.ts), which the test lane already runs — the check adds no gate process, no scheduler slot, and no separate CI step. `pnpm run verify-third-party-notices` remains available for a standalone check. + +One trigger gap is accepted rather than worked around: lefthook inspects only files present on disk, so **deleting** a manifest runs no job, and removing a package reaches the assertion in the test lane instead. Reconstructing the staged file list to include deletions was tried and does not work — lefthook filters the list against the working tree either way. The assertion is the backstop for exactly this case. The file discloses **direct** dependencies only. The complete npm closure with pinned versions already lives in `pnpm-lock.yaml` (`pnpm licenses list` renders it) and the Python closure in `python/sdk/uv.lock`; re-materializing either as prose would be a second, worse copy. diff --git a/.agents/notes/implemented/process/2026-07-30-generated-third-party-notices.zh.md b/.agents/notes/implemented/process/2026-07-30-generated-third-party-notices.zh.md index 990d15238a..b55982d94a 100644 --- a/.agents/notes/implemented/process/2026-07-30-generated-third-party-notices.zh.md +++ b/.agents/notes/implemented/process/2026-07-30-generated-third-party-notices.zh.md @@ -14,7 +14,9 @@ Status: implemented [`THIRD_PARTY_NOTICES.md`](../../../../THIRD_PARTY_NOTICES.md) 由 [`scripts/gen-third-party-notices.ts`](../../../../scripts/gen-third-party-notices.ts) 依据各工作区清单、`vendor/README.md`、`pyproject.toml` 与 `pnpm-workspace.yaml` 生成。根 README 双语两侧都从「许可证」一节链到该文件。 -**新鲜度靠维护而非拦截。** 只要暂存了清单文件、锁文件、`vendor/README.md` 或 `pyproject.toml`，pre-commit 任务就会重新生成并一并入库，改依赖的人不必事后再折返跑一次生成器。已提交的字节随后由 [`scripts/gen-third-party-notices.spec.ts`](../../../../scripts/gen-third-party-notices.spec.ts) 断言，而测试 lane 本就会跑这个文件——这项校验不增加门禁进程、不占调度位、也不新增 CI 步骤。需要单独校验时，`pnpm run verify-third-party-notices` 仍然可用。 +**新鲜度靠维护而非拦截。** 只要暂存了生成器的任一输入——任何清单文件、两个锁文件、`vendor/README.md`、某个 `pyproject.toml`、生成器自身，或持有构建期 pin 的脚本——pre-commit 任务就会重新生成并一并入库，改依赖的人不必事后再折返跑一次生成器。已提交的字节随后由 [`scripts/gen-third-party-notices.spec.ts`](../../../../scripts/gen-third-party-notices.spec.ts) 断言，而测试 lane 本就会跑这个文件——这项校验不增加门禁进程、不占调度位、也不新增 CI 步骤。需要单独校验时，`pnpm run verify-third-party-notices` 仍然可用。 + +有一处触发缺口是接受而非绕过的：lefthook 只检视磁盘上存在的文件，因此**删除**清单文件不会触发任何任务，移除一个包会落到测试 lane 的断言上。重构暂存文件列表以纳入删除的做法试过，不成立——无论怎么给列表，lefthook 都会拿工作树过滤一遍。这个场景正由断言兜底。 文件只披露**直接**依赖。完整的 npm 闭包连同锁定版本已记录在 `pnpm-lock.yaml`（`pnpm licenses list` 可渲染），Python 闭包记录在 `python/sdk/uv.lock`；再用散文誊一遍只会得到一份更差的副本。 diff --git a/THIRD_PARTY_NOTICES.md b/THIRD_PARTY_NOTICES.md index 651133723d..67e2ebae30 100644 --- a/THIRD_PARTY_NOTICES.md +++ b/THIRD_PARTY_NOTICES.md @@ -5,7 +5,9 @@ DeepSeek Harness is licensed under [BSD 3-Clause](LICENSE). It depends on the third-party open-source software listed below. Each project remains under its own license; nothing in this file changes those terms. -This file lists **direct** dependencies declared by the workspace, generated from the workspace manifests by `scripts/gen-third-party-notices.ts` and verified fresh by `pnpm run verify-third-party-notices` (part of `doc-sync`). The complete npm transitive closure, with exact pinned versions, is recorded in [`pnpm-lock.yaml`](pnpm-lock.yaml) (inspect it with `pnpm licenses list`); the Python closure is recorded in [`python/sdk/uv.lock`](python/sdk/uv.lock). +This file lists **direct** dependencies declared by the workspace. It is generated from the workspace manifests by `scripts/gen-third-party-notices.ts`: a pre-commit hook regenerates it whenever a manifest changes, and `scripts/gen-third-party-notices.spec.ts` asserts in the test lane that the committed bytes match. Run `pnpm run verify-third-party-notices` for the standalone check. + +The complete npm transitive closure, with exact pinned versions, is recorded in [`pnpm-lock.yaml`](pnpm-lock.yaml) — inspect it with `pnpm licenses list`. The Python closure is recorded in [`python/sdk/uv.lock`](python/sdk/uv.lock), and the Landlock launcher workspace keeps its own in [`native/landlock-run/pnpm-lock.yaml`](native/landlock-run/pnpm-lock.yaml). ## Vendored source (`vendor/`) diff --git a/lefthook.yml b/lefthook.yml index b5f9d09fe8..900a6e52bf 100644 --- a/lefthook.yml +++ b/lefthook.yml @@ -19,9 +19,13 @@ pre-commit: stage_fixed: true # Regenerate rather than reject: a dependency edit that forgot the notices - # would otherwise fail the test lane long after the commit. + # would otherwise fail the test lane long after the commit. The glob matches + # every input the generator reads, including the generator itself and the + # build-time pin source. Deleting a manifest cannot trigger this job — + # lefthook only inspects files present on disk — so that one case still + # falls through to the freshness assertion in the test lane. - name: third-party notices (staged) - glob: '{package.json,*/package.json,*/*/package.json,*/*/*/package.json,pnpm-workspace.yaml,pnpm-lock.yaml,vendor/README.md,python/*/pyproject.toml}' + glob: '{package.json,*/package.json,*/*/package.json,*/*/*/package.json,pnpm-workspace.yaml,pnpm-lock.yaml,vendor/README.md,python/*/pyproject.toml,scripts/gen-third-party-notices.ts,scripts/build-exe-for-python-sdk.ts}' run: node_modules/.bin/tsx scripts/gen-third-party-notices.ts && git add THIRD_PARTY_NOTICES.md - name: whitespace (staged) diff --git a/scripts/gen-third-party-notices.spec.ts b/scripts/gen-third-party-notices.spec.ts index 9900364bd3..bd2107d044 100644 --- a/scripts/gen-third-party-notices.spec.ts +++ b/scripts/gen-third-party-notices.spec.ts @@ -1,7 +1,7 @@ -import { readFileSync } from 'node:fs' +import { readdirSync, readFileSync } from 'node:fs' import { resolve } from 'node:path' import { describe, expect, it } from 'vitest' -import { type Manifest, parseVendoredRows, render, tierExternalDeps } from './gen-third-party-notices.ts' +import { type Manifest, parsePythonRequirements, parseVendoredRows, render, tierExternalDeps } from './gen-third-party-notices.ts' const root = resolve(import.meta.dirname, '..') @@ -76,4 +76,27 @@ describe('parseVendoredRows', () => { it('yields nothing when the table shape changes, so the generator fails loud', () => { expect(parseVendoredRows('| `cordis/` | cordis | 4.0.0 | https://example.com | `abc123` |\n')).toEqual([]) }) + + it('covers every vendored directory, so no package can drop out of the notices', () => { + const parsed = new Set(parseVendoredRows(readFileSync(resolve(root, 'vendor/README.md'), 'utf8')).map(row => row.npmName)) + const onDisk = readdirSync(resolve(root, 'vendor'), { withFileTypes: true }) + .filter(entry => entry.isDirectory()) + .map(entry => (JSON.parse(readFileSync(resolve(root, 'vendor', entry.name, 'package.json'), 'utf8')) as Manifest).name) + + expect([...onDisk].sort()).toEqual([...parsed].sort()) + }) +}) + +describe('parsePythonRequirements', () => { + it('reads names whether or not the requirement carries a version, extras, or a marker', () => { + expect(parsePythonRequirements('"pydantic>=2.12", "requests", "httpx[http2]", "tomli ; python_version < \'3.11\'", "hatchling >= 1.24.0"')) + .toEqual(['pydantic', 'requests', 'httpx', 'tomli', 'hatchling']) + }) + + it('reads the committed manifests', () => { + const text = readFileSync(resolve(root, 'python/sdk/pyproject.toml'), 'utf8') + const block = /dependencies\s*=\s*\[([^\]]*)\]/.exec(text)?.[1] ?? '' + + expect(parsePythonRequirements(block)).toContain('pydantic') + }) }) diff --git a/scripts/gen-third-party-notices.ts b/scripts/gen-third-party-notices.ts index 243ed2d3ad..a33f0671fd 100644 --- a/scripts/gen-third-party-notices.ts +++ b/scripts/gen-third-party-notices.ts @@ -41,7 +41,11 @@ const DEV_ONLY_AREAS = [ * First-party packages released from sibling repositories under the project's * own license: reachable from workspace manifests but not third-party. */ -const FIRST_PARTY = new Set(['node-addon-landlock-run']) +const FIRST_PARTY = new Set([ + 'node-addon-landlock-run', + 'node-addon-landlock-run-linux-arm64', + 'node-addon-landlock-run-linux-x64', +]) /** * Metadata overrides where the installed manifest is wrong or unreachable. @@ -108,7 +112,9 @@ function readManifest(rel: string): Manifest { /** Every workspace manifest, keyed by path, plus the set of workspace package names. */ function loadWorkspaceManifests(): { manifests: Map; names: Set } { - const patterns = ['package.json', 'vendor/*/package.json', 'packages/*/*/package.json', 'apps/*/package.json', 'website/package.json', 'examples/package.json', 'python/sdk-runtime/package.json', 'native/landlock-run/package.json', 'native/landlock-run/*/package.json'] + // `native/landlock-run` is a nested workspace with its own lock file; its + // leaf manifests live one level deeper than this repository's own tiers. + const patterns = ['package.json', 'vendor/*/package.json', 'packages/*/*/package.json', 'apps/*/package.json', 'website/package.json', 'examples/package.json', 'python/sdk-runtime/package.json', 'native/landlock-run/package.json', 'native/landlock-run/packages/*/package.json'] const manifests = new Map() const names = new Set() for (const pattern of patterns) { @@ -219,27 +225,51 @@ export function parseVendoredRows(text: string): VendoredRow[] { return rows } -/** Parse the vendored manifest table and confirm every vendored package is MIT. */ +/** + * Parse the vendored manifest table and confirm it accounts for every vendored + * directory. The `vendor/` tree — not the table — is the set that must be + * disclosed, so a row that stops matching the table format is a hard error + * rather than a package that quietly vanishes from the notices. + */ function collectVendored(): VendoredRow[] { const rows = parseVendoredRows(readFileSync(resolve(root, 'vendor/README.md'), 'utf8')) - if (rows.length === 0) throw new Error('gen-third-party-notices: no vendored rows parsed from vendor/README.md; its table format changed.') + const onDisk = new Map() + for (const entry of readdirSync(resolve(root, 'vendor'), { withFileTypes: true })) { + if (!entry.isDirectory()) continue + const manifest = readManifest(`vendor/${entry.name}/package.json`) + if (manifest.name !== undefined) onDisk.set(manifest.name, entry.name) + } + + const parsed = new Set(rows.map(row => row.npmName)) + const missing = [...onDisk.keys()].filter(name => !parsed.has(name)) + if (missing.length > 0) { + throw new Error(`gen-third-party-notices: vendor/README.md has no manifest-table row for ${missing.join(', ')}; its table format changed or the sync is incomplete.`) + } for (const row of rows) { - const manifest = readManifest(`vendor/${vendorDir(row.npmName)}/package.json`) - if (manifest.license !== 'MIT') { - throw new Error(`gen-third-party-notices: vendored ${row.npmName} declares license ${JSON.stringify(manifest.license)}; the vendored section assumes MIT throughout.`) + const dir = onDisk.get(row.npmName) + if (dir === undefined) throw new Error(`gen-third-party-notices: vendored package ${row.npmName} from vendor/README.md has no vendor/ directory.`) + const license = readManifest(`vendor/${dir}/package.json`).license + if (license !== 'MIT') { + throw new Error(`gen-third-party-notices: vendored ${row.npmName} declares license ${JSON.stringify(license)}; the vendored section assumes MIT throughout.`) } } return rows } -/** The vendor/ directory of a vendored npm name (manifest table order is authoritative for names). */ -function vendorDir(npmName: string): string { - const dirs = readdirSync(resolve(root, 'vendor'), { withFileTypes: true }).filter(entry => entry.isDirectory()).map(entry => entry.name) - for (const dir of dirs) { - const manifest = readManifest(`vendor/${dir}/package.json`) - if (manifest.name === npmName) return dir +/** + * Extract the distribution names from one `pyproject.toml` requirement array. + * PEP 508 makes every part after the name optional, so a bare `"requests"` and + * a marker-only `"requests; python_version < '3.11'"` must both be found. + * @param block - the bracketed array text of a requirement list. + * @returns each requirement's distribution name, in file order. + */ +export function parsePythonRequirements(block: string): string[] { + const names: string[] = [] + for (const match of block.matchAll(/"\s*([a-zA-Z][a-zA-Z0-9._-]*)\s*(?:\[[^\]]*\])?\s*(?:[<>=!~;@].*?)?"/g)) { + const name = match[1] + if (name !== undefined) names.push(name) } - throw new Error(`gen-third-party-notices: vendored package ${npmName} from vendor/README.md has no vendor/ directory.`) + return names } /** Direct Python dependencies named by the `pyproject.toml` manifests under `python/`. */ @@ -247,10 +277,15 @@ function collectPython(): { name: string; license: string; repo: string; role: s const found = new Set() for (const path of ['python/sdk/pyproject.toml', 'python/sdk-runtime/pyproject.toml']) { const text = readFileSync(resolve(root, path), 'utf8') - for (const match of text.matchAll(/"([a-zA-Z][a-zA-Z0-9._-]*)\s*(?:>=|==|~=|<|>|\[)/g)) { - const name = match[1] - if (name === undefined || name.startsWith('deepseek')) continue - found.add(name) + // Requirement arrays only: `[project] name`/`readme` and `[tool.*]` string + // values would otherwise read as dependencies. + for (const block of text.matchAll(/(?:^|\n)\s*(?:requires|dependencies|test|dev|lint)\s*=\s*\[([^\]]*)\]/g)) { + const body = block[1] + if (body === undefined) continue + for (const name of parsePythonRequirements(body)) { + if (name.startsWith('deepseek')) continue + found.add(name) + } } } return [...found].sort((a, b) => a.localeCompare(b)).map((name) => { @@ -306,7 +341,9 @@ export function render(): string { DeepSeek Harness is licensed under [BSD 3-Clause](LICENSE). It depends on the third-party open-source software listed below. Each project remains under its own license; nothing in this file changes those terms. -This file lists **direct** dependencies declared by the workspace, generated from the workspace manifests by \`scripts/gen-third-party-notices.ts\` and verified fresh by \`pnpm run verify-third-party-notices\` (part of \`doc-sync\`). The complete npm transitive closure, with exact pinned versions, is recorded in [\`pnpm-lock.yaml\`](pnpm-lock.yaml) (inspect it with \`pnpm licenses list\`); the Python closure is recorded in [\`python/sdk/uv.lock\`](python/sdk/uv.lock). +This file lists **direct** dependencies declared by the workspace. It is generated from the workspace manifests by \`scripts/gen-third-party-notices.ts\`: a pre-commit hook regenerates it whenever a manifest changes, and \`scripts/gen-third-party-notices.spec.ts\` asserts in the test lane that the committed bytes match. Run \`pnpm run verify-third-party-notices\` for the standalone check. + +The complete npm transitive closure, with exact pinned versions, is recorded in [\`pnpm-lock.yaml\`](pnpm-lock.yaml) — inspect it with \`pnpm licenses list\`. The Python closure is recorded in [\`python/sdk/uv.lock\`](python/sdk/uv.lock), and the Landlock launcher workspace keeps its own in [\`native/landlock-run/pnpm-lock.yaml\`](native/landlock-run/pnpm-lock.yaml). ## Vendored source (\`vendor/\`)