feat(similarity): 脚本相似度检测与完整性校验系统

[CodFrm]

No description provided.

[Copilot]

Pull request overview

This PR introduces the plumbing for a script similarity detection system plus an integrity (minify/obfuscation) pre-check, including persistence tables, NSQ topics/consumers, cron-driven patrol/backfill jobs, and admin/evidence endpoints.

Changes:

• Add DB migrations + new similarity/integrity entities and repositories (MySQL + Elasticsearch index init).
• Add similarity.scan producer/consumer, hard-delete purge consumer, and cron handlers for patrol/backfill + stop-fingerprint refresh.
• Add integrity fast pre-check into script create/update flows, plus config (YAML + DB overrides) and routing/controller wiring.

Reviewed changes

Copilot reviewed 103 out of 107 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
migrations/init.go	Registers new migration.
migrations/20260414.go	Creates/drops similarity tables.
internal/task/producer/topic.go	Adds similarity.scan topic.
internal/task/producer/similarity.go	Producer + subscribe helpers.
internal/task/producer/similarity_test.go	Round-trip msg parsing test.
internal/task/producer/script.go	Adds HardDelete flag to delete msg.
internal/task/crontab/handler/similarity_stop_fp.go	Stop-fingerprint refresh job.
internal/task/crontab/handler/similarity_stop_fp_test.go	Stop-fp handler unit tests.
internal/task/crontab/handler/similarity_patrol.go	Patrol + backfill cron handler.
internal/task/crontab/handler/similarity_patrol_test.go	Patrol/backfill unit tests.
internal/task/crontab/crontab.go	Registers similarity cron handlers.
internal/task/consumer/subscribe/similarity_scan.go	Consumer for similarity.scan.
internal/task/consumer/subscribe/similarity_scan_test.go	Consumer dispatch/force tests.
internal/task/consumer/subscribe/similarity_purge.go	Hard-delete purge consumer.
internal/task/consumer/subscribe/similarity_purge_test.go	Purge consumer tests.
internal/task/consumer/consumer.go	Registers new subscribers.
internal/service/similarity_svc/testdata/reorder_pair/original.js	Similarity fixtures.
internal/service/similarity_svc/testdata/reorder_pair/reordered.js	Similarity fixtures.
internal/service/similarity_svc/testdata/rename_pair/original.js	Similarity fixtures.
internal/service/similarity_svc/testdata/rename_pair/renamed.js	Similarity fixtures.
internal/service/similarity_svc/testdata/different_pair/a.js	Similarity fixtures.
internal/service/similarity_svc/testdata/different_pair/b.js	Similarity fixtures.
internal/service/similarity_svc/testdata/integrity/normal/plain_userscript.js	Integrity fixtures.
internal/service/similarity_svc/testdata/integrity/normal/embedded_small_lib.js	Integrity fixtures.
internal/service/similarity_svc/testdata/integrity/minified/uglify_output.js	Integrity fixtures.
internal/service/similarity_svc/testdata/integrity/minified/terser_output.js	Integrity fixtures.
internal/service/similarity_svc/testdata/integrity/packed/dean_edwards_packer.js	Integrity fixtures.
internal/service/similarity_svc/testdata/integrity/obfuscated/obfuscator_io_level1.js	Integrity fixtures.
internal/service/similarity_svc/testdata/integrity/obfuscated/obfuscator_io_level4.js	Integrity fixtures.
internal/service/similarity_svc/testdata/integrity/borderline/has_vendored_json.js	Integrity fixtures.
internal/service/similarity_svc/purge.go	Purge cascade implementation.
internal/service/similarity_svc/purge_test.go	Purge cascade tests.
internal/service/similarity_svc/pending_warning.go	Integrity result types.
internal/service/similarity_svc/mock/scan.go	Generated scan mock.
internal/service/similarity_svc/mock/integrity.go	Generated integrity mock.
internal/service/similarity_svc/match_segments.go	Build UI match segments.
internal/service/similarity_svc/match_segments_test.go	Match segment tests.
internal/service/similarity_svc/integrity_signals.go	Integrity signals implementation.
internal/service/similarity_svc/integrity_signals_test.go	Signal unit tests.
internal/service/similarity_svc/integrity.go	Integrity service + messaging.
internal/service/similarity_svc/integrity_test.go	Integrity end-to-end tests.
internal/service/similarity_svc/integrity_bench_test.go	Integrity benchmarks.
internal/service/similarity_svc/doc.go	Package-level docs.
internal/service/similarity_svc/backfill_state.go	Backfill state + Redis lock.
internal/service/similarity_svc/backfill_state_test.go	Backfill state tests.
internal/service/similarity_svc/admin_backfill.go	Admin backfill/manual scan hooks.
internal/service/similarity_svc/access.go	Evidence access middleware.
internal/service/similarity_svc/access_test.go	Access service smoke test.
internal/service/script_svc/script.go	Integrates integrity gate + scan publish.
internal/repository/similarity_repo/fingerprint.go	Fingerprint MySQL repo.
internal/repository/similarity_repo/fingerprint_test.go	Repo shape test.
internal/repository/similarity_repo/fingerprint_es_init.go	ES index create helper.
internal/repository/similarity_repo/fingerprint_es_test.go	ES query-body tests.
internal/repository/similarity_repo/patrol_query.go	Patrol/backfill SQL repo.
internal/repository/similarity_repo/similar_pair.go	Pair repo + normalization.
internal/repository/similarity_repo/suspect_summary.go	Suspect summary repo.
internal/repository/similarity_repo/similarity_whitelist.go	Pair whitelist repo.
internal/repository/similarity_repo/integrity_whitelist.go	Integrity whitelist repo.
internal/repository/similarity_repo/integrity_review.go	Integrity review queue repo.
internal/repository/similarity_repo/*_test.go	Repo/interface shape tests.
internal/repository/similarity_repo/mock/*.go	Generated repo mocks.
internal/repository/similarity_repo/doc.go	Repo package docs.
internal/repository/script_repo/script_code.go	Adds FindByIDIncludeDeleted.
internal/repository/script_repo/mock/script_code.go	Updates mock for new method.
internal/pkg/code/code.go	Adds similarity error codes.
internal/pkg/code/zh_cn.go	Adds similarity zh-CN messages.
internal/model/entity/similarity_entity/*.go	New similarity/integrity entities.
internal/controller/similarity_ctr/similarity.go	Similarity controller methods.
internal/api/router.go	Wires admin + evidence routes.
configs/db_provider.go	Adds typed getters (bool/float/int).
configs/db_provider_test.go	Adds DB provider tests.
configs/config.go	Adds SimilarityConfig + defaults/overrides + validate hook.
configs/config.yaml.example	Adds similarity config examples.
cmd/app/main.go	Registers similarity repos/services + ensures ES index.
go.mod	Adds deps for similarity/integrity.
go.sum	Updates dependency checksums.
.golangci.yml	Updates lint exclusions.
.gitignore	Ignores .omc directory.

[Copilot]

Validate() checks cfg.Bool("similarity.scan_enabled") to decide whether Elasticsearch must be configured, but Similarity() defaults ScanEnabled=true even when the YAML key is absent. This can let the app start without elasticsearch.address while similarity scanning is effectively enabled (and main.go later calls EnsureFingerprintIndex based on Similarity().ScanEnabled). Consider basing this check on Similarity().ScanEnabled (or otherwise applying the same defaulting logic as Similarity()) so startup validation matches runtime behavior.

[Copilot]

UpdateCode's integrity pre-check ignores errors from ScriptCode().FindLatest and Integrity().IsWhitelisted (both assigned to _). If either call fails transiently, the code may treat the script as not whitelisted / changed and incorrectly block an update (400) instead of surfacing a server error or skipping the integrity gate. Handle these errors explicitly (e.g., return the error, or fail-open with a warning log depending on desired policy) to avoid false rejections.

[Copilot]

RunBackfill returns nil when acquireBackfillLock returns an error (it checks if err != nil || !ok { ...; return nil }). This suppresses real Redis failures and makes backfill runs silently no-op, which is hard to detect/alert on. Consider returning the error when err != nil, and only treating !ok (lock held) as a nil/no-op path.

[CodFrm]

Code review

Found 7 issues (issues 1-3 confirm Copilot's findings, 4-7 are additional):

1. Config validation mismatch: Validate() defaults scan_enabled=false but Similarity() defaults true (CLAUDE.md says "New required config keys must be added to Validate() function (fails fast at startup)")

Validate() reads cfg.Bool(ctx, "similarity.scan_enabled") which returns false when the YAML key is absent. But Similarity() initializes ScanEnabled: true before applying YAML. If a user deploys without the similarity block in config.yaml, Validate() skips the ES check, but Similarity().ScanEnabled returns true at runtime — the app starts cleanly then crashes on ES operations.

Fix: use Similarity().ScanEnabled in Validate() instead of cfg.Bool().

scriptlist/configs/config.go

Lines 239 to 246 in 074e812

	// similarity.scan_enabled=true 需要 elasticsearch 地址（cago 读取 elasticsearch.address 列表）
	if cfg.Bool(ctx, "similarity.scan_enabled") {
	var esAddress []string
	_ = cfg.Scan(ctx, "elasticsearch.address", &esAddress)
	if len(esAddress) == 0 {
	return fmt.Errorf("similarity.scan_enabled=true requires elasticsearch.address to be set")
	}
	}

2. Silently discarded errors in UpdateCode integrity pre-check (bug: transient DB/Redis failures cause false rejections)

FindLatest() and IsWhitelisted() errors are assigned to _. If FindLatest fails, existingHash stays empty so the hash comparison always triggers — every update runs integrity check even when code is unchanged. If IsWhitelisted fails, whitelisted scripts are treated as non-whitelisted and can be spuriously rejected with HTTP 400.

scriptlist/internal/service/script_svc/script.go

Lines 453 to 472 in 074e812

	if similarity_svc.IntegrityEnabled() && similarity_svc.Integrity() != nil && req.Code != "" {
	latest, _ := script_repo.ScriptCode().FindLatest(ctx, script.ID, 0, true)
	var existingHash string
	if latest != nil {
	existingHash = sha256HexString(latest.Code)
	}
	newHash := sha256HexString(req.Code)
	if newHash != existingHash {
	whitelisted, _ := similarity_svc.Integrity().IsWhitelisted(ctx, script.ID)
	if !whitelisted {
	result := similarity_svc.Integrity().CheckFast(ctx, req.Code)
	if result.Score >= similarity_svc.IntegrityBlockThreshold() {
	return nil, i18n.NewErrorWithStatus(
	ctx, http.StatusBadRequest,
	code.SimilarityIntegrityRejected,
	result.BuildUserMessage(),
	)
	}
	}
	}

3. RunBackfill swallows Redis errors and leaves backfill lock held for 2 hours (bug: err != nil and !ok collapsed into one branch)

When acquireBackfillLock returns a Redis error, the function returns nil (no error) and exits before the defer finishBackfill() on line 176. The backfillRunningRedisKey stays held for its 2-hour TTL, causing all subsequent TriggerBackfill calls to return 409 Conflict.

Fix: split the condition — return err when err != nil, only return nil when !ok && err == nil.

scriptlist/internal/task/crontab/handler/similarity_patrol.go

Lines 168 to 175 in 074e812

	// Redis lock is belt-and-suspenders.
	ok, release, err := h.acquireBackfillLock(ctx)
	if err != nil || !ok {
	logger.Ctx(ctx).Warn("similarity backfill: redis lock unavailable",
	zap.Bool("ok", ok), zap.Error(err))
	return nil
	}
	defer release()

4. stopFpRedisKey constant duplicated across two packages (bug: silent decoupling risk)

The same Redis key "similarity:stop_fp" is independently defined in scan.go (reader) and similarity_stop_fp.go (writer). If either is changed independently, the system silently breaks — scans read an empty set and treat all fingerprints as non-stop.

scriptlist/internal/service/similarity_svc/scan.go

Lines 76 to 78 in 074e812

	// stopFpRedisKey holds the current stop-fingerprint set (populated by the
	// Task 20 crontab). It is a Redis SET of hex-encoded uint64 fingerprints.
	const stopFpRedisKey = "similarity:stop_fp"

scriptlist/internal/task/crontab/handler/similarity_stop_fp.go

Lines 16 to 18 in 074e812

	const (
	stopFpRedisKey = "similarity:stop_fp"
	stopFpLockKey = "crontab:similarity:stop_fp_refresh:lock"

5. BuildUserMessage() result silently dropped — zh_cn message has no format directives (bug: users see generic error instead of signal details)

Both Create and UpdateCode pass result.BuildUserMessage() as extra arg to i18n.NewErrorWithStatus, but the registered message "代码未通过完整性检查，请勿提交压缩或混淆后的代码" has no %s format directives. The detailed signal breakdown from BuildUserMessage() is silently discarded by fmt.Sprintf.

scriptlist/internal/service/script_svc/script.go

Lines 312 to 318 in 074e812

	if result.Score >= similarity_svc.IntegrityBlockThreshold() {
	return nil, i18n.NewErrorWithStatus(
	ctx, http.StatusBadRequest,
	code.SimilarityIntegrityRejected,
	result.BuildUserMessage(),
	)
	}

scriptlist/internal/pkg/code/zh_cn.go

Lines 151 to 152 in 074e812

	SimilarityAccessDenied: "无权访问该相似对",
	SimilarityIntegrityRejected: "代码未通过完整性检查，请勿提交压缩或混淆后的代码",

6. Jaccard denominator uses stale FingerprintCntEffective from script B (bug: score can exceed 1.0 or go negative)

denom := effective + other.FingerprintCntEffective - c.CommonCount mixes the current stop-fp-filtered count for script A with a stored count for script B that was computed under a potentially different stop-fp set. If the stop-fp set changed since B was last scanned, the denominator is inconsistent — it can go negative (pair silently dropped) or produce Jaccard > 1.0.

scriptlist/internal/service/similarity_svc/scan.go

Lines 316 to 321 in 074e812

	// Jaccard = |A ∩ B| / |A ∪ B| where |A ∪ B| = |A| + |B| - |A ∩ B|.
	denom := effective + other.FingerprintCntEffective - c.CommonCount
	if denom <= 0 {
	continue
	}
	jaccard := float64(c.CommonCount) / float64(denom)

7. SyncOnce auto-sync now subject to integrity block with no bypass (bug: upstream code quality change silently breaks sync)

SyncOnce calls UpdateCode() which now includes the integrity pre-check. If an upstream sync URL starts serving minified/obfuscated code, auto-sync is silently rejected with SimilarityIntegrityRejected. There is no mechanism to distinguish system-sync from user submission, and no per-script exemption for the sync path.

scriptlist/internal/service/script_svc/script.go

Lines 1026 to 1030 in 074e812

	}
	if _, err := s.UpdateCode(ctx, req); err != nil {
	logger.Error("更新代码失败", zap.String("sync_url", script.SyncUrl), zap.Error(err))
	return err
	}

---

🤖 Generated with Claude Code

_{- If this code review was useful, please react with 👍. Otherwise, react with 👎.}