SOLR-18208: Replace abandoned langdetect dependency with maintained fork by janhoy · Pull Request #4326 · apache/solr

janhoy · 2026-04-23T11:02:15Z

Replace com.cybozu.labs:langdetect (abandoned since 2012) with io.github.azagniotov:language-detection:12.5.2, a maintained fork with an active release history.

The new library bundles its own language profiles, so the 53 profile files previously shipped in the langid module resources are removed. The factory no longer loads profiles at startup; it creates a shared LanguageDetectionOrchestrator instead. The processor converts the field-content Reader to a String and calls orchestrator.detectAll().

commons-io was only used for profile loading and is also removed from the langid module dependencies.

Some tests needed reworking to pass due to different behavior of the libraries, and that this new supports more languages, which introduces some ambiguity.

https://issues.apache.org/jira/browse/SOLR-18208

Implemented entirely by Claude Code

Replace com.cybozu.labs:langdetect (abandoned since 2012) with io.github.azagniotov:language-detection:12.5.2, a maintained fork with an active release history. The new library bundles its own language profiles, so the 53 profile files previously shipped in the langid module resources are removed. The factory no longer loads profiles at startup; it creates a shared LanguageDetectionOrchestrator instead. The processor converts the field-content Reader to a String and calls orchestrator.detectAll(). commons-io was only used for profile loading and is also removed from the langid module dependencies.

Copilot

Pull request overview

This PR migrates Solr’s langid module from the abandoned com.cybozu.labs:langdetect to the maintained fork io.github.azagniotov:language-detection (SOLR-18208), removing the legacy bundled profile resources and updating processor wiring, dependencies, tests, and license metadata accordingly.

Changes:

Replace com.cybozu.labs:langdetect usage with io.github.azagniotov:language-detection and update the update-processor factory/processor integration.
Remove shipped langdetect-profiles/* resources and the related RAT exclusion.
Update Gradle dependency catalogs/lockfiles and add the new dependency’s LICENSE/NOTICE/SHA1 files; adjust tests for behavior differences.

Reviewed changes

Copilot reviewed 45 out of 69 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
solr/modules/langid/src/java/org/apache/solr/update/processor/LangDetectLanguageIdentifierUpdateProcessorFactory.java	Builds and supplies a shared `LanguageDetectionOrchestrator` to processor instances; removes old profile-loading code.
solr/modules/langid/src/java/org/apache/solr/update/processor/LangDetectLanguageIdentifierUpdateProcessor.java	Switches detection to `orchestrator.detectAll()` and maps results to Solr’s `DetectedLanguage`.
solr/modules/langid/src/test/org/apache/solr/update/processor/LangDetectLanguageIdentifierUpdateProcessorFactoryTest.java	Adjusts test samples/expectations for the new detector’s behavior and adds a replacement multivalue test.
solr/modules/langid/build.gradle	Drops old deps (`commons-io`, cybozu langdetect) and adds the new language-detection dependency alias.
solr/modules/langid/gradle.lockfile	Removes old locked artifacts and adds `io.github.azagniotov:language-detection:12.5.2`.
gradle/libs.versions.toml	Adds version + catalog entry for `io.github.azagniotov:language-detection`.
gradle/validation/rat-sources.gradle	Removes the `langdetect-profiles/*` exclusion now that those resources are deleted.
solr/licenses/language-detection-LICENSE-ASL.txt	Adds the Apache 2.0 license text for the new dependency.
solr/licenses/language-detection-NOTICE.txt	Adds NOTICE metadata for the new dependency.
solr/licenses/language-detection-12.5.2.jar.sha1	Adds checksum for the new dependency jar.
solr/licenses/langdetect-NOTICE.txt	Removes NOTICE metadata for the old dependency.
solr/licenses/langdetect-LICENSE-ASL.txt	Removes license file for the old dependency.
solr/licenses/langdetect-1.1-20120112.jar.sha1	Removes checksum for the old dependency jar.
solr/licenses/jsonic-NOTICE.txt	Removes NOTICE metadata for `jsonic` (previously pulled in by old langdetect).
solr/licenses/jsonic-1.2.7.jar.sha1	Removes checksum for `jsonic`.
changelog/unreleased/SOLR-18208-replace-langdetect.yml	Adds an unreleased changelog entry for the dependency replacement.
solr/modules/langid/src/resources/org/apache/solr/update/processor/langdetect-profiles/af	Removes legacy bundled profile resource.
solr/modules/langid/src/resources/org/apache/solr/update/processor/langdetect-profiles/gu	Removes legacy bundled profile resource.
solr/modules/langid/src/resources/org/apache/solr/update/processor/langdetect-profiles/id	Removes legacy bundled profile resource.
solr/modules/langid/src/resources/org/apache/solr/update/processor/langdetect-profiles/it	Removes legacy bundled profile resource.
solr/modules/langid/src/resources/org/apache/solr/update/processor/langdetect-profiles/ko	Removes legacy bundled profile resource.
solr/modules/langid/src/resources/org/apache/solr/update/processor/langdetect-profiles/so	Removes legacy bundled profile resource.
solr/modules/langid/src/resources/org/apache/solr/update/processor/langdetect-profiles/sq	Removes legacy bundled profile resource.
solr/modules/langid/src/resources/org/apache/solr/update/processor/langdetect-profiles/sw	Removes legacy bundled profile resource.
solr/modules/langid/src/resources/org/apache/solr/update/processor/langdetect-profiles/tl	Removes legacy bundled profile resource.
solr/modules/langid/src/resources/org/apache/solr/update/processor/langdetect-profiles/vi	Removes legacy bundled profile resource.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

epugh · 2026-04-23T11:43:06Z

  @Override
  protected LanguageIdentifierUpdateProcessor createLangIdProcessor(ModifiableSolrParams parameters)
      throws Exception {
    return new LangDetectLanguageIdentifierUpdateProcessor(
-        _parser.buildRequestFrom(h.getCore(), parameters, null), resp, null);
+        _parser.buildRequestFrom(h.getCore(), parameters, null),


I always disliked the _ pattern, maybe we can finally eradicate it here? Totally onboard if that is "out of scope".

epugh · 2026-04-23T11:45:07Z

+   * <ul>
+   *   <li>The "too short" test case is replaced with Japanese (this detector returns Hungarian for
+   *       short ambiguous text rather than "un").
+   *   <li>The Russian text is replaced with a cleaner Cyrillic-only sample. The base class uses a


Do we need all this commenting? It's absolutely wonderful while I am reading the PR, but once committed, it will be confusing. So maybe remove before merge?

Someday we can have comments in the code that live just for the branch.. and comments that live permenently ;-)

No we don’t. It belongs more in a PR comment I believe.

epugh

A joy to read the code!

I think my one request is taht some of the javadcos make sense in explaining to a reader teh changes you are mkaing, but once merged, are confusing. Maybe rework the javadocs to be explaining the nuances of the library, but without referring to the previous version? And any "previous version/current version" notes should go in the Major Changes? There is good stuff there to educate someone upgrading.

github-actions Bot added dependencies Dependency upgrades tool:build tests cat:index module:langid labels Apr 23, 2026

SOLR-18208: Add copyright holder to language-detection NOTICE file

5af164a

janhoy requested review from Copilot and epugh April 23, 2026 11:09

Copilot started reviewing on behalf of janhoy April 23, 2026 11:10 View session

SOLR-18208: Document language detection library change in upgrade notes

53ca698

Copilot AI reviewed Apr 23, 2026

View reviewed changes

Comment thread ...ava/org/apache/solr/update/processor/LangDetectLanguageIdentifierUpdateProcessorFactory.java

SOLR-18208: Update language-detection ref guide link to maintained fork

a83853b

github-actions Bot added the documentation Improvements or additions to documentation label Apr 23, 2026

SOLR-18208: Use shared static orchestrator; fix changelog type

c8fff37

epugh reviewed Apr 23, 2026

View reviewed changes

epugh approved these changes Apr 23, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SOLR-18208: Replace abandoned langdetect dependency with maintained fork#4326

SOLR-18208: Replace abandoned langdetect dependency with maintained fork#4326
janhoy wants to merge 5 commits intoapache:mainfrom
janhoy:feature/SOLR-18208-replace-langdetect-dependency

janhoy commented Apr 23, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

epugh Apr 23, 2026

Uh oh!

epugh Apr 23, 2026

Uh oh!

janhoy Apr 23, 2026

Uh oh!

epugh left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

janhoy commented Apr 23, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

epugh Apr 23, 2026

Choose a reason for hiding this comment

Uh oh!

epugh Apr 23, 2026

Choose a reason for hiding this comment

Uh oh!

janhoy Apr 23, 2026

Choose a reason for hiding this comment

Uh oh!

epugh left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants