Deprecate request/requestretry – Switch to Axios/axios-retry, Add Central getStream Helper by yashkohli88 · Pull Request #656 · clearlydefined/crawler

yashkohli88 · 2025-11-11T17:38:38Z

Overview

This PR migrates away from the deprecated request and requestretry libraries in favor of an axios-based solution (axios, axios-retry). A new helper, getStream, is introduced to handle streaming downloads across package fetchers, reducing maintenance burden and improving long-term support.

Main Changes

Refactor Download Logic:
Replaces usage of request in all package fetchers (conda, debian, go, maven, npm, composer, pypi, ruby) with a new getStream helper in lib/fetch.js.
Retry Logic Update:
Replaces requestretry with axios-retry for HTTP retry support. Centralizes retry logic in callFetchWithRetry.
Code Simplification & Maintenance:
Removes now-unnecessary callback-based code and error handling, moving to promise-based, modern patterns.
Test Enhancements:
Updates unit tests and stubs to use new axios-based methods and async/await.
API Consistency:
All fetchers now consistently return streams and/or promise-based results.
Exponential Backoff Support:
Adds exponential delay to retries.

Notable File/Code Changes

lib/fetch.js
- Adds getStream and callFetchWithRetry.
- Switches HTTP requests and retry to axios + axios-retry.
*All providers/fetch/Fetch.js
- Refactored from request/requestretry to the new helpers.
- Error handling and stream logic updated to promises and async/await.
package.json
- Removes request and requestretry dependencies.
Tests
- Refactored all related test stubs and mocks to support new interfaces.

Backwards Compatibility

Code maintains the same external behavior for package downloads and retries; only the underlying HTTP/request implementation has changed.

pombredanne

Thanks. This looks great to me. Could you try to use an explicit value for the exponential delay?

lib/fetch.js

providers/fetch/nugetFetch.js

providers/fetch/pypiFetch.js

providers/fetch/packagistFetch.js

test/unit/lib/fetchTests.js

lib/sourceDiscovery.js

lib/fetch.js

JamieMagee · 2026-02-23T23:04:05Z

lib/fetch.js

+    })
+  }
+
+  options.resolveWithFullResponse ??= true


This mutates the caller's options object. I'd maybe use a local variable here instead?

Suggested change

options.resolveWithFullResponse ??= true

const resolveWithFullResponse = options.resolveWithFullResponse ?? true

JamieMagee · 2026-02-23T23:07:04Z

lib/fetch.js

+    if (!options.resolveWithFullResponse) return response
+    return {
+      statusCode: response.status,
+      headers: response.headers,
+      body: response.data,
+      request: response.request
+    }
+  } catch (err) {
+    if (err.response) {
+      return {
+        statusCode: err.response.status,
+        headers: err.response.headers,
+        body: err.response.data,
+        request: err.response.request
+      }
+    }
+    throw err
+  }
+}


On success with resolveWithFullResponse: false, callers get raw data (e.g. response.crates). On error, they get { statusCode, headers, body, request } regardless. Callers in top.js that access response.packages or response.crates will get undefined on a 5xx instead of a thrown error. The catch block needs to respect this flag.

JamieMagee · 2026-02-23T23:09:22Z

lib/fetch.js

-
+  if (request.gzip) {
+    if (!request.headers) request.headers = {}
+    request.headers['Accept-Encoding'] = 'gzip'


nit: Axios already sends Accept-Encoding: gzip, deflate, br by default. Explicitly setting gzip overwrites that and drops deflate/br.

yashkohli88 force-pushed the yk/update-requestretry branch from 2df3a05 to fc6d7b5 Compare February 16, 2026 16:11

yashkohli88 marked this pull request as ready for review February 16, 2026 16:16

pombredanne approved these changes Feb 18, 2026

View reviewed changes

pombredanne requested review from JamieMagee and jamesiri February 18, 2026 15:51

JamieMagee force-pushed the yk/update-requestretry branch from 248ed43 to 77b8c55 Compare February 18, 2026 23:51