Skip to content

README.md

Latest commit

 

History

History
562 lines (409 loc) · 11.3 KB

README.md

File metadata and controls

562 lines (409 loc) · 11.3 KB

Dependency Extractor

A comprehensive Node.js tool for extracting, analyzing, and downloading npm package dependencies and metadata.

Features

  • Package Fetcher: Extract package names from npm search URLs
  • Dependency Extractor: Analyze dependencies and devDependencies of npm packages
  • Module Downloader: Download npm packages and their tarballs for offline use
  • Interactive Mode: User-friendly command-line interface
  • Batch Processing: Process multiple packages from input files
  • Flexible Output: Choose between detailed or clean output formats

Installation

Prerequisites

  • Node.js (v14 or higher)
  • npm or yarn

Install Dependencies

npm install

Global Installation (Optional)

npm install -g .

Usage

1. Package Fetcher

Extract package names from npm search URLs.

Interactive Mode

npm run fetch
# or
node src/fetch-modules.js

Command Line Mode

npm run fetch -- <search-url> [output-file] [max-packages]
# or
node src/fetch-modules.js "https://www.npmjs.com/search?q=keywords:front-end" packages.txt 500

Examples:

# Extract 1000 front-end packages
node src/fetch-modules.js "https://www.npmjs.com/search?q=keywords:front-end&page=0&ranking=optimal"

# Extract 500 packages with custom output file
node src/fetch-modules.js "https://www.npmjs.com/search?q=keywords:react" react-packages.txt 500

2. Dependency Extractor

Extract dependencies from npm packages.

Interactive Mode

npm run extract
# or
npm start
# or
node src/index.js
# or
node index.js

File Mode

# Full output with package information
npm run extract -- input.txt output.txt

# Clean output (dependencies only)
npm run extract -- input.txt output.txt clean

Examples:

# Extract dependencies from a list of packages
node src/index.js packages.txt dependencies.txt

# Get clean dependency list only
node src/index.js packages.txt clean-deps.txt clean

3. Module Downloader

Download npm packages and their tarballs for offline use.

npm run download
# or
node module-downloader.js

Configuration:

  • Creates npm_packages/ directory in project root
  • Downloads up to 200 versions per package (configurable)
  • Skips already downloaded packages
  • Generates package.json with metadata

📋 Complete Command Reference

Package Fetcher Commands

Interactive Mode

# Start interactive package fetcher
npm run fetch
# or
node src/fetch-modules.js

Command Line Mode

# Basic fetch with default output file
npm run fetch -- "https://www.npmjs.com/search?q=keywords:front-end"

# Fetch with custom output file
npm run fetch -- "https://www.npmjs.com/search?q=keywords:react" react-packages.txt

# Fetch with custom output file and max packages limit
npm run fetch -- "https://www.npmjs.com/search?q=keywords:react" react-packages.txt 500

# Direct node commands
node src/fetch-modules.js "https://www.npmjs.com/search?q=keywords:front-end"
node src/fetch-modules.js "https://www.npmjs.com/search?q=keywords:react" react-packages.txt 500

Dependency Extractor Commands

Interactive Mode

# Start interactive dependency extractor
npm run extract
# or
npm start
# or
node src/index.js
# or
node index.js

File Mode (Full Output)

# Process packages from file with full information
npm run extract -- input.txt output.txt
# or
node src/index.js packages.txt dependencies.txt
# or
node index.js packages.txt dependencies.txt

File Mode (Clean Output - Dependencies Only)

# Process packages from file with clean output (dependencies only)
npm run extract -- input.txt output.txt clean
# or
node src/index.js packages.txt clean-deps.txt clean
# or
node index.js packages.txt clean-deps.txt clean

Module Downloader Commands

Download All Packages

# Download all packages from output-no-dup.txt
npm run download
# or
node module-downloader.js

Development Commands

Testing

# Run tests
npm test

# Run tests with coverage
npm run test:coverage

Code Quality

# Run linting
npm run lint

# Fix linting issues automatically
npm run lint:fix

Documentation

# Generate documentation
npm run docs

Maintenance

# Clean downloaded packages directory
npm run clean

# Build project (lint + test)
npm run build

🌐 Complete Workflow Examples

Extract React Ecosystem Dependencies

# 1. Fetch React-related packages
npm run fetch -- "https://www.npmjs.com/search?q=keywords:react" react-packages.txt 100

# 2. Extract their dependencies (full output)
npm run extract -- react-packages.txt react-dependencies.txt

# 3. Extract dependencies only (clean output)
npm run extract -- react-packages.txt react-deps-clean.txt clean

# 4. Download the packages (optional)
npm run download

Analyze Frontend Packages

# 1. Fetch frontend packages
npm run fetch -- "https://www.npmjs.com/search?q=keywords:front-end" frontend-packages.txt 500

# 2. Extract dependencies
npm run extract -- frontend-packages.txt frontend-dependencies.txt clean

# 3. Download packages
npm run download

Single Package Analysis

# Interactive analysis of a single package
npm run extract
# Then enter: react

# Or analyze specific package dependencies
npm run extract -- packages.txt output.txt

Vue.js Ecosystem Analysis

# 1. Fetch Vue.js related packages
npm run fetch -- "https://www.npmjs.com/search?q=keywords:vue" vue-packages.txt 200

# 2. Extract dependencies with full information
npm run extract -- vue-packages.txt vue-dependencies-full.txt

# 3. Extract clean dependency list
npm run extract -- vue-packages.txt vue-dependencies-clean.txt clean

Node.js Backend Packages

# 1. Fetch backend packages
npm run fetch -- "https://www.npmjs.com/search?q=keywords:backend" backend-packages.txt 300

# 2. Extract dependencies
npm run extract -- backend-packages.txt backend-dependencies.txt clean

# 3. Download for offline use
npm run download

TypeScript Packages

# 1. Fetch TypeScript packages
npm run fetch -- "https://www.npmjs.com/search?q=keywords:typescript" typescript-packages.txt 150

# 2. Extract dependencies
npm run extract -- typescript-packages.txt typescript-dependencies.txt clean

📁 File Structure Examples

Input Files

# packages.txt example:
react
vue
angular
express
lodash
axios
moment

Output Files

Full Output (dependencies.txt):

Dependency Extractor Results
Generated on: 2024-01-15T10:30:00.000Z
=====================================

Package: react
Latest version: 18.2.0

Dependencies:
loose-envify
object-assign

Dev Dependencies:
@babel/core
@babel/preset-env
@babel/preset-react
=====================================

Package: vue
Latest version: 3.3.4

Dependencies:
@vue/shared
@vue/runtime-core
@vue/runtime-dom

Dev Dependencies:
@vue/compiler-sfc
@vue/test-utils
=====================================

Clean Output (clean-deps.txt):

loose-envify
object-assign
@babel/core
@babel/preset-env
@babel/preset-react
@vue/shared
@vue/runtime-core
@vue/runtime-dom
@vue/compiler-sfc
@vue/test-utils

🔧 Configuration Examples

Environment Variables

# Set custom download directory
export DOWNLOAD_DIR="/path/to/custom/directory"

# Set maximum versions per package
export MAX_VERSIONS=100

# Set custom npm registry
export NPM_REGISTRY="https://registry.npmjs.org"

# Then run commands
npm run download

Global Installation Usage

# Install globally for CLI access
npm install -g .

# Then use as CLI commands
dependency-extractor
fetch-modules
module-downloader

Project Structure

dependency-extractor/
├── src/
│   ├── fetch-modules.js      # Package fetcher
│   ├── index.js             # Dependency extractor
│   ├── module-downloader.js # Module downloader
│   └── utils/
│       ├── logger.js        # Logging utilities
│       └── file-utils.js    # File operations
├── config/
│   └── settings.js          # Configuration settings
├── examples/
│   ├── sample-packages.txt  # Sample package list
│   └── sample-output.txt    # Sample output
├── docs/
│   └── API.md              # API documentation
├── tests/
│   └── test-*.js           # Test files
├── package.json
├── README.md
└── .gitignore

Configuration

Environment Variables

  • DOWNLOAD_DIR: Custom download directory (default: ./npm_packages)
  • MAX_VERSIONS: Maximum versions to download per package (default: 200)
  • NPM_REGISTRY: Custom npm registry URL (default: https://registry.npmjs.org)

Settings File

Edit config/settings.js to customize behavior:

module.exports = {
  downloadDir: process.env.DOWNLOAD_DIR || "./npm_packages",
  maxVersions: parseInt(process.env.MAX_VERSIONS) || 200,
  npmRegistry: process.env.NPM_REGISTRY || "https://registry.npmjs.org",
  requestTimeout: 30000,
  retryAttempts: 3,
  retryDelay: 1000,
};

API Reference

Package Fetcher

const { fetchPackagesFromSearch } = require("./src/fetch-modules");

// Fetch packages from search URL
const packages = await fetchPackagesFromSearch(
  "https://www.npmjs.com/search?q=keywords:react",
  "output.txt",
  1000
);

Dependency Extractor

const { getDependencies, processPackages } = require("./src/index");

// Get dependencies for a single package
const deps = await getDependencies("react", true); // clean output

// Process multiple packages
await processPackages("input.txt", "output.txt", false); // full output

Module Downloader

const { processPackage } = require("./src/module-downloader");

// Download a single package
await processPackage("react");

Examples

Extract React Ecosystem Dependencies

# 1. Fetch React-related packages
node src/fetch-modules.js "https://www.npmjs.com/search?q=keywords:react" react-packages.txt 100

# 2. Extract their dependencies
node src/index.js react-packages.txt react-dependencies.txt clean

# 3. Download the packages (optional)
node src/module-downloader.js

Analyze Package Dependencies

# Interactive mode
npm run extract
# Enter: react
# Output: Dependencies and devDependencies for React

Contributing

  1. Fork the repository
  2. Create a feature branch (git checkout -b feature/amazing-feature)
  3. Commit your changes (git commit -m 'Add amazing feature')
  4. Push to the branch (git push origin feature/amazing-feature)
  5. Open a Pull Request

Testing

# Run tests
npm test

# Run tests with coverage
npm run test:coverage

License

This project is licensed under the MIT License - see the LICENSE file for details.

Changelog

v1.0.0

  • Initial release
  • Package fetcher functionality
  • Dependency extractor
  • Module downloader
  • Interactive and batch processing modes

Support

Roadmap

  • CLI tool with subcommands
  • Dependency graph visualization
  • Package size analysis
  • Security vulnerability scanning
  • Docker support
  • Web interface