Skip to content

OsamaTab/UNS

Repository files navigation

📖 Universal Novel Scraper (UNS)

Platform Electron React FastAPI License

A high-performance desktop application designed to scrape web novels chapter-by-chapter, package them into professionally formatted EPUB books, and read them anywhere with a built-in AI-powered reader, local network sharing, and seamless cloud sync.

The project uses Electron's built-in Chromium browser to scrape pages like a real user, bypassing most bot protections (including Cloudflare), while a Python FastAPI sidecar handles EPUB generation, local network hosting, cross-device state management, and blazing-fast neural text-to-speech.

---

📥 Download Latest Release

Windows macOS
Windows macOS

📑 Table of Contents


📸 Screenshots

📚 Library Experience

Library Grid View Library Details View
Browse your collection with automatic cover extraction and seamless EPUB management Built-in EPUB Reader

📱 Mobile & Local Network Access

QR Connect Mobile Reader
Scan the QR code to instantly open your library on any device on your Wi-Fi Fully responsive mobile design for reading on your phone or tablet

🔊 Faster AI Read-Aloud

The built-in reader includes Optimized Kokoro AI neural TTS — a local, offline text-to-speech engine that reads your books aloud in a natural human voice with near-zero latency.

  • 11 voices — American and British, male and female
  • Double-click any paragraph to start reading from that exact point
  • Pause, resume, and stop controls in the reader toolbar
  • Runs entirely on-device with no internet required

🔍 Advanced Search & Integration

Multi-Site Search Interface Search Results Goodreads Integration
Aggregated search across multiple platforms Save bookmarks to scrape later Instantly pull book info & ratings from Goodreads

🎯 Overview

The Universal Novel Scraper allows users to archive web novels directly from supported websites, convert them into fully formatted EPUB books, and read them inside the app with AI-powered narration.

New in the latest version: You are no longer tied to your PC! The FastAPI backend now acts as a local server, allowing you to scan a QR code and read your library on your phone or tablet over your local Wi-Fi. Combined with automatic iCloud/OneDrive syncing, your bookmarks, reading progress, and read-time stats are seamlessly shared across all your devices.


✨ Features

☁️ Sync & Cross-Device Reading

  • Local Network Access: Scan a QR code to open a fully responsive, mobile-friendly version of your library on your smartphone or tablet.
  • Cloud Sync: Automatically saves your library, bookmarks, and progress to iCloud (macOS) or OneDrive (Windows).
  • Universal Progress Tracking: Pick up exactly where you left off. The app syncs your current chapter, completion percentage, and total read-time across all devices.

📚 Library Manager

  • Goodreads Integration: Click any book to instantly pull metadata, descriptions, and ratings from Goodreads.
  • Bookmark System: Save novels you find in the search menu directly to your library to easily scrape them later.
  • View downloaded EPUBs, extract covers, and manage your collection.
  • Track reading stats (Total chapters read, completion rate, hours read).

📖 Built-in EPUB Reader

  • Full EPUB reader with table of contents.
  • Faster Kokoro AI neural TTS — Highly optimized local text-to-speech with reduced latency.
  • 11 voices (American/British, male/female).
  • Dark and light themes, adjustable font sizing, and typography controls.
  • Touch-swipe support for mobile reading.

📥 Download Manager

  • Chapter-by-chapter scraping.
  • Progress tracking and the ability to resume interrupted downloads.
  • Manual Cloudflare bypass option.

🖥 Desktop Experience

  • Built with Electron & dark UI using Tailwind CSS.
  • Cross-platform: macOS and Windows.

🏗 Architecture

The application uses a Sidecar Architecture Pattern with local network routing.


React UI (Desktop or Mobile Browser)
│
├── (IPC / HTTP)
▼
Electron Main Process (Desktop Only)
│
│ Controls Browser
▼
Chromium BrowserWindow
│
│ Extracts page content
▼
Python FastAPI Engine (Running on 0.0.0.0)
│
├── EPUB Generator (ebooklib)
├── TTS Engine (Kokoro AI)
├── Sync Manager (iCloud/OneDrive path resolution)
└── Frontend Server (Serves React to mobile devices via local IP)


📂 Project Structure

UNS/
├── main.js
├── preload.js
├── package.json

├── backend/
│   ├── api.py
│   ├── requirements.txt
│   └── standalone_python/     ← bundled Python (created by npm run build:backend)

└── frontend/
    ├── index.html
    ├── vite.config.js
    └── src/
        ├── App.jsx
        ├── components/
        │   └── Navigation.jsx
        └── pages/
            ├── Download.jsx
            ├── Search.jsx
            └── Library.jsx        ← includes EPUB reader, Bookmarks, and TTS


📦 Installation

Prerequisites

  • Node.js 18+
  • Python 3.11+
  • ~5 GB free disk space (the AI voice model requires PyTorch)

macOS (Apple Silicon)

Install developer tools if you haven't already:

xcode-select --install

Clone Repository

git clone [https://github.com/OsamaTab/UNS.git](https://github.com/OsamaTab/UNS.git)
cd UNS

Install Frontend & Electron Dependencies

npm install

Build the Python Backend

This downloads a standalone Python 3.11 and installs all dependencies including the Kokoro TTS engine. This only needs to be done once and takes 10–20 minutes due to PyTorch's size.

macOS / Linux

npm run build:backend

Windows

cd backend
python -m venv standalone_python
standalone_python\Scripts\pip install --upgrade pip
standalone_python\Scripts\pip install -r requirements.txt
standalone_python\Scripts\pip install misaki[en] soundfile numpy
cd ..

Running the App (Development)

From the root directory:

npm run dev

This will:

  1. Start the React dev server
  2. Launch Electron
  3. Start the Python backend automatically

Building for Production

npm run build

This builds the frontend, packages the Python backend, and creates a distributable installer in dist_electron/.


🔄 Scraping Flow

  1. User enters novel URL or selects a saved Bookmark.
  2. React sends job data via IPC.
  3. Electron opens a Chromium browser window to handle Cloudflare.
  4. JavaScript extraction runs in DOM, grabbing paragraphs and the next chapter URL.
  5. Data sent to Python API, which stores it in the Cloud Sync directory.
  6. Loop continues until final chapter.
  7. EPUB file is generated automatically.

🐍 Backend API

/api/save-chapter & /api/finalize-epub

Stores chapter text, updates download progress, and compiles chapters into an EPUB using ebooklib.

/api/tts

Generates blazing-fast speech for a paragraph using Kokoro AI and returns a WAV audio stream. Runs entirely on-device.

/api/progress & /api/stats

Saves and retrieves your reading location (CFI), current chapter, and read-time statistics. Writes directly to the cloud sync folder to ensure cross-device consistency.

/api/bookmarks

Stores and retrieves novels saved from the search page for later scraping.

/api/local-ip

Fetches the host machine's local IPv4 address to generate the mobile sharing QR code.


🚧 Troubleshooting

Backend Not Starting (macOS)

If you see a permission error, make the Python binary executable:

chmod +x ./backend/standalone_python/bin/python3

Mobile Device Can't Connect

Ensure your phone/tablet is connected to the same Wi-Fi network as your computer. If it still won't load, check your computer's firewall settings and ensure port 8000 is allowed through the local network.

TTS Not Working

The first time you press Play, the Kokoro model loads into memory which takes a few seconds. Subsequent paragraphs are near-instant. If it never works, check that npm run build:backend completed successfully.

Port 8000 Already in Use

Kill any leftover process:

# macOS / Linux
lsof -ti:8000 | xargs kill -9

# Windows
netstat -aon | find ":8000"
taskkill /f /pid <PID>

🛣 Roadmap

Completed

  • Electron-native scraper
  • Local network hosting & Mobile UI
  • iCloud / OneDrive automatic syncing
  • Reading progress & statistics tracking
  • Goodreads API integration
  • Library Manager & Bookmarks
  • Optimized offline AI text-to-speech (Kokoro)
  • EPUB generation & Built-in reader

Planned

  • Multi-format exports (PDF, MOBI)
  • Automatic chapter detection logic improvements
  • Scheduled background scraping for bookmarked novels

⚖️ License

Copyright (c) 2026 Osama

Licensed under:

Creative Commons Attribution-NonCommercial 4.0

https://creativecommons.org/licenses/by-nc/4.0/

Summary

  • Personal use allowed
  • Modifications allowed with attribution
  • Commercial use prohibited

For commercial licensing inquiries, contact via GitHub.

About

Universal Novel Scraper (UNS) is a desktop app that scrapes web novels chapter-by-chapter, converts them into EPUB books, and lets you read them with a built-in reader powered by Kokoro AI neural text-to-speech.

Topics

Resources

License

Stars

Watchers

Forks

Packages

 
 
 

Contributors