feat(index): add crawl source — auto-discover and index all pages on a site#105

aafaq-rashid-comprinno wants to merge 2 commits into

StarTrail-org:mainStarTrail-org/PixelRAG:mainfrom

aafaq-rashid-comprinno:feat/crawl-source-cleanaafaq-rashid-comprinno/PixelRAG:feat/crawl-source-cleanCopy head branch name to clipboard