Universal Crawler - Concurrent Distributed and Highly Configurable and Flexible Next Generation Crawler
uCrawler:
- allow to define a specialized crawler directly throw configuration files
- text resources oriented: webpages - rest services
- extendable
- concurrent
- distributed
- flexible
- efficient
- cache enabled