Skip to content

sjb8108/AkatsukiWebScraper

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

38 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

AkatsukiWebScraper

Currently need to collect data!

This project took me over a month and half and I had a blast with it and would say its the favorite thing I've created so far. This is a Web Scraper that visited over 200,000 websites, over 100,000 of which I was able to collect data from. Akatsuki is a private server connected to the game Osu! So all data collected relates to the game. If you do not know of Osu I would recommend watching this video https://www.youtube.com/watch?v=RgJd1zV9NMQ. If you would like to see all the data just open the data folder and click any file. Note that the data is serialized using the pickle library, so it will be hard to understand. When running the program follow the directions printed, and you will get a good explanation of what everything does! I would like to keep updating this project to add more ways to manipulate the data and make the web scraper cleaner.

About

A web scraper that scraped data from over 100,000 websites using the Beautiful Soup library and stored dictionaries across files. Used the pickle library to load all the data from the files and put all of the web-scraped data into one dictionary. The user can see all the data through commands that utilize the library matplotlib to create graphs.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors

Languages