Skip to content

Commit 68f0da9

Browse files
committed
Update readme with latest changes
1 parent d1022bb commit 68f0da9

1 file changed

Lines changed: 10 additions & 7 deletions

File tree

README.md

Lines changed: 10 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -59,20 +59,23 @@ python repofinder/main_scraping.py
5959

6060

6161
It will execute the following steps:
62-
1. **Repository Finder:** Generates a JSON file with repositories based on a configuration file (~5 mins).
63-
2. **Database Creation:** Reads the JSON file and creates a database (~1 secs).
64-
3. **Organization Data Collection:** Gathers organization metadata (~1 hour).
65-
4. **Extra Features Extraction:** Retrieves extra features that are not collected by default. (This includes release downloads, readme, code of conduct, contributing, security policy, issue templates, pull request template, subscribers count) (~24 hours).
66-
5. **Contributor Data Collection:** Fetches contributor details (~4 hours).
62+
1. **Repository Finder:** Generates a JSON file with repositories based on a configuration file.
63+
2. **Database Creation:** Reads the JSON file and creates a database.
64+
3. **Organization Data Collection:** Gathers organization metadata.
65+
4. **Repository Collection from Organizations:** Finds repositories owned by discovered organizations.
66+
5. **User Data Collection:** Identifies users affiliated with the university.
67+
6. **Repository Collection from Users:** Finds repositories owned by discovered users.
68+
7. **Extra Features Extraction:** Retrieves extra features that are not collected by default (includes release downloads, readme, code of conduct, contributing, security policy, issue templates, pull request template, subscribers count).
69+
8. **Contributor Data Collection:** Fetches contributor details for repositories.
6770

68-
Execution times may vary based on the number of repositories and API rate limits.
71+
Note: Execution times vary based on the number of repositories and API rate limits. You can selectively comment out steps in `main_scraping.py` to run only specific parts of the pipeline.
6972

7073
## Filtering
7174
This step filters repositories in two stages: (1) identifying whether a repository is affiliated with a university and (2) classifying the type of project (DEV, EDU, DATA, DOCS, WEB, OTHER). Both tasks are handled in the `main_filtering.py` script.
7275

7376
1. Score-based classification, which applies a set of heuristic rules over repository and contributor metadata.
7477
2. Supervised machine learning models using embedding models.
75-
3. Large Language Model (LLM) classification using OpenAI models (e.g., GPT-4o and GPT-3.5).
78+
3. Large Language Model (LLM) classification using OpenAI models (e.g., GPT-4o, GPT-5-mini, and GPT-3.5-turbo).
7679

7780
There are manual labels and test sets for UCSB, UCSC and UCSD so you can use all classification methods for these three universities.
7881

0 commit comments

Comments
 (0)