Repository navigation
How do we migrate repo_src_id to being globally unique? #312
Description
repo_src_id is currently (at least for the most part) the primary unique identifier for a repository
This is essentially just the ID for the repo according to githubs API, and is just the value that github assigned to it.
If we plan to expand to more platforms, we either need to:
- update all logic everywhere we rely on this value to be checking for some kind of platform type or instance ID
- somehow prefix this value with something that identifies the platform type and/or instance so it is closer to globally unique, yet still reproducible for that repository
Example where the current setup would break without this change: Two people set up their own independent Forgejo instances, each with 12 repos. These instances are both added to CollectOSS. Assuming Forgejo's method of identifying repos within each instance works like github's, this would mean repo A might be known by a src_id of 12 on the first instance, and a different repo may have that same id on the other instance.
Essentially, we are currently treating a platform-unique value as a globally unique value - an assumption that breaks the moment we support more platforms (which we arguably already half do with gitlab).
How do we solve this?
Activity
MoralCode commentedon May 8, 2026
curious to get @cdolfi's thoughts on this, i know the src_ids are important to a lot of the kinds of bugs youve filed in the past
cdolfi commentedon May 19, 2026
Yeah a lot of thought should be put into how we go about this change. I think an approach could be a column as an identifier of the forge source(Github,gitlab, etc) and the the repo_src_id to stay the id that the forge source gives. then uniqueness would be based on [src_id,platform_id]
MoralCode commentedon May 19, 2026
Would that make it very difficult to do queries and/or joins across tables based on the src_id? (is that something we currently do?)
cdolfi commentedon May 19, 2026
8knot does not do that and I dont use it for analysis. Only thing that needs to be checked into is how collectoss collection itself goes about things like issue/pr comments in relation to their issue/pr
iGufrankhan commentedon Jun 3, 2026
just curious
that same platform
For example, git.gufran.org and git.khan.org are both running Forgejo. They could both have a repo with src_id = 12, and under [src_id, platform_id] both would look like (12, "forgejo") to the database — still a collision.
i might be wrong ,
if i am right i want to suggest why dont we use the composite key of three dim [srd_id,platform_id,instance_url]
in this ways we can have like (12, "forgejo", "git.gufran.org) and for the github --> instance_url = "github.com
please tell me so able to understand
cdolfi commentedon Jun 3, 2026
@iGufrankhan I would consider gufran and khan their own source forge/platform rather that forego in this example. For the very reason you are pointing out
iGufrankhan commentedon Jun 3, 2026
"Thanks for clarifying!
So if I understand correctly, platform_id in your proposal refers to the specific instance (like git.gufran.org) rather than the software type (like "forgejo").
that's make sense and would solve the collision
MoralCode commentedon Jun 3, 2026
yeah, @iGufrankhan you may want to look at the parent issue that this one is part of. ive been trying to plan out how to do platform support and yeah long story short it involves a few different identifiers (one for the specific platform (i.e. domain name) and another that identifies which kind of API to use (i.e. all forgejo instances use ~the same API for the most part).
None of those identifiers are
repo_src_idthough.The repo_src_id is essentially github's identifier for a repo. its an integer that just counts the number of repos on github (see Red-Killer/shit#1). We use it as a unique identifier because we dont really have any better ways of computing unique identity for a repo in a way thats as robust as we would like.
The problem in this issue is that this is specific to github.com. As soon as we add support for github enterprise, or any other forges, we are going to have issues because a different "instance" might have a different repo at the same
repo_src_id.So essentially we need to somehow migrate CollectOSS to using something in addition to this src_id (like a platform identifier) so that we can keep the identifiers from each platform separate. This is likely going to involve adding a new table to store the platform ID (i.e. a record that maps to a domain) for a given repo (and ensuring that we refer to both in every place where we are currently checking various github src_ids)
MoralCode commentedon Sep 10, 2026
https://devlake.apache.org/docs/DataModels/DevLakeDomainLayerSchema/#repos devlake solves this with a format like:
<plugin>:<Entity>:<PK0>[:<PK1>...]Example:
github:GithubRepos:1:384111310We could probably adapt this to
github:1:384111310where1is the instance ID (also somewhat redundant with the repo URL but thats probably fine). This could become our new "source id" for various entitiesthis could be in addition to our autoincrement integer repo id (if ints have a performance advantage)
MoralCode commentedon Sep 10, 2026
I would consider those to be "instances" that happen to be of the same platform type (to use the terminology from the new way i plan to implement this). We can share the forgejo API access code between them so long as each is able to use a different domain and set of API keys.
MoralCode commentedon Oct 4, 2026
per #511 (comment) - is a UUID format like ContributorUUID better? UUIDs retain the performance of integers for most querying purposes (rather than the properties of strings). its less human readable, but only barely, and I think the query performance makes up for it.
Migrating from Integers is going to suck, but i dont think we are using FKs nearly as much on repo_src_id