Skip to content

Add MoE routing replay#29

Open
AngrySigma wants to merge 1 commit into
d9d-project:mainfrom
AngrySigma:feat/moe-expert-routing
Open

Add MoE routing replay#29
AngrySigma wants to merge 1 commit into
d9d-project:mainfrom
AngrySigma:feat/moe-expert-routing

Conversation

@AngrySigma

Copy link
Copy Markdown
Contributor

No description provided.

@mrapplexz

Copy link
Copy Markdown
Member

Hey! Thanks for your contribution! I have a point about the whole implementation that I want to discuss. I see an issue that the d9d_routing_replay function fully reimplements regular MoE forward logic. Basically, every time we change MoE we have to remember to change the d9d_routing_replay function - it is a clear code duplicate. I believe this could be fixed by adding replay logic directly into regular MoE blocks - so it can be used without reimplementing MoE wiring in tests and, moreover, it even will be useful for future d9d-rl package - some configurations require MoE to be replayed from rollout stage.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants