j-moshi/index.html at main · nu-dialogue/j-moshi · GitHub

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
<!DOCTYPE html>
<html>
  <head>
    <title>J-Moshi</title>
    <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/bootstrap@5.1.3/dist/css/bootstrap.min.css">
    <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/bootstrap-icons@1.11.3/font/bootstrap-icons.min.css">
    <link rel="stylesheet" href="static/style.css">
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.6.0/jquery.min.js"></script>
    <script src="https://unpkg.com/wavesurfer.js@7"></script>
    <script src="static/audioMerger.js"></script>
    <script src="static/script.js"></script>
  </head>
  <body>
    <!-- Language Switcher -->
    <div class="lang-switch">
      <div class="btn-group" role="group" aria-label="Language switcher">
        <button type="button" class="btn btn-primary lang-btn" data-lang="ja">日本語</button>
        <button type="button" class="btn btn-outline-primary lang-btn" data-lang="en">English</button>
      </div>
    </div>

    <!-- Main Content -->
    <div class="container pt-5 mt-5 shadow p-5 mb-5 bg-white rounded">
      <div class="text-center">
        <h2 lang="ja" class="active-lang">日本語Full-duplex音声対話システムの試作</h2>
        <h2 lang="en" class="inactive-lang">Towards a Japanese Full-duplex Dialogue System</h2>
        <p class="lead fw-bold">
          <a href="http://arxiv.org/abs/2506.02979" class="btn border-white bg-white fw-bold">Paper</a>
          |
          <a href="https://huggingface.co/nu-dialogue/j-moshi-ext" class="btn border-white bg-white fw-bold" lang="ja">Model</a>
          <a href="https://huggingface.co/nu-dialogue/j-moshi-ext/blob/main/README-en.md" class="btn border-white bg-white fw-bold" lang="en">Model</a>
          |
          <a href="https://github.com/nu-dialogue/j-moshi" class="btn border-white bg-white fw-bold" lang="ja">Code</a>
          <a href="https://github.com/nu-dialogue/j-moshi/blob/main/README-en.md" class="btn border-white bg-white fw-bold" lang="en">Code</a>
        </p>
        <p class="mb-0">
          <a href="https://atsumoto.github.io"><span lang="ja">大橋 厚元</span><span lang="en">Atsumoto Ohashi</span></a>,
          <a href="https://shinyaaa1003.github.io/"><span lang="ja">飯塚 慎也</span><span lang="en">Shinya Iizuka</span></a>,
          <a><span lang="ja">姜 菁菁</span><span lang="en">Jingjing Jiang</span></a>,
          <a href="https://www.ds.is.i.nagoya-u.ac.jp"><span lang="ja">東中 竜一郎</span><span lang="en">Ryuichiro Higashinaka</span></a>
        </p>
        <p>
          <b lang="ja">名古屋大学 大学院情報学研究科</b>
          <b lang="en">Graduate School of Informatics, Nagoya University</b>
        </p>
      </div>
      <p>
        <b lang="ja" class="active-lang">概要:</b>
        <b lang="en">Abstract:</b>
        <span lang="ja" class="active-lang">
          人間同士の対話における発話のオーバーラップや相槌など，同時双方向的な特徴をモデル化できるfull-duplex音声対話システムは，近年注目を集めている．しかし日本語においては，full-duplex音声対話システムはほとんど見られず，full-duplex音声対話システムの開発に関する知見は不足している．本研究では，英語における主要なfull-duplex音声対話システムであるMoshi<span style="font-size: x-small; vertical-align: super;">[<a href="#fn-1">1</a>]</span> をベースとすることで，日本語で利用可能な最初のfull-duplex音声対話システム J-Moshi<span style="font-size: x-small; vertical-align: super;">[<a href="#fn-2">2</a>]</span> を試作し，公開する．
        </span>
        <span lang="en">
          Full-duplex spoken dialogue systems, which can model simultaneous bidirectional features of human conversations such as speech overlaps and backchannels, have attracted significant attention recently. However, the study of full-duplex spoken dialogue systems for the Japanese language has been limited, and the research on their development in Japanese remains scarce. In this paper, we present the first full-duplex spoken dialogue model in Japanese, which is built upon Moshi,<span style="font-size: x-small; vertical-align: super;">[<a href="#fn-1">1</a>]</span> a major full-duplex dialogue model in English. Our model, J-Moshi,<span style="font-size: x-small; vertical-align: super;">[<a href="#fn-2">2</a>]</span> is trained through a two-stage process: pre-training on a large-scale spoken dialogue data in Japanese, followed by fine-tuning on high-quality stereo spoken dialogue data. We further enhance the system's performance by incorporating synthetic dialogue data generated by a multi-stream text-to-speech system.
        </span>
      </p>
      <div class="container mb-4">
        <div class="row justify-content-center">
          <div class="col-12 col-md-10">
            <div class="ratio ratio-16x9">
              <video class="object-fit-contain" controls>
                <source src="static/video/oh-1-title.mp4" type="video/mp4">
                Your browser does not support HTML video.
              </video>
            </div>
          </div>
        </div>
      </div>
      <div class="footnote">
        <hr class="footnote-divider">
        <small class="text-muted">
          <ul style="list-style: none;">
            <li id="fn-1" lang="ja">[1] J-Moshi のベースとなった Moshi の詳細については，<a href="https://arxiv.org/abs/2410.00037">公式のテクニカルペーパー</a>を参照してください．</li>
            <li id="fn-1" lang="en">[1] For more details about Moshi, the base model of J-Moshi, please refer to the <a href="https://arxiv.org/abs/2410.00037">official technical report</a>.</li>
            <li id="fn-2" lang="ja">[2] 本ページにおける音声対話のデモ動画では，わかりやすさのためモデル名を "J-Moshi" と表記していますが，実際は，音声合成による拡張データによって学習された J-Moshi-ext を使用しています．</li>
            <li id="fn-2" lang="en">[2] In the demo videos on this page, the model name is shown as "J-Moshi" for clarity, but we actually use J-Moshi-ext, which is trained with augmented data from speech synthesis.</li>
          </ul>
        </small>
      </div>
    </div>


    <div class="container shadow p-5 mb-5 bg-white rounded">
      <h3 lang="ja">リアルタイム音声対話</h3>
      <h3 lang="en">Real-time Spoken Dialogue</h3>
      <p lang="ja">J-Moshiとユーザによる実際の音声対話のサンプル．</p>
      <p lang="en">Samples of real-time spoken dialogue between J-Moshi and users.</p>
      <div class="container py-4" id="demo-video-container"></div>
    </div>


    <div class="container shadow p-5 mb-5 bg-white rounded">
      <h3 lang="ja">対話継続（Prompted Dialogue Continuation）</h3>
      <h3 lang="en">Prompted Dialogue Continuation</h3>

      <p lang="ja">
        人間同士の10秒の対話音声（プロンプト）から，以下の各モデルが生成した20秒の対話音声サンプル．
      </p>
      <p lang="en">
        20-second audio samples generated by each model from a 10-second human-to-human dialogue audio prompt.
      </p>

      <ul lang="ja">
        <li><b>Re-synthesis:</b> 実際の20秒の対話音声を，Moshiの音声トークナイザMimiによって再合成した音声</li>
        <li><b>dGSLM:</b> 日本語音声対話データによって学習された<a href="https://aclanthology.org/2023.tacl-1.15/">dGSLM</a></li>
        <li><b>J-Moshi:</b> 日本語音声対話データによって学習されたMoshi</li>
        <li><b>J-Moshi-ext:</b> 日本語音声対話データとMulti-stream TTS<span style="font-size: x-small; vertical-align: super;">[<a href="#fn-3">3</a>]</span>による合成音声データで学習されたMoshi</li>
      </ul>
      <ul lang="en">
        <li><b>Re-synthesis:</b> Actual 20-second dialogue audio re-synthesized by Moshi's audio tokenizer Mimi</li>
        <li><b>dGSLM:</b> <a href="https://aclanthology.org/2023.tacl-1.15/">dGSLM</a> trained on Japanese spoken dialogue data</li>
        <li><b>J-Moshi:</b> Moshi finetuned on Japanese spoken dialogue data</li>
        <li><b>J-Moshi-ext:</b> Moshi finetuned on Japanese spoken dialogue data and synthetic audio data generated by multi-stream TTS<span style="font-size: x-small; vertical-align: super;">[<a href="#fn-3">3</a>]</span></li>
      </ul>

      <p lang="ja">
        以下の音声サンプルのうち，ベルが鳴るまでの10秒間がプロンプト音声であり，その後の20秒間が各モデルによって生成された音声です．
      </p>
      <p lang="en">
        In the following audio samples, the first 10 seconds until the bell rings is the prompt audio, and the following 20 seconds is the audio generated by each model.
      </p>

      <div class="container pt-3 table-responsive">
        <table class="table table-hover audio-table" id="continuation-table"></table>
      </div>
      <div class="footnote">
        <hr class="footnote-divider">
        <small class="text-muted">
          <ul style="list-style: none;">
            <li id="fn-3" lang="ja">[3] Multi-stream TTS については，<a href="https://arxiv.org/abs/2410.00037">Moshiテクニカルペーパー</a>のAppendix Cを参照してください．</li>
            <li id="fn-3" lang="en">[3] For details on Multi-stream TTS, please refer to Appendix C of the <a href="https://arxiv.org/abs/2410.00037">Moshi technical report</a>.</li>
          </ul>
        </small>
      </div>
    </div>


    <div class="container shadow p-5 mb-5 bg-white rounded">
      <h3>Multi-stream TTS</h3>
      <p lang="ja">
        Multi-stream TTSによって，対話テキストから合成されたステレオ対話音声サンプル．
      </p>
      <p lang="en">
        2-channel dialogue audio samples synthesized from dialogue text using Multi-stream TTS.
      <div class="container pt-3 table-responsive">
        <table class="table table-hover" id="mstts-table"></table>
      </div>
    </div>


    <div class="container shadow p-5 mb-5 bg-white rounded">
      <h3 lang="ja">謝辞</h3>
      <h3 lang="en">Acknowledgments</h3>
      <p lang="ja">
        本研究は，JSTムーンショット型研究開発事業，JPMJMS2011の支援を受けました．雑談対話コーパスおよび相談対話コーパスは，株式会社アイシンとの共同研究において構築しました．また本研究では，名古屋大学のスーパーコンピュータ「不老」を利用しました．最後に，Moshi のテクニカルペーパーおよびモデルを公開していただいた Kyutai Labs に感謝いたします．
      </p>
      <p lang="en">
        This research was supported by the JST Moonshot R&D Program, JPMJMS2011. Part of dialogue data were constructed in joint research with Aisin Corporation. This research also utilized Nagoya University's supercomputer "Flow". Finally, we would like to thank Kyutai Labs for releasing the Moshi technical report and the models.
      </p>
      <a href="https://avatar-ss.org/"><img src="static/image/moonshot_logo.svg" width="200"></a>
    </div>


    <div class="container shadow p-5 mb-5 bg-white rounded">
      <h3 lang="ja">お問い合わせ</h3>
      <h3 lang="en">Contact</h3>
      <p lang="ja">
        J-Moshiに関するお問い合わせは，<a href="https://www.ds.is.i.nagoya-u.ac.jp">東中研究室</a>までお願いいたします．
      </p>
      <p lang="en">
        For inquiries regarding J-Moshi, please contact the <a href="https://www.ds.is.i.nagoya-u.ac.jp">Dialogue System Research Group</a> at Nagoya University.
    </div>


    <div class="container p-5 mb-5 bg-white rounded">
      <p class="mb-0">
        This page was adapted from the <a href="https://google-research.github.io/seanet/soundstorm/examples">SoundStorm project page</a>.
      </p>
    </div>


  </body>
</html>