borrow web seed functionality to add individual files without force rechecking

[sjpotter]

Imagine one has a 1TB torrent corresponding to hundreds or thousands of files. It's seeded, but not seeded well. However, the content is available in assorted places.

One has a few choices.

1. download it slowly from seeds
2. try to download as much as one can (perhaps at various speeds) put it all together, and then force rescan
3. download the individual pieces and copy it it into place as they download and force recheck each time (this is very expensive for very large torrents)
4. leverage web seeds (i.e. download locally, and setup a local http server to enable clients to download the pieces from it). This actually works, but is slow as it iterates through every single file of the torrent each time (which might be missing and would be wasteful once downloaded, as stored twice).

I want to propose a functionality that allows one to point at a single file on the local file system and do the same thing a what I'm accomplishing with web seeds, but have it targeted at a single file so that one doesn't have to iterate through every single file in the torrent on the web server.

Another possibility would be to have a directory that one can monitor for updates to its tree and treat it like a parallel client, i.e. I can drop files into it, and libtorrent based clients can know to pick up pieces from it if available. since it's local, no slow web based iteration would be needed, it would simply see the file exists, pass the available pieces in them to the web seed ingestion path and it just work without having to rescan the entire super large torrent.

[arvidn]

there is a programmatic way to add pieces to a torrent.

torrent_handle::add_piece().

Although, it would probably be more efficient to put the file in place and tell libtorrent to check just that one file. That would be a pretty specific feature though, a bit hard to justify supporting.

[sjpotter]

interesting. I'm wondering if there's a way to add a helper function (or perhaps one exists already) that can give us a a list of piece indexes and start/end offsets in a file to make this easier to implement?

In a sense then, the pretty specific feature would then just be a simple function using those 2 functions.

example (pseudo go like code, as my brain is in go right now)

type indexInfo struct {
    file string
    pos  uint64
}

type piecePos struct {
    start indexInfo
    end   indexInfo
}

func getPiecePositions() []piecePos

func addFilePieces(filePathInTorrent string, fileOnDisk string) {
   piecePositions := getPiecePositions()

   for i, piecePosition := range piecePositions {
       if piecePosition.start.file != filePathInTorrent || piecePosition.end.file != filePathInTorrent) {
           continue
       }
       data, err := readFileOffsetBytes(fileOnDisk, piecePosition.start.pos, piecePosition.end.pos)
       if err != nil {
           return
       }
       add_piece(i, data) // assumes pieces are 0 indexed, might not be true, easy modification
   }
}

so a few obvious issues, with the most obvious being that the above approach isn't meant for a callback/non blocking oriented system.