Hangs / freezes when searching large directories, especially when JS files (e.g. maps) are found

[johnmaguire]

Similar to #19 but I have experienced this both with the_silver_search and ripgrep.

ripgrep on its own does not exhibit the same behavior, confirmed.

A good way to repro it is to go to ~ and run rg export. If you have many source code files scattered throughout your home directory (especially ones which contain JS files), you're likely to hit this. It will always hang just after listing a file name, prior to listing results found within that file.

Furthermore, if piping the tag command to something like | wc -l, it does not hang.

(P.S. Thanks for this tool - I've been using it for a long time, but never gotten around to posting this issue.)

[jasikpark]

It looks like the difference there is whether generateTags or passthrough is called?

tag/tag.go

Lines 238 to 245 in a1f5b04

	if disableTag || !isatty(os.Stdin) || !isatty(os.Stdout) {
	// Data being piped from stdin
	os.Exit(passThrough(cmd))
	}
	handleColorSetting(searchProg, finalArgs)
	os.Exit(generateTags(cmd))
	}

[jasikpark]

That leads me to believe that there must be faulty logic causing the hang in for Scanner.scan()

tag/tag.go

Lines 116 to 167 in a1f5b04

	func generateTags(cmd *exec.Cmd) int {
	cmd.Stderr = os.Stderr
	stdout, err := cmd.StdoutPipe()
	check(err)
	scanner := bufio.NewScanner(stdout)
	scanner.Split(bufio.ScanLines)
	var (
	line []byte
	colorlessLine []byte
	curPath string
	groupIdxs []int
	)
	aliasFile := NewAliasFile()
	defer aliasFile.WriteFile()
	aliasIndex := 1
	err = cmd.Start()
	check(err)
	for scanner.Scan() {
	line = scanner.Bytes()
	colorlessLine = ansi.ReplaceAll(line, nil) // strip ANSI
	if len(curPath) == 0 {
	// Path is always in the first line of a group (the heading). Extract and print it
	curPath = string(colorlessLine)
	curPath, err = filepath.Abs(curPath)
	check(err)
	fmt.Println(string(line))
	} else if groupIdxs = lineNumberRe.FindSubmatchIndex(colorlessLine); len(groupIdxs) > 0 {
	// Extract and tag matches
	aliasFile.WriteAlias(
	aliasIndex,
	curPath,
	string(colorlessLine[groupIdxs[2]:groupIdxs[3]]),
	string(colorlessLine[groupIdxs[4]:groupIdxs[5]]))
	fmt.Printf("%s %s\n", tagPrefix(aliasIndex), string(line))
	aliasIndex++
	} else {
	// Empty line. End of grouping, reset curPath context
	fmt.Println(string(line))
	curPath = ""
	}
	}
	err = cmd.Wait()
	return extractCmdExitCode(err)
	}

[jasikpark]

I think maybe it happens when there are multiple matches per line? For example, I had it just fail on a minified js file which had multiple module.exports statements when searching for export.

The entire file is just one line w/o newlines, since the js is minified.

[johnmaguire]

This patch:

diff --git a/tag.go b/tag.go
index b54e9e2..486889e 100644
--- a/tag.go
+++ b/tag.go
@@ -162,7 +162,13 @@ func generateTags(cmd *exec.Cmd) int {
                }
        }

+       if err := scanner.Err(); err != nil {
+               fmt.Println(err)
+       }
+
+       fmt.Println("waiting")
        err = cmd.Wait()
+       fmt.Println("unwaiting")
        return extractCmdExitCode(err)
 }

Generates the following error before hanging:

bufio.Scanner: token too long
waiting

[christopherthielen]

FYI I submitted a PR that bumps the maximum line buffer size from 64k to 1MB.

you can install my fork via homewbrew:

brew tap christopherthielen/tag-ag
brew install tag-ag