How can I make it work harder ?

[C4H7Cl2O4P]

load average ~0.6
CPU load ~5-60%
RAM used ~500 Mb

HDD OK

$ iostat -xz 1
Linux 6.12.90+deb13.1-amd64 (zoo) 	06/06/26 	_x86_64_	(4 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.48    2.00    0.56    0.34    0.00   96.62

Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
sda              0.27     14.08     0.05  16.80   14.11    52.85    2.36    175.98     2.33  49.70   15.63    74.69    0.00      0.00     0.00   0.00    0.00     0.00    0.60   14.45    0.05   1.36

net bandwidth is not loaded

---

Why doesn't it work hard ?

[smokingwheels]

Try crawiling 500 or more sites at once Yahoo Mail: Search, organise, conquer On Sat, 6 June 2026 at 8:42, ***@***.***> wrote: C4H7Cl2O4P created an issue (yacy/yacy_search_server#799) load average ~0.6 CPU load ~5-60% RAM used ~500 Mb HDD OK $ iostat -xz 1 Linux 6.12.90+deb13.1-amd64 (zoo) 06/06/26 _x86_64_ (4 CPU) avg-cpu: %user %nice %system %iowait %steal %idle 0.48 2.00 0.56 0.34 0.00 96.62 Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util sda 0.27 14.08 0.05 16.80 14.11 52.85 2.36 175.98 2.33 49.70 15.63 74.69 0.00 0.00 0.00 0.00 0.00 0.00 0.60 14.45 0.05 1.36 net bandwidth is not loaded Why doesn't it work hard ? — Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS and Android. Download it today! You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[C4H7Cl2O4P]

Excuse me, I am a newbie here. What settings should I change and how ?
System Administartion --> Crawler Settings
has only
Generic Crawler Settings:
HTTP Crawler Settings:
FTP Crawler Settings:
SMB Crawler Settings:
Local File Crawler Settings:

---

http://host:8090/PerformanceQueues_p.html
Thread Pool Settings:
Crawler Pool

It is already = 500

[mmeier86]

I assume you mean how to make it crawl more? As @smokingwheels said, you'd have to crawl more different sites at once. YaCy's crawler has a throttle build in which ensures that it only makes at most 4 req/s to any given host/domain. This is to avoid DDoS'ing the crawled server and is generally considered being well-behaved. Hammering a page with too many requests a once might very well get your IP blocked entirely.

I got to the 4 req/s number because when I'm doing single-host crawls, I'm getting somewhere around 240 pages per minute crawled, and I think I also read it somewhere, but couldn't find it right now.
So you will only get past the 240 PPM value and properly use your available resources by crawling multiple different sites/domains at the same time.

[C4H7Cl2O4P]

... it only makes at most 4 req/s to any given host/domain. ...

Thank you very much for your reply. Can crawling by my host help other people ? If I collect information about web-sites which I don't need is it useful to somebody. If it were so, then I guess I should have had tons of request from other nodes about requests they got from their users. I this case there should be some incoming port open but I did not set it up. Does it all mean that I collect (by crawling) information only to those people who use my node ?
I guess not, because before I started intentional crawling I already could get search results.
Where can I read how is it working ?

[mmeier86]

You're welcome. 🙂

Yes, crawling by your host does help a little bit, at least from my current understanding (I've only been running YaCy for about two weeks now, so take everything I'm saying with a grain of salt 😉 ). After you've crawled some pages, you should see log lines starting with DHT-OUT in your logs. Those indicate your host sending pieces of your local index data to other peers.

As long as you don't have an open port and your peer running in at least "Senior" mode, you don't fully participate in the P2P network. Specifically, other peers can't send you "unsolicited" pieces of their index. But they will still send you search results, because that's triggered by requests coming from your instance, while the background index exchanges are triggered by remote hosts.
In addition, you won't be send remote search requests yourself, because other instances don't have a way to actually send you their search requests, because you don't have an external port opened.

You can find a little bit of information in YaCy's docs about how all of this is working.

So in short: Yes, you're already contributing a little bit to the global P2P search index, but without an open port, you won't participate in remote searches coming from other instances, and background index exchanges will be restricted.