I want to find users of Heritrix3 engine
I have such setup, but with some problems:
- it always does full crawl (doesn't do deduplication)
- I am unable to control crawl order (don't have enough knowledge)
Syndicated from the fediverse. Read and engage on the original instance.
View original on xn--e1aghfa.xn--e1aqbccjfc.xn--p1acf
1 reply
kreuzcrawl — a high-performance Rust web crawling engine.