Up coming, you must come to a decision how you want the crawler to deal with a variety of World wide web systems. You can find an ongoing discussion about the intelligence of search engine crawlers. It isn't entirely distinct If they're full-blown headless browsers or simply glorified curl https://youtu.be/EVeTP0aV8gY?si=5FHbTy4kbLwUVoJD