How To Crawl Specific Folders In Screaming Frog Using List Mode + Regex
Screaming Frog starts from the first URL that you paste in the box and crawls through all of the links that are connected to the that pasted URL.
So, what if you wanted to crawl through specific folders. Let’s pick an example here. You want to crawl through UAE-EN folder of noon.com and KSA-EN folder of noon.com but NOT the Egypt-EN folder. Here are your links:
Open up Screaming Frog and switch from Crawl mode to List mode (Where you can provide particular URLs)
Go to Config > Include and paste the two folders that you want to crawl. The .* at the end of each line in Regex would mean everything inside those folders.
The Regex validation will mean only EN lang folders of KSA and UAE get crawled but NOT Egypt or any language’s AR folders, even if the link persists on a crawled URL.
Tha’s it. Hit the Start button and give it some time to crawl.