follow us in feedly
How To Crawl Specific Folders In Screaming Frog Using List Mode + Regex

How To Crawl Specific Folders In Screaming Frog Using List Mode + Regex

Screaming Frog starts from the first URL that you paste in the box and crawls through all of the links that are connected to the that pasted URL.

So, what if you wanted to crawl through specific folders. Let’s pick an example here. You want to crawl through UAE-EN folder of noon.com and KSA-EN folder of noon.com but NOT the Egypt-EN folder. Here are your links:

INCLUDE

https://www.noon.com/uae-en/

https://www.noon.com/saudi-en/

EXCLUDE

https://www.noon.com/egypt-en/

Open up Screaming Frog and switch from Crawl mode to List mode (Where you can provide particular URLs)


Screaming Frog List Mode.JPG


Go to Config > Include and paste the two folders that you want to crawl. The .* at the end of each line in Regex would mean everything inside those folders.

Screaming Frog Include Folders Regex.JPG

The Regex validation will mean only EN lang folders of KSA and UAE get crawled but NOT Egypt or any language’s AR folders, even if the link persists on a crawled URL.

Tha’s it. Hit the Start button and give it some time to crawl.




Analyze Bounce Rate, Page Load Time By Devices In R Studio

Analyze Bounce Rate, Page Load Time By Devices In R Studio

Difference between Variable and Value Comparison Conditions in Adobe Launch

Difference between Variable and Value Comparison Conditions in Adobe Launch