How-to Guides – advertools

How to crawl using proxies with Python

Crawling

advertools

proxy

scrapy

A Python script showing how the advertools crawl function can easily incorporate a proxy. We can also use an additional library to auomatically rotate a set of proxies as well.

Install advertools

Installation is simple and straightforward, from the command line run:

Create a broken link checker with Python

crawling

scraping

auditing

advertools

A Python script that takes a an advertools crawl file, maps the links on all pages, finds broken internal links and locates them. Also runs the same for external links.

How to Download, Parse, and Visualize XML Sitemaps

data visualization

analytics

SEO

XML sitemaps

URLs

plotly

A Python script using advertools to download, parse, and save the sitemap to a CSV file. We will also visualize the URL structure and publishing trends of the website

Getting status codes of URLs in XML sitemaps

status codes

auditing

SEO

XML sitemaps

robots.txt

A Python script using advertools to check the status codes of all URL in an XML sitemap (or sitemapindex), simply by supplying the robots.txt URL.

How to check if URLs in XML sitemaps are blocked by robots.txt

robots.txt

XML sitemaps

SEO

advertools

A Python script using advertools on how to bulk check if URLs in your XML sitemap are blocked by rules in your robots.txt file. This will be run in bulk for all combinations of user-agent:URL.

How to audit structured data on a website with Python

Python

advertools

Structured Data

Crawling

Auditing

Analysis

Intermediate

A Python script that takes a site crawled with advertools and provides mechanisms for understanding structured data; which are used (JSON-LD, Twitter, and OpenGrap), which tags are present, and what they contain.

How to download images in bulk with Python

Python

advertools

crawling

images

automation

Intermediate

A Python script for downloading images in bulk from a set of given URLs. You can also set contstraints like minimum width and/or height, as well as matching a regex for image file names.