A friend that I have, works with google adwords and oftenly he needs to check a list of urls to verify the Http status code of a list of urls. To verify that he’s not paying for a campaign that contain 404 pages.
So I took a weekend and build a simple site using Django and Celery. A simple three-page-website that uses an asynchronous task (in celery) to get the http status code of every page.
The code is open source in a github repository that you can fork, modify, create an issue, etc.
For now I set a limit of 300 urls, so it wont crash the server. The requests are sent one by one and it will use a normal user-agent. I’m working on a functionality that will allow you to add your custom user agent or a custom header on the request
Hope this helps someone.