FirsthandBot

The crawler behind firsthandcareers.com, a job search engine that reads openings directly from company career sites.

How it identifies itself

Mozilla/5.0 (compatible; FirsthandBot/1.0; +https://firsthandcareers.com/bot)

When a page needs JavaScript we render it in Chromium, and the user agent is then Chromium's own string with FirsthandBot/1.0 appended — so you can see both what is actually rendering the page and who is asking.

What it does

• Reads the listing page of a company's careers site, and each posting's own page.
• Keeps the job title, location, team, posting date, description and apply link.
• Visits once a night. One page at a time per host, with a delay between requests.
• Sends people to your posting to apply. We do not host applications, resell your listings, or republish them as our own.

What it does not do

• It does not collect personal data, email addresses, or anything behind a login.
• It does not submit forms or create accounts.
• It does not ignore robots.txt, and it does not disguise itself to get past a block. If a site disallows us, we stop — including where the rule is aimed at everyone rather than at us by name.

How to stop it

Add this to your robots.txt and we will stop on the next run:

User-agent: FirsthandBot
Disallow: /

Or narrow it to part of the site:

User-agent: FirsthandBot
Disallow: /internal/

We honour Crawl-delay as well, and use it in place of our own delay whenever it asks for more room.

If something is wrong

If we are taking too much, taking the wrong thing, or you would rather send us a feed than be crawled at all, write to hello@firsthandcareers.com and we will fix it. A feed is genuinely easier for both of us.

Place data by GeoNames, CC BY 4.0. · Firsthand · firsthandcareers.com