About our crawler

recruitlayer fetches public articles, feeds, and career-site information to support hiring intelligence and configured recruiting workflows.

Our identity

Our public content fetcher identifies itself as recruitlayer-hiring-intelligence/1.0, with a link to this page in its User-Agent. It is an automated HTTP client, not a web browser. Some existing career-site integrations use different request headers while their compatibility with this identity is evaluated.

Robots.txt and access

Our Scrapy-based Wrap crawler checks robots.txt while using a browser-style User-Agent. A wildcard rule can disallow that crawler, but it may also disallow other standards-compliant crawlers:

User-agent: *
Disallow: /

Direct article, feed, preview, and API fetches do not currently evaluate robots.txt, including requests that use our named crawler identity. The wildcard rule therefore does not block every recruitlayer request. Contact us to request an exclusion or resolve unwanted traffic. Direct HTTP fetches receiving a denial or rate-limit response are not retried against other IP addresses during the same fetch.

Contact us

Use our contact form with your domain, the affected paths, and approximate request times. Please omit passwords, access tokens, and other private information from examples.