Crawling Policy

Authorization

Use Rankelo only on public sites or resources you are authorized to inspect. Connecting or naming a target does not establish authorization. You are responsible for applicable contract, privacy, intellectual-property, computer-access, and sector requirements.

Safe access

Rankelo limits schemes to HTTP and HTTPS, resolves and pins public addresses, blocks local, private, metadata, reserved, and credential-bearing targets, bounds redirects, time, body size, pages, and concurrency, and restricts internal services. These controls must not be bypassed. OpenSERP is an internal search service and is not a route for arbitrary customer-provided targets.

Respect for sites

Rankelo should use reasonable rates, reuse unchanged content where reliable, respect legally applicable access restrictions and provider terms, and stop when blocked. It must not solve CAPTCHAs, imitate human identities deceptively, evade controls, exploit vulnerabilities, access private content, or intentionally burden a service. Search engines may block automated extraction; Rankelo reports the block and uses another permitted source where available.

robots.txt and user agent

Rankelo's crawler sends the user agent RankeloBot/1.0 (+https://rankelo.brandsap.com/crawling-policy.html). robots.txt groups for the product token RankeloBot apply to it; if no group names RankeloBot, the * group applies.

• robots.txt is read before any other page on a site, and each disallowed URL is skipped: entry pages, sampled pages, sitemap URLs, link checks and redirect destinations.
• A site whose robots.txt disallows the requested URL is not audited.
• A robots.txt that returns a server error or cannot be reached is treated as a full disallow; a 404 or other 4xx means no rules.
• Crawl-delay for RankeloBot (or *) is honoured between page requests; if the delay does not fit the audit time limit, the audit stops early instead of crawling faster.
• Rules follow RFC 9309: the longest matching path wins, Allow wins a tie, * and a trailing $ are supported.

Content and retention

Rankelo collects only data needed for the requested analysis, labels source and observation time, and follows the Privacy Policy retention schedule. Customers should exclude sensitive paths and avoid submitting secrets in URLs or page content.

Opt-out and concerns

Site operators may report a concern, authorization issue, or requested crawl exclusion to contact@brandsap.com. Include the domain, relevant URL, evidence of authority, and requested action. Rankelo may suspend a crawl while reviewing the request.