# Allow all search engine bots User-agent: * Allow: / Disallow: /api/ Disallow: /forgot/ Disallow: /news/dev Disallow: /article/dev # /assets/ is intentionally crawlable: Googlebot needs the CSS/JS there to render pages, # and the .pdf files under it must be fetchable for the noindex header below to be seen # .apk/.pdf are intentionally crawlable: they are served with X-Robots-Tag: noindex # (see staticOptions in app.js), which Google can only honour if it is allowed to fetch them # Allow social media crawlers to access images User-agent: facebookexternalhit Allow: /images/ User-agent: Twitterbot Allow: /images/ User-agent: LinkedInBot Allow: /images/ # Block security scanners User-agent: Information Security Team InfraSec Scanner Disallow: / User-agent: InfraSec Scanner Disallow: / # Block unwanted AI scrapers (training-only, no search benefit) User-agent: CCBot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Amazonbot Disallow: / # Sitemap Sitemap: https://www.pospos.co/sitemap.xml