CornerBot

CornerBot uses two user agent strings. Both name CornerBot and link to this page.

The crawler

CornerBot/1.0 (+https://getcorner.ai/bot; reads pages its owner added to their website assistant)

It reads your pages and the PDFs they link to. Some websites only show their content once JavaScript runs. For those pages it opens a headless browser with the same user agent, which loads the page and its scripts but not its images, videos or fonts.

The site check

Mozilla/5.0 (compatible; CornerBot/1.0; +https://getcorner.ai/bot)

It loads one page, the website's address, when its owner reaches the install step in Corner or asks it to check again. Corner's team can run the same check while helping an owner. It looks only at the page's Content Security Policy, to tell the owner which lines to add so the messenger can load, and keeps nothing else from the page.

CornerBot never wanders the web. It only visits a website someone gave to Corner:

• A business setting up its assistant. CornerBot reads the website once, then again every day or every week if the owner chooses (weekly is the default). The owner can also ask it to read again at any time.
• Someone trying the free demo at getcorner.ai/try. It reads up to 20 pages, and the demo and everything it read are deleted after 24 hours. Anyone can type a website into the demo, so CornerBot may read a few pages of a website whose owner has never used Corner. robots.txt applies here too.
• A demo Corner makes for a business. Corner's team can build a demo assistant from a business's website to show its owner. CornerBot reads up to 60 pages, and the demo and everything it read are deleted after 7 days, unless the owner keeps it by signing up. robots.txt applies here too.

It stays on the website it was given and doesn't follow links to other websites.

• Reads: public pages, starting from the address it was given and following links up to 4 steps away, plus the pages in your sitemap. PDFs linked from those pages are read too.
• How many: up to the owner's plan limit, from 100 to 5,000 pages per website, up to 20 for the free demo and up to 60 for a demo Corner makes for a business.
• Skips: login, account, register, cart and checkout pages, WordPress admin pages, feeds, search results, pages marked noindex in a robots meta tag (other than the address it was given), and files that aren't pages or PDFs, such as images, videos and office documents.
• Never: it doesn't sign in, fill in forms or click anything.
• Pace: up to 4 pages at a time. If your robots.txt sets a Crawl-delay, it reads one page at a time and waits that long between pages, up to 5 seconds.

To find your website's main address (for example from yourbusiness.com to www.yourbusiness.com), it loads the address it was given once before it reads robots.txt. If robots.txt then blocks it, it stops there and keeps nothing.

CornerBot follows the robots.txt file at the root of your website. To block it, add these lines:

User-agent: CornerBot
Disallow: /

To keep it out of part of your website only, name that part instead, for example Disallow: /private/. If your robots.txt has no rules for CornerBot, it follows the rules for all crawlers (User-agent: *).

A website that blocks CornerBot can't be read into Corner: whoever adds it sees that robots.txt blocks Corner. If it's your website and you want Corner to read it, allow CornerBot:

User-agent: CornerBot
Allow: /

The site check is a single request for the website's owner, so it doesn't read robots.txt. See how your assistant learns from your website for the owner's side.

If CornerBot causes a problem on your website, write to [email protected] with your website's address and we'll look into it.