LumefrontBot

It reads a business's own website. It starts at the page given as the business's website, usually its home page, or at the site's front page if that page cannot be read. It then reads up to four more pages of the same site that the first page links to, such as its contact, about, menu, services, prices or booking pages.

It reads the site's robots.txt first, and fetches only the pages it allows. For these pages, and for robots.txt, it sends at most one request a second to any one host name, and waits longer between them if robots.txt sets a Crawl-delay, up to 10 seconds.

If Lumefront measures how fast a home page loads, it does so with Google's PageSpeed Insights, which loads the page from Google's own servers once LumefrontBot has read it.

When Lumefront makes a picture of what a new home page could look like for a business, LumefrontBot fetches up to 8 of the images on that business's pages for it: each once, one at a time, one straight after another. If making the picture is cut short, it can be made again later, and the images fetched again.

It does not read robots.txt for these images. Lumefront may make the picture some time after LumefrontBot read the pages.

Add these two lines to your site's robots.txt:

User-agent: LumefrontBot
Disallow: /

They stop LumefrontBot fetching your pages. It still reads robots.txt itself, to see them. It keeps a copy of a site's robots.txt for up to a day, so a change can take a day to count. If no group in your robots.txt names LumefrontBot, the group for all bots (User-agent: *) applies to it.

The lines are not checked when it fetches images. But it only fetches images that are on pages it has read, so if the lines have been in place for a day before it reads your pages, it fetches no images either. If it read your pages before then, it may still fetch up to 8 of the images on them when Lumefront makes the picture.

Write to lumefrontsta@gmail.com with any question about LumefrontBot, or to ask Lumefront not to contact your business: say which business and which website you mean. To stop LumefrontBot fetching your pages, add the two lines above to your robots.txt.