AI Chatbot Website Crawling: What the Bot Reads

AI chatbot website crawling is the step where the assistant reads your public pages so it can answer from them, and it decides how good the first answers will be. In Talkmio you add your website to the Mio knowledge base and Mio reads every public page on the same domain, respecting your robots.txt file. That sounds simple, yet most “the bot did not know that” complaints trace back to the crawl: a page that was blocked, a page that is not public, a page that was changed after the last read or a limit that was reached. This guide explains what a crawl reads, what it skips and how to check and fix each case.

Talkmio is an AI live chat for websites. Mio answers only from the content you provide, in the visitor’s language, and hands the conversation to your team when it cannot answer. Because everything comes from your content, the quality of the crawl is the quality of the assistant.

What AI Chatbot Website Crawling Actually Does

A crawler starts at an address you give it, reads the page, finds links to other pages on the same domain and reads those too. The text it collects is stored as pieces of knowledge. When a visitor asks a question, Mio looks for the pieces that match the meaning of the question and writes an answer from them. Nothing is memorised from the open internet, and nothing outside your content is used as a source, which is why Mio does not invent prices or policies.

This design has two consequences. The assistant can only know what the crawl reached and what you added by hand, and it can be wrong only in the ways your content is wrong. Our overview of knowledge bases for AI chatbots describes the idea in general terms. Here we focus on the practical side: how to make the crawl find what matters.

What Mio Reads

According to the Talkmio documentation, under Mio AI and Knowledge base you can add your website, single pages, Q&A pairs, free text or files. When you add your website, Mio reads every public page on the same domain and follows your robots.txt rules. Files can be PDF, DOCX, TXT, MD, CSV or HTML, up to 15 MB each. Each source can be read again on demand with Re-read.

Source type	Best for	Watch out for
Website	Product pages, policies, FAQ, blog articles	Only public pages on the same domain; blocked or login pages are not read
Single page	One important page outside the main crawl, such as a help article on another address	It does not update unless you re-read it
Q&A pair	Exact answers you want worded your way	Keep them short and update them when policy changes
Free text	Facts that are on no page, such as internal rules for tone or hours	No source link for the visitor to follow
File	Price lists, manuals, terms, catalogues	Scanned images without text are hard for any text-based tool to use

The right mix is usually a crawl of the website for the broad base, files for documents that are not on the site and a handful of Q&A pairs for the questions that matter most. The guide to building an AI chatbot from a PDF shows how documents fit in.

Robots.txt: The File That Can Hide Your Best Pages

The robots.txt file sits at the root of your site and tells automated visitors which paths they may fetch. Mio respects it, which is the polite and correct behaviour. It also means a rule written for search engines can accidentally hide pages from your chatbot. A line that blocks a folder of help articles, a staging copy or a whole language directory will block them for the crawl as well.

Open your own file at yourdomain.com/robots.txt and read it line by line. Ask of each Disallow rule: do I want my assistant to answer questions from these pages? Blocking carts, account areas and internal search results is sensible, because they hold nothing worth quoting. Blocking pricing, shipping or returns pages is almost always a mistake. The introduction to robots.txt from Google explains the syntax, and robotstxt.org is a neutral reference for the format.

If you must keep a page blocked from crawlers but want the assistant to know its content, add the content as a single page, a file or free text instead. That way you choose exactly what Mio can say.

What a Crawl Cannot Reach

Pages behind a login

A crawler is a visitor without an account. Member areas, customer portals and gated downloads are not public, so they are not read. If your support answers live there, copy the relevant parts into a public help page or upload them as a file.

Other domains and subdomains

The website source covers pages on the same domain. If your help centre, shop or documentation lives at a different address, add it separately as its own source or as single pages, then check in Try Mio that its answers appear. Do not assume; test with a question that only that site can answer.

Text inside images

Prices in a banner image, delivery times in a graphic or opening hours in a photo are invisible to a text reader. Write the same facts as text on the page. That also helps search engines and visitors who use screen readers.

Pages that need a click or a form

Content that appears only after a visitor picks an option, enters a postcode or submits a form may not exist as a page the crawler can read. If a tool or calculator holds the facts, write the main rules down on an ordinary page.

Plan Limits and Choosing What to Keep

Each plan limits how many knowledge base pages Mio can keep: 20 on Free, 200 on Pro, 1,000 on Ultimate and more on the larger Enterprise plans, according to the pricing page. A site with hundreds of blog posts can easily exceed the free limit. Decide deliberately which pages deserve a place. Policies, pricing, shipping, returns, product specifications, booking rules and FAQ pages usually come first. Old news, event announcements and thin tag pages come last.

When you hit a limit, fewer good pages beat many weak ones. Mio finds meaning, not volume, so a clear returns page will answer returns questions better than ten articles that mention returns in passing. If your site has overlapping pages that disagree, fix them first; our guide to conflicting sources explains why contradictions produce vague answers.

How to Check a Crawl in Ten Minutes

Add your website under Mio AI, Knowledge base and wait for the read to finish.
List twenty questions customers really ask, using your inbox, phone log or past chats.
Ask each one in Try Mio and look at the answer and the knowledge pieces used.
For every wrong or missing answer, find the page that should have supplied it and check whether it is public, linked from somewhere and not blocked in robots.txt.
Fix the cause: unblock the path, publish the page, link it from the menu or footer, or add the content as a file, a single page or a Q&A pair.
Use Re-read on the changed source and ask the same question again.

Our checklist for testing an AI chatbot before launch adds further cases such as language and tone. Repeat the check after major site changes.

A Worked Example: Fixing One Missing Answer

Imagine a small online shop. A visitor asks, “Do you ship to Norway, and how long does it take?” Mio answers that it cannot find that information and hands over to a person. The owner opens Try Mio, asks the same question and sees that no knowledge piece matched. The first suspect is the crawl, so she checks three things in order. Is there a public page about international shipping? Yes, at an address under the shipping folder. Is the page linked from the menu or footer? No, it is only reachable from an old newsletter. Does robots.txt block the folder? Yes, an old rule hides the entire shipping directory from all crawlers.

The fix takes five minutes. She removes the blocking rule, adds a link to the page in the footer, presses Re-read on the website source and asks the question again. This time Mio quotes the list of countries and the delivery range from the page. She also adds a short Q&A pair for the Norway question, because it is asked often and she wants the wording to be exact. The lesson is general: when an answer is missing, work backwards from the question to the page, then from the page to the public address, the link and the robots rule.

Sites in Several Languages

If your website exists in more than one language, each language usually lives in its own folder or on its own address. Mio answers in the visitor’s language, and a question in one language can find an answer written in another, so a complete English site can serve visitors who write in German or Polish. That is a good reason to keep one language fully up to date before you translate the rest. Check that robots.txt does not block any language folder, and test one question per language in Try Mio.

Translated pages that differ from the original are a risk, because Mio may find both versions and the answers can disagree. If you are not sure that a translation is current, leave it out of the knowledge base and let the English page be the source of truth. Decide on one authoritative version of each policy, and remove or redirect outdated copies, since an old page that is still public will still be read.

Keeping the Knowledge Fresh

A crawl is a snapshot. When you change a price, a delivery time or a policy, the assistant still holds the older text until the source is read again. Sources are re-read on demand, so build a habit: whenever someone updates an important page, someone presses Re-read. The article on keeping the knowledge base updated suggests simple routines, such as a monthly review and a note in your release checklist. Plans and page limits are on the Talkmio pricing page.

Finally, remember that the assistant’s reach is a design choice. If a topic is risky, leave it out of the crawl on purpose and add a business instruction that sends it to a person. Deciding what Mio must not say is part of crawling well.

Frequently Asked Questions

Does Mio read my whole website automatically?

When you add your website, Mio reads every public page on the same domain and follows your robots.txt rules. Pages behind a login, pages blocked in robots.txt and pages on other domains are not included unless you add them as separate sources.

Does Mio obey robots.txt?

Yes. The documentation says the website source respects robots.txt. If a rule blocks a folder of help pages, Mio will not read them, so review your file and add any needed content as a single page, file or free text.

How do I make Mio read a page again after I change it?

Sources are re-read on demand and nothing updates silently. Open the source in the Knowledge base and use Re-read, then test the same question in Try Mio. Without a re-read, the assistant keeps answering from the older version of the page, even if the live page is correct.

Is there a limit on how many pages Mio can keep?

Yes, and it depends on your plan: 20 pages on Free, 200 on Pro and 1,000 on Ultimate, with larger Enterprise plans above. Prioritise policies, pricing, shipping, returns and FAQ pages before old news or thin archive pages.

Can Mio use prices that appear only inside an image?

Do not rely on it. Text inside images is hard for text-based tools to read, so write prices, hours and delivery times as ordinary text on the page. That also helps search engines and visitors who use screen readers.

How can I see which pages Mio used for an answer?

Use Try Mio in your account. It shows the answer, the knowledge pieces that were used and how long it took, so you can trace a wrong answer back to the page or document and fix the source.

The Bottom Line

AI chatbot website crawling is not a one-click guarantee; it is a process you check. Open your robots.txt, make sure key pages are public, add what the crawl cannot reach, stay inside your plan’s page limit and re-read after every important change. Then test twenty real questions and fix the sources behind any miss. Start free with Talkmio and see exactly what Mio reads on your site.

AI Chatbot Website Crawling: What Mio Reads, Skips and Re-Reads