What sitemaps and robots.txt actually do
Both are basic technical SEO files and people mix them up all the time. The sitemap tells Google which pages to look at, robots.txt tells it where not to go. Getting them wrong can take your site out of search.
Nearly every site has these two files, and people often mix up what they do. A sitemap points Google to certain pages, and robots.txt closes the door on certain sections.
The sitemap (sitemap.xml)
A list of the pages on your site. It gives Google a tidy list of the addresses you want it to look at.
Google already finds your site by following links, but pages that nothing links to, or that sit deep in the structure, can get missed. The sitemap covers that gap.
A good sitemap lists a last-modified date next to each address, and Google uses it to decide which pages to revisit.
What to leave out of the sitemap
Redirected addresses (list the destination instead)
Pages marked noindex, which send Google a contradictory signal
Addresses that return 404
Admin panel and login pages
On this site the sitemap is generated from the database. Publish a post and it's added automatically, move it back to draft and it drops off. Nobody has to maintain it by hand.
robots.txt
A plain text file at the root of your site. It tells search engine bots which sections not to crawl.
A typical robots.txt tells bots to stay out of the admin panel and the search results pages, to crawl everything else, and where to find the sitemap.
The most common mistake
robots.txt doesn't hide a page from Google. It only says "don't crawl this page". If Google finds a link to that page on another site, it can still show it in results without reading it, and the description line comes up empty.
If you want a page never to appear in search, put a noindex tag on it. But if you also block crawling in robots.txt, Google can't read that tag, so using both gets you exactly the result you didn't want.
The line people forget at launch
During development, "Disallow: /" goes into robots.txt to keep the site out of search. Forgetting to remove it at launch is one of the most expensive mistakes I've seen.
The site is live and everything works, but Google crawls none of it. It can go unnoticed for months.
On launch day, open yoursite.com/robots.txt and check it yourself. It takes thirty seconds.