robots.txt is a plain text file at the root of a website that tells crawlers which parts of the site they may and may not crawl.
Rules are set per user agent, so a site can treat search engines and AI crawlers differently. It can also point crawlers to the sitemap.
It controls crawling, not indexing: a blocked page can still appear in results if other sites link to it. Use a noindex tag to keep a page out of results.
Why it matters for a launch
A wrong rule can hide a whole site from search engines or AI assistants. Check it after every launch or site change.