Search
Close this search box.
What is robots.txt?

Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion Protocol.
In practice, robots.txt files indicate whether certain user agents (web-crawling software) can or cannot crawl parts of a website.

Related terms (by category)

Knowledge Graph

Google’s search results often display information sourced from the Knowledge Graph, a vast database containing billions of facts about people, places, and things.
Read More »

knowledge panels

Knowledge panels are information boxes that show up on Google when you search for specific entities like people, places, organizations, or things that are part of the Knowledge Graph.
Read More »

Parasite SEO

Parasite SEO is a strategy where marketers leverage the authority and ranking power of high-traffic websites to promote their content or links.
Read More »
Related terms (by alphabet)