To better understand the meaning and workings of robots.txt, to make it more understandable, I will make like the answer to question Regarding the article Hummingbird algorithm.
What is the robots.txt?
Robots.txt is a command for the search engine robots to explore or do not browse the pages on our blog. Arguably robots.txt is filter our blog from search engines.
Does every blog has a robots.txt?
Yes, all the blogs already have a robots.txt given by bloggers. By default robots.txt on a blog like this:
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: / search
Allow: /
Sitemap: http: // namablog / feeds / posts / default? Orderby = UPDATED
To view the default robots.txt, please type dibrowser http://namablogsobat.blogspot.com/robots.txt
What is the meaning of the code above?
User-agent: Mediapartners-Google
Google Adsense robot crawls blog
Disallow:
Which is not allowed no
User-agent: *
All robots Search Engine / Search engine
Disallow: / search
Not allowed to crawl the search folder and setrusnya, like .... / search / label and ... search / search? Updated ...
Allow: /
Allow all pages to be crawled, except written on Disallow above. Sign (/) less means the name of the blog.
Sitemap: http: // namablog / feeds / posts / default? Orderby = UPDATED
sitemap or feed address our blog.
READ ALSO :
Why label are excluded from the search?
By writing the default code Disallow: / search, it means crawling to label such http://namablog.com/search/label/Tutorial address, will not be crawled by search engines because the label is not the real URL toward a specific page.
How can I prevent the robot on a particular page?
To prevent Google from crawling certain pages on your blog, do not want to tell my friend about me pages indexed by search engines. For the URL about me on your blog eg http://namablogsobat.blogspot.com/p/about.html
So to robots.txt, please copy the code defaut above, tambahakan page that is not allowed, the result is like this:
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: / search
Disallow: /p/about.html
Allow: /
Sitemap: http: // namablog / feeds / posts / default? Orderby = UPDATED
Where is the way to edit robots.txt?
To add or editing robots.txt, go to Settings> Preferences Search, on Robots.txt click Edit and then click Yes, then write the desired code robots.txt> Save Changes
I never permutations robots.txt, how?
It's okay, your blog will still be crawled by search engine robots because as I mentioned before, every blog is already possessed default robots.txt.
Be careful with the use of robots.txt, if one could be writing your blog are ignored by search engines.
0 Comments