|
 |
|
|
Two Essential Files for Every Website
By Ray Zimmerman
There are two essential files that every website should have online for the search engines. Before a search engine’s robot crawls a website, they will look for two important files that will enable them effectively index the website’s content. If you do not have the following files uploaded to your website, stop whatever you’re doing and create them as soon as possible. File #1: robots.txt This particular file tells the search engine's robot if they are allowed to crawl your website. It then tells the robot which directories and files they are not allowed to crawl. The following information is an example of how this file may possibly look. User-agent: * Disallow: /cgi-bin/
Disallow: /private/
Disallow: /statistics/ Disallow: /httpdocs/banners/
Disallow: /httpdocs/css/
Disallow: /httpdocs/menu/
Disallow: /httpdocs/plesk_stat/
Disallow: /httpdocs/test/ Disallow: /product-screenshot.html
Disallow: /email-confirmation.html
Disallow: /exchange-links-confirmation.html The majority of robots will abide by all of the information that is contained in this file, but some will not. Actually, a robots.txt file can simply contain two lines of text. The asterisk in the following examples represents Any Robot. Allows All Robots Complete Access to Your Server User-agent: *
Disallow: Excludes All Robots From Crawling Your Server User-agent: *
Disallow: / Note:This file should be uploaded to the top-level directory of your web server where all of your web pages are being stored. There are examples on the robots.txt website that you should review prior to creating this file and uploading it to your server. A link to this website is provided below for your convenience. File #2: sitemap.xml This particular file provides the robot with a lot of very important information that will help the robot to properly index your web pages which includes the following. File Location <loc>http://crawfish-crawfish.com/index.html</loc> Last Modified <lastmod>2011-07-04</lastmod> Change Frequency <changefreq>monthly</changefreq> Page Priority <priority>1.0</priority> Although all of the information that is provided in the following example is not necessary, I would use every bit of it since Google utilizes all of this information when indexing your web pages. <?xml version="1.0" encoding="UTF-8"?> <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> <url>
<loc>http://crawfish-crawfish.com/index.html</loc>
<lastmod>2011-06-25</lastmod>
<changefreq>weekly</changefreq>
<priority>1.0</priority>
</url> <url>
<loc>http://crawfish-crawfish.com/contact-us.shtml</loc>
<lastmod>2011-05-10</lastmod>
<changefreq>yearly</changefreq>
<priority>0.1</priority>
</url> <url>
<loc>http://crawfish-crawfish.com/how-to-smoke-meat.html</loc>
<lastmod>2011-07-05</lastmod>
<changefreq>monthly</changefreq>
<priority>0.8</priority>
</url> </urlset> Note:This file should be uploaded to the top-level directory of your web server where all of your web pages are being stored. Each time that you update a web page or create a new web page you will need to update this file and upload it to your server at the same time that you upload your web pages. In the long run, your website will definitely benefit by taking the time to properly create these two files. Any improvement to your website means just one thing. More visitors... which equates to more profitability and the inevitability or more taxes to pay.
|
Contributor's Note
To improve your website's PR, check out the information that is provided on my website.
|
|
robots.txt Protocol
| sitemap.xml Protocol
| Free Online Sitemap Generator
 
Crawled by Googlebot
Crawled by Googlebot
PLEASE VISIT THE CONTRIBUTOR'S WEBSITE
 |  | frederick recommended this intel. Jul 7, 2011 |  | | Jhon s Martin liked this intel. Jul 7, 2011 |  | | Rocky appreciated this intel. Jul 7, 2011 |  | | oxytum agreed with this intel. Jul 7, 2011 |  | | Michael Jefferson recommended this intel. Jul 7, 2011 |  | | Samantha Decker liked this intel. Jul 7, 2011 |  | | R Foreman liked this intel. Jul 7, 2011 |  | | Laraine recommended this intel. Jul 7, 2011 |  | | biblefreeorg appreciated this intel. Jul 7, 2011 |  | | nick agreed with this intel. Jul 16, 2011 |
Please login or sign up to rate this intel.
Please login or sign up to add a comment.
So THAT's what the Googlebot looks like. I kinda knew he'd look like that.
 | | nick Jul 16, 2011 05:53 | |
CONTRIBUTOR'S REPLY
Surprised me!
In addition to putting the sitemap.xml on your site it is a good idea to sign up for Google webmaster tools and give them the url to your site map directly. You also get a lot of good information about what Google thinks of your site there. https://www.google.com/webmasters/tools/home?hl=en&pli=1
 | | adac Jul 19, 2011 10:47 | |
CONTRIBUTOR'S REPLY
You are absolutely right. I was trying to keep it as simple as possible. Have a great day!
The copyright for this content entitled "Two Essential Files for Every Website" has been specified by the contributor as:
Creative Commons Attribution-Share Alike 3.0
Details 
This content may be copied, distributed, and modified, as long as a) the original author is acknowledged with a link back to the content page, and b) if the work is modified, the result is distributed with this same license.
If you use this content according to the license specified, you must link to the following URL:
http://crawfish.qondio.com/
|
|
|
This intel was contributed by crawfish
crawfish
|
May, 2012
2008
January, February, March, April, May, June, July, August, September, October, November, December
2009
January, February, March, April, May, June, July, August, September, October, November, December
2010
January, February, March, April, May, June, July, August, September, October, November, December
2011
January, February, March, April, May, June, July, August, September, October, November, December
2012
January, February, March, April, May
|
|
Not a member yet?
Qondio is a powerful network for making it online. If you have a website to
promote, we can help.
Sign up and get in on the action.
|
|
Welcome to Qondio! Discover the awesome power this network can deliver by going to our About page. Or you could skip straight to the Sign Up form.
|
|
