下面是典型的一个magento站点的robots.txt具体设置, 有需要的同学可以根据实际情况做适当的调整…
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
| # Website SitemapSitemap: http://www.mydomain.com/sitemap.xml# Crawlers SetupUser-agent: *Crawl-delay: 10# Allowable IndexAllow: /*?p=Allow: /index.php/blog/Allow: /catalog/seo_sitemap/category/Allow:/catalogsearch/result/# DirectoriesDisallow: /404/Disallow: /app/Disallow: /cgi-bin/Disallow: /downloader/Disallow: /includes/Disallow: /js/Disallow: /lib/Disallow: /magento/Disallow: /media/Disallow: /pkginfo/Disallow: /report/Disallow: /skin/Disallow: /stats/Disallow: /var/# Paths (clean URLs)Disallow: /index.php/Disallow: /catalog/product_compare/Disallow: /catalog/category/view/Disallow: /catalog/product/view/Disallow: /catalogsearch/Disallow: /checkout/Disallow: /control/Disallow: /contacts/Disallow: /customer/Disallow: /customize/Disallow: /newsletter/Disallow: /poll/Disallow: /review/Disallow: /sendfriend/Disallow: /tag/Disallow: /wishlist/# FilesDisallow: /cron.phpDisallow: /cron.shDisallow: /error_logDisallow: /install.phpDisallow: /LICENSE.htmlDisallow: /LICENSE.txtDisallow: /LICENSE_AFL.txtDisallow: /STATUS.txt# Paths (no clean URLs)Disallow: /*.js$Disallow: /*.css$Disallow: /*.php$Disallow: /*?p=*&Disallow: /*?SID= |
Your blog article is very intersting and fanstic,at the same time the blog theme is unique and perfect,great job.To your success.
ReplyDelete-----------------
WOW Gold Buy WOW Gold Cheap WOW Gold
so useful for me !thanks,buy diablo 3 gold
ReplyDeleteIt doesn't matter what you like when it comes to actions, you will appreciate enough time you invest enjoying. Use the this article's tips to make use of your game playing encounter. Movie game playing is exciting and fun.
ReplyDeleteBuy WOW Gold RS Gold Buy Diablo 3 Gold