casraf/snpr
1
0
Fork 0
mirror of https://github.com/chenasraf/snpr.git synced 2026-05-18 01:39:01 +00:00
Code Issues Packages Projects Releases Wiki Activity

Update robots.txt

Browse Source
This commit is contained in:
alphaCTzo7G
2017-07-17 05:54:26 -07:00
committed by GitHub
parent 7c4d077760
commit f93d1a84f8
1 changed files with 11 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

11
public/robots.txt
View File

@@ -3,4 +3,15 @@
# To ban all spiders from the entire site uncomment the next two lines:
# User-Agent: *
# Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: AhrefsBot
Disallow: /
User-agent: *
Disallow: /snps
Disallow: /users
Crawl-delay:10
Disallow: /user_snps