Fix: Set common robots.txt to all virtual servers in Virtualmin

Linux OS Topics
Post Reply
User avatar
Saman
Lieutenant Colonel
Lieutenant Colonel
Posts: 828
Joined: Fri Jul 31, 2009 10:32 pm
Location: Mount Lavinia

Fix: Set common robots.txt to all virtual servers in Virtualmin

Post by Saman » Thu Nov 22, 2018 8:40 am

Go to Server Templates and see whether the skeleton folder it located at /etc/skel. Create a subfolder within it as public_html and store your robots.txt file. I will attach a sample robots.txt file. When you do that, robots.txt will be copied to all your newly created servers.

Sample robots.txt

Code: Select all

User-agent: Applebot
Allow: /

User-agent: baiduspider
Allow: /

User-agent: Bingbot
Allow: /

User-agent: Googlebot
Allow: /

User-agent: ia_archiver
Allow: /

User-agent: msnbot
Allow: /

User-agent: Naverbot
Allow: /

User-agent: seznambot
Allow: /

User-agent: Slurp
Allow: /

User-agent: teoma
Allow: /

User-agent: Twitterbot
Allow: /

User-agent: Yandex
Allow: /

User-agent: Yeti
Allow: /

User-agent: *
Disallow: /


Post Reply

Return to “Linux”