Multisite Robots.txt Manager

Multisite Robots.txt Manager

Wordpress plugin

Install on Wordpress

App Details

A Robots.txt Manager Plugin for WordPress Multisite Networks. Quickly and easily manage all robots.txt files on a WordPress Multisite Website Network.

!!! Network Enabled Multisite Installs Only !!!

This Plugin Was Created For Multisite Networks > Network Activations Only!

For Support & Bugs

Please contact us directly if you need assistance or have found a bug. The WordPress Support forum does not notify us of new support tickets, no idea why, so contact us directly.

View, Report Bugs, Contribute!

Visit this Plugin on Github! Clone/fork yourself a copy, report a bug or submit a ticket & pull request!

Features:

  • Network wide robots.txt file, shared across all sites.
  • Append extra robots.txt file data per-website Admin.
  • Create unique robots.txt files for each network Website.
  • Manage all Websites from Network Administration Area.
  • Manage a single Website through the Website Settings Admins.
  • Auto-generated Sitemap URL’s, Upload Path & Theme Path.
  • Mass update the all Websites on the Network in a single click.
  • Quickly publish preset robots.txt files across the Network.
  • Scan and clean old robots.txt file plugin data to clear out conflicts.

Quick Info:

  • The default “Network Wide” robots.txt file is NOT a live robots.txt file.
  • If you deactivate the plugin, no options are removed but the plugins robots.txt file(s) are no longer displayed.
  • If you delete this plugin, all options and settings will be removed from the database, for all Websites.

  • Get The Plugin: Download > Install > Network Activate

  • Network Access: Network Admin > Settings Tab > MS Robots.txt Link
  • Website Access: Website Admin > Settings Tab > MS Robots.txt Link

Make It Work:

  • Network Admin: Select either either with “Publish the network robots.txt file to all network websites” or “Publish the network robots.txt file to network websites you are a member of” then click the “update settings” button to publish the robots.txt files.
  • Website Admin: Modify the appended robots.txt file (or create a website unique robots.txt file) then click the “update website rules” button to publish your changes.

Arbitrary section

Understanding the Default Settings

When you first enter the plugin admin via the Network Admin, the displayed robots.txt file is the default “network only” copy.

The Network Append Marker

The marker {APPEND_WEBSITE_ROBOTSTXT} within the Network Robots.txt File is replaced by Website unique robots.txt file append rules. Use the marker in your customized Network robots.txt files to automatically append the Website robots.txt file rules when the Network is updated.

Robots.txt Files within Directories

  • This plugin WILL render robots.txt files within directories – however,

  • Search Engine Spiders only read robots.txt files found within the root directory of a Website. Spiders do not read robots.txt files within directories, such as: domain.com/PATH-or-FOLDER/robots.txt is NOT a valid location.

  • From Google: “The robots.txt file must be in the top-level directory of the host…..Crawlers will not check for robots.txt files in sub-directories.”

Testing Robots.txt Files

  • Use Google’s Webmaster Tools to Validate your Robots.txt Files…. with Google at least.:
  • Log into your Google Account and access the Log into your Webmaster Tools feature. Select a Website or Add a Website….

  • On the Webmaster Tools Home page, click the site you want.

  • Under Health, click Blocked URLs.
  • If it is not already selected, click the Test robots.txt tab.
  • Copy the content of your robots.txt file, and paste it into the first box.
  • In the URLs box, list the site to test against.
  • In the User-agents list, select the user-agents you want.
  • https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt

New Website Added to Network

  • If all Websites use the saved Network default robots.txt file, click the “update setting” button to copy the default robots.txt file over to any new Websites you have.
  • Per Site: Then click the “update website rules” button to copy the default robots.txt file to this Website.

Disabling

  • Disable a Website: Within the website plugin admin, for each unique website, scroll down and click the “Disable the saved robots.txt file…” checkbox then click the submit button. This will disable the robots.txt file the Website only, making the WordPress default robots.txt file display.
  • Disable Across Network: Within the network plugin admin, scroll down and click the “Disable saved robots.txt files…” checkbox, then click the submit button.

= Presets

  • This feature allows you to quickly duplicate premade robots.txt files and a sitemap structure url to the default network robots.txt file. This action does not publish the robots.txt files to the network.

Recommended Sitemap Plugins

  • Google XML Sitemaps: https://wordpress.org/extend/plugins/google-sitemap-generator/
  • Better WordPress Google XML Sitemaps: https://wordpress.org/extend/plugins/bwp-google-xml-sitemaps/
  • Search For Others: https://wordpress.org/extend/plugins/search.php?q=multisite+sitemap

For “real” Multisite HOST Networks, use the WordPress plugin: BWP Google XML Sitemaps – This plugin will list each Websites Sitemap URL’s in the Root Network Website’s robots.txt file.

Pricing

Starting from $0 per month.

Check Out the Countdown Bar Widget

By Common Ninja

Countdown BarTry For Free!

App Info

Rating

Reviewers

14 reviews

Tags

robot
robots
robots.txt
robotstxt
spiders

Developed By

tribalNerd

Quick & Easy

Find the Best Wordpress plugins for you

Common Ninja has a large selection of powerful Wordpress plugins that are easy to use, fully customizable, mobile-friendly and rich with features — so be sure to check them out!

Testimonial

Testimonial plugins for Wordpress

Contact Form

Contact Form plugins for Wordpress

Maps

Maps plugins for Wordpress

Translation

Translation plugins for Wordpress

Chat

Chat plugins for Wordpress

Slider

Slider plugins for Wordpress

Reviews

Reviews plugins for Wordpress

Contact

Contact plugins for Wordpress

Galleries

Galleries plugins for Wordpress

SEO

SEO plugins for Wordpress

Forms

Forms plugins for Wordpress

Comments

Comments plugins for Wordpress

Backup

Backup plugins for Wordpress

Privacy

Privacy plugins for Wordpress

Optimize

Optimize plugins for Wordpress

Tabs

Tabs plugins for Wordpress

Social Sharing

Social Sharing plugins for Wordpress

Events Calendar

Events Calendar plugins for Wordpress

Comments

Comments plugins for Wordpress

Social Feeds

Social Feeds plugins for Wordpress

Social Sharing

Social Sharing plugins for Wordpress

Portfolio

Portfolio plugins for Wordpress

Video Player

Video Player plugins for Wordpress

popup

popup plugins for Wordpress

SiteMap

SiteMap plugins for Wordpress

Payment

Payment plugins for Wordpress

Coming Soon

Coming Soon plugins for Wordpress

Inventory

Inventory plugins for Wordpress

Testimonials

Testimonials plugins for Wordpress

Portfolio

Portfolio plugins for Wordpress

Membership

Membership plugins for Wordpress

Forms

Forms plugins for Wordpress

Analytics

Analytics plugins for Wordpress

Events Calendar

Events Calendar plugins for Wordpress

Sliders

Sliders plugins for Wordpress

Analytics

Analytics plugins for Wordpress

Reviews

Reviews plugins for Wordpress

Security

Security plugins for Wordpress

Ads

Ads plugins for Wordpress

Music Player

Music Player plugins for Wordpress

Countdown

Countdown plugins for Wordpress

Email Marketing

Email Marketing plugins for Wordpress

Membership

Membership plugins for Wordpress

Ecommerce

Ecommerce plugins for Wordpress

Customer Support

Customer Support plugins for Wordpress

Video Player

Video Player plugins for Wordpress

Tabs

Tabs plugins for Wordpress

Social Feeds

Social Feeds plugins for Wordpress

Common Ninja Apps

Some of the best Common Ninja plugins for Wordpress

Browse our extensive collection of compatible plugins, and easily embed them on any website, blog, online store, e-commerce platform, or site builder.

Countdown Bar for Wordpress logo

Countdown Bar

Add a countdown bar to your site to create urgency, highlight limited time offers, and drive faster engagement and higher conversions.

Pomodoro Timer for Wordpress logo

Pomodoro Timer

Add a Pomodoro timer to your site so users can structure focus and break cycles, improving time management and productivity.

LinkedIn Feed Carousel for Wordpress logo

LinkedIn Feed Carousel

Show LinkedIn posts with a LinkedIn feed carousel that auto updates, presents content in a smooth layout, and keeps visitors engaged.

WhatsApp Chat for Wordpress logo

WhatsApp Chat

Add WhatsApp Chat to your site to communicate with visitors, deliver instant support, and create a smoother, more trustworthy user experience.

Testimonials Slider for Wordpress logo

Testimonials Slider

Add a testimonials slider to your site to showcase real customer feedback, build credibility, and increase trust that leads to higher conversions.

Audio Player for Wordpress logo

Audio Player

Add a no-code audio player to your site and boost engagement with music, podcasts, and voice content effortlessly.

Music Player for Wordpress logo

Music Player

Add a no-code music player to your site to stream audio, share playlists, and support multiple formats with easy setup.

Info List for Wordpress logo

Info List

An Info List presents key details in a clear, structured format to improve user experience and support conversions.

TikTok Carousel for Wordpress logo

TikTok Carousel

Show TikTok videos with a TikTok carousel that arranges clips in a smooth, customizable layout to boost engagement and keep visitors watching.

Process for Wordpress logo

Process

Use a process widget to present tasks in a clear visual sequence, showing each step with images and text to improve understanding and user engagement.

Popup Builder for Wordpress logo

Popup Builder

Add customizable popups to your site to capture attention, deliver messages, and drive actions like signups or conversions.

My Reviews on Amazon for Wordpress logo

My Reviews on Amazon

Show your Amazon product reviews with My Reviews on Amazon to build trust, boost credibility, and help visitors make confident purchase decisions.

More plugins

plugins You Might Like

Discover Apps By Platform

Discover the best apps for your website

WordPress
Wix
Shopify
Weebly
Webflow
Joomla
PrestaShop
Shift4Shop
WebsiteX5
MODX
Opencart
NopCommerce

Common Ninja Search Engine

The Common Ninja Search Engine platform helps website builders find the best site widgets, apps, plugins, tools, add-ons, and extensions! Compatible with all major website building platforms - big or small - and updated regularly, our Search Engine tool provides you with the business tools your site needs!

Multiple platforms