Questions tagged [robots.txt] (173)

0
answer

I made the right robots.txt?

https://alt-infinity.ru/ this is my wordpress site I recently created for him robots.txt to this we can say he was absent. Tell me, right? User-agent: * Disallow: /cgi-bin Disallow: /? Disallow: /wp- Disallow: *?s= Disallow: *&s= Disallow: /search Disallow: /author/ Disallow: */embed Disallow: */page/ Allow:...
tatum.Erns asked April 19th 20 at 12:45
1
answer

Do I need to provide robots.txt all site maps?

I robots.txt put only Sitemap: https://alt-infinity.ru/sitemap.xml and Sitemap: https://alt-infinity.ru/sitemap.xml.gz But I have other files in the site map https://alt-infinity.ru/post-sitemap.xml https://alt-infinity.ru/page-sitemap.xml https://alt-infinity.ru/category-sitemap.xml https://alt-infinity.ru/post_tag-sit...
1
answer

Robots.txt how to hide the index(for Yandex) main site and only allow subdomain?

The main domain of the website in English, there is a ru localization in EN-subdomain, respectively, as to hide from indexing main domain and allow only ru-intranet. the scheme is as follows: for Yandex ru only, for Google domain with all subdomains.
grady asked April 18th 20 at 12:38
0
answer

To hide from indexing a category?

Is there a website. News located on the home page. That would not be duplicates need to hide from indexing the page /news/ How to do it? What would the column is not indexed, the entries are indexed in it.
Krystal37 asked April 16th 20 at 11:17
1
answer

Telegram Python Bot. How to remove aydishnik user from a text file when meeting with a 403 error(bot in an emergency)?

import telebot import time import schedule import threading from telebot import apihelper So I have ajtishniki users enter into a text file ids.txt : @bot.message_handler(commands=['start']) def welcome(message): # Database user_id = message.from_user.id print(user_id) def add_id(filename: str, client_id: str) -> No...
Margie.Lemke asked April 16th 20 at 10:28
1
answer

To close from indexing the page?

Good afternoon. There is a page: /women - main section /women/1 /women/2 etc. They should not be closed from indexing Conclusion 10 cards/teasers on the page. Then there is pagination. In the search hit page: /women?page=13 /women?page=14 /women?page=15 etc. We all have the same header as the /women How to close those...
janae_Donnelly asked April 7th 20 at 11:18
1
answer

How to set robots.txt for merchants?

Good afternoon. Help solve the problem that part of the goods doesn't want to seem at merchants (google ads) because of an error in robots.txt (he robots.txt). Writes that robots.txt blocks the scanning of the page. Screenshots from merchants and tool "check url": Thanks in advance!
Magdalena65 asked April 7th 20 at 09:43
1
answer

Discovered not indexed Status: Deleted robots.txt?

I do not understand what it is SOS ! Google Search Console eliminated almost all pages from indexingDiscovered not indexed Status: EliminatedInformation Status Excluded Type Discovered not indexed Check Missing What kind of nonsense guglani writes ? the robot had already written so User-agent: * Allow: / what is missing ...
Cory.Skiles asked April 4th 20 at 12:20
1
answer

How to prohibit to index certain text using robots.txt?

How to prohibit to index certain text or part of text through robots.txt? Interested prevent indexing of the text only through robots.txt!
kelsie asked April 4th 20 at 12:17
1
answer

How to close from indexing junk files?

There is a category Uncategorized with about 1000 garbage news that need to be closed from indexation. But adding Disallow: /uncategorized/ the robots will do nothing, as all news links have the form http://сайт/load/vokzal_2015/17-1-0-7193, http://сайт/load/monstervill_2015/6-1-0-6868 etc. Maybe there's another way to disa...
nelson asked April 4th 20 at 01:56