################################# CRAWLER WARNING ######################################## # # - The terms of service and the robots.txt file disallows crawling of this site, # please see https://omim.org/help/agreement for more information. # # - A number of data files are available for download at https://omim.org/downloads. # # - We have an API which you can learn about at https://omim.org/help/api and register # for at https://omim.org/api, this provides access to the data in JSON & XML formats. # # - You should feel free to contact us at https://omim.org/contact to figure out the best # approach to getting the data you need for your work. # # - WE WILL AUTOMATICALLY BLOCK YOUR IP ADDRESS IF YOU CRAWL THIS SITE. # # - WE WILL ALSO AUTOMATICALLY BLOCK SUB-DOMAINS AND ADDRESS RANGES IMPLICATED IN # DISTRIBUTED CRAWLS OF THIS SITE. # ################################# CRAWLER WARNING ######################################## # Specification for robots.txt # https://developers.google.com/search/reference/robots_txt # https://www.robotstxt.org # Crawl delay, every two seconds Crawl-delay: 2 # Google specific - disallow everything except for specific paths # https://www.google.com/bot.html User-agent: Googlebot Allow: /$ Disallow: /entry/$ Allow: /entry/ Disallow: /clinicalSynopsis/$ Allow: /clinicalSynopsis/ Allow: /about Allow: /help/ Allow: /downloads Allow: /api Disallow: / # Bing specific - disallow everything except for specific paths # https://www.bing.com/bingbot.htm User-agent: Bingbot Allow: /$ Disallow: /entry/$ Allow: /entry/ Disallow: /clinicalSynopsis/$ Allow: /clinicalSynopsis/ Allow: /about Allow: /help/ Allow: /downloads Allow: /api Disallow: / # DuckDuckGo specific - disallow everything except for specific paths # https://duckduckgo.com/duckduckbot User-agent: DuckDuckBot Allow: /$ Disallow: /entry/$ Allow: /entry/ Disallow: /clinicalSynopsis/$ Allow: /clinicalSynopsis/ Allow: /about Allow: /help/ Allow: /downloads Allow: /api Disallow: / # Apple specific - disallow everything except for home page # https://support.apple.com/en-us/HT204683 # If not specifically mentioned Applebot will follow Googlebot instructions User-agent: Applebot Allow: /$ Disallow: / # Global - disallow everything except for home page User-agent: * Allow: /$ Disallow: / ##########################################################################################