### *********************

### NGINX BAD BOT BLOCKER

### *********************

### Copyright MMitchell Krog <mitchellkrog@gmail.com>

### https://github.com/mitchellkrogza

### VERSION INFORMATION #

###################################################

### Version: V2.2020.02.21

### Updated: Fri Feb 21 16:55:07 SAST 2020

### Bad Referrer Count: 6828

### Bad Bot Count: 562

###################################################

### VERSION INFORMATION ##

##############################################################################

# _ __ _ #

# / |/ /__ _(_)__ __ __ #

# / / _ `/ / _ \\ \ / #

# /_/|_/\_, /_/_//_/_\_\ #

# __/___/ __ ___ __ ___ __ __ #

# / _ )___ ____/ / / _ )___ / /_ / _ )/ /__ ____/ /_____ ____ #

# / _ / _ `/ _ / / _ / _ \/ __/ / _ / / _ \/ __/ '_/ -_) __/ #

# /____/\_,_/\_,_/ /____/\___/\__/ /____/_/\___/\__/_/\_\\__/_/ #

# #

##############################################################################

### Project URL: https://github.com/mariusv/nginx-badbot-blocker

### This file implements a checklist / blacklist for good user agents, bad user agents and

### bad referrers. It also has whitelisting for your own IP's and known good IP Ranges

### and also has rate limiting functionality for bad bots who you only want to rate limit

### and not actually block out entirely. It is powerful and also flexible.

### Version 1 Created By: https://github.com/mariusv/

### Version 2 Created By: https://github.com/mitchellkrogza/

### Version 2 has been rewritten from the ground up by mitchellkrogza who first started off using

### the original Nginx Bot Blocker created by MariusV. Due to formatting issues, complicated and uneccessary

### Regex patterns and inabilities to track commits properly, the entire layout was re-designed

### bringing to all existing users a new super powerful version of the bot blocker which is much easier to maintain and also to update.

### Tested on: nginx/1.10.3 (Ubuntu 16.04)

### This list was developed on and is in use on a live Nginx server running some very busy web sites.

### It was built from scratch using real data from daily logs and is updated almost daily.

### It has been extensively tested for false positives and all additions to the lists of bad user agents,

### spam referers, rogue IP address, scanners, scrapers and domain hijacking sites are extensively checked

### before they are added. It is monitored extensively for any false positives.

### *********

### Features:

### *********

### Clear formatting for Ease of Maintenance.

### Alphabetically ordered lists for Ease of Maintenance.

### Extensive Commenting for Ease of Reference.

### Extensive bad_bot list

### Extensive bad_referer list (please excuse the nasty words and domains)

### Simple regex patterns versus complicated messy regex patterns.

### Checks regardless of http / https urls or the lack of any protocol sent.

### IP range blocking / whitelisting.

### Rate Limiting Functions.

### *** PLEASE READ ALL INLINE NOTES ON TESTING !!!!

### We have this set up as an include in nginx.conf as

### Include /etc/nginx/conf.d/blacklist.conf

### This is loaded and available for any vhost to use in its config

### Each vhost then just needs the include file mentioned below for it to take effect.

### In Most cases your nginx.conf should already have an include statement as follows

### Include /etc/nginx/conf.d/*

### If that is the case then you can ignore the above include statement as Nginx will

### load anything in the conf.d folder and make it available to all sites.

###################################################################

# #

# _____ ____ __ _ #

# / ___/__ ___ / _(_)__ ___ _________ _/ /_(_)__ ___ #

# / /__/ _ \/ _ \/ _/ / _ `/ // / __/ _ `/ __/ / _ \/ _ \ #

# \___/\___/_//_/_//_/\_, /\_,_/_/ \_,_/\__/_/\___/_//_/ #

# /___/ #

# #

###################################################################

### All you then need to do is use the include statements below in the server {} block

### of a vhost file for it to take effect.

# server {

# #Config stuff here

# include /etc/nginx/bots.d/blockbots.conf;

# include /etc/nginx/bots.d/ddos.conf;

# #Other config stuff here

# }

### Make sure to read the updated configuration Instructions regarding the whitelisting include files otherwise you will get EMERG: errors

### Need we say, please don't just copy and paste this without reviewing what bots and

### referers are being blocked, you may want to exclude certain of them

### Also make SURE to whitelist your own IP's in the geo $bad_referer section.

### Know why you are using this or why you want to use it before you do, the implications

### are quite severe.

### *** PLEASE READ INLINE NOTES ON TESTING !!!!

### Note that:

### 0 = allowed - no limits

### 1 = allowed or rate limited less restrictive

### 2 = rate limited more

### 3 = block completely

### NEED we say always do a "sudo nginx -t" to test the config is okay after adding these

### and if so then "sudo service nginx reload" for it to take effect.

### *** MAKE SURE TO ADD to your nginx.conf ***

### server_names_hash_bucket_size 64;

### server_names_hash_max_size 4096;

### limit_req_zone $binary_remote_addr zone=flood:50m rate=90r/s;

### limit_conn_zone $binary_remote_addr zone=addr:50m;

### to allow it to load this large set of domains into memory and to set the rate limiting zones for the DDOS filter.

### ADDING YOUR OWN BAD REFERERS

### Fork your own local copy and then

### Send a Pull Request by following the instructions in the Pull_Requests folder.

# *********************************

# FIRST BLOCK BY USER-AGENT STRINGS

# *********************************

# ***************

# PLEASE TEST !!!

# ***************

# ALWAYS test any User-Agent Strings you add here to make sure you have it right

# Use a Chrome Extension called "User-Agent Switcher for Chrome" where you can create your

# own custom lists of User-Agents and test them easily against your rules below.

# You can also use Curl to test user-agents as per example below

# curl -I http://www.yourdomain.com -A "GoogleBot" << 200 OK

# curl -I http://www.yourdomain.com -A "80legs" <<< 444 Dropped Connection

# Here we also allow specific User Agents to come through that we want to allow

# PLEASE NOTE: In all lists below I use Nginx case-insensitive matching ~*

# This means regardless of how you type the word, upper or lowercase or mixed it will

# be detected by Nginx Regex. Some Names are Capitalised simply for Ease of Reading.

# Especially important for both Googlebot and googlebot to be allowed through no?

# Now we map all good and bad user agents to a variable called $bad_bot

map $http_user_agent $bad_bot {

default 0;

# ***********************************************

# Allow Good User-Agent Strings We Know and Trust

# ***********************************************

# START GOOD BOTS ### DO NOT EDIT THIS LINE AT ALL ###

"~*\bAdsBot-Google\b" 0;

"~*\bDoCoMo\b" 0;

"~*\bFeedfetcher-Google\b" 0;

"~*\bGoogle-HTTP-Java-Client\b" 0;

"~*\bGooglebot\b" 0;

"~*\bGooglebot-Image\b" 0;

"~*\bGooglebot-Mobile\b" 0;

"~*\bGooglebot-News\b" 0;

"~*\bGooglebot-Video\b" 0;

"~*\bGooglebot/Test\b" 0;

"~*\bGravityscan\b" 0;

"~*\bJakarta\ Commons\b" 0;

"~*\bKraken/0.1\b" 0;

"~*\bLinkedInBot\b" 0;

"~*\bMediapartners-Google\b" 0;

"~*\bSAMSUNG\b" 0;

"~*\bSlackbot\b" 0;

"~*\bSlackbot-LinkExpanding\b" 0;

"~*\bTwitterBot\b" 0;

"~*\bWordpress\b" 0;

"~*\badidxbot\b" 0;

"~*\baolbuild\b" 0;

"~*\bbingbot\b" 0;

"~*\bbingpreview\b" 0;

"~*\bduckduckgo\b" 0;

"~*\bfacebookexternalhit\b" 0;

"~*\bgsa-crawler\b" 0;

"~*\bmsnbot\b" 0;

"~*\bmsnbot-media\b" 0;

"~*\bslurp\b" 0;

"~*\bteoma\b" 0;

"~*\byahoo\b" 0;

# END GOOD BOTS ### DO NOT EDIT THIS LINE AT ALL ###

# **************************************************

# User-Agent Strings Allowed Throug but Rate Limited

# **************************************************

# Some people block libwww-perl, it us widely used in many valid (non rogue) agents

# I allow libwww-perl as I use it for monitoring systems with Munin but it is rate limited

# START ALLOWED BOTS ### DO NOT EDIT THIS LINE AT ALL ###

"~*\bLynx\b" 1;

"~*\bPresto\b" 1;

"~*\bWget/1.15\b" 1;

"~*\bjetmon\b" 1;

"~*\blibwww-perl\b" 1;

"~*\bmunin\b" 1;

# END ALLOWED BOTS ### DO NOT EDIT THIS LINE AT ALL ###

# **************************************************************

# Rate Limited User-Agents who get a bit aggressive on bandwidth

# **************************************************************

# START LIMITED BOTS ### DO NOT EDIT THIS LINE AT ALL ###

"~*\bAlexa\b" 2;

"~*\bBUbiNG\b" 2;

"~*\bBaidu\b" 2;

"~*\bFlipboardProxy\b" 2;

"~*\bMSIE\ 7.0\b" 2;

"~*\bProximic\b" 2;

"~*\bR6_CommentReader\b" 2;

"~*\bR6_FeedFetcher\b" 2;

"~*\bRED/1\b" 2;

"~*\bRPT-HTTPClient\b" 2;

"~*\bSpaidu\b" 2;

"~*\bUptimeRobot/2.0\b" 2;

"~*\bYandexBot\b" 2;

"~*\bYandexImages\b" 2;

"~*\barchive.org\b" 2;

"~*\bia_archiver\b" 2;

"~*\bsfFeedReader/0.9\b" 2;

# END LIMITED BOTS ### DO NOT EDIT THIS LINE AT ALL ###

# *********************************************

# Bad User-Agent Strings That We Block Outright

# *********************************************

# START BAD BOTS ### DO NOT EDIT THIS LINE AT ALL ###

"~*\b360Spider\b" 3;

"~*\b404checker\b" 3;

"~*\b404enemy\b" 3;

"~*\b80legs\b" 3;

"~*\bADmantX\b" 3;

"~*\bAIBOT\b" 3;

"~*\bASPSeek\b" 3;

"~*\bAbonti\b" 3;

"~*\bAboundex\b" 3;

"~*\bAboundexbot\b" 3;

"~*\bAcunetix\b" 3;

"~*\bAfD-Verbotsverfahren\b" 3;

"~*\bAhrefsBot\b" 3;

"~*\bAiHitBot\b" 3;

"~*\bAipbot\b" 3;

"~*\bAlexibot\b" 3;

"~*\bAllSubmitter\b" 3;

"~*\bAlligator\b" 3;

"~*\bAlphaBot\b" 3;

"~*\bAnarchie\b" 3;

"~*\bApexoo\b" 3;

"~*\bAsterias\b" 3;

"~*\bAttach\b" 3;

"~*\bAwarioRssBot\b" 3;

"~*\bAwarioSmartBot\b" 3;

"~*\bBBBike\b" 3;

"~*\bBDCbot\b" 3;

"~*\bBDFetch\b" 3;

"~*\bBLEXBot\b" 3;

"~*\bBackDoorBot\b" 3;

"~*\bBackStreet\b" 3;

"~*\bBackWeb\b" 3;

"~*\bBacklink-Ceck\b" 3;

"~*\bBacklinkCrawler\b" 3;

"~*\bBadass\b" 3;

"~*\bBandit\b" 3;

"~*\bBarkrowler\b" 3;

"~*\bBatchFTP\b" 3;

"~*\bBattleztar\ Bazinga\b" 3;

"~*\bBetaBot\b" 3;

"~*\bBigfoot\b" 3;

"~*\bBitacle\b" 3;

"~*\bBlackWidow\b" 3;

"~*\bBlack\ Hole\b" 3;

"~*\bBlackboard\b" 3;

"~*\bBlow\b" 3;

"~*\bBlowFish\b" 3;

"~*\bBoardreader\b" 3;

"~*\bBolt\b" 3;

"~*\bBotALot\b" 3;

"~*\bBrandprotect\b" 3;

"~*\bBrandwatch\b" 3;

"~*\bBuddy\b" 3;

"~*\bBuiltBotTough\b" 3;

"~*\bBuiltWith\b" 3;

"~*\bBullseye\b" 3;

"~*\bBunnySlippers\b" 3;

"~*\bBuzzSumo\b" 3;

"~*\bCATExplorador\b" 3;

"~*\bCCBot\b" 3;

"~*\bCSHttp\b" 3;

"~*\bCalculon\b" 3;

"~*\bCazoodleBot\b" 3;

"~*\bCegbfeieh\b" 3;

"~*\bCheTeam\b" 3;

"~*\bCheeseBot\b" 3;

"~*\bCherryPicker\b" 3;

"~*\bChinaClaw\b" 3;

"~*\bChlooe\b" 3;

"~*\bClaritybot\b" 3;

"~*\bCliqzbot\b" 3;

"~*\bCloud\ mapping\b" 3;

"~*\bCogentbot\b" 3;

"~*\bCollector\b" 3;

"~*\bCopier\b" 3;

"~*\bCopyRightCheck\b" 3;

"~*\bCopyscape\b" 3;

"~*\bCosmos\b" 3;

"~*\bCraftbot\b" 3;

"~*\bCrazyWebCrawler\b" 3;

"~*\bCrescent\b" 3;

"~*\bCrunchBot\b" 3;

"~*\bCurious\b" 3;

"~*\bCusto\b" 3;

"~*\bDBLBot\b" 3;

"~*\bDIIbot\b" 3;

"~*\bDSearch\b" 3;

"~*\bDTS\ Agent\b" 3;

"~*\bDataCha0s\b" 3;

"~*\bDatabaseDriverMysqli\b" 3;

"~*\bDemon\b" 3;

"~*\bDeusu\b" 3;

"~*\bDevil\b" 3;

"~*\bDigincore\b" 3;

"~*\bDigitalPebble\b" 3;

"~*\bDirbuster\b" 3;

"~*\bDisco\b" 3;

"~*\bDiscobot\b" 3;

"~*\bDiscoverybot\b" 3;

"~*\bDispatch\b" 3;

"~*\bDittoSpyder\b" 3;

"~*\bDnyzBot\b" 3;

"~*\bDomainAppender\b" 3;

"~*\bDomainCrawler\b" 3;

"~*\bDomainSigmaCrawler\b" 3;

"~*\bDomainStatsBot\b" 3;

"~*\bDotbot\b" 3;

"~*\bDownload\ Wonder\b" 3;

"~*\bDragonfly\b" 3;

"~*\bDrip\b" 3;

"~*\bECCP/1.0\b" 3;

"~*\bEMail\ Siphon\b" 3;

"~*\bEMail\ Wolf\b" 3;

"~*\bEasyDL\b" 3;

"~*\bEbingbong\b" 3;

"~*\bEcxi\b" 3;

"~*\bEirGrabber\b" 3;

"~*\bEroCrawler\b" 3;

"~*\bEvil\b" 3;

"~*\bExabot\b" 3;

"~*\bExpress\ WebPictures\b" 3;

"~*\bExtLinksBot\b" 3;

"~*\bExtractor\b" 3;

"~*\bExtractorPro\b" 3;

"~*\bExtreme\ Picture\ Finder\b" 3;

"~*\bEyeNetIE\b" 3;

"~*\bEzooms\b" 3;

"~*\bFDM\b" 3;

"~*\bFHscan\b" 3;

"~*\bFemtosearchBot\b" 3;

"~*\bFimap\b" 3;

"~*\bFirefox/7.0\b" 3;

"~*\bFlashGet\b" 3;

"~*\bFlunky\b" 3;

"~*\bFoobot\b" 3;

"~*\bFreeuploader\b" 3;

"~*\bFrontPage\b" 3;

"~*\bFyberSpider\b" 3;

"~*\bFyrebot\b" 3;

"~*\bG-i-g-a-b-o-t\b" 3;

"~*\bGT::WWW\b" 3;

"~*\bGalaxyBot\b" 3;

"~*\bGenieo\b" 3;

"~*\bGermCrawler\b" 3;

"~*\bGetRight\b" 3;

"~*\bGetWeb\b" 3;

"~*\bGetintent\b" 3;

"~*\bGigablast\b" 3;

"~*\bGigabot\b" 3;

"~*\bGo!Zilla\b" 3;

"~*\bGo-Ahead-Got-It\b" 3;

"~*\bGoZilla\b" 3;

"~*\bGotit\b" 3;

"~*\bGrabNet\b" 3;

"~*\bGrabber\b" 3;

"~*\bGrafula\b" 3;

"~*\bGrapeFX\b" 3;

"~*\bGrapeshotCrawler\b" 3;

"~*\bGridBot\b" 3;

"~*\bHEADMasterSEO\b" 3;

"~*\bHMView\b" 3;

"~*\bHTMLparser\b" 3;

"~*\bHTTP::Lite\b" 3;

"~*\bHTTrack\b" 3;

"~*\bHaansoft\b" 3;

"~*\bHaosouSpider\b" 3;

"~*\bHarvest\b" 3;

"~*\bHavij\b" 3;

"~*\bHeritrix\b" 3;

"~*\bHloader\b" 3;

"~*\bHumanlinks\b" 3;

"~*\bHybridBot\b" 3;

"~*\bIDBot\b" 3;

"~*\bIRLbot\b" 3;

"~*\bIblog\b" 3;

"~*\bId-search\b" 3;

"~*\bIlseBot\b" 3;

"~*\bImage\ Fetch\b" 3;

"~*\bImage\ Sucker\b" 3;

"~*\bIndeedBot\b" 3;

"~*\bIndy\ Library\b" 3;

"~*\bInfoNaviRobot\b" 3;

"~*\bInfoTekies\b" 3;

"~*\bIntelliseek\b" 3;

"~*\bInterGET\b" 3;

"~*\bInternetSeer\b" 3;

"~*\bInternet\ Ninja\b" 3;

"~*\bIria\b" 3;

"~*\bIskanie\b" 3;

"~*\bIstellaBot\b" 3;

"~*\bJOC\ Web\ Spider\b" 3;

"~*\bJamesBOT\b" 3;

"~*\bJbrofuzz\b" 3;

"~*\bJennyBot\b" 3;

"~*\bJetCar\b" 3;

"~*\bJetty\b" 3;

"~*\bJikeSpider\b" 3;

"~*\bJoomla\b" 3;

"~*\bJorgee\b" 3;

"~*\bJustView\b" 3;

"~*\bJyxobot\b" 3;

"~*\bKenjin\ Spider\b" 3;

"~*\bKeyword\ Density\b" 3;

"~*\bKozmosbot\b" 3;

"~*\bLNSpiderguy\b" 3;

"~*\bLWP::Simple\b" 3;

"~*\bLanshanbot\b" 3;

"~*\bLarbin\b" 3;

"~*\bLeechFTP\b" 3;

"~*\bLeechGet\b" 3;

"~*\bLexiBot\b" 3;

"~*\bLftp\b" 3;

"~*\bLibWeb\b" 3;

"~*\bLibwhisker\b" 3;

"~*\bLightspeedsystems\b" 3;

"~*\bLikse\b" 3;

"~*\bLinkScan\b" 3;

"~*\bLinkWalker\b" 3;

"~*\bLinkdexbot\b" 3;

"~*\bLinkextractorPro\b" 3;

"~*\bLinkpadBot\b" 3;

"~*\bLinksManager\b" 3;

"~*\bLinqiaMetadataDownloaderBot\b" 3;

"~*\bLinqiaRSSBot\b" 3;

"~*\bLinqiaScrapeBot\b" 3;

"~*\bLipperhey\b" 3;

"~*\bLipperhey\ Spider\b" 3;

"~*\bLitemage_walker\b" 3;

"~*\bLmspider\b" 3;

"~*\bLtx71\b" 3;

"~*\bMFC_Tear_Sample\b" 3;

"~*\bMIDown\ tool\b" 3;

"~*\bMIIxpc\b" 3;

"~*\bMJ12bot\b" 3;

"~*\bMSFrontPage\b" 3;

"~*\bMSIECrawler\b" 3;

"~*\bMag-Net\b" 3;

"~*\bMagnet\b" 3;

"~*\bMail.RU_Bot\b" 3;

"~*\bMajestic-SEO\b" 3;

"~*\bMajestic12\b" 3;

"~*\bMajestic\ SEO\b" 3;

"~*\bMarkMonitor\b" 3;

"~*\bMarkWatch\b" 3;

"~*\bMass\ Downloader\b" 3;

"~*\bMasscan\b" 3;

"~*\bMata\ Hari\b" 3;

"~*\bMauiBot\b" 3;

"~*\bMeanPath\ Bot\b" 3;

"~*\bMeanpathbot\b" 3;

"~*\bMediatoolkitbot\b" 3;

"~*\bMegaIndex.ru\b" 3;

"~*\bMetauri\b" 3;

"~*\bMicrosoft\ Data\ Access\b" 3;

"~*\bMicrosoft\ URL\ Control\b" 3;

"~*\bMister\ PiX\b" 3;

"~*\bMojeek\b" 3;

"~*\bMojolicious\b" 3;

"~*\bMorfeus\ Fucking\ Scanner\b" 3;

"~*\bMr.4x3\b" 3;

"~*\bMsrabot\b" 3;

"~*\bMusobot\b" 3;

"~*\bNICErsPRO\b" 3;

"~*\bNPbot\b" 3;

"~*\bName\ Intelligence\b" 3;

"~*\bNameprotect\b" 3;

"~*\bNavroad\b" 3;

"~*\bNearSite\b" 3;

"~*\bNeedle\b" 3;

"~*\bNessus\b" 3;

"~*\bNetAnts\b" 3;

"~*\bNetLyzer\b" 3;

"~*\bNetMechanic\b" 3;

"~*\bNetSpider\b" 3;

"~*\bNetZIP\b" 3;

"~*\bNet\ Vampire\b" 3;

"~*\bNetcraft\b" 3;

"~*\bNettrack\b" 3;

"~*\bNetvibes\b" 3;

"~*\bNextGenSearchBot\b" 3;

"~*\bNibbler\b" 3;

"~*\bNiki-bot\b" 3;

"~*\bNikto\b" 3;

"~*\bNimbleCrawler\b" 3;

"~*\bNimbostratus\b" 3;

"~*\bNinja\b" 3;

"~*\bNmap\b" 3;

"~*\bNutch\b" 3;

"~*\bOctopus\b" 3;

"~*\bOffline\ Explorer\b" 3;

"~*\bOffline\ Navigator\b" 3;

"~*\bOnCrawl\b" 3;

"~*\bOpenLinkProfiler\b" 3;

"~*\bOpenVAS\b" 3;

"~*\bOpenfind\b" 3;

"~*\bOpenvas\b" 3;

"~*\bOrangeBot\b" 3;

"~*\bOrangeSpider\b" 3;

"~*\bOutclicksBot\b" 3;

"~*\bOutfoxBot\b" 3;

"~*\bPECL::HTTP\b" 3;

"~*\bPHPCrawl\b" 3;

"~*\bPOE-Component-Client-HTTP\b" 3;

"~*\bPageAnalyzer\b" 3;

"~*\bPageGrabber\b" 3;

"~*\bPageScorer\b" 3;

"~*\bPage\ Analyzer\b" 3;

"~*\bPandalytics\b" 3;

"~*\bPanscient\b" 3;

"~*\bPapa\ Foto\b" 3;

"~*\bPavuk\b" 3;

"~*\bPeoplePal\b" 3;

"~*\bPi-Monster\b" 3;

"~*\bPicscout\b" 3;

"~*\bPicsearch\b" 3;

"~*\bPictureFinder\b" 3;

"~*\bPimonster\b" 3;

"~*\bPixray\b" 3;

"~*\bPleaseCrawl\b" 3;

"~*\bPockey\b" 3;

"~*\bProPowerBot\b" 3;

"~*\bProWebWalker\b" 3;

"~*\bProbethenet\b" 3;

"~*\bPsbot\b" 3;

"~*\bPump\b" 3;

"~*\bPxBroker\b" 3;

"~*\bPyCurl\b" 3;

"~*\bQueryN\ Metasearch\b" 3;

"~*\bQuick-Crawler\b" 3;

"~*\bRSSingBot\b" 3;

"~*\bRankActive\b" 3;

"~*\bRankActiveLinkBot\b" 3;

"~*\bRankFlex\b" 3;

"~*\bRankingBot\b" 3;

"~*\bRankingBot2\b" 3;

"~*\bRankivabot\b" 3;

"~*\bRankurBot\b" 3;

"~*\bReGet\b" 3;

"~*\bRealDownload\b" 3;

"~*\bReaper\b" 3;

"~*\bRebelMouse\b" 3;

"~*\bRecorder\b" 3;

"~*\bRedesScrapy\b" 3;

"~*\bRepoMonkey\b" 3;

"~*\bRipper\b" 3;

"~*\bRocketCrawler\b" 3;

"~*\bRogerbot\b" 3;

"~*\bSBIder\b" 3;

"~*\bSEOkicks\b" 3;

"~*\bSEOkicks-Robot\b" 3;

"~*\bSEOlyticsCrawler\b" 3;

"~*\bSEOprofiler\b" 3;

"~*\bSEOstats\b" 3;

"~*\bSISTRIX\b" 3;

"~*\bSMTBot\b" 3;

"~*\bSalesIntelligent\b" 3;

"~*\bScanAlert\b" 3;

"~*\bScanbot\b" 3;

"~*\bScoutJet\b" 3;

"~*\bScrapy\b" 3;

"~*\bScreaming\b" 3;

"~*\bScreenerBot\b" 3;

"~*\bSearchestate\b" 3;

"~*\bSearchmetricsBot\b" 3;

"~*\bSemrush\b" 3;

"~*\bSemrushBot\b" 3;

"~*\bSeoSiteCheckup\b" 3;

"~*\bSeomoz\b" 3;

"~*\bShodan\b" 3;

"~*\bSiphon\b" 3;

"~*\bSiteExplorer\b" 3;

"~*\bSiteLockSpider\b" 3;

"~*\bSiteSnagger\b" 3;

"~*\bSiteSucker\b" 3;

"~*\bSite\ Sucker\b" 3;

"~*\bSitebeam\b" 3;

"~*\bSiteimprove\b" 3;

"~*\bSitevigil\b" 3;

"~*\bSlySearch\b" 3;

"~*\bSmartDownload\b" 3;

"~*\bSnake\b" 3;

"~*\bSnapbot\b" 3;

"~*\bSnoopy\b" 3;

"~*\bSocialRankIOBot\b" 3;

"~*\bSociscraper\b" 3;

"~*\bSogou\ web\ spider\b" 3;

"~*\bSosospider\b" 3;

"~*\bSottopop\b" 3;

"~*\bSpaceBison\b" 3;

"~*\bSpammen\b" 3;

"~*\bSpankBot\b" 3;

"~*\bSpanner\b" 3;

"~*\bSpbot\b" 3;

"~*\bSpinn3r\b" 3;

"~*\bSputnikBot\b" 3;

"~*\bSqlmap\b" 3;

"~*\bSqlworm\b" 3;

"~*\bSqworm\b" 3;

"~*\bSteeler\b" 3;

"~*\bStripper\b" 3;

"~*\bSucker\b" 3;

"~*\bSucuri\b" 3;

"~*\bSuperBot\b" 3;

"~*\bSuperHTTP\b" 3;

"~*\bSurfbot\b" 3;

"~*\bSurveyBot\b" 3;

"~*\bSuzuran\b" 3;

"~*\bSwiftbot\b" 3;

"~*\bSzukacz\b" 3;

"~*\bT0PHackTeam\b" 3;

"~*\bT8Abot\b" 3;

"~*\bTeleport\b" 3;

"~*\bTeleportPro\b" 3;

"~*\bTelesoft\b" 3;

"~*\bTelesphoreo\b" 3;

"~*\bTelesphorep\b" 3;

"~*\bTheNomad\b" 3;

"~*\bThe\ Intraformant\b" 3;

"~*\bThumbor\b" 3;

"~*\bTightTwatBot\b" 3;

"~*\bTitan\b" 3;

"~*\bToata\b" 3;

"~*\bToweyabot\b" 3;

"~*\bTracemyfile\b" 3;

"~*\bTrendiction\b" 3;

"~*\bTrendictionbot\b" 3;

"~*\bTrue_Robot\b" 3;

"~*\bTuringos\b" 3;

"~*\bTurnitin\b" 3;

"~*\bTurnitinBot\b" 3;

"~*\bTwengaBot\b" 3;

"~*\bTwice\b" 3;

"~*\bTyphoeus\b" 3;

"~*\bURLy.Warning\b" 3;

"~*\bURLy\ Warning\b" 3;

"~*\bUnisterBot\b" 3;

"~*\bUpflow\b" 3;

"~*\bVB\ Project\b" 3;

"~*\bVCI\b" 3;

"~*\bVacuum\b" 3;

"~*\bVagabondo\b" 3;

"~*\bVeriCiteCrawler\b" 3;

"~*\bVidibleScraper\b" 3;

"~*\bVirusdie\b" 3;

"~*\bVoidEYE\b" 3;

"~*\bVoil\b" 3;

"~*\bVoltron\b" 3;

"~*\bWASALive-Bot\b" 3;

"~*\bWBSearchBot\b" 3;

"~*\bWEBDAV\b" 3;

"~*\bWISENutbot\b" 3;

"~*\bWPScan\b" 3;

"~*\bWWW-Collector-E\b" 3;

"~*\bWWW-Mechanize\b" 3;

"~*\bWWW::Mechanize\b" 3;

"~*\bWWWOFFLE\b" 3;

"~*\bWallpapers/3.0\b" 3;

"~*\bWallpapersHD\b" 3;

"~*\bWeSEE\b" 3;

"~*\bWebAuto\b" 3;

"~*\bWebBandit\b" 3;

"~*\bWebCollage\b" 3;

"~*\bWebCopier\b" 3;

"~*\bWebEnhancer\b" 3;

"~*\bWebFetch\b" 3;

"~*\bWebFuck\b" 3;

"~*\bWebGo\ IS\b" 3;

"~*\bWebImageCollector\b" 3;

"~*\bWebLeacher\b" 3;

"~*\bWebPix\b" 3;

"~*\bWebReaper\b" 3;

"~*\bWebSauger\b" 3;

"~*\bWebStripper\b" 3;

"~*\bWebSucker\b" 3;

"~*\bWebWhacker\b" 3;

"~*\bWebZIP\b" 3;

"~*\bWeb\ Auto\b" 3;

"~*\bWeb\ Collage\b" 3;

"~*\bWeb\ Enhancer\b" 3;

"~*\bWeb\ Fetch\b" 3;

"~*\bWeb\ Fuck\b" 3;

"~*\bWeb\ Pix\b" 3;

"~*\bWeb\ Sauger\b" 3;

"~*\bWeb\ Sucker\b" 3;

"~*\bWebalta\b" 3;

"~*\bWebmasterWorldForumBot\b" 3;

"~*\bWebshag\b" 3;

"~*\bWebsiteExtractor\b" 3;

"~*\bWebsiteQuester\b" 3;

"~*\bWebsite\ Quester\b" 3;

"~*\bWebster\b" 3;

"~*\bWhack\b" 3;

"~*\bWhacker\b" 3;

"~*\bWhatweb\b" 3;

"~*\bWho.is\ Bot\b" 3;

"~*\bWidow\b" 3;

"~*\bWinHTTrack\b" 3;

"~*\bWiseGuys\ Robot\b" 3;

"~*\bWonderbot\b" 3;

"~*\bWoobot\b" 3;

"~*\bWotbox\b" 3;

"~*\bWprecon\b" 3;

"~*\bXaldon\ WebSpider\b" 3;

"~*\bXaldon_WebSpider\b" 3;

"~*\bXenu\b" 3;

"~*\bYoudaoBot\b" 3;

"~*\bZade\b" 3;

"~*\bZauba\b" 3;

"~*\bZermelo\b" 3;

"~*\bZeus\b" 3;

"~*\bZitebot\b" 3;

"~*\bZmEu\b" 3;

"~*\bZumBot\b" 3;

"~*\bZyBorg\b" 3;

"~*\barchive.org_bot\b" 3;

"~*\barquivo-web-crawler\b" 3;

"~*\barquivo.pt\b" 3;

"~*\bautoemailspider\b" 3;

"~*\bbacklink-check\b" 3;

"~*\bcoccocbot-web\b" 3;

"~*\bcognitiveseo\b" 3;

"~*\bcom.plumanalytics\b" 3;

"~*\bcrawl.sogou.com\b" 3;

"~*\bcrawler.feedback\b" 3;

"~*\bcrawler4j\b" 3;

"~*\bdemandbase-bot\b" 3;

"~*\beCatch\b" 3;

"~*\bevc-batch\b" 3;

"~*\bfacebookscraper\b" 3;

"~*\bheritrix\b" 3;

"~*\binstabid\b" 3;

"~*\binternetVista\ monitor\b" 3;

"~*\bips-agent\b" 3;

"~*\blwp-request\b" 3;

"~*\blwp-trivial\b" 3;

"~*\bmagpie-crawler\b" 3;

"~*\bmeanpathbot\b" 3;

"~*\bmediawords\b" 3;

"~*\bmuhstik-scan\b" 3;

"~*\bnetEstate\ NE\ Crawler\b" 3;

"~*\boBot\b" 3;

"~*\bpage\ scorer\b" 3;

"~*\bpcBrowser\b" 3;

"~*\bplumanalytics\b" 3;

"~*\bs1z.ru\b" 3;

"~*\bscan.lol\b" 3;

"~*\bseoscanners\b" 3;

"~*\bserpstatbot\b" 3;

"~*\bsexsearcher\b" 3;

"~*\bsogouspider\b" 3;

"~*\bsp_auditbot\b" 3;

"~*\bspyfu\b" 3;

"~*\bsysscan\b" 3;

"~*\btAkeOut\b" 3;

"~*\btrendiction.com\b" 3;

"~*\btrendiction.de\b" 3;

"~*\bwebmeup-crawler\b" 3;

"~*\bx09Mozilla\b" 3;

"~*\bx22Mozilla\b" 3;

"~*\bxpymep1.exe\b" 3;

"~*\bzauba.io\b" 3;

"~*\bzgrab\b" 3;

# END BAD BOTS ### DO NOT EDIT THIS LINE AT ALL ###

# END BAD BOTS ### DO NOT EDIT THIS LINE AT ALL ###

# END BAD BOTS ### DO NOT EDIT THIS LINE AT ALL ###

}

# ****************************************

# SECOND BLOCK BY REFERER STRINGS AND URLS

# ****************************************

# Add here all referrer words and URL's that are to blocked.

# *****************

# PLEASE TEST !!!!

# *****************

# ALWAYS test referers that you add. This is done manually as follows

# curl -I http://www.yourdomain.com -e http://anything.adcash.com

# curl -I http://www.yourdomain.com -e http://www.goodwebsite.com/not-adcash

# curl -I http://www.yourdomain.com -e http://www.betterwebsite.com/not/adcash

# This uses curl to send the referer string to your site and you should see an immediate

# 403 Forbidden Error or No Response at all if you use the 444 error like I do.

# Because of case-insensitive matching any combination of capitilization in the names

# will all produce a positive hit - make sure you always test thoroughly and monitor logs

# This also does NOT check for a preceding www. nor does it check for it ending in .com

# .net .org or any long string attached at the end. It also does not care if the referer

# was sent with http https or even ftp.

# Please be careful with these and think carefully before you add new words.

# Remember we are trying to keep out the general riff-raff not kill your web sites.

# **********************************************************************

# Now we map all bad referer words below to a variable called $bad_words

# **********************************************************************

map $http_referer $bad_words {

default 0;

# ************************

# Bad Referer Single Words

# ************************

# These are Words and Terms often found tagged onto domains or within url query strings.

}

# ************************

# Bad Referer Domain Names

# ************************

# Now a list of bad referer urls these domains or any combination of them ie .com .net

# will be blocked out. Doesn't matter if the protocol is http, https or even ftp

# This section includes:

# **********************

# Blocking of SEO company Semalt.com (now merged into this one section)

# MIRAI Botnet Domains Used for Mass Attacks

# Other known bad SEO companies and Ad Hijacking Sites

# Sites linked to malware, adware and ransomware

# *****************

# PLEASE TEST !!!!

# *****************

# ALWAYS test referers that you add. This is done manually as follows

# curl -I http://www.yourdomain.com -e http://8gold.com

# This uses curl to send the referer string to your site and you should see an immediate

# 403 Forbidden Error or No Response at all if you use the 444 error like I do.

# Because of case-insensitive matching any combination of capitilization

# will all produce a positive hit - make sure you always test.

# curl -I http://www.yourdomain.com -e http://NOT-8gold.com

# curl -I http://www.yourdomain.com -e http://this.is.not8gOlD.net

# curl -I http://www.yourdomain.com -e ftp://8gold.com

# curl -I http://www.yourdomain.com -e ftp://www.weare8gold.NET

# curl -I http://www.yourdomain.com -e https://subdomain.8gold.com

# curl -I http://www.yourdomain.com -e https://NOT8GolD.org

# This works exactly like the bad referer word lists above and is very strict !!!

# I have gone for the simple stricter approach which blocks all variants for those

# who just hop out and but another domain name.

# So if you see a bad referer from wearegoogle.com and you want to block them just add

# them as "~*wearegoogle.com" don't ever go and do something like "~*google.com" you will

# kill all your SEO in a week.

# While you can fork your own copy and modify it as you please we appreciate you rather sending

# a Pull Request by following the instructions in the Pull_Requests folder.

# ***********************************************************************

# Now we map all good & bad referer urls to variable called #bad_referer

# ***********************************************************************

map $http_referer $bad_referer {

hostnames;

default 0;

# ************************************

# GOOD REFERERS - Spared from Checking

# ************************************

# Add your own domain names here to spare them from referer checking (one per line)

# Use the new include file method so any further updates will no longer require you to

# have to keep putting your whitelisted domains here when updating.

# PLEASE READ UPDATED CONFIGURATION INSTRUCTIONS REGARDING THESE INCLUDE FILES

include /etc/nginx/bots.d/whitelist-domains.conf;

# START BAD REFERERS ### DO NOT EDIT THIS LINE AT ALL ###

"~*\b000free\.us\b" 1;

"~*\b007angels\.com\b" 1;

"~*\b00author\.com\b" 1;

"~*\b00go\.com\b" 1;

"~*\b00it\.com\b" 1;

"~*\b00webcams\.com\b" 1;

"~*\b01apple\.com\b" 1;

"~*\b03e\.info\b" 1;

"~*\b03p\.info\b" 1;

"~*\b08800\.top\b" 1;

"~*\b0912701309f8ce\.com\b" 1;

"~*\b0c47f8422d3f\.com\b" 1;

"~*\b0daymusic\.org\b" 1;

"~*\b0lovespells0\.blogspot\.com\b" 1;

"~*\b0n\-line\.tv\b" 1;

"~*\b1000n1\.ru\b" 1;

"~*\b1001watch\.com\.ua\b" 1;

"~*\b100dollars\-seo\.com\b" 1;

"~*\b100searchengines\.com\b" 1;

"~*\b101billion\.com\b" 1;

"~*\b101flag\.ru\b" 1;

"~*\b101lesbian\.xyz\b" 1;

"~*\b101raccoon\.ru\b" 1;

"~*\b108shot\.com\b" 1;

"~*\b10bet\.com\b" 1;

"~*\b11235813\.webzdarma\.cz\b" 1;

"~*\b11pikachu\.ru\b" 1;

"~*\b123any\.com\b" 1;

"~*\b123cha\.com\b" 1;

"~*\b123kuma\.com\b" 1;

"~*\b123locker\.com\b" 1;

"~*\b123movies\.love\b" 1;

"~*\b12bet\.com\b" 1;

"~*\b12masterov\.com\b" 1;

"~*\b12u\.info\b" 1;

"~*\b1314dh\.com\b" 1;

"~*\b13tabs\.com\b" 1;

"~*\b14b\.info\b" 1;

"~*\b1688\.com\b" 1;

"~*\b178evakuator178\.ru\b" 1;

"~*\b1\-99seo\.com\b" 1;

"~*\b1\-free\-share\-buttons\.com\b" 1;

"~*\b1adult\.com\b" 1;

"~*\b1bet\.com\b" 1;

"~*\b1flag\.co\.za\b" 1;

"~*\b1hwy\.com\b" 1;

"~*\b1j7740kd\.website\b" 1;

"~*\b1kdailyprofit\.me\b" 1;

"~*\b1kinobig\.ru\b" 1;

"~*\b1millionusd\.xyz\b" 1;

"~*\b1pamm\.ru\b" 1;

"~*\b1qingdao\.com\b" 1;

"~*\b1stat\.ru\b" 1;

"~*\b1webmaster\.ml\b" 1;

"~*\b1xbet4you\.com\b" 1;

"~*\b2000k\.ru\b" 1;

"~*\b2015god\.org\b" 1;

"~*\b2020iscoming\.info\b" 1;

"~*\b202ch\.com\b" 1;

"~*\b20pascals\.nl\b" 1;

"~*\b214jaluwobapef\.cf\b" 1;

"~*\b21h2o\.com\b" 1;

"~*\b2345\.com\b" 1;

"~*\b23kw\.ru\b" 1;

"~*\b24\-ak\.ru\b" 1;

"~*\b24videos\.tv\b" 1;

"~*\b24x7\-server\-support\.site\b" 1;

"~*\b256bit\.by\b" 1;

"~*\b2728fb936f0\.com\b" 1;

"~*\b273\-fz\.ru\b" 1;

"~*\b28n2gl3wfyb0\.ru\b" 1;

"~*\b2ads\.co\.uk\b" 1;

"~*\b2daytrendingnews\.com\b" 1;

"~*\b2drittel\.de\b" 1;

"~*\b2girls1cup\-free\.com\b" 1;

"~*\b2itech\.ru\b" 1;

"~*\b2kata\.ru\b" 1;

"~*\b2nt\.ru\b" 1;

"~*\b2pxg8bcf\.top\b" 1;

"~*\b2rich4bitches\.com\b" 1;

"~*\b2x2fan\.ru\b" 1;

"~*\b300richmond\.co\.nz\b" 1;

"~*\b34\.gs\b" 1;