I think I got that last of the botters that are trying to ruin this site

Tue, 10 Dec 2013, 3:23pm	I think I got that last of the botters that are trying to ruin this site »
scott Administrator Registered: Sep, 2008 Last visit: Thu, 06 Oct 2022 Posts: 706	Found another user trying to scrape the site today: rocessing PostsController#show (for 187.184.243.34 at 2013-12-10 10:10:29) [GET] Processing TopicsController#show (for 187.184.243.34 at 2013-12-10 10:10:29) [GET] Processing PostsController#topic (for 187.184.243.34 at 2013-12-10 10:10:30) [GET] Processing PostsController#show (for 187.184.243.34 at 2013-12-10 10:10:30) [GET] Processing PostsController#topic (for 187.184.243.34 at 2013-12-10 10:10:31) [GET] Processing TopicsController#show (for 187.184.243.34 at 2013-12-10 10:10:30) [GET] Processing PostsController#show (for 187.184.243.34 at 2013-12-10 10:10:31) [GET] Processing TopicsController#show (for 187.184.243.34 at 2013-12-10 10:10:31) [GET] Processing PostsController#topic (for 187.184.243.34 at 2013-12-10 10:10:32) [GET] Processing TopicsController#show (for 187.184.243.34 at 2013-12-10 10:10:32) [GET] Processing PostsController#show (for 187.184.243.34 at 2013-12-10 10:10:33) [GET] Processing PostsController#topic (for 187.184.243.34 at 2013-12-10 10:10:33) [GET] Processing PostsController#show (for 187.184.243.34 at 2013-12-10 10:10:33) [GET] Processing PostsController#topic (for 187.184.243.34 at 2013-12-10 10:10:33) [GET] Processing TopicsController#show (for 187.184.243.34 at 2013-12-10 10:10:33) [GET] Processing PostsController#show (for 187.184.243.34 at 2013-12-10 10:10:34) [GET] Processing TopicsController#show (for 187.184.243.34 at 2013-12-10 10:10:34) [GET] Processing PostsController#topic (for 187.184.243.34 at 2013-12-10 10:10:34) [GET] Processing PostsController#show (for 187.184.243.34 at 2013-12-10 10:10:35) [GET] Processing PostsController#topic (for 187.184.243.34 at 2013-12-10 10:10:35) [GET] Processing PostsController#show (for 187.184.243.34 at 2013-12-10 10:10:36) [GET] Processing TopicsController#show (for 187.184.243.34 at 2013-12-10 10:10:36) [GET] Processing PostsController#topic (for 187.184.243.34 at 2013-12-10 10:10:37) [GET] Processing PostsController#show (for 187.184.243.34 at 2013-12-10 10:10:37) [GET] Processing PostsController#topic (for 187.184.243.34 at 2013-12-10 10:10:38) [GET] Processing TopicsController#show (for 187.184.243.34 at 2013-12-10 10:10:37) [GET] Processing PostsController#show (for 187.184.243.34 at 2013-12-10 10:10:38) [GET] Processing TopicsController#show (for 187.184.243.34 at 2013-12-10 10:10:38) [GET] Processing PostsController#topic (for 187.184.243.34 at 2013-12-10 10:10:39) [GET] Processing PostsController#show (for 187.184.243.34 at 2013-12-10 10:10:40) [GET] Processing TopicsController#show (for 187.184.243.34 at 2013-12-10 10:10:39) [GET] Processing PostsController#topic (for 187.184.243.34 at 2013-12-10 10:10:41) [GET] Processing PostsController#show (for 187.184.243.34 at 2013-12-10 10:10:42) [GET] Processing PostsController#topic (for 187.184.243.34 at 2013-12-10 10:10:43) [GET] Processing TopicsController#show (for 187.184.243.34 at 2013-12-10 10:10:42) [GET] Processing PostsController#show (for 187.184.243.34 at 2013-12-10 10:10:43) [GET] Processing TopicsController#show (for 187.184.243.34 at 2013-12-10 10:10:43) [GET] Processing PostsController#topic (for 187.184.243.34 at 2013-12-10 10:10:44) [GET] Processing PostsController#show (for 187.184.243.34 at 2013-12-10 10:10:44) [GET] Processing TopicsController#show (for 187.184.243.34 at 2013-12-10 10:10:44) [GET] Processing PostsController#topic (for 187.184.243.34 at 2013-12-10 10:10:46) [GET] Processing PostsController#show (for 187.184.243.34 at 2013-12-10 10:10:46) [GET] Processing PostsController#topic (for 187.184.243.34 at 2013-12-10 10:10:46) [GET] Processing TopicsController#show (for 187.184.243.34 at 2013-12-10 10:10:46) [GET] Processing PostsController#show (for 187.184.243.34 at 2013-12-10 10:10:47) [GET] Processing TopicsController#show (for 187.184.243.34 at 2013-12-10 10:10:47) [GET] Processing PostsController#topic (for 187.184.243.34 at 2013-12-10 10:10:48) [GET] Processing TopicsController#show (for 187.184.243.34 at 2013-12-10 10:10:48) [GET] Processing PostsController#show (for 187.184.243.34 at 2013-12-10 10:10:48) [GET] Processing PostsController#topic (for 187.184.243.34 at 2013-12-10 10:10:49) [GET] Processing PostsController#show (for 187.184.243.34 at 2013-12-10 10:10:50) [GET] Processing TopicsController#show (for 187.184.243.34 at 2013-12-10 10:10:50) [GET] Processing PostsController#topic (for 187.184.243.34 at 2013-12-10 10:10:50) [GET] Processing PostsController#show (for 187.184.243.34 at 2013-12-10 10:10:51) [GET] Processing TopicsController#show (for 187.184.243.34 at 2013-12-10 10:10:51) [GET] Please don't, you will get banned like you just did. This user had about 60,000 pages requests in the past 8 hours: root@webapp1:/usr/local/bin# cat /u1/app/wlodb/log/production.log \| grep 187.184.243.34 \| wc -l 58325 Other than that, the auto blacklist is working well and has banned about 1300 ip addresses in the past 2 weeks so far. I also got the database fixed up a lot. It's pulling 90% of the queries from the table cache now.
Offline	Link

Found another user trying to scrape the site today:

rocessing PostsController#show (for 187.184.243.34 at 2013-12-10 10:10:29) [GET]
Processing TopicsController#show (for 187.184.243.34 at 2013-12-10 10:10:29) [GET]
Processing PostsController#topic (for 187.184.243.34 at 2013-12-10 10:10:30) [GET]
Processing PostsController#show (for 187.184.243.34 at 2013-12-10 10:10:30) [GET]
Processing PostsController#topic (for 187.184.243.34 at 2013-12-10 10:10:31) [GET]
Processing TopicsController#show (for 187.184.243.34 at 2013-12-10 10:10:30) [GET]
Processing PostsController#show (for 187.184.243.34 at 2013-12-10 10:10:31) [GET]
Processing TopicsController#show (for 187.184.243.34 at 2013-12-10 10:10:31) [GET]
Processing PostsController#topic (for 187.184.243.34 at 2013-12-10 10:10:32) [GET]
Processing TopicsController#show (for 187.184.243.34 at 2013-12-10 10:10:32) [GET]
Processing PostsController#show (for 187.184.243.34 at 2013-12-10 10:10:33) [GET]
Processing PostsController#topic (for 187.184.243.34 at 2013-12-10 10:10:33) [GET]
Processing PostsController#show (for 187.184.243.34 at 2013-12-10 10:10:33) [GET]
Processing PostsController#topic (for 187.184.243.34 at 2013-12-10 10:10:33) [GET]
Processing TopicsController#show (for 187.184.243.34 at 2013-12-10 10:10:33) [GET]
Processing PostsController#show (for 187.184.243.34 at 2013-12-10 10:10:34) [GET]
Processing TopicsController#show (for 187.184.243.34 at 2013-12-10 10:10:34) [GET]
Processing PostsController#topic (for 187.184.243.34 at 2013-12-10 10:10:34) [GET]
Processing PostsController#show (for 187.184.243.34 at 2013-12-10 10:10:35) [GET]
Processing PostsController#topic (for 187.184.243.34 at 2013-12-10 10:10:35) [GET]
Processing PostsController#show (for 187.184.243.34 at 2013-12-10 10:10:36) [GET]
Processing TopicsController#show (for 187.184.243.34 at 2013-12-10 10:10:36) [GET]
Processing PostsController#topic (for 187.184.243.34 at 2013-12-10 10:10:37) [GET]
Processing PostsController#show (for 187.184.243.34 at 2013-12-10 10:10:37) [GET]
Processing PostsController#topic (for 187.184.243.34 at 2013-12-10 10:10:38) [GET]
Processing TopicsController#show (for 187.184.243.34 at 2013-12-10 10:10:37) [GET]
Processing PostsController#show (for 187.184.243.34 at 2013-12-10 10:10:38) [GET]
Processing TopicsController#show (for 187.184.243.34 at 2013-12-10 10:10:38) [GET]
Processing PostsController#topic (for 187.184.243.34 at 2013-12-10 10:10:39) [GET]
Processing PostsController#show (for 187.184.243.34 at 2013-12-10 10:10:40) [GET]
Processing TopicsController#show (for 187.184.243.34 at 2013-12-10 10:10:39) [GET]
Processing PostsController#topic (for 187.184.243.34 at 2013-12-10 10:10:41) [GET]
Processing PostsController#show (for 187.184.243.34 at 2013-12-10 10:10:42) [GET]
Processing PostsController#topic (for 187.184.243.34 at 2013-12-10 10:10:43) [GET]
Processing TopicsController#show (for 187.184.243.34 at 2013-12-10 10:10:42) [GET]
Processing PostsController#show (for 187.184.243.34 at 2013-12-10 10:10:43) [GET]
Processing TopicsController#show (for 187.184.243.34 at 2013-12-10 10:10:43) [GET]
Processing PostsController#topic (for 187.184.243.34 at 2013-12-10 10:10:44) [GET]
Processing PostsController#show (for 187.184.243.34 at 2013-12-10 10:10:44) [GET]
Processing TopicsController#show (for 187.184.243.34 at 2013-12-10 10:10:44) [GET]
Processing PostsController#topic (for 187.184.243.34 at 2013-12-10 10:10:46) [GET]
Processing PostsController#show (for 187.184.243.34 at 2013-12-10 10:10:46) [GET]
Processing PostsController#topic (for 187.184.243.34 at 2013-12-10 10:10:46) [GET]
Processing TopicsController#show (for 187.184.243.34 at 2013-12-10 10:10:46) [GET]
Processing PostsController#show (for 187.184.243.34 at 2013-12-10 10:10:47) [GET]
Processing TopicsController#show (for 187.184.243.34 at 2013-12-10 10:10:47) [GET]
Processing PostsController#topic (for 187.184.243.34 at 2013-12-10 10:10:48) [GET]
Processing TopicsController#show (for 187.184.243.34 at 2013-12-10 10:10:48) [GET]
Processing PostsController#show (for 187.184.243.34 at 2013-12-10 10:10:48) [GET]
Processing PostsController#topic (for 187.184.243.34 at 2013-12-10 10:10:49) [GET]
Processing PostsController#show (for 187.184.243.34 at 2013-12-10 10:10:50) [GET]
Processing TopicsController#show (for 187.184.243.34 at 2013-12-10 10:10:50) [GET]
Processing PostsController#topic (for 187.184.243.34 at 2013-12-10 10:10:50) [GET]
Processing PostsController#show (for 187.184.243.34 at 2013-12-10 10:10:51) [GET]
Processing TopicsController#show (for 187.184.243.34 at 2013-12-10 10:10:51) [GET]

Please don't, you will get banned like you just did. This user had about 60,000 pages requests in the past 8 hours:

root@webapp1:/usr/local/bin# cat /u1/app/wlodb/log/production.log | grep 187.184.243.34 | wc -l
58325

Other than that, the auto blacklist is working well and has banned about 1300 ip addresses in the past 2 weeks so far.

I also got the database fixed up a lot. It's pulling 90% of the queries from the table cache now.