Advanced search  

News:

CPG Release 1.6.26
Correct PHP8.2 issues with user and language managers.
Additional fixes for PHP 8.2
Correct PHP8 error with SMF 2.0 bridge.
Correct IPTC supplimental category parsing.
Download and info HERE

Pages: [1]   Go Down

Author Topic: How to prevent entire site downloads  (Read 13238 times)

0 Members and 1 Guest are viewing this topic.

Sogeri

  • Coppermine novice
  • *
  • Offline Offline
  • Posts: 46
    • http://www.orchidspng.com
How to prevent entire site downloads
« on: October 02, 2003, 06:22:51 am »

My site http://www.orchidspng.com gets an average of 25,000 page hits a day, some days peaking at over 100,000 page hits. From my web stats I can see what appears to be that some people are downloading the entire site. Yesterday's traffic was over 1GB! And that costs money. It is a hobby site.

Other than regulating access to the site for registered users only or via a throttle (I find it hard to guess what a reasonable number of hits per hour/day would be) or htaccess is there any other method to block a single IP from accessing the site to often within a given time frame.
Logged

hyperion

  • VIP
  • Coppermine addict
  • ***
  • Offline Offline
  • Posts: 1317
  • - retired -
How to prevent entire site downloads
« Reply #1 on: October 02, 2003, 07:04:27 am »

Yes, but it could get complicated.  Basically, you store the IP addresses along with a timestamp.  You then delete the IP addresses as they exceed a certain time.  You then count the number of times an IP address is in the list (or increment a counter, etc.), and redirect to an explanation page when it exceeds the number of hits in the time frame. You put the call to the function at the begining of every page by placing it in the theme.php file.

Some of those downloaders might be spiders or robots that obey commands.  Use meta tags and robot files to try and keep them under control.

Great orchid shots, BTW. :)
Logged
"Then, Fletch," that bright creature said to him, and the voice was very kind, "let's begin with level flight . . . ."

-Richard Bach, Jonathan Livingston Seagull

(http://www.mozilla.org/products/firefox/buttons/getfirefox_small.png)

gtroll

  • VIP
  • Coppermine addict
  • ***
  • Offline Offline
  • Posts: 618
    • CPG-Nuke
How to prevent entire site downloads
« Reply #2 on: October 02, 2003, 07:24:19 am »

Your site downloaders are probably using a bot, you can ban them in your .htaccess
http://www.webmasterworld.com/forum13/687.htm

Jim

  • Coppermine newbie
  • Offline Offline
  • Posts: 1
    • http://www.quadrahosting.com.au
How to prevent entire site downloads
« Reply #3 on: October 02, 2003, 07:36:04 am »

webmasterworld thread is for members only :(
Logged
Quadra Hosting

Sogeri

  • Coppermine novice
  • *
  • Offline Offline
  • Posts: 46
    • http://www.orchidspng.com
How to prevent entire site downloads
« Reply #4 on: October 02, 2003, 07:40:15 am »

:D  Thanks for that. I will upload the .htaccess file as suggested.
Logged

gtroll

  • VIP
  • Coppermine addict
  • ***
  • Offline Offline
  • Posts: 618
    • CPG-Nuke
.htaccess stops bots
« Reply #5 on: October 02, 2003, 07:44:08 am »

Here you go Jim contents of the post there
Quote

#From toolman of webmasterworld
<Files .htaccess>
deny from all
</Files>
RewriteEngine on
RewriteBase /
RewriteCond %{HTTP_USER_AGENT} ^EmailSiphon [OR]
RewriteCond %{HTTP_USER_AGENT} ^EmailWolf [OR]
RewriteCond %{HTTP_USER_AGENT} ^ExtractorPro [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mozilla.*NEWT [OR]
RewriteCond %{HTTP_USER_AGENT} ^Crescent [OR]
RewriteCond %{HTTP_USER_AGENT} ^CherryPicker [OR]
RewriteCond %{HTTP_USER_AGENT} ^[Ww]eb[Bb]andit [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebEMailExtrac.* [OR]
RewriteCond %{HTTP_USER_AGENT} ^NICErsPRO [OR]
RewriteCond %{HTTP_USER_AGENT} ^Teleport [OR]
RewriteCond %{HTTP_USER_AGENT} ^Zeus.*Webster [OR]
RewriteCond %{HTTP_USER_AGENT} ^Microsoft.URL [OR]
RewriteCond %{HTTP_USER_AGENT} ^Wget [OR]
RewriteCond %{HTTP_USER_AGENT} ^LinkWalker [OR]
RewriteCond %{HTTP_USER_AGENT} ^sitecheck.internetseer.com [OR]
RewriteCond %{HTTP_USER_AGENT} ^DIIbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^psbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^EmailCollector
RewriteRule ^.* - [F]
RewriteCond %{HTTP_REFERER} ^http://www.your-site.com$
RewriteRule !^http://[^/.]\.your-site.com.* - [F]

Tarique Sani

  • VIP
  • Coppermine addict
  • ***
  • Offline Offline
  • Gender: Male
  • Posts: 2712
    • http://tariquesani.net
How to prevent entire site downloads
« Reply #6 on: October 02, 2003, 08:05:18 am »

Don't want to be rain on the parade BUT spoofing of USER_AGENT is built into most new URL fetchers. I guess the correct way is to have Apache configured using mod_throttle OR mod_bandwidth.
Logged
SANIsoft PHP applications for E Biz

Sogeri

  • Coppermine novice
  • *
  • Offline Offline
  • Posts: 46
    • http://www.orchidspng.com
How to prevent entire site downloads
« Reply #7 on: October 02, 2003, 08:20:47 am »

I found an even more extensive .htaccess file here:

http://tech.ratmachines.com/downloads/sample_wbmw.txt

So, which file would be best to use??
Logged

epsilon

  • Coppermine regular visitor
  • **
  • Offline Offline
  • Posts: 95
    • http://www.cremita.com
Ok but...
« Reply #8 on: December 04, 2003, 05:01:58 pm »

In which directory i must put this htaccess ? in albums dir only?
Logged

Joachim Müller

  • Dev Team member
  • Coppermine addict
  • ****
  • Offline Offline
  • Gender: Male
  • Posts: 47843
  • aka "GauGau"
    • gaugau.de
Re: Ok but...
« Reply #9 on: December 04, 2003, 07:11:27 pm »

Quote from: "epsilon"
In which directory i must put this htaccess ? in albums dir only?
yes
Logged

epsilon

  • Coppermine regular visitor
  • **
  • Offline Offline
  • Posts: 95
    • http://www.cremita.com
More explicit... Please
« Reply #10 on: January 08, 2004, 03:21:41 am »

Code: [Select]
Don't want to be rain on the parade BUT spoofing of USER_AGENT is built into most new URL fetchers. I guess the correct way is to have Apache configured using mod_throttle OR mod_bandwidth.

How i can do it? i have on mod_rewrite to use the .htaccess commands, and when i will activate the throttle and bandwidth what i must do?

Thnks
Logged

Tarique Sani

  • VIP
  • Coppermine addict
  • ***
  • Offline Offline
  • Gender: Male
  • Posts: 2712
    • http://tariquesani.net
How to prevent entire site downloads
« Reply #11 on: January 08, 2004, 04:21:31 am »

See http://www.snert.com/Software/mod_throttle/

I don't use it as I don't need it
Logged
SANIsoft PHP applications for E Biz
Pages: [1]   Go Up
 

Page created in 0.026 seconds with 20 queries.