Is Baldrick a human?

[El Heggunte]

Not anymore.

Are you a human?

If you are a human click here to continue use our site or contact support@videohelp.com

Happens every time I go to www.videohelp.com 🤔.

¿What the devil is going on?

[johns0]

I've never had that happen to me.

[aedipuss]

you probably have to allow cookies, as then it will be stored after the first time.

[cholla]

I get this message rarely but I have got it.
I doubt it is cookies as I'm almost obsessive about clearing cookies.
I believe my hosts file usually blocks this type message.
Possibly one of this type:apis.google.com or akamai being blocked.

The way I set up uBlock also may block this message most of the time.

[Baldrick]

I have added the human check to some countries that has very suspcious traffic to my site.

It must be a bot network that scrapes my site. They do not identify as any search engine or ai bot.

[El Heggunte]

Thanks for the explanation.

VPN is the workaround

(not a solution).

[Baldrick]

The bot network still scrapes my site. It's always around 500 connections/minute from one specific country.

Almost all identifies as Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) and browsing strange pages in the forum. Like only members sections.

[Baldrick]

I have removed the country human verification now. The bot network has disappeared.

edit: And the bot network is back again....why are they scraping same data over and over again?

edit2: If you allow cookies you should not get the human/bot verification that often.

[thecoalman]

edit: And the bot network is back again....why are they scraping same data over and over again?

Possibly to obfuscate their intentions. Do they get login form for these pages?

I've seen similar activity on many phpBB forums. Not sure about VB but on a phpBB forum memberlist page is not available to guests. My guess is they have registered "user" that scrapes memberlist page to get the usernames, or they just just scraped them from public posts.

The only conclusion I can draw is bots will try accessing memberlist or some other non public page and get access denied with login box. They aren't accessing the login link which prevents admin from seeing they are trying to login. From there they can brute force the password or they might have list of usernames with compromised passwords. They will be successful for some accounts.

From there they can for example try and fly under the radar with post or do something like change the signature which is retroactive on all posts.

phpBB tracks failed login attempts and will block the IP after X attempts for X amount of time. If you look in users table that column maxes out at 99 and it's common to see it maxed to 99 on old accounts even with IP's being blocked for hours.

One of the things I've done on my forum is reset passwords on very old inactive accounts.

[Baldrick]

Yep, I have noticed that bots try login with compromised users with passwords(just check https://haveibeenpwned.com/ and users use same login/passwords for all sites). But I have added a javascript login check so it fails for 99% of all bots.

But this bot network doesn't seem to try login at all it just scrape random data over and over. From several thousand different IPs. Almost all identifies as Mac with cookies disabled. Not much Mac related stuff in this forum.

[thecoalman]

Is it from Brazil? That seems to be the most problematic country recently.

I use Cloudflare, the number I heard recently was they handle 15% of global traffic. Whatever the case it's an enormous amount giving them copious amounts of traffic to analyze. Even the free plan is pretty good. Plenty of tools available including a one click rule for blocking bot nets they have identified.

[Baldrick]

Mostly Brazil but also from all these countries: SG HK UY TR RU AR UZ PK ZA EC IQ CN UA IN MA BD MX VE KE TN CO JO AZ NP OM CL AE KZ AL PY PE DZ RS JM GE PS KG CG DO BO AM ML SN BG KR EG EE GT MY UA SV LY BA BN MD GE HN LB NG LV MU



I can now easily "block" most of that traffic so I really don't need Cloudfare. And my server host stops bigger DDOS attacks.

[thecoalman]

Mostly Brazil but also from all these countries: SG HK UY TR RU AR UZ PK ZA EC IQ CN UA IN MA BD MX VE KE TN CO JO AZ NP OM CL AE KZ AL PY PE DZ RS JM GE PS KG CG DO BO AM ML SN BG KR EG EE GT MY UA SV LY BA BN MD GE HN LB NG LV MU

99% of my legitimate traffic is from US, it's niche site. I could even narrow it down to Northeast US. I whitelist the US and everyone else in the world gets the "Checking your browser...." page, it's less than 1% that make it through. It's all bot traffic

CF sends country code in the header with the request if you want to do more server side. I'm going to start recording it with IP so it's displayed right on the post.

As far as DDOS there is one enormous benefit with CF, you can firewall ports 80 and 443 except CF IP's. If anything it puts an end to bots banging away at the IP on ports 80 and 443 making requests for /phpmyadmin etc. It's also critical for DDOS protection so they can't just attack the IP with custom DNS.

[Baldrick]

94% of all VideoHelps traffic is now just search engines, ai scrapers, and bot traffic.

[Baldrick]

ANother bot network from Singapore and Hong Kong. I'm getting tired of this crap.

[thecoalman]

Use the force: https://www.cloudflare.com/

The free plan is actually quite good.

Under security section go to settings and enable Bot fight mode, this will block the most egregious bot nets CF has identified. You can also enable block AI but this is for identified AI bots.

Next go to Security Rules, Create rule >>

1st rule id for worst offenders, set action to interactive challenge.:

Country >> Equals >> CN OR
Country >> Equals >> IN OR
Country >> Equals>> ?? .......

2nd rule you can whitelist and for action issue Managed Challenge to everyone else:

Country >> Does not equal >> US AND
Country >> Does not equal >> UK AND
Country >> Does not equal >> ?? AND.......

Result is China and India get issued a solvable captcha, the US and UK get nothing and the rest of the world gets some kind of challenge based on what CF determines ranging from "Checking your browser..." page to a solvable captcha. You can just set it to JSChallenge which is the Checking your browser..." page. I think the default is two hours before they get it again.

There is also rate limiting section but this has limited functionality with free plan because it will only block the IP for 10s. You need Pro plan or better for this to be really effective.

[Brazil]

94% of all VideoHelps traffic is now just search engines, ai scrapers, and bot traffic.

https://about.readthedocs.com/blog/2024/07/ai-crawlers-abuse/
https://news.ycombinator.com/item?id=43422413

[Baldrick]

Use the force: https://www.cloudflare.com/

The free plan is actually quite good.

Under security section go to settings and enable Bot fight mode, this will block the most egregious bot nets CF has identified. You can also enable block AI but this is for identified AI bots.

Next go to Security Rules, Create rule >>

1st rule id for worst offenders, set action to interactive challenge.:

Country >> Equals >> CN OR
Country >> Equals >> IN OR
Country >> Equals>> ?? .......

2nd rule you can whitelist and for action issue Managed Challenge to everyone else:

Country >> Does not equal >> US AND
Country >> Does not equal >> UK AND
Country >> Does not equal >> ?? AND.......

Result is China and India get issued a solvable captcha, the US and UK get nothing and the rest of the world gets some kind of challenge based on what CF determines ranging from "Checking your browser..." page to a solvable captcha. You can just set it to JSChallenge which is the Checking your browser..." page. I think the default is two hours before they get it again.

There is also rate limiting section but this has limited functionality with free plan because it will only block the IP for 10s. You need Pro plan or better for this to be really effective.

I might try that. Thanks for info!

[thecoalman]

You need to install and configure mod_remoteip. CF sends a whole bunch of custom headers to origin server including the original IP which you are going to want to restore.

If you want to fully take advantage of the DDOS protection you need to firewall ports 80 and 443 except CF IP's. You also need to remove anything that can identify origin IP. Email server needs to be on another IP, no remote files like uploading avatars from URL etc.

There will probably be some hiccups, the WAF uses OWASP mod security rule set. What you might want to do is before setting up any blocking rules is see what else is getting blocked. Once you have any issues sorted then engage the blocking rules.

When somoe is blocked for whatever reason there is ray id on the page, that can be used to lookup the exact request that was blocked and why.