YisouSpider 是不是很无耻？

V2EX = way to explore

V2EX 是一个关于分享和探索的地方

For Existing Member Sign In

Recommended Services

› Amazon Web Services

› LeanCloud

› New Relic

› ClearDB

This topic created in 3066 days ago, the information mentioned may be changed or developed.

YisouSpider 完全不遵守 robots 啊，一次来一组 IP，一组 IP 有 5 个，轮着来！ KAO，太无耻了！加了一组 IP 到 Deny，改天又来新的一组 IP ！

无耻

deny

kao

robots

16 replies • 2020-02-22 04:06:50 +08:00

mydns

Dec 4, 2017

防火墙列表里全是 yisou 和神马搜索这两个的 IP 爬起来和 cc 一样~~~

Had

Dec 4, 2017

直接按 UA 过滤呗...

TKKONE

PRO

Dec 4, 2017

你没见过更无耻的 MJ12bot 吧

loveminds

Dec 4, 2017

@mydns 本来就是一个，前身和雅虎中国 /一搜有关系

enenaaa

Dec 4, 2017

@mydns 感觉神马搜索很高冷啊。专门去提交 url 还不理人。

handwork

Dec 4, 2017

@tumbzzc 88.198.16.12 - - [04/Dec/2017:04:06:01 +0800] "GET /robots.txt HTTP/1.1" 200 1485 "-" "Mozilla/5.0 (compatible; MJ12bot/v1.4.7; http://mj12bot.com/)"
还守规矩啊，爬了 robots 就悄悄的走了，没其它痕迹……

flyingghost

Dec 4, 2017

@huaxing0211 #6 于是换个 UA 再来。/机智

qdcanyun

Dec 4, 2017

是的。。。他们的爬虫就像 cc 攻击一样，超级烦

JoeyChan

Dec 4, 2017

吓得我赶紧查了一下，还好已经在黑名单了，return 444。

mydns

Dec 4, 2017

YisouSpider
MJ12bot
AhrefsBot
DotBot
SemrushBot
以上就是这几年屏蔽的搜索引擎列表 robots 都无效用防火墙封 ip 段解决的每次一上站 cpu 就 100% 而且持续一整天

mydns

Dec 4, 2017

补充 IP 段：
42.156.254.37-42.156.254.42
103.47.210.1-103.47.210.254
42.156.136.103
42.120.161.103
42.120.160.103
42.156.139.103
42.156.138.103
42.156.137.103
180.97.106.162
180.97.106.161
180.97.106.37
42.120.161.72
42.156.138.72
42.120.160.72
42.156.137.72
42.156.136.72
42.156.139.72
5.9.6.51
69.30.198.178
193.70.39.162
163.172.68.136
72.90.76.89
173.234.159.250
144.76.29.66
93.219.71.27
163.172.32.175
136.243.152.18
144.76.7.106
178.203.146.96
193.70.37.50
42.156.137.107
42.120.160.107
42.156.138.107
42.156.139.107
213.251.184.38
42.156.139.46
42.120.161.107
91.209.51.22
42.156.138.46
42.156.136.46
42.156.137.46
62.138.8.181
42.156.139.106
42.120.161.106
42.120.160.106
42.156.137.106
42.156.136.106
42.120.160.83
42.156.138.83
42.156.136.83
42.156.254.47
42.156.136.108
42.156.137.108
42.156.138.108
42.120.160.108
42.120.161.108
42.156.139.108
216.244.66.247
51.255.65.88
137.74.201.99
164.132.161.7
164.132.161.8
51.255.71.118
164.132.161.59
182.118.33.6
175.2.133.236
175.15.110.180
175.15.118.111

handwork

Dec 4, 2017

@mydns
42.156.139.62
42.156.138.62
42.156.137.96
42.156.137.62
42.156.136.62
42.120.160.62
42.156.136.33
42.156.139.33
42.120.161.33
42.156.137.33
42.156.138.33
106.11.152.186
106.11.156.169
106.11.157.173
106.11.155.161
106.11.154.169
106.11.153.181
106.11.159.197
106.11.158.214
106.11.152.198
106.11.153.204
106.11.157.220
106.11.156.193
106.11.157.148
106.11.154.153
106.11.158.139
106.11.155.158
106.11.152.134
106.11.159.138