请问爬虫高手为什么每次 scrapy 用 xpath 的 extract_first() 或者 get() 总是会有这个 class 的前缀出现???
helloworld000 · 2020-08-28 22:55:28 +08:00 · 1134 次点击这是一个创建于 1549 天前的主题，其中的信息可能已经有所发展或是发生改变。
比如 stackoverflow 的 这个 userpage https://stackoverflow.com/users/542251/liam
我要得到用户的自我介绍这个 class 里的内容
response.xpath("//div[@class='grid--cell mt16 s-prose profile-user--bio ']").get()
最后的结果(如下)
'<div class="grid--cell mt16 s-prose profile-user--bio ">\r\n<p>Father, Husband, Rock Climber and Developer with 20 years experience in the industry (in that order).</p>\n\n<hr>\n\n<p><strong>Any fool can write code that a computer can understand. Good programmers write code that humans can understand.</strong></p>\n\n<p><a href="https://en.wikiquote.org/wiki/Martin_Fowler" rel="nofollow noreferrer">Martin Fowler (2008)</a></p>\n\n<hr>\n\n<p>Programmers waste enormous amounts of time thinking about, or worrying about, the speed of noncritical parts of their programs, and these attempts at efficiency actually have a strong negative impact when debugging and maintenance are considered. <strong>We should forget about small efficiencies, say about 97% of the time: premature optimization is the root of all evil</strong>. Yet we should not pass up our opportunities in that critical 3%.</p>\n\n<p><a href="https://en.wikiquote.org/wiki/Donald_Knuth" rel="nofollow noreferrer">Donald Ervin Knuth (Professor Emeritus at Stanford University, and winner of the 1974 Turing Award)</a></p>\n\n<hr>\n </div>'
怎么前面和后面还是有这个<div class="grid--cell mt16 s-prose profile-user--bio ">?
难道这个前缀不应该没了吗? 如果我在后面加了个 text()的话就只剩下 ( '\r\n' ) 了...
scrapy 里的 xpath 不能像 lxml 里的 html 一样直接取 text_content()吗?
 |
1
crella 2020-08-29 11:36:01 +08:00 via Android
看看有没有.innerHtml 的函数？不懂 python
|
 |
2
Kobayashi 2020-08-29 13:30:13 +08:00 via Android
不能。把标签内所有内容拼接起来一次返回，目前只有 lxml.html 和 bs4 支持，后者其实也是封装前者。
scrapy 内置解析器是 parsel，也是依赖 lxml 。先取出这个 div HTML 代码，临时引入 lxml.html 包装为 HTML Element，再调用 text_content() 就可以了。 |
 |
3
Guidance3204 2020-08-29 16:38:27 +08:00
xpath 写的不对，大概应该是 //div[@class="classname"]//p//text(), 数据没在 <div> 下面，在 <p> 里面呢
|
Select Language