{ "version": "https://jsonfeed.org/version/1", "title": "Hadoop", "description": "The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using a simple programming model.", "home_page_url": "https://www.v2ex.com/go/hadoop", "feed_url": "https://www.v2ex.com/feed/hadoop.json", "icon": "https://cdn.v2ex.com/navatar/82aa/4b0a/162_large.png?m=1458164266", "favicon": "https://cdn.v2ex.com/navatar/82aa/4b0a/162_normal.png?m=1458164266", "items": [ { "author": { "url": "https://www.v2ex.com/member/NeroKamin", "name": "NeroKamin", "avatar": "https://cdn.v2ex.com/gravatar/1e5faebe3ace131d79c2faade4b32eb6?s=73&d=retro" }, "url": "https://www.v2ex.com/t/1020336", "date_modified": "2024-03-04T09:29:49+00:00", "content_html": "\u80cc\u666f\uff1a\u65b0\u5165\u804c\u7684\u4e00\u5bb6\u4f20\u7edf\u516c\u53f8\uff0c\u5f00\u53d1\u8fd0\u7ef4\u7684\u6d3b\u90fd\u8981\u81ea\u5df1\u5e72\u3002\u9886\u5bfc\u4e0d\u61c2\u6280\u672f\uff0c\u4eca\u5e74\u5b9a\u4e0b\u7684\u76ee\u6807\u662f\u8981\u5728\u516c\u53f8\u5185\u641e\u4e00\u4e2a\u5927\u6570\u636e\u5e73\u53f0\uff0c\u6307\u5b9a\u8981\u90e8\u7f72\u4e00\u5957 Hadoop \u96c6\u7fa4\uff0c\u8ba9\u6211\u6765\u8d1f\u8d23\u3002\u6211\u611f\u89c9\u6309\u76ee\u524d\u516c\u53f8\u7684\u4e1a\u52a1\u6765\u8bf4\u6839\u672c\u7528\u4e0d\u5230\uff0c\u57fa\u672c\u4e0a\u5c5e\u4e8e\u662f\u7ee9\u6548\u5de5\u7a0b\uff0c\u6211\u81ea\u5df1\u4e5f\u5bf9\u8fd9\u65b9\u9762\u6ca1\u6709\u592a\u591a\u7814\u7a76\u3002\u5f88\u65e9\u4ee5\u524d\u63a5\u89e6\u8fc7\u4e00\u70b9 Hadoop \uff0c\u5370\u8c61\u5c31\u662f\u5b89\u88c5\u914d\u7f6e\u633a\u9ebb\u70e6\u7684\uff0c\u6240\u4ee5\u6211\u60f3\u6765\u8bf7\u6559\u4e0b\u5728\u751f\u4ea7\u73af\u5883\u4e0a\u80fd\u5426\u4f7f\u7528 Docker \u6765\u7b80\u5316 Hadoop \u7684\u5b89\u88c5\u914d\u7f6e\uff0c\u662f\u5426\u6709\u4ec0\u4e48\u5751\uff1f", "date_published": "2024-03-04T01:58:17+00:00", "title": "\u8bf7\u6559\u4e00\u4e0b Hadoop \u5728\u751f\u4ea7\u73af\u5883\u53ef\u4ee5\u901a\u8fc7 Docker \u5b89\u88c5\u5417\uff1f", "id": "https://www.v2ex.com/t/1020336" }, { "author": { "url": "https://www.v2ex.com/member/Arost", "name": "Arost", "avatar": "https://cdn.v2ex.com/avatar/abda/fb03/533684_large.png?m=1625622659" }, "url": "https://www.v2ex.com/t/879031", "title": "hadoop zookeeper", "id": "https://www.v2ex.com/t/879031", "date_published": "2022-09-09T15:15:45+00:00", "content_html": "

\u4e3a\u5565\u5728 hadoop \u548c zookeeper \u91cc\u9762\u7ecf\u5e38\u4f1a\u51fa\u73b0\u914d\u7f6e\u6587\u4ef6\u4e2d\u7684 hostname \u8bc6\u522b\u4e0d\u51fa\u6765\uff0c\u5fc5\u987b\u6539\u6210 0.0.0.0 \u624d\u884c\uff1f

\n" }, { "author": { "url": "https://www.v2ex.com/member/zywoo", "name": "zywoo", "avatar": "https://cdn.v2ex.com/avatar/39b9/db74/442552_large.png?m=1686055178" }, "url": "https://www.v2ex.com/t/876050", "title": "\u83dc\u9e21\u521d\u5b66\u8005\u5bf9 Hadoop \u4e91\u7aef\u67b6\u6784\u7684\u4e00\u4e9b\u7591\u60d1", "id": "https://www.v2ex.com/t/876050", "date_published": "2022-08-28T15:21:38+00:00", "content_html": "

Java \u5f00\u53d1\u5b9e\u4e60\u751f\uff0c\u4f46\u662f\u5b9e\u4e60\u671f\u95f4\u4e00\u76f4\u5728\u5f04\u4e9b hive \u811a\u672c\u7684\u76f8\u5173\u5de5\u4f5c\uff0c\u7ef4\u62a4\u73b0\u6709\u7684\u62a5\u8868\u3002\n\u4e4b\u524d\u6ca1\u4e86\u89e3\u8fc7\u5927\u6570\u636e\u76f8\u5173\uff0c\u81ea\u5b66\u4e86\u4e24\u5929 Hadoop \u548c Hive \u76f8\u5173\u7684\u4e1c\u897f\uff0c\u6709\u4e00\u4e9b\u7591\u95ee\uff1a\n1 \u3001\u7531\u4e8e\u516c\u53f8\u662f\u6d77\u5916\u4e1a\u52a1\uff0caws \u548c azure \u7528\u7684\u6bd4\u8f83\u591a\uff0c\u6570\u4ed3\u8fd9\u4e00\u5957\u4e1c\u897f\u5168\u90fd\u642d\u5728\u4e91\u4e0a\uff0c\u7528\u7684\u662f aws EMR \u670d\u52a1 + aws S3 \u505a\u5b58\u50a8\u3002\u8fd9\u91cc\u7b2c\u4e00\u4e2a\u95ee\u9898\uff1aS3 \u4f5c\u4e3a\u5b58\u50a8\u670d\u52a1\uff0c\u662f\u548c HDFS \u540c\u7ea7\u522b\u7684\u4e1c\u897f\u5417\uff1f\u662f S3 \u4ece\u6839\u672c\u4e0a\u4ee3\u66ff\u4e86 HDFS \uff0c\u8fd8\u662f\u8bf4 HDFS \u662f\u4e00\u79cd\u6982\u5ff5\u6216\u8005\u89c4\u8303\uff0cS3 \u662f HDFS \u7684\u4e00\u79cd\u5b9e\u73b0\u65b9\u5f0f\uff1f\n2 \u3001\u5982\u679c\u662f\u524d\u8005\uff0c\u90a3\u4e48 HDFS \u67b6\u6784\u4e2d\u7684 NameNode \uff0cDataNode \u8fd9\u4e9b\u6982\u5ff5\uff0c\u5728 S3 \u4e2d\u662f\u5fc5\u8981\u7684\u5417\uff1f S3 \u53c8\u662f\u5426\u6709\u6570\u636e\u5197\u4f59\u7684\u63aa\u65bd\u6765\u4fdd\u8bc1\u6570\u636e\u7684\u9ad8\u53ef\u7528\uff1f

\n

\u521d\u5b66\u8005\u6709\u5f88\u591a\u57fa\u672c\u7684\u6982\u5ff5\u548c\u7406\u5ff5\u8fd8\u6ca1\u6709\u641e\u6e05\u695a\uff0c\u6240\u4ee5\u8fd9\u53ef\u80fd\u662f\u4e00\u4e2a\u975e\u5e38\u8822\u7684\u95ee\u9898\u3002\u6c42\u6709\u7ecf\u9a8c\u7684\u5927\u4f6c\u4eec\u70b9\u62e8\u4e00\u4e8c

\n" }, { "author": { "url": "https://www.v2ex.com/member/pmispig", "name": "pmispig", "avatar": "https://cdn.v2ex.com/avatar/e512/4d49/69239_large.png?m=1406604193" }, "url": "https://www.v2ex.com/t/868999", "date_modified": "2022-07-27T05:41:06+00:00", "content_html": "

CPU 24 \u6838 256G \u5185\u5b58\u7684\u673a\u5668\uff0c\u6709 10 \u5757\u786c\u76d8.
\n\u8bf7\u95ee\u90e8\u7f72\u65b9\u5f0f\u662f\u9009\u62e9\u4ec5\u90e8\u7f72\u4e00\u4e2a datanode \uff0c\u8ba9\u8fd9\u4e2a datanode \u4f7f\u7528 10 \u5757\u78c1\u76d8\uff1f
\n\u8fd8\u662f\u90e8\u7f72 10 \u4e2a datanode \uff0c\u6bcf\u4e2a node \u4ec5\u4f7f\u7528\u4e00\u5757\u78c1\u76d8\uff1f
\n\u8bf7\u95ee\u4e24\u79cd\u59ff\u52bf\u5404\u6709\u4ec0\u4e48\u4f18\u52bf\u548c\u7f3a\u70b9\u5462\uff1f

\n", "date_published": "2022-07-27T05:01:09+00:00", "title": "\u8bf7\u95ee hadoop \u591a\u5757\u78c1\u76d8\u53ea\u90e8\u7f72\u4e00\u4e2a datanode \u597d\u8fd8\u662f\u6bcf\u5757\u78c1\u76d8\u5404\u90e8\u7f72\u4e00\u4e2a datanode", "id": "https://www.v2ex.com/t/868999" }, { "author": { "url": "https://www.v2ex.com/member/komorebi12138", "name": "komorebi12138", "avatar": "https://cdn.v2ex.com/avatar/7494/2e94/546486_large.png?m=1729040762" }, "url": "https://www.v2ex.com/t/848155", "title": "\u6c42\u52a9\uff1a Hadoop \u4e2d\u4f7f\u7528 Hbase \u5f02\u5e38", "id": "https://www.v2ex.com/t/848155", "date_published": "2022-04-20T07:22:20+00:00", "content_html": "

\u4e07\u80fd\u7684 V \u53cb\uff0c\u5728\u4f7f\u7528 Hbase \u4e2d\uff0c\u542f\u52a8\u65e0\u4efb\u4f55\u5f02\u5e38\uff0c\u4f46\u662f\u4f7f\u7528 list \u67e5\u770b\u7684\u65f6\u5019\u62a5\u9519\uff0c\u7f51\u4e0a\u8c37\u6b4c\u7684\u95ee\u9898\u90fd\u4e0d\u884c\u4e0b\u9762\u9644\u4e0a\u622a\u56fe\n\"image.png\"\nhbase-env.sh \u914d\u7f6e\n\"image.png\"\nhbase-site.xml \u914d\u7f6e\n\"image.png\"\njps\n\"image.png\"

\n" }, { "author": { "url": "https://www.v2ex.com/member/zxCoder", "name": "zxCoder", "avatar": "https://cdn.v2ex.com/gravatar/a442bec87d1dc5a6c29869ad5add87ee?s=73&d=retro" }, "url": "https://www.v2ex.com/t/805518", "title": "\u6709\u4e2a\u9009\u4fee\u8bfe\u4f5c\u4e1a\u9700\u8981\u5199\u4e00\u4e2a\u5173\u4e8e hadoop \u6280\u672f\u7684\u8bba\u6587\u7efc\u8ff0\uff0c\u6709\u54ea\u4e9b\u6bd4\u8f83\u91cd\u8981\u7684\u65b9\u9762\u53ef\u4ee5\u5199", "id": "https://www.v2ex.com/t/805518", "date_published": "2021-10-01T00:53:59+00:00", "content_html": "

\u8001\u5e08\u8bf4\u4e0d\u8ba9\u5199\u8bfe\u4e0a\u8bb2\u8fc7\u7684\u90a3\u4e9b\u57fa\u7840

\n" }, { "author": { "url": "https://www.v2ex.com/member/lin0kin", "name": "lin0kin", "avatar": "https://cdn.v2ex.com/avatar/a6c9/3f71/235009_large.png?m=1500100454" }, "url": "https://www.v2ex.com/t/797929", "title": "\u5927\u4f6c\u4eec, \u8bf7\u95ee\u81ea\u5b66 hadoop \u7cfb\u5217, \u6709\u4ec0\u4e48\u6559\u7a0b\u6216\u8005\u8d44\u6599\u5417?", "id": "https://www.v2ex.com/t/797929", "date_published": "2021-08-25T06:54:04+00:00", "content_html": "

\u670b\u53cb\u7ed9\u4e86\u4e00\u672c hadoop \u6743\u5a01\u6307\u5357. \u8fd8\u6709\u522b\u7684\u8d44\u6599\u53ef\u4ee5\u53c2\u8003\u4e00\u4e0b\u5417?

\n" }, { "author": { "url": "https://www.v2ex.com/member/VWWWV", "name": "VWWWV", "avatar": "https://cdn.v2ex.com/avatar/aeb8/1f11/440459_large.png?m=1665194473" }, "url": "https://www.v2ex.com/t/793131", "date_modified": "2021-08-02T06:39:05+00:00", "content_html": "

\u6211\u8fd9\u8fb9\u662f\u51e0\u4e2a\u670d\u52a1\u5668\u7ec4\u7684 hadoop \u96c6\u7fa4\uff0c\u4e3b\u8981\u662f ETL \u6e05\u6d17\u63a8\u9001\uff0c\u4eca\u5929\u7a81\u7136\u63a5\u5230\u4e2a\u8981\u6c42\uff0c\u8bf4\u8981\u51cf\u5c11\u5230\u4e00\u4e2a\u670d\u52a1\u5668\uff0c\u8fd9\u3002\u3002\u3002 \u6709\u54ea\u4e9b\u7406\u7531\u53ef\u4ee5\u548c\u7532\u65b9\u626f\u76ae\u5417\uff1f

\n", "date_published": "2021-08-02T04:30:16+00:00", "title": "\u7a81\u7136\u63a5\u5230\u7532\u65b9\u7684\u4e00\u4e2a\u9700\u6c42\uff0c\u8981\u6c42\u4ece hadoop \u96c6\u7fa4\u56de\u5230\u5355\u673a....", "id": "https://www.v2ex.com/t/793131" }, { "author": { "url": "https://www.v2ex.com/member/Mindzy", "name": "Mindzy", "avatar": "https://cdn.v2ex.com/gravatar/2855d936df9f4465c016a41d886432e1?s=73&d=retro" }, "url": "https://www.v2ex.com/t/774167", "title": "\u6709\u4ec0\u4e48\u90e8\u7f72 Hadoop \u7684\u4e3b\u6d41\u65b9\u6cd5\u4e48", "id": "https://www.v2ex.com/t/774167", "date_published": "2021-04-29T09:48:34+00:00", "content_html": "

CDH \u4ece\u4eca\u5e74\u5f00\u59cb\u4e0d\u518d\u514d\u8d39\uff0c\u4f46\u597d\u50cf\u80fd\u4e0b\u8f7d 6.3.3 \u53ca\u4ee5\u524d\u7248\u672c\n\u73b0\u5728\u90e8\u7f72 hadoop \u3001hive \u3001Spark \u6709\u4ec0\u4e48\u4e3b\u6d41\u65b9\u6cd5\u4e48\n\u4e4b\u524d\u770b\u5e16\u5b50\u6709 hadoop on k8s\n\u4f46\u67e5\u4e86\u8d44\u6599\u8c8c\u4f3c\u4e0d\u662f\u5b98\u65b9\u652f\u6301\u7684\u65b9\u6cd5\uff0cHDFS \u548c k8s \u8c8c\u4f3c\u4e5f\u4e0d\u517c\u5bb9

\n

\u96be\u9053\u8981\u81ea\u5df1\u4e0b Apache \u7684\u53d1\u884c\u7248\uff0c\u7eaf\u624b\u5de5\u90e8\u7f72\u4e48

\n" }, { "author": { "url": "https://www.v2ex.com/member/txc106", "name": "txc106", "avatar": "https://cdn.v2ex.com/gravatar/86d3bfbb9d99ba9123dbe72f04d3d732?s=73&d=retro" }, "url": "https://www.v2ex.com/t/768067", "title": "\u722c\u866b\u6bcf\u5929\u4f20\u4e00\u6b21\u5230 hdfs \u80fd\u7528 flume \u5417\uff1f", "id": "https://www.v2ex.com/t/768067", "date_published": "2021-04-05T03:33:22+00:00", "content_html": "

\u722c\u866b\u6bcf\u5929\u4f20\u4e00\u6b21\u5230 hdfs \u80fd\u7528 flume \u5417\uff1f\u5df2\u7ecf\u914d\u597d\u4e86 flume \u4e3a\u4e86\u8bba\u6587\u591a\u70b9\u5185\u5bb9\u4e5f\u60f3\u5c3d\u91cf\u7528\u4e0a \u4f46\u53d1\u73b0 flume \u597d\u50cf\u8bbe\u6eda\u52a8\u7684\u8bdd\u4f1a\u4e00\u76f4\u4ea7\u751f\u65b0\u6587\u4ef6 \u8bbe\u6eda\u52a8\u65f6\u95f4\u957f\u5728\u65f6\u95f4\u5230\u4e4b\u524d\u597d\u50cf\u53c8\u662f\u4e00\u76f4\u662f tmp \u72b6\u6001 \u6709\u529e\u6cd5\u53ef\u4ee5\u4f7f csv \u6587\u4ef6\u6bcf\u5929\u901a\u8fc7 flume \u4f20\u4e00\u6b21\u5230 hdfs \u5417\uff1f\u8fd8\u662f\u53ea\u80fd\u8bbe\u7f6e\u5b9a\u65f6\u4efb\u52a1\u5230\u65f6\u95f4 put \u4e00\u6b21\u5462\uff1f\u611f\u8c22\u89e3\u7b54

\n" }, { "author": { "url": "https://www.v2ex.com/member/xiaopanzi", "name": "xiaopanzi", "avatar": "https://cdn.v2ex.com/avatar/a2a3/d5b8/206459_large.png?m=1699876192" }, "url": "https://www.v2ex.com/t/763624", "title": "\u5173\u4e8e Hadoop3 \u7684 HADOOP_MAPRED_HOME", "id": "https://www.v2ex.com/t/763624", "date_published": "2021-03-21T04:14:19+00:00", "content_html": "

\u6211\u6bd4\u8f83\u4e86 Hadoop 2 \u548c Hadoop 3 \u7684\u5173\u4e8e\u4f2a\u5206\u5e03\u5f0f\u7684\u90e8\u7f72\u6587\u6863\uff0c\u5176\u4e2d\u4e00\u4e2a\u533a\u522b\u662f Hadoop 3 \u8981\u6c42\u8bbe\u7f6emapreduce.application.classpath\uff0c\u800c\u8be5\u503c\u4e2d\u4f7f\u7528\u4e86\u73af\u5883\u53d8\u91cfHADOOP_MAPRED_HOME\u3002

\n

\u5b9e\u9645\u4e0a\uff0c\u6211\u5728\u6ca1\u6709\u624b\u52a8\u8bbe\u7f6eHADOOP_MAPRED_HOME\u7684\u60c5\u51b5\u4e0b\uff0c\u91cc\u9762\u7684 example \u8fd8\u662f\u80fd\u591f\u6b63\u5e38\u8fd0\u884c\u3002\u6240\u4ee5 Hadoop \u5185\u90e8\u662f\u5982\u4f55\u5f97\u5230HADOOP_MAPRED_HOME\u7684\u9ed8\u8ba4\u503c\u7684\uff1f

\n

\u6b64\u5916\uff0c\u5373\u4f7f\u662f\u5728 Hadoop 2 \u4e2d\uff0c\u914d\u7f6e\u6587\u4ef6\u91cc\u9762\u4e5f\u6709mapreduce.application.classpath\u8fd9\u4e2a\u5c5e\u6027\uff0c\u53ea\u662f\u6587\u6863\u91cc\u9762\u6ca1\u6709\u8ba9\u8bbe\u7f6e\u3002\u90a3\u4e48\uff0c\u7a76\u7adf\u662f\u4ec0\u4e48\u573a\u666f\u4e0b\u9700\u8981\u7528\u5230\u8fd9\u4e2a\u503c\uff1f

\n" }, { "author": { "url": "https://www.v2ex.com/member/foxyier", "name": "foxyier", "avatar": "https://cdn.v2ex.com/gravatar/1f81470bfe2e5790748810e44269bbc9?s=73&d=retro" }, "url": "https://www.v2ex.com/t/748200", "title": "\u8bf7\u6559\u4e00\u4e0b\u5b58\u50a8\u7684\u95ee\u9898", "id": "https://www.v2ex.com/t/748200", "date_published": "2021-01-25T08:17:26+00:00", "content_html": "\u76ee\u524d\u6709\u9700\u6c42\u573a\u666f\uff1a
\u5199\u5165 100w+\u6570\u636e\u91cf\u6bcf\u5929\uff0c \u573a\u666f\u4e3a\u79bb\u7ebf\uff0c \u4e14\u5199\u591a\u8bfb\u5c11\uff0c \u76ee\u524d\u7528\u7684\u662f hbase+hive

\u9047\u5230\u7684\u95ee\u9898\uff1a
hbase \u7684\u670d\u52a1\u7aef rpc \u9891\u7e41\u8d85\u65f6\uff0c \u5bfc\u81f4\u6570\u636e\u5806\u79ef

\u5df2\u91c7\u7528\u7684\u65b9\u6848:
\u4e0e\u670d\u52a1\u7aef\u6c9f\u901a\uff0c \u589e\u52a0 hbase \u79bb\u7ebf\u96c6\u7fa4\uff0c\u589e\u52a0\u670d\u52a1\u7aef\u6570\u91cf\uff08\u7ed3\u679c\uff1a \u8f7b\u5ea6\u7f13\u89e3\u8d85\u65f6\u95ee\u9898\uff0c \u4f46\u4ecd\u672a\u89e3\u51b3\u95ee\u9898\uff09

\u5f85\u9009\u65b9\u6848:
1. \u66f4\u6362\u5b58\u50a8\uff0c \u6539\u4e3a mongodb \u6216\u8005\u5176\u4ed6\u5b58\u50a8
2. \u76f4\u63a5\u628a\u6570\u636e\u5199\u5165\u5230 hive \u8868\uff08\u8fd9\u91cc\u60f3\u8bf7\u6559\u4e00\u4e0b\u4e24\u4e2a\u95ee\u9898\uff1a1. hive \u8868\u4e2d\u5916\u90e8\u8868\u548c\u5185\u90e8\u8868\u5728\u8bfb\u5199\u6027\u80fd\u65b9\u9762\u662f\u5426\u6709\u533a\u522b\uff0c2. hive \u8868\u7684\u5199\u5165\u6548\u7387\u5982\u4f55\uff0c \u662f\u5426\u80fd\u652f\u6301 100w/\u5929\u7684\u5199\u5165\u6548\u7387\uff09

\u60f3\u8bf7\u6559\u4e00\u4e0b\u5404\u4f4d\u5927\u5927\uff0c \u5f85\u9009\u65b9\u6848\u662f\u5426\u53ef\u884c\uff0c \u4ee5\u53ca\u662f\u5426\u6709\u66f4\u597d\u7684\u89e3\u51b3\u65b9\u6848" }, { "author": { "url": "https://www.v2ex.com/member/MPAmber", "name": "MPAmber", "avatar": "https://cdn.v2ex.com/gravatar/b768ac0e6da086ec5adafb392a91c359?s=73&d=retro" }, "url": "https://www.v2ex.com/t/735711", "date_modified": "2020-12-15T08:35:19+00:00", "content_html": "\u52a0\u5165 PayPal \u662f\u4ec0\u4e48\u6837\u7684\u4f53\u9a8c\uff1f

\u2022 \u8f7b\u677e\u6109\u5feb\u7684\u5de5\u4f5c\u6c1b\u56f4\uff0c\u4f17\u591a\u4f18\u79c0\u4ee5\u53ca nice \u7684\u5c0f\u4f19\u4f34 \u2022 \u8d85\u957f\u7684\u5047\u671f \u2014\u2014 \u6bcf\u5e74 15 \u5929\u5e74\u5047\u8d77\uff0c\u6bcf\u5de5\u4f5c\u6ee1 1 \u5e74\u52a0\u4e00\u5929\uff08\u6700\u591a 20 \u5929\uff09\uff1b\u9664\u6b64\u4ee5\u5916\uff0c\u5de5\u4f5c\u6ee1 5 \u5e74\u8fd8\u5c06\u83b7\u5f97\u989d\u5916\u7684 4 \u5468\u5e26\u85aa\u5047\u671f\uff01\u4ee5\u53ca 15 \u5929\u5e26\u85aa\u75c5\u5047\u7b49\u7b49\u7b49\u3002 \u2022 \u5305\u5bb9\u3001\u591a\u5143\u4ee5\u53ca\u56fd\u9645\u5316\u7684\u516c\u53f8\u6587\u5316 \u2022 \u5e02\u503c 2000 \u4ebf\u7f8e\u5143\uff0c\u5168\u6c11\u6301\u80a1\uff0c\u80a1\u7968\u6fc0\u52b1\uff0c\u85aa\u916c\u4e0d\u8f93 996 \u516c\u53f8\u7684\u201c\u798f\u62a5\u201d\u3002work-life balance\uff0c\u751f\u6d3b\u4e0d\u5e94\u53ea\u6709\u52a0\u73ed\u3002

PayPal \u7684\u5927\u6570\u636e\u7ec4\u4f1a\u505a\u4ec0\u4e48\uff1f

At PayPal Global Data Science(GDS) team, we develop machine learning platform and AI applications to improve PayPal\u2019s global business. Machine learning and AI is one of the core competitive advantage of PayPal, which significantly reduced payment risk loss, brought million dollars\u2019 revenue and expanded to multiple domains rapidly. As an engineer in GDS, you will work closely with analytical team, understand the requirement with cutting-edge algorithm, contribute to the core platform, make the research work to a real product. We are looking for strong technologists who are passionate to solve machine learning problems and able to continuously deliver AI solutions in scalable way.

\u8fd9\u4e2a\u804c\u4f4d\u7684\u8981\u6c42\u662f\u4ec0\u4e48\uff1f

Qualifications \u2022 BS, MS, or PhD in Computer Science or related technical discipline (or equivalent). \u2022 8+ years\u2019 work experience in software development area with at least 5+ years\u2019 experience in Java programming. \u2022 Excellent understanding of computer science fundamentals, data structures, and algorithms. \u2022 Excellent problem solving skills, can triage and resolve critical tech issues without supervision. \u2022 Expertise required in object-oriented design methodology and application development in Java. \u2022 Experience in big data technology such as Hadoop/Spark/Pig/HBASE/Streaming \u2022 Mastering at least one scripting language such as Unix Shell/Python/Perl/JS \u2022 Hands on web application development skill (HTML5/CSS/JS) is a very big plus \u2022 Knowledge on Machine Learning application pipeline is a very big plus \u2022 Proven results oriented person with a delivery focus in a high velocity, high quality environment. \u2022 Strong communication skills in Oral and Written English. \u2022 Working Experience in Multi-national Company is a plus. \u2022 Geek style is a big plus.

\u8fd8\u6709\u9644\u52a0\uff01\uff01\uff01\uff01\uff01\uff01\uff01\uff01\uff1a

\u5982\u679c\u6709\u5174\u8da3\u7684\u5c0f\u4f19\u4f34\uff0c\u540c\u65f6\u53c8\u6709\u70b9\u62c5\u5fc3\u5e74\u7ec8\u5956\u7684\u635f\u5931\u7684\u8bdd\u3002 \u4e0d\u7528\u62c5\u5fc3\uff01\u57fa\u4e8e\u9762\u8bd5\u60c5\u51b5\uff0cPayPal \u4f1a\u8003\u8651\u7ed9\u4e88\u989d\u5916\u5956\u91d1\u6216\u8005 Sign-On Bonus \u4e4b\u7c7b\u51cf\u5c11\u4f60\u7684\u5e74\u7ec8\u5956\u635f\u5931\uff0c \u8ba9\u4f60\u5728\u5e74\u5e95\u65e2\u80fd\u8f7b\u677e\u641e\u5b9a\u65b0\u7684 Exciting \u7684\u5de5\u4f5c\u673a\u4f1a\uff0c\u53c8\u80fd\u5373\u65f6\u5f97\u5230\u4e00\u4e9b\u8865\u507f\uff0c\u4f55\u4e50\u4e0d\u4e3a\u5462\uff1f", "date_published": "2020-12-15T08:33:52+00:00", "title": "PayPal \u4e0a\u6d77\u62db\u5927\u6570\u636e\u7814\u53d1\u5de5\u7a0b\u5e08\u5566\uff01", "id": "https://www.v2ex.com/t/735711" }, { "author": { "url": "https://www.v2ex.com/member/hzt1234h", "name": "hzt1234h", "avatar": "https://cdn.v2ex.com/avatar/9c84/b0ae/353895_large.png?m=1596510307" }, "url": "https://www.v2ex.com/t/726400", "title": "\u5927\u6587\u4ef6\u901a\u8fc7 Mapreduce \u5207\u5206\u5b58\u5165 hbase", "id": "https://www.v2ex.com/t/726400", "date_published": "2020-11-17T13:31:40+00:00", "content_html": "

\u5927\u6587\u4ef6\u901a\u8fc7 Mapreduce \u5207\u5206\u6210\u591a\u4e2a 5mb \u7684\u5c0f\u6587\u4ef6\u5e76\u5b58\u5165 hbase \u8fd9\u79cd\u64cd\u4f5c\u5982\u4f55\u5b9e\u73b0\u554a

\n" }, { "author": { "url": "https://www.v2ex.com/member/ysn2233", "name": "ysn2233", "avatar": "https://cdn.v2ex.com/gravatar/ecc2f2d57e316a59784430ba269b2da5?s=73&d=retro" }, "url": "https://www.v2ex.com/t/700363", "title": "Hadoop \u8bfb\u53d6 10 \u4e07\u4e2a\u6587\u4ef6\u521d\u59cb\u5316\u8fc7\u7a0b\u5f88\u6162\u6709\u65b9\u4fbf\u7684\u89e3\u51b3\u529e\u6cd5\u4e48\uff1f", "id": "https://www.v2ex.com/t/700363", "date_published": "2020-08-21T12:56:40+00:00", "content_html": "

\u6211\u662f\u7528 Flink \u7684 DataSet API \u53bb\u8bfb\u53d6\u6587\u4ef6\uff0c\u7528\u7684 Hadoop Compatability \u91cc Hadoop \u7684 FileInputFormat\uff0c\u53d1\u73b0\u5728\u521b\u5efa\u5206\u7247\u7684 getSplits()\u65b9\u6cd5\u91cc\uff0c\u4f1a\u53bb\u904d\u5386\u6bcf\u4e00\u4e2a\u6587\u4ef6\u83b7\u53d6 block locations\uff0c\u6709 10 \u4e07\u4e2a\u6587\u4ef6\u7684\u8bdd\u5c31\u5f97\u53d1 10 \u4e07\u6b21\u8bf7\u6c42\uff0c\u6240\u4ee5\u4efb\u52a1\u521d\u59cb\u5316\u975e\u5e38\u6162\uff0c\u8bf7\u95ee\u4e00\u4e0b\u6709\u4ec0\u4e48\u597d\u7684\u89e3\u51b3\u529e\u6cd5\u4e48\uff1f

\n" }, { "author": { "url": "https://www.v2ex.com/member/yellowmarlboro", "name": "yellowmarlboro", "avatar": "https://cdn.v2ex.com/gravatar/1fef4d523494279913e63b5fccaac887?s=73&d=retro" }, "url": "https://www.v2ex.com/t/675347", "title": "\u516c\u53f8\u8981\u505a\u4e2a\u6570\u4ed3\uff0c\u5355\u673a hadoop \u662f\u5426\u6ca1\u4ec0\u4e48\u610f\u4e49\uff0c\u4f46\u662f\u53c8\u60f3\u81ea\u5df1\u52a8\u624b\u505a\u4e00\u4e0b\u3002\u6c42\u7ed9\u5efa\u8bae\uff01", "id": "https://www.v2ex.com/t/675347", "date_published": "2020-05-25T10:10:11+00:00", "content_html": "

\u80cc\u666f

\n\n

\u9700\u6c42

\n\n

\u60f3\u6cd5\u548c\u95ee\u9898

\n\n

\u603b\u4e4b\u5c31\u662f\u60f3\u7528\u8fd9\u4e2a\u673a\u4f1a\u5b66\u4e60\u52a8\u624b\u70b9\u65b0\u4e1c\u897f\u800c\u4e14\u505a\u597d\uff0c\u4f46\u662f\u8fd9\u4e2a\u67b6\u6784\u4ec0\u4e48\u7684\u5bf9\u6211\u6765\u8bf4\u8fd8\u662f\u6bd4\u8f83\u96be\uff0c\u6240\u4ee5\u60f3\u5f97\u5230\u5404\u4f4d\u5927\u4f6c\u7684\u5efa\u8bae

\n" }, { "author": { "url": "https://www.v2ex.com/member/zdkk", "name": "zdkk", "avatar": "https://cdn.v2ex.com/avatar/6d27/ed88/337630_large.png?m=1759663563" }, "url": "https://www.v2ex.com/t/654917", "title": "\u6c42\u52a9\uff1a hadoop \u7684 wordcount \u4e00\u76f4\u662f\u6302\u8d77\u72b6\u6001 ACCEPTED: waiting for AM container to be allocated, launched and register with RM.", "id": "https://www.v2ex.com/t/654917", "date_published": "2020-03-21T10:40:12+00:00", "content_html": "

\u642d\u5efa\u5355\u8282\u70b9\u4f2a\u5206\u5e03\u5f0f\u7cfb\u7edf\u65f6\uff0c\u914d\u7f6e\u8fd0\u884c\u65b9\u5f0f\u4e3a yarn \uff08\u6539\u6210 local \u65f6\u8fd0\u884c wordcount \u6b63\u5e38\uff09\u65f6\uff0c\u4e00\u76f4\u65e0\u54cd\u5e94\uff0c\u88ab\u6302\u8d77\u3002\n\u770b\u4e86\u5f88\u591a\u5e16\u5b50\uff0c\u5df2\u68c0\u67e5 hosts \u6b63\u5e38\u3002\u914d\u7f6e\u6587\u4ef6\u7b49\u90fd\u6b63\u5e38

\n

\u4f46\u662f\u7a0b\u5e8f\u4f9d\u65e7\u4e0d\u80fd\u6b63\u5e38\u8fd0\u884c\u3002

\n

\u95ee\u9898\u8be6\u7ec6\uff1a https://bbs.csdn.net/topics/396194270?page=1#post-411038079

\n" }, { "author": { "url": "https://www.v2ex.com/member/ysn2233", "name": "ysn2233", "avatar": "https://cdn.v2ex.com/gravatar/ecc2f2d57e316a59784430ba269b2da5?s=73&d=retro" }, "url": "https://www.v2ex.com/t/652079", "title": "\u5927\u6570\u636e\u73af\u5883\u4e2d\u538b\u7f29\u683c\u5f0f\u7528\u4ec0\u4e48\u6bd4\u8f83\u597d\uff1f", "id": "https://www.v2ex.com/t/652079", "date_published": "2020-03-12T02:08:03+00:00", "content_html": "

\u56e0\u4e3a\u6587\u4ef6\u5927\u5c0f\u53ef\u80fd\u4e0d\u4e00\uff0c\u9700\u8981\u652f\u6301 splittable \u7684\uff0c\u76ee\u524d\u770b\u5230\u7684\u8c8c\u4f3c\u6709 Bzip2 \u548c lzo \uff08\u9700\u8981\u5efa\u7d22\u5f15\uff09\uff0c\u54ea\u4e2a\u76f8\u5bf9\u6bd4\u8f83\u597d\u7528\uff1f

\n" }, { "author": { "url": "https://www.v2ex.com/member/ysn2233", "name": "ysn2233", "avatar": "https://cdn.v2ex.com/gravatar/ecc2f2d57e316a59784430ba269b2da5?s=73&d=retro" }, "url": "https://www.v2ex.com/t/651838", "title": "\u6709\u4ec0\u4e48\u597d\u7684\u529e\u6cd5\u53ef\u4ee5\u7528 Flink/Spark \u9ad8\u6548\u7387\u5e76\u884c\u5904\u7406\u5927\u91cf\u5927\u5c0f\u4e0d\u4e00\u7684\u538b\u7f29\u6570\u636e", "id": "https://www.v2ex.com/t/651838", "date_published": "2020-03-11T05:30:18+00:00", "content_html": "

\u6570\u636e\u683c\u5f0f\u90fd\u662f gzip \u538b\u7f29\uff0c\u90fd\u6ca1\u6cd5\u5207\u5206\uff0c\u53ea\u80fd\u4e00\u4e2a\u7ebf\u7a0b\u8bfb\u4e00\u4e2a\u6587\u4ef6\uff0c\u5f88\u591a\u65f6\u5019\u5c0f\u6587\u4ef6\u65e9\u5c31\u5904\u7406\u5b8c\u4e86\uff0c\u4f46\u5927\u6587\u4ef6\u4f1a\u975e\u5e38\u6162\u3002\u6709\u6ca1\u6709\u4ec0\u4e48\u597d\u7684\u529e\u6cd5\u53ef\u4ee5\u8ba9 gzip \u53d8\u5f97 splittable

\n" }, { "author": { "url": "https://www.v2ex.com/member/cirton", "name": "cirton", "avatar": "https://cdn.v2ex.com/avatar/28a7/f1a2/308039_large.png?m=1539244114" }, "url": "https://www.v2ex.com/t/631916", "title": "\u8bf7\u6559\u4ece\u5916\u90e8\u8bbf\u95ee\u7528 docker \u642d\u5efa\u7684 hadoop \u96c6\u7fa4\u7684\u95ee\u9898\uff1f", "id": "https://www.v2ex.com/t/631916", "date_published": "2019-12-24T08:40:26+00:00", "content_html": "

\u7528 docker \u642d\u5efa\u7684 hadoop \u6d4b\u8bd5\u73af\u5883\uff0c\u73b0\u5728\u60f3\u4ece\u5916\u90e8\u8bbf\u95ee\u96c6\u7fa4\uff0c\u6bd4\u5982 \u60f3\u4ece\u96c6\u7fa4\u5916\u8bbf\u95ee hbase \u6570\u636e\u5e93\uff0c\u8fd9\u65f6\u9700\u8981\u628a hadoop \u96c6\u7fa4\u4e2d\u914d\u7f6e\u6587\u4ef6(yarn-site.xml, hbase-site.xml \u7b49)\u62f7\u8d1d\u5230\u672c\u5730\u9879\u76ee\u4e2d\uff1b

\n

\u95ee\u9898\u662f\uff1a\u62f7\u8d1d\u8fc7\u6765\u7684\u914d\u7f6e\u6587\u4ef6\uff0c\u91cc\u9762\u6d89\u53ca\u5230\u7aef\u53e3\u7684\u5185\u5bb9\uff08\u5bf9\u5916\u7684 web \u7aef\u53e3\u548c\u5185\u90e8\u901a\u4fe1\u7aef\u53e3\uff09\u9700\u4e0d\u9700\u8981\u8c03\u6574\uff1f\u8fd8\u662f\u4fdd\u6301\u539f\u6765\u7684\u914d\u7f6e\uff1f

\n" }, { "author": { "url": "https://www.v2ex.com/member/hiouyuu", "name": "hiouyuu", "avatar": "https://cdn.v2ex.com/gravatar/4301e0589071203fd07b5789f18f9bdd?s=73&d=retro" }, "url": "https://www.v2ex.com/t/604646", "title": "\u8bf7\u6559\uff1a hadoop \u4e0d\u4f1a\u5c06\u5927\u8868\u683c\u5206\u5272\u4e3a block", "id": "https://www.v2ex.com/t/604646", "date_published": "2019-09-27T01:02:45+00:00", "content_html": "\u4e0d\u662f dba\uff0c\u53ea\u662f\u7528\u6237\u3002

\u6211\u4e4b\u524d\u4f7f\u7528\u7684\u96c6\u7fa4\uff0c\u5728 sqoop \u5bfc\u5165\u3001create\u3001insert \u7b49\u64cd\u4f5c\uff0c\u90fd\u4f1a\u5c06\u5927\u7684\u8868\u683c\u5206\u5272\u4e3a 100-200M \u7684\u5c0f\u6587\u4ef6\u3002\u4f46\u76ee\u524d\u6240\u7528\u7684 hadoop \u96c6\u7fa4\u4e0a\u9762\u64cd\u4f5c\u65f6\uff0c\u6c38\u8fdc\u662f\u5728\u539f\u6587\u4ef6\u4e0a append\uff0c\u5bfc\u81f4\u6587\u4ef6\u8d8a\u6765\u8d8a\u5927\uff0chdfs dfs ls \u67e5\u770b\u5230\u8868\u683c\u4ec5\u7531\u5355\u4e2a\u6587\u4ef6\u7ec4\u6210\uff0c\u4e2a\u522b\u8868\u683c\u6587\u4ef6\u5927\u5c0f\u751a\u81f3\u53ef\u80fd\u8d85\u8fc7\u6570\u5341 GB\u3002

\u6211\u4e2a\u4eba\u4e5f\u4e0d\u77e5\u9053\u662f\u5426\u56e0\u4e3a\u8fd9\u4e2a\u539f\u56e0\u5bfc\u81f4\u901f\u5ea6\u975e\u5e38\u5dee\u3002

\u8bf7\u95ee\u662f\u5426\u6709\u914d\u7f6e\u9700\u8981\u4fee\u6539\uff1f\u6709\u6ca1\u6709\u53ef\u80fd\u5bf9\u73b0\u6709\u8868\u683c\u8fdb\u884c\u5206\u5272\uff1f" }, { "author": { "url": "https://www.v2ex.com/member/andj4cn", "name": "andj4cn", "avatar": "https://cdn.v2ex.com/avatar/8ed8/e758/422978_large.png?m=1564553194" }, "url": "https://www.v2ex.com/t/587779", "title": "\u5173\u4e8e\u5206\u5e03\u5f0f\u7cfb\u7edf\u7684\u5b66\u4e60", "id": "https://www.v2ex.com/t/587779", "date_published": "2019-07-31T02:46:42+00:00", "content_html": "

\u6700\u8fd1\u5728\u5b66\u5206\u5e03\u5f0f\uff0c\u6211\u60f3\u95ee\u4e0b\u5927\u5bb6\uff0c\u5206\u5e03\u5f0f\u73af\u5883\u662f\u4f9d\u6258\u4ec0\u4e48\u642d\u5efa\u7684\u3002\u672c\u8eab\u81ea\u5df1\u662f\u4e00\u4e2a\u5b66\u751f\u515a\uff0c\u76ee\u524d\u7684\u51e0\u4e2a\u65b9\u6848\u90fd\u6709\u8fc7\u5c1d\u8bd5\uff0c\u6216\u8005\u51c6\u5907\u5c1d\u8bd5\uff0c\u5982\u4e0b\uff1a

\n
    \n
  1. \u516c\u6709\u4e91\u73af\u5883\u642d\u5efa\uff0c\u4f46\u662f\u8d39\u7528\u6709\u70b9\u8d35\u3002
    2. \n
  2. \u6811\u8393\u6d3e\uff0c\u6709\u4e9b\u5982 redis \u642d\u5efa\u8d77\u6765\u8981\u81f3\u5c11 6 \u4e2a\u8282\u70b9\uff0c\u611f\u89c9\u4e5f\u5f88\u5403\u94b1\u3002\u53e6\u5916\u90fd\u5f97\u4e0a\u6563\u70ed\uff0c\u4e5f\u662f\u4e2a\u95ee\u9898\u3002
    3. \n
  3. Docker\u3002\u76ee\u524d\u5f88\u591a\u5982 Zookeeper\u3001hadoop \u8fd9\u4e9b\u4f3c\u4e4e\u90fd\u6709\u955c\u50cf\u53ef\u4ee5\u4f7f\u7528\uff0c\u4e0d\u8fc7 Docker \u6bd5\u7adf\u4e0d\u662f\u771f\u7684\u8282\u70b9\uff0c\u76ee\u524d\u6b63\u5728\u8e29\u5751\u3002
    4. \n
  4. \u865a\u62df\u673a\u3002\u672c\u5b50\u4e0d\u591f\u597d\uff0c\u5f00\u5f88\u591a\u865a\u62df\u673a\u5185\u5b58\u6d88\u8017\u5403\u4e0d\u8d77\u3002
    5. \n
\n

Java \u540e\u7aef\u5165\u95e8\u5c0f\u751f\uff0c\u5f88\u591a\u7684\u4e2d\u95f4\u4ef6\u3001\u6846\u67b6\u7b49\u90fd\u9700\u8981\u5206\u5e03\u5f0f\u73af\u5883\uff0c\u5e0c\u671b\u5404\u8def\u5927\u795e\u5206\u4eab\u4e0b\u5b9d\u8d35\u610f\u89c1\uff01

\n" }, { "author": { "url": "https://www.v2ex.com/member/yeboo", "name": "yeboo", "avatar": "https://cdn.v2ex.com/gravatar/f7a0ad757ee301806bcb160ca3804724?s=73&d=retro" }, "url": "https://www.v2ex.com/t/578166", "title": "springboot \u6574\u5408 spark, \u96c6\u7fa4\u65b9\u5f0f\u542f\u52a8\u9700\u8981\u600e\u4e48\u914d\u7f6e\u5462\uff1f", "id": "https://www.v2ex.com/t/578166", "date_published": "2019-06-27T16:26:31+00:00", "content_html": "

springboot \u4e2d\u8c03\u7528\u4e86 spark \u65b9\u6cd5\uff0cSparkConf \u4e2d\u4ee5 local[*] \u542f\u52a8\u53ef\u4ee5\u6b63\u5e38\u8fd0\u884c\uff0c\u4f46\u662f\u8bbe\u7f6e\u6210 yarn-client \u6a21\u5f0f\u5c31\u4f1a\u62a5\u9519\uff0c\u9879\u76ee\u662f\u76f4\u63a5\u542f\u52a8\u7684 java -jar\uff0c\u6709\u5927\u4f6c\u8fd9\u6837\u64cd\u4f5c\u8fc7\u5417\uff1f

\n" }, { "author": { "url": "https://www.v2ex.com/member/cirton", "name": "cirton", "avatar": "https://cdn.v2ex.com/avatar/28a7/f1a2/308039_large.png?m=1539244114" }, "url": "https://www.v2ex.com/t/576255", "title": "\u5173\u4e8e zookeeper \u96c6\u7fa4 leader \u8282\u70b9\u7684\u9009\u4e3e\u95ee\u9898?", "id": "https://www.v2ex.com/t/576255", "date_published": "2019-06-21T09:35:11+00:00", "content_html": "

zk \u96c6\u7fa4\u662f\u4f7f\u7528\u7b97\u6cd5\u9009\u4e3e Leader \u8282\u70b9\u7684\u3002

\n

\u4e3a\u4ec0\u4e48\u4e0d\u53ef\u4ee5\u968f\u673a\u6307\u5b9a\u4e00\u4e2a\u6d3b\u8dc3\u8282\u70b9\u4f5c\u4e3a leader \u8282\u70b9\u5462\uff1f\n\u4f7f\u7528\u7b97\u6cd5\u8fdb\u884c\u9009\u4e3e\u6709\u4ec0\u4e48\u597d\u5904\uff1f

\n" }, { "author": { "url": "https://www.v2ex.com/member/bxb100", "name": "bxb100", "avatar": "https://cdn.v2ex.com/avatar/5f39/34ee/80678_large.png?m=1774171047" }, "url": "https://www.v2ex.com/t/553948", "title": "\u5982\u4f55\u4ece afs \u4e0a\u4e0b\u8f7d\u6570\u636e", "id": "https://www.v2ex.com/t/553948", "date_published": "2019-04-11T01:34:25+00:00", "content_html": "

\u4e0d\u4f7f\u7528 hadoop \u5982\u4f55\u4e0b\u8f7d afs \u4e0a\u7684\u6570\u636e

\n" }, { "author": { "url": "https://www.v2ex.com/member/cirton", "name": "cirton", "avatar": "https://cdn.v2ex.com/avatar/28a7/f1a2/308039_large.png?m=1539244114" }, "url": "https://www.v2ex.com/t/551936", "title": "\u5982\u4f55\u4f7f\u7528\u4ee3\u7801\u83b7\u53d6\u4e24\u4e2a Namenode \u7684\u8282\u70b9\u4fe1\u606f\uff1f", "id": "https://www.v2ex.com/t/551936", "date_published": "2019-04-04T02:28:48+00:00", "content_html": "

Hadoop \u63d0\u4f9b\u4e86\u83b7\u53d6 datanode \u7684 host \u548c ip \u7684 api \uff1b\n\u4e5f\u63d0\u4f9b\u4e86\u83b7\u53d6 active namenode \u4fe1\u606f\u7684 api;

\n

\u4f46\u662f\u6ca1\u627e\u5230\u83b7\u53d6\u5168\u90e8 namenode(\u4e3b\u5907\u6a21\u5f0f\uff0c\u4e24\u4e2a namenode)\u4fe1\u606f\u7684\u63a5\u53e3\u3002

\n

\u600e\u6837\u7528\u4ee3\u7801\u83b7\u53d6\u5168\u90e8 namenode \u7684\u8282\u70b9\u4fe1\u606f\u5462\uff1f

\n" }, { "author": { "url": "https://www.v2ex.com/member/KOSKERS", "name": "KOSKERS", "avatar": "https://cdn.v2ex.com/gravatar/f66bd8079b1f4bd57be19c9feeaa0bde?s=73&d=retro" }, "url": "https://www.v2ex.com/t/551826", "title": "SQOOP \u5bfc\u51fa\u6570\u636e\u5230 MYSQL \u62a5\u9519", "id": "https://www.v2ex.com/t/551826", "date_published": "2019-04-03T13:53:59+00:00", "content_html": "

mysql \u8868\u7684\u5b57\u6bb5\u548c hadoop \u91cc\u7684\u8868\u5b57\u6bb5\u4e00\u6837\u3002\u5bc6\u7801\u548c\u7528\u6237\u4e5f\u5bf9\u4f46\u662f\u5c31\u662f\u62a5\u9519\u3002\u5bfc\u51fa\u4e0d\u4e86\u3002\n\u5404\u4f4d\u6709\u9047\u5230\u8fd9\u4e2a\u60c5\u51b5\u5417\uff1f\n\u5173\u952e\u4ece\u8f93\u51fa\u4e0a\u770b\u4e0d\u5230\u8be6\u7ec6\u65e5\u5fd7\u3001\u3002\u3002\u3002\u3002

\n

19/04/03 21:44:35 INFO mapreduce.Job: The url to track the job: http://cdh1:8088/proxy/application_1553483085921_3757/\n19/04/03 21:44:35 INFO mapreduce.Job: Running job: job_1553483085921_3757\n19/04/03 21:44:40 INFO mapreduce.Job: Job job_1553483085921_3757 running in uber mode : false\n19/04/03 21:44:40 INFO mapreduce.Job: map 0% reduce 0%\n19/04/03 21:44:45 INFO mapreduce.Job: map 100% reduce 0%\n19/04/03 21:44:45 INFO mapreduce.Job: Job job_1553483085921_3757 failed with state FAILED due to: Task failed task_1553483085921_3757_m_000003\nJob failed as tasks failed. failedMaps:1 failedReduces:0

\n

19/04/03 21:44:45 INFO mapreduce.Job: Counters: 13\nJob Counters\nFailed map tasks=1\nKilled map tasks=3\nLaunched map tasks=3\nData-local map tasks=2\nRack-local map tasks=1\nTotal time spent by all maps in occupied slots (ms)=40720\nTotal time spent by all reduces in occupied slots (ms)=0\nTotal time spent by all map tasks (ms)=5090\nTotal vcore-milliseconds taken by all map tasks=5090\nTotal megabyte-milliseconds taken by all map tasks=41697280\nMap-Reduce Framework\nCPU time spent (ms)=0\nPhysical memory (bytes) snapshot=0\nVirtual memory (bytes) snapshot=0\n19/04/03 21:44:45 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead\n19/04/03 21:44:45 INFO mapreduce.ExportJobBase: Transferred 0 bytes in 12.0892 seconds (0 bytes/sec)\n19/04/03 21:44:45 INFO mapreduce.ExportJobBase: Exported 0 records.\n19/04/03 21:44:45 ERROR tool.ExportTool: Error during export:\nExport job failed!\nat org.apache.sqoop.mapreduce.ExportJobBase.runExport(ExportJobBase.java:439)\nat org.apache.sqoop.manager.SqlManager.exportTable(SqlManager.java:931)\nat org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:80)\nat org.apache.sqoop.tool.ExportTool.run(ExportTool.java:99)\nat org.apache.sqoop.Sqoop.run(Sqoop.java:147)\nat org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)\nat org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)\nat org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)\nat org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)\nat org.apache.sqoop.Sqoop.main(Sqoop.java:252)

\n" }, { "author": { "url": "https://www.v2ex.com/member/lcdxiangzi", "name": "lcdxiangzi", "avatar": "https://cdn.v2ex.com/avatar/4a13/cfaa/318465_large.png?m=1578628414" }, "url": "https://www.v2ex.com/t/551330", "date_modified": "2019-04-02T09:20:25+00:00", "content_html": "\u7528\u597d\u51e0\u53f0\u673a\u5668\u5c1d\u8bd5\u8fc7\uff0c\u666e\u901a\u7528\u6237\u90fd\u5728\u62a5\u9519\uff0csudo \u4e5f\u4e0d\u884c\u3002\u4e00\u5b9a\u8981 root \u7528\u6237\u76f4\u63a5\u64cd\u4f5c\u624d\u53ef\u4ee5\u5417\uff1f
+======================================================================+
| Error: Unable to find a compatible version of Java on this host,|
| either because JAVA_HOME has not been set or because a |
| compatible version of Java is not installed. |
+----------------------------------------------------------------------+
| Please install either: |
| - a supported version of the Oracle JDK from the Oracle Java web |
| site: |
| > http://www.oracle.com/technetwork/java/javase/index.html < |
| OR |
| - a supported version of the OpenJDK from your OS vendor. Help for |
| some OSes are available at: |
| > http://openjdk.java.net/install/ < |
| |
| Cloudera Manager requires Oracle JDK or OpenJDK 1.8 or later. |
| NOTE: Cloudera Manager will find the Oracle JDK when starting, |
| regardless of whether you installed the JDK using a binary |
| installer or the RPM-based installer. |
+======================================================================+
error: %pre(cloudera-manager-server-6.2.0-968826.el7.x86_64) scriptlet failed, exit status 1
Error in PREIN scriptlet in rpm package cloudera-manager-server-6.2.0-968826.el7.x86_64
\u9a8c\u8bc1\u4e2d : cloudera-manager-agent-6.2.0-968826.e 1/3
\u9a8c\u8bc1\u4e2d : cloudera-manager-daemons-6.2.0-968826 2/3
\u9a8c\u8bc1\u4e2d : cloudera-manager-server-6.2.0-968826. 3/3

\u5df2\u5b89\u88c5:
cloudera-manager-agent.x86_64 0:6.2.0-968826.el7

\u5931\u8d25:
cloudera-manager-daemons.x86_64 0:6.2.0-968826.el7
cloudera-manager-server.x86_64 0:6.2.0-968826.el7



\u53ea\u8981\u662f sudo\uff0c\u5c31\u4f1a\u62a5\u8fd9\u4e2a\u9519\uff0c\u5982\u679c su - root \u540e\uff0c\u5c31\u53ef\u4ee5\u6210\u529f\u3002
\u65b0\u624b\u4e0a\u8def\u6709\u70b9\u61f5\uff0c\u4e00\u822c\u60c5\u51b5\u4e0b root \u7528\u6237\u90fd\u4e0d\u5927\u7528\u7684\u554a\u3002", "date_published": "2019-04-02T09:20:13+00:00", "title": "\u8bf7\u6559\uff0c CDH \u96c6\u7fa4\u7ba1\u7406\u5de5\u5177\uff0c centos \u666e\u901a\u7528\u6237\u53ef\u4ee5\u5b89\u88c5\u5417\uff1f\u8fd8\u662f\u5fc5\u987b\u4f7f\u7528 root \u7528\u6237", "id": "https://www.v2ex.com/t/551330" }, { "author": { "url": "https://www.v2ex.com/member/shallyy", "name": "shallyy", "avatar": "https://cdn.v2ex.com/avatar/14ed/f56c/68129_large.png?m=1505783828" }, "url": "https://www.v2ex.com/t/549801", "title": "\u5173\u4e8e HDFS \u5c0f\u6587\u4ef6\u5f52\u6863\u6709\u4ec0\u4e48\u597d\u529e\u6cd5\u5417\uff1f", "id": "https://www.v2ex.com/t/549801", "date_published": "2019-03-29T02:16:28+00:00", "content_html": "

archive \uff1f

\n" }, { "author": { "url": "https://www.v2ex.com/member/lcdxiangzi", "name": "lcdxiangzi", "avatar": "https://cdn.v2ex.com/avatar/4a13/cfaa/318465_large.png?m=1578628414" }, "url": "https://www.v2ex.com/t/549611", "title": "\u8bf7\u6559 9 \u53f0\u673a\u5668\u7684 hadoop \u96c6\u7fa4\u89d2\u8272\u5206\u914d\u95ee\u9898", "id": "https://www.v2ex.com/t/549611", "date_published": "2019-03-28T09:14:12+00:00", "content_html": "zookeeper\uff0chive\uff0chbase\uff0c\u90fd\u5206\u522b\u5e94\u8be5\u600e\u6837\u5206\u914d\uff1f

\u6211\u672c\u6765\u4ee5\u4e3a zookeeper\u3001hbase \u90fd\u5e94\u8be5\u5168\u96c6\u7fa4\u90e8\u7f72\uff0chive \u662f\u54ea\u4e2a\u673a\u5668\u9700\u8981\u7528\uff0c\u5c31\u88c5\u5728\u54ea\u4e2a\u673a\u5668\u3002

\u4f46\u662f\u770b\u4e86\u51e0\u4e2a\u5e16\u5b50\uff0c\u53d1\u73b0\u6709\u4eba\u662f\u90e8\u5206\u673a\u5668\u90e8\u7f72\u90e8\u5206\u529f\u80fd\u3002

\u65b0\u624b\uff0c\u4e0d\u61c2\uff0c\u8bf7\u5927\u4f6c\u6307\u70b9\uff0c\u8c22\u8c22\u3002

\u4e0a\u9762\u63d0\u5230\u7684\u5e16\u5b50
https://blog.csdn.net/lepton126/article/details/60866664" }, { "author": { "url": "https://www.v2ex.com/member/KOSKERS", "name": "KOSKERS", "avatar": "https://cdn.v2ex.com/gravatar/f66bd8079b1f4bd57be19c9feeaa0bde?s=73&d=retro" }, "url": "https://www.v2ex.com/t/542399", "title": "cloudera agent \u542f\u52a8\u5931\u8d25", "id": "https://www.v2ex.com/t/542399", "date_published": "2019-03-08T02:36:17+00:00", "content_html": "

cloudera 5.14.2 \u7248\u672c\nagent \u542f\u52a8\u4e00\u6bb5\u65f6\u95f4\u4f1a\u4e2d\u65ad\u3002\u770b\u65e5\u5fd7\u663e\u793a

\n

\u4e4b\u524d\u662f\u597d\u7684\u3002\u91cd\u542f\u4e4b\u540e\u7a81\u7136\u4e0d\u884c\u4e86\u3002\u975e\u5e38\u5947\u602a

\n

Error: Another program is already listening on a port that one of our HTTP servers is configured to use. Shut this program down first before starting superv\nisord.

\n

[08/Mar/2019 09:18:07 +0000] 98011 MainThread agent INFO SCM Agent Version: 5.14.2\n[08/Mar/2019 09:18:07 +0000] 98011 MainThread agent INFO Re-using pre-existing directory: /opt/CDH/cm-5.14.2/run/cloudera-scm-agent\n[08/Mar/2019 09:21:21 +0000] 98463 MainThread agent INFO SCM Agent Version: 5.14.2\n[08/Mar/2019 09:21:21 +0000] 98463 MainThread agent INFO Re-using pre-existing directory: /opt/CDH/cm-5.14.2/run/cloudera-scm-agent\n[08/Mar/2019 09:54:53 +0000] 2843 MainThread agent INFO SCM Agent Version: 5.14.2\n[08/Mar/2019 09:54:54 +0000] 2843 MainThread agent INFO Re-using pre-existing directory: /opt/CDH/cm-5.14.2/run/cloudera-scm-agent\n[08/Mar/2019 10:15:19 +0000] 2482 MainThread agent INFO SCM Agent Version: 5.14.2\n[08/Mar/2019 10:15:19 +0000] 2482 MainThread agent INFO Re-using pre-existing directory: /opt/CDH/cm-5.14.2/run/cloudera-scm-agent\n[08/Mar/2019 10:30:21 +0000] 4208 MainThread agent INFO SCM Agent Version: 5.14.2\n[08/Mar/2019 10:30:21 +0000] 4208 MainThread agent INFO Re-using pre-existing directory: /opt/CDH/cm-5.14.2/run/cloudera-scm-agent\n[08/Mar/2019 10:30:29 +0000] 4397 MainThread agent INFO SCM Agent Version: 5.14.2\n[08/Mar/2019 10:30:29 +0000] 4397 MainThread agent INFO Re-using pre-existing directory: /opt/CDH/cm-5.14.2/run/cloudera-scm-agent

\n" }, { "author": { "url": "https://www.v2ex.com/member/lcdxiangzi", "name": "lcdxiangzi", "avatar": "https://cdn.v2ex.com/avatar/4a13/cfaa/318465_large.png?m=1578628414" }, "url": "https://www.v2ex.com/t/534996", "date_modified": "2019-02-14T08:25:59+00:00", "content_html": "

\u65b0\u624b\u4e0a\u8def\uff0c\u8bf7\u6559\u95ee\u9898\u3002

\n

\u81ea\u5df1\u5199\u4e86\u4e00\u4e2a\u7b80\u5355\u7684 MapReduce \u5e94\u7528\u3002\u7528\u6765\u89e3\u6790\u96c6\u7fa4 HDFS \u4e0a\u9762\u7684\u4e00\u4e2a\u6587\u4ef6\u5185\u5bb9\u3002

\n

\u672c\u5730\u5728 idea \u4e2d\u8c03\u8bd5\uff0cmain \u51fd\u6570\u6267\u884c\u662f\u6b63\u5e38\u7684\uff0c\u5f88\u5feb\u53ef\u4ee5\u8dd1\u51fa\u7ed3\u679c\u3002

\n

maven \u6253\u6210 jar \u5305\u540e\uff0c\u672c\u5730 hadoop jar \u6267\u884c\u4e5f\u662f\u6b63\u5e38\u7684\u3002

\n

\u4f46\u662f\u62ff\u5230\u96c6\u7fa4\u4e0a\u9762\uff08\u4e00\u4e2a\u5c0f\u96c6\u7fa4\uff0c\u4e00\u4e3b\u4e8c\u4ec6\uff09\uff0c\u518d\u4f7f\u7528 hadoop jar \u53bb\u8fd0\u884c\u5c31\u4f1a\u4e00\u76f4\u5361\u4f4f\u3002

\n

\u524d\u9762\u51e0\u884c\u65e5\u5fd7\u662f\u6b63\u5e38\u6253\u5370\u7684\uff0c\u4f46\u662f\u5230\u4e86

\n

2019-02-14 16:02:34,991 INFO mapreduce.Job: Running job: job_1542766536312_0001

\n

\u4e4b\u540e\u5c31\u6302\u5728\u90a3\u91cc\u4e86\uff0c\u4e45\u4e45\u6ca1\u6709\u53cd\u5e94\u3002

\n

\u6c42\u6307\u70b9\uff0c\u662f\u6211\u54ea\u91cc\u64cd\u4f5c\u6709\u95ee\u9898\u5417\uff1f\u8fd8\u662f\u53ef\u80fd\u54ea\u91cc\u7684\u95ee\u9898\uff1f

\n

\u591a\u8c22

\n", "date_published": "2019-02-14T08:24:48+00:00", "title": "MapReduce \u5e94\u7528\u5728\u672c\u5730\u8c03\u8bd5\u6b63\u5e38\uff0c\u96c6\u7fa4\u4e0a\u8dd1\u4e0d\u52a8", "id": "https://www.v2ex.com/t/534996" }, { "author": { "url": "https://www.v2ex.com/member/belcher", "name": "belcher", "avatar": "https://cdn.v2ex.com/gravatar/9b98b77192bd44e37af204902b5fb765?s=73&d=retro" }, "url": "https://www.v2ex.com/t/528667", "title": "spark \u7684 shuffle \u7b97\u5b50\u4e00\u5b9a\u4f1a\u4ea7\u751f\u5bbd\u4f9d\u8d56\u5417\uff1f", "id": "https://www.v2ex.com/t/528667", "date_published": "2019-01-19T12:56:29+00:00", "content_html": "

\u4eca\u5929\u670b\u53cb\u629b\u51fa\u4e00\u4e2a\u5f88\u6709\u610f\u601d\u7684\u7ed3\u8bba\uff1a\nshuffle \u4e0d\u4e00\u5b9a\u4ea7\u751f\u5bbd\u4f9d\u8d56\uff0c\u4f46\u662f\u5bbd\u4f9d\u8d56\u4e00\u5b9a\u662f\u7531 shuffle \u4ea7\u751f\u7684\u3002\n\u6211\u60f3\u4e86\u597d\u4e45\uff0c\u6709\u6ca1\u6709\u8fd9\u79cd\u60c5\u51b5\uff0c\u5728\u6781\u7aef\u60c5\u51b5\u4e0b\uff0c\u5c31\u7b97\u4f7f\u7528\u4e86 shuffle \u7b97\u5b50\uff0c\u7236\u5b50 rdd \u7684 partition \u4f9d\u7136\u662f\u4e00\u5bf9\u4e00\u7684\u5173\u7cfb\uff0c\u8fd9\u6837\u5b9e\u9645\u5c31\u6ca1\u6709\u5f62\u6210\u5bbd\u4f9d\u8d56\u867d\u7136\u4f7f\u7528\u4e86 shuffle \u7b97\u5b50\u3002\n\u719f\u6089 spark \u7684\u5927\u4f6c\uff0c\u8c01\u5bf9\u8fd9\u4e00\u5757\u6bd4\u8f83\u719f\uff0c\u4e0d\u541d\u8d50\u6559\uff01

\n" }, { "author": { "url": "https://www.v2ex.com/member/engineer9", "name": "engineer9", "avatar": "https://cdn.v2ex.com/gravatar/e92bc2013eee02c242d7c295737153e0?s=73&d=retro" }, "url": "https://www.v2ex.com/t/527878", "date_modified": "2019-01-17T03:52:38+00:00", "content_html": "

\u597d\u53cb\u5173\u7cfb\uff1a

\n
1 2\n1 3\n1 4\n1 5\n1 6\n1 7\n1 8\n2 3\n3 9\n9 10\n7 9\n2 7\n
\n

\u6211\u60f3\u5b9e\u73b0\u5982\u4e0b\u529f\u80fd\uff0c\u6c42 3 \u8282\u70b9\u4e0e\u5b83\u7684\u4e8c\u8df3\u8282\u70b9\u7684\u597d\u53cb\u4e4b\u95f4\u7684\u5171\u540c\u597d\u53cb\u4e2a\u6570\uff0c\u5b9e\u73b0\u7c7b\u4f3c QQ \u597d\u53cb\u63a8\u8350\u529f\u80fd\uff1a

\n
3 \u548c 5 \u7684\u5171\u540c\u597d\u53cb\u4e2a\u6570\u4e3a\uff1a1\n3 \u548c 10 \u7684\u5171\u540c\u597d\u53cb\u4e2a\u6570\u4e3a\uff1a1\n3 \u548c 6 \u7684\u5171\u540c\u597d\u53cb\u4e2a\u6570\u4e3a\uff1a1\n3 \u548c 7 \u7684\u5171\u540c\u597d\u53cb\u4e2a\u6570\u4e3a\uff1a3\n3 \u548c 8 \u7684\u5171\u540c\u597d\u53cb\u4e2a\u6570\u4e3a\uff1a1\n3 \u548c 4 \u7684\u5171\u540c\u597d\u53cb\u4e2a\u6570\u4e3a\uff1a1\n\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\u2014\n\u6392\u5e8f\u4e4b\u540e\uff0c\u8fd9\u91cc\u53ef\u4ee5\u7ed9 3 \u63a8\u8350 7 \u8ba4\u8bc6\u3002\n
\n
    \n
  1. \u7b80\u5355\u7684\u901a\u8fc7 for \u5faa\u73af(\u5f97\u5230\u597d\u53cb\u8282\u70b9\u7684\u4f1a\u89e6\u53d1 action)\u7b97\u51fa\u8fd9\u6837\u7684\u5173\u7cfb\u5728 spark \u4e2d\u4f30\u8ba1\u4e5f\u6ca1\u4ec0\u4e48\u4f18\u52bf\u5427\uff08\u5343\u4e07\u8282\u70b9,\u4ebf\u6761\u5173\u7cfb\uff09\uff1f\u4f46\u662f\u6c42\u4e8c\u8df3\u8282\u70b9\u901a\u8fc7 graphx \u81ea\u5e26\u7684 api \u8fd8\u662f\u6709\u5f88\u5927\u4f18\u52bf\u7684\u3002
    2. \n
  2. saprk graphx \u53ef\u4ee5\u901a\u8fc7\u5b83\u81ea\u5e26\u7684 pregel API \u76f4\u63a5\u6c42\u51fa\u4e00\u4e2a\u8282\u70b9\u7684\u4e00\u8df3\u5230 N \u8df3\u7684\u8282\u70b9\u3002
    3. \n
  3. neo4j \u4e00\u6761\u8bed\u53e5\u5b9e\u73b0\u5982\u4e0b:
    4. \n
\n
match (n:Person {name:\"A\"})-[:Follow]->(m)<-[:Follow]-(b) with n,b,count(m) as summ where (not (n)-[:Follow]-(b) )   return b,summ\n
\n\n
    \n
  1. https://cloud.tencent.com/developer/article/1030789
    2. \n
  2. https://cloud.tencent.com/developer/article/1005999
    3. \n
\n", "date_published": "2019-01-17T03:51:13+00:00", "title": "spark graphx \u600e\u4e48\u5b9e\u73b0\u7c7b\u4f3c QQ \u597d\u53cb\u63a8\u8350\u529f\u80fd\uff1f neo4j \u4e00\u6761\u8bed\u53e5\u5b9e\u73b0", "id": "https://www.v2ex.com/t/527878" }, { "author": { "url": "https://www.v2ex.com/member/trafficMGR", "name": "trafficMGR", "avatar": "https://cdn.v2ex.com/gravatar/2874e868ef0399a74ad8222e43d60940?s=73&d=retro" }, "url": "https://www.v2ex.com/t/525496", "date_modified": "2019-01-09T12:58:40+00:00", "content_html": "

\u96c6\u7fa4\u89c4\u5212\u548c\u642d\u5efa\u8fc7\u7a0b

\n

\u642d\u5efa\u8fc7\u7a0b\u548c\u914d\u7f6e\u6587\u4ef6\u4e13\u95e8\u7528 VuePress \u8bb0\u5f55\u4e86\u4e00\u4e0b\uff1a\u524d\u5f80\u65e5\u5fd7

\n\n

\u6570\u636e\u5904\u7406

\n

\u4efb\u52a1

\n

\u4f7f\u7528 Hive \u5bfc\u5165\u793e\u4ea4\u7528\u6237\u6570\u636e CSV \u6587\u4ef6\uff0c\u4f7f\u7528 SQL \u585e\u9009\u51fa\u7528\u6237\u8868\u4e2d\u7684\u5171\u540c\u7231\u597d\u3001\u5171\u540c\u597d\u53cb\u3002

\n

Map Reduce \u4f5c\u4e1a\u95ee\u9898

\n

\u5728\u8fd9\u4e2a\u96c6\u7fa4\u4e0a\u505a\u7b80\u5355\u7684 Map \u548c Reduce \u4f5c\u4e1a\u662f\u6781\u5176\u7f13\u6162\u7684\uff0c\u4ece 0% \u8df3\u5230 100%\u53ef\u80fd\u4f1a\u7ecf\u5386\u6570\u5341\u5206\u949f\uff0c\u6548\u7387\u582a\u5fe7\u3002\u7531\u4e8e\u5404\u4e2a\u8282\u70b9\u662f\u540c\u5b66\u81ea\u5df1\u4e70\u7684\uff0c\u4e0d\u80fd\u8d70 VPC \u5185\u7f51\u4e92\u901a\uff0c\u53ea\u597d\u7528\u516c\u7f51\u4e92\u901a\u3002\ud83d\udc48\u6000\u7591\u95ee\u9898\u4f1a\u51fa\u73b0\u5728\u8fd9\u91cc

\n
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1\n2019-01-09 19:56:25,499 Stage-1 map = 0%,  reduce = 0%\nINFO  : Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1\nINFO  : 2019-01-09 19:56:25,499 Stage-1 map = 0%,  reduce = 0%\n2019-01-09 19:57:26,238 Stage-1 map = 0%,  reduce = 0%\nINFO  : 2019-01-09 19:57:26,238 Stage-1 map = 0%,  reduce = 0%\n2019-01-09 19:58:26,818 Stage-1 map = 0%,  reduce = 0%\nINFO  : 2019-01-09 19:58:26,818 Stage-1 map = 0%,  reduce = 0%\n2019-01-09 19:59:27,374 Stage-1 map = 0%,  reduce = 0%\nINFO  : 2019-01-09 19:59:27,374 Stage-1 map = 0%,  reduce = 0%\n2019-01-09 20:00:27,878 Stage-1 map = 0%,  reduce = 0%\n\n...\n\n2019-01-09 20:17:32,700 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 1.69 sec\nINFO  : 2019-01-09 20:17:32,700 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 1.69 sec\n2019-01-09 20:18:33,218 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 1.69 sec\nINFO  : 2019-01-09 20:18:33,218 Stage-1 map = 100%,  reduce = 0%, Cumulative CPU 1.69 sec\n
\n

\u6d4b\u8bd5

\n

\u4f7f\u7528\u76f8\u540c\u7684 SQL \u8bed\u53e5\u6362\u505a\u5728 E-MapReduce \u4e0a\u80fd\u5f88\u5757\u8dd1\u51fa\u7ed3\u679c\uff08\u52a0\u94b1\u4e16\u754c\u53ef\u53ca\uff09\u3002

\n

\u8fd8\u672a\u7ed3\u675f

\n\n", "date_published": "2019-01-09T12:48:52+00:00", "title": "\u81ea\u5efa 5 \u4e2a\u8282\u70b9\u7684 Hadoop \u96c6\u7fa4\uff0c\u4ee5\u53ca\u5b8c\u6210 MapReduce \u4f5c\u4e1a", "id": "https://www.v2ex.com/t/525496" }, { "author": { "url": "https://www.v2ex.com/member/lcdxiangzi", "name": "lcdxiangzi", "avatar": "https://cdn.v2ex.com/avatar/4a13/cfaa/318465_large.png?m=1578628414" }, "url": "https://www.v2ex.com/t/525281", "date_modified": "2019-01-09T02:57:47+00:00", "content_html": "\u6700\u8fd1\u5728\u7ffb hadoop \u6743\u5a01\u6307\u5357\uff0c\u53d1\u73b0\u4e66\u91cc\u9762\u63d0\u5230\u6709\u4e9b property \u7684 kv \u5bf9\u662f\u9488\u5bf9\u4e00\u4e2a job \u7684\uff0c\u6709\u4e9b\u662f\u9488\u5bf9 node \u7684\uff0c\u6709\u4e9b\u662f\u9488\u5bf9 cluster \u5168\u5c40\u7684\u3002
\u4f46\u662f\u6ca1\u6709\u4e00\u4e2a\u6bd4\u8f83\u7cfb\u7edf\u7684\u8bf4\u660e\uff0c\u8bf8\u591a\u914d\u7f6e\u9009\u9879\u7684\u751f\u6548\u8303\u56f4\uff0c\u6216\u8005\u8bf4\u4f5c\u7528\u57df\u3002

\u8bf7\u95ee\u54ea\u91cc\u6709\u7c7b\u4f3c\u7684\u4e66\u6216\u8005\u6587\u7ae0\u53ef\u4ee5\u53c2\u8003\u4e00\u4e0b\u5417\uff1f
\u662f\u901a\u8fc7\u5c06\u914d\u7f6e\u5185\u5bb9\u653e\u5728\u4e0d\u540c\u7684\u914d\u7f6e\u6587\u4ef6\u4e2d\u6765\u5b8c\u6210\u4f5c\u7528\u57df\u7684\u63a7\u5236\uff0c\u8fd8\u662f\u6709\u5176\u4ed6\u7684\u673a\u5236\uff1f

\u65b0\u624b\u4e0a\u8def\uff0c\u8bf7\u591a\u6307\u70b9\uff0c\u591a\u8c22", "date_published": "2019-01-09T02:56:51+00:00", "title": "\u8bf7\u6559 hadoop \u914d\u7f6e\u6587\u4ef6\u4e2d\u914d\u7f6e\u9879\u7684\u4f5c\u7528\u57df", "id": "https://www.v2ex.com/t/525281" }, { "author": { "url": "https://www.v2ex.com/member/cirton", "name": "cirton", "avatar": "https://cdn.v2ex.com/avatar/28a7/f1a2/308039_large.png?m=1539244114" }, "url": "https://www.v2ex.com/t/523715", "title": "yarn \u53ef\u4ee5\u76d1\u63a7 hadoop \u6216\u8005 spark \u7684\u670d\u52a1\u5417\uff1f", "id": "https://www.v2ex.com/t/523715", "date_published": "2019-01-04T01:38:31+00:00", "content_html": "

yarn \u53ef\u4ee5\u76d1\u63a7 hadoop \u7684\u4efb\u52a1\uff0c\u5b83\u53ef\u4ee5\u540c\u65f6\u76d1\u63a7 hadoop \u7684\u670d\u52a1\u5417?\u6bd4\u5982\u8bf4\u8fd0\u884c\u5728 namenode \u8282\u70b9\u4e0a\u7684\u670d\u52a1\u3002

\n" }, { "author": { "url": "https://www.v2ex.com/member/dhairoot", "name": "dhairoot", "avatar": "https://cdn.v2ex.com/avatar/c10b/de50/153036_large.png?m=1734948416" }, "url": "https://www.v2ex.com/t/510945", "title": "\u6709\u5f88\u591a xml \u6587\u4ef6\uff0c\u5982\u4f55\u6309\u7167\u6587\u4ef6\u53bb\u5e76\u884c\uff0c\u800c\u4e0d\u662f\u4e00\u4e2a\u5927\u6587\u4ef6\u5207\u5206\u6210\u5c0f\u5757\u6765\u505a\u5e76\u884c\u8ba1\u7b97\u5462\uff1f", "id": "https://www.v2ex.com/t/510945", "date_published": "2018-11-23T13:36:04+00:00", "content_html": "

\u6709\u5f88\u591a xml \u6587\u4ef6\uff0c\u5982\u4f55\u6309\u7167\u6587\u4ef6\u53bb\u5e76\u884c\uff0c\u800c\u4e0d\u662f\u4e00\u4e2a\u5927\u6587\u4ef6\u5207\u5206\u6210\u5c0f\u5757\u6765\u505a\u5e76\u884c\u8ba1\u7b97\u5462\uff1f\n\u6bd4\u5982\u73b0\u5728\u6211\u6709\u5f88\u591a\u7684 xml \u6587\u4ef6\uff0c\u6211\u60f3\u8981\u505a\u7684\u662f\u5229\u7528 spark \u53bb\u540c\u65f6\u89e3\u6790\u591a\u4e2a xml \u6587\u4ef6\uff0c\u800c\u4e0d\u662f\u4e00\u4e2a xml \u6587\u4ef6\u88ab\u4e14\u5206\u6210\u5c0f\u5757\u53bb\u5904\u7406

\n" }, { "author": { "url": "https://www.v2ex.com/member/dhairoot", "name": "dhairoot", "avatar": "https://cdn.v2ex.com/avatar/c10b/de50/153036_large.png?m=1734948416" }, "url": "https://www.v2ex.com/t/507999", "date_modified": "2018-11-15T01:32:56+00:00", "content_html": "

\u5927\u91cf zip \u538b\u7f29\u5305\uff0c\u6bcf\u4e00\u4e2a\u90fd\u5f88\u5c0f\uff0c10M \u4ee5\u5185\uff0c\u5982\u4f55\u5229\u7528 MapReduce \u5b9e\u73b0\u5feb\u901f\u89e3\u538b\u5462\uff1f

\n", "date_published": "2018-11-15T01:32:11+00:00", "title": "\u5927\u91cf zip \u538b\u7f29\u5305\uff0c\u6bcf\u4e00\u4e2a\u90fd\u5f88\u5c0f\uff0c 10M \u4ee5\u5185\uff0c\u5982\u4f55\u5229\u7528 MapReduce \u5b9e\u73b0\u5feb\u901f\u89e3\u538b\u5462\uff1f", "id": "https://www.v2ex.com/t/507999" }, { "author": { "url": "https://www.v2ex.com/member/PanameraM", "name": "PanameraM", "avatar": "https://cdn.v2ex.com/avatar/73bd/dce1/193938_large.png?m=1553088475" }, "url": "https://www.v2ex.com/t/493382", "title": "\u8bf7\u6559\u5173\u4e8e HDFS DELEGATION TOKEN \u8fc7\u95ee\u9898\u3002", "id": "https://www.v2ex.com/t/493382", "date_published": "2018-09-27T23:02:19+00:00", "content_html": "

\u63cf\u8ff0\uff1aSpark Streaming Long Running Job\uff0c\u4ee5 client \u6a21\u5f0f\u8dd1\u5728 YARN \u96c6\u7fa4\u4e0a\uff0cHDFS \u542f\u7528 HA\uff0c\u6574\u4e2a\u96c6\u7fa4\u542f\u7528 Kerberos \u8ba4\u8bc1\u3002\u5728\u6267\u884c\u4efb\u52a1\u65f6\uff0cApplicationMaster \u53ef\u4ee5\u6b63\u5e38\u91cd\u65b0 renew \u65b0\u7684 HDFS DELEGATION TOKEN\uff0cexecutor \u7aef\u4e5f\u53ef\u4ee5\u6b63\u5e38\u62ff\u5230 AM \u65b0\u751f\u6210\u7684 Token \uff08\u7531 AM \u5199\u5230 HDFS \u7684\u4e00\u4e2a\u6587\u4ef6\u4e0a\uff09\u3002\u4f46\u662f Driver \u7aef\u65e0\u6cd5\u6b63\u5e38\u62ff\u5230\u65b0\u7684 Token\u3002\u603b\u662f\u62a5 GSS Error \u548c token not found in cache\u3002

\n

\u73af\u5883\uff1aCDH 5.13.1\uff0chadoop-2.6.0-cdh5.13.1\uff0cspark 2.2.0 \u793e\u533a\u7248

\n

Google \u4e86\u4e00\u5708\uff0c\u52a0\u4e86 --conf spark.hadoop.fs.hdfs.impl.disable.cache=true\uff0c\u8bf4\u662f\u5728 HDFS HA \u6a21\u5f0f\u4e0b\uff0c\u5fc5\u987b\u5173\u6389 cache \u529f\u80fd\uff0c\u4f46\u662f\u52a0\u4e0a\u4e86\u5e76\u6ca1\u6709\u4ec0\u4e48\u7528\u3002

\n

\u6709\u9047\u5230\u8fc7\u7c7b\u4f3c TOKEN \u95ee\u9898\u8fc7\u671f\u7684\u5927\u4f6c\u5417\uff1f\u6307\u70b9\u4e00\u4e0b

\n" }, { "author": { "url": "https://www.v2ex.com/member/Nirlan", "name": "Nirlan", "avatar": "https://cdn.v2ex.com/avatar/f500/ba3d/227952_large.png?m=1496560162" }, "url": "https://www.v2ex.com/t/493372", "title": "\u5173\u4e8e Spark \u8bfb\u53d6\u9884\u5206\u533a Hbase \u95ee\u9898", "id": "https://www.v2ex.com/t/493372", "date_published": "2018-09-27T16:26:39+00:00", "content_html": "

RT.

\n

\u6211\u6709\u4e00\u5f20\u9884\u5206\u533a\u7684 Hbase \u8868, split key \u662f 000| 001| ... 199|\u8fd9\u6837,200 \u4e2a\u5206\u533a.

\n

\u6211\u7684 rowkey \u662f\u8fd9\u6837\u8bbe\u8ba1\u7684 001|20180928001122+ \u4e1a\u52a1 ID + 6 \u4f4d\u968f\u673a\u6570

\n

\u8fd9\u6837\u8bbe\u8ba1\u7684\u8bdd\u907f\u514d\u4e86 Spark \u8bfb\u53d6\u65f6\u6570\u636e\u503e\u659c\u554a,\u63d2\u5165\u65f6\u6570\u636e\u70ed\u70b9\u95ee\u9898.

\n

\u4f46\u662f\u6211\u60f3\u7528 Spark \u8bfb\u53d6\u67d0\u4e00\u5929\u7684\u6570\u636e,\u8fd8\u60f3\u7528 scan \u64cd\u4f5c\u7684\u8bdd,\u8c8c\u4f3c\u5f88\u96be\u5b9e\u73b0.

\n

\u6bd4\u5982\u6211\u7684\nstartrow=001|2018092800 + 0000 + 0000 + 000000\nendrow=001|2018092899 + 0000 + 0000 +000000

\n

\u6211\u60f3\u8bfb\u53d6\u5b8c\u8fd9\u4e00\u5929\u7684\u6570\u636e,\u96be\u9053\u5f97\u5faa\u73af 200 \u4e2a region \u5417?

\n

\u5355\u673a\u591a\u7ebf\u7a0b\u7684\u8bdd\u662f\u53ef\u4ee5\u8fd9\u4e48\u505a\u7684,\u4f46\u662f\u6211\u60f3\u7528 spark \u5206\u5e03\u5f0f\u73af\u5883\u6765\u64cd\u4f5c.

\n

\u6211\u67e5\u9605\u4e86 TableSnapshotScanner \u7c7b,\u5bf9\u5176 regions \u5c5e\u6027\u4e0d\u751a\u7406\u89e3,\u671b\u9ad8\u624b\u7ed9\u4e2a\u601d\u8def(\u7ed9\u4e2a demo \u6700\u597d\u4e86...

\n" }, { "author": { "url": "https://www.v2ex.com/member/ClaireWei", "name": "ClaireWei", "avatar": "https://cdn.v2ex.com/avatar/b7da/1317/342083_large.png?m=1538300548" }, "url": "https://www.v2ex.com/t/492424", "title": "[\u817e\u8baf\u5c97\u4f4d] \u5927\u6570\u636e\u5f00\u53d1\u5de5\u7a0b\u5e08\uff08\u6df1\u5733\uff09", "id": "https://www.v2ex.com/t/492424", "date_published": "2018-09-25T09:02:29+00:00", "content_html": "

\u5de5\u4f5c\u804c\u8d23\n1.\u8d1f\u8d23\u6d77\u91cf\u5927\u6570\u636e\u7684\u6e05\u6d17\u3001\u7279\u5f81\u6316\u6398\u5de5\u4f5c\uff1b\n2.\u8d1f\u8d23\u7528\u6237\u6807\u7b7e\u6316\u6398\u4e0e\u753b\u50cf\u4f53\u7cfb\u6784\u5efa\uff1b\n3.\u57fa\u4e8e Spark \u6846\u67b6\u5927\u6570\u636e\u5206\u5e03\u5f0f\u6570\u636e\u6316\u6398\u5e73\u53f0\u7684\u8bbe\u8ba1\uff0c\u5f00\u53d1

\n

\u4efb\u804c\u8d44\u683c\n1.\u7cbe\u901a Scala/Python \u7a0b\u5e8f\u5f00\u53d1(\u81f3\u5c11\u4e00\u79cd)\uff0c\u719f\u6089 Linux/Unix \u5f00\u53d1\u73af\u5883\uff1b\n2. 2 \u5e74\u4ee5\u4e0a\u5206\u5e03\u5f0f\u5927\u6570\u636e\u5f00\u53d1\u3001\u6570\u636e\u6316\u6398\u3001\u673a\u5668\u5b66\u4e60\u7b49\u9886\u57df\u7684\u5f00\u53d1\u7ecf\u9a8c\uff1b\n3. \u719f\u6089\u5e38\u7528\u5206\u7c7b\u3001\u805a\u7c7b\u7b97\u6cd5\uff0c\u5982\u6734\u7d20\u8d1d\u53f6\u65af\uff0cKNN\uff0cSVM\uff0c\u903b\u8f91\u56de\u5f52\u7b49\uff1b\n4. \u719f\u6089 Spark(Streaming/MLlib)\u3001\u6709 Hadoop \u7684\u5206\u5e03\u5f0f\u6570\u636e\u6316\u6398\u5f00\u53d1\u7ecf\u9a8c

\n

\u6709\u610f\u8005\u8bf7\u8054\u7cfb Claire 13751127682 \uff08\u624b\u673a&\u5fae\u4fe1\uff09\n\u7b80\u5386\u53ef\u6295\u9012 Claire.Wei@elpcon.com

\n" }, { "author": { "url": "https://www.v2ex.com/member/myyou", "name": "myyou", "avatar": "https://cdn.v2ex.com/avatar/bbaf/e3f1/135646_large.png?m=1440754497" }, "url": "https://www.v2ex.com/t/477298", "title": "Apache Kylin \u5728 left outer join \u540e\u4e3a\u4f55\u4e0d\u652f\u6301+-*/\u8fd0\u7b97\u554a\uff1f", "id": "https://www.v2ex.com/t/477298", "date_published": "2018-08-06T07:44:30+00:00", "content_html": "\n
select\n  a.\"member_num\", b.\"member_num_tb\", a.\"member_num\" / b.\"member_num_tb\"\nfrom\n(\n    select\n      count(distinct mobile) as \"member_num\"\n      ,'week' as \"dtype\"\n    from member\n    where dweek <= '18.06.04~18.06.10'\n) as a\nleft outer join\n(\n    select\n      count(distinct mobile) as \"member_num_tb\"\n      ,'week' as \"dtype\"\n    from member\n    where dweek_tb <= '18.06.04~18.06.10'\n) as b on a.\"dtype\" = b.\"dtype\"\n
\n\n
That the right side of the BinaryTupleExpression owns columns is not supported for / while executing SQL: \n\"select a.\"member_num\", b.\"member_num_tb\", a.\"member_num\" / b.\"member_num_tb\" from ( select count(distinct mobile) as \"member_num\" ,'week' as \"dtype\" from member where dweek <= '18.06.04~18.06.10' ) as a left outer join ( select count(distinct mobile) as \"member_num_tb\" ,'week' as \"dtype\" from member where dweek_tb <= '18.06.04~18.06.10' ) as b on a.\"dtype\" = b.\"dtype\" \"\n
\n

\u8bf7\u95ee\u8fd9\u662f\u662f\u4ec0\u4e48\u539f\u56e0\u554a\uff1f

\n" }, { "author": { "url": "https://www.v2ex.com/member/IcyinCoupang", "name": "IcyinCoupang", "avatar": "https://cdn.v2ex.com/gravatar/b908c761f178867cbd13e74871169d11?s=73&d=retro" }, "url": "https://www.v2ex.com/t/475351", "title": "\u7f8e\u8d44\u7535\u5546 Coupang\uff08\u97e9\u56fd\u7535\u5546\u5de8\u5934\uff09\u5728\u4e0a\u6d77\u529e\u516c\u5ba4\u62db Chief Architect\uff08\u5927\u6570\u636e\u65b9\u5411\uff09", "id": "https://www.v2ex.com/t/475351", "date_published": "2018-07-30T06:53:09+00:00", "content_html": "

\u7f8e\u8d44\u7535\u5546 Coupang \uff08\u97e9\u56fd\u4e9a\u9a6c\u900a\uff09,\u4e0a\u6d77\u529e\u516c\u5ba4\u62db Chief Architect \uff08\u5927\u6570\u636e\u65b9\u5411\uff09\uff0c\u5730\u5740\uff1a\u6d66\u4e1c\u5730\u94c1 2 \u53f7\u7ebf\u91d1\u79d1\u8def\u7ad9\u957f\u6cf0\u5e7f\u573a\u3002 \u5de5\u4f5c\u73af\u5883\u597d\uff0c\u6280\u672f\u5927\u725b\u591a\uff0c\u6d77\u5916\u57f9\u8bad\u51fa\u5dee\u673a\u4f1a\u591a\uff08\u7845\u8c37\uff0c\u9996\u5c14\uff09\uff0c\u5f39\u6027\u5de5\u4f5c\u5236\uff0c\u6280\u672f\u6c1b\u56f4\u5f88\u6d53\u539a\u3002

\n

\u8981\u6c42\uff1a\u82f1\u6587\u53e3\u8bed\u6d41\u5229\uff0c\u6c9f\u901a\u597d\uff0c8 \u5e74\u5de6\u53f3 Java \u5f00\u53d1\u7ecf\u9a8c\uff0c2-3 \u5e74\u5927\u6570\u636e\u5f00\u53d1\u7ecf\u9a8c\uff0c\u613f\u610f\u843d\u5730\u505a\u5f00\u53d1\u94bb\u7814\u6280\u672f\u3002

\n

\u5982\u679c\u6709\u5174\u8da3\uff0c\u6b22\u8fce\u52a0\u5fae\u4fe1\uff1alittleicybingbing\uff0c\u6216\u8005\u76f4\u63a5\u53d1\u7b80\u5386\u5230 iliu@coupang.com

\n

Chief Architect - Fraud Detection System (Director or Sr. Manager level)\nCoupang is one of the largest and fastest growing e-commerce platforms on the planet. Our mission is to create a world in which Customers ask, \u201c How did I ever live without Coupang?\u201d We are looking for passionate builders to help us get there. Powered by world-class technology and operations, we have set out to transform the end-to-end Customer experience -- from revolutionizing last-mile delivery to rethinking how Customers search and discover on a truly mobile-first platform. We have been named one of the \u201c 50 Smartest Companies in the World \u201d by MIT Technology Review and \u201c 30 Global Game Changers \u201d by Forbes.\nCoupang is a global company with offices in Beijing, Los Angeles, Seattle, Seoul, Shanghai, and Silicon Valley.\nWho are We?\nFDS (Fraud Detection System) protects online transactions from fraudsters and abusers. We build highly sophisticated and analytics-based systems and services that scan massive amounts of transaction-related data (buyer, seller, payment, etc.) to uncover underlying correlations and detect fraud and abuse patterns, as well as tools and processes for operation teams to investigate and act on these. A robust FDS protects Coupang \u2019 s brand and image, prevents negative impacts to Customers and Sellers and reduces revenue loss.\nOur mission is to ensure that every transaction is trusted, safe, and protected from fraud. We build the brand image that Coupang is a trustworthy place to buy things and do business.\nBy leveraging state-of-the-art Big Data technologies and machine learning expertise, we try to build a data-driven culture for fraud detection and risk management. We believe in the machine learning approach to fight fraudsters and protect our customers, and are the early adopters for Big Data and Machine Learning technologies at scale in the industry.\nWe view Big Data and Machine Learning as our core strengths. We are a truly blended group with both aspired technologists and data scientists. Advanced analytics powered by strong engineering muscles creates solutions to one of the most compelling battles for e-Commerce company like Coupang. We \u2019 re the modern detectives hunting for fraudulent activities, but only with better weapons in our hands.\nResponsibilities:\n\u2022 Act as the chief architect/senior technical leader of the team throughout the software development life cycle\n\u2022 Lead team to develop high performance big data framework\n\u2022 Collaborate with product owners, stakeholders and other agile teams on business and\nproduct strategy from a technical and architectural perspective;\n\u2022 Collaborate with data scientist, verify and implement data models to realize automatic and\naccurate fraud detection; improve the risk management level of E-commerce/payment\nplatforms;\n\u2022 Analyze information acquired and compare solutions and weigh them against the actual\nneeds; provide root cause analysis affecting key business problems;\n\u2022 Play an active role in company-wide technical strategy;\nQualifications:\n\u2022 Masters/PhDs in CS machine learning, Statistics, Applied Math or a quantitative field;\n\u2022 10+ years software development experience with at least 5 years proven track record acting as the architect/leader/manager of mission critical enterprise software/e-commerce solutions\n\u2022 Excellent communication and presentation skills\n\u2022 Expert at one or more of program languages (JAVA, Scala, C, C++, etc.);\n\u2022 Experience in operation and development in Hadoop and other leading big data ecosystem\n(Spark, Hive, Pig, Sqoop, Oozie);\n\u2022 Experience with SQL (MySQL/Oracle/MSSQL/DB2 etc)\n&NoSQL(Redis/Cassandra/HBase/MongoDB);\n\u2022 Passion to learn new technologies\n\u2022 Will work with colleagues in Silicon Valley and Seoul. Overseas travel is required.\n\u2022 Fluent spoken and written English

\n" }, { "author": { "url": "https://www.v2ex.com/member/engineer9", "name": "engineer9", "avatar": "https://cdn.v2ex.com/gravatar/e92bc2013eee02c242d7c295737153e0?s=73&d=retro" }, "url": "https://www.v2ex.com/t/475230", "date_modified": "2018-07-30T02:04:28+00:00", "content_html": "

\u73b0\u6709\u4e00\u4e9b\u793e\u4ea4\u6570\u636e\u5982\u4e0b\uff0c

\n
id from to\n1   A   B\n2   A   C\n3   B   A\n4   C   A\n
\n

\u53bb\u91cd\u590d\u540e\u5982\u4e0b\uff0c

\n
id from to\n1   A   B\n2   A   C\n
\n

\u5c1d\u8bd5\u7684\u89e3\u51b3\u65b9\u6848:
\n1\u3001\u91c7\u7528 bloomfilter \u53bb\u91cd\uff0c\u7531\u4e8e bloomfilter \u672c\u8eab\u7b97\u6cd5\u95ee\u9898\uff0c\u4f1a\u4e22\u5931\u4e00\u4e9b\u6570\u636e\uff1b
\n2\u3001\u4f7f\u7528\u6570\u636e\u5e93\u67e5\u8be2\u7136\u540e\u5199\u5165\u5230\u65b0\u8868\uff0c\u901f\u5ea6\u6709\u70b9\u6162\u3002
\n3\u3001\u4f7f\u7528 BerkeleyDB \uff1f
\n4\u3001\u4f7f\u7528 hadoop \u6216\u8005 spark \u89e3\u51b3,\u7f51\u4e0a\u627e\u5230\u7684\u65b9\u5f0f\u51e0\u4e4e\u90fd\u662f\u4f7f\u7528 group by \u6216\u8005 distinct \u4f46\u8fd9\u5e76\u4e0d\u9002\u5408\u6211\u8fd9\u4e2a\u573a\u666f\uff0c\u5982\u4f55\u89e3\u51b3\u5462\uff1f
\n\u5934\u5927\u3002\u3002\u3002\u65b0\u624b\u4e0a\u8def\u3002\u3002\u3002

\n", "date_published": "2018-07-30T02:03:38+00:00", "title": "hadoop or spark \u5927\u6570\u636e\u53bb\u91cd(10 \u4ebf)", "id": "https://www.v2ex.com/t/475230" }, { "author": { "url": "https://www.v2ex.com/member/DongDongXie", "name": "DongDongXie", "avatar": "https://cdn.v2ex.com/gravatar/cf9066a688ba15afbe91f71caafcd8e3?s=73&d=retro" }, "url": "https://www.v2ex.com/t/474584", "title": "\u8bf7\u6559\u4e2a hadoop \u6307\u4ee4\u95ee\u9898", "id": "https://www.v2ex.com/t/474584", "date_published": "2018-07-27T02:12:59+00:00", "content_html": "

hadoop fs -cat $data_set_high/p*|awk -v repeat=${num} '{for(i=0;i<repeat;i++)print}' >> $local_file

\n

\u6ca1\u592a\u61c2\u8fd9\u4e00\u53e5\uff0c\u65b0\u624b\u4e0a\u8def\uff0c\u8bf7\u5404\u4f4d\u6307\u6559\u4e00\u4e0b\u3002

\n" }, { "author": { "url": "https://www.v2ex.com/member/DongDongXie", "name": "DongDongXie", "avatar": "https://cdn.v2ex.com/gravatar/cf9066a688ba15afbe91f71caafcd8e3?s=73&d=retro" }, "url": "https://www.v2ex.com/t/473813", "title": "\u8bf7\u95ee\u8fd9\u4e2a hadoop \u6307\u4ee4\u662f\u4ec0\u4e48\u610f\u601d\u554a", "id": "https://www.v2ex.com/t/473813", "date_published": "2018-07-24T12:54:14+00:00", "content_html": "

hadoop fs -cat $data_set/.pig_header>$local_file\nhadoop fs -cat $data_set/p*>>$local_file

\n

cat \u4e0d\u662f\u6253\u5370\u5417\uff0c\u540e\u9762\u8ddf\u8fd9\u4e2a>\u548c>>\u662f\u4ec0\u4e48\u610f\u601d\u554a\uff0c\u6c42\u6559

\n" }, { "author": { "url": "https://www.v2ex.com/member/s3040608090", "name": "s3040608090", "avatar": "https://cdn.v2ex.com/gravatar/8fcc6510762b10014884b3914c3fdbfe?s=73&d=retro" }, "url": "https://www.v2ex.com/t/469644", "title": "\u60f3\u8bf7\u6559\u4e00\u4e0b\u4ece oozie \u8c03\u7528 hadoop \u9047\u5230\u7684\u95ee\u9898", "id": "https://www.v2ex.com/t/469644", "date_published": "2018-07-10T08:35:42+00:00", "content_html": "

\u957f\u8bdd\u77ed\u8bf4\u3002hadoop(2.7)\u548c oozie(5.0)\u90fd\u5b89\u88c5\u5728\u4e00\u53f0\u670d\u52a1\u5668\u4e0a\uff08\u4f2a\u5206\u5e03\u5f0f\uff09\u3002\u5c1d\u8bd5 oozie \u7684\u5b98\u65b9 example\uff1aexample

\n

\u4fee\u6539 map-reduce \u7684 job.properties \u5982\u4e0b\uff1a

\n
nameNode=hdfs://localhost:9000\njobTracker=localhost:8032\nqueueName=default\nexamplesRoot=examples\n\noozie.wf.application.path=${nameNode}/user/${user.name}/${examplesRoot}/apps/map-reduce\noutputDir=map-reduce\n
\n

\u6309\u7167\u6587\u6863\u4e2d\u7684\u63d0\u793a\u8f93\u5165\u547d\u4ee4\"oozie job -oozie http://localhost:11000/oozie -config examples/apps/map-reduce/job.properties -run\"\u65f6\u62a5\u9519\uff1a

\n
\n

Error: E0501 : E0501: Could not perform authorization operation, Failed on local exception: com.google.protobuf.InvalidProtocolBufferException: Message missing required fields: callId, status; Host Details : local host is: \"Master/\u670d\u52a1\u5668 ip\"; destination host is: \"localhost\":9000;

\n
\n

\u5df2\u7ecf\u786e\u8ba4 nameNode \u7aef\u53e3\u65e0\u8bef\uff0c\u5728\u670d\u52a1\u5668 shell \u952e\u5165\"hadoop fs -ls hdfs://localhost:9000\"\u4f1a\u8fd4\u56de\u7ed3\u679c\u3002\uff0c\u53ef\u4ee5\u786e\u8ba4 oozie \u7684\u7aef\u53e3\u65e0\u8bef\uff0c\u6d4f\u89c8\u5668\u8f93\u5165\u201c http://\u670d\u52a1\u5668 ip:11000 \u201d\u53ef\u4ee5\u8fdb\u5165 oozie \u7ba1\u7406\u754c\u9762\u3002jobtracker ip \u5e94\u8be5\u6ca1\u9519\uff0c\u6211\u5728 yarm-site.xml \u91cc\u6ca1\u6709\u770b\u89c1\u7aef\u53e3\u7684\u8bbe\u7f6e\u9879\uff0c\u4f46\u662f\u5728\"netstat -tunlp\"\u4e2d\u770b\u89c1\u6709 8032 \u7aef\u53e3\u5728\u5360\u7528\uff0c\u8fd9\u5e94\u8be5\u662f resourcemanager \u7684\u9ed8\u8ba4\u7aef\u53e3\u3002\n\u5173\u952e\u70b9\u53ef\u80fd\u5728 Message missing required fields: callId, status \u8fd9\u4e2a\u63d0\u793a\u4e0a\uff0c\u653e\u72d7\u641c\u770b\u5230\u6709\u4eba\u8bf4\u662f hadoop \u7248\u672c\u592a\u9ad8\u63a5\u53e3\u4e0d\u4e00\u81f4\u4ec0\u4e48\u7684\uff0c\u4f46\u662f oozie5.0 \u5df2\u7ecf\u8981\u6c42\u5b89\u88c5\u7684 hadoop \u6700\u4f4e\u7248\u672c\u5728 2.6 \u4ee5\u4e0a\u3002\u8fd9\u8ba9\u6211\u5f88\u7591\u60d1\u3002\n\u5e0c\u671b\u80fd\u5f97\u5230\u4e00\u4e9b\u6307\u5bfc\uff0c\u611f\u8c22\u3002

\n" }, { "author": { "url": "https://www.v2ex.com/member/zzys101", "name": "zzys101", "avatar": "https://cdn.v2ex.com/gravatar/7d476b71fbdf3c87c27b0c7373d56bb4?s=73&d=retro" }, "url": "https://www.v2ex.com/t/467504", "title": "Hadoop security", "id": "https://www.v2ex.com/t/467504", "date_published": "2018-07-02T08:09:41+00:00", "content_html": "

Hadoop \u73af\u5883\u4e2d\u662f\u5426\u6709\u53ef\u4ee5\u66ff\u4ee3 kerberos \u505a\u8ba4\u8bc1\u534f\u8bae\u7684\uff1f\nhadoop \u591a\u79df\u6237\u7684\u89e3\u51b3\u65b9\u6848,\u6709\u63a8\u8350\u7684\u5417\uff1f

\n" }, { "author": { "url": "https://www.v2ex.com/member/PanameraM", "name": "PanameraM", "avatar": "https://cdn.v2ex.com/avatar/73bd/dce1/193938_large.png?m=1553088475" }, "url": "https://www.v2ex.com/t/457643", "title": "\u5927\u5bb6\u7ecf\u5e38\u53bb\u54ea\u4e9b\u5927\u6570\u636e\u7684\u8bba\u575b\u6216\u8005\u7f51\u7ad9\u5b66\u4e60\u5462\uff1f", "id": "https://www.v2ex.com/t/457643", "date_published": "2018-05-25T02:02:56+00:00", "content_html": "RR" } ] }