数据挖掘技术的有趣应用：Kaggle的实践 - 推酷
数据挖掘技术的有趣应用：Kaggle的实践
已收藏到推刊！
请填写推刊名
描述不能大于100个字符!
权限设置： 公开
仅自己可见
作者：陈运文 博士
Kaggle是创办于美国硅谷的一个近年来风头正劲的数据挖掘竞赛平台，汇集了全球83000多名数据科学家，致力于通过数据挖掘技术解决各种各样现实的问题。文本挑选了Kaggle上若干个有趣实例，让大家了解如何从海量的、看似模糊而随机的实际应用数据中，挖掘出隐含其中的有巨大价值的信息和知识，并指导我们的认知和决策的。
你的职位该给多少薪酬？
薪水的高低恐怕是职场人士们最关心的一个问题了，很多人想知道自己的职位到底在人才市场上值多少钱？而对招聘的企业来说，如果能为各个职位制定合理的薪酬，则既能招募到渴求的人才，又能有效的控制成本，何乐而不为。来自英国伦敦的数据挖掘公司Adzuna，就致力于此。Adzuna根据积累的海量职位薪酬数据，来为招聘双方提供薪酬的制定的科学依据。
图：根据积累的海量职位薪酬数据，来为招聘双方提供薪酬的制定的科学依据
Adzuna所记录的数据则非常全面，这些信息包括招聘公司(Company)、职位类型(Category)、头衔(Title)，详细描述(Full Description), 具体工作地点(Location Tree)，全职或兼职（ContractType），是否永久雇员(ContractTime)，数据来源(SourceName)，根据以上信息，对薪酬的预测采用通行的MAE（Mean Absolute Error）方法。截至目前，Adzuna竞赛中最好的预测算法MAE已达到了3796.7，预测值和实际成交值的误差接近10%，已十分精确。大数据挖掘技术不仅能为求职者提供深度的职业指导、薪酬情况和发展趋势预测，还是英国内阁高官的“幕后智囊团”。据报道，Adzuna开发的”Number 10 Dashboard”是首相官邸唐宁街10号专用的iPad应用，首相卡梅伦很喜欢通过这项应用提供的英国实时公共数据，及时了解失业率、职位空缺、薪资水平等经济发展情况的报告，并调整政府工作。
知道下一次你去超市的时间和花费的金额
去超市购物恐怕是我们日常生活中再普通不过的一件事情了，但是你可曾想过这背后有着严谨而复杂的数据分析工作？全球三大零售商之一的英国TESCO公司为此推出了海量顾客研究数据系统Dunnhumby Shop，并根据该系统中记录的消费者在过去若干年间历次购物的时间、消费金额等信息，发起举办了Dunnhumby Challenge竞赛，任务是预测每位消费者下一次来访问超市的时间和即将消费的金额。想不到计算机系统会比你更先一步了解你的购买行为吧。
图：改变以往零售行业靠经验和直觉进行决策的机制
这个数据挖掘任务属于统计学中的间断性偶发需求(Intermittent Demand)，经过激烈角逐，来自俄罗斯的数据科学家D’yakonov Alexander从全球数据挖掘高手的竞争中脱颖而出，达到了最好的预测精度。指数加权移动平均法和针对性的算法优化起到了很好的作用。TESCO使用Clubcard记录的海量用户行为数据，以及大数据挖掘技术，正在成功帮助各个门店针对不同的顾客群体采取针对性的促销手段。与此同时，供应商也可以通过这个数据库清楚地了解到商品的市场反响，并对不同区域制订合理有效的价格和配货方案。通过开发数据这座宝藏，改变了以往零售行业靠经验和直觉进行决策的机制，让TESCO在与英国超市业其它竞争对手的赛跑中遥遥领先，斩获了丰厚的回报。
告诉你道路通行所需的时间
澳大利亚悉尼的M4高速公路是新南威尔士州(NSW)的一条交通大动脉，这条道路的畅通与否，直接影响着悉尼人的生活和工作。和全球很多高速公路一样，M4公路的交通状况时好时坏，澳大利亚道路与交通管理局（RTA）在Kaggle上发起的这项数据挖掘任务，就是根据过去2年时间里，M4公路各个路段(某入口至出口)感知器24小时不间断记录下的车辆实际通行时间记录，请数据科学家们来设计智能的模型，来预测接下来的任意时刻、指定路线的驾车通行时间。
图：M4公路交通状况的预测
经过一番激烈角逐，最终来自卡耐基梅隆大学的Jose Gonzale和Guido Cortes给出了预测精度最高的方案（RMSE=191.47）。该方案对历史数据所记录的时间和空间因素进行充分的挖掘和运用，既有对时间周期(每天第几小时、每周星期几、每月第几日)的不同维度的切分和统计，也有各子路段空间距离和通行先后的相互影响的数据建模，用不同的特征和数据组合方式构成决策树，再由300棵各不相同的决策树组合成一个随机森林(Random Forest)进行综合决策。最终这个构造良好的预测模型不仅应用在了悉尼交通调度系统上，也成为了悉尼人驾车出行时提前进行路径规划的好帮手。
雾霾天气让你早知道
空气污染已经是全球性的环境问题，根据世界卫生组织的统计，全世界约有2.35亿人在糟受哮喘以及相关疾病的折磨。对于少年儿童来说，污染空气对身体的伤害尤其巨大。和传统的天气情况不同，由于空气污染和人类的活动密切相关（例如行驶车辆的密度），通过历史数据挖掘能发现环境空气质量变化的过程，提高城市空气质量预警水平。
由EMC公司和Data Science Global联合发起的Air Quality Prediction竞赛就致力于通过大数据挖掘技术来更精确的进行空气质量预测，并能细致到小时级的预报。这个数据挖掘任务使用了美国伊利诺伊州(illinois)政府近年来记录的空气质量历史数据，这些信息不仅包括各个观测点的空气质量情况，还记录有各观测点的经纬度、海拔、风向、风速、气温、气压，阳光强度等详细数据。来自美国的Ben Hamner和澳大利亚的James Petterson给出了效果最好的预测模型。随着新技术的进步，不久的将来，数据挖掘系统提供更及时准确的空气质量预警，让相关人群提前规避外出活动，能有效减少污染对人体带来的伤害。
从笔迹就能分辨你的性别
笔迹分析是一件很有趣的工作，在刑事侦查或者法庭断案时，笔迹经常被作为重要呈堂证供。这是由于每个人运笔的方式、书写的格式都千差万别，从笔迹中往往判读出很多隐藏的信息。然而由专家们人工判读的方式非常耗时耗力，那么笔迹鉴定能否由计算机来代劳呢？卡塔尔大学的科学家们在Kaggle上提出了一个有趣的命题：如何让计算机根据笔迹自动判别写作者的性别？
图：通过笔迹就能分析出你的性别
了解机器学习的同学们都熟悉，这里性别的判别是一个典型的模式识别（Pattern Recognition）问题，通过挖掘系统所记录了大量已知性别用户的笔迹信息（阿拉伯数字、英文文章等），计算机程序通过“学习”这些笔迹信息后，会“聪明”的作出判断（如图）。
近几年来，大量训练样本和大数据挖掘的新方法的提出，从特征抽取、自动筛选、特征组合或降维，到模式分类和决策等各个环节，数据挖掘技术每一天都在前进。今天，计算机系统能解决的问题也比以前大大增加了。尤其近几年深度学习（Deep Learning）技术在图像、音频、文本挖掘等方面处理精度的突飞猛进，相信不久的将来，大数据挖掘系统能淘出海量数据中所蕴藏的越来越多的宝藏。
已发表评论数()
&&登&&&陆&&Tesco - Current Prices, Offers, Deals, Tesco Clubcard, Tesco Personal Finance
Food Guides
Save on Shopping
Petrol Prices
Car and Petrol Guides
Motoring Calculators
Travel Guides
Vouchers and Discounts
Daily Deals
Competitions and Freebies
Family, Tax and Income
Banking: most popular
Credit cards
Choose supermarket offers:
Food Guides
British supermarket group Tesco is one of the world&s leading grocery and general merchandising retailers. The company is the UK&s largest retailer by both sales and domestic market share, with profits in excess of &2 billion.
Tesco currently holds a 30% share of the UK grocery market, accounting for 14.3% of all retail sales in Britain, or &1 in every &7 spent in UK stores.
Tesco specialises primarily in food and drink with its Value and Finest ranges, but has also become a major player in a range of non-food markets including consumer electricals, clothing, financial services, telecoms, fuel, internet services, music downloads and retailing and renting DVDs and CDs.
Tesco Departments
or browse by departments below.
Tesco was founded in 1919 by Jack Cohen as a grocery stall in the east end of London . The Tesco brand first appeared in 1924, but it wasn&t until 1929 when the first Tesco store opened for business in Edgware, Middlesex. Fast forward 70 years and there are more than 2,100 outlets in the UK alone.
UK Operations
In the UK, Tesco operates six different store formats, each distinguished by size and the range of products sold.
Tesco Superstores & Tesco superstores are the company&s standard store format. These large supermarkets stock and sell all groceries along with a large range of non-food products. They also offer an in-store pharmacy.
Tesco Extra & Tesco Extra stores are the retailer&s large out-of-town hypermarkets (second in size behind superstores) which stock nearly all of Tesco's product ranges.
Tesco Metro & Metro stores are medium-sized stores (sized between Tesco superstores and Tesco Express stores) that offer a range of food lines.
Tesco Express & Express stores are convenience shops that are mainly located on Esso petrol station forecourts. Most of their stock consists of food, with an emphasis on higher-margin products alongside everyday essentials.
Tesco Homeplus & Homeplus stores offer all of Tesco's ranges, except food, in warehouse-style units based in retail parks. These large units feature an Order and Collect desk where customers can purchase and collect most items without delay.
One Stop & One Stop stores are the very smallest Tesco stores. These stores work on a different pricing to all the other store formats and are also distinguished by their later opening hours.
Global Operations
Tesco boasts a global store portfolio of over 1,800 stores. Besides the UK, the company operates stores in France, China, the United States, Poland, Republic of Ireland, Japan, Malaysia, Czech Republic, South Korea, Slovakia, Thailand and Turkey .
Online Operation
Tesco runs the world's largest grocery home-shopping service, . The site was formally launched in 2000 - six years after Tesco first began operating on the internet.
Consumer goods, telecommunications and financial services are also provided via the internet.
Tesco Direct, the supermarket's catalogue/internet service, was added to the retailer&s portfolio in 2007. Customers can browse the Tesco Direct catalogue or go online to purchase non-food goods ranging from home electricals, toys and furniture to sports equipment, jewellery and even bathroom suites.
Tesco Personal Finance
Tesco Personal Finance is the banking arm of the Tesco plc. The business was previously run as a joint venture with the Royal Bank of Scotland, but in July 2008 Tesco paid &950 million to acquire the bank&s 50% share in the company.
Financial products on offer include loans, credit cards, savings accounts, mortgages, and several types of insurance, including car, home, life, dental and travel cover.
Tesco Clubcard
Tesco is one of only two UK supermarkets to offer its customers a loyalty card-scheme (the other being Sainsbury&s). Introduced in 1995, Tesco's &Clubcard& branded loyalty scheme is now the number one loyalty card in the UK, with around 13 million active Clubcard holders.
Every &1 spent in a Tesco store, online
or through Tesco Petrol, earns customers one Clubcard point. Shoppers can also collect points by paying with a Tesco Credit Card, or by using Tesco Mobile, Tesco Broadband, selected Tesco Personal Finance products or through Clubcard partners, Avis and E.ON.
Each point is worth 1p in-store when redeemed, or 4p when used with Clubcard deals on holidays, day trips, etc. Points can also be used to rack up Airmiles, or converted into coupons.
Compare broadband
Compare & save ?200+ on TV, phone &
broadband. From &2.99.
Enter your postcode to begin:
The latest money saving news.
- Thu, 23 Oct 2014
- Tue, 21 Oct 2014
- Thu, 16 Oct 2014
Our 5 most popular guides:
How to save on your energy bills without moving
Why food prices are rising at such a rapid rate
10 tips for spending less at the pumps
Why spend when you can get all this for free!
Council house tenants can purchase their home
Join the FREE money saving newsletter:
Latest prices & offers
Voucher & discount codes
Supermarkets
Save money
Money saving forum
& Supermarket.co.uk, 1996 - 2012. All rights reserved.