2013 年我接触
的时候，就听闻 Python 的能力十分强大。因此，在熟悉 Python 的基本语法之后，我就和几个一起合作，试着用 Python 的
库构建了一个贴吧 Python 。
然而，使用的过程中，我发现两个标准库的语法并不自然，甚至可以说十分反人类——用着很难受。又有，我平时使用 Python 甚少涉及到网络编程的内容。因此，Python 的网络编程就被我放下了，直到我认识了
库的宣言是
HTTP for Humans （给人用的 HTTP 库）
我们首先来验证一下。
在网络编程中，最最基本的任务包含：
反序列化打印内容
我们以 GitHub 为例，先看一下使用
要怎么做。为了把事情弄简单点，我们假设实现已经知道，GET 请求
返回的内容是个 JSON 格式的数据（实际上通过
content-type
也能判断）。
import urllib2import jsongh_url
= ''cs_user = 'user'cs_psw
= 'password'req = urllib2.Request(gh_url)password_manager = urllib2.HTTPPasswordMgrWithDefaultRealm()password_manager.add_password(None, gh_url, cs_user, cs_psw)auth_manager = urllib2.HTTPBasicAuthHandler(password_manager)opener = urllib2.build_opener(auth_manager)urllib2.install_opener(opener)handler = urllib2.urlopen(req)if handler.getcode() == 200:
text = handler.read()
d_text = json.loads(text)
for k, v in d_text.items():
print k, v
如果运行正确，那么代码应该返回：
issues_url /issuescurrent_user_repositories_url /user/repos{?type,page,per_page,sort}rate_limit_url /rate_limitrepository_url /repos/{owner}/{repo}...user_repositories_url /users/{user}/repos{?type,page,per_page,sort}team_url /teams
同样的效果，用
库则有如下代码：
import requestscs_url
= ''cs_user = 'user'cs_psw
= 'password'r = requests.get(cs_url, auth=(cs_user, cs_psw))if r.status_code == requests.codes.ok
for k, v in r.json().items():
print k, v
溢美之词就不用说了，读到这里的你心里肯定只有一声「卧槽，这才是 Python 该有的样子」。那么，接下来我们看看
都有哪些黑魔法。
最推荐的方式，是直接安装推荐过的
如果你不想安装 Anaconda，那么建议你使用
安装；只需在命令行下执行：
pip install requests
的基本用法，呃，真是不能再基本了。最基本的操作，就是以某种 HTTP 方法向远端发送一个请求而已；而
库就是这么做的。
import requestscs_url = 'http://httpbin.org'r = requests.get("%s/%s" % (cs_url, 'get'))r = requests.post("%s/%s" % (cs_url, 'post'))r = requests.put("%s/%s" % (cs_url, 'put'))r = requests.delete("%s/%s" % (cs_url, 'delete'))r = requests.patch("%s/%s" % (cs_url, 'patch'))r = requests.options("%s/%s" % (cs_url, 'get'))
从语法上看，
库设计的非常自然。所谓
requests.get
，就是以 GET 方式发送一个 REQUEST，
类的结果，保存为
中取得所有你想得到的和 HTTP 有关的信息。下面，我们以 GET 方法为例，依次介绍。
URL 传参 / 获取请求的 URL
如果你经常上网（废话，看到这里的都上过网吧……），一定见过类似下面的链接：
https://encrypted..com/search?q=hello
&协议&://&域名&/&&?&键1&=&值1&&&键2&=&值2&
库提供的 HTTP 方法，都提供了名为
的参数。这个参数可以接受一个 Python 字典，并自动格式化为上述格式。
import requestscs_url = '/s'param
= {'ie':'utf-8', 'q':'query'}r = requests.get (cs_url, params = param)print r.url
运行将得到：
/s?q=query&ie=utf-8
HTTP 状态码 / 重定向跳转
类可以方便地获取请求的 HTTP 状态码和重定向状态。
360 公司的，原名「好搜」，现在改为「360 搜索」；域名也从
。如果你在地址栏输入
，那么会经由 302 跳转到
。我们借此来演示。
import requestscs_url = '/s'param
= {'ie':'utf-8', 'q':'query'}r = requests.get (cs_url, params = param)print r.url, r.status_codecs_url = '/s'r = requests.get (cs_url, params = param)print r.url, r.status_code, r.history
/s?q=query&ie=utf-8 200/s?q=query&ie=utf-8 200 [&Response [302]&]
我们发现，
默认自动地处理了 302 跳转。在经过跳转的请求中，返回的 URL 和状态码都是跳转之后的信息；唯独在
中，用 Python 列表记录了跳转情况。
大多数情况下，自动处理是挺好的。不过，有时候我们也想单步追踪页面跳转情况。此时，可以给请求加上
allow_redirects = False
import requestscs_url = '/s'param
= {'ie':'utf-8', 'q':'query'}r = requests.get (cs_url, params = param)print r.url, r.status_codecs_url = '/s'r = requests.get (cs_url, params = param, allow_redirects = False)print r.url, r.status_code, r.history
输出结果：
/s?q=query&ie=utf-8 200/s?q=query&ie=utf-8 302 []
自动处理跳转后，返回的 URL 和状态码都符合预期了。
的超时设置以
为单位。例如，对请求加参数
timeout = 5
即可设置超时为 5 秒。
# a very short timeout is set intentionallyimport requestscs_url = ''r = requests.get (cs_url, timeout = 0.000001)
返回报错：
Traceback (most recent call last):
File "D:\test\py\test.py", line 6, in &module&
r = requests.get (cs_url, timeout = 0.000001)
File "C:\Users\username\AppData\Local\Continuum\Anaconda\lib\site-s\requests\api.py", line 69, in get
return request('get', url, params=params, **kwargs)
File "C:\Users\username\AppData\Local\Continuum\Anaconda\lib\site-packages\requests\api.py", line 50, in request
response = session.request(method=method, url=url, **kwargs)
File "C:\Users\username\AppData\Local\Continuum\Anaconda\lib\site-packages\requests\sessions.py", line 465, in request
resp = self.send(prep, **send_kwargs)
File "C:\Users\username\AppData\Local\Continuum\Anaconda\lib\site-packages\requests\sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "C:\Users\username\AppData\Local\Continuum\Anaconda\lib\site-packages\requests\adapters.py", line 419, in send
raise ConnectTimeout(e, request=request)requests.exceptions.ConnectTimeout: HTTPConnectionPool(host='', port=80): Max retries exceeded with url: / (Caused by ConnectTimeoutError(&requests.packages.urllib3.connection.HTTPConnection object at 0xAFABE0&, 'Connection to
timed out. (connect timeout=1e-06)'))
我们先来看一下
发出的 HTTP 报文默认的请求头是什么样子的。
import requestscs_url = 'http://httpbin.org/get'r = requests.get (cs_url)print r.content
返回结果：
"args": {},
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate",
"Host": "httpbin.org",
"User-Agent": "python-requests/2.7.0 CPython/2.7.10 /7"
"origin": "xx.xx.xx.xx",
"url": "http://httpbin.org/get"}
通常我们比较关注其中的
User-Agent
Accept-Encoding
。如果我们要修改 HTTP 头中的这两项内容，只需要将一个合适的字典参数传给
import requestsmy_headers = {'User-Agent' : 'From Liam Huang', 'Accept-Encoding' : 'gzip'}cs_url = 'http://httpbin.org/get'r = requests.get (cs_url, headers = my_headers)print r.content
"args": {},
"headers": {
"Accept": "*/*",
"Accept-Encoding": "gzip",
"Host": "httpbin.org",
"User-Agent": "From Liam Huang"
"origin": "xx.xx.xx.xx",
"url": "http://httpbin.org/get"}
可以看到，UA 和 AE 都已经被修改了。
作为 HTTP 请求的响应，返回的内容中也有 HTTP 头。它是一个反序列化为 Python 字典的数据结构，可以通过
Response.headers
import requestscs_url = 'http://httpbin.org/get'r = requests.get (cs_url)print r.headers
"content-length": "263",
"server": "nginx",
"connection": "keep-alive",
"access-control-allow-credentials": "true",
"date": "Fri, 26 Feb :17 GMT",
"access-control-allow-origin": "*",
"content-type": "application/json"}
字节模式 / 自动解包
长期以来，都存在带宽有限的情况。因此，网络上传输的数据，很多情况下都是经过压缩的。经由
发送的请求，当收到的响应内容经过 gzip 或 deflate 压缩时，
会自动为我们解包
。我们可以用
Response.content
来获得以字节形式返回的相应内容。
import requestscs_url = ''r = requests.get (cs_url)if r.status_code == requests.codes.ok:
print r.content
urllib2.urlopen(url).read()
如果相应内容不是文本，而是数据（比如），那么上述打印结果可能会糊你一脸。这里以图片为例，示例一下该怎么办。
import requestsfrom PIL import Imagefrom StringIO import StringIOcs_url = 'http://liam0205.me/uploads/avatar/avatar-2.jpg'r = requests.get (cs_url)if r.status_code == requests.codes.ok:
Image.open(StringIO(r.content)).show()
运行无误的话，能看到我和我爱人的。
文本模式 / 编码
如果响应返回是文本，那么你可以用
Response.text
获得 Unicode 编码的响应返回内容。
import requestscs_url = ''r = requests.get (cs_url)if r.status_code == requests.codes.ok:
print r.text
要获得 Unicode 编码的结果，意味着
会为我们做解码工作。那么
是按照何种编码去对返回结果解码的呢？
会读取 HTTP header 中关于字符集的内容。如果获取成功，则会依此进行解码；若不然，则会根据响应内容对编码进行猜测。具体来说，我们可以用
Response.encoding
来查看/修改使用的编码。
import requestscs_url = ''r = requests.get (cs_url)if r.status_code == requests.codes.ok:
print r.encoding
反序列化 JSON 数据
开篇给出的第一个
示例中，特别吸引人的一点就是
无需任何其他库，就能解析序列化为 JSON 格式的数据。
我们以 IP 查询 Google 公共 DNS 为例：
import requestscs_url
= '/service/getIpInfo.php'my_param = {'ip':'8.8.8.8'}r = requests.get(cs_url, params = my_param)print r.json()['data']['country'].encode('utf-8')
结果将输出：
模拟登录 GitHub 看看
HTTP 协议是无状态的。因此，若不借助其他手段，远程的服务器就无法知道以前和客户端做了哪些通信。Cookie 就是「其他手段」之一。
Cookie 一个典型的应用场景，就是用于记录用户在上的登录状态。
用户登录成功后，服务器下发一个（通常是加密了的）Cookie 文件。
客户端（通常是）将收到的 Cookie 文件保存起来。
下次客户端与服务器连接时，将 Cookie 文件发送给服务器，由服务器校验其含义，恢复登录状态（从而避免再次登录）。
Cookie? 你说的是小吧！
是给人类设计的 Python 库。想想使用浏览器浏览网页的时候，我们没有去保存、重新发送 Cookie 对吗？浏览器都为我们自动完成了。
中，也是这样
当浏览器作为客户端与远端服务器连接时，远端服务器会根据需要，产生一个 SessionID，并附在 Cookie 中发给浏览器。接下来的时间里，只要 Cookie 不过期，浏览器与远端服务器的连接，都会使用这个 SessionID；而浏览器会自动与服务器协作，维护相应的 Cookie。
中，也是这样。我们可以创建一个
requests.Session
，尔后在该 Session 中与远端服务器通信，其中产生的 Cookie，
会自动为我们维护好
POST 方法可以将一组用户数据，以表单的形式发送到远端服务器。远端服务器接受后，依照表单内容做相应的动作。
的 POST 方法时，可以用
参数接收一个 Python 字典结构。
会自动将 Python 字典序列化为实际的表单内容。例如：
import requestscs_url
= 'http://httpbin.org/post'my_data
= /entry/{
'key1' : 'value1',
'key2' : 'value2'}r = requests.post (cs_url, data = /entry/my_data)print r.content
"key1": "value1",
"key2": "value2"
实际模拟登录 GitHub 试试看
模拟登录的第一步，首先是要搞清楚我们用浏览器登录时都发生了什么。
GitHub 登录页面是
。我们首先清空浏览器 Cookie 记录，然后用
打开登录页面。
填入 Username 和 Password 之后，我们打开 Tamper Chrome 和 Chrome 的审查工具（找到
标签页），之后点登录按钮。
在 Tamper Chrome 中，我们发现：虽然登录页面是
，但实际接收表单的是
。若登录成功，则跳转到
首页，返回状态码
而在 Chrome 的审查元素窗口中，我们可以看到提交给
接口的表单信息。内里包含
authenticity_token
两项是定值；
分别是用户名和密码，这很好理解。唯独
authenticity_token
是一长串无规律的字符，我们不清楚它是什么。
POST 动作发生在与
接口交互之前，因此可能的信息来源只有
接口。我们打开 login 页面的源码，试着搜索
authenticity_token
就不难发现有如下内容：
&input name="authenticity_token" type="hidden" value="/entry/......" /&
原来，所谓的
authenticity_token
是明白卸载 HTML 页面里的，只不过用
模式隐藏起来了。为此，我们只需要使用 Python 的正则库解析一下，就好了。
这样一来，事情就变得简单起来，编码吧！
import requestsimport recs_url
= '/login'cs_user = 'user'cs_psw
= 'psw'my_headers = {
'User-Agent' : 'Mozilla/5.0 (M Intel Mac OS X 10_9_5) WebKit/537.36 (KHTML, like Gecko) Chrome/48.0. Safari/537.36',
'Accept' : 'text/html,application/xhtml+xml,application/q=0.9,image/webp,*/*;q=0.8',
'Accept-Encoding' : 'gzip',
'Accept-Language' : 'zh-CN,q=0.8,q=0.6,zh-TW;q=0.4'}sss
= requests.Session()r
= sss.get(cs_url, headers = my_headers)reg
= r'&input name="authenticity_token" type="hidden" value="/entry/(.*)" /&'pattern = re.compile(reg)result
= pattern.findall(r.content)token
= result[0]my_data = /entry/{
'commit' : 'Sign in',
'utf8' : '%E2%9C%93',
'authenticity_token' : token,
'login' : cs_user,
'password' : cs_psw}cs_url
= '/session'r
= sss.post(cs_url, headers = my_headers, data = /entry/my_data)print r.url, r.status_code, r.history
/ 200 [&Response [302]&]
代码很好理解，其实只是完全地模拟了浏览器的行为。
首先，我们准备好了和 Chrome 一致的 HTTP 请求头部信息。具体来说，其中的
User-Agent
是比较重要的。而后，仿照浏览器与服务器的通信，我们创建了一个
requests.Session
。接着，我们用 GET 方法打开登录页面，并用正则库解析到
authenticity_token
。随后，将所需的数据，整备成一个 Python 字典备用。最后，我们用 POST 方法，将表单提交到
最终的结果也是符合预期的：经由
跳转，打开了（
）GitHub 首页。
2013 年我接触 Python 的时候，就听闻 Python 的网络编程能力十分强大。因此，在熟悉 Python 的基本语法之后，我就和几个小伙伴一起合作，试着用 Python 的
库构建了一个百度贴吧 Python 客户端。
然而，使用的过程中，我发现两个标准库的语法并不自然，甚至可以说十分反人类——用着很难受。又有，我平时使用 Python 甚少涉及到网络编程的内容。因此，Python 的网络编程就被我放下了，直到我认识了Ask Your Question
What's wrong?I think nova 8774 port didn't open.Because I writting "service nova- nova list",and then it works. netstat -nlp | grep 8774
has nothing.
root@ubuntu:~# nova image-list
ERROR: HTTPConnectionPool(host='controller', port=8774): Max retries exceeded with url: /v2/c677d3d3cda9b4342e19d/images/detail (Caused by &class 'socket.error'&: [Errno 111] Connection refused)
root@ubuntu:~# keystone endpoint-list
+----------------------------------+-----------+-----------------------------------------+-----------------------------------------+-----------------------------------------+----------------------------------+
internalurl
service_id
+----------------------------------+-----------+-----------------------------------------+-----------------------------------------+-----------------------------------------+----------------------------------+
| 1e803f55a9a74a8fae0cbeb4 | regionOne | http://controller:8774/v2/%(tenant_id)s | http://controller:8774/v2/%(tenant_id)s | http://controller:8774/v2/%(tenant_id)s | bb5e9d0cacc1c9e5119006 |
| 987c7e6e6d60ae | regionOne |
http://controller:9292
http://controller:9292
http://controller:9292
| 415ea166b0aba3184460 |
| e24925baaadf3fbe91d7be | regionOne |
http://controller:
http://controller:
http://controller:3
| 404cff8a3b4 |
+----------------------------------+-----------+-----------------------------------------+-----------------------------------------+-----------------------------------------+----------------------------------+
root@ubuntu:~# keystone service-list
+----------------------------------+----------+----------+-------------------------+
description
+----------------------------------+----------+----------+-------------------------+
| 415ea166b0aba3184460 |
| Openstack Image Service |
| 404cff8a3b4 | keystone | identity |
Openstack Identity
| bb5e9d0cacc1c9e5119006 |
OpenStack Compute
+----------------------------------+----------+----------+-------------------------+
root@ubuntu:~# cat /etc/nova/nova.conf
dhcpbridge_flagfile=/etc/nova/nova.conf
dhcpbridge=/usr/bin/nova-dhcpbridge
logdir=/var/log/nova
state_path=/var/lib/nova
lock_path=/var/lock/nova
force_dhcp_release=True
iscsi_helper=tgtadm
libvirt_use_virtio_for_bridges=True
connection_type=libvirt
root_helper=sudo nova-rootwrap /etc/nova/rootwrap.conf
verbose=True
ec2_private_dns_show_ip=True
api_paste_config=/etc/nova/api-paste.ini
volumes_path=/var/lib/nova/volumes
enabled_apis=ec2,osapi_compute,metadata
rpc_backend = rabbit
rabbit_host=controller
rabbit_password=zh7758520.
my_ip=10.0.0.10
vncserver_listen=10.0.0.10
vncserver_proxyclient_address=10.0.0.10
auth_strategy=keystone
[database]
connection=mysql://nova:zh7758520.@controller/nova
[keystone_authtoken]
auth_url=http://controller:5000
auth_host=controller
auth_port=35357
auth_protocol=http
admin_tenant_name=service
admin_user=nova
admin_password=zh7758520.
/var/log/nova/nova-api.log
15:52:18.378 4145 INFO keystoneclient.middleware.auth_token [-] Starting keystone auth_token middleware
15:52:18.379 4145 WARNING keystoneclient.middleware.auth_token [-] This middleware module is deprecated as of v0.10.0 in favor of keystonemiddleware.auth_token - please update your WSGI pipeline to reference the new middleware package.
15:52:18.381 4145 WARNING keystoneclient.middleware.auth_token [-] Configuring admin URI using auth fragments. This is deprecated, use 'identity_uri' instead.
15:52:18.382 4145 WARNING keystoneclient.middleware.auth_token [-] Configuring auth_uri to point to the public identity
clients may not be able to authenticate against an admin endpoint
15:52:18.389 4145 INFO keystoneclient.middleware.auth_token [-] Using /var/lib/nova/keystone-signing as cache directory for signing certificate
15:52:18.393 4145 WARNING keystoneclient.middleware.auth_token [-] signing_dir is not owned by 0
15:52:18.433 4145 INFO nova.wsgi [-] osapi_compute listening on 0.0.0.0:8774
15:52:18.437 4145 INFO mon.service [-] Starting 1 workers
15:52:18.449 4145 INFO mon.service [-] Started child 4151
15:52:18.461 4151 INFO nova.osapi_compute.wsgi.server [-] (4151) wsgi starting up on http://0.0.0.0:8774/
15:52:18.464 4145 INFO nova.network.driver [-] Loading network driver 'nova.network.linux_net'
15:52:18.622 4145 INFO nova.wsgi [-] metadata listening on 0.0.0.0:8775
15 ... (more)
Sort by >>
pip install olso.rootwrap --ugrade service nova-api restart
it will be work
Had the same problem, found that in /var/log/nova/nova-api.log the error message: Stderr: 'Traceback (most recent call last):\n
File "/usr/bin/nova-rootwrap", line 6, in &module&\n
from oslo.rootwrap.cmd import main\nImportError: No module named rootwrap.cmd\n' Solution:
Reinstalled oslo with the command: pip install --upgrade oslo.rootwrap
Check your nova log files:
/var/log/nova/nova-api.log
.../nova-compute.log
.../nova-conductor.log
and other log files.
Paste your suspected part of logs somewhere accessible to us. Check your AMPQ(RabbitMQ) service on your controller node. It is out of reach from your compute nodes. Maybe due to network connectivity or misconfiguration.
verify your /etc/hosts file for host addresses that it has correct mapping of ip address and hostname Also your nova.conf file does not contains mapping of glance_host = controller (as glance host is in controller node) (may be its not required)
please try
apt-get install nova-api
Add the following line in the
file : /etc/nova/api-paste.ini # EC2 # NOTE: this is now deprecated in favor of /stackforge/ec2-api [composite:ec2]
use = egg:Paste#urlmap
/: ec2cloud [composite:ec2cloud]
use = call:nova.api.auth:pipeline_factory
noauth2 = ec2faultwrap logrequest ec2noauth cloudrequest validator ec2executor
keystone = ec2faultwrap logrequest ec2keystoneauth cloudrequest validator ec2executor [filter:ec2faultwrap]
paste.filter_factory = nova.api.ec2:FaultWrapper.factory [filter:logrequest]
paste.filter_factory = nova.api.ec2:RequestLogging.factory [filter:ec2lockout]
paste.filter_factory = nova.api.ec2:Lockout.factory [filter:ec2keystoneauth]
paste.filter_factory = nova.api.ec2:EC2KeystoneAuth.factory [filter:ec2noauth]
paste.filter_factory = nova.api.ec2:NoAuth.factory [filter:cloudrequest]
controller = nova.api.ec2.cloud.CloudController
paste.filter_factory = nova.api.ec2:Requestify.factory [filter:authorizer]
paste.filter_factory = nova.api.ec2:Authorizer.factory [filter:validator]
paste.filter_factory = nova.api.ec2:Validator.factory [app:ec2executor]
paste.app_factory = nova.api.ec2:Executor.factory
Ok, in my case, while I am trying this on Ubuntu 14.04 and Juno on the below command I would get the status as below and the logs were frozen.
service nova-api start
nova-api stop/waiting
For some strange reason nova-api had gone missing, reinstalling it fixed the problem.
Your Answer
Please start posting anonymously - your entry will be published after you log in or create a new account.
Add Answer
Get to know Ask OpenStack
Question Tools
3 followers
Seen: 7,734 times
Last updated: Dec 12 '14
Related questions
Documentation
Branding & Legal
OpenStack is a trademark of OpenStack Foundation. This site is powered by . ( ). Content on this site is licensed under a
Please note: Ask OpenStack requires javascript to work properly, please enable javascript in your browser,post - Python Request - ConnectionError - Max retries exceeded - Stack Overflow
to customize your list.
Stack Overflow is a community of 4.7 million programmers, just like you, helping each other.
J it only takes a minute:
Join the Stack Overflow community to:
Ask programming questions
Answer and help your peers
Get recognized for your expertise
I am using rauth and requests to make calls to the Beatport API. The call works but I quite occasionaly get the following error ConnectionError: HTTPSConnectionPool(host='oauth-', port=443): Max retries exceeded with url
Here is the traceback.
Traceback (most recent call last):
File "&console&", line 1, in &module&
File "scraper/songlist_top100.py", line 88, in &module&
'sortBy': 'releaseDate ASC'})
File "C:\Python27\lib\site-packages\requests\sessions.py", line 347, in get
return self.request('GET', url, **kwargs)
File "C:\Python27\lib\site-packages\rauth\session.py", line 208, in request
return super(OAuth1Session, self).request(method, url, **req_kwargs)
File "C:\Python27\lib\site-packages\requests\sessions.py", line 335, in reques
resp = self.send(prep, **send_kwargs)
File "C:\Python27\lib\site-packages\requests\sessions.py", line 438, in send
r = adapter.send(request, **kwargs)
File "C:\Python27\lib\site-packages\requests\adapters.py", line 327, in send
raise ConnectionError(e)
ConnectionError: HTTPSConnectionPool(host='oauth-', port=443):
Max retries exceeded with url: /catalog/3/tracks?perPage=150&
oauth_nonce=xxxxx&oauth_timestamp=xxxxx&facets=artistName%3A
Avicii&oauth_signature_method=HMAC-SHA1&oauth_version=1.0&
oauth_consumer_key=xxxxx&oauth_token=xxxxxx&sortBy=releaseDate+ASC
&oauth_signature=xxxxx%3D&page=3 (Caused by &class 'httplib.BadStatusLine'&: '')
Here is my script
from rauth import OAuth1Service
import requests
from hunt.models import DJ, Song
def get_obj_or_none(model, **kwargs):
return model.objects.get(**kwargs)
except model.DoesNotExist:
return None
beatport_login = 'xxx'
beatport_pass = 'xxx'
beatport = OAuth1Service(
name='beatport',
consumer_key='xxxxx',
consumer_secret='xxxxx',
request_token_url= 'https://oauth-/identity/1/oauth/request-token',
access_token_url='https://oauth-/identity/1/oauth/access-token',
authorize_url='https://oauth-/identity/1/oauth/authorize',
base_url='https://oauth-/json/catalog')
request_token, request_token_secret = beatport.get_request_token(method='POST', data={
'oauth_callback': ''})
authorize_url = beatport.get_authorize_url(request_token)
values = {
'oauth_token': request_token,
'username': beatport_login,
'password': beatport_pass,
'submit' : 'Login',
r = requests.post('https://oauth-/identity/1/oauth/authorize-submit', data=values)
verifier = r.url.split("oauth_verifier=",1)[1]
tokens = beatport.get_raw_access_token(request_token, request_token_secret, method='POST', data={
'oauth_verifier': verifier})
token_string = tokens.content
access_token = token_string[token_string.find('=')+1:token_string.find('&')]
access_token_secret = token_string[token_string.find('t=')+2:token_string.rfind('&s')]
session = beatport.get_session((access_token, access_token_secret))
for dj in DJ.objects.all():
r = session.get('https://oauth-/catalog/3/tracks', params={'facets': "artistName:"+dj.name, 'perPage': 150})
count_response = r.json()
results = []
for i in range(1, count_response['metadata']['totalPages']+1):
r1 = session.get('https://oauth-/catalog/3/tracks', params={'facets': "artistName:"+dj.name,
'page': i,
'perPage': 150,
'sortBy': 'releaseDate ASC'})
json_response = r1.json()
results += json_response['results']
song_list = []
for song in results:
[artist['name'] for artist in song['artists'] if str(artist['type'])=='artist']
remixers = [artist['name'] for artist in song['artists'] if str(artist['type'])=='remixer']
if not ((dj.name in artists) and ((dj.name not in remixers) if len(remixers)&0 else False)):
song_list.append(song)
for song in song_list:
[artist['name'] for artist in song['artists'] if str(artist['type'])=='artist']
remixers = [artist['name'] for artist in song['artists'] if str(artist['type'])=='remixer']
artist_list = ', '.join(artists)
remixer_list = ', '.join(remixers)
song_name = song['name']
if not(song_name.lower().find("feat.") == -1 ):
normal_name=song_name[0:song_name.lower().find("feat.")].rstrip()
normal_name=song_name
genre_list=[]
for genre in song['genres']:
genre_list.append(genre['name'])
genres = ', '.join(genre_list)
if not get_obj_or_none(Song, name__iexact=song_name, artist=dj):
s = Song(song_id=song['id'],
name=song_name,
title=song['title'],
normalized_name=normal_name,
artist=dj,
artists=artist_list,
remixers=remixer_list,
release_date=song['releaseDate'],
slug=song['slug'],
artwork=song['images']['large']['url'],
genres=genres)
print "Added song:", s.song_id, s.artist
Why do I get the above mentioned error?
463k6211161286
It looks as if the Beatport API is overloaded and closes the connection prematurely sometimes. Your first set of requests succeeded just fine, it was page 3 that threw the error because the response is empty.
You really should report this to Beatport, but you could perhaps work around this issue by instructing the requests module to retry requests:
from requests.adapters import HTTPAdapter
session = beatport.get_session((access_token, access_token_secret))
session.mount('https://oauth-', HTTPAdapter(max_retries=5))
would retry your requests a few more times in case an error occurred.
463k6211161286
Your Answer
Sign up or
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Post as a guest
By posting your answer, you agree to the
Not the answer you're looking for?
Browse other questions tagged
Stack Overflow works best with JavaScript enabled