간단 코드 예시


import requests


def main():
print('Hello, world!')
response = requests.get('https://httpbin.org/ip')
print(response.status_code)
print(response.headers)
print('Your IP is {0}'.format(response.json()['origin']))

if __name__ == '__main__':
main()



결과는 다음과 같다. 


Hello, world!

200

{'Connection': 'keep-alive', 'Server': 'gunicorn/19.8.1', 'Date': 'Mon, 04 Jun 2018 02:28:09 GMT', 'Content-Type': 'application/json', 'Content-Length': '26', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Credentials': 'true', 'Via': '1.1 vegur'}

Your IP is 1.1.1.1





HTTPAdapter를 이용하는 코드이다.


from requests import Session
from requests.adapters import HTTPAdapter


def main():
print('Hello, world!')

session = Session()
session.mount("http://", HTTPAdapter(max_retries=3))
response = session.get('https://httpbin.org/ip', timeout=0)

print(response.status_code)
print(response.headers)
print('Your IP is {0}'.format(response.json()['origin']))

if __name__ == '__main__':
main()




결과는 동일하다.




from requests import Session
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry


def main():
print('Hello, world!')
retries_number = 3
backoff_factor = 0.3
status_forcelist = (500, 400)

retry = Retry(
total=retries_number,
read=retries_number,
connect=retries_number,
backoff_factor=backoff_factor,
status_forcelist=status_forcelist,
)
session = Session()
session.mount("http://", HTTPAdapter(max_retries=retry))
response = session.get('https://httpbin.org/ip', timeout=0)

print(response.status_code)
print(response.headers)
print('Your IP is {0}'.format(response.json()['origin']))

if __name__ == '__main__':
main()



아래 공식에 따르면, 다음과 같다.  총 소요되는 시간은 1.8인데... 


{backoff factor} * (2 ^ ({number of total retries} - 1))



0.3 * ( 2 ^ ( 1 - 1)) = 0

0.3 * ( 2 ^ ( 2 - 1)) = 0.6

0.3 * ( 2 ^ ( 3 - 1)) = 1.2



1.8 = 0 + 0.6 + 1.2






https://urllib3.readthedocs.io/en/latest/reference/urllib3.util.html#module-urllib3.util.retry


backoff_factor (float) –

A backoff factor to apply between attempts after the second try (most errors are resolved immediately by a second try without a delay). urllib3 will sleep for:

{backoff factor} * (2 ^ ({number of total retries} - 1))

seconds. If the backoff_factor is 0.1, then sleep() will sleep for [0.0s, 0.2s, 0.4s, …] between retries. It will never be longer than Retry.BACKOFF_MAX.

By default, backoff is disabled (set to 0).






만약 타임아웃이 생기면, 중간에 쉬는 타임이 생긴다. retries와 timeout을 잘 사용하면 괜찮을 것 같다. 


response = session.get('https://httpbin.org/ip', timeout=5)

Posted by 김용환 '김용환'