[python] 웹 요청 예시 (requests, HTTPAdapter, Retry)

간단 코드 예시

import requests


def main():
 print('Hello, world!')
 response = requests.get('https://httpbin.org/ip')
 print(response.status_code)
 print(response.headers)
 print('Your IP is {0}'.format(response.json()['origin']))
 
if __name__ == '__main__':
 main()

결과는 다음과 같다. 

Hello, world!

200

{'Connection': 'keep-alive', 'Server': 'gunicorn/19.8.1', 'Date': 'Mon, 04 Jun 2018 02:28:09 GMT', 'Content-Type': 'application/json', 'Content-Length': '26', 'Access-Control-Allow-Origin': '*', 'Access-Control-Allow-Credentials': 'true', 'Via': '1.1 vegur'}

Your IP is 1.1.1.1

HTTPAdapter를 이용하는 코드이다.

from requests import Session
from requests.adapters import HTTPAdapter


def main():
 print('Hello, world!')
 
 session = Session()
 session.mount("http://", HTTPAdapter(max_retries=3))
 response = session.get('https://httpbin.org/ip', timeout=0)
 
 print(response.status_code)
 print(response.headers)
 print('Your IP is {0}'.format(response.json()['origin']))
 
if __name__ == '__main__':
 main()

결과는 동일하다.

from requests import Session
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry


def main():
 print('Hello, world!')
 retries_number = 3
 backoff_factor = 0.3
 status_forcelist = (500, 400)
 
 retry = Retry(
  total=retries_number,
  read=retries_number,
  connect=retries_number,
  backoff_factor=backoff_factor,
  status_forcelist=status_forcelist,
 )
 session = Session()
 session.mount("http://", HTTPAdapter(max_retries=retry))
 response = session.get('https://httpbin.org/ip', timeout=0)
 
 print(response.status_code)
 print(response.headers)
 print('Your IP is {0}'.format(response.json()['origin']))
 
if __name__ == '__main__':
 main()

아래 공식에 따르면, 다음과 같다.  총 소요되는 시간은 1.8인데... 

{backoff factor} * (2 ^ ({number of total retries} - 1))

0.3 * ( 2 ^ ( 1 - 1)) = 0

0.3 * ( 2 ^ ( 2 - 1)) = 0.6

0.3 * ( 2 ^ ( 3 - 1)) = 1.2

1.8 = 0 + 0.6 + 1.2

https://urllib3.readthedocs.io/en/latest/reference/urllib3.util.html#module-urllib3.util.retry

backoff_factor (float) –

A backoff factor to apply between attempts after the second try (most errors are resolved immediately by a second try without a delay). urllib3 will sleep for:

{backoff factor} * (2 ^ ({number of total retries} - 1))

seconds. If the backoff_factor is 0.1, then sleep() will sleep for [0.0s, 0.2s, 0.4s, …] between retries. It will never be longer than Retry.BACKOFF_MAX.

By default, backoff is disabled (set to 0).

만약 타임아웃이 생기면, 중간에 쉬는 타임이 생긴다. retries와 timeout을 잘 사용하면 괜찮을 것 같다. 

response = session.get('https://httpbin.org/ip', timeout=5)