Mureq is a replacement for python-requests.¶

>>> mureq.get('https://clients3.google.com/generate_204')
Response(status_code=204)
>>> response = _; response.status_code
204
>>> response.headers['date']
'Sun, 26 Dec 2021 01:56:04 GMT'
>>> response.body
b''
>>> params={'snap': 'certbot', 'interface': 'content'}
>>> response = mureq.get('http://snapd/v2/connections', params=params,
    unix_socket='/run/snapd.socket')
>>> response.status_code
200
>>> response.headers['Content-type']
'application/json'
>>> response.body
b'{"type":"sync","status-code":200,"status":"OK","result":{"established":[],
"plugs":[],"slots":[]}}'
>>> response.json()
{'type': 'sync', 'status-code': 200, 'status': 'OK', 'result': {'established':
[], 'plugs': [], 'slots': []}}

Why?¶

In short: performance (memory consumption), security (resilience to supply-chain attacks), and simplicity.

Performance¶

python-requests is extremely memory-hungry, mainly due to large transitive dependencies like chardet https://github.com/chardet/chardet that are not needed by typical consumers. Here’s a simple benchmark using Python 3.9.7, as packaged by Ubuntu 21.10 for amd64:

python3 -c "import os; os.system('grep VmRSS /proc/' str(os.getpid()) + '/status')"
VmRSS:      7404 kb
python3 -c "import os, mureq; os.system('grep VmRSS /proc/' + str(os.getpid()) + '/status')"
VmRSS:     13304 kb
python3 -c "import os, mureq; mureq.get('https://www.google.com');
            os.system('grep VmRSS /proc/' + str(os.getpid()) + '/status')"
VmRSS:     15872 kb
python3 -c "import os, requests; os.system('grep VmRSS /proc/' + str(os.getpid()) + '/status')"
VmRSS:     21488 kb
python3 -c "import os, requests; requests.get('https://www.google.com');
            os.system('grep VmRSS /proc/' + str(os.getpid()) + '/status')"
VmRSS:     24352 kb

In terms of the time cost of HTTP requests, any differences between mureq and python-requests should be negligible, except in the case of workloads that use the connection pooling functionality of python-requests. Since mureq opens and closes a new connection for each request, migrating such a workload will incur a performance penalty. Note, however, that the normal python-requests API (requests.request, requests.get, etc.) also disables connection pooling, instead closing the socket immediately to prevent accidental resource leaks <https://github.com/psf/requests/blob/a1a6a549a0143d9b32717dbe3d75cd543ae5a4f6/requests/api.py#L57-L61>. In order to use connection pooling, you must explicitly create and manage a requests.Session <https://docs.python-requests.org/en/latest/user/advanced/#session-objects> object.

It’s unclear to me whether connection pooling even makes sense in the typical Python context (single-threaded synchronous I/O, where there’s no guarantee that the thread of control will reenter the connection pool). It is much easier to implement this correctly in Go (see: https://pkg.go.dev/net/http#Client).

Security¶

Together with its transitive dependencies, python-requests is tens of thousands of lines of third-party code that cannot feasibly be audited. The most common way of distributing python-requests and its dependencies is pypi.org <https://pypi.org/>, which has relatively weak security properties: as of late 2021 it supports ‘hash pinning, but not code signing <https://flawed.net.nz/2021/02/02/PyPI-Security-State/>`. Typical Python deployments with third-party dependencies are vulnerable to supply-chain attacks <https://en.wikipedia.org/wiki/Supply_chain_attack> against pypi.org, i.e., compromises of user credentials on pypi.org (or of pypi.org itself) that allow the introduction of malicious code into their dependencies.

In contrast, mureq is approximately 350 lines of code that can be audited easily and included directly in a project. Since mureq’s functionality is limited in scope, you should be able to “install” it and forget about it.

Simplicity¶

python-requests was an essential addition to the ecosystem when it was created in 2011, but that time is past, and now in many cases the additional complexity it introduces is no longer justified:

1. The standard library has caught up to python-requests in many respects. The most important change is PEP 476 https://www.python.org/dev/peps/pep-0476/, which began validating TLS certificates by default against the system trust store. This change has landed in every version of Python that still receives security updates. 2. Large portions of python-requests are now taken up with compatibility shims that cover EOL versions of Python, or that preserve compatibility with deprecated versions of the library itself. 3. python-requests and urllib3 have never actually handled the low-level HTTP mechanics specified in RFC 7230 https://datatracker.ietf.org/doc/html/rfc7230 and its predecessors; this has always been deferred to the standard library (http.client https://docs.python.org/3/library/http.client.html in Python 3, httplib https://docs.python.org/2/library/httplib.html in Python 2). This is why it’s so easy to reimplement the core functionality of python-requests in a small amount of code.

However, the API design of python-requests is excellent and in my opinion still considerably superior to that of urllib.request <https://docs.python.org/3/library/urllib.request.html> hence the case for a lightweight third-party library with a requests-like API.

How do I use mureq?¶

The core API (mureq.get, mureq.post, mureq.request, etc.) is similar to python-requests, with a few differences. For now, see the docstrings in mureq.py itself for documentation. HTML documentation will be released later if there’s a demand for it.

If you’re switching from python-requests, there are a few things to keep in mind:

1. mureq.get, mureq.post, and mureq.request mostly work like the analogous python-requests calls https://docs.python-requests. org/en/latest/user/quickstart/#make-a-request. 2. The response type is mureq.HTTPResponse, which exposes fewer methods and properties than requests.Response. In particular, it does not have text (since mureq doesn’t do any encoding detection). Instead, the response body is in the body member, which is always of type bytes. (For the sake of compatibility, the content property is provided as an alias for body.) 3. The default way to send a POST body is with the body kwarg, which only accepts bytes. 4. The json kwarg takes an arbitrary object, which is serialized to JSON, encoded as UTF-8, and sent as the request body with the usual Content-Type: application/json header. 5. To send a form-encoded POST body, use the form kwarg. This accepts a dictionary of key-value pairs, or any object that can be serialized by urllib. parse.urlencode https://docs.python.org/3/library/urllib.parse.html#urllib.parse. urlencode. It will add the usual Content-Type: application/x-www-form-urlencoded header. 6. To make a request without reading the entire body at once, use with mureq. yield_response(url, method, **kwargs). This yields a http.client.HTTPResponse https://docs.python.org/3/library/http.client.html#httpresponse-objects. Exiting the contextmanager automatically closes the socket. 7. mureq does not follow HTTP redirections by default. To enable them, use the kwarg max_redirects, which takes an integer number of redirects to allow, e.g. max_redirects=2. 8. mureq will throw a subclass of mureq.HTTPException (which is actually just http.client.HTTPException <https://docs.python.org/3/library/http.client.html#http.client.HTTPException>) for any runtime I/O error (including invalid HTTP responses, connection failures, timeouts, and exceeding the redirection limit). It may throw other exceptions (in particular ValueError) for programming errors, such as invalid or inconsistent arguments. 9. mureq supports two ways of making HTTP requests over a Unix domain stream socket: • The unix_socket kwarg, which overrides the hostname in the URL, e.g. mureq.get(’http://snapd/’, unix_socket=’/run/snapd.socket’) • The http+unix URL scheme, which take the percent-encoded path as the hostname, e.g. http+unix://%2Frun%2Fsnapd.socket/ to connect to /run/snapd.socket.

Module Contents¶

request(method: HTTPMethod, url: str, *, read_limit: int | None = None, **kwargs: Any) → Response[source]¶

Performs an HTTP request and reads the entire response body.

This function performs an HTTP request using the specified method and URL. It handles reading the response and returning it as a Response object.

Parameters:¶

method : HTTPMethod¶

The HTTP method to use for the request (e.g., ‘GET’, ‘POST’).

url : str¶

The URL to which the request is sent.

read_limit : Optional[int]¶

The maximum number of bytes to read from the response body. If None, the entire response body will be read. Defaults to None.

**kwargs : Any¶

Additional keyword arguments to be passed to the request.

Returns:¶

Response – An instance of the Response class containing the response data.

Raises:¶

HTTPException – If an HTTP error occurs during the request.

get(url: str, **kwargs: Any) → Response[source]¶

Get performs an HTTP GET request.

post(url: str, body: str | None = None, **kwargs: Any) → Response[source]¶

Post performs an HTTP POST request.

head(url: str, **kwargs: Any) → Response[source]¶

Head performs an HTTP HEAD request.

put(url: str, body: str | None = None, **kwargs: Any) → Response[source]¶

Put performs an HTTP PUT request.

patch(url: str, body: str | None = None, **kwargs: Any) → Response[source]¶

Patch performs an HTTP PATCH request.

delete(url: str, **kwargs: Any) → Response[source]¶

Delete performs an HTTP DELETE request.

yield_response(

method: HTTPMethod,

url: str,

*,

unix_socket: str | None = None,

timeout: float = DEFAULT_TIMEOUT,

headers: HTTPHeaders | None = None,

params: QueryType | None = None,

body: bytes | None = None,

form: QueryType | None = None,

json: object | None = None,

verify: bool = True,

source_address: str | tuple[str, int] | None = None,

max_redirects: int | None = None,

ssl_context: ssl.SSLContext | None = None,

) → collections.abc.Generator[http.client.HTTPResponse, Any, None][source]¶

Exposes the actual http.client.HTTPResponse.

This function is a low-level API that exposes the actual http.client.HTTPResponse via a context manager.

Note that unlike mureq.Response, http.client.HTTPResponse does not automatically canonicalize multiple appearances of the same header by joining them together with a comma delimiter.

To retrieve canonicalized headers from the response, use response.getheader(). see: https://docs.python.org/3/library/http.client.html#http.client.HTTPResponse.getheader

Parameters:¶

method : HTTPMethod¶

The HTTP method to request (e.g., ‘GET’, ‘POST’).

url : str¶

The URL to request.

unix_socket : str | None¶

Path to Unix domain socket to query, or None for a normal TCP request.

timeout : float¶

Timeout in seconds, or None for no timeout (default: 15 seconds).

headers : HTTPHeaders | None¶

HTTP headers as a mapping or list of key-value pairs.

params : QueryType | None¶

Parameters to be URL-encoded and added to the query string, as a mapping or list of key-value pairs.

body : bytes | None¶

Payload body of the request.

form : QueryType | None¶

Parameters to be form-encoded and sent as the payload body, as a mapping or list of key-value pairs.

json : object | None¶

Object to be serialized as JSON and sent as the payload body.

verify : bool¶

Whether to verify TLS certificates (default: True).

source_address : str | tuple[str, int] | None¶

Source address to bind to for TCP.

max_redirects : int | None¶

Maximum number of redirects to follow, or None (the default) for no redirection.

ssl_context : ssl.SSLContext | None¶

TLS config to control certificate validation, or None for default behavior.

Yields:¶

HTTPResponse – The actual HTTP response object.

Raises:¶

HTTPException – If an HTTP error occurs during the request.

class Response(url: str, status_code: int, headers: http.client.HTTPMessage, body: bytes)[source]¶

Response contains a completely consumed HTTP response.

Variables:¶

url : str¶

the retrieved URL, indicating whether redirection occurred

status_code : int¶

the HTTP status code

headers : http.client.HTTPMessage¶

the HTTP headers

body : bytes¶

the payload body of the response

property ok : bool¶

Returns whether the response had a successful status code.

(anything other than a 40x or 50x).

property content : bytes¶

Returns the response body (the body member).

This is an alias for compatibility with requests.Response.

url : str¶

status_code : int¶

headers : http.client.HTTPMessage¶

body : bytes¶

raise_for_status() → None[source]¶

Raise_for_status checks the response’s success code.

raising an exception for error codes.

json()[source]¶

Attempts to deserialize the response body as UTF-8 encoded JSON.

exception TooManyRedirects[source]¶

Extends: http.client.HTTPException

TooManyRedirects error.

TooManyRedirects is raised when automatic following of redirects was enabled, but the server redirected too many times without completing.

Initialize self. See help(type(self)) for accurate signature.