std/httpclient

    Dark Mode
Search:
  Source   Edit

This module implements a simple HTTP client that can be used to retrieve webpages and other data.

Retrieving a website

This example uses HTTP GET to retrieve http://google.com:

import std/httpclient
var client = newHttpClient()
echo client.getContent("http://google.com")

The same action can also be performed asynchronously, simply use the AsyncHttpClient:

import std/[asyncdispatch, httpclient]

proc asyncProc(): Future[string] {.async.} =
  var client = newAsyncHttpClient()
  return await client.getContent("http://example.com")

echo waitFor asyncProc()

The functionality implemented by HttpClient and AsyncHttpClient is the same, so you can use whichever one suits you best in the examples shown here.

Note: You need to run asynchronous examples in an async proc otherwise you will get an Undeclared identifier: 'await' error.

Using HTTP POST

This example demonstrates the usage of the W3 HTML Validator, it uses multipart/form-data as the Content-Type to send the HTML to be validated to the server.

var client = newHttpClient()
var data = newMultipartData()
data["output"] = "soap12"
data["uploaded_file"] = ("test.html", "text/html",
  "<html><head></head><body><p>test</p></body></html>")

echo client.postContent("http://validator.w3.org/check", multipart=data)

To stream files from disk when performing the request, use addFiles.

Note: This will allocate a new Mimetypes database every time you call it, you can pass your own via the mimeDb parameter to avoid this.

let mimes = newMimetypes()
var client = newHttpClient()
var data = newMultipartData()
data.addFiles({"uploaded_file": "test.html"}, mimeDb = mimes)

echo client.postContent("http://validator.w3.org/check", multipart=data)

You can also make post requests with custom headers. This example sets Content-Type to application/json and uses a json object for the body

import std/[httpclient, json]

let client = newHttpClient()
client.headers = newHttpHeaders({ "Content-Type": "application/json" })
let body = %*{
    "data": "some text"
}
let response = client.request("http://some.api", httpMethod = HttpPost, body = $body)
echo response.status

Progress reporting

You may specify a callback procedure to be called during an HTTP request. This callback will be executed every second with information about the progress of the HTTP request.

import std/[asyncdispatch, httpclient]

proc onProgressChanged(total, progress, speed: BiggestInt) {.async.} =
  echo("Downloaded ", progress, " of ", total)
  echo("Current rate: ", speed div 1000, "kb/s")

proc asyncProc() {.async.} =
  var client = newAsyncHttpClient()
  client.onProgressChanged = onProgressChanged
  discard await client.getContent("http://speedtest-ams2.digitalocean.com/100mb.test")

waitFor asyncProc()

If you would like to remove the callback simply set it to nil.

client.onProgressChanged = nil
Warning: The total reported by httpclient may be 0 in some cases.

SSL/TLS support

This requires the OpenSSL library. Fortunately it's widely used and installed on many operating systems. httpclient will use SSL automatically if you give any of the functions a url with the https schema, for example: https://github.com/.

You will also have to compile with ssl defined like so: nim c -d:ssl ....

Certificate validation is performed by default.

A set of directories and files from the ssl_certs module are scanned to locate CA certificates.

Example of setting SSL verification parameters in a new client:

import httpclient
var client = newHttpClient(sslContext=newContext(verifyMode=CVerifyPeer))

There are three options for verify mode:

  • CVerifyNone: certificates are not verified;
  • CVerifyPeer: certificates are verified;
  • CVerifyPeerUseEnvVars: certificates are verified and the optional environment variables SSL_CERT_FILE and SSL_CERT_DIR are also used to locate certificates

See newContext to tweak or disable certificate validation.

Timeouts

Currently only the synchronous functions support a timeout. The timeout is measured in milliseconds, once it is set any call on a socket which may block will be susceptible to this timeout.

It may be surprising but the function as a whole can take longer than the specified timeout, only individual internal calls on the socket are affected. In practice this means that as long as the server is sending data an exception will not be raised, if however data does not reach the client within the specified timeout a TimeoutError exception will be raised.

Here is how to set a timeout when creating an HttpClient instance:

import std/httpclient

let client = newHttpClient(timeout = 42)

Proxy

A proxy can be specified as a param to any of the procedures defined in this module. To do this, use the newProxy constructor. Unfortunately, only basic authentication is supported at the moment.

Some examples on how to configure a Proxy for HttpClient:

import std/httpclient

let myProxy = newProxy("http://myproxy.network")
let client = newHttpClient(proxy = myProxy)

Get Proxy URL from environment variables:

import std/httpclient

var url = ""
try:
  if existsEnv("http_proxy"):
    url = getEnv("http_proxy")
  elif existsEnv("https_proxy"):
    url = getEnv("https_proxy")
except ValueError:
  echo "Unable to parse proxy from environment variables."

let myProxy = newProxy(url = url)
let client = newHttpClient(proxy = myProxy)

Redirects

The maximum redirects can be set with the maxRedirects of int type, it specifies the maximum amount of redirects to follow, it defaults to 5, you can set it to 0 to disable redirects.

Here you can see an example about how to set the maxRedirects of HttpClient:

import std/httpclient

let client = newHttpClient(maxRedirects = 0)

Types

AsyncResponse = ref object
  version*: string
  status*: string
  headers*: HttpHeaders
  body: string
  bodyStream*: FutureStream[string]
  Source   Edit 
HttpClientBase[SocketType] = ref object
  socket: SocketType
  connected: bool
  currentURL: Uri            ## Where we are currently connected.
  headers*: HttpHeaders      ## Headers to send in requests.
  maxRedirects: Natural      ## Maximum redirects, set to `0` to disable.
  userAgent: string
  timeout*: int              ## Only used for blocking HttpClient for now.
  proxy: Proxy               ## `nil` or the callback to call when request progress changes.
  when SocketType is Socket:
    onProgressChanged
  else:
    onProgressChanged
  when false:
    sslContext
  contentTotal: BiggestInt
  contentProgress: BiggestInt
  oneSecondProgress: BiggestInt
  lastProgressReport: MonoTime
  when SocketType is AsyncSocket:
      bodyStream
      parseBodyFut

  else:
    bodyStream
  getBody: bool              ## When `false`, the body is never read in requestAux.
  Source   Edit 
HttpRequestError = object of IOError
Thrown in the getContent proc and postContent proc, when the server returns an error   Source   Edit 
MultipartData = ref object
  content: seq[MultipartEntry]
  Source   Edit 
MultipartEntries = openArray[tuple[name, content: string]]
  Source   Edit 
ProgressChangedProc[ReturnType] = proc (total, progress, speed: BiggestInt): ReturnType {.
    closure, ...gcsafe.}
  Source   Edit 
ProtocolError = object of IOError
exception that is raised when server does not conform to the implemented protocol   Source   Edit 
Proxy = ref object
  url*: Uri
  auth*: string
  Source   Edit 
Response = ref object
  version*: string
  status*: string
  headers*: HttpHeaders
  body: string
  bodyStream*: Stream
  Source   Edit

Consts

defUserAgent = "Nim httpclient/1.6.6"
  Source   Edit

Procs

proc `$`(data: MultipartData): string {....raises: [], tags: [].}
convert MultipartData to string so it's human readable when echo see https://github.com/nim-lang/Nim/issues/11863   Source   Edit 
proc `[]=`(p: MultipartData; name, content: string) {.inline,
    ...raises: [ValueError], tags: [].}
Add a multipart entry to the multipart data p. The value is added without a filename and without a content type.
data["username"] = "NimUser"
  Source   Edit 
proc `[]=`(p: MultipartData; name: string;
           file: tuple[name, contentType, content: string]) {.inline,
    ...raises: [ValueError], tags: [].}
Add a file to the multipart data p, specifying filename, contentType and content manually.
data["uploaded_file"] = ("test.html", "text/html",
  "<html><head></head><body><p>test</p></body></html>")
  Source   Edit 
proc add(p: MultipartData; name, content: string; filename: string = "";
         contentType: string = ""; useStream = true) {....raises: [ValueError],
    tags: [].}

Add a value to the multipart data.

When useStream is false, the file will be read into memory.

Raises a ValueError exception if name, filename or contentType contain newline characters.

  Source   Edit 
proc add(p: MultipartData; xs: MultipartEntries): MultipartData {.discardable,
    ...raises: [ValueError], tags: [].}
Add a list of multipart entries to the multipart data p. All values are added without a filename and without a content type.
data.add({"action": "login", "format": "json"})
  Source   Edit 
proc addFiles(p: MultipartData; xs: openArray[tuple[name, file: string]];
              mimeDb = newMimetypes(); useStream = true): MultipartData {.
    discardable, ...raises: [IOError, ValueError], tags: [ReadIOEffect].}
Add files to a multipart data object. The files will be streamed from disk when the request is being made. When stream is false, the files are instead read into memory, but beware this is very memory ineffecient even for small files. The MIME types will automatically be determined. Raises an IOError if the file cannot be opened or reading fails. To manually specify file content, filename and MIME type, use []= instead.
data.addFiles({"uploaded_file": "public/test.html"})
  Source   Edit 
proc body(response: AsyncResponse): Future[string] {.
    ...raises: [Exception, ValueError], tags: [RootEffect].}
Reads the response's body and caches it. The read is performed only once.   Source   Edit 
proc body(response: Response): string {....raises: [IOError, OSError],
                                        tags: [ReadIOEffect].}

Retrieves the specified response's body.

The response's body stream is read synchronously.

  Source   Edit 
proc close(client: HttpClient | AsyncHttpClient)
Closes any connections held by the HTTP client.   Source   Edit 
proc code(response: Response | AsyncResponse): HttpCode {.
    ...raises: [ValueError, OverflowDefect].}

Retrieves the specified response's HttpCode.

Raises a ValueError if the response's status does not have a corresponding HttpCode.

  Source   Edit 
proc contentLength(response: Response | AsyncResponse): int

Retrieves the specified response's content length.

This is effectively the value of the "Content-Length" header.

A ValueError exception will be raised if the value is not an integer.

  Source   Edit 
proc contentType(response: Response | AsyncResponse): string {.inline.}

Retrieves the specified response's content type.

This is effectively the value of the "Content-Type" header.

  Source   Edit 
proc delete(client: AsyncHttpClient; url: Uri | string): Future[AsyncResponse]
Connects to the hostname specified by the URL and performs a DELETE request. This procedure uses httpClient values such as client.maxRedirects.   Source   Edit 
proc delete(client: HttpClient; url: Uri | string): Response
  Source   Edit 
proc deleteContent(client: AsyncHttpClient; url: Uri | string): Future[string]
Connects to the hostname specified by the URL and returns the content of a DELETE request.   Source   Edit 
proc deleteContent(client: HttpClient; url: Uri | string): string
  Source   Edit 
proc downloadFile(client: AsyncHttpClient; url: Uri | string; filename: string): Future[
    void]
  Source   Edit 
proc downloadFile(client: HttpClient; url: Uri | string; filename: string)
Downloads url and saves it to filename.   Source   Edit 
proc get(client: AsyncHttpClient; url: Uri | string): Future[AsyncResponse]

Connects to the hostname specified by the URL and performs a GET request.

This procedure uses httpClient values such as client.maxRedirects.

  Source   Edit 
proc get(client: HttpClient; url: Uri | string): Response
  Source   Edit 
proc getContent(client: AsyncHttpClient; url: Uri | string): Future[string]
Connects to the hostname specified by the URL and returns the content of a GET request.   Source   Edit 
proc getContent(client: HttpClient; url: Uri | string): string
  Source   Edit 
proc getSocket(client: AsyncHttpClient): AsyncSocket {.inline, ...raises: [],
    tags: [].}
  Source   Edit 
proc getSocket(client: HttpClient): Socket {.inline, ...raises: [], tags: [].}

Get network socket, useful if you want to find out more details about the connection

this example shows info about local and remote endpoints

if client.connected:
  echo client.getSocket.getLocalAddr
  echo client.getSocket.getPeerAddr
  Source   Edit 
proc head(client: AsyncHttpClient; url: Uri | string): Future[AsyncResponse]

Connects to the hostname specified by the URL and performs a HEAD request.

This procedure uses httpClient values such as client.maxRedirects.

  Source   Edit 
proc head(client: HttpClient; url: Uri | string): Response
  Source   Edit 
proc lastModified(response: Response | AsyncResponse): DateTime

Retrieves the specified response's last modified time.

This is effectively the value of the "Last-Modified" header.

Raises a ValueError if the parsing fails or the value is not a correctly formatted time.

  Source   Edit 
proc newAsyncHttpClient(userAgent = defUserAgent; maxRedirects = 5;
                        sslContext = getDefaultSSL(); proxy: Proxy = nil;
                        headers = newHttpHeaders()): AsyncHttpClient {.
    ...raises: [], tags: [].}

Creates a new AsyncHttpClient instance.

userAgent specifies the user agent that will be used when making requests.

maxRedirects specifies the maximum amount of redirects to follow, default is 5.

sslContext specifies the SSL context to use for HTTPS requests.

proxy specifies an HTTP proxy to use for this HTTP client's connections.

headers specifies the HTTP Headers.

Example:

import std/[asyncdispatch, strutils]

proc asyncProc(): Future[string] {.async.} =
  let client = newAsyncHttpClient()
  result = await client.getContent("http://example.com")

let exampleHtml = waitFor asyncProc()
assert "Example Domain" in exampleHtml
assert "Pizza" notin exampleHtml
  Source   Edit 
proc newHttpClient(userAgent = defUserAgent; maxRedirects = 5;
                   sslContext = getDefaultSSL(); proxy: Proxy = nil;
                   timeout = -1; headers = newHttpHeaders()): HttpClient {.
    ...raises: [], tags: [].}

Creates a new HttpClient instance.

userAgent specifies the user agent that will be used when making requests.

maxRedirects specifies the maximum amount of redirects to follow, default is 5.

sslContext specifies the SSL context to use for HTTPS requests. See SSL/TLS support

proxy specifies an HTTP proxy to use for this HTTP client's connections.

timeout specifies the number of milliseconds to allow before a TimeoutError is raised.

headers specifies the HTTP Headers.

Example:

import std/strutils

let exampleHtml = newHttpClient().getContent("http://example.com")
assert "Example Domain" in exampleHtml
assert "Pizza" notin exampleHtml
  Source   Edit 
proc newMultipartData(): MultipartData {.inline, ...raises: [], tags: [].}
Constructs a new MultipartData object.   Source   Edit 
proc newMultipartData(xs: MultipartEntries): MultipartData {.
    ...raises: [ValueError], tags: [].}
Create a new multipart data object and fill it with the entries xs directly.
var data = newMultipartData({"action": "login", "format": "json"})
  Source   Edit 
proc newProxy(url: string; auth = ""): Proxy {....raises: [], tags: [].}
Constructs a new TProxy object.   Source   Edit 
proc newProxy(url: Uri; auth = ""): Proxy {....raises: [], tags: [].}
Constructs a new TProxy object.   Source   Edit 
proc patch(client: AsyncHttpClient; url: Uri | string; body = "";
           multipart: MultipartData = nil): Future[AsyncResponse]
Connects to the hostname specified by the URL and performs a PATCH request. This procedure uses httpClient values such as client.maxRedirects.   Source   Edit 
proc patch(client: HttpClient; url: Uri | string; body = "";
           multipart: MultipartData = nil): Response
  Source   Edit 
proc patchContent(client: AsyncHttpClient; url: Uri | string; body = "";
                  multipart: MultipartData = nil): Future[string]
Connects to the hostname specified by the URL and returns the content of a PATCH request.   Source   Edit 
proc patchContent(client: HttpClient; url: Uri | string; body = "";
                  multipart: MultipartData = nil): string
  Source   Edit 
proc post(client: AsyncHttpClient; url: Uri | string; body = "";
          multipart: MultipartData = nil): Future[AsyncResponse]
Connects to the hostname specified by the URL and performs a POST request. This procedure uses httpClient values such as client.maxRedirects.   Source   Edit 
proc post(client: HttpClient; url: Uri | string; body = "";
          multipart: MultipartData = nil): Response
  Source   Edit 
proc postContent(client: AsyncHttpClient; url: Uri | string; body = "";
                 multipart: MultipartData = nil): Future[string]
Connects to the hostname specified by the URL and returns the content of a POST request.   Source   Edit 
proc postContent(client: HttpClient; url: Uri | string; body = "";
                 multipart: MultipartData = nil): string
  Source   Edit 
proc put(client: AsyncHttpClient; url: Uri | string; body = "";
         multipart: MultipartData = nil): Future[AsyncResponse]
Connects to the hostname specified by the URL and performs a PUT request. This procedure uses httpClient values such as client.maxRedirects.   Source   Edit 
proc put(client: HttpClient; url: Uri | string; body = "";
         multipart: MultipartData = nil): Response
  Source   Edit 
proc putContent(client: AsyncHttpClient; url: Uri | string; body = "";
                multipart: MultipartData = nil): Future[string]
Connects to the hostname specified by the URL andreturns the content of a PUT request.   Source   Edit 
proc putContent(client: HttpClient; url: Uri | string; body = "";
                multipart: MultipartData = nil): string
  Source   Edit 
proc request(client: AsyncHttpClient; url: Uri | string;
             httpMethod: HttpMethod | string = HttpGet; body = "";
             headers: HttpHeaders = nil; multipart: MultipartData = nil): Future[
    AsyncResponse]

Connects to the hostname specified by the URL and performs a request using the custom method string specified by httpMethod.

Connection will be kept alive. Further requests on the same client to the same hostname will not require a new connection to be made. The connection can be closed by using the close procedure.

This procedure will follow redirects up to a maximum number of redirects specified in client.maxRedirects.

You need to make sure that the url doesn't contain any newline characters. Failing to do so will raise AssertionDefect.

headers are HTTP headers that override the client.headers for this specific request only and will not be persisted.

Deprecated since v1.5: use HttpMethod enum instead; string parameter httpMethod is deprecated

  Source   Edit 
proc request(client: HttpClient; url: Uri | string;
             httpMethod: HttpMethod | string = HttpGet; body = "";
             headers: HttpHeaders = nil; multipart: MultipartData = nil): Response
  Source   Edit

Made with Nim. Generated: 2022-05-05 01:21:40 UTC