Multipart form data

[kamcma]

I'd love to see if anyone has taken a stab at implementing a multipart/form-data parser-printer, either ad-hoc or generally. I love the idea of swift-url-routing as a type-safe API client, but it's not too long before you run into the need to make multipart/form-data requests.

I figure unlike the provided (application/x-www-form-urlencoded) FormData parser, which is merely a parser of Data, a multipart parser would need to be a parser of all of URLRequestData, because it needs access to the body and headers, to capture the boundary string.

Speaking of capturing the boundary string, I'm not quite sure how to to do that, for use in parsing the separator in the body.

Anyway, just thought I'd throw up a discussion and see if anyone has anything to give me a head start. If I make headway, I'll follow up.

[kamcma]

Well, it's a total punt on actually parsing a multipart/form-data body, but, if I'm willing to take on a dependency such as Vapor's multipart-kit, then I can ape swift-parsing's JSON conversion like so:

import Foundation
import MultipartKit
import NIOFoundationCompat

extension Conversions {
  struct Multipart<Value: Codable>: Conversion {
    let boundary: String
    let decoder: FormDataDecoder
    let encoder: FormDataEncoder

    init(
      _ type: Value.Type,
      boundary: String,
      decoder: FormDataDecoder,
      encoder: FormDataEncoder
    ) {
      self.boundary = boundary
      self.decoder = decoder
      self.encoder = encoder
    }

    public func apply(_ input: Data) throws -> Value {
      try self.decoder.decode(Value.self, from: ByteBuffer(data: input), boundary: boundary)
    }

    public func unapply(_ output: Value) throws -> Data {
      var buffer = ByteBuffer()
      try self.encoder.encode(output, boundary: self.boundary, into: &buffer)
      return Data(buffer: buffer)
    }
  }
}

extension Conversion {
  static func multipart<Value>(
    _ type: Value.Type,
    boundary: String,
    decoder: FormDataDecoder = .init(),
    encoder: FormDataEncoder = .init()
  ) -> Self where Self == Conversions.Multipart<Value> {
    .init(type, boundary: boundary, decoder: decoder, encoder: encoder)
  }

  func multipart<Value>(
    _ type: Value.Type,
    boundary: String,
    decoder: FormDataDecoder = .init(),
    encoder: FormDataEncoder = .init()
  ) -> Conversions.Map<Self, Conversions.Multipart<Value>> {
    self.map(.multipart(type, boundary: boundary, decoder: decoder, encoder: encoder))
  }
}

Which then allows me to implement something like the following. I think flatMap is the capturing mechanism I was searching for. The loss of invertibility from flatMap's one-way operation, and recovery with .printing(), feels correct, because when parsing, the boundary string is ultimately discarded, but when printing, you need to provide some boundary value. I'm fairly certain this can be DRYed out, I was hitting weird compiler errors when I tried to do so (user-error, I'm sure), but this ugly version at least gets my meaning across:

struct Multipart<Value: Codable>: ParserPrinter {
  let boundary: String

  init(
    _ type: Value.Type,
    printingBoundary boundary: String
  ) {
    self.boundary = boundary
  }

  var body: some ParserPrinter<URLRequestData, Value> {
    Headers {
      Field("Content-Type") {
        "multipart/form-data;boundary=\""
        Prefix { $0 != "\"" }.map(.string)
        "\""
      }
    }
    .flatMap { boundary in
      Body(.multipart(Value.self, boundary: boundary))
    }
    .printing(self.printing)

  }

  var printing: some ParserPrinter<URLRequestData, Value> {
    ParsePrint {
      Skip {
        Headers {
          Field("Content-Type") {
            "multipart/form-data;boundary=\""
            self.boundary
            "\""
          }
        }
      }
      Body(.multipart(Value.self, boundary: self.boundary))
    }
  }
}

I may continue to take passes at this, because taking a transitive dependency on swift-nio just to parse/print multipart/form-data bodies is bit of a bummer for me (though absolutely fine for Vapor, because their whole ecosystem speaks that language), but for the moment, this is a start.

[stephencelis]

@kamcma This is great, thanks for sharing! With the general infrastructure you have in place you can probably eventually shed off the Vapor dependency and speed up your build. Please do share any further iteration!

[lukeredpath]

I'm also interested in this - let me know if I can help at all.

[kamcma]

Alright, so I have iterated on this a bit.

What was stumping me for quite a while was the shift from converting all of body at once to parsing it incrementally, since it is Data. I thought I'd be able to achieve this with the stock conversions, and I still wouldn't be surprised if someone shows me it's possible, but I'll be darned if I could find the right pieces to make the compiler happy. Eventually it occurred to me I might need to implement a non-provided conversion:

extension Conversions {
  struct _BytesToUTF8<Input: PrependableCollection>: Conversion where Input.SubSequence == Input, Input.Element == UTF8.CodeUnit {
    func apply(_ input: Input) throws -> Substring.UTF8View {
      Substring.UTF8View(input)
    }

    func unapply(_ output: Substring.UTF8View) throws -> Input {
      .init(output)
    }
  }
}

With that in place, I can implement this ad-hoc multipart/form-data body parser for the sample multipart/form-data body from the MDN docs. I'm ignoring the header bits for now and just focusing on parsing the body data:

/*
POST /test HTTP/1.1
Host: foo.example
Content-Type: multipart/form-data;boundary="boundary"

--boundary
Content-Disposition: form-data; name="field1"

value1
--boundary
Content-Disposition: form-data; name="field2"; filename="example.txt"

value2
--boundary--

*/

public struct _MultipartBody: ParserPrinter {
  public init() {}

  public var body: some ParserPrinter<Data, (String, String, String)> {
    From(Conversions._BytesToUTF8()) {
      "--boundary".utf8; Whitespace(1, .vertical)
      "Content-Disposition: form-data; name=\"field1\"".utf8; Whitespace(2, .vertical)
      From(.substring) {
        Prefix { !$0.isNewline }.map(.string)
      }
      Whitespace(1, .vertical)
      "--boundary".utf8; Whitespace(1, .vertical)
      "Content-Disposition: form-data; name=\"field2\"; filename=\"".utf8
      From(.substring) {
        Prefix { $0 != "\"" }.map(.string)
      }
      "\"".utf8; Whitespace(2, .vertical)
      From(.substring) {
        Prefix { !$0.isNewline }.map(.string)
      }
      Whitespace(1, .vertical)
      "--boundary--".utf8; Whitespace(1, .vertical)
    }
  }
}

Which passes the following test:

class MultipartTests: XCTestCase {
  func testParseMultipartBody() throws {

    let sample = """
--boundary
Content-Disposition: form-data; name="field1"

value1
--boundary
Content-Disposition: form-data; name="field2"; filename="example.txt"

value2
--boundary--

"""

    var body = Data(sample.utf8)
    let (field1, filename, field2) = try _MultipartBody().parse(&body)
    XCTAssertEqual("value1", field1)
    XCTAssertEqual("example.txt", filename)
    XCTAssertEqual("value2", field2)
    XCTAssertEqual(Data("".utf8), body)

    XCTAssertEqual(
      try _MultipartBody().print(("value1", "example.txt", "value2")),
      Data(sample.utf8)
    )
  }
}

So, with this Data-conversion piece, and the header-parsing boundary string capture from my previous attempt, I think it's possible to implement an ad hoc parser-printer for a multipart/form-data URLRequestData.

As far as generalizing it, that will take some more thought. The data structure of multipart/form-data is kinda key-value pairs, but then some parts have nested key-value pairs, like file names and MIME types. My own use case only has a few multipart/form-data routes, so ad hoc implementation in the meantime might be acceptable.

But also, because multipart/form-data requests often contain whole files, we might be interested streaming those request bodies, in both the parsing and printing directions. Even if swift-parsing's Stream parser were a printer, which it is not, URLRequestData's body is Data?, and doesn't have an analogous property to URLRequest's httpBodyStream: InputStream? property. So perhaps the strategy for the time being is to use swift-url-routing to prepare all of a request except for the body, and then manually attach the body stream using Foundation directly. 🤔 (I happen to be implementing an API client, which is why I'm thinking about the printing direction in particular.)

[lukeredpath]

I've had a go at trying to generalise this. My approach has been to parse the body in two stages - first parse the raw URLRequestData body into multiple parts, with each part represented as a PartData value, analogous to URLRequestData i.e. a structure better suited to parsing each part into something more domain-specific. PartData has headers (using the built-in Fields type) and a var body: Data.

I have a Multipart parser that extracts the boundary from the content-type header, uses that to parse out the part data and then passes that into a part parser that parses each part accordingly. As an example, you end up with something like this:

Route(.case(TestRoute.multipart)) {
  Method.post
  Path { "upload" }
  Multipart(boundary: "abcde12345") {
    Part {
      PartHeaders {
        Field("Content-Type") { "text/plain" }
      }
      PartBody(DataToSubstring(encoding: .utf8).string)
    }
    Part {
      PartHeaders {
        Field("Content-Type") { "application/json" }
      }
      PartBody(.json(Model.self))
    }
  }
}

The only weird thing is that you still need to provide some boundary value to the multipart parser, but it's only used when printing. This is because even though the boundary value can be parsed from the headers when parsing, this value is discarded after parsing the body into separate parts, so that value is lost. However, it doesn't seem to really matter what the boundary value is when printing as long as its something that isn't part of your content so you can supply any boundary value that makes sense when printing. It does seem like a bit of a strange API though.

I'm sure there's a lot that can be improved about my implementation but I've pushed up my WIP here:
main...lukeredpath:swift-url-routing:multipart

[kamcma]

That's really nice! Thanks for sharing.

I think the bit about the boundary string only being needed for printing actually makes sense, given how multipart works. Some multipart libraries even provide a default value, a UUID or some basic string, so the consumer doesn't even have to provide one if they have no constraint on what it should be (eg, if some web service were to constrain its value when maybe it really shouldn't).

I read somewhere the spec says newlines should specifically be CRLF, so maybe we could be even more specific than the Whitespace parser.

I still would love to solve the streaming issue, but given the current design of URLRequestData, this seems pretty close to ideal.

[lukeredpath]

Just circling back to this thread to let you know that we've finally open sourced out multipart routing extensions https://github.com/Shimmur/swift-url-routing-multipart