Not compatible with superagent 1.7.0 - 1.7.2

[jpodwys]

Superagent has made some internally breaking changes in version 1.7.0 and above. It's possible that I will adapt to these changes, but I'm currently holding off until the superagent team makes some decisions. If you'd like to follow the discussion, it can be found in this superagent issue.

[kornelski]

Can you explain what APIs do you need to make it work without peeking into superagent's internals?

[jpodwys]

Are you asking what public APIs I would need added in order to not have to evaluate superagent's prototypes and such?

In order to achieve what I'm doing with superagent-cache, I'm adding functions to/evaluating data from the Request prototype object.

[kornelski]

Yes, public or public-ish APIs. Basically, we need to agree on what we can change, and what needs to be maintained.

[jpodwys]

Awesome, I really like the idea of being able to get all the data I need from a set of public-ish APIs maintained by the superagent team. It's likely superagent needs all these functions internally anyhow, so perhaps it's simply a matter of making them accessible. Thanks for exploring this!

Here's a list of all (I think) of the things I would need superagent to be able to do and examples of where I currently do them:

• Attach functions to the Request prototype
• Get the current request's URI and REST method
• Get the current request's query params as a key value object
• Get the current request's headers as a key value object
• Get the current request's Request._redirectList
• A means of modifying the response object before superagent returns it. I implemented this as .prune(fn), and it looks like there is a PR on superagent to add .dataFilter(fn) for similar purposes
• Currently, I wrap superagent's .end() call with my own, and alias the original as ._end(). I don't know if there's a better way to do this so that I don't need to mimic certain behaviors such as recreating the redirect list or handling the callback function based on the number of provided arguments. Maybe this will just stay the same.

[jpodwys]

Just got a note that the issue fixed by the merged PR linked just above this comment has surfaced again when using superagent 1.7.2. If there isn't one yet, I need to add a unit test for this.

[jpodwys]

I believe I've resolved all incompatibilities relating to http headers in the 1-7-0-compatibility branch. I plan to merge it tomorrow.

[kornelski]

We've got toJSON now that serializes basic request properties: ladjs/superagent#906

I think we could add query string there too.

[jpodwys]

I've merged my branch and released 1.3.5 which resolves compatibility issues with > 1.6.1 while still maintaining backwards compatibility.

Moving/creating public-ish functions is great, but when I need to maintain backwards compatibility, I can't rely on those APIs. If superagent ever makes a jump to 2.x, I can consider using those APIs and no longer supporting 1.x, but I'm not sure of a better way to do things until then. I don't want to only support 1.8 and higher and I don't want to support two superagent-cache repos for different superagent version numbers.

I appreciate the desire to provide APIs so I don't have to touch superagent's internals. Still thinking a 2.x release is a long way out?

[jpodwys]

I made this issue's title more specific and am closing it.

[kornelski]

As for 2.x, I suppose we'll make it soon, as we're finding more and more stuff we may want to slightly change.

but what if we release 2.0 before it has public APIs that you need?

[jpodwys]

Of course, it would be nice to have all the public stuff (I'll just call it the developer API) at release but I'm sure we have some options. The best option I can think of is something I was already planning on which is to make superagent a peer-dependency in superagent-cache 2.x. That way, each of my releases can explicitly define which version range of superagent must be used for compatibility reasons.

However, while this makes cutting releases that have conflicting code a non-issue, it doesn't address the fact that, without a complete developer API, I would still need to make potentially frequent changes to my code until the developer API is complete. Some possible solutions to this include:

1. Waiting to support 2.x until it hits 2.1 or whatever release will have sufficient developer API support to do everything I need to do.
2. If the developer API is "good enough" when 2.x is released, meaning that I only need to evaluate a couple of minor superagent internals and we're able to agree that those will stay constant for the foreseeable future, then I could just go for it.

However, I do want to address the feasibility of the superagent team providing public APIs for every single thing I currently need and could need in the future. If we can make it work, I really want it to happen. However, I think it might be unreasonable to expect that the superagent team should have to code a solution/review and merge a PR and cut a new release any time I need a new API due to a superagent-cache feature request. Not only would it burden the superagent team, but it would also limit my freedom to release as often as I want.

Do you agree that this is an issue? If so, can you think of any solutions? Here's one idea, assuming that superagent will, in fact, attempt to house a complete developer API:

One thing that could contribute to holding up superagent releases I might need is merge conflicts. Perhaps having a completely separate repo consumed by superagent which is responsible for the developer API could make sense. This way we could separate code written exclusively to support superagent extensions from the main code body as well as allowing superagent to only need a package.json update to get the latest developer API thereby minimizing the likelihood of merge conflicts. Another benefit could be that, as long as we can silo this repo such that a code change to it will never break superagent, we can streamline PRs to it and associated superagent releases as well as opening up the collaborator's list a bit more to this repo than to the actual superagent repo. (Not sure how well this would work given that it could introduce code duplication since superagent would likely internally need many of the same functions contained in this repo. I imagine it's difficult to make a hard separation between internal and developer APIs.)

Anyhow, sorry for the essay! I'm really excited about a superagent developer API for both ease of integration and internal consistency, I just want to make sure we consider potential pain points. What are your thoughts?

[kornelski]

We could also add superagent-cache tests to our test suite, so that we know when we break stuff, and can decide what to do in each case.

[kornelski]

One thing I'm worried about that in order to provide a developer API, we'd need to make breaking changes, e.g. currently some state is set immediately when you set it, some is computed when you call .end(). We could make it consistent by computing everything in something like ._beforeEnd(), but it'd require moving setters' code, and some state currently privately visible wouldn't exist before _beforeEnd call.

[jpodwys]

If an official developer API is the goal, then pulling in unit tests for a number of superagent extensions would be really cool.

As for the concern over breaking changes, this is why I believe it's best to release a developer API as part of 2.x.