How to use cowboy with ranch 2.1.0?

[cgrothaus]

Question

What must I do to use cowboy with ranch 2.1.0? And is this a good idea at all?

Background

On a standard phoenix 1.6 app, I recently ran mix hex.outdated --all, which featured a line

Dependency              Current  Latest  Status               
ranch                   1.8.0    2.1.0   Update not possible  

Running mix hex.outdated ranch shows that it is because of cowboy:

There is newer version of the dependency available 2.1.0 > 1.8.0!

Source  Requirement  Up-to-date  
cowboy  1.8.0        No          

So apparently the standard (transitive) dependencies of cowboy are holding back ranch at 1.8.0. How to resolve this?

[essen]

Please ask in an Elixir related forum I don't know how to do that with mix. There's nothing stopping you from using Ranch 2.1 but Cowboy has to depend on 1.x until Cowboy reaches 3.0.

[epinault]

@essen The issue is that it seems that some package are dependent on Ranch 2.1, and cowboy seems to say 1.8. So we have conflicts now

Because cowboy >= 2.9.0 depends on ranch 1.8.0 and rabbit_common >= 3.13.2-rc.1 depends on ranch 2.1.0, cowboy >= 2.9.0 is incompatible with rabbit_common >= 3.13.2-rc.1.
And because your app depends on cowboy ~> 2.12.0, rabbit_common >= 3.13.2-rc.1 is forbidden.
So, because your app depends on rabbit_common ~> 3.13.4, version solving failed.

While I could use some form of override, what is the consequence of using latest cowboy and 2.x?

More context is that using OTP 27 and Elixir 1.17, rabbit_common uses. Ranch 2.x... So Cowboy is now holding back everything that uses Rabbit... since to move to OTP27 you must use a new version that depends on Ranch 2.x...

[essen]

Cowboy can't switch to a new major release of a dependency without increasing Cowboy's version to 3.0, which isn't going to happen for a while as the features that I want in Cowboy 3.0 are not yet ready.

[cgrothaus]

How about splitting that into two Cowboy versions (v3 for dependency upgrades like ranch, v4 for the new features that you mention)?

[belltoy]

Seems it's because the ranch dependency in hex_metadata.config is fixed to 1.8.0.

The cowboy is compatible with both ranch 1.7 and 2.0 since cowboy 2.7, and update ranch dep to 1.8.0 in cowboy 2.9.

To fix the "conflict" or support compatibility, the ranch dependency should be specified to ~> 1.8 or ~> 2.0.

However, I'm not familiar with erlang.mk. Instead, I use rebar3 to do this.

@epinault you can use this to see if it works in your environment.

[belltoy]

@essen Is it possible to support ~> 1.8 or ~> 2.0 in the generated hex_metadata.config via erlang.mk?

[epinault]

this a problem for many of the package that depends on cowboy. now I am seeing also esaml depends on cowboy, and bring older vesion of ranch. we definitely need a solution as a this blocking and causing dependency conflict

@belltoy that s a neat idea. Just seems like it will be a pain to maintain long term. And while I migrated to bandit at this point, there are still lot of packages that depends on cowboy that bring that similar issue. if @essen could relax a bit the requirement per @belltoy , this would help us

[essen]

A patch to Erlang.mk is necessary to override the version when generating the metadata. It should be simple enough, just needs someone to work on it.

[epinault]

hmm yea it s 7000+ lines in the makefile. But only reference I see for the version is

dep_ranch = git https://github.com/ninenines/ranch 1.8.0

I tried to put a range or different spec, and it breaks . It seems that we definitely need to patch somewhere but that a little hard to parse what needs to happen in those 7000 lines.

[essen]

You are looking at the built file, not the source. The Hex related source is at https://github.com/ninenines/erlang.mk/blob/master/plugins/hex.mk

[epinault]

seems like you wrote most of it and that quite a lot for someone not familiar with this project. any chance you could help with this? merci d'avance.

[essen]

It's not a big file if you know Make. The place to do the change is probably this line https://github.com/ninenines/erlang.mk/blob/master/plugins/hex.mk#L125 which is basically a Make-based template of an Erlang file which is run to produce the metadata.

I am currently doing Erlang.mk work so maybe I'll have time to do this one.

[epinault]

thank you!

[essen]

If I set it to ~> 1.8 will it really accept 2.x Ranch versions? I'm reading otherwise.

[essen]

OK seems that I want >= 1.8.

[essen]

Can someone please confirm that this is expected to work?

{<<"requirements">>,
 [{<<"cowlib">>,
   [{<<"app">>,<<"cowlib">>},
    {<<"optional">>,false},
    {<<"requirement">>,<<"~= 2.13.0">>}]},
  {<<"ranch">>,
   [{<<"app">>,<<"ranch">>},
    {<<"optional">>,false},
    {<<"requirement">>,<<">= 1.8.0 and < 3.0.0">>}]}]}.

[epinault]

the requirement seems good to me.

[essen]

Is setting the requirement in the metadata enough or will it fail when Cowboy is fetched and then its Makefile/rebar.config contain a dependency on Ranch 1.8.0 using git? Does Hex basically require everything to fetch from Hex to conform to its version shenanigans?

[essen]

Or Mix, if that's Mix complaining in that case.

[essen]

Basically if hex_metadata says #1591 (comment)

But Cowboy Makefile/rebar.config say Ranch 1.8.0

And amqp_client depends on Ranch 2.1.0

Will there still be an issue in a Mix project that depends on both Cowboy and amqp_client?

Meaning it's not just the Hex metadata that needs to use this version format, but also Makefile/rebar.config?

@wojtekmach Do you know?

[essen]

Right from what I'm reading I think it's deeper than just Hex. The problem isn't just that the Hex metadata is too strict, but that both Cowboy and RabbitMQ do version pinning, and that the Elixir world doesn't like / play well with version pinning.

So even if I release, say, Cowboy 3.0 which depends on upcoming Ranch 2.2, Mix will still not be happy because RabbitMQ will not depend on Ranch 2.2 until a new version is released.

The way the Elixir world does things is not fully compatible with the way the Erlang world does things (first dependency fetched wins, both Rebar3 and Erlang.mk do this) and so these issues are bound to keep happening.

I think the only way to solve this is to detect Mix and use the Hex version strings in this case for the version, but if Mix calls Make to build Cowboy this isn't possible right now because we don't have the dependency version resolver available in hex_core (or I've not seen it). If instead it uses the rebar.config then I guess a rebar.config.script could replace the versions with the Hex strings when it detects Mix, and let Mix handle it.

A sample Elixir project that uses both cowboy and amqp_client would be a great help.

[wojtekmach]

hex_core does not ship with dependency resolver. That would be a very welcome addition but there are no current plans.

Yes, the hex resolver tries to find the newest version that is compatible with all requirements and if that is not possible (say package1 depends on package3 =1.0.0 but package2 on =1.0.1) it errors and users need to use overrides.

It's easy to reproduce this error:

$ iex
iex> Mix.install [{:cowboy, "2.12.0"}, {:amqp_client, "4.0.3"}]
Resolving Hex dependencies...
Resolution completed in 0.123s
Because amqp_client >= 4.0.3 depends on rabbit_common 4.0.3 which depends on ranch 2.1.0, amqp_client >= 4.0.3 requires ranch 2.1.0.
And because cowboy >= 2.9.0 depends on ranch 1.8.0, amqp_client >= 4.0.3 is incompatible with cowboy >= 2.9.0.
And because your app depends on amqp_client 4.0.3, cowboy >= 2.9.0 is forbidden.
So, because your app depends on cowboy 2.12.0, version solving failed.

and it can be solved with overriding the conflicting dependency, here ranch:

iex> Mix.install [{:cowboy, "2.12.0"}, {:amqp_client, "4.0.3"}, {:ranch, "~> 2.1.0", override: true}]
Resolving Hex dependencies...
Resolution completed in 0.017s
New:
  amqp_client 4.0.3
  cowboy 2.12.0
  cowlib 2.13.0
  credentials_obfuscation 3.4.0
  rabbit_common 4.0.3
  ranch 2.1.0
  recon 2.5.6
  thoas 1.2.1

[essen]

@wojtekmach Thank you. What would happen if Cowboy included >= 1.8.0 and < 3.0.0 in its requirement for Ranch, but then its Makefile/rebar.config would depend explicitly on 1.8.0 using git? Is there additional dependency version checks in Mix post-fetch or will it then be "first fetch wins"?

[wojtekmach]

Not sure I'm fully understanding the question. If we publish, say, Cowboy 2.13.0 with dependency on ranch >= 1.8.0 and < 3.0.0 and then on another project we try to use cowboy, Mix/Hex only looks at the hex registry for available versions to consider (does not look into how the package was built, what's in Makefile/rebar.config, etc). Once the resolver picks a version (possibly with the assistance of the user that specified overrides) then that's that, there's no extra step that could augment the picked versions.

[essen]

Right I understand now. So it should be OK to continue pinning in Makefile/rebar.config, while still allowing Mix projects to use version ranges. I thought the Makefile/rebar.config would be involved somehow.

OK then I have a patch that is more or less ready. I will see about releasing new versions with a relaxed Hex version requirement for all my projects as well as rabbit_common in the next few days.

[essen]

Thank you!!

[wojtekmach]

No problem, I hope I didn't give false statements, it's been a while since I looked into those kinds of problems but that's my operating assumptions how this all works. Perhaps best to test it on just one package or something.

Out of curiosity what do you have in mind? Would you specify somewhere in Makefile that you depend on version range and that's for publishing but you'd additionally have somewhere the pinned version that'd be used by other tools?

[essen]

Just override the requirement when generating the Hex metadata file. If an override exists, use it, otherwise use the dep's version specified in the Makefile (rebar.config also takes this version). Patch is simple really:

diff --git a/plugins/hex.mk b/plugins/hex.mk
index 44beb65..b8bf78d 100644
--- a/plugins/hex.mk
+++ b/plugins/hex.mk
@@ -91,7 +91,7 @@ define hex_tarball_create.erl
 			<<"$(if $(subst hex,,$(call query_fetch_method,$d)),$d,$(if $(word 3,$(dep_$d)),$(word 3,$(dep_$d)),$d))">> => #{
 				<<"app">> => <<"$d">>,
 				<<"optional">> => false,
-				<<"requirement">> => <<"$(call query_version,$d)">>
+				<<"requirement">> => <<"$(if $(hex_req_$d),$(hex_req_$d),$(call query_version,$d))">>
 			},)
 		$(if $(DEPS),dummy => dummy)
 	},

[wojtekmach]

Yup, I think that sounds good.

[wojtekmach]

I briefly peeked at hex.mk and yeah, glad to see hex_core being used!

Btw for https://github.com/ninenines/erlang.mk/blob/a8d5744cc412016b0d57e0d398e2195c380ae4b6/plugins/hex.mk#L137 you can use:

{ok, Tarball} = file:read_file(Path),
{ok, #{metadata := Metadata}} = hex_tarball:unpack(Tarball, memory, hex_core:default_config()).

Can't remember why we don't accept a Path argument in unpack/3 that'd streamline this further.

[essen]

Nice, thanks for the tip!