Generic associated types (associated type constructors)

[withoutboats]

I definitely realize that this feature may not be prioritized right now, and I expect that some aspects of it probably present real implementation challenges (especially partial application of type operators and extending HRTBs), but I wanted to document and discuss what seems to me like the best incremental step toward higher-kinded polymorphism in Rust.

Rendered

[kennytm]

There is a missing ` after the "not".

[glaebhoerl]

I agree that this seems like the ideal starting point for adding higher-kinded capability to Rust. Even the idea of moving forward incrementally in this way, as opposed to "Adding Higher-Kinded Types" all at once in a "big bang", is something that hadn't occurred to me, but seems obvious in hindsight.

To stick with our own terminology consistently, instead having of a mishmash of vocabulary from Rust, Haskell, and the academic literature, I would probably prefer to call this feature "generic associated types" or "associated generic types". (I would also prefer to call "higher-kinded types" instead "higher-order generic types", contrasting with "higher-rank types" which are "higher-order generic functions", but the usage of "HKT" is incredibly widespread by this point.)

I also suspect that most of the difficult design work which needs to be done before this feature can move forward is on how to extend the type system to support it, which this RFC (self-admittedly) doesn't even attempt to address. Am I right?

[dwrensha]

Currently, an approximation of StreamingIterator can be defined like this:

trait StreamingIterator<'a> {
  type Item: 'a;
  fn next(&'a self) -> Option<Self::Item>;
}

You can then bound types as T: for<'a> StreamingIterator<'a> to avoid the lifetime parameter infecting everything StreamingIterator appears.

However, this only partially prevents the infectiveness of StreamingIterator, only allows for some of the types that associated type operators can express, and is in generally a hacky attempt to work around the limitation rather than an equivalent alternative.

I'm interested in seeing a concrete example of something that cannot be expressed using this hack. I've hit situations where I've wanted associated type operators as proposed in this RFC, but in every case it has sufficed hoist the the type parameters up to the trait itself. See, for example, these capnp traits. I agree that such hoisting can be ugly, and I feel like there are probably cases where it fails, but those cases are not immediately obvious to me.

[BurntSushi]

I'm interested in seeing a concrete example of something that cannot be expressed using this hack.

Ideally, the StreamingIterator trait should be expanded to contain many of the same utility adapter methods that the Iterator trait has.

[eddyb]

AFAIK the bulk of the implementation work needed here is removing the constraints on the "3 application levels" in the current compiler (type/trait parameters, Self and fn/method parameters).

The actual resolution of non-monomorphic associated types won't actually be that big of a problem, we already have to do that sort of transformation for generic methods.

And lastly, what I'm worried most about is checking uses of such a trait with polymorphic associated types without providing concrete bindings.
We've had a few stages of adjustment to all the problems associated types can cause in various edge cases.
I suspect this is the "difficult design work" @glaebhoerl was talking about, in which case I agree.

But overall, I like this RFC and @rust-lang/compiler will continue to work towards making such advances possible, regardless of which exact syntactic or semantic choices we end up accepting.

[krdln]

@dwrensha
I think that it's currently impossible to express something like owning_ref, but generic over reference type. Eg. the interface of generic reference could look like:

unsafe trait Ref {
    type Item<'a>;
    unsafe fn erase_lifetime<'a>(x: Item<'a>) -> Item<'static> { transmute(x) }
    unsafe fn introduce_lifetime<'a>(x: Item<'static>) -> Item<'a> { transmute(x) }
}

The main benefit from this RFC here would be that compiler would know that all types created by the same & → * constructor are transmute-compatible, and that it can apply that constructor for any lifetime. Unfortunately, the trait would have to be implemented for some phantom type, since this RFC doesn't allow traits to be implemented for type constructors, but I think it's only a minor issue.

It seems that the main benefit from this RFC comes from allowing lifetime (& → *) constructors, so I'd be happy to see it accepted even if it doesn't allow full-featured * → * ones. (analogicaly, as we have for<'a> syntax only for lifetimes)

[eddyb]

@krdln Actually, right now you can't even use transmute there, although I would be inclined to accept a same-type exception (we already erase lifetimes, so it should just see Item<'static> on both sides).

[krdln]

@eddyb Your example doesn't compile, since you can't apply an lifetime paremeter to Self::Item. Maybe you've meant to use Ref<'a> somewhere there? I've tried, but I couldn't make it compile. So I'm still not convinced that thing is possible in today's Rust (but I won't be surprised if it was possible, Rust can be really surprising).

Using your suggestion, I came up with this, so this use case might be supported in current Rust, with some fixes for transmute behaviour.

[eddyb]

@krdln Hah, that was dumb of me. Totally missed the fact that you were applying multiple times within the trait itself. It could work via free functions with R: Ref<'a> + Ref<'static> but that's probably not useful to you.

[soltanmm]

@krdln I think there does exist an injection between what is proposed here and the syntax that exists today, but it requires a few more pieces than what @dwrensha pointed out.

use std::mem;

// '* -> *
trait R2T<'a> {
    type T;
}

// * -> *
trait T2T<A> {
    type T;
}

// ('* -> *)('*)
type AppR<'a, T> where T: R2T<'a> = <T as R2T<'a>>::T;
// (* -> *)(*)
type AppT<A, T> where T: T2T<A> = <T as T2T<A>>::T;

// A trait with a higher kinded operator
unsafe trait Ref {
    type Item: for<'a> R2T<'a>;
    unsafe fn erase_lifetime<'a>(x: AppR<'a, Self::Item>) -> AppR<'static, Self::Item> { mem::transmute(x) }
    unsafe fn introduce_lifetime<'a>(x: AppR<'static, Self::Item>) -> AppR<'a, Self::Item> { mem::transmute(x) }
}

// A type with a '* -> '* -> * HKT
trait MyType
    where
        for<'a> Self::BiRefItem: R2T<'a>,
        for<'a, 'b> AppR<'a, Self::BiRefItem>: R2T<'b>
{
    type BiRefItem;
}

// Use of partial application by currying (which can always be extended to full partial application with helpers)
trait PartiallyApplied<'a, T: MyType>
    where
        Self::Ref: Ref<Item=AppR<'a, T::BiRefItem>>
{
    type Ref;
    // ...
}

fn main() {}

However, it is excessively verbose, highly error prone, works only for lifetimes in certain use-cases (because it depends on HRTBs), and there're some issues with the compiler that prevent it from compiling (see rust-lang/rust#30472 and rust-lang/rust#31580).

For a more practical application of the general approach that does work today, see #1403 (comment).

For background on the general approach, see the fine-print in the associated items RFC. n.b. the RFC uses non-existent syntax in its example.

^{and I am in no way suggesting that one take a syntax-sugar approach implementing this :-P}

[soltanmm]

Actually, now that I think about it... Unless this RFC is limited to kinds that look like '* -> *, doesn't it require extending HRTBs to types (in mechanism, not necessarily in syntax) to be useful? Otherwise we can't put bounds on the outputs of HKTs like * -> *. Seems like doing that first could be a reasonable alternative.

[eddyb]

@soltanmm It is limited to '* -> *. However, this brings an interesting point I didn't realize: T: Trait<Item<'a> = &'a [u8]> is sugar for T: Trait and for<'a> <T as Trait>::Item<'a> = &'a [u8] so still HRTB in nature, just packaged away neatly.

[withoutboats]

To stick with our own terminology consistently, instead having of a mishmash of vocabulary from Rust, Haskell, and the academic literature, I would probably prefer to call this feature "generic associated types" or "associated generic types". (I would also prefer to call "higher-kinded types" instead "higher-order generic types", contrasting with "higher-rank types" which are "higher-order generic functions", but the usage of "HKT" is incredibly widespread by this point.)

I don't feel strongly about the terminology, but I'll note that I avoided the term "higher-kinded type" in this RFC because I found it very confusing before I read TaPL, which avoids that terminology. A big part of my confusion was that calling type operators "types" suggests to me that this is a language feature that allows one to create values of a "type" of a higher kinded, like let x: Vec.

I also don't think we use the term generics that much, do we? I usually see 'type parameter' or 'parameterized type.'

I'm interested in seeing a concrete example of something that cannot be expressed using this hack.

I think others have answered this well (the most obvious thing is: * -> *), but I want to add that the real point is to make this feature easier and more intuitive to use, just as adding const parameters would be an easier easier and more intuitive feature than the type-level peano number hacks that we have right now.

You are talking about allowing any operators, but the syntax seems only defined for constructors. If you plan to support any operators, could you give an example of using * → * → * associated operator (not (*, *) → *)?

I used the term operator throughout the text, but its correct that this would only allow associated type constructors.

Actually, now that I think about it... Unless this RFC is limited to kinds that look like '* -> *, doesn't it require extending HRTBs to types (in mechanism, not necessarily in syntax) to be useful? Otherwise we can't put bounds on the outputs of HKTs like * -> *. Seems like doing that first could be a reasonable alternative.

I don't know anything about how HRTBs are implemented, some of eddyb's comments suggest this might be an easy extension, but the higher-ranked parameter is being applied in a different position in these two expressions:

for<'a> Type<'a>: Trait
Type: for<'a> Trait<'a>

This RFC explicitly says it doesn't propose allowing for<'a> Type<'a>: Trait, though obviously that would be the most useful thing to add next. I this RFC is useful even if you can't bound this new kind of associated item.

Also, this RFC only discusses lifetime -> type associated items, but I don't think there's anything in the RFC that doesn't apply equivalently to all kinds of type constructor. The main reason lifetime -> type is the only kind discussed is its the only kind for which I know of obvious, valuable use cases. Not coincidentally, I think its also the kind which can be easily explained without type theory concepts: it lets the trait define "view" types for its receiver.

Extending HRTB to be used with associated items of a kind other than lifetime -> type would require more work, though, of course.

[withoutboats]

Thinking about the syntactic and semantic complexities of bounds using associated type constructors, and the fact that all of this seems to be conceivable as just an extension of HRTBs, I think it might be a good idea to not even discuss bounds using associated type constructors in this RFC, and instead delegate that to an RFC about extending HRTBs.

[eddyb]

@withoutboats Both of those forms are allowed right now and the for<'a> actually gets lifted to the whole bound (if you don't mean (for<'a> Type<'a>): Trait, of course).

In a way, for<'a> is only syntactical, at the semantic level debruijn indices are used for "late-bound lifetimes" with "binders" which define the boundaries where they apply.

So for<'a> (for<'b> fn(&'b T) -> (&'b U, &'a V)): Trait<'a> ends up being something like fn(&'2 T) -> (&'2 U, &'1 V): Trait<'1> with a binder around the signature of the fn type and another one around the whole bound.

it might be a good idea to not even discuss bounds using associated type constructors in this RFC

It's not a good idea to punt on this matter, because the T: Trait<Assoc = U> syntax is sugar for a <T as Trait>::Assoc = U bound (which we can't express right now syntactically, sadly).

[withoutboats]

It's not a good idea to punt on this matter, because the T: Trait<Assoc = U> syntax is sugar for a <T as Trait>::Assoc = U bound (which we can't express right now syntactically, sadly).

I'm not sure why this makes it bad to punt, but I think the important takeaway is that we (yet) don't need to be able to put type constructors in where clauses, because we can use HRTB. Obviously this won't be sufficient for later use cases involving higher-kinded traits.

[withoutboats]

Another issue has occurred to me just now. We would probably also want some sort of implicit HRTBs at the declaration site of the associated type, as in:

trait Collection {
    type Elem;
    type Iter<'a>: Iterator<Item=&'a Self::Elem>;
    type IterMut<'a>: Iterator<Item=&'a mut Self::Elem>;
    ...
}

[eddyb]

@withoutboats I wonder if we should agree on using "polymorphic" or some other "painfully clear" term because I thought you meant <T as Trait>::PolyAssoc<'a>: RegularTrait instead of <T as Trait>::PolyAssoc: SomethingNew, and the latter is HKT territory as far as I'm concerned, so we're in violent agreement, I suppose.

[withoutboats]

@eddyb: Right, the RFC currently discusses <T as Trait>::PolyAssoc=TypeConstructor, but instead of allowing that that we can just use for<'a> <T as Trait>::PolyAssoc<'a>=TypeConstructor<'a>

[withoutboats]

I've edited and expanded the RFC in response to the comments so far, mainly clarifications and adding content about how this would require HRTB changes.

[soltanmm]

When's the RFC to spec out the syntax of bounds/where-clauses on HRTB-quantified types coming out? 😃

^{aaand obligatory cc #1481}

[eefriedman]

The syntax struct Foo<T> where T: for<'a> StreamingIterator<Item=&'a mut [u8]> from the example in the RFC doesn't seem right. The following expanded example is legal on nightly Rust:

trait StreamingIterator {
    type Item;
}
struct Foo<T> where T: for<'a> StreamingIterator<Item=&'a mut [u8]> {
    x: T
}

Granted, as far as I can tell there is no way to write T which satisfies this constraint at the moment, but as far as I can tell that's orthogonal to this RFC.

I think what you actually want is something closer to struct Foo<T> where T: StreamingIterator<Item<'a>=&'a mut [u8]>, with no for<>.

[withoutboats]

You're right that its wrong, but it needs both the <'a> and the for<'a>. Without the for<'a>, it would search the surrounding scope for a lifetime parameter called 'a and fail.

EDIT: Fixed.

[ExpHP]

new rendered link

[Boscop]

Is there already a plan how HKTs will look outside of traits, so that there is still "design space" to make the syntax the same as in traits?

How would it look in functions, like this?

fn foo<M<type, const usize>, T, const N: usize>() -> M<T, N>;

fn foo<M<type, const usize>, T, const N: usize>()
where for<T, N> M<T, N>: Default
-> M<T, N> {
    Default::default()
}

[gbutler69]

It would be nice if any RFC that uses an acronym provides a link or definition upon first use. HRTB for example. So, this:

Users can bound parameters by the type constructed by that trait's associated type constructor of a trait using HRTB. Both type equality bounds and trait bounds of this kind are valid:

Would become this:

Users can bound parameters by the type constructed by that trait's associated type constructor of a trait using HRTB. Both type equality bounds and trait bounds of this kind are valid:

[dylanede]

I haven't seen any mention of variance with respect to GATs yet. The RFC should note whether associated types are invariant, covariant or contravariant in their type parameters. My experiments with poor-man's GATs (for<'a> bounds and dummy traits) suggest that GATs will be severely limiting without some way of specifying variance constraints on associated types within a trait definition.

[Boscop]

@dwrensha

I'm interested in seeing a concrete example of something that cannot be expressed using this hack.

How about this?
https://play.rust-lang.org/?gist=32e56d762c3ade7439cdb6796f21569f&version=stable

That's a real-world example I'm running into:

error[E0212]: cannot extract an associated type from a higher-ranked trait bound in this context
  --> src/main.rs:50:17
   |
50 |     device: Option<T::Dev>,
   |                    ^^^^^^

Does anyone of you have any idea how I can make this work? :)

I need to be able to instantiate/store AppMgr before the lifetime 'a that process() will becalled with even exists (it needs to live in an outer scope as a member of a struct, so I have to specify the type. (The 'a that process() will be called with only starts to exist shortly before calling process().)

Or is there any way I can express/enforce that DeviceApp<'a>::Dev should be the same type for all 'a? So that rustc can determine the (for<'a> DeviceApp<'a>)::Dev because there would be only one possibility for it.

---

AFAIK, to do this I'd either need the ability to express the above statement (and I'd need rustc to then allow me to use T::Dev), or I need GATs.

I tried to do it with GATs like this:

impl DeviceApp for MyAppState {
	type BorrowedState<'a> = MyBorrowedState<'a>;

but I run into this ICE:
https://play.rust-lang.org/?gist=b0614c07a45df09e1b2a3cd2bc789e7a&version=nightly

error: internal compiler error: librustc/ty/subst.rs:439: Region parameter out of range when substituting in region 'a (root type=Some(MyBorrowedState<'a>)) (index=0)
thread 'main' panicked at 'Box', librustc_errors/lib.rs:491:9
rust-lang/rust#50115

I really need this to work. What would be the recommended way to do this before this ICE issue gets fixed? :)

Btw, how long do you think it will take until this issue gets fixed?

[Boscop]

I found a workaround, pulling Dev outside of the <'a> quantification scope:

https://play.rust-lang.org/?gist=9b7907d94dfc28aabe12f5f90f8c4592&version=stable

pub trait DeviceSpecific {
	type Dev: Device;
}

pub trait DeviceApp<'a>: DeviceSpecific {
// 	type Dev: Device;

This compiles, but it's a combo of 2 ugly workarounds (I'd prefer not to split the trait like that) that wouldn't be necessary with proper (working) GATs, right?

Can you think of any solution that doesn't require me to split the trait like that? :)

[Boscop]

Ah no, it doesn't work with the impl of AppMgr :(

pub struct AppMgr<T: for<'a> DeviceApp<'a>> {
	device: Option<T::Dev>,
	check_connection: AtFps,
	in_port_name: String,
	out_port_name: String,
	app: T,
}

impl<T: for<'a> DeviceApp<'a>> AppMgr<T> {

	// error: cannot extract an associated type from a higher-ranked trait bound in this context

	pub fn process(&mut self, now: u64, v: &mut T::BorrowedState) -> Vec<T::Event> {
	//                                          ^^^^^^^^^^^^^^^^
		if self.check_connection.is_due(now) {
			let should_reconnect = if let Some(ref mut device) = self.device {
				!device.is_connected()
			} else {
				true
			};
			if should_reconnect {
				self.device = T::Dev::connect(&self.in_port_name, &self.out_port_name).ok();
			}
		}
		if let Some(ref mut device) = self.device {
			let app = &mut self.app;
			device.frame(now, |device, input| { app.process(device, input, v) })
		} else {
			vec![]
		}
	}
}

What I need is this:

// pub struct AppMgr<T: for<'a> DeviceApp<'a>> {
pub struct AppMgr<T: DeviceApp> { // DeviceApp not depending on the lifetime 'a that process() gets called with
	// ...
}

// impl<T: for<'a> DeviceApp<'a>> AppMgr<T> {
impl<T: DeviceApp> AppMgr<T> {
	// instantiating T::BorrowedState with <'a> from the scope that process() is called in
	pub fn process<'a>(&mut self, now: u64, v: &'a mut T::BorrowedState<'a>) -> Vec<T::Event> {
		// ...
	}
}

Is there any way to do this now? If not, will this work with proper working GATs?

[Boscop]

I found a way to impl AppMgr that works:

pub struct AppMgr<T: for<'a> DeviceApp<'a>> {
	device: Option<T::Dev>,
	// ...
}

impl<T: for<'a> DeviceApp<'a>> AppMgr<T> {
	pub fn process<'a>(&mut self, now: u64, msgs: Vec<<T as DeviceApp<'a>>::Msg>, v: &'a mut <T as DeviceApp<'a>>::BorrowedState) -> Vec<<T as DeviceApp<'a>>::Event> {
		// ...

Here as reduced working example:
https://play.rust-lang.org/?gist=7522f287efe68c7b73efcddef1c5dd2a&version=stable

But I still wasn't happy about having to split the trait, so I thought, if I mandate that Dev: 'static, the compiler will be able to infer that for all 'a, <T as DeviceApp<'a>>::Dev will be the same, namely the same as <T as DeviceApp<'static>>::Dev (because Dev: 'static means it won't depend on 'a).
So I tried that, bringing Dev back into the DeviceApp trait:

/*
pub trait DeviceSpecific {
	type Dev: Device;
}

pub trait DeviceApp<'a>: DeviceSpecific {
*/
pub trait DeviceApp<'a> {
	type Dev: Device + 'static;

// ...

pub struct AppMgr<T: for<'a> DeviceApp<'a>> {
	// device: Option<T::Dev>,
	device: Option<<T as DeviceApp<'static>>::Dev>, // because Dev: 'static, i thought it will be unifyable with any 'a
	app: T,
	// ...
}

But now I get this error:

error[E0495]: cannot infer an appropriate lifetime for lifetime parameter 'a in function call due to conflicting requirements
  --> src/main.rs:79:53
   |
79 |             device.frame(now, |device, input| { app.process(device, input, v) })
   |                                                     ^^^^^^^
   |
note: first, the lifetime cannot outlive the lifetime 'a as defined on the method body at 75:5...
  --> src/main.rs:75:5
   |
75 |     pub fn process<'a>(&mut self, now: u64, msgs: Vec<<T as DeviceApp<'a>>::Msg>, v: &'a mut <T as DeviceApp<'a>>::BorrowedState) -> Vec<<T as DeviceApp<'a>>::Event> {
   |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   = note: ...so that the expression is assignable:
           expected &mut <T as DeviceApp<'_>>::BorrowedState
              found &mut <T as DeviceApp<'a>>::BorrowedState
   = note: but, the lifetime must be valid for the static lifetime...
   = note: ...so that the expression is assignable:
           expected &mut <T as DeviceApp<'_>>::Dev
              found &mut <T as DeviceApp<'static>>::Dev

https://play.rust-lang.org/?gist=d4084c0a87890d4ed17d0491e453565b&version=stable

Why can't rustc unify <T as DeviceApp<'static>>::Dev> with <T as DeviceApp<'a>>::Dev> for all 'a, when it KNOWS that Dev: 'static (the type doesn't depend on 'a, so it can only be the same for all 'a)?

---

Actually, for some reason, in my real code I get a different (similar) error instead (with nightly 2018-03-06):

error: free region `'a` does not outlive free region `'static`
   --> src\devices\mod.rs:132:15
    |
132 |         if let Some(ref mut device) = self.device {
    |                     ^^^^^^^^^^^^^^

Any idea how to make it work with Dev inside the DeviceApp trait? :)

[burdges]

I quite like the StreamingIterator example here, but another question: Should we permit higher kinded type parameters? As an example, you could borrow a data structure and use mut reborrows to iterate over it several times like:

fn process_elements<V: ?Sized,II>(x &mut V,f: fn<'a>(&mut V) -> II)
where
    II<'a>: Iterator<Item=&mut X>

or

fn process_elements<F,II>(f: &mut F)
where 
    for<'a> &'a mut F: Into<II<a'>>,
    for<'a> II<'a>: Iterator<Item=&'a mut X>

In principle, these could be done with impl Trait in certain positions, but this feature seems logically prior.

[dhardy]

Re-link the tracking issue, which is not mentioned at either the top/bottom of this thread.