Towards a better end-user experience in tooling #48
Comments
|
Pinging some maintainers, for whom this is relevant: @mpilgrem @michaelpj @fendor @Mikolaj |
|
@hasufell, thanks for the ping. From the perspective of Stack (the tool), a new GHC version is usually not a major event. The needs of other tools seem to be more demanding. For Stack, it involves updating: (a) https://github.com/commercialhaskell/stackage-content/blob/master/stack/global-hints.yaml for the package versions of the libraries that come with GHC on Windows and non-Windows; and (b) https://github.com/commercialhaskell/stackage-content/blob/master/stack/stack-setup-2.yaml for URLs, file sizes, SHA1 hashes and SHA256 hashes of GHC binary distributions. For me, (a) and (b) are manual processes. If nobody else has got there first, I hurry to do them when I read on the Haskell Community that a new GHC version has been released. Step (b) is easy if the current structure is 'like' the most recent previous structure. It requires more thought if something 'structural' changes - like 'Fedora 27' disappearing and 'Fedora 33' appearing. If it is only me doing the thinking, then Linux-based structural changes can introduce a hiatus, as I don't have much Linux experience to draw upon. |
|
I agree closer coordination is very much needed. I'd add Haddock to the mix. I suppose Hackage is covered under cabal? |
Absolutely, and in general I think most of the issues we have with our build process relate to tech debt that is possible to fix. Increased meetings / social coordination is always good, but in this case we should also simply invest in making the underlying technical situation less complicated. It's very feasible. |
@Mikolaj Fortunately (and unfortunately), the historical proximity between GHC and Haddock hasn't led to to major logistical problems. This may change in the near future but if we go on with the foundations described by Julian, I'm not worried. |
|
This is great! But it misses nix. According to the latest survey about 35% of users are relying on nix. https://taylor.fausak.me/2022/11/18/haskell-survey-results/ Haskell is, as far as I know, the only language that has two totally separate nix ecosystems. Worse, they're mutually incompatible. One is better at distributing software, the other is better at building and developing it. Documentation is a mess with posts confusing them. They have completely different user-facing APIs.
|
|
No it really does make sense that nix is not included in this very specific proposal. In its current incarnation, with the current workforce at our disposal and the current repartition of expertise in the community, nix is a burden to maintain. I mean no offence by this, nor do I aim to trigger a flamewar, but it is true.
Seems like these questions would be best addressed to IOG and the other players in the space. Until there is a massive effort to bring Nix to the masses, outside of the club of passionate people who are breaking their back maintaining the infrastructure, we can't do much but applause the principles of Nix, and shedding a tear for our martyrs. |
|
@abarbu I appreciate you bringing this up. I'm however afraid we will not see a unification of Nixpkgs haskell infra and haskell.nix, they are philosophically very different, and also server almost orthogonal goals. Hence I believe (2) is simply not attainable. Haskell.nix will focus on versatility and usability for haskell projects, and to that extend gladly pay any IFD or other penalty, if it means the user has to deal with nix less; it also means we try to support a larger set of compilers, as projects can depend on older ones that were dropped from Nixpkgs. On the other hand Nixpkgs primary haskell infrastructure is mostly based on a stackage snapshot, and the primary goal is to support haskell packages in Nixpkgs, and secondarily the development of haskell packages. Therefor depending on a stackage snapshot to increase sharing makes a lot of sense, but that's not what your custom haskell project needs. However, I'd like to emphasise that haskell.nix can also be used for distribution, it's just much worse at distributing lots of Haskell packages with lots of dependency sharing between those packages. haskell.nix relies on cabal for dependency resolution--so haskell.nix stays as close to your The issue you bring up in (1) is fundamentally the issue of Now why does adding new compilers to haskell.nix take so long? For 9.4 upstream just broke it in many ways. We now have Hadrian, which fails to properly work for cross compilers. Upstream also decided to move @Kleidukos I tend to agree that we should not focus too much on nix. I will however point out that our nix infrastructure is currently the only one that can sensibly exercise ghc as a cross compiler. I'm not sure where haskell_rules is right now though. haskell.nix is a community project, it is currently hosted in the IOG namespace, so maybe we should consider moving it somewhere else, if that would make it more obvious that it's a community project. There is one person who's primary job the maintenance of haskell.nix is, and has been consumed mostly with the fallout of upstream breakage (and ensuring all our cross targets still work). |
@angerman Just for the record, I fully agree with your statement. |
Focusing on making everything reinstallable is probably the number one thing that would benefit Nix, but it also is a high contender for the best thing to benefit everything else. So we can focus on that, not worrying about Nix in particular, and yet not neglecting Nix either. |
Isn't this already done? https://gitlab.haskell.org/ghc/ghc/-/issues/19896 |
|
firstly i will have to say i am biased as i use nix for my haskell projects. however if this issue is to be focused on the end user we shouldn't ignore the 35% that use nix + possibly something else , i do believe that including the nixpkgs haskell maintainers in these discussions would help them. also nix tests more packages than stackage for instance i went to nixpkgs hydra jobsets and then searched for haskell and found that 6 days ago they built the ABList package for x86_64-linux and aarch64-linux this package isn't on stackage and i have never heard of it before, also they maintain a ghcHEAD version of ghc which is ocasionally updated. actually there is a pr to update it right now. what i want to say is that nix should have importance close to stack in this discussion, not the main point but certainly not ignored, if we se the survey it says that 33% use nix and 12% use haskell.nix. @Kleidukos i don't know if you use stack but to anyone that does surely if someone said that including stack was a maintenance burden and implied that it should be completely ignored you would at least object right? that said if the situation is as bad as said maybe we should focus only on cabal and hls which are absolutely essential? if i am not mistaken both nix and stack internally use cabal and it is significantly harder to develop without lsp. |
|
Great discussion; just to interject 2c, I don't believe 35% of Haskell users or would-be Haskell users use nix, I think that's partly sampling bias.
|
Thanks for digging this up, so this was actually merged 🤔 |
|
So, the reason I did not bring up nix is mainly because I have no interest in it and will not work on it. This issue was mainly raised because of the problems GHCup and HLS have with bindists and release coordination. Nix does not use bindists. If nix maintainers will benefit too from joining synchronous release meetings, then they surely will be included. I'd like to focus on the mentioned issues wrt bindists and release coordination, though. Yes, the thread title is a bit broader and yes it's important to collect more topics that need attention. I think @angerman has sufficient visibility about what nix and their maintainers need from GHC team, so I guess this document can serve as a template for demanding improvements in a similar fashion. But I'm not the person driving that. |
|
I think this is all fantastic! Thanks, @hasufell, for writing up your detailed start of this conversation and for being a motive force here. One challenge I have thinking about this concretely is that it's hard to pull out the proposals from the problem statement. That is, I see a proposal for a synchronous coordination meeting in advance of releases. But, other than philosophical urgings (which are important!) I don't see much else. Maybe I missed something (I likely did), but that's why I'm writing this paragraph, as a request to pull out the proposals from the rest of the text. I do see the idea of creating a shared project. Might I suggest placing this at https://github.com/haskell/haskell? That's some prime real estate that has yet to be used. Its front page (that is, README) could contain a little map of our ecosystem, along with instructions for contributing (including where to post bugs, what communications media to announce new libraries in, our standards of discourse, etc). I urge us not to get distracted by Nix. My assessment is that Haskell has currently roughly saturated the market for early enthusiasts, and that this proposal is part of the broad push within the Haskell community to "cross the chasm". (The first 20 seconds or so of this video explains the term well; excuse the marketing for a specific book, which I have never read and am not advocating for.) The existence of the Haskell Foundation can be seen as another part of this broad push. I don't have a handle on whether Nix is also trying to cross the chasm, but my assessment there is that it hasn't done this step yet: it is a powerful tool for enthusiasts (and people who pay enthusiasts to help them), but I do not think it is yet ready for mainstream. By tying our efforts here to Nix, we will be trying to connect two enthusiast-oriented technologies and making both easier to use at the same time, which seems much harder than doing them one at a time. |
There are 6 proposals:
I think all of these proposals have clear action points and scope, while 4. is maybe a bit more vague. What exactly do you find missing? The specific immediate goals of all those proposals are to:
|
|
@goldfirere to be clear... this isn't a formal proposal that expects a vote by the board. I'm using the HF issue tracker to raise this for visibility and also communicate the points where HF can get involved. The decisions on what to do in particular depend on motivation by volunteers and communication and discussion between the stakeholders. The technical parts depend ultimately on the respective maintainers who have authority over their projects. |
The summary you have now provided. That's really it! The initial post was long and detailed (and thus forms a great base of shared understanding and conversation-starting), and I failed to extract the concrete changes you were proposing. Thanks for writing that summary -- that's really all I was looking for. I remain strongly in favor of this direction of travel. |
The idea is that if I'm hit by a bus tomorrow, someone else can pick this up and go back to this post to understand what's going on. I realized while writing it that it's indeed not that easy to explain all the context properly. Which is a strong sign that it's important to do, so it doesn't just live in the inner circle of maintainers who happen to deal with the bug reports. |
|
Thanks for writing this! You've hit the nail on the head by focusing on end-user experience in tooling. If we don't agree on the problem, we won't agree on the solution. I think we agree on the problem. I just want to say that I hope my role (Haskell Foundation DevOps) is considered as a resource in this area of effort. In fact, although my main focus is still GHC CI, my current task is literally to improve the Mac bindist, so I'm already contributing in this area. I am resisting putting on my project manager hat (gotta code fast!), so I will mainly just lurk for now... |
Ben has already scheduled this for the 9.6.1 release, i.e. the next one which is coming out in a couple months. Is it needed for the other, older supported releases? Does it need backports to be useful? Or is it sufficient that it will be available for 9.6 and onwards? |
|
I am happy to support this initiative. A holistic approach to users' experience is important, and the historically-determined boundaries between various Haskell projects are not necessarily where lines would be drawn if we were to design everything today. |
Love it. I think rectifying this the unifying theme of all the key meaty projects that the Haskell Foundation is here to help us fix. |
|
@mpickering and I talked about:
I had the feeling we're on the same page. I think the next steps here are to gather more concrete ideas of how we structure these things (when to have meetings, how to adjust release workflow if necessary). Matthew took ownership of driving this forward initially. We also talked about:
So right now, I think we're trying to figure out the right structure. My impression is that my proposed survival guide, the HF error project and a shared tooling project that tracks high-level issues (such as the GHC ABI one) are all connected in some way. For the tooling project my idea is that:
I'll pursue further input from HF and stakeholders about 3. and 4. |
I get it, but we also have a lot of arcane technical debt that then indirectly bubbles up as user experience issues. Just knowing the pain points does not uncover this. |
Surely. The scope of that issue tracker/project space is about reflecting on the tooling experience. Not so much how to improve Haskell or GHC or base etc. One might arrive at ideas from either direction: seeing user bug reports and thinking about the causes or seeing technical issues that (as you point out), will bubble up to the user. I'm just trying to make clear that this project space really has a very specific scope and isn't meant to compete with HF tech proposals, base proposals, GHC proposals etc. I think it's hard to work on Haskell tooling as a holistic experience if all the projects are disconnected. |
|
I just had a call with @hasufell. We hit on a way forward with this that seems workable. The plan is to create a space in which cross-cutting tooling issues are documented in a way that takes concrete user experiences as a starting point. For example, the documentation for an issue might begin with 'A user opens a Haskell file in VS Code on their Mac. The following error message occurs when they click "Build"'. This would have screenshots and enough details the reproduce the issue. After that, there would be a summary of the technical issues that cause this error to occur (e.g. that the error occurs because of ABI compatibility issues between tooling, or that it's due to code signing requirements from the OS, or that the user didn't have Internet access at a given moment leading to partially-installed tools - these are just for the sake of example). Following the summary should be any known work-arounds for the user. After the workarounds, the issue can then describe the specific details of what's causing the issues and link to relevant issues in the issue trackers for all the involved projects. In other words, the structure is:
To avoid announcing a big initiative that results in nothing but statements of support followed by a subtle "whoof" sound, @hasufell and I will build a bit of example content centered around a real issue or two that have been repeatedly encountered. Having such a text to start with can serve as an example to other contributors as well as guide choices about the concrete technical platform on which to host it. @hasufell will write the initial text, and I'll be a second set of eyes. Thanks for a great initiative and call, @hasufell! |
|
I wonder if @tomjaguarpaw has thoughts, I remember he had https://github.com/tomjaguarpaw/tilapia which I think had similar goals to what you're describing. I'm a little worried about adding another way of reporting issues or whatever, because it inevitably gets overlaid on top of everything else. If we do something like this, it would be nice to think about how it can reduce the use of existing systems. For example, I could imagine us saying something like "Issues which are opened on the HLS issue tracker but which relate primarily to tooling (?) should be moved to this other issue tracking location" (straw example, quite probably not what we want to do). That would actively reduce load on the HLS issue tracker, which would be nice. My other $0.02 is about Nix, as a user and general Nix partisan. I think we don't need to consider much in the way of Nix-specific issues issues, because most issues that affect Nix on the tooling side tend to be generic and things that most people would consider to be improvements even if they're not being directly affected by them (e.g. |
This is not meant for user bug reports. This is meant as a documentation and collaboration effort of the maintainers and collaborators of the core Haskell tooling. The intention is to align on shared goals, increase visibility of user-facing issues beyond project boundaries and have a place where resources on complex topics can be gathered in a refined way that connects all the small pieces together. Depending on engagement, it may also be the place where we dump meeting notes of the synchronous meetings, develop elaborate guides on how to deal with tooling issues (aka survival guides), develop ideas about user experience etc. It will not be a substitute of individual issue trackers. It may also serve as a foundation for project pitches for actual tech proposals or requests for funding. For user feedback, we may utilize the "Discussion" feature of GitHub. These are all ideas and all of this depends on the eventual engagement we generate. |
|
I wanted to make a small announcement that after a long while of being stuck, I was able to make more progress on https://gitlab.haskell.org/ghc/ghc/-/issues/17191 and hope that I am finally "over the hump" with an incremental path forward for the rest. I bring this up as an example of something that is utterly divorced what I imagine are most user would complain about, but as something that nonetheless informs a lot of that as a root cause, and certainly benefits from coordinating between tools/teams. For example, this is one things that informs where the bindist metadata ought to reside. It gets more interesting with follow-up like @bgamari's WIP https://gitlab.haskell.org/ghc/ghc/-/merge_requests/9263. |
|
Thank you @Ericson2314 thid has been overdue for a very long time 🙈 |
|
Closing. This wasn't a proper proposal to begin with and a half baked way of saying "we need haskell tooling product management". After thinking about it for a long time, I don't think we're in a position to achieve that currently. And I don't know how to propell that idea. |
Most of the points in this document were firstly discussed with @bgamari in a meeting.
So this is basically a refinement of those with additional context.
Some actions have already been taken, so we want to use this thread to track
progress and also inform the public.
Although the title is very broad, we focus on release and bindist issues here, because those are the most impactful at the moment.
Context
The tooling ecosystem today consists of at least the following major components:
The end-user experience depends on how well these tools interact and synchronize
releases/updates.
Although the tooling ecosystem has improved, we frequently get user reports about issues
that require deep knowledge about existings bugs, known workarounds or how these
tools actually interact, breaking the plug-and-play experience.
Problem statement
Most of the tools mentioned above have different maintainer teams, with
some overlaps and individuals that regularly contribute across them.
We present here a dependency table between those tools:
This gives a superficial overview of how intertwined the tools are.
The end user often doesn't know these details if they follow the
installation instructions of the VSCode haskell extension or GHCup.
Frequently raised issues include:
The reasons for these issues are multi-faceted. Some of them are not of technical
nature.
I want to focus on a couple of interactions only and provide evidence of problems:
GHC bindists
GHC bindists are consumed mainly by GHCup and by stack:
Both these tools provide means of installing GHC versions by providing a unique mapping.
GHCup uses a distro+distroversion mapping, while stack uses platform+library-sonames as index.
Both have their own caveats.
When new releases are announced, the GHCup and stack maintainers investigate the available bindists GHC has provided with the release and decide on how to update their mappings, so that users of any platform and linux distro get a working bindist.
The main issue here is that neither stack, nor GHCup maintainers have direct influence on
the decisions of what bindists are created and how, although they have high stakes in it.
They are not consulted or informed consistently (although occasionally) on every change
and are not required to sign off on any non-trivial changes.
Rather, they are more downstream consumers that, most of the time, have to deal with
the end result post-release. There are rarely testing or RC phases for bindists and
GHC maintainers are reluctant to update bindists in case they have issues (this
happened only once when profiling libs were missing).
As a consequence, GHCup maintainers have had to invest considerable amount of work
into curating GHC bindists. These efforts can be seen here: https://downloads.haskell.org/ghcup/unofficial-bindists/ghc
The reasons so far included:
The most recent disruption was an update of the Fedora bindist, which was regarded by
both GHCup and stack maintainers as the linux bindist with the highest portability. When
the CI configuration was changed to use Fedora 33 instead of 27 to build the binaries,
the portability assumption broke due to a too recent GLIBC version and no testing was able to catch it.
It is still causing disruption today:
GHC releases
GHC releases have become more frequent in recent times and have increased
pressure on tooling to keep up and support these new versions.
GHC release process until recently didn't include many downstream consumers
and there was no explicit communication. Some release candidates existed only
for major versions.
Tooling maintainers would have to follow the issue tracker and the mailing list closely
to be up to date with issues that are relevant to them.
As such, releases tended to overwhelm not only GHCup, but especially HLS maintainers, where supporting new GHC versions is non-trivial.
HLS releases
Releasing HLS is non-trivial and includes:
In terms of dependencies, the cycle of releasing a new HLS version may look like this:
Issues with GHC bindists will bubble up to HLS and impact the end-user experience of someone who just installed the Haskell VSCode extension and doesn't know anything about the tooling (see the GLIBC issues described above).
So far, this pipeline is quite separated with no tight coordination. As such, support for
GHC 9.2.5 took a long time to finally land in HLS 1.9.0.0, which has been impacting
users as well.
One major reason these things became more complicated is that HLS had to switch to
dynamically linked bindists in order to avoid issues with the internal linker and TH, see:
Ways to improve
The main goal of the suggested improvements is that the end user never knows
what is going on behind the scenes to make the tools work smoothly with each other
and have releases that are tightly in sync. The solutions are not self-serving
and should be rejected or reconsidered if they don't bring us closer to this goal.
Binary distribution metadata
https://gitlab.haskell.org/ghc/ghc/-/issues/22686
GHC maintainers should maintain a metadata file that describes each bindist (in e.g. json or yaml) and includes at least the following information:
gmp,ncurses,libcversionsgccandbinutilsversionsghc --infoThe idea behind this is that all changes to bindist are at least very strictly visible post-release and can be consumed by tools, linters or just diffed by a human to get an overview.
Synchronous release meetings
However, the previous point does not solve communication issues and does not prevent mistakes pre-release.
To achieve that, I propose to hold synchronous meetings 1-2 weeks prior to a release to discuss:
This should make it possible to catch issues before a release and make maintainers aware of upcoming issues.
Making it feasible to change bindist mappings
A bindist mapping is a mapping of a platform (e.g. Ubuntu 20.04) to a binary distribution of a GHC version. At the moment, these are the problems when changing bindist mapping in ghcup after they've already been released (mutating them, so to speak):
ghcup --infoto index package databases, which is a little more robust, but also fallible.@bgamari opines that the right solution to (1) is for boot packages to have proper ABI hashes computed by Hadrian. These could then be factored in to the unit ids of a package's dependencies. Some refactoring of GHC's internal tracking of interface hashes may be necessary for this (see https://gitlab.haskell.org/ghc/ghc/-/issues/20810). See also https://gitlab.haskell.org/ghc/ghc/-/issues/21590.
This may be a major effort that we need to find a contractor for.
It's the main reason why GHCup maintainers have to be so paranoid about bindists being correct and often leads to bugfixes having to wait for the next GHC release (as is the case right now... Fedora 27 bindists were added back to GHC release CI, but tooling maintainers were not aware and ghcup and stack metadata was already released).
Developing a holistic tooling release workflow
The current release workflow can benefit from more automation as well as more cross-checking between maintainers, e.g.:
This would synergize with the synchronous GHC release meetings and may be expanded. E.g. currently, we have monthly HLS meetings and @mpickering is already attending them.
The idea behind this is to make the tool boundaries a little bit less strict and take shared responsibility and enable to discuss the end-user experience regularly and work towards shared solutions (relevant for the next section).
Creating a shared project for tooling
I suggest to set shared goals for the entire tooling suite that focuses on the end user
and then create a project on github or an epic on gitlab to track the progress. The reasoning behind this is that a lot of issues (like GHC ABI) affect multiple or all tools and tracking progress, discussions etc. across multiple issue trackers is cumbersome and causes poor visibility of high priority issues.
Issues living in this project should naturally have high priority for all designated maintainer teams.
This could be combined with a bounty system backed by the Haskell Foundation.
Increasing the bus factor across critical tooling
Many of the mentioned issues exist, because all tools are short of maintainers and only some tools have paid contributors.
GHCup has only one maintainer, who is not paid.
Not many people are able to do HLS releases and understand all the existing issues.
Cabal is understaffed too.
If we develop no strategy on how to deal with the bus factors, all efforts to improve the situation may eventually regress when knowledge and engagement gets lost.
Open question: outsourcing of GHC bindist creation
There has been some discussion whether outsourcing of bindist creation (e.g. for esoteric linux distros or tier3 platforms) would be an option too.
Practically, this is already happening partly in GHCup, but the idea was that GHCup completely relies on its own bindists.
This is more of an open thought than an actual proposal at the moment.
Closing thoughts
As can be seen, not all issues here are technical and it has been my impression that we've focused too much on that part.
Automation and CI are all great. But they don't ultimately solve the coordination issue.
It can be debated whether we would actually need a proper release coordinator who works across tooling boundaries.
Right now, I'm not sure how that would look like. I propose to try the above improvements first and maybe some of the friction will already be gone.
There are also more topics to be covered, e.g. how to unify the GHC installation methods to cause less friction with HLS+stack. But I feel that is for another day.
The text was updated successfully, but these errors were encountered: