Zarr-Python → Refactoring #1480
Replies: 14 comments 18 replies
-
|
Thanks for coordinating @MSanKeys963! Looking forward to contributing to this working group. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @MSanKeys963 and @rabernat for getting this going! Background and objectivesZarr-python has traditionally been the canonical implementation of the Zarr spec. It has also been the place where a lot of experimentation has been done . Today, Zarr-python has a fair bit of technical debt, and importantly, an out-of-date implementation of the Zarr V3 spec. Based on the conversations we had in July, I want to highlight the two main objectives for this focus group:
While these aren't particularly sexy objectives, they are poised to be highly impactful as most users of Zarr are currently relying on Zarr-python already. If you are keen to get involved with either the V3 spec implementation or storage related refactors, this is the group for you. MeetingsWe are going to meet twice a month starting in mid-August. We will schedule the first meeting for 1-hr. Following meetings will be 30 minutes. Please fill out this poll to let us know what times you could attend the kickoff meeting. update 8/4/23 - call in detailsTo join the video meeting, click this link: https://meet.google.com/mxq-jbnu-zka During the kickoff meeting, we will attempt to map out the primary effort around getting the Zarr-V3 spec reimplemented in Zarr-python. Reading
Very quick intro to meToday, I'm the CTO at @earth-mover, a software startup that uses Zarr extensively. I've been contributing to open source for many years. I got my start in open source in graduate school when I was fortunate enough to start contributing to Xarray. I've since spent a lot of time working on various problems around the Xarray, Dask, Zarr, and Jupyter ecosystem, often wearing the @pangeo-data evangelist hat. After about 6 years at @NCAR, I spent a few years at @carbonplan, where I was the technology director where we built some cool Zarr things like this and this. |
Beta Was this translation helpful? Give feedback.
-
|
Timezones have now been activated in the poll. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @jhamman, poll complete, looking forward! |
Beta Was this translation helpful? Give feedback.
-
|
I've added call in details for the kickoff meeting scheduled for August 16, 2023 at 9a PT. |
Beta Was this translation helpful? Give feedback.
-
|
@MSanKeys963 could you please add the call details to the community calls calendar? |
Beta Was this translation helpful? Give feedback.
-
@jakirkham - Done. 💯 |
Beta Was this translation helpful? Give feedback.
-
|
Update: I've included the link for the project board in the description. |
Beta Was this translation helpful? Give feedback.
-
|
Update: The refactoring kick-off meeting is now on the Zarr Community Calendar. |
Beta Was this translation helpful? Give feedback.
-
|
Update: As discussed during the kick-off meeting, I've added bi-weekly standup meetings for the refactor group to the Zarr Community Calendar. See zarr-developers/community-calls#24. Meeting details here: https://zarr.dev/community-calls/ |
Beta Was this translation helpful? Give feedback.
-
|
Hi all. 👋🏻 We have our first standup meeting tomorrow. Meeting details are in the community calendar. CC: @jhamman @cisaacstern @tomwhite @d-v-b @maxrjones @norlandrhagen @zarr-developers/steering-council |
Beta Was this translation helpful? Give feedback.
-
|
For those following along, I've updated the project board with an initial set of V3 compat tickets. More tickets will certainly need to be added but this seemed like a good first tranche. |
Beta Was this translation helpful? Give feedback.
-
|
Meeting notes from today's meeting: https://hackmd.io/Rbh6oae8S7mNU-CPWWwPaw |
Beta Was this translation helpful? Give feedback.
-
|
Given that Earthmover (presumably including @jhamman) are presenting at the zarr refactor meeting time this week, I'm assuming we will skip this week's meeting? |
Beta Was this translation helpful? Give feedback.
-
|
works for me |
Beta Was this translation helpful? Give feedback.
-
|
Came here to propose the same thing. Let's skip this week's meeting. @MSanKeys963 - can you cancel the calendar event for this week? |
Beta Was this translation helpful? Give feedback.
-
|
Done. 💯 |
Beta Was this translation helpful? Give feedback.
-
|
We'll have our next refactor meeting today (in 45 minutes). Today, I'm going to introduce a straw-man proposal that we integrate Zarrita's Array and Group classes (which today support both v2 and v3) into Zarr-Python. cc @normanrz who has been working hard on this library lately (and unfortunately is out this week). |
Beta Was this translation helpful? Give feedback.
-
|
Are we keeping notes from this meeting? Would it be possible to publish them to the website (as we do with other Zarr meetings)? cc @MSanKeys963 @jhamman (who may have thoughts here) |
Beta Was this translation helpful? Give feedback.
-
|
We will have our next working group meeting tomorrow at 9a PT (@MSanKeys963 updated the calendar invite). In advance of this meeting, I have prepared a draft design doc for Zarr-Python 3.0 which includes many of the ideas we have been discussing at this meeting. It also provides a way to bring Zarr-Python and Zarrita back together. If folks have time to look over this (and comment) ahead of the meeting tomorrow, that would be great. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks to @normanrz, @d-v-b, @dcherian, and @monodeldiablo for early comments on this doc. |
Beta Was this translation helpful? Give feedback.
-
|
This doc has been shared more broadly now: #1569 Next week's meeting will also be focused on this topic. Time permitting, we'll turn the discussion toward early actions that we can take. |
Beta Was this translation helpful? Give feedback.
-
|
At yesterday's, there was a request to document ways for others to jump in to the v3 fun. I've started to create a list of issues that would be good ways to help with the effort: https://github.com/zarr-developers/zarr-python/issues?q=is%3Aissue+is%3Aopen+label%3AV3+label%3A%22help+wanted%22 I'm sure we'll be opening more issues as we go but this is the list that we came up yesterday. |
Beta Was this translation helpful? Give feedback.
-
|
We'll have a check in today at the regular time. Expecting it to be relatively low head count due to the upcoming holiday but I'll be there to discuss progress toward v3. |
Beta Was this translation helpful? Give feedback.
-
|
I'll be there. |
Beta Was this translation helpful? Give feedback.
-
|
Are these meetings still occurring? I tried joining the zoom link just now but no-one here... |
Beta Was this translation helpful? Give feedback.
-
|
according to this calendar the next refactoring meeting is next week |
Beta Was this translation helpful? Give feedback.
-
|
@TomNicholas: https://zarr.dev/community-calls/ is one true source for all Zarr related meetings. (there's a Google calendar at the bottom) |
Beta Was this translation helpful? Give feedback.
-
|
This meeting is happening today (in 25 minutes). Hope to see folks there! |
Beta Was this translation helpful? Give feedback.
-
|
For folks following this issue... Reminding you that this meeting is still happening (now every Friday at 7a PT). We're closing in on the 3.0 release and I want to make sure we're all aligned on the remaining release blockers. Some highlights in the past few weeks:
For tomorrow's meeting, here are some specific technical topics I'd like to discuss are:
Hope folks can make the meeting but if not, see you around the GitHub! |
Beta Was this translation helpful? Give feedback.
-
Hi everyone! 👋🏻
Recently, we had meetings on
July 6th and 7th, 2023(meeting notes) led by @rabernat to discuss and decide the path forward for Zarr-Python development. After a good discussion with the attendees and gauging their interest, we decided to divide the larger group into two groups; they are:Thank you @JackKelly and @jhamman, for stepping up! 🙏🏻
What's next? 👀
This discussion thread aims to kick off the 🧑🏻💻 Refactoring 🧑🏻💻 working group and hold any top-level discussions related to the development work.
Here's the project board to organise and track progress: https://github.com/orgs/zarr-developers/projects/5
Thank you, everyone, for joining the meetings and sharing your insights. Please feel free to ask any questions.
@jhamman, please take it from here.
Beta Was this translation helpful? Give feedback.
All reactions