Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make [Try]IntoCtx borrow instead of copying #47

Closed
wants to merge 4 commits into from

Conversation

willglynn
Copy link
Contributor

This PR changes [Try]IntoCtx to take &self instead of self. It very likely breaks every downstream project that calls cwrite() or pwrite(), but in a trivial way – callers need only specify &.

This closes #46 and also incidentally closes #42.

This assumes that #[repr(packed)] types are also Copy, which is a thing
we can't verify in this context. At least we're in good company: the
built-in #[derive]s all require #[derive(Copy)] for packed structs.

Further discussion:
  m4b#46 (comment)
@willglynn
Copy link
Contributor Author

I'm working up a big PR for goblin that addresses this and the earlier Size/Units change, which seems like a representative way to evaluate the impact. (Also, I broke it/I bought it?)

@willglynn
Copy link
Contributor Author

willglynn commented Nov 2, 2018

I opened PRs in both goblin and faerie. Good thing too; I had missed updating the IOwrite trait.

@@ -163,7 +163,7 @@ fn write_io() -> Result<(), scroll::Error> {
let mut bytes = [0x0u8; 10];
let mut cursor = Cursor::new(&mut bytes[..]);
cursor.write_all(b"hello")?;
cursor.iowrite_with(0xdeadbeef as u32, BE)?;
cursor.iowrite_with(&(0xdeadbeef as u32), BE)?;
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This and the 3 others are a huge PR, will take me a bit to review, appreciate your patience, @willglynn.
I would like to note that having to take the reference now of variables, literals, makes me pretty sad...
I feel like it jars slightly with the original and basic purpose of scroll, which was to "serialize" numbers and PODs.
I do wish there had been a little more discussion before this enormous (and amazing PR!), specifically, whether we can have our cake and eat it too w.r.t. TryInto for the primitives.
Specifically, I'm wondering if there's a compromise, so that we can still pass either references or owned values to the pwrite/iowrite.
Maybe it's just me but i do find it a usability paper cut and find it less attractive that I have to take references to pwrite now, but perhaps I'm used to the old way.
So, is there some generic magic we can use to do this? Guessing wildy and randomly:

  1. Have someone take AsRef
  2. Have scroll_derive derive the reference versions for any user type (i'm ok with taking references of user structs)
  3. ???

On the other hand, in my case where the struct was too large to be put on the stack, and I had to box it, even if we did say 2., this would still probably eventually blow the stack? Or perhaps not. Have to think about it, and its late and a Thursday :)
Similarly, and against my original sadness, if I'm able to pwrite out a huge boxed struct that auto-derives pwrite, that would actually make me happy and would counteract any bitterness of no longer being able to pwrite raw deadbeefs :D
Dunno, just some thoughts, feel free to respond whenever you get the time (and that goes for anyone else who's watch).
Regardless, thanks for your amazing work here @willglynn and the robustness and tenacity of the changes, great work!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cheers! And yeah, this is why I went and did these PRs even with limited discussion: I didn't know how this concept would look until it was done 😄

I'll ponder the generics question. There might be a way we can make pwrite() be &-agnostic while keeping TryIntoCtx take &self – that'd let you pwrite() either by value or by reference without consuming self during the write.

@willglynn
Copy link
Contributor Author

If we say:

    /// Write `n` of type `T` at offset `I` with context `Ctx`
    /// # Example
    /// ```
    /// use scroll::{Pwrite, Pread, LE};
    /// let mut bytes: [u8; 8] = [0, 0, 0, 0, 0, 0, 0, 0];
    /// bytes.pwrite_with(0xbeefbeef, 0, LE).unwrap();
    /// assert_eq!(bytes.pread_with::<u32>(0, LE).unwrap(), 0xbeefbeef);
    fn pwrite_with<T,V>(&mut self, n: V, offset: usize, ctx: Ctx) -> result::Result<usize, E>
        where T: TryIntoCtx<Ctx, <Self as Index<RangeFrom<usize>>>::Output, Error = E> + ?Sized,
              V: Borrow<T>,
    {
        let len = self.measure_with(&ctx);
        if offset >= len {
            return Err(error::Error::BadOffset(offset).into())
        }
        let dst = &mut self[offset..];
        n.borrow().try_into_ctx(dst, ctx)
    }

These work:

  • bytes.pwrite_with(0xbeefbeef, 0, LE)
  • bytes.pwrite_with(&0xbeefbeef, 0, LE)

This does not work, because now there's two type parameters:

  • bytes.pwrite_with::<u32>(0xbeefbeef, 0, LE)

These also work:

  • bytes.pwrite_with::<u32,_>(0xbeefbeef, 0, LE)
  • bytes.pwrite_with::<_,u32>(0xbeefbeef, 0, LE)
  • bytes.pwrite_with::<u32,u32>(0xbeefbeef, 0, LE)

@philipc
Copy link
Contributor

philipc commented Nov 3, 2018

Using Borrow makes sense to me. Does pwrite_with::<...> get used much in practice? I guess it's only for constants, and for those I think bytes.pwrite_with(0xbeefbeef_u32, 0, LE) is clearer.

@m4b
Copy link
Owner

m4b commented Nov 8, 2018

I'm pretty excited about this version @willglynn ! I checked out your branch this weekend and played around with it, if nothing else comes up, I think this might be the way forward :)

So we can now do stuff like this:

#[test]
fn pwrite_big_struct() {
    use scroll::{LE, Pwrite};
    #[derive(Pread, Pwrite, Copy, Clone)]
    struct Big {
        arr: [u8; 100000000],
    }
    let big: Box<Big> = box (unsafe { ::std::mem::uninitialized() });//Big { arr: [0; 100000000] };
    let mut bytes = vec![0; 100000001];
    //let big = box bytes.pread_with::<Big>(0, LE).unwrap();
    let _ = bytes.pwrite_with(&* big, 0, LE).unwrap();
    assert!(false);
}

which was my original pain point when trying to write a large struct which was boxed :D (Preading is still an issue, and boxing doesn't fix it atm, but I think we could in principle fix that too, at another time)

Just grepping through a couple of codebases, I believe @philipc is correct (as usual) and afaics only goblin atm uses pwrite::<T> and this is because it relies on eliding an into() call, which is easily fixed (and probably better with an explicit into anyway):

src/mach/symbols.rs
277:                (bytes.pwrite_with::<Nlist32>(self.into(), 0, le)?)
280:                (bytes.pwrite_with::<Nlist64>(self.into(), 0, le)?)

src/mach/segment.rs
201:            bytes.pwrite_with::<Section64>(self.into(), 0, ctx.le)?;
203:            bytes.pwrite_with::<Section32>(self.into(), 0, ctx.le)?;
374:            bytes.pwrite_with::<SegmentCommand64>(self.into(), 0, ctx.le)?;
376:            bytes.pwrite_with::<SegmentCommand32>(self.into(), 0, ctx.le)?;

So yea, overall really like this direction, seems like with a minor sacrifice we can keep the owned usage for literals (and other Copyables?). If we do end up going this route, I think we'll need some explicit documentation about why passing a type parameter via turbofish is banned/problematic, etc.

Anyway, great work !

@willglynn
Copy link
Contributor Author

Cool! I plan to:

  • Keep the [Try]IntoCtx traits taking &self,
  • Keep the scroll_derive changes around &self,
  • Change the Pwrite and IOwrite traits to use Borrow as described above,
  • Reset everything that's not in the list above,
  • Fix everything that's broken

I'll comment when that's done and I've force-pushed the results.

@m4b
Copy link
Owner

m4b commented Dec 17, 2018

Friendly ping @willglynn . Would definitely like to see this get pushed through? Do you have a branch WIP anywhere, or need to offload some work to someone else? Hoping we didn't lose steam on it as i think its definitely something that's needed :)

@willglynn
Copy link
Contributor Author

@m4b 👍 I've been traveling and haven't looped back to this. My most current work is still in this branch, and it needs to be re-worked per the plan above. I expect to have more time right at year's end.

@m4b
Copy link
Owner

m4b commented Dec 19, 2018

No worries, thanks for the update! :)

@m4b
Copy link
Owner

m4b commented Apr 6, 2019

@willglynn gentle ping :) is there any chance of continuing here, or should we explore other options? If you're busy don't worry at all, or don't want to work on it anymore, don't feel bad at all/responsible whatever, just trying to get a feel for how to proceed/what to do.
If you get a chance let me know; maybe someone else can pick it up or I can take a look at it, etc.
As I understood it its basically working, modulo using borrow to deal with the ergonomic issue of pwriting constants?

@m4b m4b mentioned this pull request Apr 6, 2019
@willglynn
Copy link
Contributor Author

Sorry! I've been traveling some more and haven't got back to this. Probably the right thing to do is to reset to master to integrate the latest changes, change the interfaces as described upthread, and then just chase the build errors until they're gone again. This is still a thing I'm interested in doing, but if someone else beats me to it, that'd be cool too 👍

@m4b
Copy link
Owner

m4b commented Sep 6, 2019

gosh i don't even know where to begin to move this as a PR into the scroll repo :(

oops i thought this PR was in old scroll_derive repo, ok never mind :)

@m4b
Copy link
Owner

m4b commented Jan 25, 2021

I cargo fmt scroll and scroll derive; was hoping something like this would get resolved, but doesn't seem like it's going to happen, and now that fmt has happened there will be conflicts everywhere. going to close this; we can revisit again, or possibly never :)

@m4b m4b closed this Jan 25, 2021
@m4b m4b mentioned this pull request Nov 6, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

IntoCtx/TryIntoCtx on borrowed values Packed structs and Scroll
3 participants