Test failures on Windows

[pombreda]

These tests fail for mysterious reasons.
They are all somehow related to writing archives.

I wonder why the new_archive_entry is reused in the paths loop when adding files rather than having a new entry? Nevertheless the failure might be due to the fact that somehow the file being added may be already opened for writing? Not sure. To investigate.. This could all be due to a single problem.
Possibly a build problem, though the built lib works rather well otherwise.

FWIW errno 22 is EINVAL

================================== FAILURES ===================================
________________________________ test_convert _________________________________

    def test_convert():

        # Collect information on what should be in the archive
        tree = treestat('libarchive')

        # Create an archive of our libarchive/ directory
        buf = bytes(bytearray(1000000))
        with memory_writer(buf, 'gnutar', 'xz') as archive1:
>           archive1.add_files('libarchive/')

tests\test_convert.py:21:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
libarchive\write.py:61: in add_files
    r = read_next_header2(read_p, entry_p)
libarchive\ffi.py:85: in check_int
    raise archive_error(args[0], retcode)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

archive_p = 3609976, retcode = -25

    def archive_error(archive_p, retcode):
        msg = _error_string(archive_p)
>       raise ArchiveError(msg, errno(archive_p), retcode, archive_p)
E       ArchiveError: <unprintable ArchiveError object>

libarchive\ffi.py:69: ArchiveError
____________________________ test_entry_properties ____________________________

    def test_entry_properties():

        buf = bytes(bytearray(1000000))
        with memory_writer(buf, 'gnutar') as archive:
>           archive.add_files('README.rst')

tests\test_entry.py:17:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <libarchive.write.ArchiveWrite object at 0x03460150>
paths = ('README.rst',), write_p = 3609224, block_size = 10240
entry_p = 47076136

    def add_files(self, *paths):
        """Read the given paths from disk and add them to the archive.
            """
        write_p = self._pointer

        block_size = ffi.write_get_bytes_per_block(write_p)
        if block_size <= 0:
            block_size = 10240  # pragma: no cover

        with new_archive_entry() as entry_p:
            entry = ArchiveEntry(None, entry_p)
            for path in paths:
                with new_archive_read_disk(path) as read_p:
                    while 1:
                        r = read_next_header2(read_p, entry_p)
                        if r == ARCHIVE_EOF:
                            break
                        entry.pathname = entry.pathname.lstrip('/')
                        read_disk_descend(read_p)
                        write_header(write_p, entry_p)
                        try:
>                           with open(entry_sourcepath(entry_p), 'rb') as f:
                                while 1:
E                               IOError: [Errno 13] Permission denied: '\\\\?\\c:\\w421\\python-libarchive-c\\README.rst'

libarchive\write.py:68: IOError
________________________________ test_buffers _________________________________

tmpdir = local('c:\\tmp\\pytest-49\\test_buffers0')

    def test_buffers(tmpdir):

        # Collect information on what should be in the archive
        tree = treestat('libarchive')

        # Create an archive of our libarchive/ directory
        buf = bytes(bytearray(1000000))
        with libarchive.memory_writer(buf, 'gnutar', 'xz') as archive:
>           archive.add_files('libarchive/')

tests\test_rwx.py:23:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
libarchive\write.py:61: in add_files
    r = read_next_header2(read_p, entry_p)
libarchive\ffi.py:85: in check_int
    raise archive_error(args[0], retcode)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

archive_p = 3668736, retcode = -25

    def archive_error(archive_p, retcode):
        msg = _error_string(archive_p)
>       raise ArchiveError(msg, errno(archive_p), retcode, archive_p)
E       ArchiveError: H??????: Couldn't visit directory (errno=22, retcode=-25, archive_p=3668736)

libarchive\ffi.py:69: ArchiveError
___________________________________ test_fd ___________________________________

tmpdir = local('c:\\tmp\\pytest-49\\test_fd0')

    def test_fd(tmpdir):
        archive_file = open(tmpdir.strpath+'/test.tar.bz2', 'w+b')
        fd = archive_file.fileno()

        # Collect information on what should be in the archive
        tree = treestat('libarchive')

        # Create an archive of our libarchive/ directory
>       with libarchive.fd_writer(fd, 'gnutar', 'bzip2') as archive:
            archive.add_files('libarchive/')

tests\test_rwx.py:45:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
c:\Python27\Lib\contextlib.py:17: in __enter__
    return self.gen.next()
libarchive\write.py:116: in fd_writer
    ffi.write_open_fd(archive_p, fd)
libarchive\ffi.py:85: in check_int
    raise archive_error(args[0], retcode)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

archive_p = 3669416, retcode = -30

    def archive_error(archive_p, retcode):
        msg = _error_string(archive_p)
>       raise ArchiveError(msg, errno(archive_p), retcode, archive_p)
E       ArchiveError: Failed to clean up compressor (errno=22, retcode=-30, archive_p=3669416)

libarchive\ffi.py:69: ArchiveError
_________________________________ test_files __________________________________

tmpdir = local('c:\\tmp\\pytest-49\\test_files0')

    def test_files(tmpdir):
        archive_path = tmpdir.strpath+'/test.tar.gz'

        # Collect information on what should be in the archive
        tree = treestat('libarchive')

        # Create an archive of our libarchive/ directory
        with libarchive.file_writer(archive_path, 'ustar', 'gzip') as archive:
>           archive.add_files('libarchive/')

tests\test_rwx.py:70:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
libarchive\write.py:61: in add_files
    r = read_next_header2(read_p, entry_p)
libarchive\ffi.py:85: in check_int
    raise archive_error(args[0], retcode)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

archive_p = 3609400, retcode = -25

    def archive_error(archive_p, retcode):
        msg = _error_string(archive_p)
>       raise ArchiveError(msg, errno(archive_p), retcode, archive_p)
E       ArchiveError: Hibarchive/: Couldn't visit directory (errno=22, retcode=-25, archive_p=3609400)

libarchive\ffi.py:69: ArchiveError
_____________________________ test_custom_writer ______________________________

    def test_custom_writer():

        # Collect information on what should be in the archive
        tree = treestat('libarchive')

        # Create an archive of our libarchive/ directory
        blocks = []

        def write_cb(data):
            blocks.append(data[:])
            return len(data)

        with libarchive.custom_writer(write_cb, 'zip') as archive:
>           archive.add_files('libarchive/')

tests\test_rwx.py:97:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
libarchive\write.py:61: in add_files
    r = read_next_header2(read_p, entry_p)
libarchive\ffi.py:85: in check_int
    raise archive_error(args[0], retcode)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

archive_p = 3666496, retcode = -25

    def archive_error(archive_p, retcode):
        msg = _error_string(archive_p)
>       raise ArchiveError(msg, errno(archive_p), retcode, archive_p)
E       ArchiveError: 9: Couldn't visit directory (errno=22, retcode=-25, archive_p=3666496)

libarchive\ffi.py:69: ArchiveError
===================== 6 failed, 5 passed in 1.53 seconds ======================

[pombreda]

@Changaco I cannot understand what this code does, why and how. Sorry for my crass ignorance, but I need your help.
https://github.com/Changaco/python-libarchive-c/blob/master/libarchive/write.py#L56

here: https://github.com/Changaco/python-libarchive-c/blob/master/libarchive/write.py#L61
Why are you trying to read the files to add to the archive as if they were archives themselves?

[Changaco]

Why are you trying to read the files to add to the archive as if they were archives themselves?

Because that's the correct way to do it. libarchive handles walking the file system and filling the entry struct, the python code reads the actual content of the files and ties everything together.

[pombreda]

FWIW, I do not care for writing archives since I use libarchive only to extract things, but I am still not sure why you would use a combo of opening and reading the file in Python and using the directory traversal of libarchive. If read things correctly, libarchive can do both traversal and file reading?

[pombreda]

@Changaco I still think that there is something that is likely not to work at all on posix for special files here: https://github.com/Changaco/python-libarchive-c/blob/master/libarchive/write.py#L68
Opening a FIFO or character/device file like this usually has funny consequences such has a hanging.
libarchive knows how to handle these alright afaik, so reading in python is likely not covering all the cases. Doing a test with a temp os.mkfifo is likely to break IMHO

[pombredanne]

@Changaco any idea on this?

[Changaco]

Looks like I forgot to answer here.

If read things correctly, libarchive can do both traversal and file reading?

Does it? I don't remember seeing a function to do the actual file reading in libarchive.