SharpDevelop Community

Get your problems solved!
Welcome to SharpDevelop Community Sign in | Join | Help
in Search

ZipInputStream: Length property throws exception InvalidOperationException

Last post 06-16-2009 10:31 AM by OliverReed. 6 replies.
Page 1 of 1 (7 items)
Sort Posts: Previous Next
  • 06-11-2009 4:20 PM

    ZipInputStream: Length property throws exception InvalidOperationException

    Hi folks,

    I have a .zip file sourced from a third party so I don't know how it was generated. When I reading the Length property it throws an exception. The exception originates because entry.size is zero.

    I've stepped through the SharpZibLib source code and found that in the first local file header, bit 3 of flags is set to 1, so the filesize in the local header is set to zero and (as I understand it) will written in the data desciptor section following the file data.

    But the length never seems to be read in and is always zero.

    Here are the properties for each entry:

            CanDecompress    true    bool
            CentralHeaderRequiresZip64    false    bool
            Comment    null    string
            comment    null    string
            compressedSize    0    ulong
            CompressedSize    -1    long
            CompressionMethod    Deflated    ICSharpCode.SharpZipLib.Zip.CompressionMethod
            crc    0    uint
            Crc    -1    long
            CryptoCheckValue    134    byte
            cryptoCheckValue_    134    byte
            DateTime    {10/06/2009 16:54:44}    System.DateTime
            DosTime    986351318    long
            dosTime    986351318    uint
            externalFileAttributes    -1    int
            ExternalFileAttributes    -1    int
            extra    null    byte[
            ExtraData    null    byte[
            flags    8    int
            Flags    8    int
            forceZip64_    false    bool
            HasCrc    false    bool
            HostSystem    0    int
            IsCrypted    false    bool
            IsDirectory    false    bool
            IsDOSEntry    true    bool
            IsFile    true    bool
            IsUnicodeText    false    bool
            known    Time    ICSharpCode.SharpZipLib.Zip.ZipEntry.Known
            LocalHeaderRequiresZip64    false    bool
            method    Deflated    ICSharpCode.SharpZipLib.Zip.CompressionMethod
            name    "Test.jar"    string
            Name    "Test.jar"    string
            offset    0    long
            Offset    0    long
            Size    -1    long
            size    0    ulong
            Version    20    int
            versionMadeBy    45    ushort
            VersionMadeBy    45    int
            versionToExtract    20    ushort
            ZipFileIndex    -1    long
            zipFileIndex    -1    long

     

    Any help would be much appreciated

    Ollie

  • 06-12-2009 2:07 AM In reply to

    Re: ZipInputStream: Length property throws exception InvalidOperationException

    Hi Oliver,

    You certainly have a somewhat difficult problem there but you have done a remarkable job of investigating and debugging, and you have a most excellent name, so I'll chip in.

    You've established that the local header does not have the file size and it should be provided in a data descriptor following the data. Can you find out what follows the data and if it is the data descriptor? Unfortunately, there are some differences in the implementation of these extra descriptors between utilities.

    These descriptors are always 4 bytes which are "PK" and two byes which give the type. But as I said, usage can vary beteen programs particularly on the lesser-used descriptors. The zipfile format flexibility can also be a hassle.

    I would either:

    * watch what is read in next in the code

    * view the file in a hex editor or dumper

    Have a look at the file format, :

    http://www.pkware.com/documents/casestudies/APPNOTE.TXT

    If you do not have a hex editor, I use the Hex Editor plugin for Notepad++. I also wrote a small program to view in hex with optional decimal and blank lines, to make my life easier. Let me know if you want it.

    Any q'd pls ask

    David

  • 06-12-2009 12:14 PM In reply to

    Re: ZipInputStream: Length property throws exception InvalidOperationException

    Hi David,

    Thanks for the links. I've downloaded Notepad++ and analysed the file according to the file format. It looks perfectly normal - yet SharpZipLib cannot determine the file length.

    If I unzip the file and then re-zip the contents using Windows XP, SharpZipLib can read it just fine.

    I've included my analysis below - all values are shown translated from their Little Endian representations in the file.

    Any thoughts would be much appreciated.

    Ollie.

     

    Local file header #1:

            local file header signature     4 bytes  (04034b50)
            version needed to extract       2 bytes  (0014)
            general purpose bit flag        2 bytes  (0008)
            compression method              2 bytes  (0008)
            last mod file time              2 bytes  (86d6)
            last mod file date              2 bytes  (3aca)
            crc-32                          4 bytes  (00000000)
            compressed size                 4 bytes  (00000000)
            uncompressed size               4 bytes  (00000000)
            file name length                2 bytes  (0026)
            extra field length              2 bytes  (0000)
        filename            (38 bytes)

        filedata            (426842 bytes)

    Data Descriptor #1:

        Data description hder sig      4 bytes  (08074b50)
            crc-32                          4 bytes  (61282b20)
            compressed size                 4 bytes  (0006835a)
            uncompressed size               4 bytes  (0006d050)

    Local file header #2:

            local file header signature     4 bytes  (04034b50)
            version needed to extract       2 bytes  (0014)
            general purpose bit flag        2 bytes  (0008)
            compression method              2 bytes  (0008)
            last mod file time              2 bytes  (86d6)
            last mod file date              2 bytes  (3aca)
            crc-32                          4 bytes  (00000000)
            compressed size                 4 bytes  (00000000)
            uncompressed size               4 bytes  (00000000)
            file name length                2 bytes  (0026)
            extra field length              2 bytes  (0000)
        filename            (38 bytes)

        filedata            (2458 bytes)

    Data Descriptor #2:

        Data description hder sig      4 bytes  (08074b50)
            crc-32                          4 bytes  (b6131be7)
            compressed size                 4 bytes  (0000099a)
            uncompressed size               4 bytes  (00000e3e)


     Central directory structure:

     File header #1:
            central file header signature   4 bytes  (02014b50)
            version made by                 2 bytes     (0014)   
            version needed to extract       2 bytes  (0014)
            general purpose bit flag        2 bytes  (0008)
            compression method              2 bytes  (0008)
            last mod file time              2 bytes  (86d6)
            last mod file date              2 bytes  (3ACA)
            crc-32                          4 bytes  (61282B20)
            compressed size                 4 bytes  (0006835a)
            uncompressed size               4 bytes  (0006d050)
            file name length                2 bytes  (0026)
            extra field length              2 bytes  (0000)
            file comment length             2 bytes  (0000)
            disk number start               2 bytes  (0000)
            internal file attributes        2 bytes  (0000)
            external file attributes        4 bytes  (00000000)
            relative offset of local header 4 bytes  (00000000)
            filename            (38 bytes)

     File header #2:
            central file header signature   4 bytes  (02014b50)
            version made by                 2 bytes     (0014)   
            version needed to extract       2 bytes  (0014)
            general purpose bit flag        2 bytes  (0008)
            compression method              2 bytes  (0008)
            last mod file time              2 bytes  (86d6)
            last mod file date              2 bytes  (3ACA)
            crc-32                          4 bytes  (b6131be7)
            compressed size                 4 bytes  (0000099a)
            uncompressed size               4 bytes  (00000e3e)
            file name length                2 bytes  (0026)
            extra field length              2 bytes  (0000)
            file comment length             2 bytes  (0000)
            disk number start               2 bytes  (0000)
            internal file attributes        2 bytes  (0000)
            external file attributes        4 bytes  (00000000)
            relative offset of local header 4 bytes  (000683ae)
            filename            (38 bytes)

    End of central directory record:
            end of central dir signature    4 bytes  (06054b50)
            number of this disk             2 bytes  (0000)
            number of the disk with the
            start of the central directory  2 bytes  (0000)
            total number of entries in the
            central directory on this disk  2 bytes  (0002)
            total number of entries in
            the central directory           2 bytes  (0002)
            size of the central directory   4 bytes  (000000a8)
            offset of start of central
            directory with respect to
            the starting disk number        4 bytes  (00068d9c)
            .ZIP file comment length        2 bytes  (0000)
            .ZIP file comment       (variable size)

     

  • 06-15-2009 1:52 AM In reply to

    Re: ZipInputStream: Length property throws exception InvalidOperationException

    Well, that data looks reasonable to me.

    Have you checked whether ZipInputStream.ReadDataDescriptor is running correctly?

    Also, what exception are you getting, and where? You said you are trying to read some Length property?

    Regards,

    David

  • 06-15-2009 12:52 PM In reply to

    Re: ZipInputStream: Length property throws exception InvalidOperationException

    Have you checked whether ZipInputStream.ReadDataDescriptor is running correctly?

    Bingo!

    Here is the code that was running...

    using (ZipInputStream zipInput = new ZipInputStream(fileStream))
    {
        ZipEntry entry = zipInput.GetNextEntry();
        while ((null != entry) && !foundFile)
        {
            if (entry.IsFile && Regex.Match(entry.Name, "^" + containedFile + "$", RegexOptions.IgnoreCase).Success)
            {
                binaryFile = new byte[zipInput.Length]; //Exception occurs here
                zipInput.Read(binaryFile, 0, binaryFile.Length);
                entryName = entry.Name;
                foundFile = true;
            }
        }
        zipInput.Close();
    }

    Stepping through the code, the ReadDataDescriptor method was not called until the next GetNextEntry method was called. This filled out the missing details of the current entry member in the ZipInputStream class - and then sets it to null in readiness for the GetNextEntry processing. No use for what I'm trying to do.

    The solution to my problem is to replace the above code with the following:

    using (FileStream fileStream = new FileStream(targetFile, FileMode.Open, FileAccess.Read, FileShare.Read))
    {
        using (ZipFile zf = new ZipFile(fileStream))
        {
            ZipEntry entry = zf.GetEntry(containedFile);

            if (entry != null)
            {
                using (Stream s = zf.GetInputStream(entry))
                {
                    binaryFile = new byte[entry.Size];
                    s.Read(binaryFile, 0, binaryFile.Length);
                }
            }
        }
        fileStream.Close();
    }

    This ensures that the entry variable is fully populated when I come to use its members and works for both file formats (those with and without data descriptor sections.)

     

    Thanks David, for pointing me in the right direction. Much appreciated.

     

     

  • 06-16-2009 2:44 AM In reply to

    Re: ZipInputStream: Length property throws exception InvalidOperationException

    Hi Oliver,

    Glad to be able to help. Makes it all worthwhile.

    ReadDataDescriptor is not called until after the data has been read because it physycally occurs after the file data in the stream. Which is why the length was not known in time. Good work figuring it out.

    May i recommend a slight change to the code. Where you have
         binaryFile = new byte[entry.Size];
    that line will cause problems if the file size is too large. You would see OutOfMemory exceptions etc. Instead, you can use a fixed size buffer and copy through that to the output stream. For example ...

        byte[ ] buffer = new byte [4096] ;
        StreamUtils.Copy(s, streamWriter, buffer);
    That assumes you have a StreamWriter to output to.

    Another idea ... If you find the entry.Size to be unreliable or unobtainable, you could use the StreamUtils.Copy to copy to a memory stream, and then convert that into a byte array.

    Best regards
    David

  • 06-16-2009 10:31 AM In reply to

    Re: ZipInputStream: Length property throws exception InvalidOperationException

    Hi David,

    Thanks for the extra info - I do understand what you're driving at here.

    At the moment, each individual file within a ZIP is no bigger than 500K and is likely to stay as such due to the nature of the data it contains.

    Since the previous method (i.e. my original code) managed to read such files with a single Read, I think that the new code will also be able to cope. (Speed of response is one consideration for leaving the code as it is too).

    But if we we do start to encounter OutOfMemory exceptions, this is the first place I'll look.

    Thanks again for taking the time to provide more valuable information.

    Ollie.

Page 1 of 1 (7 items)
Powered by Community Server (Commercial Edition), by Telligent Systems
Don't contact us via this (fleischfalle@alphasierrapapa.com) email address.