SharpDevelop Community

I'm trying to compress file using ZIp format. Though UseZip64 is set to Dynamic a simple text file that contains one line is compressed to

Zip64 archive.

 I assume that it should choose to use zip64 or not based on file size or number of files in archive.

Is my assumption correct?

Thats not quite the full story but in a broad brush sense that is so.  If the size for a file is unknown it will use Zip64.  Setting the entry size before the entry is added may be the fix for you.

Zip64 can also end up being used with DataDescriptors which are added when the outputstream connot seek.

hth, -jr- 

******************************
Please ignore this posting, and see suggestion in below posting...
******************************

Hi.

I Think there IS indeed a problem with UseZip64 set to Dynamic
It does seem to force it to use Zip64. Even using a plain filestream.

I have found numerous threads that indicate the same thing:
  http://community.sharpdevelop.net/forums/p/8898/25228.aspx#25228
  http://community.sharpdevelop.net/forums/p/7957/23479.aspx#23479
  http://community.sharpdevelop.net/forums/p/6882/19610.aspx#19610
  http://community.sharpdevelop.net/forums/p/6616/18875.aspx#18875
  http://community.sharpdevelop.net/forums/p/10637/29328.aspx#29328
and post 8 of
  http://community.sharpdevelop.net/forums/p/10954/30217.aspx#30217

I tried with the below code (simplified) and a dump of example output can be seen in post 8 above

Of course in this simple example I could get the correct filesizes before PutEntry.
But in real-world the data I'm zipping originates from some SQL, And I have no way of telling the total size up front, even though it is almost always below 4Gb. So I would much prefer if it only used Zip64 in the rare cases where it is really needed.

Best Regards
Eske Rahn

        Dim BufSiz As Integer = 65536
        Dim Buffer(BufSiz - 1) As Byte
        Dim MyZippedOutputStream As New FileStream(ZipName, FileMode.Create, FileAccess.Write)
        Dim MyZipOutputStream As New ZipOutputStream(MyZippedOutputStream)
        Try
            Dim MyZipEntry As ZipEntry
            For Each Fn As String In TextBoxSourceFiles.Lines
                MyZipEntry = New ZipEntry(Path.GetFileName(Fn)) ' strip path to keep it simple
                MyZipOutputStream.PutNextEntry(MyZipEntry)
                Try
                    Dim fsSou As FileStream = File.Open(Fn, FileMode.Open, FileAccess.Read)
                    Try
                        Do
                            Dim res As Integer = fsSou.Read(Buffer, 0, BufSiz)
                            If res = 0 Then Exit Do
                            MyZipOutputStream.Write(Buffer, 0, res)
                        Loop
                    Finally
                        fsSou.Close()
                    End Try
                Finally
                    MyZipOutputStream.Flush()
                    MyZipOutputStream.CloseEntry()
                End Try
            Next
        Finally
            MyZipOutputStream.Finish()
            MyZipOutputStream.Close()
        End Try

Hi,

It (finally) went in to my thick scull, that by the UseZip64 value "Dynamic" is meant "ByPreKnownFileSize"!!

But I have a rather simple idea to make it TRUELY dynamic!

**********************************************************************
EDIT: See below posting for actual implementation
**********************************************************************

The idea is on writeback of size into the Zip64-header to CHANGE it into a Zip32-header in the following way:

a) Change "Version made by" from 2Dh (45) to 14h (20).
b) Set correct lengths in the two base-length-fields.

This is it! I tested it manually, and it works! the 0001h extra field are ignored.

Optionally:
c) Change extradata "Lengths-data" identifier 0001h to something that is guranteed to be IGNORED by std. version 20 readers!

Here is a list of known values
  http://svn.assembla.com/svn/os2utils/unzip60f/proginfo/extrafld.txt
And a few less in this
  http://www.pkware.com/documents/casestudies/APPNOTE.TXT

An obvious choice would be "#Z"

Yes, I know this would 'waste' 20Bytes for each file less than 4Gb, but that is an acceptable 'loss'. To make normal files available to older wide-spread Zip32-software, such as build into XP.

Best Regards
Eske Rahn

I found a way to fix it.

Here are a list of the changes needed to make UseZip64=Dynamic FULLY dynamic.
This gives backwards compatible zip files, for all entries less than 4Gb.
The method is to patch back the Zip64 header to a zip32 header, with a superflous ExtraDataEntry containing the lengths.

This gives an extra cost of 20bytes per file versus Zip64=Off, and saves 20bytes per file versus Zip64=On

NOTE: The ExtendedData Zip64-length identifier 0001 is patched to 0x5A23="#Z" if zip64 is not needed
      The Zip32 lengths are patched using current position after CRC-patch, before patch of Zip64 lengths, so no extra variable is needed to keep both size-positions.

In the below, line numbers before & after "/" are for RELEASE version 0.85.5.452 resp. subversion number 508.

...To implement please start from the bottom, so linenumbers hold  :)


ZipOutputStream.CS
==================


------------------------
In .PutNextEntry L233/L246
------------------------
L320/L333 Expand&Modify
            if ( (useZip64_ == UseZip64.On) || ((entry.Size < 0) && (useZip64_ == UseZip64.Dynamic)) ) {
to
            bool TempZip64 = ((entry.Size < 0) && (useZip64_ == UseZip64.Dynamic));
            if ( useZip64_ == UseZip64.On ) {
------------------------
L349/L362 Remove these four lines
                if ( patchEntryHeader ) {
                    sizePatchPos = baseOutputStream_.Position;
                }

------------------------
L372/L385 Expand
            if (entry.LocalHeaderRequiresZip64) {
to
            if ( (entry.LocalHeaderRequiresZip64) || (TempZip64) ) {
------------------------
L405/L422 Modify
            if ( entry.LocalHeaderRequiresZip64 && patchEntryHeader ) {
to
            if (((entry.LocalHeaderRequiresZip64) || (TempZip64)) && patchEntryHeader) {
------------------------


------------------------
In .CloseEntry  L442/L467
------------------------
L461/L491 Expand before
            if (curEntry.Size < 0) {
to
            if ((useZip64_ == UseZip64.Dynamic) && (0xffffffff <= size)) {
                curEntry.ForceZip64();
            }
            bool Patch64to32 = ((useZip64_ == UseZip64.Dynamic) && (curEntry.Size < 0) && (!curEntry.IsZip64Forced()));
            if (curEntry.Size < 0) {
------------------------
L507/L542 Expand before
                baseOutputStream_.Seek(curPos, SeekOrigin.Begin);
to
                if (Patch64to32) {
                    baseOutputStream_.Seek(sizePatchPos-4, SeekOrigin.Begin);
                    WriteLeShort(0x5A23); // #Z special marker...
                } else {
                    baseOutputStream_.Seek(crcPatchPos-10, SeekOrigin.Begin);
                    WriteLeShort(curEntry.Version);
                }
                baseOutputStream_.Seek(curPos, SeekOrigin.Begin);
------------------------

------------------------
In variables block
------------------------
L811/L888 Update
        // Default is dynamic which is not backwards compatible and can cause problems
        // with XP's built in compression which cant read Zip64 archives.
to
        // Default is dynamic which is Only backwards compatible with files<4Gb. But can cause problems
        // with XP's built in compression which cant read larger Zip64-archived entries.
------------------------


That is it!
:-)