SharpDevelop Community

Get your problems solved!
Welcome to SharpDevelop Community Sign in | Join | Help
in Search

Looking for a way to use a compressed memory stream for writing and reading

Last post 05-20-2016 11:07 PM by DavidPierson. 9 replies.
Page 1 of 1 (10 items)
Sort Posts: Previous Next
  • 05-17-2016 11:00 PM

    Looking for a way to use a compressed memory stream for writing and reading

    Good time of the day everyone!

    I just came across a CodeProject page that pointed me at this community. It talked about using SharpZipLib used with Memory Stream. This is an interesting concept, but I have not been successful using the same approach :( He used this code:

                    m_msBZip2 = new MemoryStream();
                    m_osBZip2 = new BZip2OutputStream(m_msBZip2);
                    m_osBZip2.Write(Encoding.ASCII.GetBytes(txtRawData.Text), 0, txtRawData.Text.Length);
                    m_osBZip2.Finalize(); // this apparently writes out all data and makes it available in the memory stream

    Then it is possible to get the memory stream contents using ToArray(), but the stream is closed! How can I read from the underlying memory stream without having to create another copy of the compressed data in memory?

    Thank you!

  • 05-18-2016 7:01 AM In reply to

    Re: Looking for a way to use a compressed memory stream for writing and reading

    Hi,

    Don't have the code open right now, but before the Finalize, try adding

    m_osBZips.IsStreamOwner = false;

    This solves the problem for Zip streams so if the proprty is in BZip it will solve it also. It prevents the zip close from closing the underlying stream.

    Cheers, Dave

  • 05-18-2016 7:43 PM In reply to

    Re: Looking for a way to use a compressed memory stream for writing and reading

    Tried that, but there does not seem to be such property of BZip2OutputStream in the binary version that I downloaded from CodeProject.

    Also I tried building the library from the currently available sources on this site, but the build fails under VS 2012 as there is no "nameof" function. There is the property you mentioned there, so I obviously have an older version w/o that property, so if I was able to build the current your suggestion might just work.

     

    Nameof seems to be a new thing in 2013 VS, so how can I build under 2012?

     

    EDIT: I might try this solution: replace all "nameof(" with "NameOf.nameof(() => " and add the following class to the solution:

     

    using System;
    using System.Linq.Expressions;

    namespace ICSharpCode.SharpZipLib
    {
        public class NameOf
        {
            public static String nameof<T>(Expression<Func<T>> name)
            {
                MemberExpression expressionBody = (MemberExpression)name.Body;
                return expressionBody.Member.Name;
            }
        }
    }

  • 05-18-2016 8:53 PM In reply to

    Re: Looking for a way to use a compressed memory stream for writing and reading

    Okay, tried to remove Finalize and add IsStreamOwner = False but now I no longer have access to the compressed data written to the stream via ToArray(). Tried to use Flush() in place of Finalize() to no avail. Is there documentation about flushing the Zip stream?

                    m_msBZip2 = new MemoryStream();
                    m_osBZip2 = new BZip2OutputStream(m_msBZip2);
                    m_osBZip2.IsStreamOwner = false;
                    m_osBZip2.Write(Encoding.ASCII.GetBytes(txtRawData.Text), 0, txtRawData.Text.Length);
                    //m_osBZip2.Finalize();
                    //m_osBZip2.Close();
                    txtOutput.Text = Convert.ToBase64String(m_msBZip2.ToArray());  // here there are only 3 bytes in the array though compressed data should be about 100 bytes

     

  • 05-19-2016 5:05 AM In reply to

    Re: Looking for a way to use a compressed memory stream for writing and reading

    Hi,

    Sorry to hear about the nameof problem. Sounds like something coming from recent refactoring.

    Did you try just using .Close() and not the .Finalize().  That is what we have in the sample code in the wiki for Zip at

    https://github.com/icsharpcode/SharpZipLib/wiki/Zip-Samples
    See the "Create a Zip from/to a memory stream or byte array" sample.

    I haven't done much work at all with BZip2 so I'm not familiar with it as I am with the Zip/GZip. Not sure why the .ToArray would not work. Can you load the source into your VS?

    Dave

     

  • 05-19-2016 5:36 PM In reply to

    Re: Looking for a way to use a compressed memory stream for writing and reading

    The Wiki example seems to be using StreamUtils from Spring to copy data, I would like to avoid linking in any more libraries.

    So I ended up with this code:

     

                    m_msBZip2 = new MemoryStream();
                    m_osBZip2 = new BZip2OutputStream(m_msBZip2);
                    m_osBZip2.IsStreamOwner = false;
                    m_osBZip2.Write(Encoding.ASCII.GetBytes(txtRawData.Text), 0, txtRawData.Text.Length);
                    m_osBZip2.Close();

                    txtOutput.Text = Convert.ToBase64String(m_msBZip2.ToArray()); // now this has compressed bytes all right...

                    txtRawData.Clear();

                    m_msBZip2.Position = 0;
                    m_isBZip2 = new BZip2InputStream(m_msBZip2);
                    byte[ bytesUncompressed = new byte[m_isBZip2.Length];
                    m_isBZip2.Read(bytesUncompressed, 0, (int)m_isBZip2.Length); // but that returns only about 1/2 of original!
                    m_isBZip2.Close();
                    m_msBZip2.Close();
                    txtRawData.Text = Encoding.ASCII.GetString(bytesUncompressed);
                    m_isBZip2 = null;
                    m_msBZip2 = null;

    Some data seems to be lost during compression or de-compression. What am I doing wrong?

    Compressing this

     

    111 22 33333 44444 555555555 666666 7777777
    111 22 33333 44444 555555555 666666 7777777
    111 22 33333 44444 555555555 666666 7777777
    111 22 33333 44444 555555555 666666 7777777
    111 22 33333 44444 555555555 666666 7777777

     

    into this base64 representation

    QlpoOTFBWSZTWW7ZRm4AACZYADoSQAA/gCAAUIAAAmqlAMzVOl2uLFi7X5cWLqW1tepcXtYsGlg0tjSwaWDS/i7kinChIN2yjNw=

    then memory stream contents decompresses into this with loss of about 2/3:

    111 22 33333 44444 555555555 666666 7777777
    111 22 33333 44444 555555555

     Another approach was to implement Zip and I tried this code but got an exception at the commented line:

     

                    m_MemStream = new MemoryStream();
                    m_osZip = new ZipOutputStream(m_MemStream);
                    m_osZip.IsStreamOwner = false;

                    ZipEntry newEntry = new ZipEntry("zip");
                    m_osZip.PutNextEntry(newEntry);

                    m_osZip.Write(Encoding.ASCII.GetBytes(txtRawData.Text), 0, txtRawData.Text.Length);
                    m_osZip.CloseEntry();
                    m_osZip.Close();

                    m_MemStream.Position = 0;
                    txtOutput.Text = Convert.ToBase64String(m_MemStream.ToArray());

                    txtRawData.Clear();

                    m_MemStream.Position = 0;
                    m_isZip = new ZipInputStream(m_MemStream);
                    byte[ bytesUncompressed = new byte[m_isZip.Length]; // <---====  entry is null here, so exception is thrown
                    m_isBZip2.Read(bytesUncompressed, 0, (int)m_isZip.Length);
                    m_isZip.Close();
                    m_MemStream.Close();
                    txtRawData.Text = Encoding.ASCII.GetString(bytesUncompressed);
                    m_isZip = null;
                    m_MemStream = null;

  • 05-20-2016 12:33 AM In reply to

    Re: Looking for a way to use a compressed memory stream for writing and reading

    > The Wiki example seems to be using StreamUtils from Spring to copy data,

     

    Not so. The StreamCopy is about 3 lines of code in the SharpZipLib utils area.

  • 05-20-2016 12:43 AM In reply to

    Re: Looking for a way to use a compressed memory stream for writing and reading

    In your first example I think thje problem is this line

    m_isBZip2.Read(bytesUncompressed, 0, (int)m_isBZip2.Length); // but that returns only about 1/2 of original!

    In the wiki I preach about not trying to read all the data into a single buffer. I prefer to declare a 4K buffer and read through that in a loop. The StreamUtils.Copy does just that. Copy it out of the SharpZipLib code into your code. I think the error is that _isBZip2.Length is the compressed size. Since you can't always know in advance what it will uncompress to, replace that with a loop & copy approach.

    From memory it is something like

    int n = inputStreamRead(buffer, 0, size of buffer);
    while (n > 0) {
        outputStream.Write(buffer, 0 , n);
        n = inputStreamRead(buffer, 0, size of buffer);
    }

     

  • 05-20-2016 3:59 PM In reply to

    Re: Looking for a way to use a compressed memory stream for writing and reading

    That worked perfect, thanks a lot for the pointer!

     

                    byte[ buffer = new byte[4096];
                    int n = m_isBZip2.Read(buffer, 0, buffer.Length);
                    while (n > 0) {
                        txtRawData.Text += Encoding.ASCII.GetString(buffer, 0, n);
                        n = m_isBZip2.Read(buffer, 0, buffer.Length);
                    }

     

    My next question is: would there be a compression ration hit if I was to write short strings of about 100 bytes at a time into the input stream versus writing larger chunks, of say, 1 MB?

  • 05-20-2016 11:07 PM In reply to

    Re: Looking for a way to use a compressed memory stream for writing and reading

    Hi,

    I don't believe there would be any compression ratio hit from small chunks. While I haven't checked BIzip2 specifically, these compressors have their own dictionary size and tree sizes. It's these sizes whch play their part in compression ratio, and cpu time requirements. In Zip there is the Level (0-9).

    Regards, Dave

Page 1 of 1 (10 items)
Powered by Community Server (Commercial Edition), by Telligent Systems
Don't contact us via this (fleischfalle@alphasierrapapa.com) email address.