SharpDevelop Community

Hi guys,

I'm trying to combine the TextEditor with another open source project, Irony, which allows you to write parsers in c# and have them compiled. My aim is to provide an editor with syntax highlighting for user defined languages using Irony's lexer.

So I tried skippen #dev's xml based language definition files and had a go at implementing IHighlightingStrategy to take advantage of Irony's lexer. The fact that the interface exists is great and proves that you guys behind #dev know what you are doing.

Unfortunately the abstraction seems to leak. Looking at the DefaultHighlightingStrategy shows a series of huge methods that my little brain can't comprehend. When I try the text isn't rendered properly, or is being overwritten by my keystrokes.

Has anyone done anything similar, or can the original developers shed some light onto what is requires besides implementing the interface?

Thanks for the great work on #dev!

Soren Skovsboll 

 GetRuleSet and GetColor were only used when a xml-based language definition was referencing a ruleset from another highlighting strategy. I've moved those methods to a separate interface now. You can simply always return null for them.

Let's go through the remaining interface members:

Name: the name of the highlighting strategy, used in right-click>File Mode list.
Extensions: array of file extensions for which the highlighting strategy should be used

Properties: a set of string properties which other features can use to get at language-specific properties. Currently only used for the "Comment region" (Ctrl+/) command. Returning an empty dictionary is OK if you don't need to support "Comment region".

GetColorFor: gets the color of an environment element. Just implement as "return new DefaultHighlightingStrategy().GetColorFor(name)" to use the default colors. See constructor of DefautHighlightingStrategy to see the list of colors this method is used for.

MarkTokens: this is the method doing the actual highlighting of the source code. The (IDocument document)-overload gets called when the whole document should be highlighted; the (IDocument document, List<LineSegment> inputLines)-overload gets called when the specified lines where changed and should be re-highlighted. Of course the latter method may re-highlight more lines than specified by "inputLines", e.g. when the start of a multiline comment was entered in one of the changed lines.

How to highlight a line:

The job of the highlighting strategy is to tokenize the line and assign colors to the tokens. (see DefaultHighlightingStrategy.ParseLine).

Create a new List<TextWord>() and add a new TextWord(...) for each word in the line. Reuse the existing TextWord.Space/TextWord.Tab instances for spaces/tabs; unless you want to give the spaces/tabs a background color (then use new TextWord.SpaceTextWord(color)/new TextWord.TabTextWord(color)).

Then assign lineSegment.Words = yourWordList; and the text editor will use the colored tokens the next time the line is drawn. I think you have to request a document repaint:
            document.RequestUpdate(new TextAreaUpdate(TextAreaUpdateType.WholeTextArea));
            document.CommitUpdate();

Thanks for helping out, Daniel!

There's still some trouble. To troubleshoot I implementet the simplest possible class and cut away the places I manipulate the TextWords.

If I implement the two MarkTokens like this:

public void MarkTokens(IDocument document, List<LineSegment> lines)
{
 document.RequestUpdate(new TextAreaUpdate(TextAreaUpdateType.WholeTextArea));
 document.CommitUpdate();
}

public void MarkTokens(IDocument document)
{
 MarkTokens(document, document.LineSegmentCollection);
}

I would expect my text to be black but otherwise act normally.

However the editor starts to behave strange: I can only overwrite the existing text, not add new chars. And the caret moved unpredictably. Is there anything I missed?

Soren

So, Soren, have you made any new headway on this?  

I too would be very interested since I'm quite enamored of Irony myself lately, and I happen to be in need of creating a syntax highlighter for a new language myself.  It seems like there could be a wrapper of some kind we could cook up to hook up to adapt an Irony parser to be used in a highlighting strategy, but I, of course, am just stumbling on to this post now, so I'll need to look at things a bit more to see how well that works or not.  It might be that a parser for highlight is like a jet engine on a paper airplane, and perhaps we need to expose or chop the irony system up a little to just get at the lexer bits fot this purpose of tokenizing and picking colors.

It is possible to use Irony's lexer only and just get a list of tokens. But I want to have error highlighting as well in which case parsing is required. Irony is fast, and if Resharper can do a full parse+semantic analysis in the background and still perform well, it should be possible here too.

Daniel Grunwald helped me out by providing an example of how to build a custom IHighlightingStrategy, and it seems to be possible, I just haven't had the time to finnish my IronyHightlightingStrategy. If you want to race me, Daniels example can be found below ;)

 /Søren

// SharpDevelop samples
// Copyright (c) 2006, AlphaSierraPapa
// All rights reserved.
//
// Redistribution and use in source and binary forms, with or without modification, are
// permitted provided that the following conditions are met:
//
// - Redistributions of source code must retain the above copyright notice, this list
//   of conditions and the following disclaimer.
//
// - Redistributions in binary form must reproduce the above copyright notice, this list
//   of conditions and the following disclaimer in the documentation and/or other materials
//   provided with the distribution.
//
// - Neither the name of the SharpDevelop team nor the names of its contributors may be used to
//   endorse or promote products derived from this software without specific prior written
//   permission.
//
// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS &AS IS& AND ANY EXPRESS
// OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY
// AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
// CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
// DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER
// IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
// OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

using ICSharpCode.TextEditor;
using System;
using System.Collections.Generic;
using System.Drawing;
using System.Windows.Forms;
using ICSharpCode.TextEditor.Document

;

namespace SharpPad
{
       public partial class SharpPad
       {
               const string SharpPadFileFilter = "Text Documents (*.txt)|*.txt|All Files (*.*)|*.*";

               [STAThread]
               public static void Main(string[ args)
               {
                       Application.EnableVisualStyles();
                       Application.SetCompatibleTextRenderingDefault(false);
                       Application.Run(new SharpPad());
               }

               public SharpPad()
               {
                       //
                       // The InitializeComponent() call is required for Windows Forms designer support.
                       //
                       InitializeComponent();

                       textEditorControl.Document.HighlightingStrategy = new MyHighlightingStrategy();
               }

               void ExitToolStripMenuItemClick(object sender, EventArgs e)
               {
                       Close();
               }

               void OpenToolStripMenuItemClick(object sender, EventArgs e)
               {
                       using (OpenFileDialog dialog = new OpenFileDialog()) {
                               dialog.Filter = SharpPadFileFilter;
                               dialog.FilterIndex = 0;
                               if (DialogResult.OK == dialog.ShowDialog()) {
                                       textEditorControl.LoadFile(dialog.FileName);
                               }
                       }
               }

               void SaveAsToolStripMenuItemClick(object sender, System.EventArgs e)
               {
                       SaveAs();
               }

               void SaveAs()
               {
                       using (SaveFileDialog dialog = new SaveFileDialog()) {
                               dialog.Filter = SharpPadFileFilter;
                               dialog.FilterIndex = 0;
                               if (DialogResult.OK == dialog.ShowDialog()) {
                                       textEditorControl.SaveFile(dialog.FileName);
                                       textEditorControl.FileName = dialog.FileName;
                               }
                       }
               }

               void SaveToolStripMenuItemClick(object sender, System.EventArgs e)
               {
                       if (textEditorControl.FileName != null) {
                               textEditorControl.SaveFile(textEditorControl.FileName);
                       } else {
                               SaveAs();
                       }
               }
       }

       /// <summary>
       /// A highlighting strategy that colors odd digits red and everything else green.
       /// </summary>
       class MyHighlightingStrategy : IHighlightingStrategy
       {
               public string Name {
                       get {
                               return "My";
                       }
               }

               public string[ Extensions {
                       get {
                               return new string[ { ".txt" };
                       }
               }

               Dictionary<string, string> properties = new Dictionary<string, string>();
               DefaultHighlightingStrategy defaultHighlightingStrategy = new DefaultHighlightingStrategy();

               public Dictionary<string, string> Properties {
                       get { return properties; }
               }

               public HighlightColor GetColorFor(string name)
               {
                       return defaultHighlightingStrategy.GetColorFor(name);
               }

               public void MarkTokens(IDocument document, List<LineSegment> lines)
               {
                       for (int i = 0; i < lines.Count; i++) {
                               LineSegment line = lines[i];
                               line.Words = new List<TextWord>();

                               ColorLineRedGreen(document, line);

                               // and tell the text editor to redraw the changed line
                               document.RequestUpdate(new TextAreaUpdate(TextAreaUpdateType.SingleLine, line.LineNumber));
                       }
               }

               void ColorLineRedGreen(IDocument document, LineSegment line)
               {
                       int lineOffset = line.Offset;
                       int wordStart = 0;
                       bool wordIsOdd = false;
                       // Tokenize the line into words
                       // Word borders are around spaces and tabs, and where odd digits touch non-odd digits
                       for (int i = 0; i < line.Length; i++) {
                               char c = document.GetCharAt(lineOffset + i);
                               if (c == ' ' || c == '\t') {
                                       AddWord(document, line, wordStart, i, wordIsOdd);
                                       line.Words.Add(c == ' ' ? TextWord.Space : TextWord.Tab);
                                       wordStart = i + 1;
                               } else {
                                       bool isOdd = c == '1' || c == '3' || c == '5' || c == '7' || c == '9';
                                       if (isOdd != wordIsOdd) {
                                               AddWord(document, line, wordStart, i, wordIsOdd);

                                               wordStart = i;
                                               wordIsOdd = isOdd;
                                       }
                               }
                       }
                       AddWord(document, line, wordStart, line.Length, wordIsOdd);
               }

               void AddWord(IDocument document, LineSegment line, int startOffset, int endOffset, bool isOdd)
               {
                       if (startOffset == endOffset)
                               return;
                       HighlightColor color = new HighlightColor(isOdd ? Color.Red : Color.DarkGreen, false, false);
                       line.Words.Add(new TextWord(document, line, startOffset, endOffset - startOffset, color, false));
               }

               public void MarkTokens(IDocument document)
               {
                       MarkTokens(document, new List<LineSegment>(document.LineSegmentCollection));
               }
       }
}

cool. I agree that it's nice to to semantic analysis as well, but I dont' think this needs to be done at the time text is highlighted.  You can have another thread go back and asynchornously do that... At least so I would think... 

You have a valid point concerning performance. If you want to do lexing only, it's Compiler.Scanner.BeginScan();

On the other hand, if one thread is lexing and the other is parsing, the parsing thread has to do lexing as well before it can parse. I don't know yet if doing lexing twice instead of once will pose another performance problem or not. Guess we'll have to measure.

/Søren

 I was curious what came of this attempt.  I am looking at doing the exact same thing.

shortdog

I got it to a point where it was mostly functional but had a few issues left. Multi-line strings and such.

If you give it a go, be prepared to tackle one thing: IHighlightingStrategy colors per line, whereas, for instance, a multi-line string is a token that spans several lines.

A challenge but definitely possible. I just met a busy period at work and let the project hang.

/Søren

Wow I forgot by the time I got down here that this was FOR irony. sorry, brain puked on me.  Anyway, the package now has a text highlighting tool in it. Get the latest one. Wow. sorry I lost my brain for a while there.

--dave

I used the included highlighter and it was working pretty good.  The current relased version had a bug where if the there wasn't a blank line, the hightlighting didn't work.  I think it lost the EOF marker or something.

Anyway I was wanting to use some of the built in features of the Text Control here.  Like code folding, bookmarks, etc.  I have thought about trying to use #D as it is and create one or more add-ins, that would allow for the functionality I need, but the end result is not a .Net language.  Instead I am looking to generate code (ANSI C in this case) that will compile into an ERP system.

I am digging into the code for both this text editor control and Irony and would like to marry the two up via a derived component.  Instead of using the XML fileds to describe the language, I would like to use the IHightlghingStrategy interface to pull the information for hightlighting out of the ParseTree or AST from Irony.  If I can get it seamelssly integrated, I would like to submit the changes for possible inclustion here, or fork it off into its own opensource project.  At a minimum I would think a codeproject.com article would be in order.

That was my exact same intention.

If you sent your e-mail address to soren at skovsboll dot com, I can send you my unfinished version of IronyHighlightingStrategy to get you started, if you like?