A nice fake quote CSS “trick” for public WordPress blogs

I have some serious topics I’d like to write about, but they take effort, and I don’t have time for putting effort into blog-writing at the moment. But here’s one little thing that I can share; something that has annoyed me for some time…

I hate the default style for quoted text, at least on my blogs. Both of them (and I’m publishing this to both) use the same theme, which doesn’t give me access to the style sheet, and uses an ugly (but readable) font that is a little too large for my liking. I’ve learned to live with that, but what I really can’t stand is the style that’s used for quotes.

The reasons I hate the default quote style is that the quoted text is ugly, and it takes up too much space. It removes the reader’s focus from my words to the quoted text, but I want the quote to supplement what I write, not grab all the attention.

To work around it, I use my own “fake quotes”. It’s hardly a CSS trick at all; it just uses inline CSS on paragraphs, which sets the font, the font size, the line height, pads the paragraphs left and right, and justifies the text. I used this technique in this post last night.

Here’s a dummy HTML page that demonstrates how it’s done. (Note the inline CSS on the paragraph.)

<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
    <title>Fake Quote Example</title>
</head>
<body>
    <p style="font-size: 11px; font-family: 'Segoe UI'; padding-left: 16px; line-height: 16px; padding-right: 16px" align="justify">
        Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque 
        penatibus et magnis dis parturient montes, nascetur ridiculus mus. Donec quam felis, ultricies nec, pellentesque eu, pretium
        quis, sem. Nulla consequat massa quis enim. Donec pede justo, fringilla vel, aliquet nec, vulputate eget, arcu. In enim justo, 
        rhoncus ut, imperdiet a, venenatis vitae, justo. Nullam dictum felis eu pede mollis pretium. Integer tincidunt. Cras dapibus. 
        Vivamus elementum semper nisi. Aenean vulputate eleifend tellus. Aenean leo ligula, porttitor eu, consequat vitae, eleifend ac, 
        enim. Aliquam lorem ante, dapibus in, viverra quis, feugiat a, tellus. Phasellus viverra nulla ut metus varius laoreet. Quisque 
        rutrum. Aenean imperdiet. Etiam ultricies nisi vel augue. Curabitur ullamcorper ultricies nisi. Nam eget dui. Etiam rhoncus. 
        Maecenas tempus, tellus eget condimentum rhoncus, sem quam semper libero, sit amet adipiscing sem neque sed ipsum. Nam quam nunc, 
        blandit vel, luctus pulvinar, hendrerit id, lorem. Maecenas nec odio et ante tincidunt tempus. Donec vitae sapien ut libero venenatis 
        faucibus. Nullam quis ante. Etiam sit amet orci eget eros faucibus tincidunt. Duis leo. Sed fringilla mauris sit amet nibh. Donec 
        sodales sagittis magna. Sed consequat, leo eget bibendum sodales, augue velit cursus nunc,</p>
</body>
</html>

 

And here’s how it looks on this blog:

Lorem ipsum dolor sit amet, consectetuer adipiscing elit. Aenean commodo ligula eget dolor. Aenean massa. Cum sociis natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Donec quam felis, ultricies nec, pellentesque eu, pretium quis, sem. Nulla consequat massa quis enim. Donec pede justo, fringilla vel, aliquet nec, vulputate eget, arcu. In enim justo, rhoncus ut, imperdiet a, venenatis vitae, justo. Nullam dictum felis eu pede mollis pretium. Integer tincidunt. Cras dapibus. Vivamus elementum semper nisi. Aenean vulputate eleifend tellus. Aenean leo ligula, porttitor eu, consequat vitae, eleifend ac, enim. Aliquam lorem ante, dapibus in, viverra quis, feugiat a, tellus. Phasellus viverra nulla ut metus varius laoreet. Quisque rutrum. Aenean imperdiet. Etiam ultricies nisi vel augue. Curabitur ullamcorper ultricies nisi. Nam eget dui. Etiam rhoncus. Maecenas tempus, tellus eget condimentum rhoncus, sem quam semper libero, sit amet adipiscing sem neque sed ipsum. Nam quam nunc, blandit vel, luctus pulvinar, hendrerit id, lorem. Maecenas nec odio et ante tincidunt tempus. Donec vitae sapien ut libero venenatis faucibus. Nullam quis ante. Etiam sit amet orci eget eros faucibus tincidunt. Duis leo. Sed fringilla mauris sit amet nibh. Donec sodales sagittis magna. Sed consequat, leo eget bibendum sodales, augue velit cursus nunc,

Posted in Design, General | Tagged , | Leave a comment

Quack quack: A somewhat malicious program I wrote many years ago

I was reading The Daily WTF earlier, and came across this image, which reminded me of something…

Pic-8_cropped

I think it’s an amusing story, at least it is to me, so I tracked down a program I wrote many years ago, to share it now, but first, here’s the story…

It was many years ago, around 2001, I think, and I had just started working at a place called Old Mutual, the Unit Trusts department in Cape Town. I was young, innocent and idealistic, with perhaps a teeny weeny mean streak. Where I’d worked before, it was perfectly safe to get up from your desk, whether you were going for coffee, or lunch, or whatever, and leave your workstation unlocked. It was perfectly safe, because we all trusted each other.

But poor little innocent me did not suspect what might happen at the new job. I left my workstation open for just ten minutes, and upon my return discovered that somebody had taken it upon himself to teach me a lesson. You see, these people took security seriously, so when I was away, Mr. Gundi Joubert (and I’m writing his name because I will share this with him on Facebook) decided to do me a favour, and show just how harmful it could be to leave your workstation open.

He’d changed my settings on my Windows NT4 machine to use the truly hideous high contrast scheme, changed my fonts to the largest that was allowed, and set my screen resolution to 640×480. I was pissed off. Unfortunately for Gundi, he didn’t realize how unsecure his NT4 machine really was, but I did.

You see, I knew that all I needed to do was copy a program into his local user startup folder via the default C$ open share on his machine. What the malicious program did was limited only by my imagination (of a consequence that was not truly harmful but very annoying) and by the time and energy I was willing to put into my revenge. But I think my achievement was pretty damn annoying for only five minutes of work.

What the quack quack (Delphi) program does is fairly simple:

  1. It runs in a hidden window, which then assigns a variable, appropriately named havoc, to a TThread-derived class. (Delphi’s way of wrapping Win32 API threads in a strong type.)
  2. The TWindowTextReplacer thread then runs in an infinite loop, between sleeping for roughly one second.
  3. It works by enumerating all windows, then enumerating all the child windows of every window, sending them all a WM_SETTEXT message, with the value “Quack Quack”.
  4. The end result: The text of every text control and every button in the system, as well as any other window control that might accept the message (which is most of them in NT4) is set to Quack Quack.

The poor bastard didn’t know what to do, and wasn’t amused when I told him that all he needed to do was open the Task Manager, go to the Quack Quack tab, select the correct Quack Quack process, press the correct Quack Quack button, and when prompted with the confirmation dialog’s Quack Quack message, press the correct Quack Quack button there too…

I recall that I did later expand that program with a second version that also set the user wallpaper to a tiled 32×32 image of Daffy Duck, but that source code was lost a long time ago. Maybe I should port this to C#, just for fun, but not today.

Please note that I do not condone writing malicious programs, unless they are used against people who really deserve them

Download the Quack Quack source code here.

Here’s the source of the malicious thread… Though I don’t have any way of compiling Delphi code anymore… This is the only source of the program that does anything. The hidden window just creates an instance of the thread class, and sets it on it’s merry way.

unit TextReplacer;

interface

uses
  Classes, Forms;

type
  TWindowTextReplacer = class(TThread)
  private
    { Private declarations }
  protected
    procedure Execute; override;
  public
    constructor Create;
  end;

implementation

uses
  Windows, SysUtils;

{ Remove Messages from uses list }
const
  WM_SETTEXT = $000C;
  WM_SETICON = $0080;

{ Type to use as the user-defined parameter in the enumwindows callback fuctions. }
type
  PHWND = ^HWND;

procedure QuackQuack(Hwnd: THandle);
var
  sTemp: String;
begin
  sTemp := 'Quack Quack';
  SendMessage(Hwnd, WM_SETTEXT, Length(sTemp), Longint(sTemp));
end;

{ Callback function for EnumChildWindows }
function EnumChildProc(Hwnd: THandle; FoundWnd: PHWND): Bool; stdcall;
begin
  Result := True;
  QuackQuack(Hwnd);
end;

{ Callback function for EnumWindows }
function EnumWndProc(Hwnd: THandle; FoundWnd: PHWND): Bool; stdcall;
begin
  Result := True;
  FoundWnd^ := Hwnd;
  QuackQuack(Hwnd);
  SendMessage(Hwnd, WM_SETICON, ICON_SMALL, Application.Icon.Handle);
  EnumChildWindows(Hwnd, @EnumChildProc, Longint(Hwnd));
end;

{ TWindowTextReplacer }

constructor TWindowTextReplacer.Create;
begin
  FreeOnTerminate := True;
  inherited Create(False);
end;

procedure TWindowTextReplacer.Execute;
var
  WindowHandle: THandle;
  i: Byte;
begin
  while not Terminated do
  begin
    EnumWindows(@EnumWndProc, Longint(@WindowHandle));
    // not actually using WindowHandle, but could use it at this point

    // Prevent 100% CPU use and pause for roughly a second.
    // Allow window's preemptive multithreading to give process time to other threads...
    i := 0;
    while (not Terminated) and (i < 100) do
    begin
      Sleep(10);
      Inc(i);
    end;
  end;
end;

end.
Posted in Programming | Tagged , , , | 1 Comment

How to align ASPxMenu items on both left and right-hand side

Download the code: Right-aligned menu.zip

As mentioned recently, we are making extensive use of DevExpress controls in my new job.

Today I needed to align menu items on the DevExpress menu to both sides of the screen. In case you wonder why, I’ll give one example: We use the menu as a toolbar in various places above an ASPxGridView, so it looks like it belongs to the grid. We then have menu items for actionable grid items on the left, and menu items to invoke the field editor and export functionality, which I felt could go on the right, for consistency across all pages. That is, the same two buttons occur on the top right toolbar above every grid, so it becomes a standard part of the application.

I found a solution posted on the component writers’ forum here, but unfortunately that doesn’t work anymore. All I’ve done is update it, using the same concept as before, such that it works with the more recent component and browser versions.

Here’s what it looks like at runtime:

RightAlignedMenu

Implementation

Here is my markup. The way to get this right is to use two menus, side by side, where each uses CSS to float to the left and right respectively. That’s exactly what the original solution did, except that it now gets rendered on two lines. All I did was to put the two menus into two table cells.

<%@ Page Language="C#" AutoEventWireup="true" CodeFile="Default.aspx.cs" Inherits="_Default" %>

<%@ Register Assembly="DevExpress.Web.v13.1, Version=13.1.8.0, Culture=neutral, PublicKeyToken=b88d1754d700e49a"
    Namespace="DevExpress.Web.ASPxMenu" TagPrefix="dx" %>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head runat="server">
    <title></title>
</head>
<body>
    <form id="form1" runat="server">
        <div style="border-color: #8BA0BC; border-style: solid; border-width: 1px; background-image: url('Images/MenuGB.png'); background-repeat: repeat-x;">
            <table style="width: 100%; border-collapse: collapse;">
                <tr>
                    <td style="float: left;">
                        <dx:ASPxMenu ID="devxMainMenuLeft" runat="server" EnableClientSideAPI="True" ClientInstanceName="mainMenu"
                            ShowAsToolbar="True" RenderMode="Lightweight" Border-BorderStyle="None" Theme="Office2010Blue">
                            <Items>
                                <dx:MenuItem Name="ShowTotalsMenuItem" Text="test1" ToolTip="Show Totals." BeginGroup="true"
                                    Checked="false">
                                </dx:MenuItem>
                                <dx:MenuItem Name="ShowGraphMenuItem" Text="test2" ToolTip="Hide Graph." BeginGroup="true"
                                    Checked="true">
                                </dx:MenuItem>
                                <dx:MenuItem Name="ShowLabelsMenuItem" Text="test3" ToolTip="Show Graph Labels."
                                    Checked="false">
                                </dx:MenuItem>
                                <dx:MenuItem Name="FiscalCalendarMenuItem" Text="test4" ToolTip="Show Fiscal Dimension.">
                                </dx:MenuItem>
                                <dx:MenuItem Name="SaveToExcelMenuItem" Text="test5" ToolTip="Save To Excel" BeginGroup="true"
                                    ClientEnabled="true">
                                </dx:MenuItem>
                                <dx:MenuItem Name="SaveToTextMenuItem" Text="test6" ToolTip="Save To Text" ClientEnabled="true">
                                </dx:MenuItem>
                            </Items>
                            <Border BorderStyle="None" />
                        </dx:ASPxMenu>
                    </td>
                    <td style="float: right">
                        <dx:ASPxMenu ID="devxMainMenuRight" runat="server" EnableClientSideAPI="True" ClientInstanceName="mainMenu"
                            ShowAsToolbar="True" HorizontalAlign="Right" RenderMode="Lightweight" Border-BorderStyle="None" Theme="Office2010Blue">
                            <Items>
                                <dx:MenuItem BeginGroup="True" Name="ResetLayout" Text="test7">
                                </dx:MenuItem>
                                <dx:MenuItem Name="Info" Text="Info">
                                </dx:MenuItem>
                            </Items>
                            <Border BorderStyle="None" />
                        </dx:ASPxMenu>
                    </td>
                </tr>
            </table>
            <div style="clear: both">
            </div>
        </div>
    </form>
</body>
</html>

Note that I had to use some minor trickery to make it look decent… And now it is no longer properly themed. I couldn’t rely on the menus’ borders, since they are rendered on the left and right respectively, but not in the middle. Thus I turned off the menus’ borders, and used the enclosing div to set the border instead. I also used a one-pixel wide image as the div background colour so that it matches the menus’ gradient background.

Posted in Programming | Tagged | Leave a comment

An easy way to print a web modal popup window

In my new job, we are using DevExpress controls quite extensively, and that involves using an ASPxPopupControl. I need to print the contents of a modal popup, and I naively thought… How difficult can it be?

It turns out to be quite difficult after all, or so it would seem. The problem is, you can only print the contents of a window, and a popup these days is not a real window, thanks to years of bad developers abusing popup windows. Printing it will also print everything else on the page, which is normally hidden behind the fake dialog.

The solutions I found online, being this one that doesn’t work because it is incomplete, and this one on the component writer’s support site, use an outright, nasty, heavy-handed hack… placing an iframe in a separate page, populating it with the popup’s content set to that page, and then printing the iframe/page. Oh, nooooo…

It always amazes me that people just blindly accept such hacks when there is surely a better way… I took a few minutes to look for one, and found a clue here on a much easier, cleaner way. Ever heard of CSS media? (I hadn’t, as I’ve only ever used CSS to effect elements on the screen.) This can be set to screen or print.

That is, you can add a style section like this to the head section of your page:

    <style type="text/css" media="print">
    #lb1{
    display:none;
    }
    </style>

Then, put everything else on the page body into a div with an ID of lb1. Presumably you only show the popup when you need to, so we don’t have to mess with it’s media type at all. Then your print button, which runs on the client side, can just call the JavaScript window.print().

And that’s it! Easy, and this will work no matter what kind of popup window you use, so it isn’t tied to DevExpress.

Posted in Programming | Tagged , , , , | 6 Comments

Just to pass time… One way to approach a simple C# programming assessment

I posted this on my other blog earlier, so excuse the first couple of paragraphs that have nothing to do with this blog…

Why must it be so heart-breaking to drop a child at crèche? She’s happy and chatters all the way in the car, but as soon as she has to part from my arms, you’d swear her little world is over! Otherwise she is a happy little devil… she’ll be one year old on May 20th, and yesterday she walked halfway across the lounge, though she is still unsteady on her feet and prefers to crawl. She’s also learned to say her name, sort-of, which came out first as “Udder Udder Udder”, but after a bit of reinforcement and encouragement, it now sounds closer to “Aishah Aishah Aishah” and she does know that she is saying her name.

Although I still don’t have feedback from last week’s assessment, I aced yesterday’s one. I hope they give me feedback by this afternoon, and either make an offer or line up an interview. This one was at least more to my liking, as it was a pure programming assessment, consisting of four programming tasks, where each took only a few minutes to complete. Too bad I did it at home, which meant I left the solution open for hours and only got to it now and then… If I’d done this one in their office, they’d have been impressed by my finishing it in less than half an hour.

So just to pass the time, I’ll share my answers to it, in case anyone who reads this gets something similar… (I’ll paraphrase what I remember of the questions, rather than copying and pasting them here.)

Question 1: Create a Customer class, with properties Id (int), Name (string) and RegistrationNo (string). Populate a collection containing 300 000 customers, and then find the customer in the collection with Id equal to 251887.

This is very simple, and any c# programmer who can’t do this is in the wrong line of work. For this question I wrote a simple Customer class, and did the work in a console application. Here’s the customer class:

namespace Question1
{
    public class Customer
    {
        public int Id { get; set; }

        public string Name { get; set; }

        public string RegistrationNo { get; set; }

        public Customer(int id)
        {
            this.Id = id;
            this.Name = string.Format("Customer {0}", id);
            this.RegistrationNo = id.GetHashCode().ToString();
        }
    }
}

And here’s the code for the console application that uses it:

using System;
using System.Collections.Generic;
using System.Linq;

namespace Question1
{
    class Program
    {
        static void Main(string[] args)
        {
            List<Customer> customers = new List<Customer>();

            for (int i = 0; i < 300000; i++)
            {
                customers.Add(new Customer(i + 1));
            }

            var customer = customers.FirstOrDefault(c => c.Id == 251887);
            Console.WriteLine(customer.Id);
            Console.ReadLine();
        }
    }
}

Question 2: Create a class called SampleProcess, which exposes a method called Process. The method must implement a loop which counts from 1 to 10 000, which is consumed by a user interface. The method must provide status to the user interface in each loop iteration. (They could have made this question more interesting by stipulating that the process should be cancelled at any time, but they did not.)

I decided to do this one in Windows Forms, since it is really easy to use the Task-based Asynchronous Pattern (TAP) in that framework, which allows the process to happen in a separate thread seamlessly, such that you can move the form around, minimize it and so on while the task is run. If you use the old way, e.g. a BackgroundWorker, or hand-code a thread, or even use the thread pool, there is a fair amount more work to be done, whereas this way is as easy as just writing sing-threaded code, with a couple of keywords so the compiler does the magic for you. Only problem… I have not met anyone else here in Johannesburg who uses the .Net4.5 async code yet, or even knows what it is.

Since this is Windows Forms, I decided to use an event. That is, the class exposes an event which, if subscribed to by the user interface code, is triggered in each loop iteration, and passes to the event subscriber the status as an integer.

Thus it needs a simple event handler with a single property, being the status integer:

using System;

namespace Question2
{
    public class ProcessEventArgs : EventArgs
    {
        public int Status { get; private set; }

        public ProcessEventArgs(int status)
        {
            this.Status = status;
        }
    }
}

Then it needs the class they asked for, which exposes an event as well as the method asked for, and just raises the event in each loop iteration:

using System;
using System.Threading.Tasks;

namespace Question2
{
    public class SampleProcess
    {
        public event EventHandler<ProcessEventArgs> Status;

        protected virtual void OnStatus(ProcessEventArgs e)
        {
            var status = Status;

            if (status != null)
                status(this, e);
        }

        public async Task Process()
        {
            for (int i = 1; i <= 10000; i++)
            {
                OnStatus(new ProcessEventArgs(i));

                /* Simulate doing something for one second, asynchronously 
                 * because there is no need for the thread to sleep.*/
                await Task.Delay(1000);
            }

            await Task.FromResult(true);
        }
    }
}

Then the form contains a single button and a label. The button click event handler is an async method, which creates the sample process class, assigns an event handler (that just sets the label’s text to the event’s status integer), waits for the “process” asynchronously, then removes the event handler:

using System;
using System.Windows.Forms;

namespace Question2
{
    public partial class Form1 : Form
    {
        public Form1()
        {
            InitializeComponent();
        }

        private async void processButton_Click(object sender, EventArgs e)
        {
            SampleProcess sampleProcess = new SampleProcess();
            sampleProcess.Status += sampleProcess_Status;

            try
            {
                /* I made the Process method async to wait for it asynchronously. 
                 * If it was async void we would remove the event handler immediately 
                 * and only get the first event. */
                await sampleProcess.Process();
            }
            finally
            {
                sampleProcess.Status -= sampleProcess_Status;
            }
        }

        void sampleProcess_Status(object sender, ProcessEventArgs e)
        {
            statusLabel.Text = e.Status.ToString();
        }
    }
}

It looks like this:

Question2

Question 3: We have various types that need to be serialized to JSON. Create a generic base class, called SerializeBase, which exposes the Serialize and Deserialize methods. Then create two derived subclasses, instantiate them, and demonstrate how they each serialize and deserialize a type.

This question was unfortunately somewhat ambiguous… There are standard ways of doing serialization in .Net, and I gather they are not asking about those. I figured they just want to know that the programmer knows what inheritance is. The way they used the word ‘generic’ also made it unclear whether they just want a reusable base class, which is what inheritance is, or whether they actually want to know that I know how to write a class that uses the .Net Generic pattern. Thus I decided to do both.

There’s a mistake in this class below… Well, not really a mistake, but something I should have done better. Can you see it? I’ll write it in the comments.

There is already a JavaScript serializer built into the framework, and all you need to do to use it is add a reference to System.Web.Extensions. Having done that, here’s the base class:

using System.Web.Script.Serialization;

namespace Question3
{
    public class SerializeBase<T>
    {
        public virtual string Serialize(object obj)
        {
            return new JavaScriptSerializer().Serialize(obj);
        }

        public virtual dynamic Deserialize(string json)
        {
            return new JavaScriptSerializer().Deserialize<T>(json);
        }
    }
}

Next, my subclasses need to do the serialization of the type that is called T in the base class, but will be defined as some class type in the two subclasses. Since I already wrote a class called Customer in Question 1, I’ll reuse that one, and just define a second class, which for want of a better name, I shall call SomethingElse. Here are my two classes (Customer is defined again in this namespace, because I copied it in here so that I don’t have to reference the other project, which in my solution is a separate project in the same solution.)

(For the deserialization to work, I also had to add a parameterless constructor to the Customer class.)

namespace Question3
{
    public class Customer
    {
        public int Id { get; set; }

        public string Name { get; set; }

        public string RegistrationNo { get; set; }

        public Customer() { }

        public Customer(int id)
        {
            this.Id = id;
            this.Name = string.Format("Customer {0}", id);
            this.RegistrationNo = id.GetHashCode().ToString();
        }
    }

    public class SomethingElse
    {
        public int Id { get; set; }

        public string Thing { get; set; }
    }
}

Now I need two classes to derive from my simple base class, one to do Customer, and one to do SomethingElse: (And they have of course inherited the mistake I made in the base class, although they do work.)

using System;

namespace Question3
{
    public class CustomerSerialiser : SerializeBase<Customer>
    {
        public override dynamic Deserialize(string json)
        {
            if (json == null)
                throw new ArgumentNullException("json");

            if (json.GetType() != typeof(string))
                throw new ArgumentException("json must be a string.");

            var deserialized = base.Deserialize(json);

            if (deserialized.GetType() != typeof(Customer))
                throw new InvalidCastException("Object is not a customer.");

            return deserialized;
        }

        public override string Serialize(object obj)
        {
            if (obj == null)
                throw new ArgumentNullException("obj");

            if (obj.GetType() != typeof(Customer))
                throw new ArgumentException("Object is not a customer.");

            return base.Serialize(obj);
        }
    }

    public class SomethingElseSerialiser : SerializeBase<SomethingElse>
    {
        public override dynamic Deserialize(string json)
        {
            if (json == null)
                throw new ArgumentNullException("json");

            if (json.GetType() != typeof(string))
                throw new ArgumentException("json must be a string.");

            var deserialized = base.Deserialize(json);

            if (deserialized.GetType() != typeof(SomethingElse))
                throw new InvalidCastException("Object is not a SomethingElse.");

            return deserialized;
        }

        public override string Serialize(object obj)
        {
            if (obj == null)
                throw new ArgumentNullException("obj");

            if (obj.GetType() != typeof(SomethingElse))
                throw new ArgumentException("Object is not a SomethingElse.");

            return base.Serialize(obj);
        }
    }

}

Lastly for this one, I wrote a console app to instantiate the two classes, then demonstrate serializing and deserializing them.

using System;

namespace Question3
{
    class Program
    {
        static void Main(string[] args)
        {
            Customer customer = new Customer(666);
            SomethingElse somethingElse = new SomethingElse { Id = 7, Thing = "day" };

            CustomerSerialiser serializer1 = new CustomerSerialiser();
            var serializedCustomer = serializer1.Serialize(customer);
            Console.WriteLine(serializedCustomer);

            Console.WriteLine();
            SomethingElseSerialiser serializer2 = new SomethingElseSerialiser();
            var serializedSomethingElse = serializer2.Serialize(somethingElse);
            Console.WriteLine(serializedSomethingElse);

            var customer2 = serializer1.Deserialize(serializedCustomer);
            var anotherThing = serializer2.Deserialize(serializedSomethingElse);

            Console.ReadLine();
        }
    }
}

Question 4: Write a function that finds all the multiples of 3 or 5 that are below 1000, and calculates the sum of all the values.

This one is too easy and I don’t see why they made the last one so easy. Normally the last one would be challenging, but not here. Thus I wrote a method that can take any number of integer parameters, using the params keyword, and calculate their sum below a given value, which I also passed as a parameter.

I assume that the question relies on the fact that idiots don’t know that to find if any number is divisible exactly by a value (and is thus a multiple of that value), the modulus of the value will be zero. Since I’m not an idiot, I answered this question without even needing to think about it.

using System;
using System.Collections.Generic;
using System.Linq;

namespace Question4
{
    class Program
    {
        static void Main(string[] args)
        {
            Console.WriteLine(GetSum(1000, 3, 5));
            Console.ReadLine();
        }

        private static int GetSum(int threshHold, params int[] values)
        {
            var list = new List<int>();

            for (int i = 1; i < threshHold; i++)
            {
                foreach (var value in values)
                {
                    if (i % value == 0 && !list.Contains(i))
                        list.Add(i);
                }
            }

            return list.Sum();
        }
    }
}
Posted in Family, Programming | Tagged , | 3 Comments

How to exclude files from a FOR loop in a batch file

The other day, my colleague ran into an interesting problem… he needed to process a bunch of files in a directory using a batch file, and one by one, archive them and send them somewhere by FTP. The files were created programmatically by my code, and he ran into two problems:

  1. A race condition exists, since his batch file is invoked by a scheduler, and it could find a file while my program was still writing to it.
  2. He needed to pass a script file to ftp.exe, and there was no way to pass the actual filename in that script (besides generating the script inside the batch file, which is an horrendous hack).

Solving the second problem is obvious, although it wasn’t apparent to him, so sometimes it really does help if two people look at a problem. The batch file simply needed to move one file at a time, in the FOR loop, to a working directory; then the ftp command could use MPUT. (And send all the files in the directory, when there is only one file. This solves the problem because the script passed to ftp.exe no longer needs to contain the file name.)

Solving the first problem is also simple, but maybe not so obvious, especially considering that searching for it on Stack Overflow leads to several over-engineered solutions, so I’ll illustrate my solution here. The trick is to make the files hidden, and then use a dir command in the loop.

Here’s a simple example batch file that illustrates the solution.

  • Create a directory.
  • In the directory, create three files by right-clicking and selecting New Text Document. (You’ll end up with New Text Document.txt, New Text Document (2).txt, and New Text Document (3).txt.)
  • Now go to the file properties of the first file, New Text Document.txt, and set it to be a hidden file.

If you save the following as a batch file in the same directory and run it, it will print the names of the other files to the console, not the hidden file. This is because the hidden file attribute by definition hides files from the directory listing.

@echo off
for %%f in (dir *.txt) do echo %%f
pause

What you’ll get is this:

dir
New Text Document (2).txt
New Text Document (3).txt
Press any key to continue . . .

That was the solution, and if you got here by searching for how to exclude files from a batch FOR loop, that’s all you need to know. But for interest sake, here is how I created the hidden files in c#.

Since my code has full control of writing the files, I changed it to:

  • Write to a temporary file.
  • Set the hidden file attribute on the temporary file.
  • Move the file to the proper file location. (It will remain hidden.)
  • Set the file attributes normal. (Now we can be certain that the batch file will only find files that we have finished writing.)

Of course I’m leaving out exception handling here, and assume that you handle errors appropriately. Otherwise files could simply be lost if an exception is thrown while writing to a temporary file. My actual code in production handles this appropriately.

using System.IO;

namespace ContrivedExample
{
    class Example
    {
        public void SaveFile(string filename)
        {
            var tempfilename = Path.GetTempFileName();

            using (var writer = new StreamWriter(
                new FileStream(tempfilename, 
                    FileMode.Create, 
                    FileAccess.Write, 
                    FileShare.None)))
            {
                // File writing logic here
            }

            File.SetAttributes(tempfilename, FileAttributes.Hidden);
            File.Move(tempfilename, filename);
            File.SetAttributes(filename, FileAttributes.Normal);
        }
    }
}
Posted in Programming, Work | Tagged | Leave a comment

A quick and easy way to call a Web Service without the generated configuration

I recently had to call a web service without being able to use the section it adds to the application configuration file. The reason for this is that my project is a class library running in the process of a third-party payment system, whose code I neither own nor have access to, and I am not allowed to add anything to their configuration file.

My first thought was to search Stack Overflow, which I did, only to find the accepted solution was several more lines of code than my gut told me it should be. (And my gut demands that it is one line.)

My solution is quite simple:

  • Add a reference to the web service as you normally would.
  • Delete your config file (or the relevant section if you do have some other configuration that you care about).
  • Change your code that calls the generated reference’s constructor, where the reference is a class derived from System.ServiceModel.ClientBase. Use the constructor that takes two parameters: of type System.ServiceModel.Channels.Binding and System.ServiceModel.EndpointAddress (passing it a string with the URL of the endpoint it needs), as I have done below.
  • Call whatever methods your reference exposes. Since your reference is still strongly typed, this is simple.
using DOSequenceNumber.ReceiptNumberServiceReference;
using System.ServiceModel;

namespace DOSequenceNumber
{
    class Example
    {
        private static string endPointAddress = "http://03rnb-i62dev01/ReceiptNumberService/ReceiptNumberService.svc";

        public static void ExampleMethod()
        {
            IReceiptNumberService service = new ReceiptNumberServiceClient(
                new BasicHttpBinding(), new EndpointAddress(endPointAddress));
            //service.DoSomething();
        }
    }
}

The advantage of doing this is that you hardly need change your code, compared to what it would have been if you could deploy the service reference configuration.

Just in case: Before you delete the generated configuration, maybe you should check what’s in it… You already know the URL (which you used when adding the service reference), but it might be worthwhile to make sure that the service you use is being called with basic binding, as mine is. Even if it isn’t, the code should still be just as easy. My impression is that most of the generated configuration is unnecessary.

Posted in Programming | Tagged , | Leave a comment

On unwanted software

Yesterday Scott Hanselman wrote an excellent blog-post on the subject of unwanted software. It’s a pity in a way that his is a development blog, because the people who need to read it are not the ones who will get the message there.

I’ve lost count of the number of times I’ve downloaded and installed useful applications and tools that I actually use, only to find that the installers also install some other crap by default. Developers and other technical users are normally (but not always) savvy enough to customize the installations, generally by unchecking the appropriate check boxes so that the unwanted products don’t install, or by clicking Decline on acceptance agreements for products that they don’t want.

But most people are quite oblivious of this – and so they should be – because installers shouldn’t install crap that we don’t want in the first place. It is a huge issue that this unethical business practice is profitable and considered acceptable in the first place, but all of us (technical users) are aware of it, and are often asked to clean up the junk (browser toolbars, default browser search engines, crapware that runs in the tray, and so on) from other non-technical users’ computers.

So if you haven’t read Scott’s post, please go read it, and be careful when downloading and installing software from now on.

Posted in Programming | Tagged | Leave a comment

Gumby: Brain Specialist

I found this a while back, but can’t seem to stop myself from watching it… and watching it… and watching it again.

 

I’ve even resorted to quoting it, in the context of mocking a tester (who is no longer here) who’s idea of negative testing did not appear to be based on reality. To demonstrate that her testing was inappropriate, my example was a hypothetical dialog between a patient and a doctor:

Patient: Doctor, when I head-butt the wall, my brain hurts!
Doctor: Well, don’t do that then, Mr. Gumby.

OK, so maybe you had to be there, but it was supposed to indicate that your test scenarios that are based on something that never can or will happen, should be removed.

Posted in Work | Tagged , | Leave a comment

How to write an empty XML element with XmlWriter in c#

This is probably trivial to most of us, but sometimes it’s the simple stuff that is so easy to miss.

I have a class that converts an incoming flat file to XML and passes it to a payment system. If a field is missing, I thought I could just pass it null, but that didn’t work, because the system expected an attribute with the form:

<NewAccountNumber xsi:nil=”true“/>

I’m assuming you are familiar with an XmlWriter… Doing this turns out to be quite easy:

using System;
using System.Xml;
using System.Xml.Schema;

namespace Example
{
    public static class Extensions
    {
        public static void WriteEmptyElement(this XmlWriter writer, string name)
        {
            if (name == null)
                throw new ArgumentNullException("name");

            writer.WriteStartElement(name);
            try
            {
                writer.WriteAttributeString(
                    "xsi",
                    "nil",
                    XmlSchema.InstanceNamespace,
                    "true");
            }
            finally { writer.WriteEndElement(); }
        }
    }
}
Posted in Programming | Tagged | Leave a comment

My favourite article by Raymond Chen: Feature-coloured glasses

(You’ll just have to forgive me for my spelling in the title… I speak English, not American.)

The other day, I noticed with great sadness, on Raymond Chen’s blog, that Technet Magazine has been retired, and thus so has his wonderful Windows Confidential column. I will miss it a great deal and hope he gets to write another regular column elsewhere. So here’s a link to my favourite of his columns, Feature-coloured glasses. If you haven’t read it (Heck, even if you have…) I suggest you drop everything and read it right now.

Posted in Programming | Tagged | Leave a comment

The dark side of SCRUM: Two more things not to say to testers

I had a new year’s resolution to be nice to people, but sadly it didn’t last quite as long as I wanted.

Some context: I’ve been working on a debit order system, where files with the banking details and so on for presumed payments are exported (to flat files) and sent to the bank, and files with audit information, unpaid transactions and redirects (update customer account details) are imported (from flat files) back into the payment system.

Although this team has three developers, I wrote all our code (why I alone did it is a little complicated to explain here). I finished it all last year before going on leave… That is, I fixed some bugs related to the end-to-end testing with the bank. This was our first week back, and this sprint ends tomorrow, but our testers still haven’t actually started testing. You’d think they would be upset that the FTP connection to the bank is down, but they neither noticed nor even know what FTP stands for. Instead they suggested I speak to the “configuration team” about issues with the payment system that we plug into.

Of course I took offense to the suggestion that I should do their job for them, when my main problem with them is that they are testers but are not actually testing, and there is no sense of urgency from them regarding doing their actual job. They produce endless documents like test scenarios and test approaches and test cases and God knows what else. (And I’m pretty sure there is no God.) We are all external contractors here, so this whole team can be fired. It happened to another team last year. Yet I get here between 6:30 and 6:45 each morning, and one of the two testers strolled in here at 9:08 this morning (even though we had a meeting scheduled for 9AM).

So I lost it, and suggested: (names changed to protect the guilty)

I think we need additional team members, since clearly Juicy’s role is to generate documentation, and Lucy’s is to shuffle tasks.

Naturally that didn’t go down well, and they suggested I be a tester for a day (which is obviously what I was inviting) and then I followed up with:

Sure, I can be a tester. Just give me a fucking lobotomy!

OK, so I didn’t achieve anything positive, but it felt so good.

But seriously, this is a problem, for which I can’t see a solution. Our testers have settled into a pattern, where each sprint, they plan a whole bunch of tasks around the “duties” they need to perform from a governance point of view. They over-estimate substantially, and their tasks are all about generating all sorts of documents, but their focus is never on actual testing. When they eventually do test, they do not properly understand what they should focus on, but log bugs for stupid trivialities, while the bulk of the code’s functionality remains inadequately tested, or not tested at all.

I really don’t see what to do about this. If I raise an alarm, it’s only going to blow up in my face. It seems to me that testers need to be developers. Though I like these two ladies as people, I have lost almost all confidence in them, and believe a couple of junior developers would be better suited to their responsibilities.

Update and plan of action: Us two remaining developers (our companion resigned last month) had a chat with our Scrum-master this morning, and we have a plan. It involves us getting more involved with the test execution. I almost got my wish… in that the other guy is a junior developer who thinks that, since he joined late and hasn’t written any of the code anyway, can do more of the testing, with me fixing the bugs. I’m not crazy about this idea because we are only going to do it due to the testers’ incompetence, but as long as the core functionality of the code gets properly tested, that’s what counts at the end of the day.

Posted in Programming, Work | Tagged | Leave a comment

How to get the application’s directory in C#

Just a short one today…

I always used to forget how to get the path of the directory that my code is running in, and although I did remember it eventually, I also recall that along the way, I found some wrong, or at least incomplete answers. So here is the code I use…

Firstly, here’s a way that works for console applications. I don’t normally do this anymore, since it doesn’t work for everything, but just to make this complete, here it is anyway.

using System;
using System.IO;

namespace Demo
{
    class Example
    {
        private static void Main()
        {
            var directory = Path.GetDirectoryName(
                Environment.GetCommandLineArgs()[0]);

            // Your code here...
        }
    }
}

Notice that in the Main method, I removed the default arguments. They’re not necessary, and besides, they may behave differently when debugging and at runtime. The Environment.GetCommandLineArgs method fetches your arguments from the system; the first argument being the full path and name of your program.

But that only works for console applications. There’s also a method that only works on Windows Forms: Application.StartupPath. But what we need is a method that works in all cases, and here it is:

using System;
using System.IO;
using System.Reflection;

namespace Demo
{
    class Example
    {
        public Example()
        {
            var directory = new Uri(Path.GetDirectoryName(
                Assembly.GetExecutingAssembly().CodeBase))
                .LocalPath;
        }
    }
}

In order not to have you need to scroll, I wrapped those lines and added the necessary namespaces to the using clauses above. (I don’t normally do that, and this is essentially one line of code.)

This gets the application name using Reflection, and then extracts the directory part. The reason I use a Uri is that it gets the path in HTML-encoded form, and not all other code will understand it, so I just use it to extract the actual local path. This works no matter what kind of program you are writing.

Posted in Programming | Tagged , | Leave a comment

On the importance of logging exceptions

It’s been a while since I wrote a programming blog-post, so here goes…

We developers like to think that our code is perfect, that there are no bugs. But there are always bugs. And even in cases where our code is bug free, what works on my machine may not work on yours. You need to handle exceptions properly, and I’m not going to tell you how to do that. But the bottom line is, handle expected exceptions, that is, things that can and will occasionally go wrong, leaving the application in a state where you can still continue (but will probably crash and burn if you do nothing). For everything else, log the error somehow before gracefully terminating the application.

There are loads of error logging solutions out there, or you can do what I did, which is to write something of your own. I wrote my own simple logger, which was originally only for unhandled exceptions as a catch-all so that fatal exceptions could be logged, and then afterwards extended it so that I have a log viewer built into the application.

My logging takes advantage of extension methods so that, as a last step in each exception handler all I need do is type ex.Log(); It uses a bit of reflection and a bit of hacky HTML manipulation so that the viewer can display a nicely formatted table with the error details, and also supports saving the error to a file, as well as sending the error in an HTML email via SMTP.

I won’t show you all the code here, since it references other bits and pieces of my code (in extension methods or whatever I used. I don’t remember anymore…).

You can download my built RomyView application or the full RomyView source code.

My Log extension method on exceptions simply looks like this:

using System.Runtime.InteropServices.ComTypes;

namespace System
{
    /// <summary>Extension methods on <see cref="System.Exception"/> to log
    /// exceptions using my <see cref="ApplicationExceptionLogger"/>.</summary>
    public static class ExceptionExtensions
    {
        public static void Log(this Exception ex)
        {
            if (ex.HResult != (int)HResult.E_FAIL) // Logging E_FAIL is meaningless...
            {
                var ax = ex as AggregateException;

                if (ax != null)
                {
                    ax.Handle(e =>
                    {
                        Log(e);
                        return true;
                    });
                }
                else
                    LogException(ex);
            }
        }

        private static void LogException(Exception ex)
        {
            if (!(ex is OperationCanceledException) && !(ex is TimeoutException))
            {
                ApplicationExceptionLogger.LogException(ex);
            }
        }
    }
}

As you can see, it references a custom enumerated type that maps HResult errors so that I can easily refer to known COM exceptions and so on (as well as easily convert well-written C++ code in MSDN examples and so on), and ignores E_FAIL errors, since there is no value in logging those. It also logs all the nested exceptions in any AggregateException, and then hands the exception over to my ApplicationExceptionLogger class.

That class then hands the exception over to another extension method in my ErrorNotifier class. I’ll paste that code below, but since it is quite lengthy, I’ll first show you what my log viewer looks like. It logs to a CSV file, where one field in the file is HTML that the code generates. The log viewer then parses the file and displays the HTML each time you select an error. (In the case of error messages having commas, my CSV may not really be valid CSV, but that’s OK because my viewer understands how to parse it.) Like so:

LogViewer

Another thing I can do in my exception handlers is also send email notifications, although by default my application settings required for that to work are not set up correctly (because doing so would hand over my Gmail credentials to the whole world). Each error can then be saved to a separate HTML file, and most importantly for me, it logs enough information to be able to find the cause. In this case it is a stupid contrived error… which is always generated if you run the program and hit Ctrl+E in a debug build.

And here is what  my ErrorNotifier class looks like:

using System;
using System.Collections;
using System.Collections.Generic;
using System.Linq;
using System.Net;
using System.Net.Mail;
using System.Reflection;
using System.Runtime.InteropServices;

namespace Romy.Core
{
    /// <summary>Helper class that generates a nicely formatted HTML log of an Exception. Gets whatever info 
    /// it can via reflection, the stack trace, and the custom Data, if any, for the Eception.</summary>
    public static class ErrorNotifier
    {
        #region Fields

        private const string logDateFormat = "yyyy/MM/dd HH:mm:ss:ff";

        #endregion Fields

        #region Methods

        #region Public Methods

        /// <summary>Geneates HTML with as much info about the Exception as possible, 
        /// plus stack trace, which is used for logging and notfication emails.</summary>
        public static string DetailsAsHtml(this Exception ex)
        {
            var html = new List<string>();

            html.AddRange(new string[]{
                "<!DOCTYPE html><html lang=\"en\" xmlns=\"http://www.w3.org/1999/xhtml\"><head><meta charset=\"utf-8\" /><title>Error Notification</title>",
                "</head><body style=\"background-color: #E2E8EB\"><div style=\"width: 100%; text-align: left;\"><table cellpadding=\"0\" cellspacing=\"0\" ",
                "style=\"border: 1px solid #CACBD9; width: 100%; border-collapse: collapse; background-color: #F0F1EF;\">",
                string.Empty,
                "</table></div></body></html>"});

            var errorRows = new List<string>();

            // Keeps track of alternate lines to be displayed shaded in generated HTML.
            var shaded = true;

            foreach (var kvp in ExceptionDetails(ex))
            {
                errorRows.Add(string.Format("<tr{0}><td style=\"width: 120px; font-weight: bold; font-family: Verdana, Geneva, Tahoma, sans-serif; font-size: 12px; text-indent: 8px; color: #272130;\">{1}</td><td style=\"font-family: Verdana, Geneva, Tahoma, sans-serif; font-size: 11px; color: #0B2022\">{2}</td></tr>", (shaded = !shaded) ? " style=\"background-color: #CCCCCC\"" : string.Empty, kvp.Key, kvp.Value));
            }
            ExtractExceptionData(ex, errorRows, ref shaded);

            ExtractInnerExceptionDetails(ex, errorRows, ref shaded);

            html.InsertRange(html.IndexOf(string.Empty), errorRows);
            html[html.Count - 1] += "\r\n";

            // I use this in CSV, so it's important to restrict the HTML to a single line.
            return string.Join(string.Empty, html.Select(s => s)).Replace("\r\n", string.Empty);
        }

        /// <summary>Send error notification email via SMTP.</summary>
        /// <remarks>I have nly used this with gmail, and have removed my password from 
        /// the project settings. To use this, change the relevant settings.</remarks>
        public static async void NotifyByEmail(Exception ex, params MailAddress[] recipients)
        {
            try
            {
                var body = DetailsAsHtml(ex);
                var fromAddress = new MailAddress(Properties.Settings.Default.Smtp_From_Address, Properties.Settings.Default.Smtp_From_DisplayName);
                var subject = string.Format("Application error in {0}", Properties.Settings.Default.ApplicationName);

                var smtp = new SmtpClient
                {
                    Host = Properties.Settings.Default.Smtp_Host,
                    Port = Properties.Settings.Default.Smtp_Port,
                    EnableSsl = true,
                    DeliveryMethod = SmtpDeliveryMethod.Network,
                    UseDefaultCredentials = false,
                    Credentials = new NetworkCredential(fromAddress.Address, Properties.Settings.Default.Smtp_Password)
                };

                using (var message = new MailMessage
                {
                    From = fromAddress,
                    Subject = subject,
                    Body = body,
                    IsBodyHtml = true
                })
                {
                    message.To.AddRange(recipients);
                    await smtp.SendMailAsync(message);
                }
            }
            catch (Exception exc) { exc.Log(); }
        }

        #endregion Public Methods

        #region Private Methods

        private static Dictionary<string, string> ExceptionDetails(Exception ex)
        {
            var now = DateTime.Now;
            var result = new Dictionary<string, string>();

            result.Add("Message", ex.Message);
            result.Add("Exception", ex.GetType().ToString());
            result.Add("Time", now.ToString(logDateFormat));

            foreach (var kvp in ExtractMethodInfo(ex))
            {
                result.Add(kvp.Key, kvp.Value);
            }

            var comExc = ex as COMException;
            result.Add("HResult", string.Format("0x{0:X}", comExc != null ? comExc.ErrorCode : ex.HResult));
            result.Add("Stacktrace", ExtractStackTrace(ex));

            return result;
        }

        private static Dictionary<string, string> ExtractMethodInfo(Exception ex)
        {
            var result = new Dictionary<string, string>();

            MethodBase mb = ex.TargetSite;

            if (mb != null)
            {
                result.Add("Member Type", mb.MemberType.ToString());

                if (mb.Module != null)
                    result.Add("Module", mb.Module.ToString());

                if (!string.IsNullOrEmpty(mb.Name))
                    result.Add("Name", mb.Name);

                if (mb.ReflectedType != null)
                    result.Add("Reflected Type", mb.ReflectedType.FullName);
            }
            return result;
        }

        private static string ExtractStackTrace(Exception ex)
        {
            var stackTrace = ex.StackTrace;
            return !string.IsNullOrEmpty(stackTrace) ? System.Web.HttpUtility.HtmlEncode(stackTrace).Replace("\r\n", "<br />").Trim() : string.Empty;
        }

        private static void ExtractExceptionData(Exception ex, List<string> errorRows, ref bool shaded)
        {
            if (ex.Data != null && ex.Data.Count > 0)
            {
                errorRows.Add(string.Format("<tr{0}><td colspan=\"2\" style=\"font-family: Verdana, Geneva, Tahoma, sans-serif; font-size: 12px; font-weight: bold; vertical-align: bottom; text-indent: 32px; color: #0B2022;\">Data</td></tr>", (shaded = !shaded) ? " style=\"background-color: #CCCCCC\"" : string.Empty));

                foreach (DictionaryEntry entry in ex.Data)
                {
                    errorRows.Add(string.Format("<tr{0}><td style=\"width: 120px; font-weight: bold; font-family: Verdana, Geneva, Tahoma, sans-serif; font-size: 12px; text-indent: 8px; color: #272130;\">{1}</td><td style=\"font-family: Verdana, Geneva, Tahoma, sans-serif; font-size: 11px; color: #0B2022\">{2}</td></tr>", (shaded = !shaded) ? " style=\"background-color: #CCCCCC\"" : string.Empty, entry.Key, entry.Value.ToString()));
                }
            }
        }

        private static void ExtractInnerExceptionDetails(Exception ex, List<string> errorRows, ref bool shaded)
        {
            if (ex.InnerException != null)
            {
                errorRows.Add(string.Format("<tr{0}><td colspan=\"2\" style=\"font-family: Verdana, Geneva, Tahoma, sans-serif; font-size: 12px; font-weight: bold; vertical-align: bottom; text-indent: 32px; color: #0B2022;\">Inner Exception</td></tr>", (shaded = !shaded) ? " style=\"background-color: #CCCCCC\"" : string.Empty));

                foreach (var kvp in ExceptionDetails(ex.InnerException))
                {
                    errorRows.Add(string.Format("<tr{0}><td style=\"width: 120px; font-weight: bold; font-family: Verdana, Geneva, Tahoma, sans-serif; font-size: 12px; text-indent: 8px; color: #272130;\">{1}</td><td style=\"font-family: Verdana, Geneva, Tahoma, sans-serif; font-size: 11px; color: #0B2022\">{2}</td></tr>", (shaded = !shaded) ? " style=\"background-color: #CCCCCC\"" : string.Empty, kvp.Key, kvp.Value));
                }

                // Recurse
                ExtractInnerExceptionDetails(ex.InnerException, errorRows, ref shaded);
            }
        }

        #endregion Private Methods

        #endregion Methods
    }
}
Posted in Programming | Tagged , | Leave a comment

Stupid Computer Talents

Raymond Chen wrote about his carrier-screaming talent from his university days… It’s the second time he’s written about it. The first is here. In the original post, he asks his readers to submit their “stupid computer talents” too. The comments are interesting to read, for all the stupid talents, some of which are pretty cool. Like many of the commenters, I too can hear CRT TVs and monitors. Sadly I still have a CRT TV at home. I can also hear fluorescent lights, but usually only their death throes… I recently replaced my kitchen lights and can’t hear the new ones, thankfully. I wonder how common this is. Can you hear CRT monitors and fluorescent lights?

Posted in Work | Leave a comment

Bad Day At The Office

Posted in Work | Tagged , | Leave a comment

Idiot programmers are more common than you think

It’s always amused me that there are so many stupid programmers here in Johannesburg since they made job interviews really easy for me. But I never realized that that, unlike the stupid programmers here who can’t even write code, there is also a special breed of programmers who can write code, but clearly have shit for brains.

Case in point, today Raymond Chen wrote a blog to answer a strange question from a dim-witted developer who wanted to write a program that generates all possible GUIDs. (Actually I couldn’t resist commenting there… scroll down to the bottom for my comment, or search the page for “thinking outside the box”…) Here’s a similar question on Stack Overflow.

What amazes me is that when people get answers telling them that the operation they would like to do will take more years than the current age of the universe, and the mass required to store the data on SSDs would be equal to that of 100 Earth-sized planets, they still persist to ask the same question…. But that’s why I asked to make the program faster… duh.

Posted in Programming | Tagged , | Leave a comment

Stuff that I don’t understand

1. Spam

I don’t get it. What good could it possibly do sending rubbish comments with links to blog comments? Surely nobody, besides the most stupid of idiots, would ever fall for that? Or is that the target market?

No sir, I do not have kidney disease. Thank you for asking. You do however, have brain disease. There’s an easy solution though. Click off the safety on an armed hand-gun, put it to your head, and pull the trigger. God will reward you in heaven, really.

2. Bullshit

Some people took offense in our daily Scrum stand-up yesterday… because after I deliberately took tasks that amounted to less hours than those in a day and was questioned about it, my reply was “Some people are idle throughout the sprint. My tasks are done so I intend to be idle for the rest of the sprint.”

OK, so I was being sarcastic and inappropriate. People, you know I am sarcastic! And sarcasm is never appropriate. (Nor is rudeness.) If it were appropriate, it would no longer be sarcasm. It would be bullshit, like the bullshit that you tell me. Why did I write all the code myself this sprint? Why do you spend so much time testing negative cases that do not touch the actual code?

If I am assigned any bugs for an external service (that moves files into our directory) for failing to move unknown files into an error directory, when these represent files from the bank, and the bank only ever sends fucking flatfiles, I am going to bounce them right back at you with comment “NOT A BUG”.

Why does it take several hours to test something that I coded in 2 minutes?

2. Johannesburg drivers who give way to taxis

I hate taxis, but I do understand them. They stop anywhere to pick up or drop passengers, including in whatever lane they are driving in. They then do whatever they can do get back en route, including driving in empty turn-only lanes, then cutting off everybody when they get to the front and force their way into the normal lane; driving into the lane risking colliding into oncoming traffic if there is a gap, then once again forcing their way into the main lane at the front; crossing intersections (usually also from turning-only lanes) just before the traffic-light changes to green.

I get all that. That’s what they do here. What I both hate and fail to understand is the fucking morons who let them in. It’s not difficult to cut them off! I do it all the time. Why would you let somebody in, when the chances are very high that he will stop right in front of you? Are you fucking brain-dead?

Posted in General | Tagged , | Leave a comment

Shared stuff updated

No new code shared, I’m afraid, but I have continued to update my image that I use for a custom logon background in Windows 7 (which I am still using at work). The feature does not work in Windows 8, but I still like it as a wallpaper. In Windows 7, you can set it to the custom logon background via Tools –> Customize Windows Logon Background on the menu. The program has a few images that I have done, but by far the best is the one shown below. (This is based on the default Windows 7 logon background, with heavily modified colours, and lots of additional painting, in Photoshop, and the .psd file is in the source code download as well.)

You can get this program feature either by extracting and running the built application, or the source code if you have Visual Studio 2012 (or newer).

Download the built RomyView application
Download the RomyView source code

Overcooked

Posted in Programming | Tagged , , , | Leave a comment

You just can’t win

I don’t get it… Normally our burn-down is more of a burn-up… This time, it’s so well and truly on the line it looks almost like it’s not real – like we did this deliberately! I would have thought it doesn’t get any better than this! I am way ahead on the development – way ahead… and I took yesterday off to go to a family funeral.

But still our scrum master is unhappy. I don’t care if it looks tight because our testers estimated 15 hours for tasks that I reckon could be done in 10 minutes. I created around 15 dev tasks, with no fat in, for one hour each, and then still merged and removed some of them. I did way more work than I intended – virtually all the dev is already done (all of it by myself and I am sure there are no bugs) and there are still two weeks left.

In fact I am waiting on the business analysis because I’ve found some discrepancies between the file and the specification, but I did the dev anyway, and will just refactor next week if I have to.

So I wonder what’s wrong. Maybe this is too perfect?

graph

Posted in Programming | Tagged , | 2 Comments

Zoo Tang Bada-Bing Spam Pollen

I hate SPAM. WordPress is getting better, or maybe most of the spammers have given up, but I did get this in my spam-box today:

My pores and skin looked a whole lot better as I misplaced Zoo Tang Bada-Bing Spam Pollen, so I don’t know.

Of course I changed the name of their product, because I do not endorse spammers, but still, what are they trying to say? I lost the product I am feebly trying to advertise, and it turned out I am better off without it.

Idiots.

Posted in General | Tagged | Leave a comment

I am not a people person

It’s true. I love my work – I love writing the best code that I can, but when it comes to some things, like tact, when dealing with other team members, I suck.

Btw, too much work has resulted in no time for any of my personal code, so I haven’t had any to share for a while. I considered sharing my prototype code for reading flat files that come from the bank, as it is written in such a way that you can easily derive subclasses to read data with any types in different file formats, where each class just has one abstract method to implement, using virtual helpers to add the various entities so that classes for different structures are quick and easy to add in a few minutes, but I don’t think it would be ethical to share that code, and in fact it might break a confidentiality agreement with Multichoice. So apologies… there won’t be any new code shared here any time soon.

I promised myself not to write all the functional code myself this sprint, but did a major part of it last week already (by prototyping). So now all the code to read our incoming files is written and unit tested. I really thought the other guy could then do the convert my nice generic classes to XML part, and he is sick today.

Worse yet, he is working on a part of the solution that I don’t believe requires much time at all, so I am once again going to have to write all the “real” code.

To explain… we have two user stories this sprint, and I don’t believe the first one should even be a story:

  1. Receive a file (of a couple different flat file types).
  2. Process the file. (Read it and transform to XML.)

I am working on number 2 of course, but now our testers are getting pedantic about part 1, claiming they can’t write their test cases and so on because they don’t know how we receive the file.

Who cares how we receive the file? I mean, it’s trivial to write a Windows service that either polls or uses a file system watcher, and just passes the file along. Of course this results in me making statements like “You are just creating obstacles” in our daily stand-up meeting.

I hate dealing with people. They are too often stupid, lazy and just get in the way. Or maybe it’s me who’s wrong?

Posted in Programming | Tagged | 3 Comments

I visited Josh this morning

Check out my other blog for photos of my visit with my son this morning. Baby steps towards having him back for good…

Posted in Family | Tagged | Leave a comment

What went wrong

I’m writing up how the planning went wrong in our last sprint, because I find it interesting thinking about how it actually went wrong, and what I can do to prevent the same sort of issues in the next sprint.

First off, here’s what our software is supposed to do:

  • It integrates into a (payment) system developed by a third party.
  • Within the framework as defined by others, the main method in the component receives a parameter, which contains information that can be used to get the file name of a file exported by their system.
  • The file is in their common format (XML) and represents a batch of debit orders to be sent to the bank, which our code must then transform into a flat file that conforms to the bank’s format. (This is an over-simplification but is good enough for this blog.)
  • The third party kindly gave us a (StreamReader-derived) component that reads their file into strongly typed transactions via a GetNextTransaction method.
  • I undertook to do the development of transforming the transactions to the flat file format.
  • I was aware that the file also required writing one or more headers and one or more footers, but I wanted to take a step back and not be solely responsible for all the functional code this time around, so…
  • I tasked our companion (who is no longer with us, pending the result of his performance hearing) with the header.
  • No task was created for the footers. Oops.
  • No task was created for the “contra” lines, which are summary lines. Oops again.

My task planning was done solely from the point of view of the transactions.

The other developer wrote pretty patterns, a class factory to determine what to do when which bank is involved, and so on. He also didn’t notice that a bunch of things were not catered for in the development.

Lester (the departed developer) didn’t really do anything at all. Well… he did do one thing sort of… The reader component given to us gave access to the file header, also as a strong type. Our dim-witted friend saved that header into a local variable, which he then did not use. (i.e. He didn’t even write anything to the output file. He just saved a reference to something we already had into an unnecessary variable, which then immediately went out of scope. Fucking brilliant.)

Maybe if dumb-ass had done something more, he could have noticed that there was also a second header to be considered, and brought it to somebody’s attention. And maybe then we could have realized that there also two footers and the contra lines. But that didn’t happen.

What did happen was, as mentioned last time, I then did all the functional code myself while the rest of the team slept, and then the whole team had to work late into the night to test that code and fix the bugs, but at least we made it, barely.

I take responsibility for this… I mean, I did not read the fucking manual.

My typing of this entry was just interrupted by our sprint retrospective, and I thus raised RTFM as one of the “What we can do better” issues. Everyone mostly agreed on what I wrote here otherwise… except that they didn’t blame me alone for the issues. Anyway, our major stumbling blocks thus far have been mostly our insufficient pre-planning and then also planning itself.

The main user story for the next sprint is more or less the reverse of what we did this last sprint. (Convert a file that comes back from the bank into the XML format and pull into payment system, which then does what it needs to do, like update subscriber account details, and reverse presumed payments made in the event of failed debit orders.) The main difference this way around is that we don’t have a “reader” for the file; this time I am writing it (I started the prototyping already) and will make sure it does a thorough job, leaving out nothing of the specification. Prototyping it and discussing it with my colleague should ensure that the planning is not lacking this time. (And most importantly for me, though it has nothing to do with the point of this post, is that in this relatively simple project, I get to write the code that is more interesting, leaving the simpler task of writing out the transformed file, and the syntactic sugar of creating pretty code patterns, to my colleague.)

Posted in Programming | Tagged , | 2 Comments

Another Sprint Passed–Hooray!; and my impression of pitfalls for developers new to Agile

This is just my impressions, but maybe they might help others who also switch to an Agile methodology. I’ve been working with Scrum for 6 months now, and I’ve also been “lead developer” on this team for the first time in my career, and have found some issues with my approach… This is normal, I understand, but in any case, sharing them might help others not to make the same mistake.

Technically, this was not my first exposure to Scrum… I was exposed to it in 2009, but I was not quite myself then… I was wired on methamphetamines every day and spent most of my time trying to figure out which of the voices I heard were real and which were only in my head… I don’t advise you do this. Consider this lesson one: Crystal meth and agile do not mix.

But getting back to what I actually intended writing about…

Scott Hanselman wrote an interesting article the other day about rockstar programmers… In my mind, rockstar programmers and agile also don’t mix. Before you disagree, listen to the voice of reason… and this one isn’t just in my head.

Looking at it from another angle: You don’t want to be a rockstar programmer, unless you are a glutton for punishment.

Why do I say this? Put it this way; there are just two of us (developers) on this team. The third (who never contributed anything of value anyway, but for one line of code this sprint, which was wrong) has sadly been removed from the team. But nonetheless, I made the same mistake as last sprint: I did all the actual functional code myself, because that way I knew it would get done.

Except this time, there were things not in the specification, and I screwed up a little: I underestimated the scope of the work. I never normally plan anything – my mind leaps straight from business documents and user stories into objects and methods – it’s my gift and sometimes also my curse. Sometimes, when the team is going through a document together, it takes them 45 minutes to understand something that I got in the first 30 seconds. It can drive me nuts, because I feel like we’re moving in slow motion.

This means that in meetings, most of the time, most of the input in terms of how things can be implemented comes from me. I’m new to this – in the past I was always super-quiet in meetings. Anyway, what’s happened over these last few months is the team has come to trust my judgment more and more. But I’m going on intuition, and I am not perfect. Yes, it’s nice that I can just jump into a conversation with people who actually attended the relevant meetings and come up with an expert opinion, and it’s great that the team relied on my judgment and all, but this time they relied too much on my input.

I found myself and my team in the situation where we had two days to go, and more than half the development not done. I worked late that night, then came in early the next morning, and by the time the rest of the team arrived, the working code was deployed. Then we all worked right into the night until 10PM, with me fixing all the bugs I had created in the rush, and of course since I wrote all the functional code, the onus was on me to fix the bugs. We both worked on it, but I am the one who knows the code, so my turnaround time was obviously way better than my companion. And as a result, I had to do 3 days of work in 1 day.

I mentioned earlier, I never plan anything… I wing it on mostly intuition. Don’t do that. That brings me to my only real point in this post:

PLAN your Sprint properly.

Technically it wasn’t all my fault. The rest of the team should have planned properly too. They were lazy and complacent, and that’s not my fault or my responsibility. But still, I feel responsible.

That’s it. Don’t be a rockstar. Rockstar’s fuck up occasionally too. They end up in rehab after losing all their possessions, and they bounce back with their brilliance, and make up for everything they lost. (Hell, I did that literally too and I’m no fucking rockstar.)

While the lessons I learned in the process of losing everything revitalized me and turned my life around, giving me foresight and hindsight I may never have gained otherwise, these lessons are not for everyone. Don’t be a rockstar, and don’t rely on one either.

Posted in Programming | Tagged , | Leave a comment

Fun with c# code – A disabled ProgressBar

I just noticed this post somehow made the top posts list on my recovery blog, so here is an updated version. (Links updated and code repasted; otherwise it is as is from the original.) Incidentally, the code works just fine on Windows 8 as well.

It’s been a while since I wrote a programming article, so I proudly present the latest addition to my RomyView application, an owner-drawn progress bar, that displays a nicely rendered disabled state when it is disabled. An example of a place where such a control could be used is the Firefox download window. Even if a download is paused, on that window, the progress bars continue to show the annoying progress animation, making it difficult to see at a glance what is paused and what is still downloading.

To see the code described here, you can download either the working application, or the full source code. I haven’t made a small stand-alone example, so you will need my latest upload of my RomyView application.

RomyView Release (built application – requires .NET Framework version 4.5)
RomyView (full source code – requires Visual Studio 2012)

Here is a screenshot of a disabled progress bar:

Progress

I use it in all sorts of places in my application; for example, when pausing a video conversion, or when applying image filters to all images in a directory. Actually I haven’t finished making all the methods where it is used “pausable”, which means, in those places that don’t really pause, it will just draw my grey progress bar instead of the “real” one.

I used to blame the Firefox developers for the animation even for paused downloads, assuming they were guilty of some sloppy code, but the reality is, the progress bar common control simply does not make provision for a disabled state. The Windows Forms ProgressBar control is just a managed wrapper around the common control, so how did I do this? Actually it was pretty easy.

Here is my constructor:

        public CustomProgressBar()
        {
            // The drawing code will flicker unless we enable double-buffering.
            SetStyle(ControlStyles.OptimizedDoubleBuffer | ControlStyles.SupportsTransparentBackColor, true);

            /* Make sure the control can't be sized smaller than the
             * Rectangles used in OnPaint when drawing disabled. */
            MinimumSize = new Size(100, 12);

            EnabledChanged += (sender, e) =>
            {
                /* When this ProgressBar is disabled, force it to be ownerdrawn only;
                 * otherwise the default painting paints over my hard work. */
                SetStyle(ControlStyles.AllPaintingInWmPaint | ControlStyles.UserPaint, !Enabled);
                UpdateStyles();
            };
        }

First of all, I made the control double-buffered. It will flicker otherwise. Secondly, I added an optional line of code, just to make sure that nobody could set the control size smaller than it needs to be, because doing so would result in the calculations in the painting handler getting some rather strange results.

Most importantly above, is the EnabledChanged event handler. All that is does is this… It tells the system that when the control is disabled, not to draw anything at all. This is because the wrapped control always draws enabled. If you want it to draw any other way, you have to draw it from scratch.

Eh? Draw it from scratch? That sounds difficult, right? Wrong. What is a progress bar, really? Basically, it’s two rectangles; the outer one is the bar, while the inner one shows the current progress. Since we’re only drawing it disabled, we don’t need no shitty green animation. As long as we draw it in the right place, everybody will be happy.

So here is my painting handler. (OK, so I use an extension method to draw rounded rectangles, but that is not really necessary. It would look dandy with normal rectangles also.)

        /// <summary>Paints a disabled ProgressBar.</summary>
        /// <remarks><para>
        /// Since ProgressBars don't cater for a disabled appearance, this is my idea of a
        /// disabled ProgressBar's look. It uses grey gradients for a "greyed-out" affect,
        /// drawn as close as I can get to the normal ProgressBar bounds.
        /// </para><para>
        /// For ownerdrawing to work, I had to set the UserPaint and AllPaintingInWmPaint
        /// ControlStyles (when Enabled changes to False), therefore as long as the
        /// ProgressBar is disabled, it will draw nothing at all.</para></remarks>
        protected override void OnPaint(PaintEventArgs e)
        {
            if (!Enabled)
            {
                /* It sometimes gets here with zero width clip rectangle, but this control's
                 * clip rectangle is the same as its bounds anyway, so try the bounds instead. */
                if (e.ClipRectangle.Width <= 0 || e.ClipRectangle.Height <= 0 &&
                    this.Size != Size.Empty && this.Size.Width > 0 && this.Size.Height > 0)
                    e.Graphics.SetClip(this.Bounds);

                e.Graphics.InterpolationMode = InterpolationMode.HighQualityBicubic;
                e.Graphics.PixelOffsetMode = PixelOffsetMode.HighQuality;
                e.Graphics.SmoothingMode = SmoothingMode.AntiAlias;

                /* It ends up drawing a rect with height one pixel too small. To overcome this, set
                 * the clip to a slightly inflated rect, and then don't use the whole new height. */
                var clip = e.ClipRectangle;
                clip.Inflate(0, 1); // This will increase the height by 2 pixels. We then use Height - 1.
                e.Graphics.SetClip(clip);

                var borderRect = e.ClipRectangle;
                borderRect.Inflate(-1, 0);
                borderRect = new Rectangle(borderRect.Left, borderRect.Top + 1, borderRect.Width, borderRect.Height - 1);

                if (borderRect.Width > 0 && borderRect.Height > 0)
                {
                    // First draw the rectangle border.
                    using (var borderBrush = new LinearGradientBrush(borderRect, BorderGradientTopColor, BorderGradientBottomColor, LinearGradientMode.Vertical))
                    {
                        using (var pen = new Pen(borderBrush, 1F))
                        {
                            pen.LineJoin = LineJoin.Round;
                            e.Graphics.DrawRoundedRectangle(pen, borderRect, 2);
                        }
                    }

                    /* Fill the background (the bar); then fill the foreground (the progress value).
                     *
                     * In both cases, I do this with two gradient-filled rectangles, which is why I have values for top,
                     * middle and bottom colours. (Middle colour shared by both rectangles.) The top rectangle has the
                     * top corners rounded, and the bottom rectangle has the bottom corners rounded. The corners where
                     * the two rectangles meet are not rounded, so they appear to be one rectangle. */
                    var topBarRect = new Rectangle(borderRect.Left + 1, borderRect.Top, borderRect.Width - 2, 7);
                    var bottomBarRect = new Rectangle(borderRect.Left + 1, borderRect.Top + 7, borderRect.Width - 2, borderRect.Height - 8);

                    if ((borderRect.Width - 2) > 0)
                    {
                        using (LinearGradientBrush topBrush = new LinearGradientBrush(topBarRect, BackGradientTopColor, BackGradientMiddleColor, LinearGradientMode.Vertical),
                            bottomBrush = new LinearGradientBrush(bottomBarRect, BackGradientMiddleColor, BackGradientBottomColor, LinearGradientMode.Vertical))
                        {
                            e.Graphics.FillRoundedRectangle(topBrush, topBarRect, 2, RectangleEdgeFilter.TopLeft | RectangleEdgeFilter.TopRight);
                            e.Graphics.FillRoundedRectangle(bottomBrush, bottomBarRect, 2, RectangleEdgeFilter.BottomRight | RectangleEdgeFilter.BottomLeft);
                        }

                        // Needs more contrast at the bottom.
                        using (var highlightPen = new Pen(ProgressHighlightColor, 1F))
                        {
                            highlightPen.LineJoin = LineJoin.Bevel;
                            e.Graphics.DrawLine(highlightPen, new Point(bottomBarRect.Left, bottomBarRect.Bottom - 1), new Point(bottomBarRect.Right, bottomBarRect.Bottom - 1));
                        }
                    }

                    // This fills the foreground (progress value) as explained above, but not for marquee style.
                    if (Style != ProgressBarStyle.Marquee && Value > 0 && Maximum > 0)
                    {
                        var w = (int)((double)Value / Maximum * (double)(borderRect.Width - 2)) - 1;
                        var h = borderRect.Height - 2;

                        if (w > 0 && h > 0)
                        {
                            var topValueRect = new Rectangle(borderRect.Left, borderRect.Top, w, 7);
                            var bottomValueRect = new Rectangle(borderRect.Left, borderRect.Top + 7, w, h - 6);

                            using (LinearGradientBrush topBrush = new LinearGradientBrush(topValueRect, ProgressGradientTopColor, ProgressGradientMiddleColor, LinearGradientMode.Vertical),
                                bottomBrush = new LinearGradientBrush(bottomValueRect, ProgressGradientMiddleColor, ProgressGradientBottomColor, LinearGradientMode.Vertical))
                            {
                                e.Graphics.FillRoundedRectangle(topBrush, topValueRect, 2, RectangleEdgeFilter.TopLeft | RectangleEdgeFilter.TopRight);
                                e.Graphics.FillRoundedRectangle(bottomBrush, bottomValueRect, 2, RectangleEdgeFilter.BottomRight | RectangleEdgeFilter.BottomLeft);
                            }
                        }
                    }
                }
            }
            else base.OnPaint(e);
        }

And that’s as easy as it is. It wouldn’t be much effort to use this code as a starting point for a progress bar control that always gets ownerdrawn, completely ignoring the built-in one. In fact, writing your own control for pretty-much anything in Windows Forms is quite easy.

The Romy.Controls class library in the source code zip file contains several other controls.

Enjoy.

Posted in Programming | Tagged , | Leave a comment

SPAM is SPAM

Of all the types of spam comments I hate, those actually deliberately posted by actual human beings, as apposed to spam bots, are the ones I hate the most.

That is, posting nonsense links about your very own c# scheduling components in response to my post about a custom TaskScheduler, or similar such posts, are not welcome here.

Btw, that was just intended as an example above, but I’ll explain further anyway… a TaskScheduler is part of the framework’s way of scheduling how it runs asynchronous threads. It’s something you don’t generally need to change, but understanding how it works and when it is invoked, particularly the relationship between a TaskFactory and the scheduler, is something I find useful – in that it provides some insight into how the various parts of the async machinery work with one-another. As such, mine is designed specifically for offloaded tasks running in parallel. You can only really understand how useful my code really is (or IS NOT) once you understand how all the parts of the framework work with one-another… at which point it no longer matters how good or bad my code actually is – the lesson learned in this case is more important than the code involved. But all this has bugger-all to do with a custom scheduling component! An idiot programmer just used the name of the classes involved to hijack the post and post advertising links to his or her own rubbish code. And besides, writing your own scheduler component, which is presumably what the fool was trying to advertise, is not so difficult. Any junior programmer should be able to do so with relative ease.

My code might not be the greatest, but it’s my code, and I have shared it proudly. This is not a place to advertise your code, however clever or stupid it may be. Any such comments will be deleted. That’s fair, isn’t it?

While we’re here, a note on what this blog is, and what it is not

I happen to have a member on my Scrum team whose attitude to programming differs utterly to mine… He downloads code, then refactors it to do what his requirements are. (He often does this without understanding either the code he downloads or his requirements, and as dev lead, it becomes my problem to get him to “rewrite” his code so that it actually works, or just code it myself. I prefer to code it myself in his case, because I know and trust my own code whereas what comes from him is completely random. But as each sprint draws to an end, if there are bugs, and if I wrote his code for him, the inherent learning curve implies that I have to cover more code in terms of possible bug-fixes. So far I’m coping, due to some combination of luck and skill where I suspect the former has greater weight than the latter. As a result, I have no respect or patience for the Lesters of this world.) This blog is not intended for “coders” like Lester, who does not so much refactor code as he molests it.

The code I share is code that is useful to me, but is not necessarily an end to end solution for anything at all. We, as programmers, use the tools at our disposal, including samples that solve parts of the problems we address, to create code deliverables for our specific needs. Thus, when I write an article about importing paradox data to SQL server, don’t expect a solution that is an application doing everything you need. Rather expect a sample of code that arms you technically to be able to do what is required. (It generates a temporary SQL database using an example simple code-generating algorithm; yet somebody complained about the performance of the SQL database.)

In short, my blog has some shared code for programmers like myself, people who can think for themselves, and use the samples to help fill gaps in their code domains. If you are a copy-and-paste monkey like Lester, fuck off – this blog is not for you.

Posted in General | Tagged | Leave a comment

Caching in a desktop application using ObjectCache and LocalDB

Last Wednesday I decided to convert my application’s caching from a file-based cache to a database-based cache implementation, mostly to try out LocalDB, which I had never used before. Unfortunately, due to power outages after that, I could only test it at home on my decent-spec computer on Sunday, at which point I decided that the database-based cache is way too slow.

The application caches thumbnails, and one of my requirements is that when the file browser is used to browse a directory that’s already cached, the thumbnails should be retrieved almost instantly. My file-based cache, which uses binary serialization, does just that… But the database does not. I suppose I should have anticipated that; the round-trip to the database takes a little longer.

Although the database-based caching was too slow for this particular need, it could be used for other applications. From that perspective, I am happy to have tried out LocalDB, because now I know how easy it is to use. For that reason, and also because it took so long to type up the draft last week, I’ve decided to publish this post anyway. What follows is the post as I typed it up last Wednesday, before my decision not to use that caching implementation.

RomyViewWithDBCache.zip

My application has used the System.Runtime.ObjectCache caching pattern for a long time now, but I resisted writing about it, since my implementation used binary serialization to the file system, which didn’t create the most easy-to-read code. In fact, some parts of it were downright ugly, like working around the possibility of any files in the cache having a length greater than the infamous MAX_PATH character limit. (Any workaround of that will always be a hack.) Also the mess that this produced on the hard drive was the ugliest part, and working backwards from the cache files to the files and directories they represented was always a mess. (Update: Ugly but fast – the file-based implementation is unfortunately still the one being used, and the shared download linked in this post is instead a branch that uses the DB cache, which I’m keeping only as a LocalDB code example.)

I have just finished implementing the (thumbnail) caching in my application, still using the System.Runtime.ObjectCache caching pattern, with the cache implemented in a LocalDB SQL server.

Hold on… Maybe I need to take a couple of steps back to put this into context:

  • Why cache thumbnails? My application can view images and play videos. It also has a limited file browser (and displays the file types that it can view plus one or two others), thus it displays thumbnails, and has its own cache, much like popular programs ACDSee or XNView.
  • Why System.Runtime.ObjectCache? ObjectCache is a really useful abstract caching class, added to the .Net Framework way back when version 4 was released. I’ve never seen any articles written about it anywhere and I don’t understand why it isn’t more popular. It defines a familiar pattern you can use for caching (similar to ASP.Net), but leaves you to implement the actual cache store however you choose.
  • Why LocalDB? LocalDB is something I’d never used before. It’s the current “embedded” database you can use in Visual Studio 2012/2013, and is actually a full working SQL server… This gives you all the power of a full SQL server, even for an application that isn’t connected to anything.

What it all comes down to is, now I can reuse some boiler-plate code for opening queries/stored procedures, so my cache implementation is much simpler. Oh, there are still some complex bits in the code, but I will not write much about that apart from mentioning it here:

  • The static method I use to get a file or directory’s thumbnail is also in this class, but I won’t detail how it works because that is outside the scope of the caching and SQL helper classes. (Getting thumbnails gets complicated. I use the Windows ShellItemImageFactory for some thumbnails, but not others. For small or transparent images and icons, I don’t like the shell thumbnail because I don’t want to lose the transparency. Directory thumbnails get returned with a white background, which I have to remove programmatically using a hacky flood-fill-the-background technique.)

To my surprise, the database-based cache store turns out to be slower than the file-based cache store, which was so fast that cache retrieval was almost instant. (Although I have not tested it on a fast PC. I wrote it at work yesterday in my spare time, intending to take it home and fine-tune it on my new PC at home, but thanks to a power failure, I couldn’t even switch on the PC.)

To make up for this, I added a “2nd layer” to the cache, just for items cached in the current application session. For this, I took advantage of the fact that my cache item class was already serializable, and just serialized it in binary to a dictionary.

My thumbnail database is very simple: just a single table and a couple of simple stored procedures for getting, setting and deleting cache items. I will most probably change and improve things; this is just my first implementation using a database, but the reason I am writing this is to share how easy it is to come up with your own personal cache storing implementation based on the abstract ObjectCache class… And also how quick it is… It literally took less than 5 minutes to implement. (Granted my boilerplate code for calling SQL – written such that it doesn’t care what parameters you pass to SQL, and it reads the schema of whatever datasets are returned – was mostly already written.)

I’m not going to past any of the actual cache class implementation code here, because it’s really as easy as:

  • Derive a class from ObjectCache and click the arrow that adds dummy methods for all the abstract methods you need to implement.
  • Add the implementation to get/set the cache items as needed according to whatever your backing store is.

Instead, what follows is some code from my SQL helper class…  Also not rocket science, but I find having a few helper methods like this that I can then reuse with whatever database going forward, is really convenient. I had written most of this helper code already; but added the synchronous methods to it when working on this. All the methods are overloads of a couple that do all the work, which are as follows:

using System;
using System.Collections.Generic;
using System.Data;
using System.Data.SqlClient;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;

namespace Romy.Core.Misc
{
    public static class Example
    {
        private static int commandTimeout = 300;

        private static string connectionString;

        public static int CommandTimeout
        {
            get { return Example.commandTimeout; }
        }

        public static string ConnectionString
        {
            get
            {
                var connectionString = Example.connectionString;
                var upperCaseConnectionString = connectionString.ToUpperInvariant();

                if (!upperCaseConnectionString.Contains("CONNECTION TIMEOUT") &&
                    !upperCaseConnectionString.Contains("CONNECTTIMEOUT"))
                {
                    connectionString += ";Connection Timeout=60;";
                }

                if (!upperCaseConnectionString.Contains("ASYNCHRONOUS PROCESSING=TRUE"))
                    connectionString += ";Asynchronous Processing=true";

                return connectionString;
            }
            set { connectionString = value; }
        }

        public static DataTable OpenQuery(string query, Dictionary<string, object> parameters, CommandType commandType = CommandType.StoredProcedure)
        {
            try
            {
                using (var connection = new SqlConnection(ConnectionString))
                {
                    using (var adapter = new SqlDataAdapter(query, connection))
                    {
                        var table = new DataTable();
                        adapter.SelectCommand.CommandType = commandType;
                        adapter.SelectCommand.CommandTimeout = CommandTimeout;

                        foreach (var kvp in parameters)
                            adapter.SelectCommand.Parameters.AddWithValue(kvp.Key, kvp.Value);

                        if (connection.State != ConnectionState.Open)
                            connection.Open();

                        using (var reader = adapter.SelectCommand.ExecuteReader())
                        {
                            using (var schemaTable = reader.GetSchemaTable())
                            {
                                foreach (DataRow row in schemaTable.Rows)
                                    table.Columns.Add((string)row["ColumnName"], Type.GetType(row["DataType"].ToString()));
                            }

                            var columns = table.Columns.ToArray<DataColumn>();

                            while (reader.Read())
                                table.Rows.Add(columns.Select(c => reader[c.Ordinal]).ToArray());
                        }
                        return table;
                    }
                }
            }
            catch (Exception ex) { ex.Log(); }
            return new DataTable();
        }

        public static int ExecuteQuery(string query, Dictionary<string, object> parameters, CommandType commandType = CommandType.StoredProcedure)
        {
            using (var connection = new SqlConnection(ConnectionString))
            {
                using (var adapter = new SqlDataAdapter(query, connection))
                {
                    adapter.SelectCommand.CommandType = commandType;
                    adapter.SelectCommand.CommandTimeout = CommandTimeout;

                    foreach (var kvp in parameters)
                        adapter.SelectCommand.Parameters.AddWithValue(kvp.Key, kvp.Value);

                    if (connection.State != ConnectionState.Open)
                        connection.Open();

                    return adapter.SelectCommand.ExecuteNonQuery();
                }
            }
        }

        public static Task<int> UpdateSqlAsync(string query, CancellationToken cancellationToken)
        {
            if (cancellationToken.IsCancellationRequested)
                return Task.FromResult(0);

            using (var connection = new SqlConnection(ConnectionString))
            {
                using (var adapter = new SqlDataAdapter("Not used", connection))
                {
                    adapter.UpdateCommand = new SqlCommand(query, connection);
                    adapter.UpdateCommand.CommandType = System.Data.CommandType.Text;
                    adapter.UpdateCommand.CommandTimeout = CommandTimeout;

                    if (connection.State != ConnectionState.Open)
                        connection.Open();

                    return adapter.UpdateCommand.ExecuteNonQueryAsync();
                }
            }
        }

        private static async Task<DataTable> OpenQueryAsync(string query, Dictionary<string, object> parameters, CommandType commandType, CancellationToken cancellationToken)
        {
            using (var connection = new SqlConnection(ConnectionString))
            {
                using (var adapter = new SqlDataAdapter(query, connection))
                {
                    var table = new DataTable();
                    adapter.SelectCommand.CommandType = commandType;
                    adapter.SelectCommand.CommandTimeout = CommandTimeout;

                    foreach (var kvp in parameters)
                        adapter.SelectCommand.Parameters.AddWithValue(kvp.Key, kvp.Value);

                    if (connection.State != ConnectionState.Open)
                        connection.Open();

                    using (var reader = await adapter.SelectCommand.ExecuteReaderAsync(cancellationToken))
                    {
                        using (var schemaTable = reader.GetSchemaTable())
                        {
                            foreach (DataRow row in schemaTable.Rows)
                            {
                                cancellationToken.ThrowIfCancellationRequested();
                                table.Columns.Add((string)row["ColumnName"], Type.GetType(row["DataType"].ToString()));
                            }
                        }

                        var columns = table.Columns.ToArray<DataColumn>();

                        while (await reader.ReadAsync())
                        {
                            cancellationToken.ThrowIfCancellationRequested();
                            table.Rows.Add(columns.Select(c => reader[c.Ordinal]).ToArray());
                        }
                    }
                    return table;
                }
            }
        }

        private static async Task<int> ExecuteQueryAsync(string query, Dictionary<string, object> parameters, CommandType commandType, CancellationToken cancellationToken)
        {
            using (var connection = new SqlConnection(ConnectionString))
            {
                using (var adapter = new SqlDataAdapter(query, connection))
                {
                    var table = new DataTable();
                    adapter.SelectCommand.CommandType = commandType;
                    adapter.SelectCommand.CommandTimeout = CommandTimeout;

                    foreach (var kvp in parameters)
                        adapter.SelectCommand.Parameters.AddWithValue(kvp.Key, kvp.Value);

                    if (connection.State != ConnectionState.Open)
                        connection.Open();

                    return await adapter.SelectCommand.ExecuteNonQueryAsync();
                }
            }
        }
    }
}
Posted in Programming | Tagged , , , | Leave a comment

Updates to my shared code

I’ve made several updates to my shared code, and the files in that share. I don’t think any of them justify an exclusive post for themselves, so I’m devoting this post to all of them (that I can remember) instead.

As always, the zip files are available as public read-only shares on my SkyDrive.
RomyView source code
Romyview Release

For some unknown reason, my Windows Live Writer is doing some weird shit in source view (mouse caret vanishes and typing overwrites some text), so if either or both of those links are broken, let me know.

Changes that I can remember

CreateBackup

I removed the “Zipper” project from the solution. That was a console application that used my async extensions to the .Net Framework 4.5 zip support helper class, because I never used it for both zip and unzip. I replaced it with the almost identical “CreateBackup” project. This one only zips the solution. I invoke it one of two ways; from the IDE… as in Debug, start new instance – to make a backup of the solution (RomyView.zip), or from my build batch file when the solution is closed, to produce both of the files that you can download.

I really should find out how to write a Visual Studio extension instead, but the documentation I downloaded was too vague. (If I don’t get a feeling for how to code something within a few seconds of starting to read the documentation, unless it is a requirement at work that I must implement, I probably don’t do it.) I assume there must be some sort of API that accesses the solution code, but my console app uses the file system.

Windows 7 Custom Logon Background Image

I never did get around to updating the separate application I originally wrote for this – which used my old monitor images in it’s form (image of monitor with 4:3 aspect ratio). I’m still using Windows 7 at work.

The form, which you can open from the Tools menu, looks like this:

GlassForm

The program’s feature to set one of my predefined images (or any other you copy into it’s directory) only works on Windows 7. The form also uses Aero Glass. You can still open it on Windows 8 (and confirm that it looks like shit). It was a fun form to write, and I suppose could be used by a junior programmer who doesn’t know anything about overriding the Paint method, as an example of one approach to custom drawing. i.e. There is a panel with a static background image (my Photoshop-drawn monitors, complete with Romy logo) but when you select any of the ListView items with the mouse (or keyboard since the ListView has focus) it dynamically loads the image and draws it on top of the panel. I experimented with a few ways of drawing it… drawing onto the panel that gets painted automatically vs. drawing everything onto a Bitmap, then drawing that onto the panel – but they made very little difference to performance. I don’t remember which way I left it in the end – but I think it was the one with less code.

And the image after spending (too) many hours in Photoshop is this:

Overcooked

If you are curious how I came up with it… It is the original Windows 7 logon background image, with a colour overlay in blue, set to exclusion (i.e. remove most of the blue from the mostly blue image). On top of that, I added a few layers to further modify the colours and contrast, and manually painted some stuff.

Oops… and remember kiddos… if you create your own Windows 7 background image, make sure the JPG file is smaller than 250kB! Mine wasn’t, and my code accidently checked for 256kB… I’ve just reuploaded it now. (That’s what happens when you work on Windows 8, with a feature that only works on Windows 7.)

Code to check if this is the foreground application

Since I wrote the application mostly for myself (I use it as my default image viewer and movie player), I broke a few rules w.r.t. windows and stealing focus. That is, it annoyed me that when I clicked a toolstrip button on the browser, it would take two clicks if the app was not focused. So I made the window steal focus if the mouse moves over certain buttons. That is kind of rude… (Actually rude enough that I will immediately uninstall any other application that does the same thing.)

This added a requirement to only steal focus from our own windows, but do nothing if the application doesn’t have focus (that is, only act if this is the application the user is working with right now).

Here’s the code:

using System;
using System.Diagnostics;
using System.Runtime.InteropServices;
using System.Security;

namespace Romy.Core
{
    public static class Example
    {
        /// <summary>Indicates whether this application has input focus.</summary>
        /// <returns><b>True</b> if this application owns the foreground window; 
        /// otherwise <b>false</b>.</returns>
        public static bool IsForegroundApplication()
        {
            var foregroundWindowHandle = NativeMethods.GetForegroundWindow();

            if (foregroundWindowHandle == IntPtr.Zero)
                return false;

            int activeProcessId;
            NativeMethods.GetWindowThreadProcessId(foregroundWindowHandle, out activeProcessId);

            return activeProcessId == Process.GetCurrentProcess().Id;
        }
    }

    [SuppressUnmanagedCodeSecurity]
    internal static class NativeMethods
    {
        [DllImport("user32.dll", CharSet = CharSet.Auto, SetLastError = true)]
        public static extern int GetWindowThreadProcessId(IntPtr handle, out int processId);

        [DllImport("user32.dll", CharSet = CharSet.Auto, ExactSpelling = true)]
        public static extern IntPtr GetForegroundWindow();
    }
}

That code actually lives in my base class for enforcing a single instance application (so that I can use it like Romy.UI.Instance.IsForegroundApplication()), which itself breaks a rule resulting in a code analysis warning… something like “Don’t add static methods to generic types, or specify type arguments, you tit!” which I suppressed.

Convenience methods to run async Tasks using the configured custom TaskScheduler

I’ve written a few posts about my custom TaskScheduler, but haven’t made it all that clear how to invoke it. Well, the main way I use it is when I have lots of tasks that need to be offloaded, because running the tasks grouped into batches, in parallel, is very effective – and that’s what my custom TaskSchedulers are designed to do. I added these to my static Tasks class, which is defined in the same source file as my global TaskFactory.

Now calling it is similar to the built-in class… I just call it using Tasks.Run (instead of Task.Run). This is the code:

using System;
using System.Threading;
using System.Threading.Tasks;
using System.Threading.Tasks.Schedulers;

namespace Romy.Core
{
    /// <summary>Set up some common options for async Tasks. The <b>TaskSchedulers.Parallel</b>
    /// scheduler references the static <b>Scheduler</b> property of this class. The value of
    /// <b>Factory.UseThreadPool</b> defines which <B>ParallelTaskSchedulerBase</B> 
    /// implementation is used and hence whether to run ThreadPool threads or dedicated threads.
    /// When using dedicated threads, <b>Factory.UseForeGroundThreads</b> configures whether they 
    /// are foreground or background threads.</summary>
    /// <remarks>For offloading Tasks to the custom TaskScheduler, call <see cref="Romy.Core.Tasks.Run()"/>.</remarks>
    public static class Tasks
    {
        private static ParallelTaskFactory factory = new ParallelTaskFactory();

        public static ParallelTaskFactory Factory
        {
            get { return Tasks.factory; }
        }

        public static TaskScheduler Scheduler
        {
            get { return factory.Scheduler; }
        }

        public static Task Run(Func<Task> body)
        {
            return factory.StartNew(async () => await body()).Unwrap();
        }

        public static Task<T> Run<T>(Func<Task<T>> body)
        {
            return factory.StartNew(async () => await body()).Unwrap();
        }

        public static Task<T> Run<T>(Func<Task<T>> body, CancellationToken cancellationToken)
        {
            return factory.StartNew(async () => await body(), cancellationToken, TaskCreationOptions.PreferFairness, Scheduler).Unwrap();
        }
    }

    /// <summary>A TaskFactory to define which custom TaskScheduler to use.</summary>
    public class ParallelTaskFactory : TaskFactory
    {
        private bool useForeGroundThreads;

        private bool useThreadPool = true;

        private static Lazy<ParallelDedicatedThreadsTaskScheduler> dedicatedThreadsScheduler = new Lazy<ParallelDedicatedThreadsTaskScheduler>(() => new ParallelDedicatedThreadsTaskScheduler());

        private static Lazy<ParallelThreadPoolTaskScheduler> threadPoolScheduler = new Lazy<ParallelThreadPoolTaskScheduler>(() => new ParallelThreadPoolTaskScheduler());

        public ParallelTaskFactory()
            : base(CancellationToken.None,
                TaskCreationOptions.DenyChildAttach | TaskCreationOptions.HideScheduler,
                TaskContinuationOptions.ExecuteSynchronously, ThreadPoolScheduler) { }

        public bool UseForeGroundThreads
        {
            get { return useForeGroundThreads; }
            set
            {
                useForeGroundThreads = value;
                DedicatedThreadsScheduler.UseForegroundThreads = useForeGroundThreads;
            }
        }

        public bool UseThreadPool
        {
            get { return useThreadPool; }
            set { useThreadPool = value; }
        }

        public static ParallelDedicatedThreadsTaskScheduler DedicatedThreadsScheduler
        {
            get { return ParallelTaskFactory.dedicatedThreadsScheduler.Value; }
        }

        public static ParallelThreadPoolTaskScheduler ThreadPoolScheduler
        {
            get { return ParallelTaskFactory.threadPoolScheduler.Value; }
        }

        public new TaskScheduler Scheduler
        {
            get { return useThreadPool ? (TaskScheduler)ThreadPoolScheduler : (TaskScheduler)DedicatedThreadsScheduler; }
        }
    }
}

That’s it for today, unless I remember some more that could be included here. Enjoy!

Posted in Programming | Tagged , , , , | Leave a comment

How to make McAfee SiteAdvisor go away

Update: I just read Mark Russinovich’s blog post entitled Hunting Down and Killing Ransomware. Hmm… I didn’t know that malware was now so aggressive, or that it was fairly prevalent, or that it could possibly use (for example) an IE vulnerability to install. (I have only ever seen installations where the user was stupid enough to be tricked into actively infecting their own system.)

With the above knowledge, I have decided to add this disclaimer: This post is intended for advanced users/developers only. (Typically, they would have figured out this stuff for themselves, so they don’t really need it, but never mind.) McAfee is a reputable company, and it’s software is comprised of tools that are genuine. I just find them annoying and I don’t need them. My own system was infected with malware once, around 9 years ago (I think). I am also intimately familiar with the Autoruns tool written about in the post mentioned above. If you are not like me, that is not a programmer, or not experienced and confidant that your system will not become infected, or that you would easily be able to remove any infections yourself (without breaking a sweat and while holding your breath) , then don’t follow the advice in this post. If your system regularly becomes infected with malware, or if you have read these two paragraphs several times and are still struggling to grasp an understanding of it’s meaning, then I suggest a running jump out of your balcony door, over your railings, and to the sweet relief your demise will bring to everybody who knows you.

A couple of weeks ago, some new software, McAfee SiteAdvisor version 3.5, appeared on my work machine. It was apparently deployed with some sort of corporate policy. It runs inside my browser, and does I know not what, apart from hanging the browser and tripping over it’s virtual shoelaces occasionally.

I Googled to find out how to uninstall this unwanted software, but all I found was some rather useless instructions on how to remove it from the Control Panel. Marvelous, except that requires stopping the service, and of course I do not have permission to stop the service.

Of course there’s a whole bunch of useless software being sold online that promises to uninstall anything and everything forever and again and forevermore, but fortunately I am smart enough to know that none of it will actually achieve anything at all when it will also be unable to stop or remove the service, to which it also has zero permission, so I will not waste any time on that.

Some people might say they choose to remove the software because it is intrusive, or something to that effect. As for me, I don’t like it because it is ugly. I mean, look at this shit. It must be using some sort of hook/hack, and it isn’t even in the right place on the window.

SiteAdvisorAlive

At this point, McCrappy might expect me to give up. “Oh dear me, I don’t have permissions to their shitty service! Oh woe is me!”

However, thinking just a little way outside of the box is enough to recall that this must be running as a plugin in my browser. I browse with Firefox, and it apparently runs each plugin in a separate instance of it’s plugin container process. (I have no idea if that statement is generally true, but no matter… it applies to this process.) This process can safely be terminated and won’t disturb the browser itself, or any of it’s other plugins.

I use Process Explorer out of habit, but I imagine you can do something similar with Task Manager.

Kill

Just select the process, and hit the Delete key. Goodbye ShiteAdvisor. It was not nice knowing you. Ashes to ashes. Dust to dust. May you rest in pieces.

SiteAdvisorDead

Posted in General | Tagged | Leave a comment

Developer inferiority complex

Scott Hanselman just published a post answering an email from a concerned developer who was asking himself the question, “Am I a developer or just a Googler?”. He’s written about this subject before.

Funny, I’ve wondered about the same thing over the years. I started out not knowing what I wanted to do with my life, and eventually studied Electrical Engineering at a technical college just because my brother did that, and I had to study something, so it seemed as good an option as anything else. I never found it interesting though, and it never even dawned on me that my only enjoyment there was when I got my HP calculator, and it became my mission to write the best possible program for it – one that could solve a mother of an equation for any given operands. (I don’t remember the details now.) Also, I bought my calculator on a Friday, and by the Monday that followed, I was explaining to a couple of guys (who’d had their calculators for six months already) how to write better programs.

I forgot all about those days, then ended up working for a company with a software division as well, and somehow slipped into a programming role years later, almost by accident, after convincing somebody that I could code in Delphi, having bought one of the “Mastering Delphi” books and written some (actually very little) code for a whole two days. It was at that time I realized that I can somehow grasp abstract concepts rapidly, and within a year I was a fully-fledged programmer, loving it, competing with people who already had 10 years experience, and supremely confident.

Only after a couple of years did the doubts start to set in. I realized that my peers often had Computer Science degrees, while I had none. I started to doubt myself. I started to question whether I was really good enough, and felt like maybe I’d just been lucky.

I’ve never had the issue of being a copy-and-paster, even at the beginning, despite the fact that I am the most lazy person I know. The fun in programming is in figuring out how to solve whatever problem is at hand, with the tools that you have. I wouldn’t do it otherwise. So I normally first write my own code – then Google it afterwards to confirm that I went down the right road – and I almost always get it right; often writing better code than the SO solutions I find for people asking the same questions – otherwise I learn and adapt my style accordingly.

So after much thought on this subject, I’ve concluded that my own self-doubt is perfectly normal, especially for somebody who used to be awfully shy. And questioning your code is healthy – it forces you to hold yourself to high standards. Thus I’ve decided to share my thoughts on this subject, since they are largely positive. (For a change… I share my negative thoughts as jokes, like the previous post.)

Although I know there are a large number of developers who should be doing something else, especially here in Johannesburg (and I know this mainly because of job interviews where they have had to dumb down their programming assessments due to all the applicants failing, such that I could score 100%), I believe that if you ask yourself that question (Am I really a programmer, or am I just lucky/a good Googler?), you probably are a decent developer – or at the very least you are pushing yourself; you are improving, and you are growing as a developer and in your career.

In Scott’s original post where he describes feeling like he is a phony, he goes on to write about a condition where people who are supremely confident are actually incompetent, and are unaware of their limitations. Actually this reminds me of a great video of John Cleese. (I can’t remember the URL, sorry.) He describes in his brilliant witty style how “truly stupid people will never know they are stupid”.

It’s kind of like the Idols TV show. People show up who can’t even conceive of what it might be like to hold a note (and are tone deaf anyway), thinking they can become popstars.

To summarize, if you question your design and your code, you are probably worth your weight in gold as a developer. If you think you’re the greatest developer ever to walk the Earth, you’re almost certainly a fucking moron.

Posted in Programming | Tagged | 1 Comment

Internet crimes and their suggested punishments

I originally published this post a while back on my other blog.

Having upgraded my home PC, I decided to search for Windows 8 Power Toys. Hmm… a bad idea. It seems that people’s ideas of power toys has changed over the years. These days, it could mean just about anything, but to writers of technical, or so called “power user” blogs, it generally means utility applications that they can’t do without. Often, these useless utilities provide functionality that is already present in Windows, but the power users are too fucking stupid to know that.

Thus it is time to republish my list of internet crimes and their suggested punishments. (I can always hope that a serial killer or psychopath reads them and applies them.)

Note that I have updated my notes for crime number 3. It now applies to all lists of any sort published on the web.

Here follows the original post:

The internet has changed the way we interact with others. It has come to my attention that in this virtual society, a number of serious offenses, which our laws do not yet recognize as crimes,  have been committed and have gone unpunished for far too long. I thusly propose this list of crimes and recommended punishments, which I highly recommend be put into practice immediately.

1. Epic fail

This phrase has been misused for far too long. It was inappropriate to begin with, since epic was never really correct, and then later was adopted by people whose lack of brain cells is surpassed only by their inability to express anything in their own words. Anybody using this phrase in any comments or message anywhere at all shall be publically flogged.

2. My vote of 1 (CodeProject)

Stick it up your bum. And by that I mean a hot poker. Anybody who submits a vote of 1 to any article on CodeProject will be subjected to a hot poker up their anus.

3. The top 10 [anything] of all time (YouTube)/ and any comparative list published anywhere

Anybody who watches any top 10 video shall have their eyes gauged out with a hot poker, after it is removed from the arse of a “1 vote” criminal above.

Submission of a “top 10” video is a heinous crime. There is no need even to try to explain why to any of these fools. As an example I offer any of the “top 10 most terrifying scenes in horror movies” or the like. (But don’t go and watch one! You have been warned.) These people are too stupid to understand that creating horror movies is a craft for experts. Removing a scene from its original context doesn’t just spoil the movie; the scene is no longer scary. Thus the list creates a disjointed list of scary moments that is not scary at all, which can ruin up to ten movies for someone who hasn’t seen them yet.

Note that this crime now (24 August 2013) applies to any comparative list published anywhere on the world wide web, since any such lists are, without exception, bullshit.

For punishment, convicted serial killers will be allowed to choose an assassination method of their liking from any top 10 movies they find on YouTube. (After which their eyes will be gauged out. There can be no exceptions.)

4. <sarcasm>Sarcasm Tags</sarcasm>

Anyone who has used tags to denote sarcasm shall be shipped off to Afghanistan, where they will remain for one month, forced to spend their days outside mosques, chanting “Allahu Akbar motherfucker” as they burn the Quran. When approached, they must defend themselves with “I was just being sarcastic”, because that is the same as sarcasm tags. Anyone who somehow survives the experience will be shot.

And just in case you don’t get the last one, since I was not being sarcastic when I wrote it, I shall explain:

Sarcasm is by definition meant to insult/deride/hurt the person at which it is directed. If the person is female and does not immediately burst into tears, your attempt at sarcasm was weak. You need to try harder.

If you need to enclose your written sarcasm in sarcasm HTML tags, I must regretfully inform you that you are too stupid to be sarcastic. You aimed for sarcasm, but your low intellect and lack of wit resulted in facetiousness instead. (You don’t already know this because other people’s sarcasm and wit usually flies right over your head.)

Posted in General | Tagged , , , | Leave a comment

On Windows 8, insanity, painting and idiots

It occurred to me that this blog’s header promises the occasional descent into madness, but I haven’t done that for a while. So now is as good a time as any, right?

Last weekend I finally bought a new PC. I was sick of my 1.5GB memory and all the time-wasting that it caused. I could make a cup of coffee while waiting for a new tab to load in the browser… 12 GB is so much better.

Only thing is… I had to upgrade to Windows 8, and I’d read various people’s complaints about it being so very different, that they were lost. I’d never even seen it, so I’m more than a little surprised. Um, so Microsoft shuffled a couple of GUI elements around. So what? Using it is almost exactly the same as it used to be if you normally run programs directly or even if you work from Windows Explorer, and they didn’t remove any of the Windows Key shortcuts. Sure, if you spend your days in Windows 7 scrolling around in the start menu, you’ll be a little lost… but if you spend your days scrolling through the start menu, you’re pretty fucked up to begin with.

And if you are a programmer and you don’t find your way around Window 8 in less than 2 minutes, you probably shouldn’t be programming. You should be grazing in a pasture somewhere with all your bovine brothers and sisters.

The only thing I miss about Windows 7 is Aero Glass, which I thought was cool. Of course, there are many idiots out there who are spending a great deal of effort hacking Glass into Windows 8. I thought that was hilarious when I found out. I mean, if you want Glass, just run Windows 7! What the fuck is wrong with you?

Lately I have been playing around with Photoshop – creating a better-looking logon background screen for Windows 7. I guess that ends now. I’d never actually used the painting tools much in Photoshop, and they’re actually kind of cool. In fact, they feel quite similar to painting for real. (Although honestly I find painting with actual oil paint and physical brushes considerably easier.)

Here’s what my latest modification of the Win 7 background looks like:

Overcooked

Anyway, the feature is built into my RomyView application.

Mostly in the wavy line things… I added lots of highlights, by manually painting them. The result is quite similar to what you get when painting for real. I need to get back to that… Anyway, you can see what I mean if you compare it to my actual oil paintings.

That post is on my meth recovery blog. Actually I only ever painted for about 2 years, and only started in my late thirties. I think I should get back to it, but I honestly am not certain if I have the right temperament for it. That is, I know I am lucky enough to have a gift for it, but I don’t know if I have the patience. All my paintings were done under the influence of copious amounts of methamphetamine. And it’s all very well to painstakingly add layer upon layer of oil paint while you are a meth zombie and tweaking yourself to kingdom come… but in a normal state of mind? I don’t know if I can still do it.

Posted in General | Tagged , , , | Leave a comment

My Parallel TaskScheduler revisited

Recall my post A configurable TaskScheduler that can be changed by the user at runtime

Yesterday I used ILSpy (I don’t use Reflector because it is no longer free) to disassemble the built-in ThreadPoolTaskScheduler class, just out of curiosity. Some code almost jumped out at me to smack me in the face! This class, which I presume is the default TaskScheduler, does one thing I forgot all about when it runs a Task. That is it checks if the Task’s TaskCreationOptions includes LongRunning, and if so, runs the respective Task on a dedicated thread.

Since my classes didn’t respect the LongRunning option and treated all tasks equally, some refactoring was in order. Now long running tasks are catered for, in a similar manner to the default scheduler.

You can download the code here.

To reiterate the summary of what this comes down to, backwards, just because I like the arse about face approach, the end result of this as far as the user is concerned is:

TaskSchedulerOptions

Above: RomyView application options dialog

And that means that normally my ParallelThreadPoolTaskScheduler is used, unless the user chooses not to use the ThreadPool, in which case the ParallelDedicatedThreadsTaskScheduler comes into play. Then, the behaviour can be further customized by choosing to use foreground threads. (And wondering why the application sometimes takes very long to exit. Hint: Even the most useless background task now runs as a foreground thread.)

To cater for long running tasks, I added a new method to the underlying interface that the base class implements. (With a signature almost identical to the existing method.)

namespace System.Threading.Tasks.Schedulers
{
    /// <summary>This is the interface implemented by <see cref="ParallelTaskSchedulerBase"/>. It declares <b>ParallelInvoke</b> 
    /// as abstract, forcing derived classes to implement it. <b>LongRunningTasksInvoke</b> is implemented as a virtual method;
    /// by default running long-running tasks on dedicated threads similarly to the default TaskScheduler. This may of course be 
    /// overridden.</summary>
    /// <remarks><b>ParallelDedicatedThreadsTaskScheduler</b> overrides <b>LongRunningTasksInvoke</b> to runs Tasks on dedicated 
    /// threads, setting each as a forground or background according to it's <b>UseForegroundThreads</b> property.</remarks>
    public interface IParallelInvoker
    {
        void ParallelInvoke(Action<Task> action, params Task[] tasks);

        void LongRunningTasksInvoke(Action<Task> action, Task[] tasks);
    }
}

The class diagram now looks like this:

TaskSchedulers

Next since all the methods have almost the same signature, and the implementation comes in two flavours; ThreadPool or Thread – I moved the relevant code into a helper class with a couple of extension methods.

using System.Collections.Generic;
using System.Linq;

namespace System.Threading.Tasks.Schedulers
{
    internal static class TasksExtensions
    {
        public static void InvokeActionOnTasksThreaded(this IEnumerable<Task> tasks, Action<Task> action, bool isBackground = true)
        {
            if (tasks == null) throw new ArgumentNullException("tasks");
            if (action == null) throw new ArgumentNullException("action");

            var exceptions = new List<Exception>();
            var threads = new List<Thread>();

            foreach (var task in tasks)
            {
                var thread = new Thread(t =>
                {
                    try { action(task); }
                    catch (Exception ex)
                    { lock (exceptions) exceptions.Add(ex); }
                }) { IsBackground = isBackground };

                threads.Add(thread);
                thread.Start();
            }

            foreach (var thread in threads)
                thread.Join();

            if (exceptions.Count > 0) throw new AggregateException(exceptions);
        }

        public static void InvokeActionOnTasksThreadPooled(this IEnumerable<Task> tasks, Action<Task> action)
        {
            if (tasks == null) throw new ArgumentNullException("tasks");
            if (action == null) throw new ArgumentNullException("action");

            var count = tasks.Count();
            if (count == 0) return;

            using (ManualResetEvent mre = new ManualResetEvent(false))
            {
                var remaining = count;
                var exceptions = new List<Exception>();

                foreach (var task in tasks)
                {
                    ThreadPool.UnsafeQueueUserWorkItem(_ =>
                    {
                        try { action(task); }
                        catch (Exception ex)
                        { lock (exceptions) exceptions.Add(ex); }
                        finally
                        {
                            if (Interlocked.Decrement(ref remaining) == 0)
                                mre.Set();
                        }
                    }, null);
                }

                mre.WaitOne();

                if (exceptions.Count > 0) throw new AggregateException(exceptions);
            }
        }
    }
}

I copied this part from my last post because I am lazy…

I took what I have learned in writing a custom TaskScheduler type, and defined an abstract class that can be used to derive custom schedulers that schedule their tasks to be run in parallel. The IParallelInvoker interface is unnecessary, but it helps clarify that the important method is the abstract ParallelInvoke method on the base class. Derived classes must then implement it to define how their tasks will be run in parallel.

I have derived two scheduler types.

  • The ParallelThreadPoolTaskScheduler runs its tasks in parallel on the managed ThreadPool.
  • The ParallelDedicatedThreadsTaskScheduler creates Thread instances, and can be configured to either run them in the foreground or in the background.

I then created a custom TaskFactory class, the ParallelTaskFactory, with two properties introduced, UseThreadPool, to set which scheduler will be returned by the Scheduler property, and UseForegroundThreads, to configure whether the ParallelDedicatedThreadsTaskScheduler instance runs its threads in the foreground or background.

Note that the TaskFactory.Scheduler property is not virtual, so it was necessary to use the new keyword to hide (and reimplement) the inherited property.

All that is still true… with the new virtual LongRunningTasksInvoke method added to the base class, and overridden by the ParallelDedicatedThreadsTaskScheduler, and all the implementations simplified thanks to the extension methods.

The base class now looks like this:

using System.Collections.Generic;
using System.Linq;

namespace System.Threading.Tasks.Schedulers
{
    /// <summary>Base class for a TaskScheduler that, when processing work items, 
    /// groups them into batches, and processes the Tasks of each batch in parallel. 
    /// The <see cref="ParallelInvoke"/> method is abstract so that derived classes 
    /// must specify how they run the Tasks in parallel.</summary>
    public abstract class ParallelTaskSchedulerBase : TaskScheduler, IParallelInvoker
    {
        #region Fields

        [ThreadStatic]
        private static bool currentThreadIsProcessingItems;

        private int maxDegreeOfParallelism;

        private volatile int runningOrQueuedCount;

        private readonly LinkedList<Task> tasks = new LinkedList<Task>();

        #endregion Fields

        #region Constructors

        protected ParallelTaskSchedulerBase(int maxDegreeOfParallelism)
        {
            if (maxDegreeOfParallelism < 1)
                throw new ArgumentOutOfRangeException("maxDegreeOfParallelism");

            this.maxDegreeOfParallelism = maxDegreeOfParallelism;
        }

        protected ParallelTaskSchedulerBase() : this(Environment.ProcessorCount) { }

        #endregion Constructors

        #region Properties

        public override int MaximumConcurrencyLevel
        {
            get { return maxDegreeOfParallelism; }
        }

        #endregion Properties

        #region Methods

        #region Public Methods

        public virtual void LongRunningTasksInvoke(Action<Task> action, params Task[] tasks)
        {
            tasks.InvokeActionOnTasksThreaded(action, true);
        }

        public abstract void ParallelInvoke(Action<Task> action, params Task[] tasks);

        #endregion Public Methods

        #region Protected Methods

        protected override bool TryDequeue(Task task)
        {
            lock (tasks) return tasks.Remove(task);
        }

        protected override bool TryExecuteTaskInline(Task task,
            bool taskWasPreviouslyQueued)
        {
            if (!currentThreadIsProcessingItems) return false;

            if (taskWasPreviouslyQueued) TryDequeue(task);

            return base.TryExecuteTask(task);
        }

        protected override IEnumerable<Task> GetScheduledTasks()
        {
            var lockTaken = false;
            try
            {
                Monitor.TryEnter(tasks, ref lockTaken);
                if (lockTaken) return tasks.ToArray();
                else throw new NotSupportedException();
            }
            finally { if (lockTaken) Monitor.Exit(tasks); }
        }

        protected override void QueueTask(Task task)
        {
            lock (tasks) tasks.AddLast(task);

            if (runningOrQueuedCount < maxDegreeOfParallelism)
            {
                runningOrQueuedCount++;
                RunTasks();
            }
        }

        #endregion Protected Methods

        #region Private Methods

        private void RunTasks()
        {
            var taskList = new List<Task>();
            var longRunningTaskList = new List<Task>();

            currentThreadIsProcessingItems = true;
            try
            {
                while (true)
                {
                    lock (tasks)
                    {
                        if (tasks.Count == 0)
                        {
                            runningOrQueuedCount--;
                            break;
                        }

                        var t = tasks.First.Value;

                        if ((t.CreationOptions & TaskCreationOptions.LongRunning) != TaskCreationOptions.None)
                            longRunningTaskList.Add(t);
                        else
                            taskList.Add(t);

                        tasks.RemoveFirst();
                    }
                }

                if (longRunningTaskList.Count > 0)
                    LongRunningTasksInvoke(t => base.TryExecuteTask(t), longRunningTaskList.ToArray());

                if (taskList.Count > 0)
                {
                    var batches = taskList.GroupBy(
                        task => taskList.IndexOf(task) / maxDegreeOfParallelism);

                    foreach (var batch in batches)
                        ParallelInvoke(t => base.TryExecuteTask(t), batch.ToArray());
                }
            }
            finally { currentThreadIsProcessingItems = false; }
        }

        #endregion Private Methods

        #endregion Methods
    }
}

Here is the ParallelThreadPoolTaskScheduler. (It’s name also changed.)

namespace System.Threading.Tasks.Schedulers
{
    /// <summary>A TaskScheduler that, when processing work items, groups 
    /// them into batches, and processes the Tasks of each batch in parallel. 
    /// This <b>ParallelInvoke</b> implementation runs the work on background 
    /// threads, using the THreadPool.</summary>
    public sealed class ParallelThreadPoolTaskScheduler : ParallelTaskSchedulerBase
    {
        /// <summary>Invoke the specified Tasks, executing the specified 
        /// action for each on the ThreadPool in parallel.</summary>
        public override void ParallelInvoke(Action<Task> action, params Task[] tasks)
        {
            tasks.InvokeActionOnTasksThreadPooled(action);
        }
    }
}

And here’s the other one…

namespace System.Threading.Tasks.Schedulers
{
    /// <summary>A TaskScheduler that, when processing work items, groups 
    /// them into batches, and processes the Tasks of each batch in parallel. 
    /// This <b>ParallelInvoke</b> implementation runs the work on dedicated 
    /// threads, and can be configured to use foreground threads.</summary>
    public sealed class ParallelDedicatedThreadsTaskScheduler : ParallelTaskSchedulerBase
    {
        #region Fields

        private bool useForegroundThreads;

        #endregion Fields

        #region Properties

        public bool UseForegroundThreads
        {
            get { return useForegroundThreads; }
            set { useForegroundThreads = value; }
        }

        #endregion Properties

        #region Methods

        #region Public Methods

        public override void LongRunningTasksInvoke(Action<Task> action, params Task[] tasks)
        {
            tasks.InvokeActionOnTasksThreaded(action, !UseForegroundThreads);
        }

        public override void ParallelInvoke(Action<Task> action, params Task[] tasks)
        {
            InternalTasksInvoke(action, tasks);
        }

        #endregion Public Methods

        #region Private Methods

        /// <summary>Invoke the specified Tasks, executing the specified action for each 
        /// on a dedicated thread, in parallel. <b>UseForegroundThreads</b> determines 
        /// whether threads are run in the foreground or background.</summary>
        private void InternalTasksInvoke(Action<Task> action, Task[] tasks)
        {
            tasks.InvokeActionOnTasksThreaded(action, !UseForegroundThreads);
        }

        #endregion Private Methods

        #endregion Methods
    }
}

Last time I wrote this (and it still applies)

And lastly, here is the code for my static Tasks class, and my custom TaskFactory, which are both in one file. Note that I had declared all the scheduler types in the code as TaskScheduler types originally (I figured it makes better polymorphic sense), but changed them a few minutes ago because declaring them using the actual types shows the relationships properly on the class diagram.

using System;
using System.Threading;
using System.Threading.Tasks;
using System.Threading.Tasks.Schedulers;

namespace Romy.Core
{
    /// <summary>Set up some common options for async Tasks. The <b>TaskSchedulers.Parallel</b>
    /// scheduler references the static <b>Scheduler</b> property of this class. The value of
    /// <b>Factory.UseForeGroundThreads</b> defines which <B>ParallelTaskSchedulerBase</B> 
    /// implementation is used, and whether the underlying threads will be foreground threads 
    /// or background TheadPool threads.</summary>
    public static class Tasks
    {
        private static ParallelTaskFactory factory = new ParallelTaskFactory();

        public static ParallelTaskFactory Factory
        {
            get { return Tasks.factory; }
        }

        public static TaskScheduler Scheduler
        {
            get { return factory.Scheduler; }
        }

        public static Task Run(Func<Task> body)
        {
            return factory.StartNew(async () => await body()).Unwrap();
        }

        public static Task<T> Run<T>(Func<Task<T>> body, CancellationToken cancellationToken)
        {
            return factory.StartNew(async () => await body(), cancellationToken, TaskCreationOptions.LongRunning | TaskCreationOptions.PreferFairness, Scheduler).Unwrap();
        }
    }

    /// <summary>A TaskFactory to define which TaskScheduler will 
    /// be used for most async Tasks in the application.</summary>
    public class ParallelTaskFactory : TaskFactory
    {
        private bool useForeGroundThreads;

        private bool useThreadPool;

        private static Lazy<ParallelDedicatedThreadsTaskScheduler> dedicatedThreadsScheduler = new Lazy<ParallelDedicatedThreadsTaskScheduler>(() => new ParallelDedicatedThreadsTaskScheduler());

        private static Lazy<ParallelThreadPoolTaskScheduler> threadPoolScheduler = new Lazy<ParallelThreadPoolTaskScheduler>(() => new ParallelThreadPoolTaskScheduler());

        public ParallelTaskFactory()
            : base(CancellationToken.None,
                TaskCreationOptions.DenyChildAttach | TaskCreationOptions.HideScheduler,
                TaskContinuationOptions.ExecuteSynchronously, ThreadPoolScheduler) { }

        public bool UseForeGroundThreads
        {
            get { return useForeGroundThreads; }
            set
            {
                useForeGroundThreads = value;
                DedicatedThreadsScheduler.UseForegroundThreads = useForeGroundThreads;
            }
        }

        public bool UseThreadPool
        {
            get { return useThreadPool; }
            set { useThreadPool = value; }
        }

        public static ParallelDedicatedThreadsTaskScheduler DedicatedThreadsScheduler
        {
            get { return ParallelTaskFactory.dedicatedThreadsScheduler.Value; }
        }

        public new TaskScheduler Scheduler
        {
            get { return useThreadPool ? (TaskScheduler)ThreadPoolScheduler : (TaskScheduler)DedicatedThreadsScheduler; }
        }

        public static ParallelThreadPoolTaskScheduler ThreadPoolScheduler
        {
            get { return ParallelTaskFactory.threadPoolScheduler.Value; }
        }
    }
}

And now my code is a bit fairer for long running tasks.

Posted in Programming | Tagged , , , | Leave a comment

Improving performance in Windows Forms by temporarily suspending redrawing a window

Sometimes the default way a window gets redrawn wreaks havoc with our code that’s updating the window, because the window gets redrawn at some point(s) when we have made some but not all of our changes. This can cause anything from windows that look like badly created stop-motion animations while they are being resized, to container windows that display child windows being added, then flicker and repaint as the child windows are reordered, to windows that look normal but are unresponsive to user input.

At such times, it would be good to have a way of suspending the painting of the respective windows, but all we get built-in for this is the SuspendLayoutResumeLayout pattern, which does not do quite what we need here.

A mistake many developers make here is calling LockWindowUpdate. Raymond Chen wrote a great series explaining why you shouldn’t do so, amongst other things, a few years ago.

If you have read the linked post, you know the solution already: WM_SETREDRAW

I wrote a helper class in c# to do the grunt work for me. It is available in my RomyView application.

In addition to wrapping the SendMessage call and reference counting it so that it can handle nested calls, the code that redraws the window uses the native RedrawWindow API function. This is because the function takes an options parameter that allows us to optimize the redrawing code, passing different flags for container windows to child windows.

The helper class has a SuspendDrawing and ResumeDrawing method, where ResumeDrawing has an overload that accepts a bool parameter to indicate whether the window is a container or child; as well as a ResumeDrawing overload that exposes the options as a parameter so that you can specify your own combination of flags for a specific window if needed.

Thus I use the code in a couple of similar, but subtly different places. For example:

  • When toggling the image viewer window between full screen and windowed mode – which not only resizes the window after changing the window state between normal and maximized, but also changes the border style as well as a few child-window properties. Without temporarily suspending the redrawing, a few horrendously ugly repaints were rendered, with the window in an intermediate state that I never even imagined, let alone expected to be visible.

  • I also use it to insert thumbnails into a FlowLayoutPanel. Of course ControlCollection doesn’t have an Insert method, but my custom collection does. The Insert implementation adds a thumbnail but prevents it from being drawn, then changes the thumbnail’s index and resumes it’s redrawing.

If you don’t want the whole application just for this code, here is the relevant class…

using System;
using System.Runtime.InteropServices;
using System.Security;
using System.Threading;
using System.Windows.Forms;

namespace Romy.Controls
{
    /// <summary>Allows suspending and resuming redrawing a Windows Forms window via the <b>WM_SETREDRAW</b> 
    /// Windows message.</summary>
    /// <remarks>Usage: The window for which drawing will be suspended and resumed needs to instantiate this type, 
    /// passing a reference to itself to the constructor, then call either of the public methods. For each call to 
    /// <b>SuspendDrawing</b>, a corresponding <b>ResumeDrawing</b> call must be made. Calls may be nested, but
    /// should not be made from any other than the GUI thread. (This code tries to work around such an error, but 
    /// is not guaranteed to succeed.)</remarks>
    public class DrawingHelper
    {
        #region Fields

        private int suspendCounter;

        private const int WM_SETREDRAW = 11;

        private IWin32Window owner;

        private SynchronizationContext synchronizationContext = SynchronizationContext.Current;

        #endregion Fields

        #region Constructors

        public DrawingHelper(IWin32Window owner)
        {
            this.owner = owner;
        }

        #endregion Constructors

        #region Methods

        /// <summary>This overload allows you to specify whether the optimal flags for a container 
        /// or child control should be used. To specify custom flags, use the overload that accepts 
        /// a <see cref="Romy.Controls.RedrawWindowFlags"/> parameter.</summary>
        /// <param name="isContainer">When <b>true</b>, the optimal flags for redrawing a container 
        /// control are used; otherwise the optimal flags for a child control are used.</param>
        public void ResumeDrawing(bool isContainer = false)
        {
            ResumeDrawing(isContainer ? RedrawWindowFlags.Erase | RedrawWindowFlags.Frame | RedrawWindowFlags.Invalidate | RedrawWindowFlags.AllChildren :
                RedrawWindowFlags.NoErase | RedrawWindowFlags.Invalidate | RedrawWindowFlags.InternalPaint);
        }

        [CLSCompliant(false)]
        public void ResumeDrawing(RedrawWindowFlags flags)
        {
            Interlocked.Decrement(ref suspendCounter);

            if (suspendCounter == 0)
            {
                Action resume = new Action(() =>
                {
                    NativeMethods.SendMessage(owner.Handle, WM_SETREDRAW, new IntPtr(1), IntPtr.Zero);
                    NativeMethods.RedrawWindow(owner.Handle, IntPtr.Zero, IntPtr.Zero, flags);
                });
                try { resume(); }
                catch (InvalidOperationException)
                {
                    synchronizationContext.Post(s => ((Action)s)(), resume);
                }
            }
        }

        public void SuspendDrawing()
        {
            try
            {
                if (suspendCounter == 0)
                {
                    Action suspend = new Action(() => NativeMethods.SendMessage(owner.Handle, WM_SETREDRAW, IntPtr.Zero, IntPtr.Zero));
                    try { suspend(); }

                    catch (InvalidOperationException)
                    {
                        synchronizationContext.Post(s => ((Action)s)(), suspend);
                    }
                }
            }
            finally { Interlocked.Increment(ref suspendCounter); }
        }

        #endregion Methods

        #region NativeMethods

        [SuppressUnmanagedCodeSecurity]
        internal static class NativeMethods
        {
            [DllImport("user32.dll")]
            public static extern bool RedrawWindow(IntPtr hWnd, IntPtr lprcUpdate, IntPtr hrgnUpdate, RedrawWindowFlags flags);

            [DllImport("user32.dll")]
            public static extern IntPtr SendMessage(IntPtr hWnd, Int32 wMsg, IntPtr wParam, IntPtr lParam);
        }

        #endregion NativeMethods
    }

    #region RedrawWindowFlags

    [Flags(), CLSCompliant(false)]
    public enum RedrawWindowFlags : uint
    {
        ///<summary>Invalidates lprcUpdate or hrgnUpdate (only one may be non-NULL). 
        ///If both are NULL, the entire window is invalidated.</summary>
        Invalidate = 0x1,

        ///<summary>Causes a WM_PAINT message to be posted to the window regardless of 
        ///whether any portion of the window is invalid.</summary>
        InternalPaint = 0x2,

        ///<summary>Causes the window to receive a WM_ERASEBKGND message when the window 
        ///is repainted. The <b>Invalidate</b> flag must also be specified; otherwise, 
        ///<b>Erase</b> has no effect.</summary>
        Erase = 0x4,

        ///<summary>Validates lprcUpdate or hrgnUpdate (only one may be non-NULL). If both 
        ///are NULL, the entire window is validated. This flag does not affect internal 
        ///WM_PAINT messages.</summary>
        Validate = 0x8,

        ///<summary>Suppresses any pending internal WM_PAINT messages. This flag does not 
        ///affect WM_PAINT messages resulting from a non-NULL update area.</summary>
        NoInternalPaint = 0x10,

        ///<summary>Suppresses any pending WM_ERASEBKGND messages.</summary>
        NoErase = 0x20,

        ///<summary>Excludes child windows, if any, from the repainting operation.</summary>
        NoChildren = 0x40,

        ///<summary>Includes child windows, if any, in the repainting operation.</summary>
        AllChildren = 0x80,

        ///<summary>Causes the affected windows (as specified by the <b>AllChildren</b> and <b>NoChildren</b> flags) to 
        ///receive WM_NCPAINT, WM_ERASEBKGND, and WM_PAINT messages, if necessary, before the function returns.</summary>
        UpdateNow = 0x100,

        ///<summary>Causes the affected windows (as specified by the <b>AllChildren</b> and <b>NoChildren</b> flags) 
        ///to receive WM_NCPAINT and WM_ERASEBKGND messages, if necessary, before the function returns. 
        ///WM_PAINT messages are received at the ordinary time.</summary>
        EraseNow = 0x200,

        ///<summary>Causes any part of the nonclient area of the window that intersects the update region 
        ///to receive a WM_NCPAINT message. The <b>Invalidate</b> flag must also be specified; otherwise, 
        ///<b>Frame</b> has no effect. The WM_NCPAINT message is typically not sent during the execution of 
        ///RedrawWindow unless either <b>UpdateNow</b> or <b>EraseNow</b> is specified.</summary>
        Frame = 0x400,

        ///<summary>Suppresses any pending WM_NCPAINT messages. This flag must be used with <b>Validate</b> and 
        ///is typically used with <b>NoChildren</b>. <b>NoFrame</b> should be used with care, as it could cause parts 
        ///of a window to be painted improperly.</summary>
        NoFrame = 0x800
    }

    #endregion RedrawWindowFlags
}
Posted in Programming | Tagged , , | Leave a comment

Updated application code

I’ve updated my shared application for download:

RomyView.zip download

Click the image for larger version…

RomyViewUI

This code includes everything I published recently, since I really use this application for most of my prototyping code.

The screenshot is shown using my dark colour theme. I think it looks the best of my three themes.

The ReOpenFile managed wrapper that I wrote about this morning is there, in Romy.Core.StreamExtensions.

The utility to change the Windows logon startup background is there with the Glass Form. (Shown.) I also worked some more on my Photoshopped monitor image (It is much improved and has a screen with 16:9 aspect ratio.), which I am also using for the program’s icons.

After commenting that I have never needed to write editors in the 2nd part of my post on playing with the PropertyGrid, I have added a couple in the solution. That’s my options dialog displayed, which allows you to select a directory path for the cache directory, using the  newer Windows common dialog (without using the Windows API Code-Pack), although I haven’t written about any of that. (I might still get around to it.) The same dialog also uses a custom UITypeEditor that shows a fully expanded  ColorDialog when you edit the colour property.

But this code is not production code and it is not bug-free, as it is always a work in progress. And BTW, if you would like the video player to be useable you need to have decent DirectShow codecs installed, such as the K-Lite Codec Pack.

I think this application is a good example of code that uses async everywhere. There’s very little synchronous code in the solution. In that regard, I have learned some lessons along the way… Like most developers I thought that asynchronous code runs faster… Just throw more threads at your problems. That is a fallacy! Asynchronous code generally runs slower than synchronous code. But compute-heavy tasks can be parallelized; and the more processor cores you have, the better it performs.

Think of synchronous code like a single sprinter in some vast wasteland. He can run super-fast, but the asynchronous code he’s up against is a zombie horde. They might move really slowly, but there’s potentially millions of them surrounding him. Who do you think is going to win?

Posted in Programming | Tagged | 1 Comment

Fascinating stuff – found a copy of Stephen Toub’s Managed ThreadPool that used to be hosted on the now defunct GotDotNet.com

I’m still waiting for our sprint planning meeting to start… it’s only at 2PM… so I have spent the whole day reading interesting articles; mostly programming-related.

I’ve been feeling more than a little disappointed that the push on all the Microsoft blogs is about Visual Studio 2013 (Frankly, I don’t give a fuck am not interested.), and the Parallel Programming Blog hasn’t had a post in 4 months.

That means I’ve had nothing to read from my favourite blog author. Seriously, I find Mr Toub’s posts are always fascinating, and he has a way of making complex code very simple that I envy. I’ve always regarded concise code to be of the highest quality (to the point where I detest code that is over-engineered). Sadly I will probably never write such excellent code myself, but that doesn’t mean I won’t keep trying. (Perfection is my goal even if I can never achieve it.)

Anyway, along the way today I found that he wrote a managed implementation of the ThreadPool in C# but it was hosted on GotDotNet.com which has since GotDot.Gone. I was highly amused to find that the most widely adopted “custom threadpool” is something called a Smart ThreadPool which is based on Stephen’s code. Read between the lines… someone who was incapable of implementing a thread pool himself took Stephen’s code and added a bunch of bells and whistles.

Actually I used to use that implementation a couple of years ago before I ever heard of Stephen. And no offense to the smart refactorer… I am also incapable.

Anyway, I found a copy of Stephen’s code in a public share somewhere. Here it is:

// Stephen Toub
// stoub@microsoft.com
//
// Very simple threadpool in C#.
// 4/27/04

#region Namespaces
using System;
using System.Threading;
using System.Collections;
#endregion

namespace Toub.Threading
{
    /// <summary>Implementation of Dijkstra's PV Semaphore based on the Monitor class.</summary>
    public class Semaphore
    {
        #region Member Variables
        /// <summary>The number of units alloted by this semaphore.</summary>
        private int _count;
        /// <summary>Lock for the semaphore.</summary>
        private object _semLock = new object();
        #endregion

        #region Construction
        /// <summary> Initialize the semaphore as a binary semaphore.</summary>
        public Semaphore() : this(1) 
        {
        }

        /// <summary> Initialize the semaphore as a counting semaphore.</summary>
        /// <param name="count">Initial number of threads that can take out units from this semaphore.</param>
        /// <exception cref="ArgumentException">Throws if the count argument is less than 0.</exception>
        public Semaphore(int count) 
        {
            if (count < 0) throw new ArgumentException("Semaphore must have a count of at least 0.", "count");
            _count = count;
        }
        #endregion

        #region Synchronization Operations
        /// <summary>V the semaphore (add 1 unit to it).</summary>
        public void AddOne() { V(); }

        /// <summary>P the semaphore (take out 1 unit from it).</summary>
        public void WaitOne() { P(); }

        /// <summary>P the semaphore (take out 1 unit from it).</summary>
        public void P() 
        {
            // Lock so we can work in peace.  This works because lock is actually
            // built around Monitor.
            lock(_semLock) 
            {
                // Wait until a unit becomes available.  We need to wait
                // in a loop in case someone else wakes up before us.  This could
                // happen if the Monitor.Pulse statements were changed to Monitor.PulseAll
                // statements in order to introduce some randomness into the order
                // in which threads are woken.
                while(_count <= 0) Monitor.Wait(_semLock, Timeout.Infinite);
                _count--;
            }
        }

        /// <summary>V the semaphore (add 1 unit to it).</summary>
        public void V() 
        {
            // Lock so we can work in peace.  This works because lock is actually
            // built around Monitor.
            lock(_semLock) 
            {
                // Release our hold on the unit of control.  Then tell everyone
                // waiting on this object that there is a unit available.
                _count++;
                Monitor.Pulse(_semLock);
            }
        }

        /// <summary>Resets the semaphore to the specified count.  Should be used cautiously.</summary>
        public void Reset(int count)
        {
            lock(_semLock) { _count = count; }
        }
        #endregion
    }

    /// <summary>Managed thread pool.</summary>
    public class ManagedThreadPool
    {
        #region Constants
        /// <summary>Maximum number of threads the thread pool has at its disposal.</summary>
        private const int _maxWorkerThreads = 25;
        #endregion

        #region Member Variables
        /// <summary>Queue of all the callbacks waiting to be executed.</summary>
        private static Queue _waitingCallbacks;
        /// <summary>
        /// Used to signal that a worker thread is needed for processing.  Note that multiple
        /// threads may be needed simultaneously and as such we use a semaphore instead of
        /// an auto reset event.
        /// </summary>
        private static Semaphore _workerThreadNeeded;
        /// <summary>List of all worker threads at the disposal of the thread pool.</summary>
        private static ArrayList _workerThreads;
        /// <summary>Number of threads currently active.</summary>
        private static int _inUseThreads;
        /// <summary>Lockable object for the pool.</summary>
        private static object _poolLock = new object();
        #endregion

        #region Construction and Finalization
        /// <summary>Initialize the thread pool.</summary>
        static ManagedThreadPool() { Initialize(); }

        /// <summary>Initializes the thread pool.</summary>
        private static void Initialize()
        {
            // Create our thread stores; we handle synchronization ourself
            // as we may run into situtations where multiple operations need to be atomic.
            // We keep track of the threads we've created just for good measure; not actually
            // needed for any core functionality.
            _waitingCallbacks = new Queue();
            _workerThreads = new ArrayList();
            _inUseThreads = 0;

            // Create our "thread needed" event
            _workerThreadNeeded = new Semaphore(0);
            
            // Create all of the worker threads
            for(int i=0; i<_maxWorkerThreads; i++)
            {
                // Create a new thread and add it to the list of threads.
                Thread newThread = new Thread(new ThreadStart(ProcessQueuedItems));
                _workerThreads.Add(newThread);

                // Configure the new thread and start it
                newThread.Name = "ManagedPoolThread #" + i.ToString();
                newThread.IsBackground = true;
                newThread.Start();
            }
        }
        #endregion

        #region Public Methods
        /// <summary>Queues a user work item to the thread pool.</summary>
        /// <param name="callback">
        /// A WaitCallback representing the delegate to invoke when the thread in the 
        /// thread pool picks up the work item.
        /// </param>
        public static void QueueUserWorkItem(WaitCallback callback)
        {
            // Queue the delegate with no state
            QueueUserWorkItem(callback, null);
        }

        /// <summary>Queues a user work item to the thread pool.</summary>
        /// <param name="callback">
        /// A WaitCallback representing the delegate to invoke when the thread in the 
        /// thread pool picks up the work item.
        /// </param>
        /// <param name="state">
        /// The object that is passed to the delegate when serviced from the thread pool.
        /// </param>
        public static void QueueUserWorkItem(WaitCallback callback, object state)
        {
            // Create a waiting callback that contains the delegate and its state.
            // At it to the processing queue, and signal that data is waiting.
            WaitingCallback waiting = new WaitingCallback(callback, state);
            lock(_poolLock) { _waitingCallbacks.Enqueue(waiting); }
            _workerThreadNeeded.AddOne();
        }

        /// <summary>Empties the work queue of any queued work items.  Resets all threads in the pool.</summary>
        public static void Reset()
        {
            lock(_poolLock) 
            { 
                // Cleanup any waiting callbacks
                try 
                {
                    // Try to dispose of all remaining state
                    foreach(object obj in _waitingCallbacks)
                    {
                        WaitingCallback callback = (WaitingCallback)obj;
                        if (callback.State is IDisposable) ((IDisposable)callback.State).Dispose();
                    }
                } 
                catch{}

                // Shutdown all existing threads
                try 
                {
                    foreach(Thread thread in _workerThreads) 
                    {
                        if (thread != null) thread.Abort("reset");
                    }
                }
                catch{}

                // Reinitialize the pool (create new threads, etc.)
                Initialize();
            }
        }
        #endregion

        #region Properties
        /// <summary>Gets the number of threads at the disposal of the thread pool.</summary>
        public static int MaxThreads { get { return _maxWorkerThreads; } }
        /// <summary>Gets the number of currently active threads in the thread pool.</summary>
        public static int ActiveThreads { get { return _inUseThreads; } }
        /// <summary>Gets the number of callback delegates currently waiting in the thread pool.</summary>
        public static int WaitingCallbacks { get { lock(_poolLock) { return _waitingCallbacks.Count; } } }
        #endregion

        #region Thread Processing
        /// <summary>Event raised when there is an exception on a threadpool thread.</summary>
        public static event UnhandledExceptionEventHandler UnhandledException;

        /// <summary>A thread worker function that processes items from the work queue.</summary>
        private static void ProcessQueuedItems()
        {
            // Process indefinitely
            while(true)
            {
                _workerThreadNeeded.WaitOne();

                // Get the next item in the queue.  If there is nothing there, go to sleep
                // for a while until we're woken up when a callback is waiting.
                WaitingCallback callback = null;

                // Try to get the next callback available.  We need to lock on the 
                // queue in order to make our count check and retrieval atomic.
                lock(_poolLock)
                {
                    if (_waitingCallbacks.Count > 0)
                    {
                        try { callback = (WaitingCallback)_waitingCallbacks.Dequeue(); } 
                        catch{} // make sure not to fail here
                    }
                }

                if (callback != null)
                {
                    // We now have a callback.  Execute it.  Make sure to accurately
                    // record how many callbacks are currently executing.
                    try 
                    {
                        Interlocked.Increment(ref _inUseThreads);
                        callback.Callback(callback.State);
                    } 
                    catch(Exception exc)
                    {
                        try
                        {
                            UnhandledExceptionEventHandler handler = UnhandledException;
                            if (handler != null) handler(typeof(ManagedThreadPool), new UnhandledExceptionEventArgs(exc, false));
                        }
                        catch{}
                    }
                    finally
                    {
                        Interlocked.Decrement(ref _inUseThreads);
                    }
                }
            }
        }
        #endregion

        /// <summary>Used to hold a callback delegate and the state for that delegate.</summary>
        private class WaitingCallback
        {
            #region Member Variables
            /// <summary>Callback delegate for the callback.</summary>
            private WaitCallback _callback;
            /// <summary>State with which to call the callback delegate.</summary>
            private object _state;
            #endregion

            #region Construction
            /// <summary>Initialize the callback holding object.</summary>
            /// <param name="callback">Callback delegate for the callback.</param>
            /// <param name="state">State with which to call the callback delegate.</param>
            public WaitingCallback(WaitCallback callback, object state)
            {
                _callback = callback;
                _state = state;
            }
            #endregion

            #region Properties
            /// <summary>Gets the callback delegate for the callback.</summary>
            public WaitCallback Callback { get { return _callback; } }
            /// <summary>Gets the state with which to call the callback delegate.</summary>
            public object State { get { return _state; } }
            #endregion
        }
    }
}
Posted in Programming | Tagged , | Leave a comment

Convert a FileStream opened in synchronous mode to one in asynchronous mode, in C#

Raymond Chen published an interesting blog post today explaining how to convert a synchronous file handle to an asynchronous one, via the ReOpenFile Windows API function.

But that function is not implemented in the .Net BCL. Thus I wrote a managed wrapper for it quickly – an extension method on FileStream. (I’m just sitting around waiting for us to start our sprint planning meeting anyway.)

Actually I have not tested this at all, but I imagine it works. Anyway, here’s the code:

using Microsoft.Win32.SafeHandles;
using System;
using System.IO;
using System.Runtime.InteropServices;

namespace Romy.Core.Extensions
{
    public static class Example
    {
        private const uint FILE_FLAG_OVERLAPPED = 0x40000000;

        public static FileStream Reopen(this FileStream source, FileAccess access, FileShare share, int bufferSize, bool useAsync)
        {
            var fileAccess = EFileAccess.GenericWrite;

            switch (access)
            {
                case FileAccess.Read:
                    fileAccess = EFileAccess.GenericRead;
                    break;
                case FileAccess.ReadWrite:
                    fileAccess = EFileAccess.GenericRead | EFileAccess.GenericWrite;
                    break;
                case FileAccess.Write:
                    fileAccess = EFileAccess.GenericWrite;
                    break;
            }

            var fileShare = EFileShare.Read | EFileShare.Write;

            switch (share)
            {
                case FileShare.Delete:
                    fileShare = EFileShare.Delete;
                    break;
                case FileShare.Inheritable:
                    fileShare = source.CanRead ? EFileShare.Read : source.CanWrite ? EFileShare.Write : EFileShare.None;
                    break;
                case FileShare.None:
                    fileShare = EFileShare.None;
                    break;
                case FileShare.Read:
                    fileShare = EFileShare.Read;
                    break;
                case FileShare.ReadWrite:
                    fileShare = EFileShare.Read | EFileShare.Write;
                    break;
                case FileShare.Write:
                    fileShare = EFileShare.Write;
                    break;
            }

            return new FileStream(NativeMethods.ReOpenFile(source.SafeFileHandle, (uint)fileAccess, (uint)fileShare, useAsync ? FILE_FLAG_OVERLAPPED : 0), access, bufferSize, useAsync);
        }

        #region Interop

        private static class NativeMethods
        {
            [DllImport("kernel32", SetLastError = true)]
            public static extern SafeFileHandle ReOpenFile(SafeFileHandle hOriginalFile, uint dwAccess, uint dwShareMode, uint dwFlags);
        }

        [Flags]
        private enum EFileAccess : uint
        {
            // Standard Section
            AccessSystemSecurity = 0x1000000,   // AccessSystemAcl access type
            MaximumAllowed = 0x2000000,     // MaximumAllowed access type

            Delete = 0x10000,
            ReadControl = 0x20000,
            WriteDAC = 0x40000,
            WriteOwner = 0x80000,
            Synchronize = 0x100000,

            StandardRightsRequired = 0xF0000,
            StandardRightsRead = ReadControl,
            StandardRightsWrite = ReadControl,
            StandardRightsExecute = ReadControl,
            StandardRightsAll = 0x1F0000,
            SpecificRightsAll = 0xFFFF,

            FILE_READ_DATA = 0x0001,        // file & pipe
            FILE_LIST_DIRECTORY = 0x0001,       // directory
            FILE_WRITE_DATA = 0x0002,       // file & pipe
            FILE_ADD_FILE = 0x0002,         // directory
            FILE_APPEND_DATA = 0x0004,      // file
            FILE_ADD_SUBDIRECTORY = 0x0004,     // directory
            FILE_CREATE_PIPE_INSTANCE = 0x0004, // named pipe
            FILE_READ_EA = 0x0008,          // file & directory
            FILE_WRITE_EA = 0x0010,         // file & directory
            FILE_EXECUTE = 0x0020,          // file
            FILE_TRAVERSE = 0x0020,         // directory
            FILE_DELETE_CHILD = 0x0040,     // directory
            FILE_READ_ATTRIBUTES = 0x0080,      // all
            FILE_WRITE_ATTRIBUTES = 0x0100,     // all

            // Generic Section
            GenericRead = 0x80000000,
            GenericWrite = 0x40000000,
            GenericExecute = 0x20000000,
            GenericAll = 0x10000000,

            SPECIFIC_RIGHTS_ALL = 0x00FFFF,
            FILE_ALL_ACCESS =
            StandardRightsRequired |
            Synchronize |
            0x1FF,

            FILE_GENERIC_READ =
            StandardRightsRead |
            FILE_READ_DATA |
            FILE_READ_ATTRIBUTES |
            FILE_READ_EA |
            Synchronize,

            FILE_GENERIC_WRITE =
            StandardRightsWrite |
            FILE_WRITE_DATA |
            FILE_WRITE_ATTRIBUTES |
            FILE_WRITE_EA |
            FILE_APPEND_DATA |
            Synchronize,

            FILE_GENERIC_EXECUTE =
            StandardRightsExecute |
              FILE_READ_ATTRIBUTES |
              FILE_EXECUTE |
              Synchronize
        }

        [Flags]
        private enum EFileShare : uint
        {
            None = 0x00000000,
            /// <summary>
            /// Enables subsequent open operations on an object to request read access.
            /// Otherwise, other processes cannot open the object if they request read access.
            /// If this flag is not specified, but the object has been opened for read access, the function fails.
            /// </summary>
            Read = 0x00000001,
            /// <summary>
            /// Enables subsequent open operations on an object to request write access.
            /// Otherwise, other processes cannot open the object if they request write access.
            /// If this flag is not specified, but the object has been opened for write access, the function fails.
            /// </summary>
            Write = 0x00000002,
            /// <summary>
            /// Enables subsequent open operations on an object to request delete access.
            /// Otherwise, other processes cannot open the object if they request delete access.
            /// If this flag is not specified, but the object has been opened for delete access, the function fails.
            /// </summary>
            Delete = 0x00000004
        }

        #endregion

    }
}
Posted in Programming | Tagged , , | Leave a comment

How to start an external process that is associated with a file type, and bring it’s window to the front, in C#

I found a number of answers regarding how to do this online, but none worked exactly the way I wanted. This is because I actually have one extra requirement. That is, if the file type does not have an associated program, I want the Windows dialog to be shown so that I can select and/or associate a default program.

For example (and to make this dull post prettier)… I created my program’s Icon in Photoshop (I use layers for the three different themes), but being obsessive, for the last few days I found myself staring at it… I don’t like the shading in those pixels over there… so I need to open it in Photoshop, and why not do so from my own application?

OpenWithShellExample

To summarise, the code posted here will:

  1. Launch an external process for any given file name. If the file does not have an associated program, the user is prompted to choose one, with the option of associating it as the default program for the particular file type.
  2. Bring the main window of the external process to the front of the Z-order, if it is a program with a Windows GUI, and thus a message pump.

First of all, the solution is simply to call Process.Start, specifying the file name, and setting UseShellExecute to true. That’s what I found everywhere, but it does not solve the issue of what to do when there isn’t an associated program. For that to work, the Verb must also be set.

Here’s my code.

using System;
using System.Diagnostics;
using System.Runtime.InteropServices;
using System.Security.Permissions;
using System.Threading.Tasks;

namespace Romy.Core
{
    public static class Example
    {
        public const int SW_RESTORE = 9;

        internal static class NativeMethods
        {
            [DllImport("user32.dll")]
            [return: MarshalAs(UnmanagedType.Bool)]
            internal static extern bool IsIconic(System.IntPtr hWnd);

            [DllImport("user32.dll")]
            [return: MarshalAs(UnmanagedType.Bool)]
            internal static extern bool SetForegroundWindow(System.IntPtr hWnd);

            [DllImport("user32.dll")]
            [return: MarshalAs(UnmanagedType.Bool)]
            internal static extern bool ShowWindowAsync(System.IntPtr hWnd, int nCmdShow);

            [DllImport("user32.dll")]
            internal static extern System.IntPtr SetActiveWindow(System.IntPtr hWnd);
        }

        public static async void ShellExecuteFile(string filename)
        {
            new SecurityPermission(SecurityPermissionFlag.UnmanagedCode).Demand();

            var startInfo = new ProcessStartInfo()
            {
                FileName = filename,
                Verb = "open",
                UseShellExecute = true,
                ErrorDialog = true
            };

            var p = Process.Start(startInfo);

            try
            {
                await Task.Run(async () =>
                {
                    try
                    {
                        p.WaitForInputIdle();
                        IntPtr handle = p.MainWindowHandle;

                        while (handle == IntPtr.Zero)
                        {
                            await Task.Delay(TimeSpan.FromMilliseconds(250D));
                            handle = p.MainWindowHandle;
                        }

                        if (handle != IntPtr.Zero)
                        {
                            if (NativeMethods.IsIconic(handle))
                                NativeMethods.ShowWindowAsync(handle, SW_RESTORE);

                            if (NativeMethods.SetForegroundWindow(handle))
                                NativeMethods.SetActiveWindow(handle);
                        }
                    }
                    catch (InvalidOperationException) { }
                    catch (PlatformNotSupportedException) { }
                    catch (NotSupportedException) { }
                    catch (Exception ex) { ex.Log(); }
                }).TimeoutAfter(TimeSpan.FromSeconds(3D));
            }
            catch (TimeoutException) { }
        }
    }
}

It should be straight-forward to follow what it does… just launches a process as explained, then waits for that process to enter an idle state. Then, if the MainWindowHandle property is still not defined, checks it in a loop. If it gets a valid handle, it sets focus on that window; otherwise it gives up after three seconds. Thanks to async code, we don’t need to call Sleep. (Actually I have never called Sleep, but that’s another story.)

Note: There’s only so much you can do when the process doesn’t belong to your application. When I first searched to find out how to do this (before deciding to write it myself as I had already decided I probably would), I found some strange code out there. One person used AttachThreadInput. Um… don’t do that!

The only thing to be careful of with this approach, is of course that the user might close the application during the time that our code is calling Task.Delay, which will then cause the code that accesses the Process.WindowHandle property (which internally calls Process.getMainWindowHandle() in the framework code) to throw an InvalidOperationException. Hence the try…catch. Actually you should always add an exception handler for those exceptions when using the System.Diagnostics.Process class anyway.

Of course, there is no such method as Task.TimeoutAfter. That’s an extension method defined on a post on the Parallel Programming team blog: Crafting a Task.TimeoutAfter Method.

I modified the code (very slightly), so here is the version that I am currently using:

        #region Task.TimeoutAfter

        /* Task.TimeoutAfter extension methods: Based on sample code by Joe Hoag of Microsoft, from the article at
         * http://blogs.msdn.com/b/pfxteam/archive/2011/11/10/10235834.aspx, Crafting a Task.TimeoutAfter Method.
         * I did not change the original code; only added the overloads that take a TimeSpan argument.
         * 
         * Note that the returned Task on a timeout is set to the faulted state with a TimeoutException, so when
         * calling this you need to catch the TimeoutException, when it times out. */

        public static Task TimeoutAfter(this Task task, TimeSpan timeoutSpan)
        {
            return task.TimeoutAfter((int)timeoutSpan.TotalMilliseconds);
        }

        public static Task<TResult> TimeoutAfter<TResult>(this Task<TResult> task, TimeSpan timeoutSpan)
        {
            return task.TimeoutAfter((int)timeoutSpan.TotalMilliseconds) as Task<TResult>;
        }

        public static Task TimeoutAfter(this Task task, int millisecondsTimeout)
        {
            // Short-circuit #1: infinite timeout or task already completed
            if (task.IsCompleted || (millisecondsTimeout == Timeout.Infinite))
            {
                // Either the task has already completed or timeout will never occur.
                // No proxy necessary.
                return task;
            }

            // tcs.Task will be returned as a proxy to the caller
            TaskCompletionSource<VoidTypeStruct> tcs = new TaskCompletionSource<VoidTypeStruct>();

            // Short-circuit #2: zero timeout
            if (millisecondsTimeout == 0)
            {
                // We've already timed out.
                tcs.SetException(new TimeoutException());
                return tcs.Task;
            }

            // Set up a timer to complete after the specified timeout period
            Timer timer = new Timer(state =>
            {
                // Recover your state information
                // Fault our proxy with a TimeoutException
                ((TaskCompletionSource<VoidTypeStruct>)state).TrySetException(new TimeoutException());
            }, tcs, millisecondsTimeout, Timeout.Infinite);

            // Wire up the logic for what happens when source task completes
            task.ContinueWith((antecedent, state) =>
            {
                // Recover our state data
                var tuple = (Tuple<Timer, TaskCompletionSource<VoidTypeStruct>>)state;

                // Cancel the Timer
                tuple.Item1.Dispose();

                // Marshal results to proxy
                MarshalTaskResults(antecedent, tuple.Item2);
            }, Tuple.Create(timer, tcs), CancellationToken.None, TaskContinuationOptions.ExecuteSynchronously, TaskScheduler.Default);

            return tcs.Task;
        }

        public static Task TimeoutAfter<TResult>(this Task<TResult> task, int millisecondsTimeout)
        {
            // Short-circuit #1: infinite timeout or task already completed
            if (task.IsCompleted || (millisecondsTimeout == Timeout.Infinite))
            {
                // Either the task has already completed or timeout will never occur.
                // No proxy necessary.
                return task;
            }

            // tcs.Task will be returned as a proxy to the caller
            TaskCompletionSource<TResult> tcs = new TaskCompletionSource<TResult>();

            // Short-circuit #2: zero timeout
            if (millisecondsTimeout == 0)
            {
                // We've already timed out.
                tcs.SetException(new TimeoutException());
                return tcs.Task;
            }

            // Set up a timer to complete after the specified timeout period
            Timer timer = new Timer(state =>
            {
                // Recover your state information
                // Fault our proxy with a TimeoutException
                ((TaskCompletionSource<TResult>)state).TrySetException(new TimeoutException());
            }, tcs, millisecondsTimeout, Timeout.Infinite);

            // Wire up the logic for what happens when source task completes
            task.ContinueWith((antecedent, state) =>
            {
                // Recover our state data
                var tuple = (Tuple<Timer, TaskCompletionSource<TResult>>)state;

                // Cancel the Timer
                tuple.Item1.Dispose();

                // Marshal results to proxy
                MarshalTaskResults(antecedent, tuple.Item2);
            }, Tuple.Create(timer, tcs), CancellationToken.None, TaskContinuationOptions.ExecuteSynchronously, TaskScheduler.Default);

            return tcs.Task;
        }

        internal static void MarshalTaskResults<TResult>(Task source, TaskCompletionSource<TResult> proxy)
        {
            switch (source.Status)
            {
                case TaskStatus.Faulted:
                    proxy.TrySetException(source.Exception);
                    break;
                case TaskStatus.Canceled:
                    proxy.TrySetCanceled();
                    break;
                case TaskStatus.RanToCompletion:
                    Task<TResult> castedSource = source as Task<TResult>;
                    proxy.TrySetResult(
                        castedSource == null ? default(TResult) : // source is a Task
                            castedSource.Result);                 // source is a Task<TResult>
                    break;
            }
        }

        #endregion
Posted in Programming | Tagged , , , , | Leave a comment

The last chapter about our ex Scrum-Master

Remember the posts I wrote about our ex Scrum-Master, who was with us in the first month of this job? I thought I’d never hear about him again, but he turned up for the last chapter last week, so I thought I’d share it.

It turned out his decision to resign on the Monday after being notified of his suspension, without even waiting for the hearing, was not the smart choice I thought he realized needed to be done. Instead, it was a tactic that went along with his story that tried to make a case for constructive dismissal, which he took to the CCMA. He tried to get them to pay him the other 11 months of the 12 month contract.

The case came up last week on Tuesday, which happened to be the same day as Josh’s case, so I wasn’t able to be a witness against him. Actually I had a funny feeling about the way the case would turn out, and I turned out being correct. None of the witnesses against him even got to speak in the court.

But I’ll get to that soon enough. First, take a walk with me, backwards in time to the day he was notified of his suspension…

Imagine the surprise for our HR manager, after she notified him, to be subjected to the same bullshit as the rest of us! He tried to explain to her what it was like here, from his distorted and sadly confused point of view. And as I may have mentioned before, whenever he spoke about anything, somehow the explanation turned into a lecture and a lesson about Scrum. Always! The same shit every time; this drawing he did on the board, a particular way he explained the various stages of the Scrum process, and some odd personalization of the various roles to explain part of it, in his comical South African Indian accent… In explaining complexity and task planning… “I’ve done this before, I know how long it takes, Da da da”. When it went on for hours, day after day, it lost the funny part though, but I will never forget him. Those words and the way they were always followed by Da da da will remain with me forever.

Needless to say, it doesn’t take anyone very long to get the feeling that something is not quite right about the guy, so he did a bang-up job of letting HR know that he was indeed the problem here, dispelling any doubts she may have had.

I confess I don’t know what the court’s decision was, but he did exactly what I thought he would; so overconfident he is – he represented himself. The others don’t even know what he said… they could just see him through a glass door… speaking from 9AM until around 6PM, after they had already gone home.

Maybe he missed the school English classes where they studied a particular type of poem that he reminded me of since day one: A dramatic monologue. Too bad, because he did in court exactly what he did here; talked himself right into his grave. In a dramatic monologue, the speaker airs his often lofty opinions, runs others down, maybe tells a whole bunch of lies; the end result being he inadvertently reveals his temperament, his true intentions, his weaknesses… everything, and gains some new enemies in the process.

Well done, stupid. No witnesses are required when you talk yourself right out of your case. Anyway, though I don’t know the result, I do know that magistrates are not stupid. Nobody would let someone talk on for so long by himself unless he was doing a better job verbally crucifying himself than anybody else ever could.

I had felt empathy for him, until the last day. In the last half-hour before he left, he made a nasty comment about the other team members, who were not in the room. He started it off with something along the lines of “I hope nobody is sensitive around here” and then he gloated. Too bad – actually I am sensitive – I don’t watch sad movies because I get upset and then often I cry more than girls do. I’m not embarrassed about it. It’s who I am. But my sensitivity had made me sympathetic for him. I felt much better when he inadvertently informed me that he did not deserve any empathy.

So farewell, you fucking loon. I hope we never meet anytime soon. And da da fucking da.

Posted in Work | Tagged | Leave a comment

Reading PSD files with c#

Just about every image viewing application these days has support for Photoshop .psd files, so when I searched for code to read such files in c#, I was surprised that there isn’t a “standard” way of doing so.

The code described here was not written by myself. I did add a little to make the code easier for me to use, and will explain my changes further on.

The most reliable c# code I have found for this purpose was written by a Mr Frank Blumenberg. I’ve found various versions online, but they are all clearly copies of his code, whether credit is given to him or not.

I was interested only in the actual classes used, but if you want his latest code, which is a Paint.Net plugin, obviously especially useful if you use Paint.Net, then ignore the rest of this article, and just get his code.

My code is based on what I found on this page on StackOverflow. The very first comment to the question contains a link to an SVN repository which contains the code I used. It’s from an open source screenshot application.

But let’s take a step back and think about how one would use a PSD file parser in c#. There are only two scenarios, in my limited imagination. (And considering that I am not interested in writing to them, only reading them.)

  • To display the composite image. (If the file was saved with a composite image.)
  • To iterate the layers, build a collection of images, and do something with them. i.e. present the user with the option of exporting them to png or something to that affect.

Since I am lazy, I’m only really interested in the first option, and thus the optimizations I’ve made to the code are simply to allow quickly and easily loading the composite image. (Also I’m tired and writing this as well as sharing the code before I go to bed. If I take more than 5 minutes to write this post, I fear I shall write nonsense something along the lines of how to count sheep in c#.)

The code I’m sharing can also load the layers. I just didn’t add anything to that. (Read the code. It’s well-written and hence easy to read.) If I cared about the layers, I’d probably have added a collection of images and a way of having the class easily do the boiler-plate code job of loading all of them. (Maybe I’ll add that in a part 2 post. I don’t really plan these things – I just write what’s important to me and on my mind at the moment.)

Actually it’s probably a really bad idea for an application’s default handling of all .psd files to be to export the layers. That’s fine for simple little files that us programmers create, with maybe 6 or 7 layers. (I’m assuming most programmers who are also comfortable with Photoshop have usage patterns similar to mine. Forgive me if I am wrong.) But designers are a different breed. Creativity and artistic inspiration can lead to files with many layers, limited only by the memory on their machines. And they may have glorious names like Layer 67 Copy 14. You don’t want to be automatically exporting .png files and pissing your users off.

My additions to Frank’s awesome code

Whenever I load images, I use either File.ReadAllBytes or my own async equivalent (depending whether the code I’m calling from is synchronous or asynchronous), then create a temporary MemoryStream around the image bytes, and load the image from the temporary stream, to return a copy of the image and avoid the risk of locking the file for the duration that the image is displayed. My async implementation just leverages the Stream’s async methods that were added in .Net 4.5 and gets the benefits of asynchronous IO.

But the PSDFile class that I downloaded doesn’t support loading from a Stream. It only has a parameterless constructor, and a Load method that loads a file. Of course it’s Load implementation uses a FileStream internally though. So my first change was to refactor Load into two public methods, the second one simply allows passing in a Stream. That is, all I did was select the code inside the using block when it loaded a file stream, and extract a new method; then made that method public after changing the parameter type from FileStream to Stream.

Next, I don’t want to go to the trouble of instantiating a PSDFile instance every time, then call Load; then Photoshop.ImageDecoder.Decode(psdFile). That’s a lot of trouble just to load an image. The obvious next step was thus to add two constructors, one accepting a filename and the other a stream, and have them internally load the composite image, which is then populated in a new public property. So that’s exactly what I did.

In the spirit of OCP, I only added the two new constructors. Calling the default constructor does not follow my introduced behaviour, and then even calling Dispose is unnecessary because the (Bitmap property’s memory) allocation didn’t happen. (Technically I had to add all three constructors, because the code as is didn’t even declare one.)

Unfortunately, the calling code can’t be a single line as I originally intended, because adding a public Image property (Image is disposable) necessitates implementing IDisposable, but it’s not too bad. Calling the code now is as simple as this:

 using (var psdFile = new PsdFile(filename))
 {
     image = psdFile.CompositImage.Clone() as Bitmap;
 }

The reason I call Clone is simply that my Dispose implementation destroys the image. I also followed the practice of declaring the CompositImage property as an Image (the base class for a Bitmap), even though I implemented it with a BitmapA best practice that comes naturally to me by now, yet I see it frequently forgotten in practice.

Unfortunately, the composite image that this code loads is not transparent.

The code downloads… I didn’t write a small application that just demos this code, but it is built in to my RomyView application. (Sorry, that download is getting rather large.) I also zipped the PSD parsing code. (Ok, so I didn’t create a small demo that uses it, but if you can’t figure out how to call the code yourself, you shouldn’t be a programmer. You should be a donkey.)

RomyView

Psd File Parser

And for interest sake, the code I use for loading all the image types I support in my application is now this:

using Photoshop;
using System;
using System.Drawing;
using System.IO;
using System.Threading.Tasks;

namespace Romy.Core
{
    public static class ImageFile
    {
        /// <summary>Opens and returns an in-memory copy of the specified image, so as not to lock the file.</summary>
        public static Bitmap OpenBitmap(string filename)
        {
            Bitmap image = null;

            var disposeImageOnError = new Action(() =>
            {
                if (image != null)
                {
                    image.Dispose();
                    image = null;
                }
            });

            try
            {
                if (string.Compare(Path.GetExtension(filename), ".gif", StringComparison.InvariantCultureIgnoreCase) == 0)
                    image = Image.FromFile(filename) as Bitmap;
                else
                {
                    using (var stream = new MemoryStream(File.ReadAllBytes(filename)))
                    {
                        if (string.Compare(Path.GetExtension(filename), ".psd", StringComparison.InvariantCultureIgnoreCase) != 0)
                            image = new Bitmap(stream);
                        else
                        {
                            using (var psdFile = new PsdFile(stream))
                            {
                                image = psdFile.CompositImage.Clone() as Bitmap;
                            }
                        }
                    }
                }
            }
            catch (IOException) { disposeImageOnError(); }
            catch (UnauthorizedAccessException) { disposeImageOnError(); }
            catch (Exception ex)
            {
                ex.Log();
                disposeImageOnError();
            }

            return image;
        }

        /// <summary>Opens and returns a Bitmap using asynchronous IO, using the Task-based Asynchronous Pattern (TAP) pattern.
        public static async Task<Bitmap> OpenBitmapAsync(string filename)
        {
            Bitmap image = null;

            try
            {
                if (string.Compare(Path.GetExtension(filename), ".gif", StringComparison.InvariantCultureIgnoreCase) == 0)
                    image = Image.FromFile(filename) as Bitmap;
                else
                {
                    using (var memStream = new MemoryStream(await FileAsync.ReadAllBytesAsync(filename, Constants.LargeBufferSize)))
                    {
                        if (memStream.Length > 0) // My ReadAllBytesAsync method can return an empty array. Via Enumerable.Empty<byte>().ToArray()
                        {
                            if (string.Compare(Path.GetExtension(filename), ".psd", StringComparison.InvariantCultureIgnoreCase) != 0)
                                image = new Bitmap(memStream);
                            else
                            {
                                using (var psdFile = new PsdFile(memStream))
                                {
                                    image = psdFile.CompositImage.Clone() as Bitmap;
                                }
                            }
                        }
                    }
                }
            }
            catch (Exception ex) { ex.Log(); }

            return image;
        }
    }
}
Posted in Programming | Tagged | 3 Comments

Weirdness in Photoshop

I have been using Photoshop since a version so many years ago, I forget which one, but my usage in the past was mostly touching up actual photos, for which I used mostly the cloning brush, and maybe the burn or dodge tool occasionally. And believe it or not, those tools were enough for me to touch up just about any photo, even when I had to recreate areas of a photograph that had been torn away years before.

So designing objects like the header image in here (as well as buttons and so on in my previous job while the actual designer was on maternity leave) is quite a departure for me.

Photoshop is an excellent tool, but one thing that gets to me is that they seem to have made some drastic change regarding smart objects in the last couple of versions. In my last job, I designed the header image above using Photoshop CS4. Having grouped a few layers together as a “smart object”, I applied some lighting affects, and they became a smart filter. But the way the smart filter is applied seems to have changed completely in Photoshop CS6. I found a workaround for it, but can’t for the life of me remember what it is.

All I can say is, if you don’t want to suffer hours of frustration, do not apply lighting affects to smart objects. You will be sorry if you do. Really.

Posted in Design | Tagged | Leave a comment

Putting it all together – Configurable TaskSchedulers that can be changed by the user at runtime

This will be my last post about TaskSchedulers and TaskFactories. It will be easier to understand this post if you have read my previous entries on implementing a custom TaskScheduler, but that is not a requirement, as all the relevant code will be included here.

Again, the source code for the whole solution is here.

Using what I have learned recently with regard to TaskSchedulers and TaskFactories, I decided to go one step further, and create a couple of different schedulers that can be changed at runtime by the user.

I’ll do this backwards for a change… Below is the end result of today’s post. (The part the user can see; that is the UI to configure the schedulers.)

I’m not going to explain either how I connected the settings with a property grid again, or how the context menu allows resetting the connected item’s properties, but if you haven’t read my two posts on that topic, you can find them here: Part 1 and Part 2.TaskUserSettings

I’ve added a new class diagram to the solution because I think it makes this code easier to visualize. Here it is:

TaskSchedulers

I took what I have learned in writing a custom TaskScheduler type, and defined an abstract class that can be used to derive custom schedulers that schedule their tasks to be run in parallel. The IParallelInvoker interface is unnecessary, but it helps clarify that the important method is the abstract ParallelInvoke method on the base class. Derived classes must then implement it to define how their tasks will be run in parallel.

I have derived two scheduler types.

  • The ParallelPoolThreadsTaskScheduler runs its tasks in parallel on the managed ThreadPool.
  • The ParallelDedicatedThreadsTaskScheduler creates Thread instances, and can be configured to either run them in the foreground or in the background.

I then created a custom TaskFactory class, the ParallelTaskFactory, with two properties introduced, UseThreadPool, to set which scheduler will be returned by the Scheduler property, and UseForegroundThreads, to configure whether the ParallelDedicatedThreadsTaskScheduler instance runs its threads in the foreground or background.

Note that the TaskFactory.Scheduler property is not virtual, so it was necessary to use the new keyword to hide (and reimplement) the inherited property.

Here’s the code for the base scheduler type. (It will look very familiar if you’ve read my other posts on custom schedulers.)

using System.Collections.Generic;
using System.Linq;

namespace System.Threading.Tasks.Schedulers
{
    /// <summary>Base class for a TaskScheduler that, when processing work items, 
    /// groups them into batches, and processes the Tasks of each batch in parallel. 
    /// The <see cref="ParallelInvoke"/> method is abstract so that derived classes 
    /// must specify how they run the Tasks in parallel.</summary>
    public abstract class ParallelTaskSchedulerBase : TaskScheduler, IParallelInvoker
    {
        #region Fields

        [ThreadStatic]
        private static bool currentThreadIsProcessingItems;

        private int maxDegreeOfParallelism;

        private volatile int runningOrQueuedCount;

        private readonly LinkedList<Task> tasks = new LinkedList<Task>();

        #endregion Fields

        #region Constructors

        protected ParallelTaskSchedulerBase(int maxDegreeOfParallelism)
        {
            if (maxDegreeOfParallelism < 1)
                throw new ArgumentOutOfRangeException("maxDegreeOfParallelism");

            this.maxDegreeOfParallelism = maxDegreeOfParallelism;
        }

        protected ParallelTaskSchedulerBase() : this(Environment.ProcessorCount) { }

        #endregion Constructors

        #region Properties

        public override int MaximumConcurrencyLevel
        {
            get { return maxDegreeOfParallelism; }
        }

        #endregion Properties

        #region Methods

        #region Public Methods

        public abstract void ParallelInvoke(Action<Task> action, params Task[] tasks);

        #endregion Public Methods

        #region Protected Methods

        protected override bool TryDequeue(Task task)
        {
            lock (tasks) return tasks.Remove(task);
        }

        protected override bool TryExecuteTaskInline(Task task,
            bool taskWasPreviouslyQueued)
        {
            if (!currentThreadIsProcessingItems) return false;

            if (taskWasPreviouslyQueued) TryDequeue(task);

            return base.TryExecuteTask(task);
        }

        protected override IEnumerable<Task> GetScheduledTasks()
        {
            var lockTaken = false;
            try
            {
                Monitor.TryEnter(tasks, ref lockTaken);
                if (lockTaken) return tasks.ToArray();
                else throw new NotSupportedException();
            }
            finally { if (lockTaken) Monitor.Exit(tasks); }
        }

        protected override void QueueTask(Task task)
        {
            lock (tasks) tasks.AddLast(task);

            if (runningOrQueuedCount < maxDegreeOfParallelism)
            {
                runningOrQueuedCount++;
                RunTasks();
            }
        }

        #endregion Protected Methods

        #region Private Methods

        private void RunTasks()
        {
            List<Task> taskList = new List<Task>();

            currentThreadIsProcessingItems = true;
            try
            {
                while (true)
                {
                    lock (tasks)
                    {
                        if (tasks.Count == 0)
                        {
                            runningOrQueuedCount--;
                            break;
                        }

                        var t = tasks.First.Value;
                        taskList.Add(t);
                        tasks.RemoveFirst();
                    }
                }

                if (taskList.Count > 0)
                {
                    var batches = taskList.GroupBy(
                        task => taskList.IndexOf(task) / maxDegreeOfParallelism);

                    foreach (var batch in batches)
                    {
                        ParallelInvoke(t => base.TryExecuteTask(t), batch.ToArray());
                    }
                }
            }
            finally { currentThreadIsProcessingItems = false; }
        }

        #endregion Private Methods

        #endregion Methods
    }
}

Here are the two derived custom schedulers:

using System.Collections.Generic;

namespace System.Threading.Tasks.Schedulers
{
    /// <summary>A TaskScheduler that, when processing work items, groups 
    /// them into batches, and processes the Tasks of each batch in parallel. 
    /// This <b>ParallelInvoke</b> implementation runs the work on background 
    /// threads, using the THreadPool.</summary>
    public sealed class ParallelPoolThreadsTaskScheduler : ParallelTaskSchedulerBase
    {
        /// <summary>Invoke the specified Tasks, executing the specified 
        /// Action for each on the ThreadPool in parallel.</summary>
        public override void ParallelInvoke(Action<Task> action, params Task[] tasks)
        {
            if (tasks == null) throw new ArgumentNullException("tasks");
            if (action == null) throw new ArgumentNullException("action");
            if (tasks.Length == 0) return;

            using (ManualResetEvent mre = new ManualResetEvent(false))
            {
                int remaining = tasks.Length;
                var exceptions = new List<Exception>();

                foreach (var task in tasks)
                {
                    ThreadPool.UnsafeQueueUserWorkItem(_ =>
                    {
                        try
                        {
                            action(task);
                        }
                        catch (Exception ex)
                        {
                            lock (exceptions) exceptions.Add(ex);
                        }
                        finally
                        {
                            if (Interlocked.Decrement(ref remaining) == 0) mre.Set();
                        }
                    }, null);
                }
                mre.WaitOne();
                if (exceptions.Count > 0) throw new AggregateException(exceptions);
            }
        }
    }
}
using System.Collections.Generic;

namespace System.Threading.Tasks.Schedulers
{
    /// <summary>A TaskScheduler that, when processing work items, groups 
    /// them into batches, and processes the Tasks of each batch in parallel. 
    /// This <b>ParallelInvoke</b> implementation runs the work on dedicated 
    /// threads, and can be configured to use foreground threads.</summary>
    public sealed class ParallelDedicatedThreadsTaskScheduler : ParallelTaskSchedulerBase
    {
        private bool useForegroundThreads;

        public bool UseForegroundThreads
        {
            get { return useForegroundThreads; }
            set { useForegroundThreads = value; }
        }

        /// <summary>Invoke the specified Tasks, executing the specified 
        /// Action for each on a dedicated thread in parallel.</summary>
        public override void ParallelInvoke(Action<Task> action, params Task[] tasks)
        {
            var exceptions = new List<Exception>();

            List<Thread> threads = new List<Thread>();

            foreach (var task in tasks)
            {
                var thread = new Thread(t =>
                {
                    try
                    {
                        action(task);
                    }
                    catch (Exception ex)
                    {
                        lock (exceptions) exceptions.Add(ex);
                    }
                }) { IsBackground = !UseForegroundThreads };
                thread.Start();
                threads.Add(thread);
            }

            foreach (var t in threads)
                t.Join();

            if (exceptions.Count > 0) throw new AggregateException(exceptions);
        }
    }
}

Notice the pattern used for exceptions. I took this from a MSDN Magazine article by Stephen Toub. It is very important to stick to this pattern, especially if you write libraries and other developers will end up using you code. That is, when you have several Tasks running, they should capture any exceptions in an AggregateException, so that one faulting Task doesn’t break the rest, and the calling code can choose to handle (and possibly suppress) the errors just by adding an exception handler and handling any AggregateException.

And lastly, here is the code for my static Tasks class, and my custom TaskFactory, which are both in one file. Note that I had declared all the scheduler types in the code as TaskScheduler types originally (I figured it makes better polymorphic sense), but changed them a few minutes ago because declaring them using the actual types shows the relationships properly on the class diagram.

using System;
using System.Threading;
using System.Threading.Tasks;
using System.Threading.Tasks.Schedulers;

namespace Romy.Core
{
    /// <summary>Set up some common options for async Tasks. The <b>TaskSchedulers.Parallel</b>
    /// scheduler references the static <b>Scheduler</b> property of this class. The value of
    /// <b>Factory.UseForeGroundThreads</b> defines which <B>ParallelTaskSchedulerBase</B> 
    /// implementation is used, and whether the underlying threads will be foreground threads 
    /// or background TheadPool threads.</summary>
    public static class Tasks
    {
        private static ParallelTaskFactory factory = new ParallelTaskFactory();

        public static ParallelTaskFactory Factory
        {
            get { return Tasks.factory; }
        }

        public static TaskScheduler Scheduler
        {
            get { return factory.Scheduler; }
        }

        public static Task Run(Func<Task> body)
        {
            return factory.StartNew(async () => await body()).Unwrap();
        }
    }

    /// <summary>A TaskFactory to define which TaskScheduler will 
    /// be used for most async Tasks in the application.</summary>
    public class ParallelTaskFactory : TaskFactory
    {
        private bool useForeGroundThreads;

        private bool useThreadPool;

        private static Lazy<ParallelDedicatedThreadsTaskScheduler> dedicatedThreadsScheduler = new Lazy<ParallelDedicatedThreadsTaskScheduler>(() => new ParallelDedicatedThreadsTaskScheduler());

        private static Lazy<ParallelPoolThreadsTaskScheduler> threadPoolScheduler = new Lazy<ParallelPoolThreadsTaskScheduler>(() => new ParallelPoolThreadsTaskScheduler());

        public ParallelTaskFactory()
            : base(CancellationToken.None,
                TaskCreationOptions.DenyChildAttach | TaskCreationOptions.HideScheduler,
                TaskContinuationOptions.ExecuteSynchronously, ThreadPoolScheduler) { }

        public bool UseForeGroundThreads
        {
            get { return useForeGroundThreads; }
            set
            {
                useForeGroundThreads = value;
                DedicatedThreadsScheduler.UseForegroundThreads = useForeGroundThreads;
            }
        }

        public bool UseThreadPool
        {
            get { return useThreadPool; }
            set { useThreadPool = value; }
        }

        public static ParallelDedicatedThreadsTaskScheduler DedicatedThreadsScheduler
        {
            get { return ParallelTaskFactory.dedicatedThreadsScheduler.Value; }
        }

        public new TaskScheduler Scheduler
        {
            get { return useThreadPool ? (TaskScheduler)ThreadPoolScheduler : (TaskScheduler)DedicatedThreadsScheduler; }
        }

        public static ParallelPoolThreadsTaskScheduler ThreadPoolScheduler
        {
            get { return ParallelTaskFactory.threadPoolScheduler.Value; }
        }
    }
}

I won’t include the user settings code, because it should be fairly obvious how to hook that up. (Just add two boolean type user settings, and change the values of the two relevant properties on the global TaskFactory when the user settings change. This then ripples through to setting/configuring the scheduler.)

As far as the performance of the code goes, I can’t see a difference. (I have not timed them.) Both schedulers work better than the default scheduler in my application, and I tend to stick with the one that uses the ThreadPool, which is in theory more resource-friendly. (It doesn’t have the overhead of creating threads all the time.)


If anyone does download the application and would like to compare the performance of the schedulers, the best place to do so (and also the only place where you can visually see the results of the schedulers directly) is when populating thumbnails. (Thumbnails that are not already cached by my application, and not in the Windows thumbnail cache.)

To clear my application’s cache, just run the ClearCache.cmd batch file from the root of the source directory.

To clear the Windows thumbnail cache, you have to run the CleanMgr application. It allows you to save sets of different options that it can run, called sage sets. I always have one set just to delete the Windows thumbnails, which I set up by running CleanMgr /sageset:2 and run by running CleanMgr /sagerun:2.

Having cleared the cached thumbnails, the results of the schedulers can be seen clearly when navigating directories with many JPG images or movie files, since the thumbnails are initially drawn with placeholders before their images are loaded.

Posted in Programming | Tagged , , , , , | 1 Comment

Diving deeper into the TaskScheduler work item pool

Two days ago, I spent my hours of insomnia thinking about the question: Just how parallel is my ParallelTaskScheduler custom TaskScheduler implementation, really?

It divides its work items into batches (of Task objects), and then process each batch’s items in parallel, on a single ThreadPool thread… But, thinking of that a little further, I know that a call to Parallel.ForEach accepts an optional ParallelOptions parameter. One of the properties that can be set on ParallelOptions is a TaskScheduler. I don’t pass it one, but that’s not the point. I can safely assume then that it is scheduled by TaskScheduler.Default. Although I haven’t seen its source code, I know how I would have implemented a default scheduler. One optimization I would have made was that if a Task got queued from a ThreadPool thread, since it is already a background thread, I’d run it inline. In fact, I’m pretty sure that it does just that.

So what does that mean for my ParallelTaskScheduler? Nothing good, I’m afraid. If the calls run inline, I could just as well have written them in an ordinary for or foreach loop. This is actually a considerable deviation  from the behaviour I intended. I have improved it, and I will include that code at the end, but first, I’ll write about my futile exercise yesterday implementing a TaskScheduler using Windows operating system fibers.

An experiment; implementing a custom TaskScheduler using Fibers

Fibers are also often referred to as lightweight threads, although they aren’t threads at all. They are units of execution with a stack and set of registers, all running on a single thread, that you schedule yourself manually by switching between them.

They do seem to be very popular, and besides, they are not implemented in .Net (but are to be found in the unmanaged API), so I thought it would be a fun exercise to implement Fibers in C# by platform invoking the unmanaged functions.

Actually it was a waste of a day. My FiberPerTaskScheduler runs nicely, but for a reason beyond my control, unmanaged Fibers should not be used in .Net. I’ll explain why further on, but for interest, here is the code anyway.

First off, here is my very simple managed Fiber wrapper. (It’s very simple because I do no scheduling from this code. If you search for Fiber implementations on the web, you’re likely to find coroutine schedulers written from scratch. I didn’t need to do that, because I’m calling this from a TaskScheduler, which does the scheduling already. All this does is the bare minimum to run a number of Tasks on Fibers.)

using System;
using System.Collections.Generic;
using System.Linq;
using System.Runtime.InteropServices;
using System.Threading;
using System.Threading.Tasks;

namespace Romy.Core
{
    public static class ManagedFiberWrapper
    {
        public static void InvokeFibers(IEnumerable<Task> tasks, Action<Task> action)
        {
            int remaining = tasks.Count();
            var exceptions = new List<Exception>();

            using (ManualResetEvent mre = new ManualResetEvent(false))
            {
                var primaryFiber = new Thread(() =>
                {
                    uint primaryFiberId = NativeMethods.ConvertThreadToFiber(0);

                    if (primaryFiberId != 0)
                    {
                        try
                        {
                            List<uint> fiberIds = new List<uint>();

                            foreach (var task in tasks)
                            {
                                try
                                {
                                    uint fiberId = 0;

                                    fiberIds.Add(fiberId = NativeMethods.CreateFiber(0, lParam => action(task), 0));

                                    NativeMethods.SwitchToFiber(fiberId);
                                    NativeMethods.DeleteFiber(fiberId);
                                }
                                catch (Exception exc)
                                {
                                    lock (exceptions) exceptions.Add(exc);
                                }
                                finally
                                {
                                    if (Interlocked.Decrement(ref remaining) == 0) mre.Set();
                                }
                            }
                        }
                        finally { NativeMethods.DeleteFiber(primaryFiberId); }
                    }
                });

                primaryFiber.Start();
            }

            if (exceptions.Count > 0) throw new AggregateException(exceptions);
        }

        static class NativeMethods
        {
            public delegate void FiberProc(uint param);

            [DllImport("Kernel32.dll")]
            public static extern uint ConvertThreadToFiber(uint lpParameter);

            [DllImport("Kernel32.dll")]
            public static extern uint CreateFiber(uint dwStackSize, FiberProc lpStartAddress, uint lpParameter);

            [DllImport("Kernel32.dll")]
            public static extern void DeleteFiber(uint lpFiber);

            [DllImport("Kernel32.dll")]
            public static extern void SwitchToFiber(uint lpFiber);
        }
    }
}

And here is my FiberPerTaskScheduler that runs the above code.

using Romy.Core;
using System.Collections.Generic;
using System.Linq;

namespace System.Threading.Tasks.Schedulers
{
    /// <summary>Custom TaskScheduler that processes work items in batches, 
    /// with each work item in the batch processed by a Fiber.</summary>
    public class FiberPerTaskScheduler : TaskScheduler
    {
        [ThreadStatic]
        private static bool currentThreadIsProcessingItems;

        private int maxDegreeOfParallelism;

        private volatile int runningOrQueuedCount;

        private readonly LinkedList<Task> tasks = new LinkedList<Task>();

        public FiberPerTaskScheduler(int maxDegreeOfParallelism)
        {
            if (maxDegreeOfParallelism < 1)
                throw new ArgumentOutOfRangeException("maxDegreeOfParallelism");

            this.maxDegreeOfParallelism = maxDegreeOfParallelism;
        }

        public FiberPerTaskScheduler() : this(Environment.ProcessorCount) { }

        public override int MaximumConcurrencyLevel
        {
            get { return maxDegreeOfParallelism; }
        }

        protected override bool TryDequeue(Task task)
        {
            lock (tasks) return tasks.Remove(task);
        }

        protected override bool TryExecuteTaskInline(Task task,
            bool taskWasPreviouslyQueued)
        {
            if (!currentThreadIsProcessingItems) return false;

            if (taskWasPreviouslyQueued) TryDequeue(task);

            return base.TryExecuteTask(task);
        }

        protected override IEnumerable<Task> GetScheduledTasks()
        {
            var lockTaken = false;
            try
            {
                Monitor.TryEnter(tasks, ref lockTaken);
                if (lockTaken) return tasks.ToArray();
                else throw new NotSupportedException();
            }
            finally { if (lockTaken) Monitor.Exit(tasks); }
        }

        protected override void QueueTask(Task task)
        {
            lock (tasks) tasks.AddLast(task);

            if (runningOrQueuedCount < maxDegreeOfParallelism)
            {
                runningOrQueuedCount++;
                RunTasks();
            }
        }

        private void RunTasks()
        {
            List<Task> taskList = new List<Task>();

            currentThreadIsProcessingItems = true;
            try
            {
                while (true)
                {
                    lock (tasks)
                    {
                        if (tasks.Count == 0)
                        {
                            runningOrQueuedCount--;
                            break;
                        }

                        var t = tasks.First.Value;
                        taskList.Add(t);
                        tasks.RemoveFirst();
                    }
                }

                if (taskList.Count > 0)
                {
                    var batches = taskList.GroupBy(
                        task => taskList.IndexOf(task) / maxDegreeOfParallelism);

                    foreach (var batch in batches)
                    {
                        ManagedFiberWrapper.InvokeFibers(batch, t => base.TryExecuteTask(t));
                    }
                }
            }
            finally { currentThreadIsProcessingItems = false; }
        }
    }
}

Even though the above code works well, there is one problem: If an exception is thrown in the Action executed on a Fiber, and I don’t mean an unhandled exception, any exception at all, the managed exception handler fails to catch the exception. Not only does it fail, it throws an unhandled StackOverflowException in “module unknown”, and brings down the whole process in a blaze of… well, nothing.

The issue is apparently related to TLS (Thread Local Storage), which is heavily used by both Fibers and the managed Exception handlers, and has something to do with the fact that all the Fibers are running on the same thread.

Since I make heavy use of Cancellation in my Tasks, my old friend OperationCancelledException is not welcome in the Fiber family home.

That sucks, almost as much as the fact that I spent most of the day yesterday reading about Fibers, and was initially chuffed when my managed wrapper implementation of them seemed to work perfectly straight away.

I have seen some managed Fiber implementations using C# iterators, but I don’t like the code, so I will not use such a pattern myself.

And thus, I fixed my ParallelTaskScheduler implementation so that it now really does invoke its Tasks in parallel. It is now very much faster than it was.

The improved ParallelTaskScheduler

using System.Collections.Generic;
using System.Linq;

namespace System.Threading.Tasks.Schedulers
{
    /// <summary>Custom TaskScheduler that processes work items in batches, with the 
    /// work items of each batch processed by a ThreadPool thread, in parallel.</summary>
    public class ParallelTaskScheduler : TaskScheduler
    {
        [ThreadStatic]
        private static bool currentThreadIsProcessingItems;

        private int maxDegreeOfParallelism;

        private volatile int runningOrQueuedCount;

        private readonly LinkedList<Task> tasks = new LinkedList<Task>();

        public ParallelTaskScheduler(int maxDegreeOfParallelism)
        {
            if (maxDegreeOfParallelism < 1)
                throw new ArgumentOutOfRangeException("maxDegreeOfParallelism");

            this.maxDegreeOfParallelism = maxDegreeOfParallelism;
        }

        public ParallelTaskScheduler() : this(Environment.ProcessorCount) { }

        public override int MaximumConcurrencyLevel
        {
            get { return maxDegreeOfParallelism; }
        }

        public static void ParallelInvoke(Action<Task> action, params Task[] tasks)
        {
            if (tasks == null) throw new ArgumentNullException("tasks");
            if (action == null) throw new ArgumentNullException("action");
            if (tasks.Length == 0) return;

            using (ManualResetEvent mre = new ManualResetEvent(false))
            {
                int remaining = tasks.Length;
                var exceptions = new List<Exception>();

                foreach (var task in tasks)
                {
                    ThreadPool.UnsafeQueueUserWorkItem(_ =>
                    {
                        try
                        {
                            action(task);
                        }
                        catch (Exception ex)
                        {
                            lock (exceptions) exceptions.Add(ex);
                        }
                        finally
                        {
                            if (Interlocked.Decrement(ref remaining) == 0) mre.Set();
                        }
                    }, null);
                }
                mre.WaitOne();
                if (exceptions.Count > 0) throw new AggregateException(exceptions);
            }
        }

        protected override bool TryDequeue(Task task)
        {
            lock (tasks) return tasks.Remove(task);
        }

        protected override bool TryExecuteTaskInline(Task task,
            bool taskWasPreviouslyQueued)
        {
            if (!currentThreadIsProcessingItems) return false;

            if (taskWasPreviouslyQueued) TryDequeue(task);

            return base.TryExecuteTask(task);
        }

        protected override IEnumerable<Task> GetScheduledTasks()
        {
            var lockTaken = false;
            try
            {
                Monitor.TryEnter(tasks, ref lockTaken);
                if (lockTaken) return tasks.ToArray();
                else throw new NotSupportedException();
            }
            finally { if (lockTaken) Monitor.Exit(tasks); }
        }

        protected override void QueueTask(Task task)
        {
            lock (tasks) tasks.AddLast(task);

            if (runningOrQueuedCount < maxDegreeOfParallelism)
            {
                runningOrQueuedCount++;
                RunTasks();
            }
        }

        private void RunTasks()
        {
            List<Task> taskList = new List<Task>();

            currentThreadIsProcessingItems = true;
            try
            {
                while (true)
                {
                    lock (tasks)
                    {
                        if (tasks.Count == 0)
                        {
                            runningOrQueuedCount--;
                            break;
                        }

                        var t = tasks.First.Value;
                        taskList.Add(t);
                        tasks.RemoveFirst();
                    }
                }

                if (taskList.Count > 0)
                {
                    var batches = taskList.GroupBy(
                        task => taskList.IndexOf(task) / maxDegreeOfParallelism);

                    foreach (var batch in batches)
                    {
                        ParallelInvoke(t => base.TryExecuteTask(t), batch.ToArray());
                    }
                }
            }
            finally { currentThreadIsProcessingItems = false; }
        }
    }
}
Posted in Programming | Tagged , , , , | Leave a comment

A sneaky little critter of a bug and I forget to use common sense to debug

Last night, I decided to put my code of late (the custom TaskScheduler and the pattern I’ve come up with using several TaskScheduler implementations) through some more rigorous testing.

Imagine my surprise when a nasty bug showed up right away… I happened to open Process Explorer and had a look at the thread count while running the application. Disaster! Every time I traversed a directory, or even refreshed the current one, new threads would be added, and they would never be removed.

This is roundabout where I lost my common sense.

Common sense says: If your code is spinning up hundreds of threads, open the Threads debug window, and look at what code they are running.

So what do I do? I jump to conclusions… Panicking, thinking “Oh no. I have written two articles about TaskSchedulers and managing your async threads, and published unspeakably stupid code. Oh, woe is me!”

Half an hour later, after going down completely the wrong road and trying to fix code that was never broken, I finally looked at the Threads window.

All of the rogue threads were in WorkStealingTaskScheduler.DispatchLoop.

It turns out my code that updates the status strip, which is called in several places, including when refreshing the directory, called Task.Factory.StartNew with an overload that accepts a TaskScheduler parameter and created a new instance of a WorkStealingTaskScheduler each time. While there is no recommendation that says you must not create multiple instances of the same scheduler type, common sense does dictate that if you should decide to do so, at least understand what the code does. (Usually if the framework expects you only to create one instance of some type, there will be a clue in the pattern in which it is consumed. For example, ToolStripManager.Renderer in Windows Forms indicates that if you create a custom renderer, you should set it to replace the global default renderer, which implies there should only be a single instance.)

WorkStealingTaskScheduler is initialized with a maxConcurrencyLevel parameter, which defaults to Environment.ProcessorCount * 2. It then spins up that number of threads (i.e. instances of the Thread type), which will wait for work to process in the DispatchLoop method. Thus every time the files were refreshed, a new scheduler would be added, adding another Environment.ProcessorCount * 2 threads. The scheduler would never be used again, but would also never be garbage collected. (I don’t know enough about garbage collection to know if a TaskScheduler would be cleaned up otherwise, but I’m guessing it would.)

I was deliberately creating new instances, because of some paranoid fear that an instance of the scheduler might be too busy, and I want status info always updated immediately, regardless of how busy the application is with other tasks. But thinking about it, this design didn’t make sense. If all your threads are busy, or worse, deadlocked, it of course indicates that something else is wrong.

Anyway, the fix was obviously not to create multiple schedulers of this type. I did the same for other types as well, even my own ParallelTaskScheduler, just in case.

Posted in Programming | Tagged , , | Leave a comment

More thoughts on TaskSchedulers and TaskFactories

I would never have expected it, but it seems that in my application, the custom TaskScheduler I came up with the other day actually performs so well that I now use it as the default in most places. That is, I use a couple of schedulers where I need finer-grained control of the work being scheduled. Of course controlling the Task scheduling makes more sense in some places than others, and perhaps is not needed in many applications.

Since my software is shared, the point of this post is to try explaining how I have used the various TaskShedulers in the shared code, and how it has worked for me. I’m by no means an expert, but having started playing with async code around the 2nd Async CTP, I hope that my lessons learned may be of value to others.

Get the latest code here.

I wrote last time that you probably won’t normally need to use anything besides the default TaskScheduler, but if you find yourself in the situation where you have several concurrent asynchronous operations running, and you need to define explicitly what order they run in, or even if you occasionally need to pre-empt one already running asynchronous operation with another, then it makes sense to use different schedulers for different kinds of Tasks.

What can you get out of this?
Hopefully, it should prove to be a useful example of how different TaskSchedulers can be used for scheduling different work items concurrently

I’m using a few of the sample schedulers from the Microsoft sample code. That is:

  • QueuedTaskScheduler
  • WorkStrealingTaskScheduler
  • StaTaskScheduler
  • and my own ParallelTaskScheduler
  • Almost forgot… A UI-Synchronized TaskScheduler, via TaskScheduler.FromCurrentSynchronizationContext

The QueuedTaskScheduler runs on top of another scheduler, allowing you to then add priority queues. Since much of my application’s async work is to build thumbnails, I created three priority queues on top of my own ParallelTaskScheduler. (although I normally only use two of the queues)

I’ve written about this before, so won’t go into too much detail here, but the the main thing is, thumbnails are added in batches. I found that when using my own scheduler for this, it worked well, but the newest added batches somehow got higher priority than those already being added. (When adding thumbnails in several batches for hundreds or thousands of files at a time, this caused thumbnails added later, only visible if you scroll down in the application, to load their images before those at the top.)

It so happened that using a higher priority queue for thumbnails based on their “Showing” (where Showing is a property on the thumbnail control that returns true only when the control is both visible and its bounds are within the visible portion of the screen), solved the issue. That is, I thought it would not be enough (because only the first batch being added should be showing) but it worked out that it always causes all the controls being added to fetch their images and update in the correct order.

What happens now in the application when a user traverses a directory is:

  • Thumbnails are added in batches via the UI-synchronized scheduler. As the batches are added, each one’s Path property is set, which will cause the asynchronous loading of its image.
  • Setting the Path so soon would cause an issue if the thumbnails were added synchronously: They would not paint anything initially because they would already be loading their images. Adding using the UI-synchronized scheduler actually slows things down, but also allows them to paint placeholder images before they load their images. (unless the images are already cached)
  • The images are loaded in parallel batches in more or less the order most wanted. This is done efficiently. That is, it performs fairly well on my home machine, with only 1.5GB RAM and 2 processor cores, and absolutely flies on my work machine, with 8 cores and 8GB RAM.
  • In the meantime, if a user selects one or more files, a WorkStealingTaskScheduler is used to update the status strip with info on the selection. (Or a StaTaskScheduler if they clicked a video, because a call is using DirectShowLib.) Actually it probably doesn’t really matter what scheduler is used for this, as long as it is not the same one adding thumbnails. (which would leave the status strip without an update until after all the thumbnails finish loading their images.)

To help make the various schedulers accessible (except for the UI-synchronized one, which is inherited from my base Form), I created a class that contains properties of the various scheduler types, like so:

namespace System.Threading.Tasks.Schedulers
{
    public class TaskSchedulers : IDisposable
    {
        #region Fields

        private static volatile bool cleanupScheduled;

        private bool disposed;

        private QueuedTaskScheduler queued;

        private TaskScheduler highPriority;

        private TaskScheduler lowerPriority;

        private TaskScheduler normalPriority;

        private TaskScheduler sta = new StaTaskScheduler(Environment.ProcessorCount);

        private TaskScheduler workStealing = new WorkStealingTaskScheduler();

        private static readonly TaskSchedulers schedulers = new TaskSchedulers();

        #endregion Fields

        #region Constructors

        ~TaskSchedulers()
        {
            Dispose(false);
        }

        public TaskSchedulers()
        {
            queued = new QueuedTaskScheduler(Parallel, Environment.ProcessorCount);
            normalPriority = queued.ActivateNewQueue(1);
            highPriority = queued.ActivateNewQueue(0);
            lowerPriority = queued.ActivateNewQueue(2);

            if (!cleanupScheduled)
            {
                cleanupScheduled = true;
                System.Windows.Forms.Application.ApplicationExit += (sender, e) => schedulers.Dispose();
            }
        }

        #endregion Constructors

        #region Properties

        public TaskScheduler HighPriority
        {
            get { return highPriority; }
        }

        public TaskScheduler LowerPriority
        {
            get { return lowerPriority; }
        }

        public TaskScheduler NormalPriority
        {
            get { return normalPriority; }
        }

        public static TaskScheduler Parallel
        {
            get { return Romy.Core.Tasks.Scheduler; }
        }

        public TaskScheduler Queued
        {
            get { return queued; }
        }

        public TaskScheduler Sta
        {
            get { return sta; }
        }

        public TaskScheduler WorkStealing
        {
            get { return workStealing; }
        }

        public static TaskSchedulers Schedulers
        {
            get { return TaskSchedulers.schedulers; }
        }

        #endregion Properties

        #region Methods

        #region Public Methods

        public void Dispose()
        {
            Dispose(true);
            GC.SuppressFinalize(this);
        }

        #endregion Public Methods

        #region Protected Methods

        protected virtual void Dispose(bool disposing)
        {
            if (!disposed)
            {
                if (disposing)
                {
                    var cleanup = new Action<dynamic>(s =>
                    {
                        if (s != null)
                        {
                            var disposable = s as IDisposable;

                            if (disposable != null)
                                disposable.Dispose();

                            s = null;
                        }
                    });

                    cleanup(queued);
                    cleanup(sta);
                    cleanup(workStealing);
                }

                disposed = true;
            }
        }

        #endregion Protected Methods

        #endregion Methods
    }
}

The cleanup action above is unnecessary, of course. It’s a leftover from some code when I was playing around with the idea of easily disposing objects, without being sure they are disposable. Its usefulness is dubious…

I also use my ParallelTaskScheduler explicitly from some lower-level code. For example, when finding files, or when using anything in the UI that operates on all the files of a directory, I call ForEachAsync, an extension method from a sample by Stephen Toub. The original calls Task.Run. But I have changed my one as follows:

        /// <summary>A ForEachAsync implementation. Based on a sample in an article by Stephen Toub,
        /// <a href="http://blogs.msdn.com/b/pfxteam/archive/2012/03/05/10278165.aspx">
        /// Implementing a simple ForEachAsync, part 2</a>.</summary>
        /// <remarks>
        /// I've changed this from calling <b>Task.Run</b> to call <b>Task.Factory.StartNew</b> in order to have it use my 
        /// custom ParallelTaskScheduler rather than the default. (This was intended to be an experimental change, but I 
        /// decided to leave it like this.)</remarks>
        public static Task ForEachAsync<T>(this IEnumerable<T> source,
            int maxDegreeOfParallelism,
            Func<T, Task> body)
        {
            return Task.WhenAll(
                from partition in Partitioner.Create(source).GetPartitions(maxDegreeOfParallelism)
                select Task.Factory.StartNew(async () =>
                {
                    using (partition)
                        while (partition.MoveNext())
                            await body(partition.Current);
                }, CancellationToken.None,
                TaskCreationOptions.DenyChildAttach | TaskCreationOptions.HideScheduler,
                new ParallelTaskScheduler(maxDegreeOfParallelism)).Unwrap());
        }

        /// <summary>An asynchronous ForAsync implementation.</summary>
        /// <remarks>It simply creates an <b>Enumerable.Range</b> and wraps <b>ForEachAsync</b>.</remarks>
        public static Task ForAsync(int fromInclusive, int toExclusive, int maxDegreeOfParallelism, Func<int, Task> body)
        {
            return Enumerable.Range(
                fromInclusive, toExclusive).
                ForEachAsync(maxDegreeOfParallelism, async i => await body(i));
        }

I also recently added a global TaskFactory, so that it can contain convenience shortcut methods, similar to Task.Run, which I will be able to access via Tasks.Run, like so:

using System;
using System.Threading;
using System.Threading.Tasks;
using System.Threading.Tasks.Schedulers;

namespace Romy.Core
{
    public static class Tasks
    {
        private static TaskFactory factory = new TaskFactory(CancellationToken.None,
           TaskCreationOptions.DenyChildAttach | TaskCreationOptions.HideScheduler,
           TaskContinuationOptions.ExecuteSynchronously, Scheduler);

        private static readonly TaskScheduler scheduler = new ParallelTaskScheduler();

        public static TaskFactory Factory
        {
            get { return Tasks.factory; }
        }

        public static TaskScheduler Scheduler
        {
            get { return Tasks.scheduler; }
        }

        public static Task Run(Func<Task> body)
        {
            return factory.StartNew(async () => await body()).Unwrap();
        }
    }
}
Posted in Programming | Tagged , , , | Leave a comment

A custom TaskScheduler in C#

Yesterday I expressed my confusion around TaskSchedulers. I was struggling with wondering what happens when I have several different TaskScheduler implementations in my application. Actually, this is a non-issue.

A TaskScheduler is just a simple abstraction for a low-level object that runs your tasks. That is, it implements the logic to schedule the running of the underlying threads, and so on. Normally the default implementation, which intelligently decides whether to run tasks on the ThreadPool or inline (on the current thread), and whether to use work-stealing (which is a way of load balancing), is perfect. You only need to change it if you need lower-level control of the underlying threads; for example, you want them to run in a particular order, or you want some to have priority over others.

There is precious little documentation of TaskSchedulers online, so it turns out the easiest way to understand them is just to write one yourself. That is, implement the abstract TaskScheduler class. And that is exactly what I did today.

Introducing the ParallelTaskScheduler. I wrote this in a few minutes, plugged it into my application, replacing the scheduler I used for populating my cache files with this one, and it works like a charm.

And I kid you not… Thanks to the huge amount of high quality sample parallel programming code from Microsoft, this literally took only a few minutes to write. (There are several custom TaskScheduler implementations in the Parallel Extensions project, which is part of the samples linked to.)

All mine does is, it groups all the tasks queued to it into batches, the size of which is determined by the maxDegreeOfParallelism parameter passed to its constructor, then for each batch, runs its tasks on the ThreadPool in parallel. More detail below.

using System.Collections.Generic;
using System.Linq;

namespace System.Threading.Tasks.Schedulers
{
    /// <summary>Custom TaskScheduler that processes work items in batches, where 
    /// each batch is processed by a ThreadPool thread, in parallel.</summary>
    /// <remarks>
    /// This is used as the default scheduler in several places in this solution, by, 
    /// for example, calling it directly in <see cref="TaskExtensions.ForEachAsync"/>, 
    /// or by accessing the relevant property of the static <see cref="TaskSchedulers"/> 
    /// class.</remarks>
    public class ParallelTaskScheduler : TaskScheduler
    {
        [ThreadStatic]
        private static bool currentThreadIsProcessingItems;

        private int maxDegreeOfParallelism;

        private volatile int runningOrQueuedCount;

        private readonly LinkedList<Task> tasks = new LinkedList<Task>();

        public ParallelTaskScheduler(int maxDegreeOfParallelism)
        {
            if (maxDegreeOfParallelism < 1)
                throw new ArgumentOutOfRangeException("maxDegreeOfParallelism");

            this.maxDegreeOfParallelism = maxDegreeOfParallelism;
        }

        public ParallelTaskScheduler() : this(Environment.ProcessorCount) { }

        public override int MaximumConcurrencyLevel
        {
            get { return maxDegreeOfParallelism; }
        }

        protected override bool TryDequeue(Task task)
        {
            lock (tasks) return tasks.Remove(task);
        }

        protected override bool TryExecuteTaskInline(Task task,
            bool taskWasPreviouslyQueued)
        {
            if (!currentThreadIsProcessingItems) return false;
            
            if (taskWasPreviouslyQueued) TryDequeue(task);

            return base.TryExecuteTask(task);
        }

        protected override IEnumerable<Task> GetScheduledTasks()
        {
            var lockTaken = false;
            try
            {
                Monitor.TryEnter(tasks, ref lockTaken);
                if (lockTaken) return tasks.ToArray();
                else throw new NotSupportedException();
            }
            finally { if (lockTaken) Monitor.Exit(tasks); }
        }

        protected override void QueueTask(Task task)
        {
            lock (tasks) tasks.AddLast(task);

            if (runningOrQueuedCount < maxDegreeOfParallelism)
            {
                runningOrQueuedCount++;
                RunTasks();
            }
        }

        /// <summary>Runs the work on the ThreadPool.</summary>
        /// <remarks>
        /// This TaskScheduler is similar to the <see cref="LimitedConcurrencyLevelTaskScheduler"/> 
        /// sample implementation, until it reaches this method. At this point, rather than pulling 
        /// one Task at a time from the list, up to maxDegreeOfParallelism Tasks are pulled, and run 
        /// on a single ThreadPool thread in parallel.</remarks>
        private void RunTasks()
        {
            ThreadPool.UnsafeQueueUserWorkItem(_ =>
            {
                List<Task> taskList = new List<Task>();

                currentThreadIsProcessingItems = true;
                try
                {
                    while (true)
                    {
                        lock (tasks)
                        {
                            if (tasks.Count == 0)
                            {
                                runningOrQueuedCount--;
                                break;
                            }

                            var t = tasks.First.Value;
                            taskList.Add(t);
                            tasks.RemoveFirst();
                        }
                    }

                    if (taskList.Count == 1)
                    {
                        base.TryExecuteTask(taskList[0]);
                    }
                    else if (taskList.Count > 0)
                    {
                        var batches = taskList.GroupBy(
                            task => taskList.IndexOf(task) / maxDegreeOfParallelism);

                        foreach (var batch in batches)
                        {
                            batch.AsParallel().ForAll(task =>
                                base.TryExecuteTask(task));
                        }
                    }
                }
                finally { currentThreadIsProcessingItems = false; }

            }, null);
        }
    }
}

There are three abstract methods that must be implemented. The only really important one is QueueTask. All that my one does is:

  • Add the task to an internal list.
  • Remove and execute all the queued tasks at that moment in time.

The actual running of the tasks is dead simple. Break them into batches, and run them via Parallel.Foreach to process each batch in turn.

The other abstract methods are:

  • TryExecuteTaskInline: To execute a task on the current thread.
  • GetScheduledTasks: Return an enumerable of tasks queued to the scheduler.

My TryExecuteTaskInline implementation removes the task from the internal list as well, if necessary, and because of this, I also implemented TryDequeue, which is a virtual method. A pity really – I had wanted to use a ConcurrentBag<T> for my internal list, which would have made the processing of tasks random (since it’s unordered). I couldn’t do that though, since I then had no way of removing specific tasks from the queue.

Update (2013-06-26): This is the slightly optimized and improved version:

  • QueueTask: The initial version was a little too keen to start running the threads, which would cause frequent ThreadPool threads running only a single Task. That should only happen when either there is only one task being scheduled, or the number of tasks being divided into batches leaves a remainder of one.
     
  • GetScheduledTasks: This was implemented as…
        lock (tasks) return tasks.ToArray();
    But that could hang the debugger if a lock is owned by a frozen thread. Thus it now uses Monitor.TryEnter, the same as the Microsoft sample code.

This code performs surprisingly well, and I’m now using it in most places of my RomyView application where I specify a TaskScheduler.

 

As you can see, TaskSchedulers are quite simple beasts… My confusion yesterday was unwarranted because the TaskScheduler runs there and then, as you queue the Task to it. How it does so is completely up to you, if you implement your own.

Posted in Programming | Tagged , , | Leave a comment

Getting more advanced knowledge is difficult

There was an annoying bug in my RomyView application the other day, a bug that caused what is in my mind, the worst kind of situation when it comes to asynchronous code: My code that populates a thumbnail cache item (which gets the image from the Windows Shell for .jpg files and movie files, and creates it directly for .png files and icons) was running on the main thread. Disaster! This happened, of course, while populating thousands of thumbnails, and brought the entire application to a grinding halt.

So just for the hell of it, I googled “Async in C# 5” and sure enough, I found a copy of the book for free. I guess I shouldn’t admit that. Sorry!

Anyway, to my surprise, there is very little in that book that I do not already know. Very little. Purchasing it would have felt like a waste of money.

Update: I decided to buy the e-book instead. I did only browse through it in a couple of minutes, and clearly a lot of work went into the book. It’s only $9.99 on the O’Reilly site, which works out cheaper than buying the e-book on a South African site.

It turns that I am running an overload of Task.Factory.StartNew that takes TaskCreationOptions and a TaskScheduler as parameters in the offending method. The TaskCreationOptions parameter needed to change to TaskCreationOptions.LongRunning to force the underlying thread to run on the thread pool.

I have also been playing around with different TaskScheduler implementations to see which is most performant for this purpose. But it feels a little like stumbling around in the dark. I am using several different schedulers in the application, and while I understand each one in itself, I do not understand how all of them relate to each other in the overall application. That is, if I have three levels of priority in a queued scheduler that’s running on the IO threadpool, but also have some code retrieving thumbnails via a round-robin scheduler, and some threads fighting over populating the thumbnails via a work-stealing scheduler, which scheduler runs when?

I was hoping the book might lead me to the light… but it did not. All I can say is, to anyone else running similar code, do not have all your code run in queued schedulers, because you will be sure to run into issues when code that you would like to run now is not in the front of the queue.

So far, at least in my experience, work-stealing schedulers are generally the better way to go, and for each discrete task being performed, having its own dedicated scheduler seems preferred. But don’t listen to me. I’m making it up as I go along.

Posted in Programming | Tagged , , , | Leave a comment

The Email Virus (that never was)

I used to enjoy reading the Daily WTF. But now it is full of articles like this.

OK, so it is an interesting story about a nasty worm virus, and how a dumb manager got her just desserts for forwarding it to the entire company with a “Do not open the attachment – It is a virus” warning, and having many of the cattle they had for staff opening it and toast their PCs.

Interesting, and obviously not true.

When last did you see an email virus that did anything more than propagate? (And hence cause chaos w.r.t. network traffic) In reality, that’s all they do, and it’s been that way for a long time.

Virus programmers were a dumb bunch to begin with, and these days they are surely just “scripting kiddies”. If decent programmers wrote viruses, then yes, there is plenty of nasty stuff that could be done (I will not elaborate, in case one of the dumbasses reads this, although they would probably only toast their own machine), but it just doesn’t happen.

Conclusion

Sites like The Daily WTF are only funny when their posts are true. Contrived stories are not funny, especially when they are written poorly. There are plenty of imaginative people out there who are gifted in writing who could do a better job making up stories anyway.

Daily WTF administrator, what the fuck is wrong with you?

Posted in General | Tagged | Leave a comment

Confused by my own design – behaviour through inheritance – How not to achieve it

I wonder, do we all get to that point where our own code, written days or weeks or months or maybe even years back, makes us ask the question, “How the fuck did I do that?”

Everything here refers to my RomyView application.

Take for instance, my conversion progress dialog. This is part of my video converter class library.

At design time, it looks like this:

ConversionProgressDesignTime

But at run-time, it looks like this:

ConversionProgressRunTime

Ignoring the colours being different, which is part of my custom Themes implementation, the noticeable difference is that somehow my custom code that paints shadows for buttons and the panel ran, even in the forms designer, but not for the ComboBox. Why?

It turns out that along the way I created quite a complex forms inheritance hierarchy, and this dialog inherits from Romy.Controls.BaseDialog, which inherits from Romy.Controls.BaseForm. And my base form class implements a paint handler like so:

Base Form Paint Handler
  1. protected override void OnPaint(PaintEventArgs e)
  2. {
  3.     if (InheritedDrawShadows)
  4.     {
  5.         foreach (var c in Controls.ToArray<Control>().
  6.             Where(cc => cc.Visible && ((cc is Panel && cc.GetType() != typeof(ShadowLessPanel) &&
  7.                 cc.GetType() != typeof(BlackPanel)) || cc is Button || cc is PropertyGrid || cc is Thumbnail)))
  8.         {
  9.             var drawShAadow = true;
  10.  
  11.             if (c is Thumbnail)
  12.             {
  13.                 var thumb = c as Thumbnail;
  14.  
  15.                 if (thumb.IsCustomSize && thumb.BorderStyle == BorderStyle.None)
  16.                     drawShAadow = false;
  17.             }
  18.  
  19.             if (drawShAadow)
  20.                 e.Graphics.DrawShadow(c);
  21.         }
  22.     }
  23.     base.OnPaint(e);
  24. }

 

What that says is, for all derived forms that don’t set InheritedDrawShadows false, for all buttons, panels, thumbnails and property grids (excluding certain derived custom types that I especially created not to have shadows), and unless the control is an instance of my Thumbnail control using a specific optimization that allows a Thumbnail just to draw an Image and nothing else, draw the damn shadow. This code is the reason that certain shadows are drawn even in the designer.

And now I remember, I didn’t add ComboBoxes to that code because I might not always want them to have shadows. In fact, usually they shouldn’t have. But I added an OnPaint handler to the progress dialog for that one.

OK. Now it makes sense. It also explains why the dialog below, part of my image filters dialogs class library in the solution, looks the same at design time and run-time:

ResizeDialog

But that doesn’t explain what happened here… This dialog, also part of the same class library, doesn’t do any of my drawing at design time:

HslDialogDesignTime

But at run-time, it’s behaving as it should be:

HslDialogRunTime

Actually, this reveals a limitation of my base class paint handler. It can only paint shadows for certain types of controls that are dropped directly onto the form.

Oops. And how did I make it draw the shadows above? With this horrible hack:

Base Dialog Paint Handler
  1. #region Protected Methods
  2.  
  3. /// <summary>Button shadow affects for dialogs.</summary>
  4. /// <remarks>Defined here, rather than the base form, because I typically
  5. /// have buttons in SplitContainers on dialogs, not forms. And if I do have
  6. /// any on forms, I may not want this default shadow drawing.</remarks>
  7. protected override void OnCreateControl()
  8. {
  9.     /* The base form’s paint handler doesn’t handle nested controls, thus it won’t paint the
  10.      * shadows for buttons on the split container panels. Painting them requires handlers for
  11.      * any SplitterPanels that contain buttons. We add them here. */
  12.     foreach (var s in Controls.ToArray<Control>().Where(ss => ss is SplitContainer))
  13.     {
  14.         foreach (var p in s.Controls.ToArray<Control>().Where(pp => pp is SplitterPanel && pp == ((SplitContainer)s).Panel1))
  15.         {
  16.             p.Paint += (sender, args) =>
  17.             {
  18.                 PaintPanelShadows(p, args);
  19.             };
  20.         }
  21.  
  22.         foreach (var p in s.Controls.ToArray<Control>().Where(pp => pp is SplitterPanel && pp == ((SplitContainer)s).Panel2))
  23.         {
  24.             p.Paint += (sender, args) =>
  25.             {
  26.                 PaintButtonShadows(p, args);
  27.             };
  28.         }
  29.     }
  30.  
  31.     foreach (var p in Controls.ToArray<Control>().Where(pp => pp is Panel))
  32.     {
  33.         p.Paint += (sender, args) =>
  34.         {
  35.             PaintButtonShadows(p, args);
  36.         };
  37.     }
  38.  
  39.     base.OnCreateControl();
  40. }
  41.  
  42. #endregion Protected Methods
  43.  
  44. #region Private Methods
  45.  
  46. private static void PaintButtonShadows(Control p, PaintEventArgs args)
  47. {
  48.     foreach (var b in p.Controls.ToArray<Control>().Where(bb => bb is Button && bb.Visible))
  49.     {
  50.         args.Graphics.DrawShadow(b);
  51.     }
  52. }
  53.  
  54. private static void PaintPanelShadows(Control p, PaintEventArgs args)
  55. {
  56.     foreach (var b in p.Controls.ToArray<Control>().Where(bb => bb is Panel && bb.Visible))
  57.     {
  58.         args.Graphics.DrawShadow(b);
  59.     }
  60. }
  61.  
  62. #endregion Private Methods

 

Conclusion

While this code works, it is rather flimsy. Controls in different containers will not inherit the intended behaviour at all.

So what did I do wrong?

While the approach was heading in the right direction, my implementation is, well, backwards. Instead of trying to paint every type I could think of from the forms, and going as far as deriving specific types that should not inherit the intended behaviour, I should have created types specifically to inherit the intended behaviour. That is, create derived Buttons and so on that should always paint shadows, no matter what their containers might be. It would have involved less code, and been easier to read as well.

I hope that this might point out to somebody else who is heading down the wrong road (or heading up the right road instead of down) that they need to rethink their design. These things are easy to get wrong while you are coding in that zone deep in the code…

Note: Obviously Graphics.DrawShadow is an extension method and is irrelevant to the point of this article.

Posted in Programming | Tagged , , | Leave a comment