Chris Brumme

When I learned in 2018 that Chris had passed away, I was very sad.

I first met Chris in the fall of 2003. Erik Meijer had asked me to attend a devlab in Redmond for Xen and since I needed the trip to get to KLM Flying Dutchman Platinum status I decided to go.

The Xen devlab was canceled, but I did go to spend a lovely week driving and hiking around beautiful Washington state and met up with Chris in his office. We talked about the intricacies of running Java on the CLR (both technical and legal).

We started e-mailing earlier in 2003. Below is the archive of these mails. I made a couple of redactions to remove names, two phone numbers and some personal details. I updated some links and added clarifying titles to some terms.

In memory of a brilliant, but above all very kind and generous person.

Christopher Brumme
blog@jeroen.nu
Wed 5/7/2003 17:29
privatescope
Jeroen,

I’m not sure I fully understand your scenario.

I think you want to have a method on a class which satisfies an interface contract. Today, this means the method must be virtual.

You want to be sure that the subtype can replace your stubbed method body with something reasonable. So long as the subtype mentions the Interface on its ‘implements’ list again, we will trigger a new layout of this interface on that subtype.

You want to be able to use any old name for the class method. This implies you are using a MethodImpl to match the method to the interface contract. Any name will work, so long as it is unique on this class. So you could come up with a mangling scheme, using characters that are illegal in the source language but legal in the metadata. It sounds like you want to use ‘privatescope’ to avoid the uniqueness requirement. (Is this the only reason you need privatescope?)

You want the class method to be unavailable to external callers, because it is an implementation detail. Any of private, privatescope, or internal would achieve this. Private is the most restrictive, and arguably the most suitable here. This is what C# does, as you know.

Are those your requirements? If so, I would personally use a ‘private virtual newslot final’ method, just like C#. And I would come up with a name-mangling scheme for the name. Unlike C#, I might pick a mangling scheme that includes ‘$’ or some other bizarre character other than ‘.’. But I would have to check the list of legal characters.

IKVM looks very interesting. If you get really stuck on something, feel free to ask. I may be able to help. However, I want to avoid getting deep into Java / JVM issues. I can only help you with issues targeting the CLR.

Chris.

http://blogs.gotdotnet.com/cbrumme/

Jeroen Frijters
Christopher Brumme
Fri 5/9/2003 09:24
RE: privatescope

Hi Chris,

 

> I’m not sure I fully understand your scenario.

 

I dug up my original notes and I'll explain what I was trying to do (note that I'm not looking for a solution, when I found out that privatescope methods aren't allowed to be virtual, I chose to use another way of doing of it).

 

This is a somewhat bizarre scenario, but since I want my JVM to be as compatible as possible I did think it was important to support. Here is the Java code (won't compile, but the equivalent .class files will run).

 

class Base {

  protected void Foo() {}

}

interface IFoo {

  public void Foo();

}

class Derived implements IFoo {

}

class MoreDerived extends Derived {

  protected void Foo() {}

}

 

Note that Derived is not abstract, but it does *not* implement IFoo.Foo, because Base.Foo is protected it doesn't qualify as an implementation of IFoo.Foo. If you instantiate Derived on a JVM (this is allowed!) and then call IFoo.Foo, it throws an IllegalAccessError.

 

Now when I compiled this to MSIL, I inserted a privatescope Foo method in Derived that implements IFoo.Foo and made that method throw an IllegalAccessError. To my surprise I found that MoreDerived.Foo actually overrides the Derived.Foo privatescope method. I hadn't expected this because I (incorrectly) assumed that the CLR wouldn't ever take the name of a privatescope method into account (basically, I though of privatescope as equivalent to "this has no name").

 

The solution was, of course, trivial. Mangle the name of the Derived.Foo method and make it private instead of privatescope (because I had since learned from Serge Lidin that privatescope virtual wasn't a legal combination anyway). Everyone happy, except for the fact that I felt some disappointment at the fact that privatescope wasn't as nice a construct as I had originally thought when I was reading the ECMA spec for the first time. I had hoped it would do away with the need for name mangling, but in this case I still had to use it.

 

> It sounds like

> you want to use ‘privatescope’ to avoid the uniqueness

> requirement.  (Is this the only reason you need privatescope?)

 

Exactly. I thought it was the most elegant way of mapping the (broken) Java classes onto CLR types.

 

> You want the class method to be unavailable to external

> callers, because it is an implementation detail.  Any of

> private, privatescope, or internal would achieve this. 

> Private is the most restrictive, and arguably the most

> suitable here.  This is what C# does, as you know.

 

An additional property of privatescope that I liked, was the fact that it wasn't private. When the Java application would use reflection to look at the methods in the type, it would be really easy for me to filter out the privatescope methods (which weren't supposed to be there, from the Java point of view). In the new situation I had to use a custom attribute to mark the stub method as an implementation detail that shouldn't be visible.

 

> Are those your requirements?  If so, I would personally use a

> ‘private virtual newslot final’ method, just like C#.  And I

> would come up with a name-mangling scheme for the name.

 

This is exactly what I ended up with.

 

> IKVM looks very interesting.  If you get really stuck on

> something, feel free to ask.  I may be able to help. 

> However, I want to avoid getting deep into Java / JVM issues.

>  I can only help you with issues targeting the CLR.

 

Thanks. Let me explain a little of what prompted my original comment. I think I have a pretty deep understanding of both the JVM and CLR and I must say that I *love* the CLR. In fact, I like the CLR so much, that every time I find a tiny blemish, I'm disappointed ;-) I'd like the CLR (and the specification) to be all it can be, so I occasionally try to challenge you (and others at MS) in the hope that either it will be explained to me why something was designed in such a way (in other words, prove me wrong) or that the design (or implementation) might be improved (in the long run. I don't really care about the small service pack style bugs that can easily be worked around).

 

BTW, thanks for your blog, it is really good to read about the design decisions and trade offs that were/are considered. Are you attending the PDC? I'd love to chat with you over a couple of beers :-)

 

Regards,

Jeroen

Christopher Brumme
Jeroen Frijters
Fri 5/9/2003 18:37
RE: privatescope

Thanks, that all makes perfect sense.

 

I'll try to chase down the discrepancies between ECMA and our implementation with respect to privatescope.

 

I'm not sure if I'll be going to the PDC.  I've consistently missed every PDC for the last 6 years (and every company meeting, too).  But I'm seriously thinking of going to this one.  If I go, a beer sounds great.

 

Chris.

http://blogs.gotdotnet.com/cbrumme/

 

 
Jeroen Frijters
Christopher Brumme
Thu 6/5/2003 09:06
A possible question for your Blog

Hi Chris,

 

I've got a question that might be appropriate for your blog. I would like to know why the CLR has finalizers and what your take on them is. To give some context: IMHO, finalizers should only be used as a last resort for freeing unmanaged resources and nothing else, because of this, I would have put an InheritanceDemand on Finalize for the UnmanagedCode permission. At the moment, untrusted code can run at a higher priority by high-jacking the finalizer thread (In Java, untrusted code can change the thread priority, I'm not sure that .NET even tries to prevent this). More important, I believe that untrusted code can prevent an AppDomain from being unloaded -- again -- by running in the finalizer thread.

 

Second question: Will you be at the Whidbey SDR on the 25/26 of June?

 

Regards,

Jeroen

Christopher Brumme
Jeroen Frijters
Thu 6/5/2003 15:01
RE: A possible question for your Blog

For your second question, I shall be on Maui from 6/11 thru 7/4.

However, I have broadband and wireless out by the pool.  So -- if my wife permits -- I shall still blog a little.

 

I'm thinking of going to the PDC in L.A. in late October this year.

 

Chris.

http://blogs.gotdotnet.com/cbrumme/

 
Jeroen Frijters
Christopher Brumme
Thu 6/5/2003 15:08
RE: A possible question for your Blog

 

> For your second question, I shall be on Maui from 6/11 thru 7/4.

> However, I have broadband and wireless out by the pool.  So -- if my

> wife permits -- I shall still blog a little.

 

That's fun too ;-)

 

> I'm thinking of going to the PDC in L.A. in late October this year.

 

I know, I already asked you. I may have even promised to buy you a beer ;-)

 

Regards,

Jeroen

 
Christopher Brumme
Jeroen Frijters
Thu 6/5/2003 15:14
RE: A possible question for your Blog

I think you said "a couple of beers", which could be interpreted as up to 3.

 

Chris.

http://blogs.gotdotnet.com/cbrumme/

Jeroen Frijters
Christopher Brumme
Fri 7/25/2003 12:49
A few questions

Hi Chris,

 

I hope you had a good vacation!

 

The Whidbey SDR was fun and the alpha bits are very nice.

 

I have a few questions, possibly for your blog or you can just ignore them and I'll get the answers at the PDC, while you're under the influence of large amounts of alcohol ;-)

 

- Why no return type co-variance (and argument type contra-variance)?

- Why no delegate structural equivalence?

- What is the output of this code:

 

class Test {

  public static void Main() {

    int i = int.MinValue;

    System.Console.WriteLine(-i);

    System.Console.WriteLine(i * -1);

    System.Console.WriteLine(i / -1);

  }

}

 

Why?

 

- Why don't arrays implement IList<T>, ICollection<T> and IEnumerable<T>? (I think I understand why, but it might make a good blog entry to explain). Would it be a good idea to add a static helper to System.Array to wrap arrays into a generic class that does implement these interfaces?

- What does ildasm mean when it says ".assembly *legacy* library mscorlib" on Whidbey assemblies?

 

That's enough for now ;-). Feel free to ignore me if you're busy.

 

Regards,

Jeroen

Christopher Brumme
Jeroen Frijters
Fri 7/25/2003 15:26
RE: A few questions

Some of these are easy.

 

No co/contravariance -- we haven't had time to do it yet.  It's not a controversial feature.  You might see some limited places show up in the next release.  But not where you are expecting it (i.e. not with normal virtual methods).

 

No delegate structural equivalence -- we haven't had time to do it yet.

This one is even more desirable.  You might see us make it easier for you to form a single delegate type over multiple related signatures, before you see us actually treat structurally equivalent delegates as type-substitutable.

 

For the math output, presumably you are asking about the exception.  The code is:

 

Mov eax, 80000000

Mov ecx, ffffffff

Cdq

Idiv ecx

 

This generates an integer overflow exception.  From the ECMA spec on

'div':

 

      Integral operations throw ArithmeticException if the result

      cannot be represented in the result type. This can happen

      if value1 is the smallest representable integer value, and

      value2 is -1.

 

I haven't been following the generics effort as closely as I should.  My initial answer, without thinking about it, is that variant arrays (MyType[] substitutes for Object[]) would be something of a lie.  But the lie is no worse than the substitution, so this may not be the reason.  I'll check into this.

 

Part of the Whidbey versioning story is that assemblies can be tagged to indicate something about how they version.  Based on what ILDASM is saying, it sounds like the tag isn't present in mscorlib yet.  Therefore ILDASM is categorizing it as the default.  The 'legacy' part indicates that the assembly isn't participating properly in the new plan yet.

 

Chris.

Jeroen Frijters
Christopher Brumme
Fri 7/25/2003 15:57
RE: A few questions

Chris, thanks for your answers.

 

> For the math output, presumably you are asking about the exception. 

> The code is:

>

> Mov eax, 80000000

> Mov ecx, ffffffff

> Cdq

> Idiv ecx

>

> This generates an integer overflow exception.  From the ECMA spec on

> 'div':

 

Right, but this is an x86-ism. The CLR could handle the exception and pretend the result of the idiv is -1. This is what Java does and I think that makes for a more consistent story, especially since the CLR has the .ovf versions (but the div.ovf is conspicously absent).

 

So what I would have done is: div.ovf = current behavior and div = catch the overflow exception and resume on the next instruction (leaving the -1 in eax).

 

> I haven't been following the generics effort as closely as I should. 

> My initial answer, without thinking about it, is that variant arrays

> (MyType[] substitutes for Object[]) would be something of a lie.  But

> the lie is no worse than the substitution, so this may not be the

> reason.  I'll check into this.

 

I thought it had to do with the type information sharing that arrays do, but after thinking about it a bit more I don't think there is any reason why it wouldn't work. Performance for value types would probably not be ideal (unless specialised code for each array type is generated, but that would be wasteful), but it would be a powerful feature.

 

This reminds me, I noticed the private FastArray<T> class in mscorlib and I assume it is there to get around the cost of array covariance, but it looks to be more complicated than it needs to be. Someone else (I can't take credit unfortunately) came up with this pattern:

 

      public sealed class FastArray<T>

      {

           struct ValueContainer<Ti>

           {

                 internal Ti t;

           }

 

           ValueContainer<T>[] array;

 

           public FastArray(int count)

           {

                 array = new ValueContainer<T>[count];

           }

 

           public int Length

           {

                 get { return array.Length; }

           }

     

           public T this[int index]

           {

                 get { return array[index].t; }

                 set { array[index].t = value; }

           }

      }

 

Regards,

Jeroen

Christopher Brumme
Jeroen Frijters
Fri 7/25/2003 16:41
RE: A few questions

The ECMA rules for division are that the sign of the result is positive if both operands have the same sign, and it's negative if both operands have different signs.  This makes sense to me.

 

Since you are dividing two negative operands, the result must be positive.  Returning -1 seems like the wrong thing to do here.  The actual result cannot be represented in a signed integer of this size.

 

I'll dig into the array / generics issues and get back to you later.

 

Chris.

 
Jeroen Frijters
Christopher Brumme
Fri 7/25/2003 16:52
RE: A few questions

 

> The ECMA rules for division are that the sign of the result is

> positive if both operands have the same sign, and it's negative if

> both operands have different signs.

> This makes sense to me.

 

OK, but why not apply the same rule to multiplication then?

 

-2147483648 * -1 = -2147483648

 

 is just as nonsensical as

 

-2147483648 / -1 = -2147483648

 

but only the latter one throws an exception.

 

Regards,

Jeroen

Christopher Brumme
Jeroen Frijters
Fri 7/25/2003 16:55
RE: A few questions

Working on the execution engine rather than the JIT, I don't' spend much time looking at IL.  I have to admit that I never noticed the asymmetry.

I'll look into this, too.  I'm curious if there was a rationale.

 

Chris.

Christopher Brumme
Jeroen Frijters
Tue 8/19/2003 19:02
RE: A few questions

Sorry, I've been gone for a while on some fire drills.

Your FastArray is certainly convenient to express.  But it creates new structs and new arrays of structs for each expansion of a different reference type T.  Today, we share code very aggressively for reference types, but we do a poor job of sharing code for different structs.  Therefore, your approach will generate many distinct method bodies and stubs.  In the future, we could do a better job here.

And here's the internal story for the div/mul thing from a dev involved with the decisions.  I doubt that this is our official story, so I don't think I would publicize it:

As you say, div really should be called div.ovf, and thus there is symmetry there with mul.ovf, that part is perfectly reasonable.

Really the only issue is why we don’t have a div without overflow.    The simple answer here is that it did not add a whole lot of value (where would you use div non-overflow?)

Of course historically this is not how it happened.  In reality div and mul came first before the overflow variants, and the goal of the IL was simply to find an abstraction for many hardware platforms.  Hardware DID support overflow for division but not multiplication.    Later we added mul.ovf.   Why go back at that point and go to some trouble to create a ‘broken’ version of div (since it really does have to return the wrong value for at least on case), just for the sake of symmetry?

 

Chris.

http://blogs.gotdotnet.com/cbrumme/
Jeroen Frijters
Christopher Brumme
Wed 8/20/2003 17:31
RE: A few questions

Hi Chris,

 

> Your FastArray is certainly convenient to express.  But it creates new

> structs and new arrays of structs for each expansion of a different

> reference type T.  Today, we share code very aggressively for

> reference types, but we do a poor job of sharing code for different

> structs.  Therefore, your approach will generate many distinct method

> bodies and stubs.

>  In the future, we could do a better job here.

 

I also noticed that when I change FastArray to a value type, it becomes a lot slower. I assume this is due to the extra type information that needs to be passed in each method call.

 

Another thing I've been thinking about (and this is really far out there) is running managed code in kernel mode. Do you think that in the future (and I'm talking about far into the future, past Longhorn/Orcas) it will ever become possible to run (verifiable) managed applications in kernel mode? It would obviously require a lot of work (esp. duplicating all the APIs on the kernel side), but the performance and scalability gains could be significant. I don't know if you're familiar with the IBM AS400, but its architecture inspired me to think about managed code in kernel mode.

 

Obviously this requires a 64 bit architecture, because currently kernel memory space is already cramped, but I think that this might be an interesting future path.

 

Thanks.

 

Regards,

Jeroen

Christopher Brumme
Jeroen Frijters
Wed 8/20/2003 18:24
RE: A few questions

Long term, we would all like to see managed execution in the kernel.

Suspension for GC has to be done differently.  NGEN rather than JIT becomes more important.

 

But we all see the benefits of having type safety and bounds checking inside device drivers.  They will need to step outside of type-safety in some places, but those places can be clearly marked for audit (like C# 'unsafe' blocks).

 

Eventually, I'm confident that this will happen.  Whether it's the CLR we built, or whether the OS folks build their own... who knows.

 

Chris.

http://blogs.gotdotnet.com/cbrumme/

 
Jeroen Frijters
Christopher Brumme
Thu 8/21/2003 14:39
RE: A few questions

Hi Chris,

 

I just read your most recent blog entry. Awesome! Thanks for taking the time (and having the courage) to write this stuff up.

 

At the end you write: "This means aggressively chasing the weaknesses of our present system, like the fact that locally installed assemblies by default run with FullTrust throughout their execution."

 

This (FullTrust for local code) was my biggest disappointment with .NET when I first encountered it, it's good to see that it is on the agenda. Another thing that annoyed me recently was a 3rd party application that decided (during its installation) that it would be a good idea to modify the .NET framework security policy (without telling me). I hope that for Longhorn, application sandboxing will become the norm rather than the exception, but I realise that this is a difficult task (and probably not related to what you are doing).

 

I know you probably can't talk about this (hopefully at the PDC), but I'm wondering about the combination of Palladium (eh, NGSCB) and managed code. I hope that *only* managed code will be allowed to run as a Palladium agent. That way running encrypted code can be prevented (which I believe is critical to the success of Palladium, imagine a worm that uses Palladium to encrypt itself securely).

 

About the kernel mode code, do you think it will ever become viable to run application code in kernel space? Below you said: "NGEN rather than JIT becomes more important". Is this because of bootstrapping issues, or because you don't want something with the complexity of the JIT running in kernel space? Maybe one day the JIT will be written in managed code as well ;-)

 

Regards,

Jeroen

 
Christopher Brumme
Jeroen Frijters
Thu 8/21/2003 16:31
RE: A few questions

More personal opinions:

 

By the time LH rolls around, I think we'll have a much more rational and restricted security world for applications to live in.  And a reasonable way for apps to escape that restricted world.

 

NGSCB is necessarily a lot less than Win32.  Therefore the CLR cannot run in NGSCB without a significant port.  I think everyone agrees that this is a long-term sensible way to execute ultra-secure application code.  However, it's not clear whether either the CLR or NGSCB are mature enough for us to invest in that marriage today.  And as always we are very pressed for resources.

 

I'm not sure why we would want to run application code in kernel space.

Device drivers make sense to me.

 

I think existing kernel developers would be more comfortable with NGEN than the JIT for kernel-side managed execution (should this ever occur).

The notion of generating code on the fly -- inside the kernel -- and then executing whatever you ended up with, is horrifying to them.  They really want to know what they are going to be executing and they really don't want any randomization based on the sequence in which JITting occurs, or based on which .cctors win a race condition, etc.  They want absolute predictability and I cannot blame them.

 

It's perfectly reasonable for JITs to be written in managed code.  There are bootstrapping issues, but they are addressed with NGEN or interpretation or with a special bootstrapping unmanaged JIT.  I think you will see managed JITs at some point.

 

Chris.

http://blogs.gotdotnet.com/cbrumme/
Christopher Brumme
Jeroen Frijters
Thu 10/2/2003 18:56
Exceptions

Could you explain the use of the ‘volatile’ prefix before any non-dependent ldfld or stfld?  We will order stores for you.  It’s true that loads might be hoisted.  Does Java have a stronger memory model than X86, which forces your compiler to emit these prefixes?

 

Although Java may allow transfers in or out of finally blocks, your example doesn’t seem to take advantage of this.  So it seems like you could compile your current example more simply than you do.  Can’t this example be handled by emitting try/catch/finally clauses exactly as they appear in the source?

 

And (you’ll have to excuse my extreme ignorance of Java) you say that try/finally is compiled in Java to try/catch.  Presumably the Java compiler branches from the catch to some common code that’s shared with the non-exceptional finally pathway.  Then at the end it either returns or throws depending on whether the finally was reached exceptionally or not.  You also say that Java supports branching in and out of catch & finally and that a return from a finally is legal.  Do you have a simple example that uses these freedoms?  I would like to think about it for a while.

 

Chris.

 

http://blogs.gotdotnet.com/cbrumme/

 
Jeroen Frijters
Christopher Brumme
Thu 10/2/2003 22:34
RE: Exceptions

Hi Chris,

 

> Could you explain the use of the ‘volatile’ prefix before any

> non-dependent ldfld or stfld?  We will order stores for you.

 

The example is from Java (source) code that might also run on other VMs that do require volatile. I wrote the code, but I didn't have the CLR memory model in mind at the time, the Java model is weaker.

 

> It’s true that loads might be hoisted.  Does Java have a stronger

> memory model than X86, which forces your compiler to emit these

> prefixes?

 

Rather it's the other way around ;-) Java has a weaker memory model, so the source contains the a volatile modifier on the "running" field. When I compile the Java bytecode to CIL, I have to respect the fact that the field is volatile.

 

This code is not a very good example of the memory model issues, because it is broken on several counts (by design, there is no safe way to write the thread startup/teardown in Java, it really should be done in native code).

 

> Although Java may allow transfers in or out of finally blocks, your

> example doesn’t seem to take advantage of this.

 

Not on the surface, but I don't actually compile the source, I compile the .class file. A try finally is compiled as follows (pseudo code):

 

try {

  ...

  jsr finally_block;

} catch(Throwable t) {

  jsr finally_block;

  throw t;

}

 

finally_block:

   ....

   ret

 

So here you have branches out of both the try and catch blocks. I compile jsr by pushing an integer representing the call site and then branching to the address and ret is compiled as a bunch of beq instructions (jumping the return address corresponding to the call site).

 

> So it seems like you could compile your current example more simply

> than you do.  Can’t this example be handled by emitting

> try/catch/finally clauses exactly as they appear in the source?

 

If I were compiling source that would be the obvious solution, but now I don't know. For typical code I could figure out what constitutes a finally block (for known compilers anyway), but that would fail for unknown compilers or handwritten bytecode and that would cause the semantics to be different based on whether my CIL compiler was able to parse the bytecode correctly. This seems undesirable.

 

> And (you’ll have to excuse my extreme ignorance of Java) you say that

> try/finally is compiled in Java to try/catch.

> Presumably the Java compiler branches from the catch to some common

> code that’s shared with the non-exceptional finally pathway.  Then at

> the end it either returns or throws depending on whether the finally

> was reached exceptionally or not.

 

As you saw above, it's worse than that. They have a very evil jsr/ret construct (which makes their verifier very complicated).

 

> You also say that Java supports branching in and out of catch &

> finally and that a return from a finally is legal.

> Do you have a simple example that uses these freedoms?  I would like

> to think about it for a while.

 

I hope the above made it clear. I don't have any good examples of returning from a finally block (which is obviously an evil thing to do). I'm vacationing in WA next week, wanna have a cup of coffee?

 

Regards,

Jeroen

 
Christopher Brumme
Jeroen Frijters
Fri 10/3/2003 01:10
RE: Exceptions

I need to think about the exceptions more.  I don't drink coffee, but would certainly enjoy meeting with you while you are out here.

 

You have my work email.  The phone number here is [Redacted].

 

Of course, I still have a beer or two coming my way at the PDC.

 

Chris.

http://blogs.gotdotnet.com/cbrumme/

 
Jeroen Frijters
Christopher Brumme
Fri 10/3/2003 08:43
RE: Exceptions

 

> I need to think about the exceptions more.  I don't drink coffee, but

> would certainly enjoy meeting with you while you are out here.

>

> You have my work email.  The phone number here is [Redacted].

 

OK. I'll give you a call when I'm in the neighborhood. I'm not taking my laptop with me, so I don't if I'll be able to check my e-mail.

 

> Of course, I still have a beer or two coming my way at the PDC.

 

Absolutely!

 

Regards,

Jeroen

 
Christopher Brumme
Jeroen Frijters
Fri 10/3/2003 15:58
RE: Exceptions

If you can't reach me at my desk with the number below, you can also try my cell phone [Redacted].

 

Chris.

http://blogs.gotdotnet.com/cbrumme/

 
Christopher Brumme
Jeroen Frijters
Thu 10/9/2003 18:44
Class constructor ordering

Here’s an example of a program which – I think – would fail a Java conformance test.

class A  {

static {

                        B.testF();

          }

}

class B extends A  {

            static Object f = new Object();

            public static void testF() {

                        f.toString();

            }

}

class C  {

            public static void main(String[] args)  {

                        new B();

            }

}

On a conforming implementation, I suspect that A’s .cctor executes before B’s.  So the call from A’s .cctor to B.testF can trigger B’s .cctor.

On the CLR, B’s .cctor will start executing first.  If B’s .cctor chains to A’s .cctor, then we will consider this to be a cycle.  We will allow the thread to see uninitialized state in B (the alternative now being a deadlock).  This will cause an AV since B.f is still null.

Chris.

http://blogs.gotdotnet.com/cbrumme/

 
Jeroen Frijters
Christopher Brumme
Mon 10/13/2003 09:38
RE: Class constructor ordering

Hi Chris,

 

I'm safely back home again. I enjoyed meeting you, thanks.

 

You wrote:

> Here’s an example of a program which – I think – would fail a Java

> conformance test.

 

It doesn't. It throws an ExceptionInInitializerError (caused by NullPointerException) on Sun's JRE 1.4.1 (and on IKVM). I just reread the JVM specification section on class initialization and it explicitly states that the base class initializer runs *after* starting the initialization of the current class.

 

When talking to you I felt kind of stupid for not having noticed a problem with class initialization (other than the deadlock vs seeing uninitialized state issue), but now I'm a little more confident again that I didn't miss anything.

 

At the PDC I'm having dinner with Lee [Redacted] and some of the J# developers (and Miguel [Redacted]), I'll ask if they can explain the issue they are running into. BTW, if you'd like to join us that would be cool (but maybe you'd need to invite a lawyer too ;-)). I don't know what night it'll be, but it will be after the "alternate programming languages" BOF.

 

> On a conforming implementation, I suspect that A’s .cctor executes

> before B’s.  So the call from A’s .cctor to B.testF can trigger B’s

> .cctor.

 

Fortunately (for me) this turns out not to be the case and the JVM algorithm is implementable on the CLR. I'm trying to think of a reason why you'd want the behavior you're describing (vs the specified JVM behavior), but it's early and I've only had one cup of coffee ;-)

 

> On the CLR, B’s .cctor will start executing first.  If B’s .cctor

> chains to A’s .cctor, then we will consider this to be a cycle.  We

> will allow the thread to see uninitialized state in B (the alternative

> now being a deadlock).

 

Java does the same thing if the access is on the same thread, but in multithreaded scenarios it is possible to deadlock (whereas on the CLR you'd still see the unitialized state).

 

Regards,

Jeroen

 
Christopher Brumme
Jeroen Frijters
Tue 10/14/2003 07:12
RE: Class constructor ordering

So those guys complained that we weren't conforming, without checking their facts?  I'm glad I told them we wouldn't fix this one for them.

 

I'm not sure about a dinner with the J# folks.  Let me know when you pick an evening.  But I think I need to check before I can accept.  Definitely let's get together one evening during the PDC.  I think Monday is the only night I'm currently committed.

 

Chris.

 
Jeroen Frijters
Christopher Brumme
Tue 10/28/2003 21:07
dinner

Chris,

 

I'm not sure what we're doing for dinner, but I'm going to be at the Ask the Experts "Java and .NET: Migration and Interop" tonight at 7pm (to chat with the J# people). I understand Miguel is also going to be there, so maybe we'll do dinner afterwards, but I'm not sure.

 

Regards,

Jeroen

 
Jeroen Frijters
Christopher Brumme
Sat 11/15/2003 14:02
weak references

Hi Chris,

 

I finally got around to implementing weak reference support in IKVM and I thought you might be interested in what I found.

 

First a little bit of introduction. Java has three types of weak references:

- SoftReference: Only cleared when GC is about to run out of memory

- WeakReference: Like a short .NET System.WeakReference

- PhantomReference: A long weak reference, but the referent is not retrievable (!)

 

In addition to typical weak reference functionality, these classes also support a ReferenceQueue notion. When a reference gets cleared by the GC, it can optionally post the reference into a ReferenceQueue. PhantomReference is only useful in combination with a ReferenceQueue. The PhantomReference object gets added to the queue after the referent is finalized, the other two get added when they are cleared.

 

(BTW, all three classes derived from java.lang.ref.Reference)

 

Here are my implementation experiences:

I couldn't find a way to implement SoftReference, so at the moment I treat them the same as a WeakReference. The reference part of WeakReference and PhantomReference was easy.

 

The trickiest was to find a way to detect that a reference is cleared so that it can be enqueued.

 

I came up with the following implementation of java.lang.ref.Reference (pseudo code):

 

abstract class Reference {

  private cli.System.WeakReference referent;

  private ReferenceQueue queue;

 

  Reference(object o, ReferenceQueue queue) {

    referent = new cli.System.WeakReference(o, this instanceof PhantomReference);

    this.queue = queue;

    if(queue != null) new QueueWatcher();

  }

 

  private class QueueWatcher {

    protected void finalize() {

      if(referent.IsAlive) {

        if(!cli.System.Environment.HasShutdownStarted) {

          new QueueWatcher();

        }

      } else {

        enqueue();

      }

    }

  }

 

  public void enqueue() {

    queue.add(this);

  }

 

  public object get() {

    return referent.Target;

  }

 

  public void clear() {

    referent.Target = null;

  }

}

 

(In case your Java is a bit rusty, the QueueWatcher inner classes has an implicit reference to the instance of the outer class that constructed it.)

 

This works, but there are some obvious problems with the QueueWatcher hack:

- It's a hack

- It's inefficient

- A future JIT could decide to compile "new QueueWatcher()" as "(new QueueWatcher()).finalize()"

 

However, overall, I'm pretty happy with it.

 

I've thought about what functionality the CLR could provide to make this easier, but I haven't been able to come up with anything that isn't just a rip off of the Java model.

 

Regards,

Jeroen

 
Jeroen Frijters
Christopher Brumme
Mon 11/17/2003 09:21
Partially trusted code can leak GCHandles
test.il

Hi Chris,

 

I found a bug in System.WeakReference. The class is not sealed and neither is its finalizer, so any untrusted code can create an unbounded number of GCHandles that will never get cleaned up (by creating a subclass of WeakReference that overrides Finalize and not calls the base class Finalize).

 

See attached code for a reproduction.

 

Regards,

Jeroen

 
Christopher Brumme
Jeroen Frijters
Mon 11/17/2003 17:58
RE: Partially trusted code can leak GCHandles

Thanks.  I'll file the bug.  Unfortunately, sealing the class now would be a breaking change.  And Denial of Service isn't considered a strong security threat -- particularly since defense against denial of service attacks remains so difficult.

 

It won't surprise me if this bug is indefinitely punted.

 

Chris.

http://blogs.gotdotnet.com/cbrumme/

 
Jeroen Frijters
Christopher Brumme
Mon 11/17/2003 18:15
RE: Partially trusted code can leak GCHandles

It is kind of nasty that the leak stays around even after the AppDomain is unloaded.

 

It would be fixable by making the Finalize method final. Unfortunately, C# doesn't allow this (big design flaw, IMHO), but I guess that would also be a breaking change. How about a full trust inheritance demand on the Finalize method? That way untrusted code can't leak and it's hard to come up with a useful scenario where partially trusted code needs a Finalize anyway.

 

Regards,

Jeroen

 
Christopher Brumme
Jeroen Frijters
Mon 11/17/2003 19:25
RE: Partially trusted code can leak GCHandles

The leak shouldn't stay around after the AppDomain is unloaded.  We bulk unmap the entire handle table for the doomed AppDomain.  (Each AppDomain gets its own handle table, though we have to be clever with handles to objects that are agile with respect to AppDomain boundaries).  If this were not the case, we would crash on the first GC after the unload since you can be sure that the MethodTables of the instances have been unmanpped.

 

I think that SQL Server may have a pre-verification pass to reject "safe" code that defines Finalize methods.  If so, Finalize would remain available to some partially trust assemblies -- but not to the assemblies with the lowest possible trust.

 

It would be interesting to formalize this, so that a process host could decide whether it wants to make itself immune to this sort of denial of service.  There is a tension here, because the host can no longer run arbitrary code that was written for some other environment.

 

I'll add some of your thoughts to the bug report.

 

Chris.

http://blogs.gotdotnet.com/cbrumme/

 
Jeroen Frijters
Christopher Brumme
Mon 11/17/2003 19:49
RE: Partially trusted code can leak GCHandles

> The leak shouldn't stay around after the AppDomain is unloaded.  We

> bulk unmap the entire handle table for the doomed AppDomain.

> (Each AppDomain gets its own handle table, though we have to be clever

> with handles to objects that are agile with respect to AppDomain

> boundaries).  If this were not the case, we would crash on the first

> GC after the  unload since you can be sure that the MethodTables of

> the instances have been unmanpped.

 

Aha. My test case used an empty string object (ldstr "") to put in the WeakReference, so that probably explains why they stayed around (I checked with perfmon).

 

Regards,

Jeroen

 
Jeroen Frijters
Christopher Brumme
Tue 2/3/2004 09:22
Potential finalization security issue
extendhack.il

Hi Chris,

 

I'm glad to see you're blogging againg. The reason I'm e-mailing is that I've found a new finalization issue. The attached (verifiable) code creates a subclass of System.Reflection.Assembly (an unsealed type, but the constructor is inaccessible), and proceeds to create an instance of this subclass. This is obviously not a good thing :-)

 

This one is easy to fix (the verifier should treat types without an accessible constructor as sealed), but when the constructor is accessible you can still create instances without calling it and I think this is much harder to solve.

 

I would like to take this opportunity once again to plead for restricting Finalize to fully trusted code. I know it is a breaking change, but for Whidbey you might be able to get away with it, with every release it will become harder.

 

BTW, I think Peter [Redacted] ([Redacted]@microsoft.com) already filed a bug for this (I discovered this while having a private e-mail discussion with him).

 

Regards,

Jeroen

 
Christopher Brumme
Jeroen Frijters
Tue 2/3/2004 19:24
RE: Potential finalization security issue

Thanks.  I changed your test slightly to avoid the race with the

finalizer:

 

    IL_000b:  call       void [mscorlib]System.GC::Collect()

Add>>      call       void

[mscorlib]System.GC::WaitForPendingFinalizers()

    IL_0010:  ldsfld     class ExtendHack ExtendHack::cheat

 

I'm guessing that the finalizer was getting scheduled for you ahead of the app's main thread, but for me it wasn't running.

 

Clearly we need to fix this bug.  But we will also be talking about the larger issue of whether finalization should be revoked from partial trust, and the behavior of partially constructed objects.

 

I'll get back to you with the decision.  And I really appreciate the heads up here.

 

Chris.

http://blogs.msdn.com/cbrumme

 
Jeroen Frijters
Christopher Brumme
Thu 2/5/2004 11:35
RE: Potential finalization security issue

Hi Chris,

 

> Thanks.  I changed your test slightly to avoid the race with the

> finalizer:

>

>     IL_000b:  call       void [mscorlib]System.GC::Collect()

> Add>>      call       void

> [mscorlib]System.GC::WaitForPendingFinalizers()

>     IL_0010:  ldsfld     class ExtendHack ExtendHack::cheat

>

> I'm guessing that the finalizer was getting scheduled for you ahead of

> the app's main thread, but for me it wasn't running.

 

Yeah, I knew I should've put that in. I thought of it as I was editing the IL, but my mind has been made weak by Visual Studio .NET intellisense, and I was too lazy to look up the exact method name ;-)

 

> Clearly we need to fix this bug.  But we will also be talking about

> the larger issue of whether finalization should be revoked from

> partial trust, and the behavior of partially constructed objects.

 

I think it would also be a good idea to restrict GC.SupressFinalize/ReRegisterForFinalize to fully trusted code.

 

Something completely different, but I notice that the GC class has ClearCache event for the Reflection cache, it would be really nice if similar functionality would be available in the public API.

 

> I'll get back to you with the decision.  And I really appreciate the

> heads up here.

 

Thanks!

 

Regards,

Jeroen

 
Jeroen Frijters
Christopher Brumme
Tue 2/24/2004 12:28
Finalization

Hi Chris,

 

Thanks for the blog about finalization. As you probably would have expected, I disagree with you on the need for partially trusted code to have finalizers. I just don't see a compelling scenario. If you really want to enable pooling in this way (IMHO a bad design pattern), add notification to weak references (like Java has). This works much better (although I must admit that I don't see a way to implement this without making GCHandles less efficient).

 

Another issue wrt finalization that bothers me is the fact that objects that override Finalize are automatically added to the RegisteredForFinalization queue. For some programming languages/environments it might make sense to make this decision on an instance by instance basis, instead of class based. It would be nice if there was an attribute to mark your type "DontRegisterForFinalize". That way you could call ReRegisterForFinalize only when needed. Of course, this really should have been the default, but its too late for that now ;-) That also would have prevented the JIT bug where it registers an object for finalization that shouldn't have been created yet.

 

 

>>However, many other denial of service attacks are possible from partial trust, so this is uninteresting.<<

 

While I can appreciate the pragmatism behind this, it still saddens me to read this ;-)

 

BTW, any chance I can get a look at the Whidbey changes (I'm on the alpha and under NDA), or will the (rumored soon) next alpha drop contain the new functionality already?

 

Regards,

Jeroen

 
Christopher Brumme
Jeroen Frijters
Tue 2/24/2004 15:41
RE: Finalization

Yes, I knew you wouldn't be happy about the "finalization from partial trust".  But ultimately we should be able to fix the denial of service issues here.  In fact, the work was on our Whidbey schedule.  Of course, it was eventually punted to a future release.

 

Delegates use an internal finalization mechanism that is instance-based.

When we marshal a delegate to unmanaged code, we create a thunk for the eventual unmanaged->managed callbacks.  We share these thunks pretty aggressively, and we drop refcounts on them as the marshaled delegates are collected.  The vast majority of delegates are never marshaled out in this fashion, so they don't need cleanup side effects.

 

I hadn't realized this was a common pattern.  What kind of places do you run into this?  Suppress/ReRegisterForFinalization gives you some instance-level control.  But it clearly expects the common case to be that all instances of a type require finalization.

 

As for requiring the program to explicitly ask for finalization during construction, I suspect we would quickly get customer requests for a type-level marker.  With all the rules about multiple calls to a finalize method, possibly concurrently with other method calls, any Finalize method must be written in an extremely careful way.  If the Finalize method isn't robust in the face of partially constructed or partially destructed objects, it's broken.  (Yes, I bet that many of our FX Finalize methods would not pass this test).  Personally, I like the guarantee that if the object was even allocated, it will Finalize.

 

I would love to send you the missing pieces from the finalization and hosting blogs.  Who did you deal with on the NDA?  Are you on any of our advisory councils?  If you give me a name, I'll follow up over here.

 

To be honest, I haven't been paying any attention to the schedule.  So although I think we are making an interim build publicly available, I haven't looked to see when it is happening.  I should follow up.  If it contains the code I was writing about, I won't have to wait for Beta before I can post the missing pieces.

 

It's really too bad that you don't want to come work on the CLR.

 

Chris

http://blogs.gotdotnet.com/cbrumme

 
Jeroen Frijters
Christopher Brumme
Tue 2/24/2004 16:46
RE: Finalization

Hi,

 

> Yes, I knew you wouldn't be happy about the "finalization from partial

> trust".  But ultimately we should be able to fix the denial of service

> issues here.  In fact, the work was on our Whidbey schedule.

> Of course, it was eventually punted to a future release.

 

I don't mind actually, as long as it gets done for Orcas ;-) I think that DoS prevention willo be much more important for Longhorn.

 

> Delegates use an internal finalization mechanism that is

> instance-based.

[...]

> I hadn't realized this was a common pattern.  What kind of places do

> you run into this?

 

It is something that one of the Eiffel developers asked about and I ran into it yesterday when I was working on my object model mapping stuff for IKVM. I rename some methods (Equals -> equals, GetHashCode -> hashCode), this is done based on some XML mapping file, so obviously the code also emitted a Finalize -> finalize rename (i.e. a method named finalize that overrides Finalize and doesn't do anything except call the base class method). Obviously not a good idea (the workaround is, of course, trivial). This sort-of coincided with my previous thinking on finalization (and the bug I mentioned) to form the idea that finalization should be more flexible. So, it's one of those "nice to have" features, but as far as I'm concerned it has no priority whatsoever. I'm sure you get lots of these "it would be really cool if..." requests.

 

> Suppress/ReRegisterForFinalization gives you some instance-level

> control.  But it clearly expects the common case to be that all

> instances of a type require finalization.

 

Yes, the Eiffel guys tried calling SuppressFinalization in their constructor, but performance was still pretty bad.

 

> As for requiring the program to explicitly ask for finalization during

> construction, I suspect we would quickly get customer requests for a

> type-level marker.

 

Yeah, I thought it would be solvable in the compiler, but now I realise that because object allocation/constructor invocation is a single instruction you cannot do that.

 

> If the Finalize method isn't robust in the face of partially

> constructed or partially destructed objects, it's broken.

 

I agree and that certainly wasn't my motivation for the feature.

 

> Personally, I like the

> guarantee that if the object was even allocated, it will Finalize.

 

Well, there is the downside that it makes the whole base constructor invocation verification (almost) useless. This is also where the instance based finalization would be helpful, you could override Finalize with an empty method and make it final to prevent subclasses from obtaining uninitialized instances.

 

> I would love to send you the missing pieces from the finalization and

> hosting blogs.  Who did you deal with on the NDA?

 

Barbara [Redacted] handled the NDA. BTW, I haven't read the hosting blog yet, I hope to do that tonight.

 

> Are you on any of our advisory councils?

 

No, never been asked.

 

> To be honest, I haven't been paying any attention to the schedule.  So

> although I think we are making an interim build publicly available, I

> haven't looked to see when it is happening.  I should follow up.  If

> it contains the code I was writing about, I won't have to wait for

> Beta before I can post the missing pieces.

 

I don't know, but it might be that the next alpha (if there is one) will not be made public. We got the current version in July and weren't allowed to talk about them until the PDC.

 

> It's really too bad that you don't want to come work on the CLR.

 

If you get clearance to send me the design documents, I'd love to give my feedback.

 

Regards,

Jeroen

 
Jeroen Frijters
Christopher Brumme
Thu 2/26/2004 09:12
RE: Finalization

Hi,

 

I just realized that my reference to the JIT bug in my last mail probably didn't make any sense to you. The bug I was talking about was not the verifier bug I reported earlier, but a bug in the JIT I discovered a while ago: http://weblog.ikvm.net/CommentView.aspx?guid=b199682c-6546-46fd-bead-bc355c4fc21f

 

Summary: When you do new "Foo(GetConstructorArgs())", the JIT allocates the object before calling GetConstructorArgs and that has the side effect of running finalization on an object that shouldn't have been allocated yet (if GetConstructorArgs throws an exception).

 

Regards,

Jeroen

P.S. Any news on the NDA?

 
Christopher Brumme
Jeroen Frijters
Thu 2/26/2004 19:07
RE: Finalization

Thanks.  I hadn't seen this case.  I see Peter already logged the bug.

 

I hadn't heard back from Barbara yet, so I just pinged her in email again.  I'll try phoning her later today if I don't get a reply in email.

 

Chris

http://blogs.gotdotnet.com/cbrumme

 
Christopher Brumme
Jeroen Frijters
Mon 3/1/2004 20:42
RE: Finalization

I've added a bunch of legalese to remind you of your NDA.  Hopefully I did it correctly.  Obviously you shouldn't show these docs to anyone.

And when I post the updated blogs upon disclosure, I would recommend throwing away these attached copies.

 

 

Chris.

http://blogs.msdn.com/cbrumme

 
Jeroen Frijters
Christopher Brumme
Tue 3/2/2004 09:59
RE: Finalization

Hi,

 

> I've added a bunch of legalese to remind you of your NDA.  Hopefully I

> did it correctly.  Obviously you shouldn't show these docs to anyone.

> And when I post the updated blogs upon disclosure, I would recommend

> throwing away these attached copies.

 

Thanks, I won't and I will, respectively.

 

BTW, some of this stuff is already public (although somewhat vague):

http://longhorn.msdn.microsoft.com/lhsdk/ref/ns/system.runtime.interopservices/c/safehandle/safehandle.aspx

 

After reading it through (but not letting it sink in yet), my first thought is that I like it. Of course, I also have a few (contradictory) questions:

1) Why was CF generalized? Why is SafeHandle not enough?

2) Why is CFO a base class and not an attribute?

3) Wouldn't it be a good idea to require CFO derived classes to be sealed?

 

>>This means that a buffer flush can precede the close of the underlying

>>handle.<<

 

IHMO promoting this idea is bad. Depending on finalizers to flush buffers will still cause hard to find bugs. Sure the typical case is solved with CF, but what about layered streams (e.g. a block encryption stream on top of a buffering stream). Or user defined finalizers?

This reminds me of another use for instance finalizers, when you put assertions in your finalizers (e.g. assert that the buffer is empty, instead of flushing it automatically), it would be nice if that was something that could be enabled with a switch (somewhat like a customer debug probe).

 

>>This would be an intolerable situation for a Security feature, but

>>it’s perfectly acceptable when we’re just trying to increase the

>>scalability and reliability of naively written database

>>applications.<<

 

I'm a bit worried that someone, somewhere will at some point in the future rely HPA for security.

 

>>In Whidbey, it is possible to avoid inducing asynchronous Aborts onto

>>threads that are performing backout (i.e. filter, finally, catch or

>>fault blocks) or that hold locks.<<

 

I assume you mean that on Whidbey the runtime will never introduce asynchronous Aborts into threads that are performing backout or holding locks, but the quoted text isn't entirely clear to me.

 

I'll probably have more to say when I've thought about it a bit more.

 

Regards,

Jeroen

 
Christopher Brumme
Jeroen Frijters
Tue 3/2/2004 20:54
RE: Finalization

>> Why was CF generalized? Why is SafeHandle not enough?

 

The System.Data folks have lighter-weight resources that aren’t handle-based, like buffers with offsets.  For these resources, they still want guaranteed cleanup, reduced graph promotion, GC.KeepAlive behavior and enforcement of the above based on the type system.  But they don’t need the cost of preventing handle recycling, because they don’t expose these handles out directly to the app.  Since the app cannot control their lifetime directly, there are no risks so it isn’t worth paying the price of interlocking on each call.

 

>> Why is CFO a base class and not an attribute?

 

If it was an attribute, you would add it to some existing class.  Then we won’t get the benefits of reduced graph promotion.

 

>> Wouldn't it be a good idea to require CFO derived classes to be

 

>> sealed?

 

Each subtype needs to judge this for itself.  But yes, in general that makes sense.  Note that you must be trusted to subtype one of these types (transitively) so some of the usual concerns about unsealed types don’t apply here.

 

>> IHMO promoting this idea is bad. Depending on finalizers to

 

>> flush buffers will still cause hard to find bugs.

 

I think everyone agrees with you.  The issue was that our critical finalizer code actually made the problem worse than it used to be.  We don’t want you creating references between the file and the stream instances and we don’t want the finalizable objects pointing at anything, because of graph promotion.  So we added this tweak just so we wouldn’t cause regressions.

 

>> I'm a bit worried that someone, somewhere will at some point in

 

>> the future rely HPA for security.

 

This is a big concern.  Even within the company we’ve had a lot of confusion on this point.

 

>> I assume you mean that on Whidbey the runtime will never

 

>> introduce asynchronous Aborts into threads that are performing

 

>> backout or holding locks, but the quoted text isn't entirely

 

>> clear to me.

 

I wish I had said that.  It would be a reasonable position for us to take, but it would be a breaking change.  Instead, we give you ways to abort nicely.  And we may add a Customer Debug Probe for the legacy case, so the dev understands the risk of what he is doing.

 

This is one of those nasty places where we are trading off long term correctness of the platform against the short term costs of breaking application compatibility.  It’s still being debated.

 

 

Chris.

 

http://blogs.msdn.com/cbrumme

 
Jeroen Frijters
Christopher Brumme
Wed 3/3/2004 09:04
RE: Finalization

Hi,

 

Thanks for the explanations!

 

> >> Why was CF generalized? Why is SafeHandle not enough?

>

> The System.Data folks have lighter-weight resources that aren’t

> handle-based, like buffers with offsets.

 

Ah I see. i had forgotten about those (not the System.Data specific ones, I didn't know about them, but the general idea of unmanaged memory buffers). I actually spent quite a bit of time thinking about this (in relation to memory mapped files, a tricky problem in a managed environment), so I should have remembered that.

 

> >> Why is CFO a base class and not an attribute?

>

> If it was an attribute, you would add it to some existing class.  Then

> we won’t get the benefits of reduced graph promotion.

 

You can apply the same reasoning to subclassing. Arguably, a subclasser is more likely not to notice the parentage of its parent than someone deliberatly applying the attribute. How about the following ideas? If an attribute is used, all subclasses should also have the attribute (not as a security precaution, but to make sure the developer of the subclass is aware of the issues). Or, in the CFO base class scheme, you could only allow subclassing in the same assembly (or "friend" assemblies).

 

I know that you shouldn't have outward facing CF classes, but I'm sure there'll be third party libraries doing the wrong thing and it would be good to try to minimize the opportunity for that.

 

> >> Wouldn't it be a good idea to require CFO derived classes to be

> >> sealed?

>

> Each subtype needs to judge this for itself.  But yes, in general that

> makes sense.  Note that you must be trusted to subtype one of these

> types (transitively) so some of the usual concerns about unsealed

> types don’t apply here.

 

I fear that most people will still be developing full-trust-only code in the Whidbey "timeframe" (to use MS speak ;-)), so while not a security issue, it'll be good to guide people in the right direction.

 

> >> IHMO promoting this idea is bad. Depending on finalizers to flush

> >> buffers will still cause hard to find bugs.

>

> I think everyone agrees with you.  The issue was that our critical

> finalizer code actually made the problem worse than it used to be.  We

> don’t want you creating references between the file and the stream

> instances and we don’t want the finalizable objects pointing at

> anything, because of graph promotion.  So we added this tweak just so

> we wouldn’t cause regressions.

 

OK, I see what you're saying now. Basically you are just giving an example of a potential regression that would have been created if you didn't have the ordering rule (which makes *perfect* sense BTW). If I may make a suggestion, it would be great if you added a little clarification to the text that explains why it is still a bad idea to depend on this.

 

> >> I'm a bit worried that someone, somewhere will at some point in the

> >> future rely HPA for security.

>

> This is a big concern.  Even within the company we’ve had a lot of

> confusion on this point.

 

The name of the attribute and the fact that it is a CAS attribute doesn't help. It's probably too late, but have you considered renaming it to something like HostScalabilityAdviceAttribute (or whatever)?

 

> >> I assume you mean that on Whidbey the runtime will never introduce

> >> asynchronous Aborts into threads that are performing backout or

> >> holding locks, but the quoted text isn't entirely clear to me.

>

> I wish I had said that.  It would be a reasonable position for us to

> take, but it would be a breaking change.  Instead, we give you ways to

> abort nicely.

 

I see. So presumably there'll be a new method Thread.SafeAbort().

 

> This is one of those nasty places where we are trading off long term

> correctness of the platform against the short term costs of breaking

> application compatibility.  It’s still being debated.

 

I know it's not a democracy, but I'll cast my vote anyway ;-) I'd like to see the existing Abort method fixed (funnily enough, in both senses of the word, "corrected" and like having your dog fixed).

 

As an aside, I got a notice today on the subclassing by finalization bug, saying that it didn't work in partial trust (verification exception), so I'm glad to see it has been fixed :-)

 

As another aside, I know it is not your area, but there are two unnecessary restrictions in the spec that I'd like to see fixed. I'd ask [Redacted] but at the PDC I got the impression he doesn't like me very much (no idea why, maybe it's just in my head). Anyway, the CLI spec says that all interfaces members must be public, this is a bogus rule for static members, in particular for .cctor. A second point is that the spec disallows nested types in interfaces, this restriction also doesn't make much sense (and, in fact, the CLR doesn't enforce it).

The relevant references are:

 

Partition I, Working Draft 2.2, June 2003, section 8.5.3.2:

"1. Members defined by an interface shall be public."

 

Partition II, Working Draft 2.2, June 2003, section 9.6:

"Interfaces may be nested inside of classes and value types, but classes and value types shall not be nested inside of interfaces."

 

If you agree that these restrictions are pointless, maybe you could forward it to the appropriate person. However, as usual, feel free to ignore me. I won't hold it against you.

 

Regards,

Jeroen

 
Jeroen Frijters
Christopher Brumme
Sat 3/13/2004 23:28
Main thread keeps TLS alive when it's undead
Class1.cs

Hi,

 

I ran across an interesting bug (I think). When the main thread ends, I think its thread local storage stays alive. I assume this is because the main thread actually stays around to manage the CLR life cycle. See the attached repro.

 

I ran into this when I was trying to hack my own thread death event, by using finalization of a TLS object as the event trigger. Really lame, but I couldn't think of anything better.

 

BTW, the Longhorn command line shell preview released a new Whidbey build (2.0.31113) that includes the CriticalFinalizerObject. I installed it, but haven't played with it much, but I intend to.

 

Regards,

Jeroen

 
Christopher Brumme
Jeroen Frijters
Tue 3/23/2004 19:21
RE: Finalization

Your ECMA suggestions are now on our internal list.  Obviously I cannot speak to whether the external ECMA committee will accept them, since it is made up of representatives from many companies.  However, they seem non-controversial and I would guess that they will pass.

 

As for feeling that [Redacted] didn't like you, I would not read anything into that.  I'm sure you just caught him at a bad time.  Many of us on the team have been through similar circumstances with him, when we caught him at a bad time.

 

If it's not obvious, there are a bunch of people on the CLR team who know who you are, what you do, and hope that one day you might consider joining the team.  (I realize that this isn't in the cards right now, but we're all in this for the long term).

 

Chris.

http://blogs.msdn.com/cbrumme

 
Jeroen Frijters
Christopher Brumme
Fri 4/2/2004 17:14
RE: Finalization

Hi,

 

> Your ECMA suggestions are now on our internal list. 

 

Thanks!

 

> If it's not obvious, there are a bunch of people on the CLR team who

> know who you are, what you do, and hope that one day you might

> consider joining the team.  (I realize that this isn't in the cards

> right now, but we're all in this for the long term).

 

Thanks. If I ever go looking for another job, the CLR team would definitely be at the top of my list :-)

 

BTW, The "Visual Studio 2005 Community Technology Preview" was given out at VSLive, so I think you can update your blog entries now.

 

Regards,

Jeroen

 
Christopher Brumme
Jeroen Frijters
Tue 4/27/2004 00:00
RE: Finalization

I finally posted the updates.

 

Chris.

http://blogs.msdn.com/cbrumme

 
Jeroen Frijters
Christopher Brumme
Wed 8/11/2004 15:59
Whidbey verifier change

Hello Chris,

 

I just noticed that in beta 1 the verifier complains about interchanging references from a System.Byte[] with a System.SByte[]. According to the 1.0 CLI spec this was allowed and I use this feature in IKVM to make interoperating with Java's signed byte arrays easier with the CLS unsigned byte arrays.

 

Is this an intentional change and is the CLI spec going to be updated?

 

I actually think this is a good change, when I originally discovered that the verifier allowed mixing the two I complained about it (but then I also started using it ;-)).

 

BTW, what's the deal with the co-/contravariant generic type parameters? Yesterday I found out that it is already working (http://weblog.ikvm.net/), but there appears to be absolutely no documentation. Has it not yet been decided whether that feature will make it into the final release?

 

Thanks.

 

Regards,

Jeroen

 
Christopher Brumme
Jeroen Frijters
Wed 8/11/2004 16:21
RE: Whidbey verifier change

I'll check into the Byte[]/SByte[] change and get back to you.  I will be surprised if the change was intentional.

 

The co-/contravariant generic type parameters are definitely going to be in the final release.  However, none of the MS languages except ILASM have any support for them.  So you aren't going to see them used in WinFX either.

 

I assume that Eiffel uses them, but haven't checked.

 

I'm personally nervous whenever we have a feature for which we have no internal consumers.  In the past, we've discovered security or functionality holes when we've done this.

 

It doesn't look like you need documentation for this feature... you've already figured out exactly how it works.  I'm told that it was documented for the ECMA submission on generics.  But I'll also mention it as a doc hole to UE.

 

I saw further down your blog that we don't properly inject assemblies into all the AppDomains that use them, when the assemblies are being shared.  A lot of work went into fixing that one for Whidbey.  But, as far as I know, you are the only person who actually noticed the old broken behavior.

 

Chris.

http://blogs.msdn.com/cbrumme

 
Christopher Brumme
Jeroen Frijters
Wed 8/11/2004 16:26
FW: Whidbey verifier change

Oops.  I forgot to say something friendly like:

 

I hope everything is going well for you and that you are enjoying the summer.  It's great to see Whidbey now on a solid path to shipping.

Obviously we are still some distance from saying the same thing about Orcas.  I'll be interested in hearing your views on Whidbey generally.

From over here, it feels like we put a ton of effort and features into it.  But the customer is always right and so I'm curious to know if we left out some important stuff that's going to cause pain out there.

 

Chris.

http://blogs.msdn.com/cbrumme

 
Jeroen Frijters
Christopher Brumme
Wed 8/11/2004 16:49
RE: Whidbey verifier change

 

> I'll check into the Byte[]/SByte[] change and get back to you.  I will

> be surprised if the change was intentional.

 

Thanks.

 

> The co-/contravariant generic type parameters are definitely going to

> be in the final release.  However, none of the MS languages except

> ILASM have any support for them.  So you aren't going to see them used

> in WinFX either.

>

> I assume that Eiffel uses them, but haven't checked.

 

That's what I assumed too. I only discoverd them through the System.GenericParameterAttributes enum. I then had to twiddle with the bits of an assembly and use ILDASM to figure out the syntax ;-)

 

> I'm personally nervous whenever we have a feature for which we have no

> internal consumers.  In the past, we've discovered security or

> functionality holes when we've done this.

 

I know exactly what you mean. This seems like a relatively straight forward feature though.

 

> It doesn't look like you need documentation for this feature... you've

> already figured out exactly how it works.  I'm told that it was

> documented for the ECMA submission on generics.  But I'll also mention

> it as a doc hole to UE.

 

I don't think the ECMA submissions are public yet. I'd love to be proven wrong though.

 

> I saw further down your blog that we don't properly inject assemblies

> into all the AppDomains that use them, when the assemblies are being

> shared.  A lot of work went into fixing that one for Whidbey.  But, as

> far as I know, you are the only person who actually noticed the old

> broken behavior.

 

Yeah, that was a very strange and tricky bug to find (I didn't actually discover it, a user of IKVM discovered it and I had to repro it based on a vague description of the problem).

 

> Oops.  I forgot to say something friendly like:

 

:-)

 

> I hope everything is going well for you and that you are enjoying the

> summer.

 

We just had a heat wave (5 days above 25 degrees C, three of which above 30 deg C). That wasn't very pleasant, fortunately it is a bit cooler now. We don't generally have AC here (except in cars). I hope you're having nice summer like last year when I was there for the Whidbey SDR.

 

> It's great to see Whidbey now on a solid path to shipping.

 

Definitely. Beta 1 looks very good.

 

> Obviously we are still some distance from saying the same thing about

> Orcas.

 

If there is going to be a SDR (or even something earlier than that), I'd definitely be interested.

 

> I'll be interested in hearing your views on Whidbey generally.

> From over here, it feels like we put a ton of effort and features into

> it.  But the customer is always right and so I'm curious to know if we

> left out some important stuff that's going to cause pain out there.

 

In general I'm very happy. For IKVM I would have liked to have seen some more improvements to Reflection (and Reflection.Emit).

 

Regards,

Jeroen

 
Jeroen Frijters
Christopher Brumme
Fri 8/13/2004 10:29
RE: Whidbey verifier change

Hi,

 

> I'll check into the Byte[]/SByte[] change and get back to you.  I will

> be surprised if the change was intentional.

 

Any news?

 

Also, I ran across a Whidbey regression relating to type names with double backslashes in them in Reflection.Emit scenarios. I haven't been able to build a concise repro, but beforing spending a lot of time on it I wanted to check with you if it might be a known issue already.

 

So far I see two, probably related, issues:

 

AssemblyName name = new AssemblyName();

name.Name = "test";

AssemblyBuilder ab = AppDomain.CurrentDomain.DefineDynamicAssembly(name, AssemblyBuilderAccess.Run); ModuleBuilder mod = ab.DefineDynamicModule("test"); TypeBuilder tb = mod.DefineType(@"foo\\1"); Console.WriteLine(tb.FullName); Console.WriteLine(tb.CreateType().FullName);

 

The above code prints out:

foo\\1

foo\\\\1

 

The other problem is that in a larger program I get a System.Runtime.InteropServices.COMException: Exception from HRESULT: 0x80131130. This is also caused by blackslashes in the type name (I use blackslashes to mangle names that would otherwise clash), because the problem goes away if I use a forward slash in the name mangling. I tried to build a small repro, but so far I couldn't get that to fail.

 

Thanks.

 

Regards,

Jeroen

 
Christopher Brumme
Jeroen Frijters
Fri 8/13/2004 16:07
RE: Whidbey verifier change

I hadn't heard any resolution from the verifier dev yet, so I just ping'ed him again.  Given the ECMA rules, I think we'll have no choice but to fix this.

 

Thanks for the repro on the emit problem.  I'll send this to the reflection emit dev.  Given that we have a customer app that has been broken, this is likely something we want to / have to fix, also.

 

Chris.

http://blogs.msdn.com/cbrumme

 
Christopher Brumme
Jeroen Frijters
Fri 8/13/2004 22:42
RE: Whidbey verifier change

For the Byte[]/SByte[] problem, I see where the code got broken when we tightened up the rules for generics.  I filed a bug and we'll get it fixed.

 

For the \\ change, the dev cleaned up the grammar for type names and inadvertently broke this.  Given that we broke a customer's app, he is now trying to achieve his cleanup in a non-breaking way.  So we're going to try to fix this one, but I could imagine he might get stuck.  If that happens, I'll get back to you.  Otherwise you should assume that we will fix it.

 

Thanks for pointing out these bugs.

 

Chris

http://blogs.msdn.com/cbrumme

 
Jeroen Frijters
Christopher Brumme
Tue 8/17/2004 14:32
Repro: System.Runtime.InteropServices.COMException (0x80131130)
Program.cs

Hi,

 

Since the e-mail exchange with Chris [Redacted] wasn't very productive, I spent a little time to repro the COMException I was getting.

 

The attached program works on Everett, but throws on Whidbey.

 

This is clearly related to the backslash in the type name (if you remove that, it works on both Everett and Whidbey).

 

BTW, was it my lack of clarity in my mails to Chris [Redacted], or did you understand my proposed solution, or did I totally miss the point?

 

Regards,

Jeroen

 
Christopher Brumme
Jeroen Frijters
Mon 8/23/2004 20:00
RE: Repro: System.Runtime.InteropServices.COMException (0x80131130)

I apologize for taking so long to respond.  Something came up on this end.

 

I gave your program to Chris [Redacted], without any of your below email.

 

It was not your lack of clarity.  There are two people on reflection, and one of them recently [personal information removed].  He hasn't been back in the office yet.

 

I'm sure the bug will get resolved properly when he's back in the office, if Chris doesn't get to it before then.

 

Chris

http://blogs.msdn.com/cbrumme

 
Jeroen Frijters
Christopher Brumme
Mon 8/23/2004 20:04
RE: Repro: System.Runtime.InteropServices.COMException (0x80131130)

Hi,

 

> I apologize for taking so long to respond.  Something came up on this

> end.

 

No problem.

 

> It was not your lack of clarity.  There are two people on reflection,

> and one of them recently [personal information removed]

> He hasn't been back in the office yet.

 

Do you mean [Redacted]? I hope he's doing alright.

 

> I'm sure the bug will get resolved properly when he's back in the

> office, if Chris doesn't get to it before then.

 

Thanks.

 

Regards,

Jeroen

 
Christopher Brumme
Jeroen Frijters
Mon 8/23/2004 20:18
RE: Repro: System.Runtime.InteropServices.COMException (0x80131130)

[Personal information removed]

 

He sent some email saying he would be in the office later this week.

We'll see.

 

Chris.

http://blogs.msdn.com/cbrumme

 
Jeroen Frijters
Christopher Brumme
Wed 9/8/2004 08:55
Memory model

Hi Chris,

 

Sorry to bother you with this, but this is something that worries me:

http://blogs.msdn.com/grantri/archive/2004/09/07/226355.aspx

 

Either [my understanding of] the CLI spec is seriously broken or I'm not going to be running any mission critical code on my AMD64 machine...

 

Thanks.

 

Regards,

Jeroen

 
Christopher Brumme
Jeroen Frijters
Wed 9/8/2004 15:04
RE: Memory model

You are certainly not bothering me.

 

I was unaware of the blog, but I'm quite aware of the issue.  I had several meetings on this particular bug and I recently found a much more compelling and wide-spread example where this optimization can bite us (thread-safe event firing, where we deliberately use a local to avoid issues with asynchronous RemoveOn callers).

 

My current plan is to get some DCRs approved in time for Beta2 (officially I'm already too late), which would condition these JIT64 optimizations on the presence of a CompilationRelaxationsAttribute.  I'm hoping that we can get the .NET frameworks cleaned up enough to safely add this attribute to our own assemblies, but your code would have to opt-in to this aggressive optimization.  So user assemblies will run correctly between X86 & AMD64, by default.

 

I should stress that this is my plan, rather than the plan of record -- especially given the usual resource constraints.  But I talked to several people about it over the last week and there seems to be enough support for it.

 

Public blogs are definitely a two-edged sword.

 

Chris.

http://blogs.msdn.com/cbrumme

 
Jeroen Frijters
Christopher Brumme
Wed 9/8/2004 15:52
RE: Memory model

> You are certainly not bothering me.

 

Well if I am, don't hesitate to say so.

 

> I was unaware of the blog, but I'm quite aware of the issue.  I had

> several meetings on this particular bug and I recently found a much

> more compelling and wide-spread example where this optimization can

> bite us (thread-safe event firing, where we deliberately use a local

> to avoid issues with asynchronous RemoveOn callers).

 

So the optimization is indeed allowed per the spec? That surprises me quite a bit. Programming is looking more and more like quantum physics :-) A local variable can contain a superposition of multiple values...

 

> My current plan is to get some DCRs approved in time for Beta2

> (officially I'm already too late), which would condition these JIT64

> optimizations on the presence of a CompilationRelaxationsAttribute. 

> I'm hoping that we can get the .NET frameworks cleaned up enough to

> safely add this attribute to our own assemblies, but your code would

> have to opt-in to this aggressive optimization.  So user assemblies

> will run correctly between X86 & AMD64, by default.

 

Is this optimization really that significant that it is worth all this trouble? I honestly don't see anyone adding this attribute to their code (and of the people that would add it, some (most?) probably don't understand the consequences).

 

In any case, the CLI spec should definitely be clarified that this type of optimization is allowed. I'm still trying to find language in the spec that prohibits this optimization :-)

 

How about this:

"Values are stored in locations. A location can hold only one value at a time. All locations are typed. The type of the location embodies the requirements that shall be met by values that are stored in the location. Examples of locations are local variables and parameters. "

 

Interesting thought: A "secure" method must do a volatile read before doing any parameter validation, otherwise the value of the parameter might change after the validation (if the method is inlined and this optimization happens).

 

> Public blogs are definitely a two-edged sword.

 

:-) I hope you get time to post to your blog soon.

 

Another thing that I recently ran into: What's with the System.Runtime.CompilerServices.IsBoxed modifier? Wouldn't it be better to finally implemented boxed references, instead of this hack? I know that would mean that these C++/CLI methods wouldn't be callable by other languages, but as it stand the language interop story isn't that great either (with the ValueType parameter types).

 

Thanks.

 

Regards,

Jeroen

 
Christopher Brumme
Jeroen Frijters
Wed 9/8/2004 17:52
RE: Memory model

      >> So the optimization is indeed allowed per the spec?

 

It depends on your definition of "side effect".  Historically C++ has used this same language to allow exactly this sort of thing, so that would be the typical reading of the spec.

 

Note that the CLI spec also specifies an extremely weak memory model that is heavily influenced by IA64.  Over here, we generally believe that the spec is too lenient and that our customers deserve a stronger memory model.  We would use the same custom attribute to allow the system assemblies to opt-in to a weaker memory model on IA64 without forcing all our customers to deal with the high test burden and complicated programming model that it implies.

 

 

      >> Is this optimization really that significant that it is worth

      >> all this trouble?

 

For line-of-business and web apps, the optimizations are not worth the trouble.  For high end ISV apps (like perhaps one day AutoCAD) or for the OS Shell, these optimizations are critical.

 

 

      >> I honestly don't see anyone adding this attribute to their

      >> code (and of the people that would add it, some (most?)

      >> probably don't understand the consequences).

 

Agreed.  It's there for people who need the performance of traditional unmanaged code and are willing to pay all the costs of achieving it.

This means that our customers generally will ignore this capability, though they will certainly enjoy the performance benefits of having most of the frameworks built this way.  We just need to be sure that we don't create races and other subtle bugs in the frameworks by chasing this extra perf.  On IA64 the risk is higher than with AMD64.  For X86 we are assuming no new risk in Whidbey.

 

 

      >> A "secure" method must do a volatile read before doing any

      >> parameter validation, otherwise the value of the parameter

      >> might change after the validation (if the method is inlined

      >> and this optimization happens).

 

Thank-you.  I have raised this issue more broadly.  You really should be working here.

 

 

      >> Wouldn't it be better to finally implemented boxed references,

      >> instead of this hack?

 

Eventually, we still intend to implement boxed references.  Of course, we've been saying that for about 5 years so it's hard to know whether to take us seriously.  You shouldn't read too much into the fact that this class is now defined in mscorlib.  We basically took a bunch of markers from the C++ team (as we had already done with some other language

teams) and dumped them into mscorlib.  The CLR still has nothing to do with the C++ illusion of boxed references.

 

Chris.

http://blogs.msdn.com/cbrumme

 
Jeroen Frijters
Christopher Brumme
Wed 9/8/2004 18:24
RE: Memory model

>     >> So the optimization is indeed allowed per the spec?

>

> It depends on your definition of "side effect".  Historically C++ has

> used this same language to allow exactly this sort of thing, so that

> would be the typical reading of the spec.

 

I sort of figured that that's where it came from. I, however, see source code as a much more abstract representation than IL (which is much more like a CPU instruction set), so they can get away with this much easier. Also, in C++ you mark a variable as volatile, but in the CLR (and in the CPU) it's the individual instructions that have the acquire/release semantics. To me the fact that a local variable can change its value feels like a CPU register that changes its value (that reminds of the old days under DOS, where the high 16 bits of the 32 bit registers where sometimes randomly destroyed by broken memory extenders).

 

>     >> Is this optimization really that significant that it is worth

>     >> all this trouble?

>

> For line-of-business and web apps, the optimizations are not worth the

> trouble.  For high end ISV apps (like perhaps one day AutoCAD) or for

> the OS Shell, these optimizations are critical.

 

I can imagine reordering loads and stores are important, but I have a hard time believing that the particular one that Grant's blog referred to makes a difference. After all, loading the value from the stack (an indirection through the stack pointer) versus loading it from the object (an indirection through the this pointer) doesn't make much difference.

 

> We just need to be sure that we don't create races and other subtle

> bugs in the frameworks by chasing this extra perf.

 

It seems like a difficult trade-off to me. The time spent to audit the code could also be used to improve perf in another way. Or is it possible to have static analysis tools detect these issues reliably?

 

>     >> Wouldn't it be better to finally implemented boxed references,

>     >> instead of this hack?

>

> Eventually, we still intend to implement boxed references.  Of course,

> we've been saying that for about 5 years so it's hard to know whether

> to take us seriously.

 

<g> It seems like a trivial feature (of course, almost everything looks trivial from the outside), I'm surprised the C++ guys haven't been able to convince the CLR team to do it. The thing I worry about is that once C++/CLI "popularizes" the current hack, there will be even less incentive to fix the CLR, because it will introduce another, incompatible, way of expressing the same thing. You probably can't answer this, but with the new Longhorn/WinFX story does that mean that Longhorn will ship with Whidbey instead of Orcas? If so, Whidbey is really going to have to be the CLR version that gets everything right.

 

BTW, did you see how they encode the type? I think it's rather ugly:

 

class ValueType modopt(Int32) modopt(IsBoxed) i

 

I would've preferred something like this:

 

class ValueType modopt(IsBoxed`1<Int32>) i

 

but I vaguely seem to remember that modifiers don't allow generics.

 

Regards,

Jeroen

 
Jeroen Frijters
Christopher Brumme
Fri 9/10/2004 10:53
RE: Memory model

Hi Chris,

 

I thought a lot about the optimization and I'm now somewhat more willing to believe that it is worthwhile and I have to reluctantly agree that adding a bit to CRA is the best thing to do. I really don't like CRA though, e.g. does link understand it (and refuse to link together modules with different flags)? How does it interact with Reflection.Emit or DynamicMethods? Global switches like that add conceptual weight to the platform.

 

I feel kinda funny, because in the Java world I'm always arguing that they are too conservative adding features to the VM.

 

Aside from the optimization, if the current CLI spec really allows this optimization, then I'm having a hard time figuring out how to (efficiently) compile Java bytecode to IL. Java doesn't allow this optimization (I had to re-read chapter 17 of the JLS to be sure [compared to the CLI spec they have a really detailed and formal description of their memory model [even if it is a little flawed]]), so how would I do a field load? I could always use a volatile.l[s]fld, but that has far too strong guarantees and disallows many optimizations. The only other option I could come up with was to do a virtual method call after each field load, that would also preclude the JIT from doing the optimization (as long as there is no way for it to prove where the virtual call ends up).

 

Finally, I can't be sure, but I think my suggestion to solve the "method arguments" problem (do a volatile load) doesn't work. As I understand it now, even a volatile load doesn't preclude the copy propagation of the non-volatile load. So you'd have to have a full memory barrier or do a virtual call at the top of your method (or modify the JIT to disallow copy propagation on arguments).

 

In summary, I think where we should end up is: CLI spec disallows this optimization, but when the CRA attribute is present (and has the correct flag set) the JIT is allowed to do optimizations that aren't spec compliant.

 

Regards,

Jeroen

 
Christopher Brumme
Jeroen Frijters
Fri 9/10/2004 17:17
RE: Memory model

Linking already ignores a large number of assembly-level concepts.  I think their current goal is for linking to mechanically succeed, and any semantic issues are for the developer to resolve.  Eventually I could imagine this being better productized.

 

The V1 ECMA spec is way too lax about the memory model.  With a weak

(IA64-style) memory model, there's little point in discouraging codegen optimizations... the chip is going to screw up the developer's assumptions anyway.  I personally would like to see the ECMA spec updated to a stronger memory model.  This will give developers a rational world to code in.

 

Regardless of whether the ECMA spec moves to a strong memory model, the CLR will guarantee a strong model by default.  (It's still not clear which technique we will use on IA64 to achieve this). So as a practical matter you can convert byte codes to IL for the CLR without any volatile operations.  However, I realize that changing the ECMA spec is far more desirable for you.

 

I'm writing up a DCR today anyway.  I'll include the proposal to ECMA as part of that work.  However, I already know that some of the non-Microsoft ECMA attendees will be very opposed to these changes.

 

Chris.

http://blogs.msdn.com/cbrumme

 
Jeroen Frijters
Christopher Brumme
Thu 12/2/2004 11:20
AppDomain.Unload bug?

Hello Chris,

 

I ran into something interesting again. I was writing a test case for IKVM to test static initialization dependencies, and to do this I iterate thru all types in my class library and fire up an AppDomain to initialize that type and then I unload the AppDomain again and move on to the next type.

 

I noticed that for types that fire up the AWT thread (basically the Windows event loop) the AppDomain couldn't be unloaded. This turned out to be because the thread is blocking inside a Win32 API waiting for a message (and since there aren't any windows, the message loop is unlikely to receive any messages).

 

It's not terribly important, but it might be a good idea to have special support in Thread.Abort for Windows message loop threads, that seems like a simple thing, but it probably is more complicated (isn't it always ;-)).

 

I could be wrong about my diagnosis, it's consistent with what I see on Everett, but on Whidbey (beta 1) the unload often succeeds for some reason.

 

I added a simple workaround to my code (I set up a WinForms timer that fires every 100ms).

 

Regards,

Jeroen

 
Christopher Brumme
Jeroen Frijters
Mon 12/6/2004 18:57
RE: AppDomain.Unload bug?

Hi.

 

I apologize for the delay...  I was at an offsite for the last week.

They kept us busy from 7:00 AM until 10:00 PM, so I had very little time left over for email in the evenings and mornings.

 

It's true that any unmanaged blocking will defeat the CLR's ability to take control of the thread.  That's one reason why we wrap [Msg]WaitForMultipleObjects[Ex] with WaitHandle.WaitAny/All, for example.

 

I had thought that WinForms used PeekMessage or MsgWait* or other tricks to either poll or efficiently block when there are no messages for the GUI thread to process.  But I looked at the WinForms code to confirm this and it looks like they will call WaitMessage is nothing is happening.  This looks like it will be a problem.

 

At some point I need to go look at what Avalon is doing here, too.  I have some general questions about how they do add-ins and whether they can cleanly support AppDomain.Unload of those add-ins.

 

All of this will shake out naturally when we get our act together with a strong add-in model and implementation.  But we are way behind on this effort and it obviously is now post-Whidbey.

 

How are things going generally?  Sometimes I get mail on Mono's progress, and I see that IKVM is the way to execute Java code on their runtime.

 

And is the Applications business healthy in Europe?  Microsoft certainly seems to be screwing up its business opportunities there.  Of all our businesses, Applications seems to be the one where we are struggling the most and all of those problems seem to be self-inflicted.  I still feel that over time this will be a huge business for us.  But it looks like it will take a lot more time than I had hoped.

 

 

Chris.

http://blogs.msdn.com/cbrumme

Jeroen Frijters
Christopher Brumme
Mon 12/6/2004 21:31
RE: AppDomain.Unload bug?

Hi,

 

> I apologize for the delay...  I was at an offsite for the last week.

> They kept us busy from 7:00 AM until 10:00 PM, so I had very little

> time left over for email in the evenings and mornings.

 

Ugh. Was that some internal Microsoft event?

 

> I had thought that WinForms used PeekMessage or MsgWait* or other

> tricks to either poll or efficiently block when there are no messages

> for the GUI thread to process.  But I looked at the WinForms code to

> confirm this and it looks like they will call WaitMessage is nothing

> is happening.  This looks like it will be a problem.

 

It's a tricky problem. Who owns these things? It's tempting to say that the individual libraries should prevent this, but at the end of the day the owner of Thread.Abort/AppDomain.Unload is probably more aware of the issues (to put it nicely ;-))

 

> At some point I need to go look at what Avalon is doing here, too.  I

> have some general questions about how they do add-ins and whether they

> can cleanly support AppDomain.Unload of those add-ins.

 

I need to look at Avalon too. I've installed the CTP, but so far I haven't really played with it.

 

> How are things going generally?  Sometimes I get mail on Mono's

> progress, and I see that IKVM is the way to execute Java code on their

> runtime.

 

Yeah, that's going great. IKVM is really starting to pick up users, but unfortunately I'm still the only developer.

 

> And is the Applications business healthy in Europe?

 

I don't know in general, but we're doing good (but we're small). The company whose product we're piggybacking on continues to blunder along and so far seems unaffected by Microsoft's move into the space.

 

Something else, when is Orcas planning really going to get going? I've considered writing up my wishlist (in detail), would that be helpful?

 

Regards,

Jeroen

 
Christopher Brumme
Jeroen Frijters
Wed 12/15/2004 19:34
RE: AppDomain.Unload bug?

It was indeed a Microsoft event, though it had zero technical content.

But it gave me a chance to meet with a lot of people here who I normally wouldn't interact with (lawyers, business folks and a few architects), so I did get some value out of it.

 

Generally the CLR pushes the frameworks to be friendly to an Unload.  We have guidelines for this which we have reviewed with all the teams.

It's part of the hardening effort that's required to get onto SQL Server's white list so you can be used inside the database.  But ultimately we rely on different teams having scenarios that involve Unload and testing it themselves.  When we get our add-in story going properly, I think this issue will resolve itself.  Both WinForms & Avalon would test add-ins aggressively for the client and we already have ASP.NET and SQL Server testing Unload aggressively on the server.

 

As for Orcas planning, we've actually had some high level planning already.  This has included presentations to various VPs, so it's somewhat baked.  However, a lot is going to change between now and when we can move lots of bodies from Whidbey over to Orcas.  (I think a few people are working on Orcas deliverables, but it's at the noise level).

I personally am not yet happy with some of the broad directions of the current Orcas plan.  Realistically I think it will be early spring before we have a plan we can believe in and we are really executing on.

 

Any time between now and February is probably a good time for input on the plan.

 

Chris.

http://blogs.msdn.com/cbrumme

 
Jeroen Frijters
Christopher Brumme
Mon 1/17/2005 17:06
byte[] == sbyte[]

Hello Chris,

 

A while ago I asked you if the changed verifier behavior towards signed/unsigned arrays was intentional and you said it probably wasn't. I just installed the December CTP on my x64 system and peverify still complains about this method:

 

.method public static int8[] cast(unsigned int8[] A_0) cil managed {

  .maxstack  1

  IL_0000:  ldarg.0

  IL_0001:  ret

}

 

As I said back then, I don't really mind, it's just a reminder to me that I need to change my code to represent Java byte arrays as unsigned byte arrays (not because of this, but to be CLS compliant).

 

Regards,

Jeroen

 
Christopher Brumme
Jeroen Frijters
Mon 1/17/2005 17:31
RE: byte[] == sbyte[]
arr.cs

There was a lot of email chatter about fixing this and a bunch of related regressions, though I didn't review those fixes myself.

 

I just tried the attached program with a 12/24 private build.  It executes off a network share and it satisfies peverify.exe.

 

Then I used ILDASM / ILASM to paste in the method you show below.

Peverify still likes it.

 

So I think we must have fixed it, but it didn't make the CTP.

 

Chris.

http://blogs.msdn.com/cbrumme

 
Jeroen Frijters
Christopher Brumme
Mon 1/17/2005 23:15
RE: byte[] == sbyte[]

> I just tried the attached program with a 12/24 private build.  It

> executes off a network share and it satisfies peverify.exe.

 

Interesting that casting now works consistently with the verifier. In Everett you couldn't do that.

 

> So I think we must have fixed it, but it didn't make the CTP.

 

It turns out that the December CTP includes a pretty old version of the CLR (build 41115) it's main focus is the Team System stuff. I only installed it because the November CTP DVD didn't have a working x64 installer (the setup file contained all zeroes, how on earth did that happen :-0).

 

Regards,

Jeroen

 
Jeroen Frijters
Christopher Brumme
Tue 1/18/2005 11:29
RE: byte[] == sbyte[]
chris.cs

> > I just tried the attached program with a 12/24 private build.  It

> > executes off a network share and it satisfies peverify.exe.

>

> Interesting that casting now works consistently with the verifier. In

> Everett you couldn't do that.

 

Upon reflection I think that this is bad idea. It's a breaking change (assuming that isinst behaves consistently with castclass) and offers not much benefit.

 

I'm not sure if it's worth losing any sleep over, but I've attached a contrived example that is perfectly safe on Everett, but would have security hole on Whidbey.

 

Regards,

Jeroen

 
Jeroen Frijters
Christopher Brumme
Tue 1/25/2005 18:41
GCHandle reuse attack
Class1.cs

Hi Chris,

 

I have no idea what made me think of this, but attached is a program that uses WeakReference to get access to a GCHandle it shouldn't have access to.

 

BTW, who on earth is responsible for making the Whidbey version of ILDASM unusable in interactive mode, by making the tree list the full type names at every level?

 

Regards,

Jeroen

 
Christopher Brumme
Jeroen Frijters
Tue 1/25/2005 18:49
RE: GCHandle reuse attack

Argh!

 

I'll forward your feedback on the ILDASM to the owner.  (Do you know Serge?  He's the dev who wrote the ".NET IL Assembler" book).

 

And I'll follow up on the security problem with GcHandle immediately.

Thank-you for finding this.

 

-- I haven't forgotten about the byte[]/sbyte[] security threat you sent me.  It's just that on my first casual read of it, I didn't understand what it was doing.  So I need a quiet 30 minutes, which is hard to come by.

 

Chris

http://blogs.msdn.com/cbrumme

 
Christopher Brumme
Jeroen Frijters
Tue 1/25/2005 22:14
RE: GCHandle reuse attack

Serge had this to say:

 

"...full class name display in graphic mode can be disabled by unchecking menu choice View/FullClassNames."

 

I don't know if this satisfies you, given that it's not a persistent setting.  It's not clear whether Serge can add persistence before Whidbey locks down, though I know he would like to.

 

 

Chris.

http://blogs.msdn.com/cbrumme

 
Jeroen Frijters
Christopher Brumme
Tue 1/25/2005 22:21
RE: GCHandle reuse attack

Thanks. I had looked for an option on the view menu (it's not in beta 1, or I must be really blind ;-)), but that's sufficient for me. I don't really mind if it doesn't persist.

 
Jeroen Frijters
Christopher Brumme
Wed 1/26/2005 12:59
RE: GCHandle reuse attack

Hi,

 

> I'll forward your feedback on the ILDASM to the owner.  (Do you know

> Serge?  He's the dev who wrote the ".NET IL Assembler" book).

 

I never met him, but we've e-mailed. I didn't know he was still the one responsible for ILDASM or I would have flamed him directly ;-)

 

> And I'll follow up on the security problem with GcHandle immediately.

> Thank-you for finding this.

 

You're welcome. I'm curious what fix they'll come up with.

 

> -- I haven't forgotten about the byte[]/sbyte[] security threat you

> sent me.  It's just that on my first casual read of it, I didn't

> understand what it was doing.  So I need a quiet 30 minutes, which is

> hard to come by.

 

It's very contrived so the code doesn't make all that much sense, but the general principle is that the CLR should avoid trying to do too many surprising things (programmers generally study a programming language and not the underlying abstraction [unfortunately]). This especially applies to changing the existing functionality, of course.

 

<noise>

Yesterday I thought I had found a JIT bug. I wanted to fix an outstanding issue in IKVM's handling of JNI (Java Native Interface) where it didn't support calling methods that return a value type. When I wrote a test case for this behavior and ran it, it failed in a very strange way. Here is a (much) distilled version of the code I generated:

 

    .maxstack  9

    .locals init (valuetype [IKVM.Runtime]IKVM.Runtime.JNI/Frame V_0,

             native int V_1,

             valuetype [IKVM.GNU.Classpath]java.lang.CharSequence V_2)

 

    ldloca.s   V_0

    initobj    [IKVM.Runtime]IKVM.Runtime.JNI/Frame

 

    ldloca.s   V_0

    ldloc.1

    ldloca.s   V_0

    ldsfld     object class0::'__<classObject>'

    brtrue.s   IL_0052

 

    ldtoken    class0

    call       object [IKVM.Runtime]IKVM.Runtime.ByteCodeHelper::GetClassFromTypeHandle(valuetype [mscorlib]System.RuntimeTypeHandle)

    stsfld     object class0::'__<classObject>'

IL_0052:

    ldsfld     object class0::'__<classObject>'

    call       instance native int [IKVM.Runtime]IKVM.Runtime.JNI/Frame::MakeLocalRef(object)

 

    ldftn      native int class0::NativeFunc(native int, native int)

    calli      native int(native int,native int)

 

    dup

    call       void [mscorlib]System.Console::WriteLine(int32)

 

    call       instance object [IKVM.Runtime]IKVM.Runtime.JNI/Frame::UnwrapLocalRef(native int)

    stloc.2

 

    ldloc.2

    ret

 

The crucial bit is that V_2 is a value type and that UnwrapLocalRef returns object. The stloc.2 caused the JIT to "optimize" the call the UnwrapLocalRef by passing in the address of V_2 (because it assumed that the method would return a value type, since I was [incorrectly] treating the return value as a value type).

 

Before I had figured out what was wrong I worked around it by moving the "ldloca.s V_0" to after the "calli" (and I had previously run the code on Whidbey, where the problem also didn't show), so I was "conviced" it was a JIT bug. Only this morning when I was trying to build a concise repro I figured out what was going on and I realized that it wasn't a JIT bug after all.

</noise>

 

Regards,

Jeroen

 
Jeroen Frijters
Christopher Brumme
Thu 1/27/2005 15:21
Bug fix or regression?
test.il

Hi again,

 

I just did a little more IKVM testing on x64 and I ran into a problem relating to how MethodImpls are handled. On Everett when you override a method that has a MethodImpl, you end up overriding the original method (in other words, the MethodImpl method occupies the same vtable slot as the original method), in Whidbey (build 41115) this no longer works. See the attached example.

 

Was this change intentional?

 

Regards,

Jeroen

 
Christopher Brumme
Jeroen Frijters
Simon [Redacted]
Thu 1/27/2005 16:48
FW: Bug fix or regression?

I see "I'm Derived" for this test case on a recent Whidbey build, just like Everett.

 

I believe this was an accidental regression during Whidbey that was noticed by the C# team and that has been corrected.  Simon can confirm this.

 

Thanks for the heads up.

 

Chris

http://blogs.msdn.com/cbrumme

 
Simon [Redacted]
Christopher Brumme; Jeroen Frijters
Thu 1/27/2005 16:56
RE: Bug fix or regression?

Hi Jeroen,

 

Actually, the change was not accidental; it was an attempt to move closer to what methodImpls were originally intended to be. We originally wanted methodImpls to be a one-time, one-slot method body substitution rather than a form of virtual slot unification. However, the C# team pointed out that they were relying on the previous behaviour and that probably others were as well, so we changed it back. Assuming that you're working with Beta 1, you'll see the behaviour reverted in Beta 2 when it is released.

 

Thanks,

Simon

 
Christopher Brumme
Jeroen Frijters
Fri 2/4/2005 01:21
RE: byte[] == sbyte[]

We're still talking about the generalization of your example (like some cases involving dispatch on boxed enums).  There's no question that it's interesting and a potential problem.  So far, the examples are all contrived and it looks like we have more threatening problems for this release.  But it's too early to be sure.

 

Regardless of where we end up, thank-you very much for bringing it to our attention.

 

Chris

http://blogs.msdn.com/cbrumme