159The Law of Sudden Riches

Author: Dr. Heinz M. KabutzDate: 2008-05-05Java Version: 5Category: Concurrency

Abstract: We all expect faster hardware to make our code execute faster and better. In this newsletter we examine why this is not always true. Sometimes the code breaks on faster servers or executes slower than on worse hardware.

Welcome to the 159th issue of The Java(tm) Specialists' Newsletter, sent from Athens. My two older kids and I went camping at Menies this week, a fairly deserted beach where people used to come from all over the world to worship Diktinna. We clambered up a hill with an amazing view of the Cretan sea and found a whole bunch of parts of ancient columns. An archeologists dream!

javaspecialists.teachable.com: Please visit our new self-study course catalog to see how you can upskill your Java knowledge.

The Law of Sudden Riches

We are looking at a series of laws to make sense of Java concurrency. Just to remind you, here they are again. In this newsletter, we will look at the Law of Sudden Riches.

1. The Law of the Sabotaged Doorbell
2. The Law of the Distracted Spearfisherman
3. The Law of the Overstocked Haberdashery
4. The Law of the Blind Spot
5. The Law of the Leaked Memo
6. The Law of the Corrupt Politician
7. The Law of the Micromanager
8. The Law of Cretan Driving
9. The Law of Sudden Riches
10. The Law of the Uneaten Lutefisk
11. The Law of the Xerox Copier

The Law of Sudden Riches

Imagine receiving an email from a stranger, telling you that you have inherited part of an estate from someone you never met. In the days before mass email scams, we received an email like that. My wife had inherited a motor home in the USA from someone who only vaguely knew her father. Nowadays I would just delete an email like that, but in those days, we were trusting enough to believe them. As a result, we received a bequest of several thousand dollars. She put the bequest to good use and we are still receiving the blessings from it today.

Now imagine for a moment that instead of it having been a few thousand dollars, if it had been a few hundred million dollars. Our lives would most definitely have changed and I am not sure for the better! Very often, these sudden wins or massive inheritances, cause seemingly stable systems to start shaking.

We saw this happen with a company that had bought a faster server. The older server was coping fairly well with the Java application. The new server was 4x faster and had more cores. However, the new server would occasionally stop users from logging in to the system, typically when there was a high load. So on Friday afternoons, when the client reports needed to be generated, the system would not allow them to log in anymore.

This baffled us initially, since we would expect the opposite to happen. We thought that with better hardware, the system should be faster. However, we have seen this in several systems already, where better hardware makes the overall program perform worse (or not at all).

We traced the problem to a DOM tree that got corrupted due to a data race that happened less frequently in the older system. See also The Law of the Corrupt Politician.

Seemingly Stable System

I need to make the point here that the bug was in the system all along, even in the old system. However, it occurred seldomly in the old system. Once a year, perhaps, the users could not log in. The administrator would restart the server and we would be none the wiser. With the faster server, which also corresponded with an increase in users, we got to the data race more quickly, thus causing the error once a week.

Let's look at some code that demonstrates this. To run this effectively, you need two CPUs, one modern and one old. We start with a BankAccount with a thread racing condition:

public class BankAccount {
  private int balance;

  public BankAccount(int balance) {
    this.balance = balance;
  }

  public void deposit(int amount) {
    balance += amount;
  }

  public void withdraw(int amount) {
    deposit(-amount);
  }

  public int getBalance() {
    return balance;
  }
}

To demonstrate that we have a problem in the code, we write a test where two threads continually deposit and withdraw 100. A third thread prints out the value of balance once a second.

import java.util.concurrent.*;

public class BankAccountTest {
  public static void main(String[] args) {
    final BankAccount account = new BankAccount(1000);
    for (int i = 0; i < 2; i++) {
      new Thread() {
        {
          start();
        }

        public void run() {
          while (true) {
            account.deposit(100);
            account.withdraw(100);
          }
        }
      };
    }
    ScheduledExecutorService timer =
        Executors.newSingleThreadScheduledExecutor();
    timer.scheduleAtFixedRate(new Runnable() {
      public void run() {
        System.out.println(account.getBalance());
      }
    }, 1, 1, TimeUnit.SECONDS);
  }
}

If I run this with JDK 6 using the -client hotspot compiler, I see the following effect on my older machine. We see that the racing condition becomes evident, but only after running the code for some time.

    Old Machine, JDK 1.6.0_05 -client
    1000
    1000
    1100
    1000
    1000
    1200
    1100
    1200
    1100
    1100
    1200
    1200
    1200
    1300  // incorrect
    1300  // incorrect
    1400  // incorrect
    1500  // incorrect
    1500  // incorrect   

In the next test, I run the exact same code on my new MacBook Pro, with a 2.6GHz dual core 64-bit processor. I don't know which the exact processor model is, so if any of you clever Mac freaks have any idea how to figure that out, please drop me a line :-) The default Apple Java VM only supports the -server HotSpot compiler, so I run this example with the SoyLatte JVM distribution:

    New MacBook Pro
    java version "1.6.0_03-p3" Client VM
    -118100
    -116400
    -112500
    -107500
    -102800
    -98100
    -94500
    -90600
    -86800

Right off the bat, this version was completely wrong. Thus if we have code that works with my old computer, it could go horrendously wrong in my new computer.

We now add the -server HotSpot compiler to the mix. In Java 6, the -server compiler optimizes really quite aggressively. However, this can make the system appear correct, even though it isn't. Let's run the code again using the -server hotspot compiler:

    New MacBook Pro
    java version "1.6.0_03-p3" Server VM
    1000
    1000
    1000
    1000
    1000
    1000
    1000
    1000
    1000
    1000
    1000
    1000

It appears as if the system works correctly, without the need for synchronization, since we always get correct results. However, let's run the code again:

    New MacBook Pro
    java version "1.6.0_03-p3" Server VM
    3241600
    3241600
    3241600
    3241600
    3241600
    3241600
    3241600
    3241600
    3241600

You can probably see what is happening here - the value of the balance is "hoisted" into the thread stack, after which it is not updated, even if the two threads do actually update it. We can test this premise by making the balance variable volatile. This prevents the thread from caching the value. See The Law of the Blind Spot.

public class BankAccount {
  private volatile int balance;

  public BankAccount(int balance) {
    this.balance = balance;
  }

  public void deposit(int amount) {
    balance += amount;
  }

  public void withdraw(int amount) {
    deposit(-amount);
  }

  public int getBalance() {
    return balance;
  }
}

When we run the code again with the -server option, we get more realistic results:

    New MacBook Pro
    balance set to "volatile"
    java version "1.6.0_03-p3" Server VM
    -415480100
    -826545300
    -1243123400
    -1648635600
    -2019877800
    1875291096
    1460629096
    1056496196
    645105696
    230742396
    -183617704
    -597808704
    -1007827804

The lesson here is to be really careful when moving to better and faster hardware. Your system might be incorrect, without you realizing it and the new hardware might show you up.

Kind regards

Heinz

Comments

We are always happy to receive comments from our readers. Feel free to send me a comment via email or discuss the newsletter in our JavaSpecialists Slack Channel (Get an invite here)

Related Articles

Browse the Newsletter Archive