182Remote Screenshots

Author: Dr. Heinz M. KabutzDate: 2010-05-12Java Version: 6Category: GUI

Abstract: In this newsletter, we describe how we can generate remote screen shots as compressed, scaled JPGs to build a more efficient remote control mechanism.

A warm welcome to the 182nd edition of The Java(tm) Specialists' Newsletter, sent to you from Regina in Saskatchewan Canada. I went to a fantastic restaurant last night called "The Diplomat", where I tasted a wonderful fillet with lemon potatoes. As soon as I put the first potato in my mouth, I knew the nationality of the chef - Greek! I asked the waitress and she was quite surprised that I knew. The potatoes tasted exactly like those made by Irene at her taverna up the road from where we live! A Java programmer from Albania subscribed recently, so our list of subscriber countries has grown to 121. Special welcome to Adonis Arifi!

javaspecialists.teachable.com: Please visit our new self-study course catalog to see how you can upskill your Java knowledge.

Remote Screenshots

Programmers do not like it when you stand behind them watching how they code, especially if they are not confident on how to solve a problem. Many years ago, I wrote a very simple Java program that allowed me to watch my students' screens, with their knowledge of course, without the feeling of being watched.

Unfortunately my very basic program only scaled up to 6 PCs, after that it became too slow and it was hard to follow all those windows. It also did not work well over a DSL internet connection.

Last year I was away from home for 150 days, due to all the courses I had to present. Our plan is to offer more courses from our conference facility on our island in the Mediterranean. To do this, I needed a program to let me view the approach my students were using in solving the exercises. I thus took another look at my old program and improved it to reduce the bandwidth.

The way this program works is that we start a server on our side to which the students can connect. The server then sends a request to the student program to please send a "screen shot". It also tells the student program what zoom factor we want to see, which well help to reduce bandwidth on screens that I am not currently looking at. I can send mouse clicks to the student screen, which I would do if I needed to show them something or check that their answers are correct.

Similar to our first approach, we will use the interface RobotAction, which represents a command that I send to the client machine.

import java.awt.*;
import java.io.*;

public interface RobotAction extends Serializable {
  Object execute(Robot robot) throws IOException;
}

The RobotActionQueue extends a Deque implementation, so we can peek at the last element in the queue.

import java.util.concurrent.*;

public class RobotActionQueue extends
    LinkedBlockingDeque<RobotAction> {
}

The client machine would run the Student class (below), which would read in the RobotAction objects from the server and then insert them into a queue. One of the issues that we had with our old solution was that the same requests would start filling up the queue, so that new requests would take a very long time to be processed. We thus use jobs.peekLast() to see whether the same type of job is already waiting and if it is, we discard it.

When the robot action is executed, any non-null result is sent back to the server. In our current implementation, the only information that we want to send back to the Teacher is a screen shot as a byte[], so that is all that we would expect. However, we could easily expand this program to let the student press a button to call for help.

import java.awt.*;
import java.io.*;
import java.net.*;

public class Student {
  private final ObjectOutputStream out;
  private final ObjectInputStream in;
  private final Robot robot;
  private final RobotActionQueue jobs = new RobotActionQueue();
  private final ProcessorThread processor;
  private final ReaderThread reader;

  public Student(String serverMachine, String studentName)
      throws IOException, AWTException {
    Socket socket = new Socket(
        serverMachine, TeacherServer.PORT);
    robot = new Robot();
    out = new ObjectOutputStream(socket.getOutputStream());
    in = new ObjectInputStream(
        new BufferedInputStream(socket.getInputStream()));
    out.writeObject(studentName);
    out.flush();
    processor = new ProcessorThread();
    reader = new ReaderThread();
  }

  private class ReaderThread extends Thread {
    public void run() {
      try {
        RobotAction action;
        while ((action = (RobotAction) in.readObject()) != null) {
          if (!action.equals(jobs.peekLast())) {
            jobs.add(action);
            System.out.println("jobs = " + jobs);
          } else {
            System.out.println("Discarding duplicate request");
          }
        }
      } catch (EOFException eof) {
        System.out.println("Connection closed");
      } catch (Exception ex) {
        System.out.println("Connection closed abruptly: " + ex);
      }
    }
  }

  private class ProcessorThread extends Thread {
    public ProcessorThread() {
      super("ProcessorThread");
      setDaemon(true);
    }

    public void run() {
      try {
        while (!isInterrupted()) {
          try {
            RobotAction action = jobs.take();
            Object result = action.execute(robot);
            if (result != null) {
              out.writeObject(result);
              out.reset();
              out.flush();
            }
          } catch (InterruptedException e) {
            interrupt();
            break;
          }
        }
        out.close();
      } catch (IOException e) {
        System.out.println("Connection closed (" + e + ')');
      }
    }
  }

  public void start() {
    processor.start();
    reader.start();
  }

  public static void main(String[] args) throws Exception {
    if (args.length != 2) {
      System.err.println("Parameters: server studentname");
      System.exit(1);
    }
    Student student = new Student(args[0], args[1]);
    student.start();
  }
}

The first action we want to look at is MouseMove. We want to send a request to the client PC to move the mouse to a certain position. The student program would then call the execute method, which would move the pointer to the desired location.

Instead of using the standard hashcode algorithm, where we multiply the first field by 31 and then add the second, we will rather do bit shifting to get unique values every time. There is otherwise a very good chance of a collision. We are not using the MoveMouse objects for hashing, but on principle I think we should write good hashCode functions even if the object is currently not intended as a key.

import java.awt.*;
import java.awt.event.*;

public class MoveMouse implements RobotAction {
  private final int x;
  private final int y;

  public MoveMouse(Point to) {
    x = (int) to.getX();
    y = (int) to.getY();
  }

  public MoveMouse(MouseEvent event) {
    this(event.getPoint());
  }

  public Object execute(Robot robot) {
    robot.mouseMove(x, y);
    return null;
  }

  public String toString() {
    return "MoveMouse: x=" + x + ", y=" + y;
  }

  public boolean equals(Object o) {
    if (!(o instanceof MoveMouse)) return false;
    MoveMouse mm = (MoveMouse) o;
    return x == mm.x && y == mm.y;
  }

  public int hashCode() {
    return (x << 16) + y;
  }
}

The next action is ClickMouse. Here we specify which mouse button was pressed and how often. With these two actions, I can now control the remote PC to help anyone who gets stuck.

import java.awt.*;
import java.awt.event.*;

public class ClickMouse implements RobotAction {
  private final int mouseButton;
  private final int clicks;

  public ClickMouse(int mouseButton, int clicks) {
    this.mouseButton = mouseButton;
    this.clicks = clicks;
  }

  public ClickMouse(MouseEvent event) {
    this(event.getModifiers(), event.getClickCount());
  }

  public Object execute(Robot robot) {
    for (int i = 0; i < clicks; i++) {
      robot.mousePress(mouseButton);
      robot.mouseRelease(mouseButton);
    }
    return null;
  }

  public String toString() {
    return "ClickMouse: " + mouseButton + ", " + clicks;
  }

  public boolean equals(Object o) {
    if (!(o instanceof ClickMouse)) return false;
    ClickMouse cm = (ClickMouse) o;
    return clicks == cm.clicks && mouseButton == cm.mouseButton;
  }

  public int hashCode() {
    return 31 * mouseButton + clicks;
  }
}

The last robot action that we need to look at is the ScreenShot, where the Robot creates an image of the screen and then sends that over the wire as a JPEG. In this code we used several tricks to reduce the bandwidth used.

First off, when I am not actively watching what the student is doing, I set the zoom factor to 30%. This allows me to keep lots of student screens open. As soon as I select his screen, it changes the zoom factor to 100%. I tried several options to make the scaling look nice. After some experiments I decided to use SCALE_AREA_AVERAGING, but that is a bit slow. If the student machine is too slow to cope, then we can also change to the SCALE_FAST algorithm.

The second trick is to reduce the JPG quality, so that I can still recognize what is happening, but where we reduce the bandwidth significantly. This way we were able to reduce the images to 1/50th of their size. In our program we chose a compression quality of 30%. You can see the artifacts of the JPG compression if you look carefully, but at least it is really fast.

The third trick is that I only send the image if it has not changed since the last time, which I store in the ThreadLocal called "previous". It would be even better if we stored the raw image, so that we could only send across the region that was changed last. This is how the commercial solutions work.

import javax.imageio.*;
import javax.imageio.stream.*;
import java.awt.*;
import java.awt.image.*;
import java.io.*;
import java.util.*;

public class ScreenShot implements RobotAction {
  // this is used on the student JVM to optimize transfers
  private static final ThreadLocal<byte[]> previous =
      new ThreadLocal<byte[]>();
  private static final float JPG_QUALITY = 0.3f;

  private final double scale;

  public ScreenShot(double scale) {
    this.scale = scale;
  }

  public ScreenShot() {
    this(1.0);
  }

  public Object execute(Robot robot) throws IOException {
    long time = System.currentTimeMillis();
    Toolkit defaultToolkit = Toolkit.getDefaultToolkit();
    Rectangle shotArea = new Rectangle(
        defaultToolkit.getScreenSize());
    BufferedImage image = robot.createScreenCapture(shotArea);
    if (scale != 1.0) {
      image = getScaledInstance(image);
    }
    byte[] bytes = convertToJPG(image);
    time = System.currentTimeMillis() - time;
    System.out.println("time = " + time);
    // only send it if the picture has actually changed
    byte[] prev = previous.get();
    if (prev != null && Arrays.equals(bytes, prev)) {
      return null;
    }
    previous.set(bytes);
    return bytes;
  }

  private byte[] convertToJPG(BufferedImage img)
      throws IOException {
    ImageWriter writer =
        ImageIO.getImageWritersByFormatName("jpg").next();
    ImageWriteParam iwp = writer.getDefaultWriteParam();
    iwp.setCompressionMode(ImageWriteParam.MODE_EXPLICIT);
    iwp.setCompressionQuality(JPG_QUALITY);

    ByteArrayOutputStream bout = new ByteArrayOutputStream();
    writer.setOutput(new MemoryCacheImageOutputStream(bout));
    writer.write(null, new IIOImage(img, null, null), iwp);
    writer.dispose();
    bout.flush();
    return bout.toByteArray();
  }

  public BufferedImage getScaledInstance(BufferedImage src) {
    int width = (int) (src.getWidth() * scale);
    int height = (int) (src.getHeight() * scale);

    Image scaled = src.getScaledInstance(width, height,
        BufferedImage.SCALE_AREA_AVERAGING);
    BufferedImage result = new BufferedImage(
        width, height, BufferedImage.TYPE_INT_RGB
    );
    result.createGraphics().drawImage(
        scaled, 0, 0, width, height, null);
    return result;
  }

  public String toString() {
    return "ScreenShot(" + scale + ")";
  }

  public boolean equals(Object o) {
    if (!(o instanceof ScreenShot)) return false;
    return Double.compare(((ScreenShot) o).scale, scale) == 0;
  }

  public int hashCode() {
    long temp = Double.doubleToLongBits(scale);
    return (int) (temp ^ (temp >>> 32));
  }
}

TeacherServer

Our main class is the TeacherServer. When a new student connects, the first object that is sent is the student name. We then generate two threads to communicate with the student. In our system, the bottleneck is the network and not the number of threads. It thus does not matter that we are constructing two threads per connection. I would not be able to monitor that many students at a time anyway. However, if it ever became an issue, then we could simply replace the server with non-blocking IO.

The SocketWriterThread controls the zoom factor and the delay between ScreenShot requests. Whilst the window is active, we want to see updates as quickly as possible, so we set the zoom factor to 100% and the wait time to 300 milliseconds. When we stop looking at the window, the zoom goes back to 30% and the wait time to 3 seconds between taking screen shots.

import java.awt.event.*;
import java.io.*;
import java.util.concurrent.*;

class SocketWriterThread extends Thread {
  private final RobotActionQueue jobs = new RobotActionQueue();
  private final String studentName;
  private final ObjectOutputStream out;
  private volatile boolean active = false;

  public SocketWriterThread(String studentName,
                            ObjectOutputStream out) {
    super("Writer to " + studentName);
    this.studentName = studentName;
    this.out = out;
  }

  public void setActive(boolean active) {
    this.active = active;
    askForScreenShot();
  }

  private double getZoomFactor() {
    return active ? 1.0 : 0.3;
  }

  public long getWaitTime() {
    return active ? 500 : 3000;
  }

  public void clickEvent(MouseEvent e) {
    if (active) {
      jobs.add(new MoveMouse(e));
      jobs.add(new ClickMouse(e));
    }
    active = true;
    askForScreenShot();
  }

  private void askForScreenShot() {
    jobs.add(new ScreenShot(getZoomFactor()));
  }

  public void run() {
    askForScreenShot();
    try {
      while (!isInterrupted()) {
        try {
          RobotAction action = jobs.poll(
              getWaitTime(),
              TimeUnit.MILLISECONDS);
          if (action == null) {
            // we had a timeout, so do a screen capture
            askForScreenShot();
          } else {
            System.out.println("sending " + action +
                " to " + studentName);
            out.writeObject(action);
            out.reset();
            out.flush();
          }
        } catch (InterruptedException e) {
          interrupt();
          break;
        }
      }
      out.close();
    } catch (IOException e) {
      System.out.println("Connection to " + studentName +
          " closed (" + e + ')');
    }
    System.out.println("Closing connection to " + studentName);
  }
}

The SocketReaderThread reads the screen shot results from our student process and then passes them to the TeacherServer, who then displays them immediately. This is rather simple code:

import java.io.*;

class SocketReaderThread extends Thread {
  private final String studentName;
  private final ObjectInputStream in;
  private final TeacherServer server;

  public SocketReaderThread(
      String studentName,
      ObjectInputStream in,
      TeacherServer server) {
    super("Reader from " + studentName);
    this.studentName = studentName;
    this.in = in;
    this.server = server;
  }

  public void run() {
    while (true) {
      try {
        byte[] img = (byte[]) in.readObject();
        System.out.println("Received screenshot of " +
            img.length + " bytes from " + studentName);
        server.showScreenShot(img);
      } catch (Exception ex) {
        System.out.println("Exception occurred: " + ex);
        ex.printStackTrace();
        server.shutdown();
        return;
      }
    }
  }

  public void close() {
    try {
      in.close();
    } catch (IOException ignore) {
    }
  }
}

The TeacherServer class sets up the network connections with the server and creates the TeacherFrame, which then displays the screen shots sent by the student program.

import javax.swing.*;
import java.io.*;
import java.net.*;
import java.lang.reflect.*;

public class TeacherServer {
  public static final int PORT = 5555;

  private final SocketWriterThread writer;
  private final TeacherFrame frame;
  private final SocketReaderThread reader;

  public TeacherServer(Socket socket)
      throws IOException, ClassNotFoundException,
      InvocationTargetException, InterruptedException {
    ObjectOutputStream out = new ObjectOutputStream(
        socket.getOutputStream());
    ObjectInputStream in = new ObjectInputStream(
        new BufferedInputStream(
            socket.getInputStream()));
    System.out.println("waiting for student name ...");
    final String studentName = (String) in.readObject();

    reader = new SocketReaderThread(studentName, in, this);
    writer = new SocketWriterThread(studentName, out);

    final TeacherFrame[] temp = new TeacherFrame[1];
    SwingUtilities.invokeAndWait(new Runnable() {
      public void run() {
        temp[0] = new TeacherFrame(studentName,
            TeacherServer.this, writer);
      }
    });
    frame = temp[0];

    reader.start();
    writer.start();

    System.out.println("finished connecting to " + socket);
  }

  public void showScreenShot(byte[] bytes) throws IOException {
    frame.showScreenShot(bytes);
  }

  public void shutdown() {
    writer.interrupt();
    reader.close();
  }

  public static void main(String[] args) throws Exception {
    ServerSocket ss = new ServerSocket(PORT);
    while (true) {
      Socket socket = ss.accept();
      System.out.println("Connection From " + socket);
      new TeacherServer(socket);
    }
  }
}  

The last class we need is the TeacherFrame, where we display what our student can see. When we click on the image, we at the same time send an two events to our student, MoveMouse and ClickMouse. We use the windowActivated and windowDeactivated events to decide what zoom factor to send to our student.

import javax.imageio.*;
import javax.swing.*;
import java.awt.event.*;
import java.awt.image.*;
import java.io.*;

public class TeacherFrame extends JFrame {
  private final TeacherServer server;
  private final SocketWriterThread writer;
  private final JLabel iconLabel = new JLabel();

  public TeacherFrame(String studentName, TeacherServer server,
                      SocketWriterThread writer) {
    super("Screen from " + studentName);
    this.server = server;
    this.writer = writer;

    add(new JScrollPane(iconLabel));
    iconLabel.addMouseListener(new MouseAdapter() {
      public void mouseClicked(MouseEvent e) {
        TeacherFrame.this.writer.clickEvent(e);
      }
    });
    addWindowListener(new WindowAdapter() {
      public void windowActivated(WindowEvent e) {
        TeacherFrame.this.writer.setActive(true);
      }
      public void windowDeactivated(WindowEvent e) {
        TeacherFrame.this.writer.setActive(false);
      }
      public void windowClosing(WindowEvent e) {
        TeacherFrame.this.server.shutdown();
      }
    });

    pack();
    setVisible(true);
  }

  public void showScreenShot(byte[] bytes) throws IOException {
    final BufferedImage img = ImageIO.read(
        new ByteArrayInputStream(bytes));
    SwingUtilities.invokeLater(new Runnable() {
      public void run() {
        iconLabel.setIcon(new ImageIcon(img));
        pack();
      }
    });
  }
}

To try out this program, it would be best to use two computers. The first computer would run the TeacherServer program, for example:

    java eu.javaspecialists.tjsn.jmatia.server.TeacherServer

The second computer runs the student, for which we need to specify the address of the server and the name of the student, for example:

    java eu.javaspecialists.tjsn.jmatia.Student 192.168.1.7 "Max Guy"

It is really interesting how quickly the images are now sent from the student to the teacher. You can learn a lot by watching how people approach a new exercise. It certainly helps to then explain to them how it should have been answered. It also helps to be able to watch lots of students at once.

Kind regards, and hope to see you in Crete, either in person or via the internet :-)

Heinz

Comments

We are always happy to receive comments from our readers. Feel free to send me a comment via email or discuss the newsletter in our JavaSpecialists Slack Channel (Get an invite here)

Related Articles

Browse the Newsletter Archive