Lab: Keyboard Control

Introduction

In this lab, you’ll build an alternative computer keyboard using any of the USB-native boards (Nano 33 IoT, the MKR series, the Due, the Leonardo, the Micro, and the Feather M0 series all fit this requirement). You’ll also learn some techniques for determining when a user takes a physical action. In the Digital Lab and Analog Lab, you learned how to read the input from a digital or analog input, but you didn’t learn how to interpret the stream of data as an event. There are a few everyday physical interactions that are fairly easy to pick out from a stream of data. You’ll see a few of them in this lab.

What You’ll Need to Know

To get the most out of this lab, you should be familiar with the following concepts. You can check how to do so in the links below:

Things You’ll Need

Figures 1-5 show the parts you’ll need for this exercise. Click on any image for a larger view.

A short solderless breadboard with two rows of holes along each side. There are no components mounted on the board.
Figure 1. A solderless breadboard.
Photo of an Arduino Nano 33 IoT module. The USB connector is at the top of the image, and the physical pins are numbered in a U-shape from top left to bottom left, then from bottom right to top right.
Figure 2. Arduino Nano 33 IoT
Three 22AWG solid core hookup wires. Each is about 6cm long. The top one is black; the middle one is red; the bottom one is blue. All three have stripped ends, approximately 4 to 5mm on each end.
Figure 3. 22AWG solid core hookup wires.
Resistors. Shown here are 220-ohm resistors. You can tell this because they have two red and one brown band, followed by a gold band.
Figure 4. Resistors. Shown here are 220-ohm resistors.  For this exercise, you’ll need 10-kilohm resistors (10K resistors are brown-black-orange. You’ll also need 1 220-ohm resistor. For more, see this resistor color calculator)
Photo of four breadboard-mounted pushbuttons
Figure 5. Pushbuttons.  For this exercise you’ll need five pushbuttons or any five momentary switches
Photo of a handful of LEDs
Figure 6. LEDs. The long leg goes to voltage and the short leg goes to ground

About Keyboard control

NOTE: The sketches contained in this lab will cause the microcontroller to take control of your keyboard. Make sure they’re working properly before you add the keyboard commands. The example doesn’t introduce the Keyboard commands until the end of the lab. Instead, messages are printed to the serial monitor to tell you what should happen. When you’ve run this and seen the serial messages occurring when you think they should, then you can add the mouse commands safely.

The circuit diagrams below show an Arduino Leonardo, but you can use any USB-capable board that you wish.

Note on Mouse, Keyboard, and Serial Ports

The Mouse and Keyboard libraries on the SAMD boards (MKRZero, MKR1xxx, Nano 33IoT) have an unusual behavior: using them changes the serial port enumeration. When you include the Keyboard library in a sketch, your board’s serial port number will change. For example, on MacOS, if the port number is /dev/cu.usbmodem14101, then adding the Keyboard library will change it to /dev/cu.usbmodem14102. Removing the Keyboard library will change it back to /dev/cu.usbmodem14101. Similarly, if you double-tap the reset button to put the board in bootloader mode, the serial port will re-enumerate to its original number.

Windows and HID Devices

You may have trouble getting these sketches to work on Windows. On Windows, the default Arduino drivers must be uninstalled so the system can recognize the Arduino as both serial device and HID device. Read this issue and follow the instructions there.

Recovering From a Runaway HID (Mouse or Keyboard) Sketch

Programs which control your keyboard and mouse can make development difficult or impossible if you make a mistake in your code. If you create a mouse or keyboard example that doesn’t do what you want it to, and you need to reprogram your microcontroller to stop the program, do this:

Open a new blank sketch.

This sketch is your recovery:

void setup() {

}
void loop() {

}

Programming the board with this blank sketch removes your mistaken sketch and gives you control again. To do this, however, you need the current sketch to stop running. So:

Put the microcontroller in bootloader mode and upload the blank sketch

On the SAMD boards, you can do this by double-tapping the reset button. The on-board LED will begin glowing, and the bootloader will start so that you get a serial port enumeration, but the sketch won’t run. On the Leonardo and other 32U4-based boards, hold the reset down until you’ve given the upload command. The 32U4 bootloader will take a few seconds before it starts the sketch, so the uploader can take command and load your blank sketch.

Once you’ve got the blank sketch on the board, review the logic of your mistaken Keyboard or Mouse sketch and find the problem before uploading again.

Prepare the breadboard

Connect power and ground on the breadboard to power and ground from the microcontroller. On the Arduino module, use the 5V and any of the ground connections as shown below in Figure 7. If using the Arduino Nano connect the 3.3V and ground pins according to Figure 8.

An Arduino Leonardo on the left connected to a solderless breadboard, right. The Leonardo's 5V output hole is connected to the red column of holes on the far left side of the breadboard. The Leonardo's ground hole is connected to the blue column on the left of the board. The red and blue columns on the left of the breadboard are connected to the red and blue columns on the right side of the breadboard with red and black wires, respectively. These columns on the side of a breadboard are commonly called the buses. The red line is the voltage bus, and the black or blue line is the ground bus.

Figure 7. An Arduino Leonardo on the left connected to a solderless breadboard, right.
Arduino Nano on a breadboard.

Figure 8. Breadboard view of an Arduino Nano connected to a breadboard. The +3.3 volts and ground pins of the Arduino are connected by red and black wires, respectively, to the left side rows of the breadboard. +3.3 volts is connected to the left outer side row (the voltage bus) and ground is connected to the left inner side row (the ground bus). The side rows on the left are connected to the side rows on the right using red and black wires, respectively, creating a voltage bus and a ground bus on both sides of the board.

Made with Fritzing


Add Several Pushbuttons

Attach pushbuttons to digital pin 2 through 6 as shown below in Figure 9. For Arduino Nano connections, see Figure 10. For the schematic view, see Figure 11. Connect one side of each pushbutton to voltage, and the other side of the pushbuttons to 10-kilohm resistors. Connect the other end of each resistor to ground. Connect the junction where each pushbutton and resistor meet to a digital pin, using pins 2 through 6. (For more on this digital input circuit, see the Digital Input Lab).

Breadboard drawing of an Arduino Leonardo connected to five pushbuttons. The pushbuttons straddle the center divide of the breadboard in rows 4-6, 9-11, 14-16, 19-21, and 24-26, respectively. 10-kilohm resistors straddle the center divide in rows 7, 12, 17, and 22, respectively. Each resistor's row on the left center side is connected to the row above it with a wire, thus connecting the pushbuttons and the resistors. Each resistor's row on the right center side is connected to the right side ground bus with a black wire. Each pushbutton's upper row (that is, rows 4, 9, 14, 19, and 23) on the right center side is connected to the right side voltage bus with a red wire. The junction rows, that is, rows 6, 11, 16, 21, and 26, are connected to digital inputs 2 through 6 on the Leonardo, respectively, with blue wires
Figure 9. Breadboard drawing of an Arduino Leonardo connected to five pushbuttons at the Leonardo’s digital pins 2-6.
Breadboard drawing of an Arduino Nano connected to five pushbuttons. The pushbuttons straddle the center divide of the breadboard in rows 20-22, 25-27, 30-32, 35-37, and 40-42, respectively. 10-kilohm resistors connect the left side of each pushbutton to the breadboard's GND bus at rows 22, 27, 32, 37, 42. Blue wires connect the right side of each pushbutton at pins 22, 27, 32, 37, 42 to the arduino digital pins 2-6 respectively. Red wires connect the left side of each pushbutton to the voltage bus at rows 20, 25, 30, 35, 40
Figure 10. Breadboard drawing of an Arduino Nano connected to five pushbuttons at the Nano’s digital pins 2-6.
Schematic drawing of five pushbuttons attached to an Arduino Leonardo. The pushbuttons is attached to digital pin 2 through 5 on one side, and to +5 volts on the other. There is also a 10-kilohm resistor attached to each of digital pins 2 through 6. The other side of each resistor is attached to ground.
Figure 11. Schematic drawing of five pushbuttons attached to an Arduino Leonardo.

Add an LED

Add an LED and 220-ohm resistor on pin 13 as shown below in Figure 12. For Arduino Nano see Figure 13. For the schematic view, see Figure 14. You’ll use this to keep track of the state of the keyboard control.

Breadboard drawing of an Arduino Leonardo connected to five pushbuttons. The pushbuttons straddle the center divide of the breadboard in rows 4-6, 9-11, 14-16, 19-21, and 24-26, respectively. 10-kilohm resistors straddle the center divide in rows 7, 12, 17, and 22, respectively. Each resistor's row on the left center side is connected to the row above it with a wire, thus connecting the pushbuttons and the resistors. Each resistor's row on the right center side is connected to the right side ground bus with a black wire. Each pushbutton's upper row (that is, rows 4, 9, 14, 19, and 23) on the right center side is connected to the right side voltage bus with a red wire. The junction rows, that is, rows 6, 11, 16, 21, and 26, are connected to digital inputs 2 through 6 on the Leonardo, respectively, with blue wires. A 220-ohm resistor is connected to pin 13 of the board, and straddles the center divide of the breadboard in row 1. The anode of an LED is attached to the other side of the resistor, and the cathode is attached to the right side ground bus.
Figure 12: Breadboard drawing of an Arduino Leonardo connected to five pushbuttons at digital pins 2-6. The Leonardo’s digital pin 13 is connected to a 220 ohm resistor in series with an LED.
Breadboard drawing of an Arduino Nano connected to five pushbuttons. The pushbuttons straddle the center divide of the breadboard in rows 20-22, 25-27, 30-32, 35-37, and 40-42, respectively. 10-kilohm resistors connect the left side of each pushbutton to the breadboard's GND bus at rows 22, 27, 32, 37, 42. Blue wires connect the right side of each pushbutton at pins 22, 27, 32, 37, 42 to the arduino digital pins 2-6 respectively. Red wires connect the left side of each pushbutton to the voltage bus at rows 20, 25, 30, 35, 40. A blue wire connects the Arduino's digital pin 13 to a a 220-ohm resistor in series with a green LED. The anode of an LED is attached to the anode of the resistor, the cathode of the LED is attached to the ground bus with a black wire.
Figure 13. Breadboard drawing of an Arduino Nano connected to five pushbuttons at digital pins 2-6. The Nano’s pin 13 is connected to a 220 ohm resistor in series with an LED.
Schematic drawing of five pushbuttons attached to an Arduino Leonardo. The pushbuttons is attached to digital pin 2 through 5 on one side, and to +5 volts on the other. There is also a 10-kilohm resistor attached to each of digital pins 2 through 6. The other side of each resistor is attached to ground. A 220-ohm resistor is connected to pin 13. The anode of an LED is connected to the other side of the resistor, and its cathode is connected to ground.
Figure 14. Schematic drawing of five pushbuttons and an LED attached to an Arduino Leonardo.

In the Sensor Change Lab, you learned how to detect the state change of a digital input. You’ll use the same technique here.

You’ll use the pushbutton on pin 2 to track whether the microcontroller is acting as a keyboard or not. Pressing this button will change the state of a global variable called keyboardIsActive. It will also call the Keyboard.begin() and Keyboard.end() commands as needed to take or relinquish control of the keyboard. The other four pushbuttons are intended to imitate keystrokes. The first will send the shift key. The last three will send the letters a, s, and d. With these options, you’ll learn to send both regular keystrokes as well as key combinations.

To make this work, you’ll need to know not only when each button is pressed and released, but also when it is still being pressed. For the keyboard activator button, you’ll change the state of the keyboard whenever the button is pressed. For the shift button, you’ll send a message on press and release. For the other buttons, you’ll send a message on press and release, and when the button is still being pressed, you’ll send the keystroke repeatedly, after a short delay.

Start by including the Keyboard library, then add two global arrays to track the button states and the previous button states. Also add a boolean variable to track whether the microcontroller is acting as a keyboard or not, a variable for the key repeat rate in milliseconds, and an array holding the keys you want to press with four of the five keys:

#include "Keyboard.h"

// Global variables:
int buttonState[5];      // states of the buttons
int lastButtonState[5];  // previous states of the buttons
// whether or not the Arduino is controlling the keyboard:
bool keyboardIsActive = false;
int repeatRate = 50;    // key repeat rate, in milliseconds

// keys to be pressed by the keyboard buttons:
int key[] = {KEY_LEFT_SHIFT, 'a', 's', 'd'};

Now, in the setup() function, set all the keys to be inputs. You can do this quickly with a for loop. Set the LED pin as an output as well:

void setup() {
  // initialize serial communication:
  Serial.begin(9600);
  // set the pins as INPUT_PULLUPs:
  for (int b = 2; b < 7; b++) {
    pinMode(b, INPUT_PULLUP);
  }
  // make the LED pin an output:
  pinMode(13, OUTPUT);
}

Detect the State Change of the Buttons

Now in your main loop() function you need to detect whether each button is pressed, released, or repeating. This is a slight variation on the digital in state change algorithm from the Sensor Change Lab. Again, you can use a for loop to do this for all the buttons like so:

void loop() {
  // iterate over the buttons:
  for (int b = 0; b < 5; b++) {
    // pin number = array index number + 2:
    buttonState[b] = digitalRead(b + 2);
    // see if the button has changed:
    if (buttonState[b] != lastButtonState[b]) {
      Serial.print("button");
      Serial.print(b);
      if (buttonState[b] == HIGH)
      { // pressed
        Serial.println("pressed");
      } else { // released
        Serial.println("released");
      }
    } else {
      if (buttonState[b] == HIGH) {
        Serial.println("still pressed");
      }
    }
    // save button's current state as previous state for next loop:
    lastButtonState[b] = buttonState[b];
  }
}

If you run the code now, you’ll see the messages “pressed”, “released” and “still pressed” for each key. You need to do something different foreach of those states for  each key, as follows:

  • Keyboard state button (pin 2): on press, toggle the keyboard state
  • Shift key button (pin 3): If keyboardIsActive is true, on press, send shift key press. On release, send shift key release. On still pressed, do nothing
  • Other key buttons:  If keyboardIsActive is true, on press, send key press. On release, send key release. On still pressed, send key press after key repeat delay

You can write a function in which you pass the button number and the state of the button to make this happen. First, add the following function after the loop() function:

void buttonAction(int buttonNumber, int buttonState) {
  // the keyboard state change button
  if (buttonNumber == 0) {
    // on press, toggle keyboardIsActive
    if (buttonState == HIGH) {
      keyboardIsActive = !keyboardIsActive;
    }
  }
  // the other three buttons:
  if (keyboardIsActive) {
    if (buttonNumber > 0 ) {
      if (buttonState == HIGH) {
        Serial.println("pressed");
      }
      if (buttonState == LOW) {
        Serial.println("released");
      }
    }
    // all the buttons other than the shift button and the keyboard state button:
    if (buttonNumber > 1) {
      if (buttonState == 2) { // still pressed
        Serial.println("repeating");
        delay(repeatRate);
      }
    }
  }
}

Then replace all the Serial.println() functions in the main loop() function with calls to this function, like so:

void loop() {
  // iterate over the buttons:
  for (int b = 0; b < 5; b++) {
    // pin number = array index number + 2:
    buttonState[b] = digitalRead(b + 2);
    // see if the button has changed:
    if (buttonState[b] != lastButtonState[b]) {
      Serial.print("button ");
      Serial.print(b);
      if (buttonState[b] == HIGH)
      { // pressed
        buttonAction(b, 0);
        //Serial.println(" pressed");
      } else { // released
        buttonAction(b, 1);
        //Serial.println(" released");
      }
    } else {
      if (buttonState[b] == HIGH) {
        buttonAction(b, 2);
        //Serial.println("still pressed");
      }
    }
    // save button's current state as previous state for next loop:
    lastButtonState[b] = buttonState[b];
  }
}

When you run this sketch now, you should see that when you press the first button, it enables the other three to print out “pressed”, “released” and “repeating” messages.

Add commands to control the Keyboard

Finally, change the Serial.println() statements in the buttonAction() functions to Keyboard.press() or Keyboard.release() functions like so. Also add a block of code to activate or deactivate the Keyboard library and to set the LED (n.b. an audio alternative to the LED follows this code block). Note that you’re getting the key value from the key[] array, using the button number as the index:

void buttonAction(int buttonNumber, int buttonState) {
  // the keyboard state change button
  if (buttonNumber == 0) {
    // on press, toggle keyboardIsActive
    if (buttonState == HIGH) {
      keyboardIsActive = !keyboardIsActive;
      // activate the keyboard:
      if (keyboardIsActive) {
        Keyboard.begin();
      } else {
        Keyboard.end();
      }
      // change the LED to reflect the keyboard state:
      digitalWrite(13, keyboardIsActive);
    }
  }
  // the other three buttons:
  if (keyboardIsActive) {
    if (buttonNumber > 0 ) {
      if (buttonState == HIGH) {
        Keyboard.press(key[buttonNumber - 3]);
        // Serial.println(" pressed");
      }
      if (buttonState == LOW) {
        Keyboard.release(key[buttonNumber - 3]);
        // Serial.println(" released");
      }
    }
    // all the buttons other than the shift button and the keyboard state button:
    if (buttonNumber > 1) {
      if (buttonState == 2) { // pressed
        Keyboard.press(key[buttonNumber - 3]);
        // Serial.println(" repeating");
        delay(repeatRate);
      }
    }
  }
}

If you prefer an audio notification instead of the LED, put a buzzer on pin 13 and change the lines following the one that changes the keyboardIsActive variable as follows:

eyboardIsActive = !keyboardIsActive;
// activate the keyboard:
if (keyboardIsActive) {
  Keyboard.begin();
  //buzz once for active:
  digitalWrite(13, HIGH);
  delay(300);
  digitalWrite(13, LOW);
} else {
  Keyboard.end();
  //buzz twice for inactive:
  digitalWrite(13, HIGH);
  delay(300);
  digitalWrite(13, LOW);
  delay(300);
  digitalWrite(13, HIGH);
  delay(300);
  digitalWrite(13, LOW);
}

Once you upload this change, your board will be controlling the keyboard. Open a text editor and press the keyboard activation button. The LED will light up to indicate that keyboard control is active. Then press the last three keys, and it should send a, s, and d to the text editor. Hold shift while you press the other keys and you’ll get A, S, and D.

Shouldn’t I Use the Shift Key to Modify The Behavior of Others in Code?

You may be thinking that the code above is not finished, and that you should be sending “a” when the shift key is unpressed, but “A” when it is. That is what is happening, but that’s not the microcontroller’s job. It only has to report which keys are pressed. The operating system on the receiving computer decides what to do with the results.

The Keyboard Library Commands

In the sketch above, you used the Keyboard.begin() and Keyboard.end() commands to take or relinquish control of the keyboard, and the Keyboard.press(), Keyboard.release(), and Keyboard.write() commands to send keystrokes. There is more you can do with the Keyboard library. You can use many of the same commands you’ve used in the Serial library, like print() and println(). You can even program the Arduino to reprogram itself. Check out the Keyboard library documentation for more.

The full sketch for this can be found on the phys comp github repository, called LabKeyboardControl.