{"id":1737,"date":"2014-08-23T10:40:04","date_gmt":"2014-08-23T14:40:04","guid":{"rendered":"https:\/\/itp.nyu.edu\/physicalcomputing\/?page_id=1737"},"modified":"2022-11-06T18:00:46","modified_gmt":"2022-11-06T23:00:46","slug":"interpreting-serial-data","status":"publish","type":"page","link":"https:\/\/itp.nyu.edu\/physcomp\/lessons\/interpreting-serial-data\/","title":{"rendered":"Interpreting Serial Data"},"content":{"rendered":"\n

<\/span>Introduction<\/span><\/h2>\n\n\n\n

Serial data is passed byte by byte from one device to another, but it\u2019s up to you to decide how each device (computer or microcontroller) should interpret those bytes, when the beginning of a message is, when the end is, and what to do with the bytes in between.  Serial communication protocols<\/strong> define the structure of communication between devices. These notes explain how serial data is interpreted by computers.<\/p>\n\n\n\n

To get the most out of these notes, you should know what a microcontroller is <\/a>and have an understanding of the basics of microcontroller programming<\/a>. You should also understand the basics of serial communication<\/a>. You should have some understanding of programming a personal computer as well, ideally in a language that can access the serial ports of the computer, like p5.js<\/a>, the Processing programming environment<\/a>, the node.js programming environment<\/a>, Python, or Java.<\/p>\n\n\n\n

This is a useful page to return to when you have a specific advanced serial communication problem to solve. <\/p>\n\n\n\n

<\/span>ASCII and Binary<\/span><\/h2>\n\n\n\n

Related video: ASCII Bytes Explained<\/a><\/p>\n\n\n\n

Imagine that you’re sending the value of one sensor from a microcontroller to a personal computer. If the sensor’s value is always less than 255, you know it can fit in a single byte. This kind of message is easy. Just send the value over and over, and the receiver can read the latest byte to have your whole message. In Arduino, you can do this using the Serial.write()<\/tt> command<\/a>, as shown in the Arduino sketch below:<\/p>\n\n\n\n

<\/span>Binary Serial Arduino Program<\/span><\/h3>\n\n\n
\nvoid setup() {\n  Serial.begin(9600);\n}\n\nvoid loop() {\n  \/\/ read the sensor:\n  int sensorValue = analogRead(A0);\n  \/\/ divide by 4 to reduce the range to 0-255:\n  sensorValue = sensorValue \/ 4;\n  \/\/ send it:\n  Serial.write(sensorValue);\n}\n<\/pre><\/div>\n\n\n
Imagine a typical stream of bytes sent by the program above:<\/p>\n\n\n\n
23 23 23 23 24 24 25 25 26 28 27 27 27<\/code><\/p>\n\n\n\n
Every value you send can range from 0 to 255, the full range that can fit in a byte.  A protocol like this is often called a binary protocol<\/strong>, because there’s no other meaning to each byte’s value other than the value of the number itself.<\/p>\n\n\n\n
Now imagine you want to send two sensor readings.  You might think “No problem; just add another analogRead() command<\/a> and another Serial.write() command<\/a>.” But the resulting stream of data would look the same. Each byte would range from 0 to 255, and you’d have no way to know which byte represents which sensor. You need some punctuation bytes.<\/p>\n\n\n\n
If you\u2019re sending more than one value (and you usually are), then the receiving computer has to know when the message starts and when it ends, and it needs to know how to put the bytes together into a message.<\/p>\n\n\n\n
Computers use numbers to represent alphanumeric characters (letters and numbers and punctuation) in bytes. The first standard code for doing this was called the ASCII code<\/strong> which stands for American Standard Code for Information Interchange.<\/em> ASCII assigns each number or letter a specific byte value from 0 to 127.  For example, capital A is ASCII value 65. Capital B is 66. A space is ASCII value 32. The numeral 0 is ASCII 48. ASCII includes only the characters for the English alphabet, though, so a newer protocol, the Unicode protocol<\/a>, includes the ASCII character set and includes codes for other alphabets as well. The simplest subset of Unicode, UTF-8<\/a>, is compatible with ASCII.<\/p>\n\n\n\n
The ASCII table<\/a> and Unicode set can be found in many computer manuals\u2019 indexes, and all over the place online. These codes are used for number-to-character translation in every computer operating system and programming language.<\/p>\n\n\n\n
Because ASCII and Unicode assign each alphanumeric character a unique value, including punctuation symbols, you can now differentiate the values in your serial stream by ASCII-encoding them.  Change the Serial.write()<\/code> in the program above to a Serial.print() command<\/a>, and add one more line:<\/p>\n\n\n\n
<\/span>ASCII-encoded Serial Arduino Program<\/span><\/h3>\n\n\n
\nvoid setup() {\n  Serial.begin(9600);\n}\n\nvoid loop() {\n  \/\/ read the sensor:\n  int sensorValue = analogRead(A0);\n  \/\/ divide by 4 to reduce the range to 0-255:\n  sensorValue = sensorValue \/ 4;\n  \/\/ send it:\n  Serial.print(sensorValue);\n  Serial.print(\",\");\n}\n<\/pre><\/div>\n\n\n
Now you’ve got a string of bytes representing numeric characters, AND a byte representing a comma.  What would the values of the bytes be?<\/p>\n\n\n\n
50 51 44 50 51 44 50 51 44 50 51 44 50 52 44 50 52 44 50 53 44 50 53 44 50 54 44 50 56 44 50 55 44 50 55 44 50 55 44<\/code><\/p>\n\n\n\n
Why is it so much longer, and what are all those 44s doing in there so regularly?  If you look at the ASCII table, you’ll see why. The sensor values are ASCII-encoded now.  That means that the value 23 is represented by the character “2” followed by the character “3”. In ASCII, “2” has the value 50 and “3” has the value 51. And “,” has the value 44.  So the numbers above, read as ASCII, translate to:<\/p>\n\n\n\n
\"23,23,23,23,24,24,25,25,26,28,27,27,27\"<\/code><\/p>\n\n\n\n
It takes more bytes to send data this way, but you have a more human-readable protocol, because most receiving programs will default to displaying byte values using the ASCII or unicode characters assigned to them.<\/p>\n\n\n\n
<\/span>Serial Terminal Programs<\/span><\/h2>\n\n\n\n
Related video: Serial 4 – Devices and Bytes<\/a><\/p>\n\n\n\n
Most serial terminal programs assume that when you’re receiving serial data, it should be interpreted as ASCII characters, This is why you’ll see random characters when you open the Serial Monitor in Arduino after uploading the binary serial program above: the Arduino’s using a binary protocol, but the Serial Monitor thinks it’s an ASCII protocol.<\/p>\n\n\n\n
The freeware program CoolTerm<\/a> is a useful Serial Terminal application, because it can show you both ASCII and raw binary values. Download it, then open it. Click the Options icon, then choose your serial port from the Serial Port menu:<\/p>\n\n\n\n
\"Screenshot<\/a>
Figure 1. CoolTerm options menu<\/figcaption><\/figure><\/div>\n\n\n\n
Click OK, then click Connect (Figure 1). You should see the same random characters you were seeing in the Arduino IDE’s Serial Monitor as shown in Figure 2. (make sure you have the Serial Monitor closed before you connect in CoolTerm because Serial ports can only be controlled by one program at a time)<\/span>. But if you click on the View Hex icon, you’ll see a very different view:<\/p>\n\n\n\n
\"CoolTerm<\/a>
Figure 2. The CoolTerm serial terminal application showing the hexadecimal view.<\/figcaption><\/figure><\/div>\n\n\n\n
Now you’re looking at the ASCII characters for each byte in the right hand column, and the raw binary values in the center column (in hexadecimal notation) as shown in Figure 3. This is really handy when you’re trying to interpret a binary protocol.<\/p>\n\n\n\n
Click the Disconnect icon in CoolTerm  (because  Serial ports can only be controlled by one program at a time<\/span>) and upload the ASCII-encoded Serial Arduino program above. Once the program’s uploaded, connect in CoolTerm again and look at how the hex view has changed:<\/p>\n\n\n\n
\"The<\/a>
Figure 3. The CoolTerm serial terminal application showing the hexadecimal view. This time you see the numeric values of the ASCIII characters.<\/figcaption><\/figure><\/div>\n\n\n\n
With this version of the program, the ASCII view is more readable. In the hex view, you can tell the commas from the regular data, because every third byte is 0x2C, or 44 in decimal, or “,” in ASCII.<\/p>\n\n\n\n
Related video: Serial 7 – Reading Strings<\/a><\/p>\n\n\n\n
Make the following final improvement to the program above:<\/p>\n\n\n\n
<\/span>ASCII-Encoded Serial Arduino Sketch with Line Break<\/span><\/h3>\n\n\n
\nvoid setup() {\n  Serial.begin(9600);\n}\n\nvoid loop() {\n  \/\/ read the sensor:\n  int sensorValue = analogRead(A0);\n  \/\/ divide by 4 to reduce the range to 0-255:\n  sensorValue = sensorValue \/ 4;\n  \/\/ send it:\n  Serial.print(sensorValue);\n  Serial.print(\",\");\n\n  \/\/ read another sensor:\n  int sensorValue2 = analogRead(A1);\n  \/\/ divide by 4 to reduce the range to 0-255:\n  sensorValue2 = sensorValue2 \/ 4;\n  \/\/ send it:\n  Serial.println(sensorValue2);\n}\n<\/pre><\/div>\n\n\n
When you view the output of this, you’ll see you get a string of two numbers, with a linefeed after each string. In hex view, these will be represented as 0x0A (linefeed) and 0x0D (carriage return). This is a simple multi-value protocol. You’ve got a comma separating the values, and a linefeed and carriage return separating each set of values. \u00a0This protocol is both easy to read, and easy for most personal computer programming environments to interpret. For more on that, see the Serial Duplex Lab using P5.js and p5.webserial<\/a><\/a>.<\/p>\n\n\n\n
<\/span>Managing the Serial Buffer<\/span><\/h2>\n\n\n\n
Before you read this section, try\u00a0the Serial Duplex Lab using P5.js and p5.webserial<\/a><\/a>. This section will make more sense when you’ve seen the programs in that lab in operation.<\/p>\n\n\n\n
The reason this form of  serial communication is called  “asynchronous” is that the data between the two devices involved is not synchronized. The sender can send when the receiver is not listening, and vice versa. Both sides typically maintain a serial buffer, which is a place in memory to store incoming serial data before it is used, as mentioned in the serial basics notes<\/a>. On computers with an operating system, this serial buffer is maintained even when your program isn’t running. So when you stop your program and re-start it, there may be data in the buffer from a previous run of the program. This can cause errors. Most programming  environments automatically flush this buffer every time you restart the program, but if the one you are using does not, then look for a function that flushes the buffer. It’s often called flush()<\/code> and it’s good to run it right after you open the serial port. In Processing and p5.webserial it’s called clear()<\/code>. <\/p>\n\n\n\n
In Processing:<\/p>\n\n\n
\n\/\/ in your setup() after you create a new serial object called myPort:\n  myPort.clear();\n<\/div>\n\n\n
In p5.js with p5.webserial:<\/p>\n\n\n
\nfunction openPort() {\n  \/\/ wait for the serial.open promise to return,\n  \/\/ then call the initiateSerial function\n  serial.open().then(initiateSerial);\n\n  \/\/ once the port opens, let the user know:\n  function initiateSerial() {\n    serial.clear();\n  }\n}\n<\/pre><\/div>\n\n\n
<\/span>Make Sure There’s Any Data To Read<\/span><\/h3>\n\n\n\n
The first thing you can do wrong is to assume there’s data there to read when there isn’t. Imagine the following situation:<\/p>\n\n\n\n
Your microcontroller is continually sending a string of three sensors values to a program on your personal computer, ASCII-encoded, comma-separated, and terminated by a newline, like so:<\/p>\n\n\n\n
234,23,142\\n<\/code><\/p>\n\n\n\n
The controller is not waiting for a response from your desktop program, it’s just continually sending as fast as it can. The desktop program reading this data is waiting until it sees a newline character, then reading the whole buffer. Then it splits the incoming string into an array and converts the values to integers. You want to read only when a newline character comes in. <\/p>\n\n\n\n
In p5.js using the p5.webserial library it might look like this:<\/p>\n\n\n
\nfunction serialEvent(){\n  let inputString = serial.readStringUntil(\"\\r\\n\");\n  \/\/ split the string into an array:\n  let sensorReadings = split(inputString, \",\");\n}\n<\/pre><\/div>\n\n\n
In Processing, it might look like this:<\/p>\n\n\n
\n\/\/ in the setup(), configure the serial object to generate serial events\n\/\/ only when a newline arrives, like so:\nmyPort.bufferUntil(‘\\n’);\n\n\/\/ then your serialEvent function looks like this:\nvoid serialEvent(Serial myPort){\n  String inputString = myPort.readString();\n  \/\/ split the string into an array:\n  String[] sensorReadings = split(inputString, “,”);\n}\n<\/div>\n\n\n
The receiving program may not be reading as frequently as the microcontroller is sending, and the personal computer’s serial buffer contains the unread bytes. You stop the desktop program, and there’s still a byte in the buffer, like this:<\/p>\n\n\n\n
\\n<\/code><\/p>\n\n\n\n
The next time you start your program, there’s a newline in the buffer even if the Arduino’s not running, so a serial data event is generated and your serialEvent()<\/code> function is called. But there’s no string preceding the newline character, so when you try to split<\/a> the string into an array, you get an error .  The serial communication between the devices was working properly, but because it’s asynchronous, it’s your job to check that the data in the buffer is what you think it is. That’s your job as programmer. You might address this problem by checking that there’s a valid string to read first:<\/p>\n\n\n\n
in p5.js with p5.webserial:<\/p>\n\n\n
\n let inputString = myPort.readStringUntil(\"\\r\\n\");\n  if (inputString != null) {\n    \/\/ when you know you've got a good string, take action:\n    \/\/ split the string into an array:\n    let sensorReadings = split(inputString, \",\");\n  }\n<\/pre><\/div>\n\n\n
in Processing:<\/p>\n\n\n
\n String inputString = myPort.readString();\n  if (inputString != null) {\n    \/\/ when you know you’ve got a good string, take action on it:\n    \/\/ split the string into an array:\n    String[] sensorReadings = split(inputString, “,”);\n  }\n<\/div>\n\n\n
Similar problems can happen in all serial environments.<\/p>\n\n\n\n
<\/span>Clear Any Old Data Before Reading<\/span><\/h3>\n\n\n\n
Although this solution works if you’re reading the serial buffer as a string, it doesn’t solve every problem. Another way to avoid this particular problem is to make sure that you’ve cleared out the serial buffer at the beginning of your program before you start reading new data. Once you’ve configured your serial object and opened the port, clear the buffer by calling serial.clear().<\/p>\n\n\n\n
<\/span>Make Sure All The Data Has Been Received<\/span><\/h3>\n\n\n\n
You can still get errors even if you’ve cleared the buffer and made sure there’s data to read if the data there to read doesn’t match your expectations. In the example above, your data sentence includes three ASCII-encoded numbers and a newline.  But what if the microcontroller doesn’t send that?  Maybe there’s an error in its program, or maybe there’s still data in the serial buffer (because you didn’t clear the buffer).  You should check to make sure that everything you expect is present before you operate on it. For example, imagine you’re splitting the input data into an array of three strings, then copying those strings into global variables<\/a> like so:<\/p>\n\n\n\n
in Processing:<\/p>\n\n\n
\n\/\/ global variables:\nint xPosition, yPosition, zPositon;\n\n\/\/ then your serialEvent function looks like this:\nvoid serialEvent(Serial myPort){\n  String inputString = myPort.readString();\n  \/\/ split the string into an array:\n  String[] sensorReadings = split(inputString, “,”);\n  \/\/ copy the first element into xPosition:\n  xPosition = int(sensorReadings[0]); \n  \/\/ copy the second element  into yPosition:\n  yPosition = int(sensorReadings[1]); \n  \/\/ copy the third element into zPosition: \n  zPosition = int(sensorReadings[2]);  \n}\n<\/div>\n\n\n
In p5.js with p5.webserial:<\/p>\n\n\n
\n\/\/ global variables:\nlet xPosition, yPosition, zPositon;\n\n\/\/ then your serialEvent function looks like this:\nfunction serialEvent(){\n  let inputString = serial.readStringUntil(\"\\r\\n\");\n  \/\/ split the string into an array:\n  let sensorReadings = split(inputString, \",\");\n  \/\/ copy the first element into xPosition:\n  xPosition = Number(sensorReadings[0]);\n  \/\/ copy the second element  into yPosition:  \n  yPosition = Number(sensorReadings[1]); \n  \/\/ copy the third element into zPosition: \n  zPosition = Number(sensorReadings[2]);  \n}\n<\/pre><\/div>\n\n\n
This works great until you don’t have three elements in the array. If there wasn’t a full sentence of data, the array might have only one or two elements, and your error is back. To solve this, make sure the length of the array is as long as the number of elements you’re trying to read from it like so:<\/p>\n\n\n\n
in Processing:<\/p>\n\n\n
\n\/\/ global variables:\nint xPosition, yPosition, zPositon;\n\n\/\/ then your serialEvent function looks like this:\nvoid serialEvent(Serial myPort){\n  String inputString = myPort.readString();\n  \/\/ split the string into an array:\n  String[] sensorReadings = split(inputString, “,”);\n  if (sensorReadings.length > 2) {  \n    \/\/ copy the first element into xPosition:\n    xPosition = int(sensorReadings[0]);  \n    \/\/ copy the second element  into yPosition:\n    yPosition = int(sensorReadings[1]);  \n    \/\/ copy the third element into zPosition:\n    zPosition = int(sensorReadings[2]);  \n  }\n}\n<\/div>\n\n\n
In p5.js with p5.webserial:<\/p>\n\n\n
\n\/\/ global variables:\nlet xPosition, yPosition, zPositon;\n\n\/\/ then your serialEvent function looks like this:\nfunction serialEvent(){\n  let inputString = serial.readStringUntil("\\r\\n");\n  \/\/ split the string into an array:\n  let sensorReadings = split(inputString, ",");\n  if (sensorReadings.length > 2) {  \n    \/\/ copy the first element into xPosition:\n    xPosition = Number(sensorReadings[0]);\n    \/\/ copy the second element  into yPosition:  \n    yPosition = Number(sensorReadings[1]); \n    \/\/ copy the third element into zPosition: \n    zPosition = Number(sensorReadings[2]);\n  }  \n}\n<\/pre><\/div>\n\n\n
If all of the data isn’t there for any reason, this if statement will prevent an error by skipping the part where you look for data that’s not there.  When you combine it with the check for a null string above, and the clearing of the serial buffer, you make your serial reading much more stable.  The final result might look like this:<\/p>\n\n\n\n
in Processing:<\/p>\n\n\n
\n\/\/ global variables:\nint xPosition, yPosition, zPositon;\n\n\/\/ then your serialEvent function looks like this:\nvoid serialEvent(Serial myPort){\n  String inputString = myPort.readString();\n  if (!inputString) return;\n  \/\/ split the string into an array:\n  String[] sensorReadings = split(inputString, “,”);\n  if (sensorReadings.length > 2) {  \n    \/\/ copy the first element into xPosition:\n    xPosition = int(sensorReadings[0]);  \n    \/\/ copy the second element  into yPosition:\n    yPosition = int(sensorReadings[1]);  \n    \/\/ copy the third element into zPosition:\n    zPosition = int(sensorReadings[2]);  \n  }\n}\n<\/div>\n\n\n
In p5.js with p5.webserial:<\/p>\n\n\n
\n\/\/ global variables:\nlet xPosition, yPosition, zPositon;\n\n\/\/ then your serialEvent function looks like this:\nfunction serialEvent(){\n  let inputString = serial.readStringUntil("\\r\\n");\n  if (!inputString) return;\n  \/\/ split the string into an array:\n  let sensorReadings = split(inputString, ",");\n  if (sensorReadings.length > 2) {  \n    \/\/ copy the first element into xPosition:\n    xPosition = Number(sensorReadings[0]);\n    \/\/ copy the second element  into yPosition:  \n    yPosition = Number(sensorReadings[1]); \n    \/\/ copy the third element into zPosition: \n    zPosition = Number(sensorReadings[2]);\n  }  \n}\n<\/pre><\/div>\n\n\n
When combined with a handshaking methodology as seen in the Serial Duplex Lab using P5.js and p5.webserial<\/a>, you also ensure that the serial buffer is only filled when you’re ready for new data. All of these practices are good serial communication practices and will make your projects more stable.<\/p>\n\n\n\n
<\/span>Parsing Text in Arduino<\/span><\/h2>\n\n\n\n
There are some tools in the Serial library of Arduino that make it simpler to parse text strings. For example if you know you are getting a string of numbers separated by commas, you can use Serial.parseInt()<\/code> like so:<\/p>\n\n\n
\nvoid setup() {\n  \/\/ initialize serial\n  Serial.begin(9600);\n}\n\nvoid loop() {\n  if (Serial.available()) {\n    int x = Serial.parseInt();\n    int y = Serial.parseInt();\n    int z = Serial.parseInt();\n    Serial.print(\"x = \");\n    Serial.print(x);\n    Serial.print(\", y = \");\n    Serial.print(y);\n    Serial.print(\", z = \");\n    Serial.println(z);\n  }\n}\n<\/pre><\/div>\n\n\n
Try sending comma-separated strings of three numbers to this from the Serial Monitor. You should get an output like this:<\/p>\n\n\n\n
x = 34, y = 45, z = 56<\/pre>\n\n\n\n
Sometimes you will get 0 values if a non-numeric character arrives (like a newline or carriage return). You can catch these by checking for the right number of bytes. For example, 34, 56, 78<\/code> followed by carriage return and newline is 10 bytes. So if you know you can expect at least 10 bytes, you could use if (Serial.available() > 10)<\/code>. There are a number of other useful finding and parsing functions in the Serial library, including:<\/p>\n\n\n\n