{"id":312,"date":"2018-06-11T16:47:55","date_gmt":"2018-06-11T16:47:55","guid":{"rendered":"https:\/\/itp.nyu.edu\/adjacent\/issue-3\/?p=312"},"modified":"2024-10-08T20:28:41","modified_gmt":"2024-10-08T20:28:41","slug":"ml5-friendly-open-source-machine-learning-library-for-the-web","status":"publish","type":"post","link":"https:\/\/itp.nyu.edu\/adjacent\/issue-3\/ml5-friendly-open-source-machine-learning-library-for-the-web\/","title":{"rendered":"ml5: Friendly Open Source Machine Learning Library for the Web"},"content":{"rendered":"

Technology is not enough. <\/span><\/p>\n

Consider the technology as a tool which, in itself, could do nothing.<\/span><\/p>\n

Treat the technology as something that everyone on the team could learn, understand, and explore freely.<\/span><\/p>\n

\u00a0— Red Burns<\/span><\/p>\n<\/blockquote>\n

Each fall at ITP, many students with a variety of backgrounds from all over the world see code for the first time in a course called \u201cIntroduction to Computational Media\u201d.\u00a0This is not a computer science course, rather it is a course about learning to think computationally and apply computation to one\u2019s passion whether that be art, dance, biology, social justice, fashion, and more. There are no tests, nor algorithms to memorize, only creative projects to make and the sole requirement is to simply give it a try. This is a key mission of ITP \u2014 find ways to make technology and coding more accessible and inclusive, s<\/span>omething that everyone can learn, understand, and explore freely and creatively.<\/span><\/p>\n

\"An<\/p>\n

The primary vehicle for the introductory course is <\/span>p5.js<\/span><\/a>, a JavaScript library from <\/span>The Processing Foundation<\/span><\/a> whose mission aligns with the course: <\/span>\u201cto empow<\/span>er people of all interests and backgrounds to learn how to program and make creative work with code, especially those who might not otherwise have access to these tools and resources.\u201d Casey Reas and Ben Fry, in <\/span>A Modern Prometheus: The History of Processing<\/span><\/a>, write:<\/span><\/p>\n

The original mission of Processing was to create software that made learning to code accessible for visual people (designers, artists, architects) and to help a more technical audience work fluidly with graphics. We aspired to empower people to become software literate\u200a\u2014\u200ato learn to <\/span>read<\/span><\/i> and <\/span>write<\/span><\/i> software. We wanted to change curriculum in universities and art schools around the world. Instead of teaching students how to <\/span>use<\/span><\/i> software, we thought it was just as important to teach them how to <\/span>create<\/span><\/i> software.<\/span><\/p><\/blockquote>\n

With Processing, an artist can invent her own tools rather than relying on those created and maintained by others. Today, being \u201csoftware literate\u201d is arguably more important than ever. Algorithms affect our lives in ways we couldn\u2019t have anticipated and the explosion of so-called \u201cdeep\u201d learning research has further extended their reach. Our interactions are now negotiated not just with each other, but with autonomous vehicles, ever-listening assistants, cameras that identify our faces and poses. Without access to and understanding of the machine learning models, underlying data, and outputs driving the software, how can we meaningfully engage, question and propose alternatives? Machine learning faces a similar challenge of approachability as simply learning to code did more than fifteen years ago. This is where <\/span>ml5.js<\/span><\/a> comes in: a beginner-friendly library for the web to introduce coders to machine learning, from algorithms and data to models and training.<\/span><\/p>\n

Barriers to Machine Learning While Learning<\/span><\/h2>\n

Beginning programmers face two main barriers with <\/span>current machine learning tools. Firstly, having to install and configure a development environment and secondly, having to implement low-level <\/span>mathematical operations<\/span>. <\/span><\/p>\n

Take Google\u2019s TensorFlow, one of the most popular open-source library for machine learning. The installation instructions require setting up and configuring a python environment through a complex sequence of commands entered via a terminal interface as shown in the following image.<\/span><\/p>\n

\"A<\/p>\n

In the summer of 2017, deeplearn.js paved the way for making machine learning easier to access by bringing it to the web. Deeplearn.js was an <\/span>open source<\/span><\/a> hardware-accelerated machine intelligence javascript library developed by <\/span>Nikhil Thorat and Daniel Smilkov, researchers from Google\u2019s <\/span>Big Picture Group<\/span><\/a> led by<\/span> Fernanda Vi\u00e9gas and Martin Wattenberg. Unlike the python predecessors, no environment needed to be installed and configured to use deeplearn.js. A student, artist, eductor, tinkerer, researcher could simply go to a URL and run a machine learning model in the browser. On March 30, 2018, Google announced that deeplearn.js became <\/span>TensorFlow.js<\/span><\/a>, in turn expanding the capabilities, features, and development team to support the project.<\/span><\/p>\n

Despite the ease of running a machine learning model in the browser, beginners who want to code their own still face the challenge of mathematical notation and low-level, technical code. Machine learning frameworks are commonly geared for people with advanced knowledge of calculus, linear algebra, statistics, data science, and several years of programming in a language like python or C++. While important for research and<\/span> development of new machine learning models and architectures, starting from this point can turn away newcomers with other backgrounds. Rather than thinking creatively about how to use machine learning as an artistic platform, beginners may be overwhelmed by fine distinctions between scalars, vectors, matrices, operations, inputs layers, outputs, and more. <\/span><\/p>\n

Additionally, the TensorFlow.js examples are written using the latest features of the JavaScript language itself which might be unfamiliar to anyone in a beginner setting. Here is some example TensorFlow.js code from the \u201cGetting Started\u201d example:<\/span><\/p>\n

[gist https:\/\/gist.github.com\/stephkoltun\/28dbb6c05f4e92b94ad3861c6d197370]<\/p>\n

This isn\u2019t a criticism of TensorFlow.js; the documentation and examples are excellent and thorough. It\u2019s simply for a different audience. While appropriate for an experienced data scientist with advanced JavaScript knowledge, it is not well suited for a beginning coder. In the fall of 2018 Daniel Shiffman, on behalf of ITP, received a Google Faculty Research Award to collaborate with the TensorFlow.js team and develop a higher-level library to address this gap.<\/span><\/p>\n

In the meantime, Cristobal Valenzuela (ITP \u201818) had already created a github repository called \u201c<\/span>p5deeplearn: deeplearn.js meets p5<\/span><\/a>\u201d. This repository was the genesis of ml5.js, with the \u201c5\u201d paying homage to p5.js and the Processing philosophy itself. At ITP we organized a weekly working group to discuss developing the library itself and invited guest artists and researchers to participate in its development. The guests and collaborators included Jabril Ashe, Kate Compton, Hannah Davis, Darius Kazemi, Gene Kogan, Kyle McDonald, and Yining Shi.<\/span><\/p>\n

\"Headshots<\/p>\n

Walking Through The Machine Learning Door<\/span><\/h2>\n

In an introductory programming course, is it possible to include a one-week module on machine learning? What tool could be used? The TensorFlow.js team opened the door to machine learning at a single URL with the increasingly ubiquitous programming language of the web: JavaScript. With ml5.js, we invite beginners to walk through the door. Built on top of TensorFlow.js with no other external dependencies, the library provides access to machine learning algorithms and models in the browser with examples that use beginner-friendly JavaScript. By introducing beginners to machine learning, we hope to inspire creative experimentation and provoke discussions about data, training, and algorithmic bias and ethics.<\/span><\/p>\n

So, how does this happen? What is the first example to introduce the topic of machine learning to a beginner? Let\u2019s consider the common task of image classification. ml5.js <\/span>(via TensorFlow.js) provides an interface for loading a \u201cpre-trained\u201d model from the cloud with one line of code.<\/span><\/p>\n

https:\/\/gist.github.com\/cvalenzuela\/09a40c72a5580ef9c5ccc59e51879f87<\/p>\n

This model, known as \u201c<\/span>MobileNet<\/span><\/a>\u201d was trained on <\/span>ImageNet<\/span><\/a>, a database of approximately 15 million images labeled with 1,000 different classifications. What the model classifies a given image is entirely dependent on that training data — what is included, excluded, and how those images are labeled (or mislabeled).<\/span><\/p>\n

As demonstrated on ml5js.org, the model and <\/span>ImageClassifier<\/span> object can be used to classify an image.<\/span><\/p>\n