Beyond Scratch – Creative Software in the Real World | Postcards from a Creative Coder
October 10, 2019
For lots of kids, teens and non-technical adults just beginning learn to code, educational tools like Scratch that can be deployed to create games and animations using block-based coding languages are an accessible starting point. But how might this translate to actual products and projects that go beyond the platform, and where can aspiring creative coders go from here?
As part of our series Postcards from a Creative Coder – Dispatches from Silicon Valley, our Chief Tinkerer, GP, explains the link between learning and creative coding tools, and shares his shortlist of creative coding tools that you can tinker with to create real-world projects.
Coding is an important basic skill for students, and there is a lot of effort to build tools that allow learners to get started quickly. Your children are probably familiar with tools like Scratch and MakeCode. But an important question is: Can kids build real-world skills using those systems, and what can they learn next?
As it happens, there are a lot of programming environments designed for learners and non-coders to get powerful results. These fall in the category of “creative coding” tools, and include everything from code-based environments to multimedia and game development systems. In my work as a creative technologist, I’ve had the opportunity to use a lot of these. These systems are a great next step for learners, and are a great way to apply the skills learned from educational systems.
The connection between learning and creative coding tools
Creative technology tools are often designed to enable the “creative” side of the equation by making powerful technology available to artists, designers and creators whose primary focus may not be writing code. The focus is instead on getting something working quickly, and allowing for fast iteration.
There are many features these systems have in common with educational software:
- Easy to get started: they all present high level modules that can be used directly, without a lot of setup
- Iteration: the cycle of creating and testing is very quick, you can see results of your changes almost immediately
- Very rich and responsive multimedia output: they all support high quality, live video and audio presentation
- Extensible: they can all be extended to support new capabilities, in particular new and unusual input devices
- Polished results: the presentation of output can look crisp and finished
Since creative coding tools have so much in common with educational environments, they are a great next step for learners – with real world applications.
Tools for creative coding
Here, I’ll explore three different categories of development environments that are used in the real world, that share a lot of the same heritage as those that students use to learn coding:
1. Multimedia creation environments
These are software tools designed to allow creators to build interactive works without writing a lot of code. They include ways of defining installations that use a graphical user interface, where blocks and modules can be connected on a screen to create an installation.
Other examples of multimedia creative tools include the famous Max/MSP, which has been around for 30 years and is now part of the Ableton family of creative tools. Max was one of the very first systems that allowed a user to create software by linking together modules visually.
Another great coding tool is Touch Designer, by Derivative Inc. This tool is particularly well known for its visual quality and high performance realtime graphics. Touch Designer was first released in 2008, and is closely related to the Houdini visual effects program. It can integrate a large variety of physical sensors and other systems. When the built-in blocks are not enough, a developer can write programs in Python that extend Touch Designer.
In my work, we’ve used Touch Designer to build large-scale interactive video walls, and for playback of very high resolution videos. This giant living wall would respond as users walked in front of it:
2. Game Development Systems
Video game developers have many of the same challenges as creative technologists, trying to express a creative vision that is immersive and responsive. In recent years, tools used for developing video games have become increasingly used for building interactive installations.
At Britelite Immersive where I work, we are big fans of Unity, a popular game development system. Unity is known for letting a developer build software once that can be used on a number of different devices, such as mobile phones, computers and video game consoles. Unity is also liked for its well-stocked Asset Store, an online market place for software snippets, 3D models and everything else that you could need to build a game. A strong point of Unity is the very strong support for AR (augmented reality) and VR (virtual reality) systems. It is possible to quickly build an interactive 3D scene and make it available to users of many different systems. Unity supports programming in C#, a modern language that is based on C++ but has many additional features that make it more robust and reliable.
We used Unity to build this robotic surgery simulation game, that was presented at the opening gala for a new hospital in San Francisco. For this project we used a special screen that responds to touch and to objects placed on it.
We also used Unity as the software to drive these automatic beer pouring kiosks. On this project, we integrated Unity with a variety of sensors, including a pressure map to detect when a user approached, an NFC (near-field communication) payment sensor and a depth camera for taking AR (augmented reality) selfies of a user wearing their favorite team’s colors as facepaint.
There are other game systems as well. We use Unreal Engine by Epic in situations where very a realistic rendering of 3D systems is required. It also offers a way to program without writing code, through what are called “Blueprints”. This is a node based system reminiscent of the creative coding systems like Max and Touch Designer. If you do need to write code to achieve something that is not possible with Blueprints, you can write code in C++.
3. Creative Coding Frameworks
When ultimate flexibility is required, there is no substitute for writing code. Many systems have emerged that make it easier for creators to build code-based projects. There are programming frameworks for multimedia, embedded devices and machine learning, among others.
A very well-known example is the Processing environment, which provides an easy-to-use development environment for writing code in Java. It is free and open-source, and includes a vast number of extensions for different creative effects and interfaces.
However, Java is subject to significant performance limitations, so there is a lot of work to make higher performance languages accessible to creators.
The Cinder framework is a library of modules that support graphics and interactivity using the C++ language. While harder to use than purely graphical systems, C++ is able to provide very high performance in the finished apps, and frameworks like Cinder save a lot of development time.
We used Cinder to build the control software for this elaborate LED illuminated tree:
Another framework similar to Cinder is OpenFrameworks, which also allows creators to quickly build media rich applications.
Finally, we often have to program tiny computers that will connect to specialist sensors and devices. The Arduino environment, originally developed for education, has grown over the years to support a vast number of embedded systems. We frequently use Arduino to build software to integrate physical sensors.
An important subject that has emerged in recent years is Machine Learning and Artificial Intelligence. Since this is a new field, it is particularly important for more people to learn about this domain, and the learning tools are very good. These learning tools, due to their power and ease of use, can also be used in real-world applications.
One notable example is Jupyter Notebooks, a system that allows you to run Python code in a web page, and output graphical or textual results directly on the page. Python is extremely popular for working with Machine Learning applications. The Zumi robot from Robolink lets learners experiment with machine learning and AI in the context of a self-driving car. It uses real AI tools like Tensorflow, accessed via Jupyter Notebooks.
The learning path
Systems like Scratch and MakeCode are excellent for learning the basics of programming, including concepts like iteration, conditions and variables. In fact, MakeCode goes one step further by letting you switch between seeing your program as blocks, and as TypeScript (a modern evolution of Javascript).
A student who starts from Scratch, moves on to MakeCode, then Javascript or Python is well positioned to use something like Max/MSP or Touch Designer in ‘real’ projects.
The game design engines are also quite accessible, and clearly interesting to most students! Unity in particular has a wonderful set of tutorials oriented towards beginners.
So for a parent, teacher or student that is asking “what’s next?”, the creative coding tools are a wonderful path to explore.