Code goes in, art comes out: everyone is an NFT artist

This talk was given at Clojure/conj 2018 and focuses on some interesting issues about practicing generative art, techniques that have been used inNFTcarried out.

welcome everybody. Thank you. Thank you very much for coming. I'm excited to talk about a topic that I really love, which is artwork, and specifically, I'm going to talk about artwork created through programming: what you might call generative artwork. This talk will not provide a technical practical guide like making generative art, but an exploration of some interesting philosophical questions that arise around the practice. Generative art is a relatively new medium, and it's very different from some of the traditional art forms that came before it, so there are a lot of new problems that I think are very interesting, and they're particularly relevant to us programmers, and I'll spend some time talking about My actual software setup, and the tools I use.

The artwork I make usually has no input. It is not any kind of data visualization or image manipulation. Everything needed to generate an image is contained within the program itself, and whatever the program outputs is the final product. Usually, I work with digital images as the final product, but I don't strictly do so. Sometimes I break into the physical world and I create something like this, which is a plotter painting. It's a real painting, but it was created with a plotter, a simple robot that I'll get to later. I mostly work with abstract images, but lately I've been trying to integrate hand drawn components, so I wrote a custom interface that allows me to use the input of digital drawing via a Wacom tablet as a starting point for the algorithm. In this case, I draw the outline of a figure and pass it to an algorithm that takes over from there and breaks it down into different shapes and assigns all the colors and details, but mainly this [final image] is Something I've been working on for the past five years or so.

Here are the tools I'm using: I'm using Quil, which is a Clojure library. It's a wrapper around another Java library called Processing, which I think has been around for 10 or 15 years. Processing is a basic graphics library. It has a relatively simple API for working with shapes, lines and images. Processing was originally written as a Java library with its own weird IDE, but you can wrap it in Clojure using Quil. Processing was also ported to Java Script, and Quil wraps it in ClojureScript as well. You can choose which platform you want to use.

It's just a very simple program to give you an idea of ​​what it's like to set it up. There are very few boiler plates. This program doesn't draw anything particularly interesting. It's just a bunch of jumbled lines around the image, but the nice thing about this is that you can just start creating the image, and the API is relatively simple.

At the bottom, I get (begin-shape), a bunch of calls to (vertex), and then (end-shape), which is 90% of the way I use these APIs. Occasionally, I'll do more advanced stuff like layers and masks, but that's the core of it. You can use the system as vector graphics or raster graphics, so you have access to the actual pixel array. You can choose a different renderer to use. You can work in 2D or 3D, although I prefer to work in 2D myself. You can do animations with this, so on top of that I have a (no loop) call to stop it from doing that, but normally it calls draw repeatedly for each frame, so you can refresh and redraw, so people create like animation. They also create interactive software this way. You can accept keyboard input, mouse input, Wacom tablets, it so why did I choose Clojure for that?

It got even better once I found out that there are actually tools for this. Unfortunately, I started with Matplotlib, which is as painful to use as you can imagine, but Clojure is much better, and I think the main reason is that creating visual images is actually a kind of data processing. You are dealing with these shapes, curves, points, pixels. All of this is very conveniently represented as simple data structures, so it maps very well to the typical Clojure-style operations of setting up these data transformation pipelines on simple data structures. It's very applicable to this area, and it's also very applicable to audio, so there are Clojure libraries like Overtone that let you make generative audio, and they work very well.

Of course, everyone here knows that Clojure is fast to write, and that's one of the reasons I think we all love it, but especially with artwork, you don't really plan that architecture up front. You're more of an exploratory experimentation mode, just plugging things in and running it to see what it looks like. So especially with the integration with the REPL, where you can reload the modules and re-run the code, it's a very tight feedback loop that's perfect for this kind of exploratory work. Of course, one of the big advantages of Clojure over other Lisp dialects is that you get access to the JVM, which is both a great piece of engineering and a great ecosystem. It's nice to have all these Java libraries you randomly need out there and not have to write everything from scratch every time. Of course, the performance of the JVM is relatively high. It works well with large heap sizes.

If you ever find yourself needing to optimize a bunch of array operations, you can use Java, and do it very easily, although I find this to be very uncommon. Of course, if you really want to deploy to the browser, you can do it through ClojureScript. If you're making interactive software, you start to have to deal with shared state that may operate concurrently, so the standard Clojure toolkit of atoms and agents, especially agents, tend to be useful for this way of working, you also have to be careful not to block the animation thread , so tools like core.async are very handy for doing this style of work. That last point is a bit speculative, but I feel like having a more flexible language puts you in a better frame of mind when doing this type of creative work.

So, in the few years I've been working on generative art, there are some big questions that I've spent a lot of time thinking about, and the answers to those questions aren't that simple. Maybe it's not surprising, but I think they make for some interesting stuff. So, the first one I want to talk about is how to programmatically make interesting artwork. I mean, that's really the key to generative artwork. However, I need to be careful what I say here: "interesting art" is a heavy term. What exactly does this mean? It can mean many things. Art can be interesting in many ways, but I think a relative property here is that good art does things in a way that we didn't expect. It has the ability to surprise us or teach us something new, or to see things from a different perspective. This is the difference between good art and craft. But we also have a need to understand artworks.

We really don't like it if we can't wrap our heads around it. Some artwork can be hard to digest, and that's okay, but ultimately we want to feel like there's some meaning, structure, or organization behind the artwork. It can't just be bundled together haphazardly. Even if we cannot develop a rational understanding of it, we must have some intuitive understanding of a work of art in order to appreciate it. So, if we appreciate art we didn't expect, does that get us into trouble when programming? Programming is one of the strengths that make it a useful tool, it's very predictable. We can write these systems that let us know what's going to happen in any given state, and barring any kind of error, it's a very predictable way of working.

This might seem to conflict with our desire to make unpredictable artwork, but many generative artists have tackled this problem quite successfully, so I'll discuss some workarounds. My own approach, and one that several artists I follow seem to use as well, is really a programming style that's more of a guideline than an accurate description of what should be created. To give you a visual example here, these are two images created from the same program:

Without any code changes, just by running it multiple times, I get two completely different results, but they still seem very aesthetically related in some ways. Barely a single pixel matches, but somehow they are tied together. The way to achieve this is to use randomness very selectively and very carefully, from high-level structure to fine details, and elements such as color choices.

The algorithm is mostly based on a kind of recursive triangle subdivision, with randomness used to do things like instead of perfectly splitting each triangle, it might be slightly off center or use curves instead of straight lines. It may recurse to a depth in some areas and not in others, and it tries to keep those distributions in an interesting way. The color might be inherited from a parent triangle to it's children, but it might also randomly switch to another color with various probabilities, so it's really... the program is I'm moving in a certain direction advancing, but without clearly laying out what needs to happen. I'm still trying to give the program room to surprise me with unexpected results, it's just a bootstrap randomness.

I really like another artist (Manolo Gamboa Naon). His name is hard for me to pronounce, so I'll avoid butchering it, but I have two images here, which I believe are from the same program:

Again, you can see that even though they're completely different, they maintain this aesthetic consistency, so it's a popular way of working. You can find other artists who seem to take this approach. There are also very different approaches that tend to rely on more emergent properties and emergent structures, so what I mean by emergent in this context is that you have a very simple set of rules that don't seem to do anything interesting, But if you run it repeatedly over time, these large superstructures start to emerge that you can't predict just by looking at the rules.

The basis of this really has to do with chaos theory, and a very simple example of a chaotic system is the triple pendulum. This is basically three interconnected pendulums. Although these objects are very predictable objects in their own right, when you combine them you end up with chaotic systems, meaning that any small difference in the initial state can lead to a huge difference in the final result.

So, here I am with some really nice artwork by Jonathan McCabe. He was working on an advanced form of cellular automata. A lot of you probably know what that is, but for those who don't, maybe you've heard of Conway's Game of Life, which is a cellular automaton. Essentially you have a grid with cells that can be turned on or off and you can step through and have some simple rules that control when the cells turn on and off like they have a certain number of Neighbors open like they'll be in the next step.

This is a more complex system. This is what he calls a multiscale Turing model, so he uses cellular automata, one for macroscale features and one for more detailed features, but there's a wide variety of outputs here, and every output of this program looks at It looks completely different. It's really wild, but it still has this coherent structure that we still somehow understand. This is very remarkable.

Another common approach is a bit like cellular automata, but without the grid, and something more complex than cells, so in this case we have what I call agents or actors that move within the system, And in some ways affects it greatly. And these also tend to interact with each other, so if two agents collide, maybe they'll change the color they're drawing. You can set these up with simple rules, like with a cellular automaton, you run through many steps and it builds this complex image. This also works really well for animations, but this is just a still.

One of the recent developments is machine learning. I think neurology is a great term. This is artwork made by Mario Klingemann, I have some of his work here. This artwork was made using something called a GAN, a generative adversarial network. It's a neural network, and it's trained in a certain style, but really the two levers an artist has when they use these are, first, the training set, and therefore the collection of images that the network is trained on. Another real point of control for them is the objective function. So that's how the network scores itself or scores itself to see if it's doing a good job, and how it should change to do a better job, but there's a lot of variation. This is just the smallest glimpse of what he does. I definitely recommend checking out more.

These artworks are all very different in style, but the common thread that I think ties them all together is that they do a great job of mixing randomness and structure. They're able to maintain this balance between the two until it's still surprising and unpredictable, but they still have some type of order that we can understand if only intuitively. One image that makes this very clear to me is the work of John Greene who makes tile art:

Obviously a tiling system like this is very orderly, but he also manages to introduce randomness in a way that creates a lot of really nice variety. So, I think we can satisfactorily answer the question "How do you make interesting art with programming?" We can say it is possible. There are different ways to programmatically create surprising results.

So, I guess the next really interesting question has more to do with the philosophy of generative art and where it might go in the future. We can translate our aesthetics into code, right. What I'm trying to do when I'm writing this program is somehow encode what I like into the program, rather than make it very precise, but the art is a very intuitive practice. It's guided by a lot of intuition and intuition, you can make false decisions about what to do, and programming is pretty much the opposite of that. This is very clear. It's very organized. It's very organized. So, this is a very unusual attempt to mix, add these things. As a result, many generative artists start to wonder:

I'm going to narrow this problem down to color. Imagine that you are tasked with creating a program that picks a color for each shape in an image to produce a pleasing effect. This seems like a doable task. Color is a three-dimensional space, very simple. But when you dig deeper, this color is extremely relatable. Any one color is affected by all the colors around it, and by the proximity of those colors, and by their size. Not only that, but colors have all these cultural connotations for us. We associate colors with different emotions. We attach colors to objects, locations and even time periods. Really good artists use color in unexpected or ironic ways, so it's clear that color isn't actually just a three-dimensional space.

I mean, all it takes to really do that is powerful artificial intelligence, at which point the definition of art becomes even murkier than it is now. Matt Pearson has an interesting short story (Death of an Artist, in Novelty Waves) that talks about someone writing an AI artist. I recommend checking it out. So, I think, the answer to the question, can we translate our aesthetics into code, can we do that, but not all. It has to be a collaboration between the artist and the show, and that's okay.

So, the last question, and perhaps the most important one, why do you want to do this? Why make art with code? I think the most straightforward answer is basically that it's fun. We spend all our time using a very powerful tool to develop these incredibly complex skills, but we rarely use it in interesting ways.

How many of you actually sit down and write code with no predetermined goal? Yep... like the five of you, it's probably pretty rare for a few of you. It's not something we would normally do, and I think we really should. Doing interesting things with these skills now and then is a waste. In addition to the artistic challenges, there are many interesting engineering challenges of doing this type of work.

If you want to write really fast code, you can write a real-time GPU shader, these things have very high throughput. If efficient compact code is your thing, there's demo scene art that's all about generating stuff from incredibly small executables. That's a whole world that fits in a 4 KB executable. This is just a still image of a video. It's a whole terrain with mountains, smells, clouds, oceans, and it's very exciting.

If you like robots, we have robots. Somebody got these crazy nine-axis industrial things and painted portraits. I had to satisfy myself with a simple three-entry robot.

This is an AxiDraw. It's made by a small company in California. I use it for sketching and drawing, but sometimes it's really nice to be able to do something in the physical world. It's really an underexplored space, so if you like doing something, be the first to do it, it's really easy here. You can walk in almost any direction in a fraction of the time, and you're in new fertile soil. As far as motivation goes, it's really great. With just a few hours of work, you can get some instant visual feedback. l This can be bad feedback, but not always. You generally enjoy what you create, so this is not a project that will take months or years to achieve the desired result. At least for my personality, having this kind of motivation is useful.

I think it goes without saying: you're not going to do any of this.

I touched on this a bit, but honestly, we should be able to have fun with these skills we develop. We really don't have a good reason not to. I think we should get into the habit of relaxing a little more often. But, on a more serious note, I do feel that we should actually devote some time and energy to these things for society. We have an understanding of technology and programming that most of the world doesn't. I mean, to most people, it's a complete mystery to them, and artwork can sometimes bridge that gap.

For example, consider state-of-the-art machine learning. If I go to someone on the street and ask them "what are the strengths and weaknesses of state-of-the-art machine learning algorithms?", they don't know. For some of you here, that might even be hard to answer. However, if I ask them to listen to a short piece of music composed by machine learning algorithms, or have them read a poem generated by these algorithms, or look at a piece of art, they will quickly and intuitively understand what is going on with it. They will be able to hear. Well, maybe it can get the details right. It can string these short note sequences together in pleasing fashion, but it ignores the bigger picture. It doesn't have any drama. There is no story. No tension.

I think the same intuitive understanding can happen not only in this case of machine learning, but also in various works of art created through programming. I think there is an educational aspect, a very important communication aspect. Generative art also tends to explain things to us about the world, since the universe itself is generative, based on laws and probabilities. These lead to certain patterns, emergent patterns, and generative artwork-heavy processing patterns. Thus, whether intentional or not, generative works of art often stumble upon some of the same patterns we find in the world. So, a cool example is seashells. I don't know what species it is, but the shell is just right, and the patterns in there happen to be very, very similar to Stephen Wolfram creating (or discovering) a cellular automaton called Rule 30, except that it has some glitches. But it's really remarkable how these things are reflected in nature.

To delve into the deeper layers of philosophy, I think generative artworks also say something important about us as human beings. Art is fundamentally about being human in many ways. When we try to write rules about works of art, we try to make a more explicit attempt to describe what we appreciate about works of art. And this approach quickly revealed its limitations. The complexity and high dimensionality of works of art became crystal clear, and the gap between how programs work and how humans work became more apparent. We're starting to see that maybe these programs can address the 'how' of making art, but they can't really address the 'why', so it highlights the importance of the 'why' in art, and how we as human beings.

So, I think this approach to generative art, at least to me, reveals a lot about the nature of human life. The last part I want to say is that any culture without artwork is dystopian. You want artwork to belong to any healthy culture. Art encourages discovery. It encourages reflection, communication and an appreciation for life. Given that programming is basically eating the world - it's becoming more important every day and our lives are increasingly surrounded by technology - we need to make sure that we embrace the importance of art, that we invest some if we want to make sure Building a healthy culture for all of us as we move into the future, we will spend our time and energy enjoying it and creating it.

Here are my best answers to those questions. I hope some of you will want to learn more about generative artwork or make some of your own. With this, I'll say thank you.