Transcript

Scot: 0:06 This is the sound and data channel. And I'm talking to Brian Lu. And I'm gonna let him introduce himself just because he knows more about what he's doing. So we've never actually physically met. But I've been at Meijer of his work for quite a while. Brian, can you give a little bio?

Brian: 0:28 Sure, yeah, I'm, I'm some currently the data visualization artist at the Museum of Natural History in New York City. My background is in computer science and visual arts. For this talk, I'll probably be mostly speaking about my sonification work, which I've been doing for the past few years, more as a creative endeavor, through a project called data driven DJ. And that's where I explore different and think about ways in which we could use music as a medium to communicate the underlying experience that you should get, I guess, out of that out of that particular dataset. So that's kind of the nutshell.

Scot: 1:14 And you're composing background that led you to feel confident about doing that.

Brian: 1:20 Zero composition background. And that's, that's kind of how I got into, you know, data sonification, or data music, and then have that music composition background, I had to figure out how to make music, using my existing skill set being, you know, computer science, data science, and general kind of design aesthetic, right. So so it's kind of, you know, I kind of used the the idea of like collage or like, you know, hip hop music, where I sampled different things, you know, different references, different aesthetics, different data sets in order to produce these pieces. And, yeah, it's just almost by necessity, because I just, I just can't, you know, manually compose a song myself, Well,

Scot: 2:12 I mean, to my ear, that's very musical work, actually more musical than a lot of certifications that I've, you know, and I'm kind of ravenous about shaking everything up. Because I got the sound and data channel and I, myself am working on his research and on the area of what I've kind of been referring to as ways of listening for information. Because everybody's eyes glaze over when you say sonification, right, which is the whole thing, that's the data driven DJ thing is kind of in that same area. And our project is called Data stethoscope so that it's also easy public read. So there is this kind of distance between, you know, what people are expecting and understand and what, what's actually going on, I think, with what I refer to as sauna fires, like yourself, you know, and so, but you've done a bunch of work, man, it's like, you should go to Brian foods calm Brian foo. Oh, one word. And one thing your computer scientists, I can't help but ask foo. Is that like, it's not like,

Brian: 3:23 interviews, real name. I mean,

Scot: 3:27 I didn't think about until I was talking to a computer science guy, and he went like, Oh, yeah. Gotta be a nom de plume, you know?

Brian: 3:38 yesterday and before, before I was born, I guess. Yeah, right. Right.

Scot: 3:45 Okay, so, um, so these aren't in chronological order on the page, I'm looking at the breakdown of the pieces, which started with the early artwork, and then go up. And what I noticed the first piece was like memories of underground is that maybe?

Brian: 4:03 Yeah, memory underground? Yeah. Sorry. Yeah, there's definitely a trajectory of my work. Around that time, I started to kind of combined my computer science and my art slash design background. And before that, those two worlds were mostly separate. You know, I kind of had my, you know, computer development practice. And then my kind of personal art stuff, which is mostly visual. It was kind of when I started working at the New York Public Library, and that was I worked there for about five years. Starting in 2012, or something like that. That's kind of where I got the bug for kind of combining those two worlds using the glue of like, digital humanities slash, you know, Information Science, slash data science. Basically. simply trying to think of ways to, you know, to combine algorithms and to combine, you know, massive collections or data sets to create interesting, compelling experiences, both to, you know, just make those underlying materials more accessible and more visible to the public.

Scot: 5:22 Well, one, one question I have just on the overall work, since I've kind of reviewed everything is that given your visual that you're beginning as a visual artist, all of your pieces are tied, they're multimodal. In other words, there's always a visual component that that integrates with the sound even if the sound is dominating. I noticed that Yeah. And that's a that's sort of a question area in the overall thing. I mean, we're working on some things that are only auditory, you know, faces, and you have the old, you know, Alexa and Google Home audio interface thing, which hasn't really interfaced with, with us. Ways of listening for information, sonification or anything like that. So. So when you are doing memories, memory underground, you use the tools that you'd already developed? Or you had to, how did you how did you start out? Is it like chalk? Or c sound? Or what's the audio? Or did you write your own code? You started this thing, beginning with this piece of memory underground. And you had to build up from the beginning. And you said you, what did you start with? How did that? Describe the?

Brian: 6:38 Yeah, I think, are you referring to the train song? Yes. So basically, that whole project, that agent, data driven DJ project, which is a collection of 10 different songs, right? I've used a, I kind of developed my own process for it. And it primarily uses three components. One is Python for the data processing part. Uh, huh. Chuck is kind of used for the kind of sequencing and kind of generation of the audio. Uh huh. And I use processing for the visual, and everything is done using code. So I'm not using any user interfaces or anything like that. And that's because that's where I'm most comfortable and most comfortable kind of coding things manually, rather than learning interfaces and things like that. Also, yeah,

Scot: 7:33 so I don't use the article or when using check your you just using check commands and that kind of

Brian: 7:37 stuff. Yeah, yeah. So it's basically I press a button and it plays the song. So it creates for a very interesting kind of creative process, where, you know, typically, when you think about composing a song, you would kind of maybe focus on a part of the song and develop it or, you know, tweak little things. Whereas this is I maybe tweak the algorithm, but it completely changes the song, you know, in its entirety. Because it, you know, changes the rules. So, so every time I hear the song, it's, it's a, it's completely new, and then be it's the first time I heard I heard it. And that's also because I don't really have a great music composition background. So I can't really envision what the song might sound like, you know, if I, if I create these rules, so

Scot: 8:31 well, you have a, I mean, there's a musical flair there because there's a musicality to the result that happens, which is very interesting.

Brian: 8:38 Yeah. I mean, that was definitely part of the The goal of this project, because, you know, this kind of was born out of me just wanting to make music, you know, that was just something I wanted to do. So I did want to, you know, intentionally think about aesthetics and kind of the musical aspect of these of these pieces. And and, you know, even though there are visual components, I did want the songs to be able to stand alone.

Scot: 9:10 Doing I mean, yeah, but there's a rich visual, like, it's a supporting if S aspect of the sound is definitely the dominant. Yeah, sensory input from them pieces.

Brian: 9:23 Yeah, the visuals are definitely meant to kind of put things in context or just kind of give you a frame. Similar to how you know, like a chart might have an axes or something, just because there isn't a common, you know, auditory language for it for these for communicating information with sound right?

Scot: 9:43 I just I call a vague taxonomy on the jury kind of problem. That's right. Have you had anyone like approach you that was like, I don't know. involved? In a neuroscientists or a social scientists for mixed attractions or you know, someone who's involved in global refugee movement, contact you to see if they could use this in a functional kind of way at all.

Brian: 10:16 Um, yeah, I mean, I have had people contact me for for various use cases. There was one song about mapping Costa Louisiana's plan loss with music, right? to Ben. Yeah, to blue. And there was interest about that, regarding communication, various land loss, you know, around the country. You know, a lot of the, a lot of the feedback I typically get that are interesting or not as much as use for science or research. It's more like, for example, the one about the brainwave data, right where the song? Um, you know, I was approached with other people who had epilepsy who wanted to hear their data, you know, in music. And,

Scot: 11:14 you know, they know about Chris's thing. Is that how they heard about it, or?

Brian: 11:18 I don't know, I haven't I didn't ask about that. Yeah.

Scot: 11:22 But yeah, typically, just for the listener, Chris chafe has the cerebella project, which is kind of the same with the same kind of material of onset of a epileptic seizure. That wraps it in gray, which is one of Brian's pieces is all about. Yeah. But so different. I mean, he uses I mean, I've heard both pieces and that they're, they both use voice, though. Interestingly, you use the imaging at CES. And he uses a voice synthesizer within Chuck itself. So it's kind of interesting.

Brian: 11:59 Yeah, I typically try to match the sounds, the appropriate sounds for each particular song. Yeah, I know, that sounds but the the kind of the genre,

Scot: 12:12 I think is very strong part of what what you're up to, actually, because it communicates directly. You don't have to write a four paragraph thing for someone to understand what the violins are the rising tide, and I interrupted. You know, one thing I'm curious about is like, the progression here, is it like track one, track two, track three, I'm looking just at the data driven DJ, calm now, because stuff was mixed with your visual stuff. And I didn't want to confuse what we're talking about. And there's a total of 10 tracks. Are they in chronological order? Or was it Nick, did you do the track numbering according to like, an aesthetic decision? Or was that about?

Brian: 12:56 Those are in chronological order in which I've made them? Okay, so that was the first appears. Yeah.

Scot: 13:02 Okay. Well, that'll be interesting for people that do go to data driven dj.com? and check it out? Yeah.

Brian: 13:09 Yeah, I did, I didn't really have a master plan to begin with, I just had a number of tracks that I was interested in. And it was more of me just thinking about each piece as almost like a mini research project, you know, something that I was interested in just learning more about. And then digging into what data sets are available and what ways in which, you know, it can provide a lens on those different data sets, to to produce, you know, a compelling song.

Scot: 13:41 So how, what was the inspiration for like, for example, two trains? No one approached you, that was just like a creative act, right?

Brian: 13:51 Yeah. So all of these are personal, non commissioned pieces. And this is all just, you know, purely personal, personal work. So, yeah, I think there's different ways I kind of get into a particular idea. One is either I find an amazing, interesting data set, and I feel like there's something in there, um, or I have a particular interest in a particular topic, and then, or an idea for a song, and then seek if there's any kind of datasets available for that. Um, but a lot of times requires a bit of data visualization or other ways to look at the, to explore the data itself, to understand if it's a good candidate for a song or not. Uh huh. Um, you know, typically when I think about like a, you know, an interesting song, it wouldn't have a lot of contrast, color, movement, you know, how it would have a particular progression or something like that. So, so typically, that might translate into See, if you're looking at a graph, you know, it might have peaks or anomalies or, you know, something that that's interesting. That would translate into a song that has a lot of contrast of color, and when it'd be flat, so to speak. But there's some songs where that was almost the point, you know, so that was like, there was one song called color balance. Uh huh. I would kind of diversity of gender, race and ethnicity and movies. And the point is that, that that song doesn't, it's kind of boring, you know, because there is a lot of diversity in the song, there's a lot of diversity in data. So, so yeah, it's it's kind of really thinking about what the strengths are, you know, of music and sound, right. And think, ways in which, you know, I can leverage that with a particular dataset.

Scot: 15:57 Now, since your background isn't in music, but some of this stuff is incredibly tonal, or using like a scale functions of Chuck, when you are generating pitches, like in music elliptica, sir.

Brian: 16:11 No, so I mean, all of the all of the pieces are sample based. I see, you know, the sounds, so that that was also an aesthetic decision. But it also created another interesting creative act of like selecting the right samples, right?

Scot: 16:29 Because, for example, Rhapsody and gray, I played the imaging heap piece against it, like they just did with each other. Anyway, we're about to say that,

Brian: 16:43 yes, I use samples. So usually, for each piece, I think about maybe, kind of, for combining four different three or four different things, though, one is, you know, obviously, the data set itself. Then I try to think about what is the right kind of music genre, or kind of general aesthetic that I'm looking for, um, and then seeking the right samples

Scot: 17:12 to use know that prom and interesting, actually, yeah,

Brian: 17:17 yeah. And then, for some of them, there's some, like, specific compositional strategies that I actually sampled from usually, like, you know, minimalist composers like Steve rice, or Terry Riley. Uh huh. And you know, the

Scot: 17:34 airplane. Yeah, yeah. Yeah.

Brian: 17:37 Um, so yeah, I kind of, I think of it like a collage where I'm taking all these different pieces, and just kind of putting them together, creating new ideally. And kind of, you know, so so more that the selection and how I map, you know, the data to specific sounds is more of the creative act acts that I'm doing here. Rather than thinking about the right, you know, scales or melodies or things.

Scot: 18:08 So you set up this sort of, you know, Magister ludi Glass Bead game setup, and you push the button and stand back and let it roll. Is it?

Brian: 18:18 Yeah, that's Yeah. And kind of the hidden the hidden work here is that you know, that every song that you hear is probably like the 200th song. So, you know, good is that, you know, you should probably hear some of the first things that kind of, you know, come on to my speakers. Yeah. And even good. Yeah. You know, even the entire kind of algorithm sometimes changes, you know, or the, you know, it your approach. Really, yeah. So, um,

Scot: 18:53 have you ever, like, just given up on a sample that you thought was going to work and move on to a different approach that way?

Brian: 18:59 Oh, absolutely. Yeah. Yeah. I

Scot: 19:02 painstaking processes.

Brian: 19:05 Yeah. Yeah. It's, it's, it's both like, it's very, like, there's a strength to it, and that I can just generate completely new songs complete, completely different sounding songs. Yeah. But, um, you know, if it's, if it if it kind of doesn't work, like, I can't massage it, you know, like, in order to kind of be true to the, to the, to the algorithm and data, right. Um, you know, either works or doesn't, you know, I can kind of tweak some of the parameters, you know, to make things you know, change a little but I know that it is it is kind of just kind of mixing mixing things together to see what works and and you're doing this all on the command line. So

Scot: 19:49 there's no like, you know, slider design gesture and exactly most Yeah, yeah, that's an interesting and very powerful way to do it. Actually. I remember minded of the the very first data pieces I did were like with Hollerith cards and Fortran on a Cal comp water, you know, and it was the same kind of thing it turned out, and then you get, you get back this piece of tape that just went for the next day. And, yeah, so I remember working like that, and I really kind of, it's inspiring to talk to you about it. So, um, the other thing I've noticed, with this listen to your work is a sort of accessibility of it. In other words, you can play any of these pieces for a broad range of audience, it's not like, you know, playing Xenakis or, you know, even, I don't know, certain punk bands and that kind of stuff. Instead, it's, it's more a very sort of accessible and not a, you know, you don't pay attention to it that being dissonant, or any of that kind of stuff. You're more paying attention to the contour of it. That was clearly in aesthetic and a decision on your part.

Brian: 21:08 Yeah, absolutely. And I think accessibility and aesthetics is again, like something that was was a goal in the beginning. And yeah, I think it is interesting, and I kind of, you know, learned this along the way is that, you know, you know, one of the strengths of music, in addition to, you know, some of the, you know, kind of the more quantitative ideas of, you know, hearing contrast hearing change, is that it it's it's sticky, you know, if it's done, right, and if you have the right sounding song, you know, it gets stuck in people's heads. Yeah, and in this case, you know, there's also information attached to it or a particular subject. So that's that, that that also gets stuck in their head, which is interesting. And that's not something you can typically do with a chart, you know, as well, you know, you

Scot: 21:58 can have, right, so long term memory, I guess. I mean, it's built up out of short term memory, so enough repetition, and you start to develop, yeah, I've kind of got a theoretical basis for some of that, in terms of the cognitive part of what makes good sonification work. And this is a class of them, my, you know, weird, or trying to figure it out, because I'm really fascinated that it hasn't scaled, I mean, you know, you got the Geiger counter, I got the, you know, a Vario for hang glider pilots and a turn signal keeps clicking, you know, you're not looking at it. That's about it. The whoosh sound when you send a mail on a Mac, you know, but, you know, it's it's very strange to me, because it's such a powerful thing. And I think this approach of using sort of, I don't even say nostalgia or emotion as a way of Have you ever heard of Michael winter, you know, him at all?

Brian: 22:56 I'm not familiar, I

Scot: 22:58 don't think, oh, he's working on the idea of sonification and emotion. And he's kind of in the same zone of like, creating an aesthetic, memorable experience that's related to the data's shape, so that you can have a long term memory of the thing, you know,

Brian: 23:17 yeah, that's, that's definitely something I think about, you know, with these pieces, you know, what is, you know, how should the listener feel, right. Which is, which, again, it's kind of in this gray area between being an artist and being a, you know, data scientists. Well, your,

Scot: 23:32 your day job, right? Is it the Natural History Museum of New York? Yes, yeah. Yeah. So you're doing this kind of thing in public display and internal documents for them? I imagine.

Brian: 23:43 Yeah. Yes, I do. sonification in my work, here at the museum. Oh, really? It is. It isn't a it's it's more of a, I guess, more of a scientific approach, or it's supposed to, you know, be supposed to convey information more. I guess, in a less creative way, I guess,

Scot: 24:08 in terms of like, a hearing data, that's part of an exhibit that kind of thing, or,

Brian: 24:17 yeah, yeah. So one example is that I designed these interactives for the whole planet Earth. Okay, um, the section that talks about climate change. And there's this one graph that shows temperature over time, global temperature over time, so you can imagine they're kind of going up. For so you could so you can visually, you know, look at this graph, you could zoom in and out, you could zoom into a year, you could zoom out 250 years. And then there's a horizontal slider that allows you to kind of slide back and forth and play play the sound of the graph. And that that's supposed to teach this idea of kind of short term noise versus string. Long term trends. So right? Oh, yeah. So when you kind of zoom into one year, and you play the graph, it kind of sounds like random noise, because, you know, that's kind of the concept of, you know, short term noise or weather. And then you kind of zoom out 250 years and get here clear, you know, you know, scale kind of going up. Yeah. So it's, it's a, you know, there's a lot more parameters, and I have to work with, you know, and doing things for the museum, because there has to be, it has to go through a lot of repeat scientific reviews. So Wow. So they care about accuracy and kind of conveying the right message and things like that. So everything from like, what scale should I use, you know, what particular SAT, you know, what types of sounds should I use? So,

Scot: 25:52 so you, you're in that context, not your data driven DJ thing, but the stuff for the History Museum, you have to keep this same idea of general pleasing aesthetic for a product for a broad audience and still meet the requirements of the scientific community that's overlooking your work? Yes. Yeah. That's, that's correct. Yeah. And that's, and that's, that's the real thing. You're, you're doing the real thing right there. That's, that's actual, like sonification in the world.

Brian: 26:23 Yeah. But we I mean, that we have seen that there has been research internally, that we have demonstrated that using sound as an additional type of feedback. Yeah, it is very compelling, especially to the younger audiences. Yep. And it reinforces learning and all these all these great things. So

Scot: 26:45 I have a question, though. Do any of the scientists you're working with go, Hey, could you make me a tool that does the sound because I really want to use that on a day to day basis with my stuff. That had nothing to do with a public exhibit or any of that kind of thing?

Brian: 26:58 Right. I haven't had any of those types of requests. Yeah. See?

Scot: 27:05 That's the Willy Wonka ticket of this stuff. Yeah.

Brian: 27:07 Yeah. Yeah.

Scot: 27:10 Well, that's really interesting. I mean, I've gotten, I always ask these kinds of questions of whoever I'm talking to, because I'm really interested in that. Like, you know, they have that thing in finance, about jumping the chasm where something early adopters, and then there's like this thing where you jump and then it goes to a mass market. And sonification has never done that. And it's just a really strange. I mean, maybe never will. I mean, and why is that? That's like, Whoa, it's like really weird, you

Brian: 27:36 know? Yeah, I don't know. Yeah. Especially with the increasing popularity of things like podcasts and just sound based smart speakers.

Scot: 27:47 I mean, yeah, yeah, I can, I can imagine just hearing you know, having your smart speaker go wrap, you know, you know, it's gonna be cloudy tomorrow, and you know, but I don't know if that's ever gonna happen. So Brian food's really been great talking to you. Let's, you know, reach out anytime. If there's some project you want to, you know, talk about in more detail specifically about that. And just generally, this is such an interesting approach you have I love that idea of just pushing the button and having it. Okay, well, that's enough of that. I really, thank you so much, and I'll let you get back to work and I really appreciate you taking the time to talk to me.

Brian: 28:25 So thanks for having me.

Scot: 28:26 Okay, well, that was Brian foo, really doing some very, very interesting work. I hope you get a chance to check it out. data driven dj.com and also just Brian food calm as a larger picture of also his visual work. Amazing stuff. This is Scot Gresham-Lancaster for the sound and data channel on the creative disturbance podcast aggregate. Hope you can tune in next time. Transcribed by https://otter.ai