Transcript

Scot: 0:11 And Mark Bollora on the line here. And what we're what we're talking about and reason why I really thought to reconnect with you was that I'm having trouble finding people working in the sonification area that have had success, having scientists adopt their the research that they've done on a regular level. And I got the impression at least your heart murmur things could possibly have have gained some traction as a used technology in cardiology. Is that true at all? Or am I?

Mark: 0:50 No, they haven't gained traction? No, that's an unfinished project from my past. We got as far as proof of concept, but no, nobody's using it for anything.

Scot: 0:59 Do you have any projects that that you find, have been accepted? In any discipline as as a useful item? I mean, Geiger counter level useful?

Mark: 1:13 Well, I don't know Geiger counter levels a pretty high order. But ...

Scot: 1:17 yeah, well, you know, heading that way.

Mark: 1:19 I mean, what changed? I think what changed was when I had the opportunity to work on the the rhythms of the universe film that that Mickey Hart was making with George Smoot. So that was, that wasn't research, that was outreach and communications. But that that was kind of a paradigm shift for me, because it got me thinking, you know, maybe, maybe trying to change the research environment, all of a sudden, with a brilliant new invention is unlikely to happen. I don't seem to be doing it. And neither is anybody else. And maybe it's better to focus on making cool educational components, and then make it part of what kids get in their science classes at school so that they just grow up thinking of sound is a normal part of how you study science. And then by the time they grow up, they'll be the ones coming up with the killer apps, and making the great discoveries with it.

Scot: 2:23 Wow, well, so that's really interesting.

Mark: 2:27 That was work on that film and having some of my stuff in Mickey Hart sound library that he's used on albums and on stage. It's not exactly a research milestone, but it was certainly a, I think it was an important step for sonification as a field, actually, I mean, to to have the Grateful Dead community talking about sonification. There's a lot of people there. So that opened some doors for me, and I got it accepted at Penn State. So I started getting asked to give presentations at ... there was a thing called polar day that they did each year, where they talked about climate issues relating to the polar areas. And so that brought some attention to it. And that led to an ongoing collaboration I've been doing with a meteorologists named Jenny Evans. And that's, that's kind of done both things. She She wanted me to create some sounds of tropical storm dynamics, which is what she works on. And she liked them. I added them to some satellite videos, like you see online and like you see behind the weather people on television, and she liked them. She wanted to take them to a conference in Korea, and she did. And it was twofold. One, it made the videos a lot more interesting to look at it made them a lot more visceral. That's a word that a lot of people seem to say.

Scot: 4:06 It was like multi modal, you're saying

Mark: 4:08 Exactly. Multi modal. And, I mean, it was like the difference between a silent movie in a movie with sound, you know, it suddenly brings you into that world, but she also is a researcher found it quite useful to be able to hear all of these simultaneous parameters. Like she said that she studies the phase space of tropical storms. That's her, that's one of her big contributions to the field. Usually when a storm gets more symmetrical when it gets rounder, it's more intense. It's like when an ice skater pulls her arms in and spins faster. It's the same thing with a with a storm. So she can see on a video when it's getting more symmetrical, but sometimes the storms would get more intense when Without getting more symmetrical, and she couldn't very well see that, but she could hear it in the data. So that was like, Yes. Golden spike there.

Scot: 5:07 Oh yeah, yeah, there you go.

Mark: 5:11 So that's everybody, whenever I give talks on sonification, and I show that one everybody likes that one, everybody, or if they asked me to give a talk to say, Oh, are you going to show the storms? I would consider that one to be successful. So we're looking to do more of that. And as far as as an educational component that strikes me is particularly promising.

Scot: 5:40 But other meteorologists aren't like talking to her and say, Who is that guy? Let's do more of this kind of thing.

Mark: 5:47 Well, I went I visited an oceanographer at University of Rhode Island. I'll talk to you about that next week. But when I was there, she said, Oh, why don't you make this meteorologist and show them your storm Sonifications, a fellow named Isaac Guinness. And so I played them for him. And he said, You know, this is, this is really interesting, because he works in he works with disaster relief agencies and people who try to communicate the dangers of storms and and take action, right, or storms. And he said, you know, the problem is that we researchers sometimes get our research done. And so we're real clear on everything. And so we think everybody else is too. And that's sometimes not the case. Sometimes our graphs and our research just don't get the point across. And this could be really helpful. So he was having an open house, and he asked if he could include my sonifications in his open house. So that was kind of like somebody asking, Who was that guy? Can I use his stuff? So we're, you know, we're thinking that this is a promising thing to make as part of an educational package perhaps in in concert with a science museum or something like that. It's it's a pretty easy case to make now that storm dynamics are something that it's important that we have a better understanding of ask anybody in Houston, and I think that they'll agree with me,.

Scot: 7:22 or Puerto Rico.

Mark: 7:23 Yeah, yeah. You know, so it's a current and compelling issue that I need to know about in. And so we're thinking that this would be a good way to help with that. It's, it's interesting, a lot of scientists are realizing now that communications are about as important as research in terms of action being taken on their work.

Scot: 7:52 Well with this change in what they think science is a really bizarre time. I'm talking politically right now. But, you know, this is an interesting kind of thing that I found. I mean, I've been really scouring the literature, looking for real instances of, you know, scientists really finding like, what you're saying, like the golden app, or the, you know, the magic bullet kind of thing. And I'm not finding it at all, but this, but what I do find is this kind of thing like PR outreach, like the Higgs boson stuff is is the hugest case scenario of that someone just last night, Carla Scaletti, did a sonification concert that I heard about a friend of mine, went to the concert and talked to her afterwards. And she was basically saying the same thing. And she's been involved in this stuff for, you know, for as long as I known about it, you know,

Mark: 8:48 Oh, well, she's the one who named it. She's the one who, actually she didn't quite name it, but she's the one who contributed the term sonification to the auditory display community. Yeah, that was back in 1992.

Scot: 9:00 Right, right.

Mark: 9:00 We we had the we had the international auditory display conference at Penn State. Last summer. I was I was the chair of it. And Carla was one of the keynotes.

Scot: 9:10 Right, right.

Mark: 9:12 all that basic. I love Carla.

Scot: 9:14 Yeah, she's fantastic. I mean, I had a Kyma for a bunch of years at my studio at CSU East Bay, back in the day, but anyway, the and so I was in touch with her a lot then. And I also I've interviewed Greg Kramer, and I think that UR sources. Very interesting. The other person I'm trying to get a hold of is Thomas Herman. Oh, yeah. I'm sure he was at that conference as well. Probably Yeah. Most of the work is going on right now.

Mark: 9:43 Yeah, the whole outreach thing is is interesting. It's like I A few years ago, I would I was spending time running around Penn State, you know, rubbing sticks together and talking to people. And it's like someone time I met with had a nice cordial coffee with Richard Alley, who's one of the world's leading climate scientists. You know, you see him on CNN sometimes as a talking head and that kind of thing. He's one of the people that everybody looks to. And so I'm talking to him, you know, sound is data, you know, a lot of possibilities. And he's kind of nodding, and he's kind of cautiously interested, but he's not quite sure about it. And he's like, yeah, okay, you know, we'll keep this in mind. And we, you know, we parted on good terms. And, and, you know, it's one of these things that happens. I didn't hear from him. Yeah. And then a few years later, when I had this opportunity to present it polar day. And so we thought, well, we need to find somebody who can give us some data. Oh, Mark, why don't you go to Richard Alley's pizza lunch that he has on Friday with his graduate students? And I said, Oh, yeah, that'd be great. And, it seems they didn't have anything on the agenda for that day. So I said, Well, hey, you know, I've got this 22 minutes film rhythms of the universe, you know, that I worked on this film by Mickey Hart and George Smoot, if you want I could show that. And they were like, yeah, okay, sure. So I go there, and I show this film. And Richard Alley is absolutely, you know, doing backflips. He's like, Wow, this is great. fantastic method of outreach. Wow. You know, a few years and a good project early makes all the difference. And, but it's interesting, because he liked it, and that was great. Some people see it right away, other people don't, or sometimes people would come up to me, like after I've given a talk or something. And they have kind of this look like I'm a used car salesman or something. So what assessments have you collected on this? And?

Scot: 11:59 Yeah, like the whole CHI (Computer Human Interface) kind of approach?

Mark: 12:02 Yeah. Which is, you know, for somebody coming from the arts? That's just an odd question. We don't we don't deal with that. But people from the sciences do. And so I've gone from being kind of snarky and resistant to that to thinking, Well, you know, I, I really need to get something. You know, it's just a question of what surely there are good ways to assess this. And so that's, that's a ...

Scot: 12:25 Boy I mean, you're right in the zone that we are, we've arrived at actually completely. We're trying to have hope our standardized modules for types of sonification. And I mean, part of the reason why I remembered you had to do with that early heart work that you did, which I thought was very profoundly well thought out. I mean, I encourage any listener to go check out your that what was that project called? Again? I'm sure I don't have it right in front of me.

Mark: 12:55 It was heart rate variability.

Scot: 12:57 Yeah, heart rate variability. And I mean, it was such a clear example of, of a way of looking at something that that was familiar enough, because it was like, kind of stethoscop"ish", You know, but it required the extra step of what computer technology has given to us in terms of the audio world, and it was done in a very artistic and in a craftsmanship way, which is another component of this whole thing that doesn't get discussed as much, in my opinion, is that you really do need musicians and composers, because, well, I was just rereading Curtis Rhodes book, "Composing Electronic Music, a new Aesthetic", and he had the section where he's quoting Stockhausen from 1989, that's about a physicist would just do an audification. I would call like a direct transfer and say that that was equivalent, but that a composer is needed in order to sort of bring it out and make it available on a sort of leit motive level or something or on a conceptual level, you know?

Mark: 14:06 Yeah. Well, I Well, first of all, I'm gratified that you found those heart rate sonifications, musical. I mean, that was that was some time ago, and I've, let's say I've come away since then. So when I listen to that now, I kind of think that's kind of crude. But I mean, it's not bad. But I do better work than that now, but this

Scot: 14:30 Yeah, I think we're all we're all moving along.

Mark: 14:32 But I wanted to tell you about this. Yeah. Well, that was the thing about getting to do things with Mickey Hart, was our music was first and foremost, I knew that I would not be having any deep intellectual discussions with him about sonification. I mean, I knew that he would want it to have integrity, but I also knew that when I uploaded stuff, and he would download it, I would have like two seconds and it would either grab him or it wouldn't. And all the discussion in the world was not going to undo that first impression. So, having having contributed to something where it really had to function at a musical level was, I think, a really important step, and that plays into the whole assessment thing.

Scot: 15:17 This project was with a former Grateful Dead drummer Mickey Hart. And I went when I heard you are working with him, I had to tell this story. I mean, I just the thing is, I grew up in San Carlos, California. And there was Mickey Hart's drum world, which was owned by Mickey Hart, Father, and I bought and I bought my first 12 string guitar from Mickey Hart when he was working at his father's store. Wow, that's weird. You know,

Mark: 15:53 we really are distant relatives.

Scot: 15:56 So I thought it was a colorful anecdote to throw in, you know,

Mark: 16:02 So this whole issue of communications and so forth has been ... I mean, different people are starting to see this. So what came out of the work with Janny Evans. And the other work I got to do at Penn State was this opportunity to go to a ... They call it a conference, which was more like a workshop Think Tank by the Keck futures initiative. So their, their whole idea was to choose an important scientific theme, and then bring people together from different disciplines to brainstorm on how they might approach it. So last year, it was the deep blue ocean. And so I, you know, you applied online, my associate dean found it and said, Hey, Mark, is this sound interesting to you? And I was like, Oh, boy, does it. So I, you know, I applied online, and by that time, I had enough of a track record that I made a convincing enough case that they accepted me. So I so I went there. And I was, I met some people who found this interesting. So among them was an oceanographer named Karen Wishner. She's at University of Rhode Island. And, and she thought, Well, yeah, this, this could be pretty cool in introductory oceanography classes, when I'm presenting graphs of things to students who are new to this, what if we played them music? Or something musical, but how would that effect them? And I'm like, Yes, this is what I need. This is a whole class of undergraduates who are being exposed to science through sound. And then if we can get a science educator involved, and actually gauge how effective that is, that would be really cool. So we applied for a Naqvi seed grant, and we got it. Which, you know, which has been great. That's like, you know, a big foundation saying, here's a bunch of money, go do some sonifications. And I'm like, Wow, thank you so much. Yeah, it's. So that's why I was in University of Rhode Island a little while ago. That's where I met Isaac Guinness, who I was telling you about.

Scot: 18:16 Right, right. I got it

Mark: 18:17 but in the fall that's what we're going to be doing. There's there. I mean, we went from talking about this one module, it would be maybe a couple of classes where they show this one particular graph that illustrates the the types of little little sea creatures you find at different depths. And I thought that's what we were going to work on. But then they started saying, well, gee, we could do your storms too. And we could do things like the Keeling Curve for climate change, because this is an introductory earth science class. stuff all through the term. And I'm like, Oh, really. So that would be like a whole current? Sorry, pardon the pun, but current class is playing stuff for these introductory students.

Scot: 19:02 Okay, now, this takes me to a tech takes takes me to a technical question, are for each of the sonification that you just mentioned, are you using the same technique? That's a time series representation of, you know, like a Cartesian graph, or are using different techniques for each of those?

Mark: 19:21 Well a lot of them are. Yeah, I don't know that the one where, where, where she's showing people, the populations of different sea critters at different depths is is not a time domain graph. That's, that's she's plotting depth on one axis, and then she's plotting the populations of different creatures on another axis. So you have something like that. Then you say, Okay, so what, what is my time component going to be? So maybe what I need is a sound for the sounds like we're going underwater. I need a sound to represent increasing depth levels. And then as we go down, so it's like the sound of an elevator going down.

Scot: 20:04 Like it's a data narrative kind of thing.

Mark: 20:07 Exactly. Yeah, exactly.

Scot: 20:08 Yeah. So So I mean, that visualization. That's, that's a, that's a common theme in the training on visualization. And I, we were thinking of these training modules for the types of sonification modules that we're developing, one of which is the Cartesian graph, one, you know, but so, so this is really helpful to, to, you know, listen to your experience about it. And it confirms what we're sort of assuming. So that's great.

Mark: 20:35 Yeah, but so I, you know, I think that's going to be real promising that that got me a chance to show definitively that this is a good thing. And she's not an artist, but also for her for her, they may be able to hear some things by listening to them in this way. So I mean, that's what's so important is to have buy in from a from an important researcher like that from somebody who really wants to work with you. And and, you know, through it with you, I mean, when I worked with Jenny Evans, it took me, you know, maybe three or four iterations to get the storm sonification into a form that she liked. Aha, you know, the fact that she was willing to say, Well, okay, I see what you're doing. But could you change this? Alright, now that you've done that, maybe you could change this. And it wasn't like, some people that I've worked with her. It's like, Well, okay, Mark, let me see what you got. Oh, yeah. Well, okay, that's, that's not quite it. But But thanks so much. And it's like, you know, the performing monkey didn't quite come through if you are on that one, damn it. Somebody who will really work with you and develop something who's committed enough to it and who sees the potential in it, that they that they want to make it work. That's, what makes all the difference. So I would say that Jenny Evans wanting to continue working with me and Karen Wisner wanting to work with me, I would call those I mean, these are the bases of clinical success of sonification. Just to make sure I answer the very first question that you asked me,

Scot: 22:11 I'm going through the same experience, we got a similar type of grant, a DARPA grant for neuroscience, and I'm working with this guy, Gagan Wig, who's a neuroscientist, and, and he's very excited about, you know, we've done several iterations with him. And that's really is the key, I think, maybe, is that you have to get that kind of buy in. and it also sort of mutual respect, which is hard to develop. Because I, and this is a little blunt and frank, but I think a lot of scientists don't have this level of respect for for the sort of depth of craftsmanship, it really does take and, and sort of, you know what I'm saying?

Mark: 22:52 I do you and that's it? Well, it's crossing that divide. And I and I think that that's coming down, right? The science, what I'm finding, at least from my side of the fence is that scientists more and more are acknowledging the importance of the arts, in helping them put their word out. And also the division between the artsy people and the sciency people are kind of breaking down, which I think is, is healthy, it kind of takes us back a few 100 years to the age of the natural philosopher where you weren't just a a physicist or a violinist, but you were a renaissance person who did all kinds of things in each informed the other.

Scot: 23:40 Well, that's the classic Leonardo model, you know?

Mark: 23:44 Exactly. Yeah, I think I think that we're coming back to that. I mean the fact that Naqvi the Keck futures initiative would have these annual conferences and make sure that they had artists there. Right. And that they would say, yeah, this proposal between a sonification guy and an oceanographer sounds great. Here's $50,000.

Scot: 24:05 Yeah, well, that's the same thing that happened with this DARPA deal. I mean, they insisted that we have a performance connected to the research we're doing with the neuroscientists, which is like what, you know, but that we raised Yeah, so we did this thing in Seattle. I mean, you know, so So yeah, and I've heard the same thing in the EU as a as a big science art initiative. Of course, I'm working with Roger Malina who's kind of like, a linchpin in this kind of area. So he, he hears about all this stuff. So I get it kind of firsthand, or first row seat on the way it's flowing. But I but I think your approach is the thing and we have to refine the way that we're ... with the expectations that we have for the work and also the way of doing assessments as you were talking about before, and just in general, kind of try to understand why this hasn't been as successful as it as we Intuit it should be, I think, and it was As where I'm coming at. I mean, you know, like, why isn't there a new Geiger counter that's based on data? Or what we were calling a data stethoscope? You know?

Mark: 25:09 Yeah. Well, it's, you know, it's interesting. Carla scality when she gave her keynote at ICAD, and she showed some of these first line and area graphs that William Playfair produced in the 1780s. Uh huh. You know, and that's, that's regarded by many as the birth of visualization.

Scot: 25:29 Right, I know what you're talking about, actually. Yeah.

Mark: 25:32 And we look at these now. And it's just second nature to us. It's like, Oh, yeah. But she said, You know, there were 10 pages of explanation that went along with that. That's what was required to explain it. The first time anybody ever saw this? They didn't just immediately look at this and go, Oh, yeah, that tells me everything I need to know.

Scot: 25:49 Yeah, we're running into this training problem. I mean, you know, that, clearly, cuz we're trying different types of sonification. And, you know, every time you do something in a different way, you have to sort of give it a context, even though we, as the creators totally understand it. That doesn't mean anybody gets what the hell we're doing. You know, it's like,

Mark: 26:07 well, yeah, now you're talking like eyes against it. The meteorologist is like, yeah, I've done all my research. So why don't you get it?

Scot: 26:13 Yeah, right.

Mark: 26:15 And, and so that's why I think it's so important. That's why I'm I'm presently more interested in working in the educational domain in exposing kids to this and making this just a part of their science world. So that it's not an unusual novelty, but it's just part of what you do. While you listen to graphs. Of course, you do.

Scot: 26:39 in our, in our setup, it's certainly one of our use cases, which is a K 12. advanced science curriculum that we're contributing sonification research to, in the context of, you know, it's just identical to the use case you're talking about. So and it's probably is the richest, it's the one we've had the most success with, that's for sure.

Mark: 27:03 With kids.

Scot: 27:05 Yeah. So far. I mean, you know, certainly when we give a demo at the Art science lab, the things that work in terms of demonstrating the idea of what sonification is, or go over the best in that kind of context, you know,

Mark: 27:20 Wow,you should present that it I can, you know, and, and, and give us the number, you know, put the numbers in, you know, get get people who do the numbers who know how to do that? Or maybe you know how to do that? I don't know,

Scot: 27:32 that's one of the things I mean, this is what's great with working with Roger is he's, he's the one who's working on the assessment part of it. Uh huh. Which is great, you know, I can only do so much, you know, and and so, so we're, we're, we're trying to develop a kind of a workflow. And I, you know, I'll talk to you some more offline, when we're not doing a podcast. But I think we've covered enough for now everybody's brain is going to start hurting here. Pretty soon. Oh, sorry.

Mark: 27:55 No, no, no.

Scot: 27:57 I'm just saying I try to keep these down to about a half an hour and, and we just hit the half hour mark. So I'm going to i, the thing is, I'll call you again. And we'll talk some more like in maybe four or five months and see our let's say, you're going to do this next fall, is that what you're saying?

Mark: 28:11 No, this spring, this spring,

Scot: 28:12 Okay, so maybe, maybe early summer, I'll give you another ring. And then we can get an update on on your success and that kind of thing. I plan for this to be an ongoing thing.

Mark: 28:21 Oh great !

Scot: 28:21 In terms of the sound and data channel. And I Likewise, if you come across anyone that you would like to do a similar sort of interview and conversation with please let me know. And we can include you as a host on the same channel, because we're trying to build this up and get sort of a dialogue going about this kind of very specific area of the threshold between, I guess, the sonification, application creation and the art side as well, because there's just a ton of work that's been done. That is well, I don't know if Have you seen this guy, Martin Keary's YouTube, about sonification. on YouTube. It's a little snarky, but it's it's interesting. And he makes it makes a really some really good points. Actually, earlier today, I was trying to reach him. He has twin baby boys. And I guess something happened. He didn't get get ahold me, but I'm also interviewing him because his point of view is really on point in a way, you know. Yeah. But he sort of, I think your approach is much more positive, because it's true, that we have to introduce these ideas of what sonification is into the culture before we can expect there to be like a broader usage, I think,

Mark: 29:30 yeah, that it's I just accepted that it would take time that it wouldn't happen overnight, like flipping on a light switch.

Scot: 29:37 Right.

Mark: 29:37 If my third grader can grow up, thinking that this is normal, and by the time he's my age, you know, take the long view look ahead 30 - 40 years and hope that by then it has an effect that that that will be a lot less pressure.

Scot: 29:59 Yeah. Carla Scaletti's example of what was going on with the early visualization, right? Yeah, yeah, I think that's a really apt point. Anyway. Okay, so I guess that's it for now. I'm gonna stop the recording but let's just keep talking for a second after that. Okay, sir thing.

Mark: 30:16 Thanks for having me. Yeah

Scot: 30:16 Mark Ballora just came up with some amazing points on this whole area. And so we'll be getting back to him next spring INTRO 30:30 Sound and data