From Shane: I worked with Ryan back in the summer of 2017 to discuss accessibility options for a groundbreaking game he had developed for iOS earlier that year. It's called Blackbox, and is still available on the app store today. At the time, most mainstream game developers, especially those working on mobile operating systems, were not making a significant effort to make their games accessible. With a product like Ryan's, I fully expected the same response, but he jumped on, recognizing the real opportunity here. It had been years since wecaught up after the Blackbox project, and below is a transcript of our conversation.

‍

Shane: What's going on my friends? This is Shane Lowe coming at you from Super cents presents. today I'm hanging out with my friend Ryan McLeod. We used to work together, way back in the day back in 2017. How're you doing?

Ryan: Very good to be here

Shane: It's good to talk to you, man. What's your background? I guess let's start there. Where did you start? And what are you doing?

Ryan: What am I doing is a good one. Where did I come from? Let's see, I've always had an interest in computers and doing a lot of like, development type stuff. But ended up getting into web design, and then trying to switch over to iOS to be able to like design more interface type stuff, which…

Shane: When did you make the switch?

Ryan: I don't know developing on the web can be painful, and it's, it's kind of up on a screen,and I think at the time, like the stuff that was happening with iPhones is really exciting to be able to have this like direct interaction. It somehow feels much more personal. There's people developing like, just really interesting models of interaction. 

To that. Yeah, so let's see, you have your simple things like, pulled a refresh and stuff like that. But just being able to kind of directly manipulate content, you know, some of the initial stuff, like being able to pinch and zoom on photos, or now we kind of have like drag and drop that you kind of pick something up and move from one app to another with that content, like kind of that Minority Report feature that we wanted on like a small scale. So just being able to manipulate data, versus everything on the web tended to be kind of constricted to each page, and then right, moving between pages and links. They're very just like, it's moving between, like, you know, PowerPoint slides, and we've gotten more interactive stuff since then. But there's just something about using the mouse that somehow like distances you from it, I think there was just a lot of really good design work being done on the iPhone for some reason that sort of pulled me into that.

Shane: I see what you're saying so. So how did you get started with that, then? What were some of your first projects like?

Ryan: Yeah, I taken one class in college for iOS development, but didn't really make much of it. And then, when I was kind of trying to retrain an iOS, that's when I started working on black box, that's kind of like a test app. I've made one other little game for the class. But just working on that, and like sort of discovering what was possible in the platform, I think was where that began. It's sort of like it's very continuous blend into where we are today.

Shane: Yeah, yeah. Black Box is huge. For people who don't know. Tell me about it.

Ryan: It's a collection of puzzles, where you sort of solve mysterious challenges without touching the screen, for the most part, a lot of using things like motion, audio, or the phase of the moon. And it's a total of like 80 puzzles or so now. Has been attempted by about 8 million people I think has been finished by under 1000. At the moment, so.

Shane: Wow, that's really cool. Yeah, I remember. That's how we first got in touch. Actually, I would say 2017 When Blackbox really hit the market? Yeah, we were. We were playing it and experimenting with it and talking a lot about accessibility for blind people. Blackbox is still completely accessible for voiceover. 

Ryan: Yeah. So trying to keep that up. If anyone finds bugs, please let me know. Yeah, it had like a really viral peak at the beginning of 2017. That's probably where you and other people reached out to me from around that, and I think to sort of ask, like, hey, is this accessible? And it was not.

Shane: But this is where like, the story gets really interesting because most of the time as a blind person who cares about accessibility and values, you know, actually being able to interact with the world the same way everybody else interacts with the world. Most people when you reach out and say, 'Hey, this product that you've devoted  thousands of hours to is not accessible. It doesn't work for an entire sector of the market.' They're like, 'well sucks for you.' That's the end of it. We have way other things. Because we have other stuff going down the pipeline, and you're not among it. so yeah.

Ryan: Yeah, I can imagine how frustrating that is

Shane: Yeah, it’s super frustrating and so that's why we have these legislature in the United States, like the ADA,and things to try so, if we can't, if the developers won't do it willingly, that we kind of have to coerce them with legislature, sometimes just really unfortunate. But Ryan was like, 'yeah, man, let's, work on this.' And you put a bunch of time into it, which is why now four years later, I was thinking we were talking about developing mainstream products that are also accessible, that have inclusive aspects for the blind community, and you were the first person I thought of because of that experience, that was really cool. So what is it? What does it take? So once we reached out and started harassing you about making things accessible? But what did that process look like on your side?

Ryan: um, yeah, so with the web stuff, I didn't really know about the accessibility tags and stuff like that. But then with iOS, I started learning about what was making apps accessible, I think it started becoming something that we talked about more in the iOS community, and something that Apple started talking about more, and so I knew it was a thing. But I hadn't thought it was possible to make black box accessible until that point in time. So I immediately knew there were things like I hadn't gone through and manually check that accessibility values and hints and things like that were set correctly, I knew that was the first thing to do. But then there was obviously, this other problem that the app didn't even really have sound effects at the time. That's like UI sounds. So actually, being able to play the game and enjoy it like it was purely visual. So that was kind of the next challenge, I guess, solving that. What's interesting about Blackbox is since it kind of has this main constraint of, you can't touch the screen. The options are very limited for what's left, and then you get directed to meeting challenges with buttons or the accelerometer or whatever. So once you take out being all these visuals to communicate anything, the obvious answer was to use sound.

Shane: What did that look like for you? Are you doing a sound design?

Ryan: No. So I had a friend who does a lot of sound engineering, and was able to work with him to kind of come up with these, like sonic interfaces, as I have described them. What was interesting is like, we very quickly realized in the process of ideating, for that very first challenge, which you probably remember, at route, which uses a sort of tilting the device around and it changes the visual, that sort of represents that there is no good way to use VoiceOver to sort of announce like the orientation the phone, like first of all, you already know what it is when you're holding it. Second, it doesn't give you any the nuance and like third, it completely takes away from the satisfaction like figuring it out because if it announces orientation, you quickly know that's what it is looking for. Right. So I think the idea came pretty quickly from that one that like what if we modulating a sine wave type thing, or like what if you modulate it, the loudness, the amplitude of it in one axis and the frequency in another, like then you could sort of train someone quote unquote, like to, to understand what was going on the same way that the visual would train them, and I've compared this before to the sort of noisy crinkly things you like attached to baby's hands and feet, and they're learning how to control those, it kind of helps you understand, Oh, when I do this, this happens. So then from there, it was like, 'Huh, that's really cool. Could that apply to the rest of the challenges?' We thought through it a little bit, and it seemed like, it would be hard but possible, and from there, we kind of figured out this audio system that's like, super, not normal for a video game. So often, you trigger sound effects, or even have procedural music, where like, what's happening in the game affects how the music's playing. But the system under Blackbox is essentially a synthesizer. So there's kind of like a synthesizer for each different challenge, and I don't know if you can kind of imagine like a DJ setup. But like every sensor, we can have kind of adjust the music that's being created live. So

Shane: Really neat! How long did it take for you to implement that?

Ryan: We used a really great open source framework called audio kit, which handled a lot of that like, crazy low level stuff, and it didn't actually take that long, I flew to New York to meet up with Gus, who does all the sound, and we had like a sort of crazy sprint week or two, and made a lot of it, you have to remember the game was like, half the size at the time.

Shane: Oh, that’s true.

Ryan: Yeah, and discovered a lot of other sort of interesting accessibility issues I can get into later. But yeah, that was the gist of it. Now. Yeah, it's still pretty quick. Gus is pretty good at like, looking at a challenge, understanding my idea of how the sound should work to sort of teach people what's going on, and then, you know, might take a day or two to kind of wire up and get working correctly. But he does some really interesting stuff that I'd love to. I can touch on more.

Shane: Yeah, what’s the response been for that?

Ryan: You told me I think it's been quite positive. It's what's interesting to me is like, I don't get a lot of comments on the sounds itself. I still remember like, it's very memorable to me, the first time I heard you solve the first challenge. I don't know if you remember that. But that was…

Shane: Tell the story, because I don't remember it actually, super well.

Ryan: Yeah, I don't know if we had like a beta with only the first challenge. Or it was, it was the first beta and maybe we had like six or so challenges with the sonic interfaces. But we had a Skype call, and you went through the tutorial, and completed all the challenges and like, made no big deal, and I was just sitting there, because, you know, we're on Skype, I can't see or anything, I'm used to watching people solve the puzzle. But realizing that you did it. Using an interface that has not existed to anyone else was wild to me, like the thing that I think of as the game and a lot of people think of as the game was not the game you were playing, if that makes sense.

Shane: It does, actually, and that's, that's what makes it so versatile. I think that's really interesting. Like, I hear that. I hear that a lot with artists, you know, like performing artists you're talking about, they write a song, and then it becomes something completely different. But this was the first time when you and I had that conversation a couple years ago, and I was going through the few the first challenges. I feel like that was the first time I really saw that art take shape. Like the different perspectives on that art take shape in the programming world in app development is really interesting

Ryan: Cool, I’m happy to hear that. I like to think of it as an art form.  It's wild thinking about how just the that. Yeah, I mean, the interfaces itself are each level is supposed to kind of be able to look like not like art that you would frame or something. But like any freeze frame of it should look really nice. Yeah, I think, Gus did a good job with the sound to where it's kind of it strikes that balance too. It's not like it's not just--

Shane: Absolutely

Ryan: --utilitarian sounds are kind of can be beautiful.

Shane: I need to revisit this game now. I haven't played it since we did all of the testing. 

Ryan: And I totally understand I'm, I'm the same way, as a game developer. I have barely any real time to play games, which sometimes feels silly. 

Well, if anyone else wants to ever test it, or has feedback, I'm always all ears, you can always send me an email.

Shane: That's awesome. Yeah, absolutely. We'll put your contact at the end here. Also, you mentioned a couple of other accessibility issues that that came up that you fixed. What are those, these are always really helpful for programmers who are looking at how to make apps more accessible. So it's better for us to have.

Ryan: So I think like the main thing that I think is important to know, with the audio, the sonic interfaces, or whatever it was, I initially thought, like, we don't have to make or I don't have to make Blackbox accessible because it's understood that it's like impossible to make a game like this accessible, and then I sort of was reflecting on that, and was trying to think like, is there a way? And that's how we came up with this. So just not seeing the problem as impossible from the outset, I think is huge‍

Shane: I can see how that would help!‍

Ryan: Yeah. Which is, in part why it's good to have external pressure. Because obviously, those people think it's possible, of course, and from there, I think it's just about coming up with appropriate solutions. I'm sure there's some really interesting, good parallels for this. But just like I was saying, we could have made the sounds, we could have just made them spoken descriptions. Or we could have, you know, just made the sounds not that nice or whatever. But realizing that like this is someone else's entire interface to the game, potentially, for someone sighted, it might just be an additional source of information and pretty sounds. But if someone's blind in playing it, like they should be experiencing the same level of mystery and feedback to have the same level of satisfaction. Otherwise, you're just kind of like, sort of turning down the difficulty or something, you know.

Shane: Yeah, absolutely. I imagine that's really difficult, because it's definitely the easier thing to just put in spoken descriptions, which kind of signify what the puzzle is looking for, and a lot of people don't think about the aesthetics of accessibility. They don't think about the experience as much like okay, well, the blind people can see it. So. Yeah, even,and really, that's all we expect. You know, I didn't expect you to fly to New York and implement soundscapes. So I was really the next level of accessibility, which is what was really cool.‍

Ryan: Yeah

Shane: It really did take accessibility, the experience of the game, and translate it into an accessible format instead of just providing the bare bones stuff they like, 'well, you can't sue me anymore.'  (Ryan and Shane combined) Yeah. It was awesome. That you did that.

Ryan: Thanks. Yeah, I mean, I think I always wanted to have sound effects for the game, but it just was a lot of work and or money or time or whatever. So in a way, it was a great excuse to do that. But in another sense, like, the sounds never would have been this cool and complicated. If we weren't trying to make them accessible,and that's one thing that's so interesting is the sounds communicate extra information. So I'm always bummed that people have their phones on silent and they're not able to solve a challenge that they want hints. That's like if you turn off that ringer switch you'll get some information like Gus baked some really interesting things in were like the the camera shutter sound or something related to sounds from charging the phone or kind of like blended in in a really weird way. Is that maybe your brain is picking up on? Maybe not?

Shane: Oh, that's so cool. This is serving two purposes, because now I really want to play again, I'm excited. Do you have anything to say, as a response to people who say that it's, it's too costly, you know, either in both time or money to make a product like this accessible?

Ryan: I don't know, should just find a way to do it, I understand. It can be really difficult to prioritize. I've had a lot of difficulty answering this question in the past for like, indie developers and stuff like that as if I had to make Blackbox from scratch all over again, I think it would be so intimidating, I might not start. By building it up slowly. Over time, I was kind of able to approach it bit by bit, and accessibility was one part of that. But I think like anything I start now even if it didn't have as ambitious of a sort of like accessible interface, it would be accessible from day one in like that minimal way you were talking about what you expect, and I think everyone needs to be a little basically trained on what to expect and how to use their own products to at least do that. But then like, beyond that should realize there's all these unforeseen benefits from it as well, if you want to talk about it from like, the pure business standpoint, or whatever is motivating larger companies. You know, there's been a lot of good that's come from it for everyone, like I mentioned, you know, what, improve the quality of the game overall versus this, like, other way of sort of understanding the puzzles and whatnot, so…

Shane: Yeah absolutely, and there are a lot of games that are implementing more accessibility features. I mean, 'The Last of Us'

Ryan: Totally, oh my God, yes.

Shane: Is groundbreaking

Ryan: Yeah

Shane: In the way they've implemented their accessibility features. There are a lot of games, you know, 'Gears 5' is accessible. For the most part.

Ryan: 'The Last of Us' stuff was just crazy, and that's like, a was a great example to me of like, being able to improve things that even I thought were like, Oh, you just can't, that's just one thing that you can't make accessible. Like they kind of proved 24 times that you can, which I think just goes to prove again, there's nothing that you can't make better, and sometimes it requires kind of outside the box Thinking to get there, but there's always some sort of solution.

Shane: And they have, it has this massive menu of accessibility settings. It's nothing. It's amazing. I'm just waiting for factions, their online multiplayer mode. 

Ryan: Cool

Shane: If they can make that accessible, then there's no more barriers. There's no excuses for any game developer, if they can make online PvP in that kind of arena accessible. Anyone can do it for any game

Ryan: I'd say one other addition to your question, like, people should seek out more of this and try to see how these things are working or how people are playing. I've, I've been looking to like surround myself with a lot of that online, and from some of the accessibility attention Blackbox has got, but a lot of people just don't know what is happening in the accessibility space. They don't know what these things are like for people. So.‍

Shane: Absolutely. That's very true. What are you working on now? You're still working on blackbox,and also anything else you're doing that you want to tell people about?‍

Ryan: Not that I can talk about right now. Yeah, I'm still just really, it's not a huge secret. I'm just still basically doing Blackbox. But there's always have ideas for other stuff. So certainly seeing the puzzles. ‍

Shane: That’s what you do for work, is Blackbox?

Ryan: Yeah

Shane: That's awesome. I love that it's still so successful.

Ryan: Still a full time thing, and still new people finding it. Yeah.

Shane: that is really cool. I'm really happy to hear it.

Ryan: Super lucky

Shane: to see that the game is still growing and expanding and evolving. That's about it. Man. I really, thank you so much for sitting down with me and chatting. Having a conversation. It's been really cool to catch up.

Ryan: Yeah, very good, and, yeah, let's see if it happens in another few years.

Shane: Yeah. In 2025 We'll schedule our next four-year check-in.

Ryan: Like that's awesome. Be surprised.

Shane: How can people get in touch if they want to follow your projects to see what you're doing or interact?

Ryan: I'm "Warpling" on all social media stuffs, W-A-R-P-L-I-N-G,and if anyone has longer things to write or questions, I'm Ryan@growpixel.com. So yeah, happy to answer any accessibility questions or not, and always looking for new accessibility testers for beta tests. I also don't have enough of those right now. 

Shane: That's awesome. Thank you so much, my friend. It was great to catch up to you. Thanks for being on Supersense presents. 

Ryan: It's gonna be great. Thanks for having me. It's gonna be fun. Thank you so much.

‍