Transcript
Thanks to Richard Park, Rodrigo Girão Serrão and Rory Kemp for their work on this transcript.
Conor Hoekstra [CH] 0:00
For folks that are interested and that want to learn, what's the best way to go about it? Do we recommend a single language? Is it, is it, is it q? Is it, is it J? Is it APL?
Richard Park [RP] 0:09
That's a different episode, Conor.
[CH] 0:10
Oh, is it?
[RP] 0:12
Array languages face-off: there can be only one.
[Music Theme] 0:16
[CH] 0:26
Welcome to the second episode of array cast. My name is Conor Hoekstra, I'll be your host for this episode. And as I said before all following episodes. First, we just want to say thank you so much to all of the folks that have provided feedback, we've had an overwhelming amount of positive feedback. And folks that have just been saying that they're super excited about the podcast and are looking forward to more episodes. So it's super encouraging on our end when we get that feedback. And I think, Bob, when we go around and do our brief introductions, might say a little bit more about that. And then we'll hop into our episode, which is going to be talking about iteration versus loops and how you can get started in learning the array programming languages. And we'll also have a short announcement, before we do that. We're going to briefly introduce ourselves, we'll start with Stephen, then go to Bob, and then go to Rich.
Stephen Taylor [ST] 1:11
I am Stephen Taylor. I've been an APL developer for decades. And for the last five years, I've been the Kx librarian in charge of documenting the q language...
Bob Therriault [BT] 1:23
And I'm Bob Therriault. And I'm a J enthusiast. I've been playing around with J for about last 19 years. I do not program professionally, but I'm certainly interested in the language. And by the way, just to add one more thing, we have 1593 subscribers, which is substantially more than our first episode's estimates which I think Conor said he'd be happy if we had anybody. Well, we have 1593 more than that.
[RP] 1:49
Woohoo! And I'm Richard Park, been working for the APL vendor Dyalog for about two and a half years maybe. I'm mainly focused on training, teaching people APL, and also a little bit of evangelism spreading the word of APL outside of those circles.
[CH] 2:08
And as mentioned earlier, my name is Conor Hoekstra. I am your resident non-APLer. I'm a professional C++ developer. But I'm a huge APL enthusiast, and also interested in learning about the other languages J, k, q. So I'm here to ask the questions and hopefully learn just as much as the listeners. So before we hop into the episode, I'll kick it over to Bob, who's got a short announcement.
[BT] 2:29
Yeah, kind of well, sad news this week, it turned out, and I heard this through a post of Roger Hui's, Lawrence Breed, Lawrence Moser Breed passed away, May 16th. Now, for those of you who know about Larry Breed, I didn't know Larry Breed. So what I'm putting together is from Roger's email, and also his Wikipedia site. But having said I didn't know of Larry Breed, I wish I knew Larry Breed. And the reason is, Larry Breed was one of the original implementers of APL when it went from being a notation to actually implemented on a computer language. It was Larry Breed, and Philip S. Abrams, who put it together for the first time. This is where it starts to get interesting. Before he did this, as an undergraduate at Stanford University, he created the first computer animation language and system, and used it at Stanford football halftimes to coordinate images produced by 100 foot by 100 foot arrays of fans holding up colored cards. So before he was even writing APL, he was already working arrays and stands. So it just it sort of it goes on from there. One of the things I thought was really cool about him was his management style. Jim Brown, who I think worked for him, was talking about Larry, he says in 1969 he got a Summer position from the APL group at IBM Yorktown research. Due to an administrative error, the Summer job was never terminated, and lasted three years. And then Jim Brown also says that Larry is pretty much responsible for my APL career. He was my first manager at Yorktown research. Although I didn't know he was a manager until the exit interview. He messed up the termination and I just kept working from Syracuse. So he's kind of got my management style, which is barely. And then after he retired, done all this stuff and after – Oh, and by the way, he won the Grace Hopper award. You know, because that's just something you do when you invent this kind of stuff. When he retired, he get involved with Burning Man under the playa name of Ember. He was one of the copy editors for the blog, Black Rock Gazette, which became the Black Rock Beacon. He, he was in fact, one of the sculptures that he, he created was called “Chaotick” and it was one of the, along with the actual Burning Man, it was one of the longest lasting sculptures at Burning Man festival. So, and I guess to wrap it up using what Roger, Roger also said, and this is Roger saying this “Larry's programming skills are the stuff of legend.”. And if Roger says that his programming skills are the stuff of legend, they are the stuff of legend. And Roger wraps up saying, “Yes, APL (in parentheses, I am including J) has quite a character set, as in what a bunch of characters.”. So cheers to Larry Breed a life well lived.
[CH] 5:38
Ahh cheers. Thanks for that Bob. And I also did not personally know Larry, but I'm somewhat of a programming language historian. And I've spent a lot of time studying the history of all of the array programming languages. And it's, it's really clear that I think the APL and array programming language community at large, has a lot to be thankful for him. I don't think I don't think APL and these languages would be where they are today if it weren't for Larry. So. But transitioning to our topic for today. So I think we'll we'll let Stephen start because this was I think, initially, your idea, Stephen, talking about the difference between sort of languages where you iterate versus languages, like the array programming paradigm languages, where you don't typically iterate, you don't have looping mechanisms, or you have alternatives for them. So I'll just let you take it away. And we'll go from there.
[ST] 6:34
Thanks, Conor. We were talking in the last episode about what it is that distinguishes the array languages that we're talking about here. And we geniuses managed to come up with it was arrays. I can't remember how long it took us to get there. But we got it done. It was arrays. And in the time since then, I've been thinking that's not an entirely satisfactory answer. We can maybe do better than that. It's not even that distinctive an answer, because when APL was first designed, arguably, it was the only language which took this ending like its approach to arrays or treated them as first class objects. But now you go to pretty much any, any language, most of them have got some way of dealing with arrays. So somebody coming in to the Iverson array languages could reasonably say, what's the big deal about arrays. And thinking about that led me to “Well, it's not just the arrays, it's the way in which they interact with the primitives and the way that they iterate”. So if you come into this conversation from the outside, you've heard fairly early on, we don't write loops. And it's easy to miss how much we don't write loops. And I guess that's the burden of what I'm proposing here is the iteration that you do in an array language is buried so deeply into the language and new coders, or people coming into the languages for the first time, I think miss that. I think it takes a while to get into it. I often see people who are beginning and as in q or in APL, take some expression which works on a simple single object, and say “now I'm going to apply this to a whole list of them”, and they take the expression and they, in APL you use an 'each' operator, in q, you'd use an 'each' iterator, you basically each it over that, you know, we call that a map in other languages now, so it doesn't look like a big deal. But it looks like a first “Hey ma, I'm programming without loops. Look, no hands, no for loops, no counters, I'm using an each expression”. And if you're lucky if you're the newbie, and if you're lucky, somebody with more experience is going to come along and just run a pencil through the 'each'. In APL, 'each' is one character, it might be one character in q and you get to see the primitives in your expression already do the iteration. You didn't need to put the iterator in there. And some of those primitives which we call in some of the languages "atomic primitives", will just keep doing that iteration all the way down until they find atomic, you know, single integers or single characters. The first time I began to work up to this and that it applies not, this is relevant not only to applying functions to data, but also the indexing, was when somebody took a Boolean matrix, and just used it to index a two character vector. And suddenly I had a, I had a chart, no no function, nothing at all it just index. So using a, an array to index into a vector got me an array. Way over over my head here. So that's what I wanted to propose as our second distinguishing feature of the Iverson array languages: that we have iteration, which is mostly handled either atomic or with near-atomic iteration in the, in the primitives, and then very, very concisely with what we call adverbs, or operators, or iterators, usually a single character, and only if you're absolutely desperate, would you use a loop expression.
[CH] 11:09
Yeah, I, I had a huge grin on my face when you said, “We don't like loops”, and probably to a certain percentage of our listeners that are C++ devs, and might be listening because, because of a tweet that I sent out, they'll know the anecdote that I'm about to tell. But there's a very, very popular expression that was popularized by Sean Parent, who is a principal scientist at Adobe, very senior, sort of a luminary of the C++ community. And he proposes that, you know, the one thing that you should do to improve the quality of your C++ code is to avoid raw loops. And so the the anecdotal phrase that he has is “no raw loops”. And it's not to say that you shouldn't use loops in your code. But if, if you've got, you know, a 50 line piece of code, and in that 50 lines, there's a 10 line loop, odds are that that loop is an algorithm. And you can replace that with a better named function, maybe something from a library, a standard library, or a third party library. And this has become like a huge part of my philosophy. And as I was learning about APL and J and q, I stumbled across a website called www.nsl.com. And I was like, NSL, what does that stand for? And sure enough, I find out that it stands for no stinking loops. And I almost fell off my chair, because I was like, “Wait a second”, like, I'm not I'm not an array programming programmer. I'm coming from C++ land. And like, Sean Parent is like one of my heroes, I look up to him. He's got a lot of great stuff to teach. And then I come over to this community, and you have a whole website dedicated to no stinking loops. And it's a bunch of examples of how you know, you can solve these problems without any loops. Anyway, so there's like a dotted line between the C++ community and this sort of “no raw loops” movement, and the array programming languages, which I just, I just think is like, I thought it was the best thing in the world.
[RP] 13:01
Yeah, I've been thinking about these sorts of ideas, recently, the sort of distinguishing features of APL. And I started to think about, I guess the types of things you learn when you start learning to program and you start by building up these loopy solutions, and then after a while you're doing more complex problems. So your loopy solutions get really long and unwieldy, you don't want to copy and paste that code everywhere. So then you learn abstraction, which is sort of hiding away all those nasty details under a name, and then you can refer to the name. But then after a while, maybe you get this thing, though, sort of in the Iversonian languages all along, where you find you're spotting patterns of you doing very similar things in lots of different contexts. And maybe the way that APLs sort of abstract those patterns away into those select primitives, is a different kind of abstraction to like what's done in those other traditional languages where you have a bespoke solution to one very specific thing. Whereas here, we're saying, oh, you're doing the same sorts of things all over the place in lots of different places. I mean, maybe that's what some of the C++ standard library algorithms are hinting to, and this idea of no raw loops that actually, you don't need to get too complicated with it.
[BT] 14:20
And actually, the key thing I think, is, the language takes care of a lot of that complication that you would need to put in if you are writing a loop. There's something there that's going to do the same things. But it's already been made performant, it's already been debugged, it's already ready to go. You just have to know the right things to hit to make it work for you. And that's a much easier way into what's going on than having to start from scratch and writing all your iterators and everything and it makes the language a lot cleaner. And one of the other interesting things I found, one of the areas I learned about J was “J for C Programmers” which Henry Rich wrote – excellent book. And he, I think he's got six or seven different types of loops. So there's loops, like if you were adding to arrays, and that the size of the shapes matched, there's a loop to match each single, you know, item in those arrays and come up with an array of the same shape, which is the sum of those individual integers. There's the loop, which is like a partition loop where it breaks everything down into different lengths, depending on, on what you've got is your your original noun, the original argument you're working with. So if you had something that was 100 characters long, and you were cutting based on spaces, you might break it into seven or eight irregularly shaped, well, that's a different type of a loop. And there's a loop that will do that within the array languages as well. So, your loops are sort of almost like mental concepts. And in some strange ways, it's forced me to think more deeply about loops, because I don't use them.
[CH] 16:04
Yeah, that's that's a great point. I think, you know, and this, I'll pose this as a question to get all of your answers on, because I'm curious. But I think one of the primary reasons to prefer algorithms or, you know, if you're in an array programming language, a primitive, to a for loop, is that it's more readable, you're communicating more to the reader of the code. You can have two very similar pieces of code in C++, one that's doing a 'count_if' reduction, where it's just counting the number of elements that match some predicate. Or, you could be doing some sort of 'any_of' where you're just trying to see “Is there a single element that matches this?”; one can be sort of short-circuited, one can't. And if you look at the code for each of them, they're very, very similar. It's just, you know, a couple of, one's got a break in the if statement, the other one doesn't. So it's very easy to confuse them and hard to discern the difference. But when you have two differently named algorithms, it becomes abundantly clear, you know, what's the difference. So that's one of the things that I think makes algorithms, you know, more preferable to for loops, but I would pose the question to the three of you. Why, how would you convince someone that's coming from either Python or Java or C++ or JavaScript, and they're saying, you know, every single one of these languages has for loops. And I can go from Java to C++, and, you know, essentially, the for loops are roughly the same. And I don't I don't need to, there's no cognitive, you know, barrier to entry of me figuring out what this code does, you know, what do what do we try? How do we convince folks that it is worth going and learning the algorithms or the primitives of these languages? So that we can lean on them more than for loops?
[BT] 17:46
Well, I would say that part of it is that there's there's there's a two-way communication, when you you have somebody who's written the construct of a loop, if we're talking about the way these languages do it for you. One way is, you know what that will they know what you're going to use. So that can make them much more efficient, because on their side of things, they know how you're going to interact with this. And they can get really good efficiencies. As a person actually writing in the language, they also have a very set group of expectations as to how you're going to use it. So that makes it a lot simpler. You don't have to say, “Oh, I'm partway through this, am I going to branch out this way or that?”, you either can or can't do it. And if you do, if there is something you can do, there's a specific way that you can do it. And that means that there's a lot of consistency. And I think that makes them easier to read that way. Because when you go in, you know how the loop works. And by the way, it's structured, you know what it's going to do. You're not having a sort of an ad hoc approach to creating loops.
[CH] 18:49
Stephen, Rich, do you got any...
[RP] 18:52
People think that maybe you'd lose sight of what we're abstracting away what the computer is doing, but actually, you have a very concrete view of exactly what the computer is doing. And being able to reason about the code, just by looking at a line of a few primitives can be a lot simpler than trying to untangle some really complex bespoke piece of code... Guess that doesn't make it more straightforward in some senses. People do worry about “can you, for a particular problem, for a particular solution, can I then go and implement some of the more obvious performance improvements?”. For certain types of problems that does actually become a bit of a challenge in APL, you do want to unpack for a very particular style of problem. But for large swathes of problem, you the code that you write is going to be performant. If you write it the sort of idiomatic APL way, which is something that's quite hard to explain, describe, I guess, especially over voice, but uh...
[CH] 19:55
I think, I think it makes sense, at least to me what you're saying is that what I'm picturing is a recent problem, that now Bob has actually just released a great YouTube video solving it in J, which, if we have show notes, I think we do, we should, we should link to that. But, on my other podcast, my co-host, and I have spent a lot of time talking about is finding the maximum number of ones in a sequence of ones and zeros. And to solve that using for loops, and if statements, in a language like C++, the resulting code that you get, in my opinion, is more difficult to reason about, then the APL or the J solution, like in APL, you can just use a partitioned eclose with the commute operator. And that's going to immediately give you your nested list of ones in boxes in a nested array. So if you have 1, if your starting sequence is, 1100111, doing a partitioned enclose using that array as the mask itself is going to give you a nested array, where the first one has two ones in it, and the second nested array has three ones in it. And then from there, all you have to do is a plus reduction. So, sum up those values, or you could even take the length, they both work, and then just take the maximum. So you've got two different reductions, a max reduction, a sum reduction. And you've also got a partition happening in there, and the equivalent code using just for loops and if statements, I would have to stare at longer to understand what it was actually doing. If I didn't if that if that if it wasn't in a function with a good name that was explaining what the code was doing, I would be able to figure out much quicker reading the APL code, or the J code, what the code was doing, because it's, it's relying on your primitives that you know exactly what they do. Which is, I think what you're trying to communicate is, it's we're not actually going down the, you know, abstraction ladder, using these primitives in these languages is is going up the ladder, which makes it easier to understand; if I understood, what you were trying to say, correctly.
[RP] 22:02
Yeah, that that I've heard that described as this idea of subordination of detail as opposed to abstracting it away into obscurity, right? The detail is still visible from the, from the same piece of code, but also the intention at that high level of “What is it you're actually trying to do?”. They're sort of visible at the same time, which is something that's, again, not super, you don't see it a lot in these other programming languages. And it's also quite hard to express without just I don't know, showing people lots of examples or having someone actually learn APL for a while. So in that case, of the example you just gave, I guess it also for people with performance concerns when you're traversing in that case, you've got to traverse the entire argument anyway, right? Because you're, because there might be a bigger collection of 1s further down or something. So in that case, the partition is just fine. What's an example? Well, we know you can short circuit a 'and' reduction or something. But I guess you might not think of that if you're handwriting the, handwriting the solution in C by yourself. Because there's something that Bob alluded to earlier, the idea of the optimizations for certain, certain algorithms, certain primitives has already been taken care of for you or continues to be improved on over time. So you're not bogged down by that either. I know I'm not explaining this incredibly lucidly, but it really does, it feels like having your cake and eating it, and eating it too, when it comes to working on computational problems.
[ST] 23:42
I guess, I guess I got two answers to your question, Conor. Depends on how smart I think the person I'm talking to is. You mentioned nsl.com, no stinking loops, and that sites owned by Stevan Apter, who's one of the smartest programmers I know and an absolutely gun, a vector programmer. If I were trying to persuade someone like Stevan Apter, to take up an array programming language – which of course, is something he did years ago, being as smart as he is – I would, I would be arguing for how it enables him to think and reason about the code more easily, very much along the lines, you were saying. If I was talking to someone like myself, for whom my mental powers are a scarce resource, I simply say “I really don't have the mental powers to spare to futz around with counters and loops.”. I just, I just need to stick with a problem.
[CH] 24:48
That is uh, I've actually heard that exact reasoning on another pod, one of my favorite podcasts, “Lambda Cast”, which is a functional programming podcast and I'm going to misremember who this quote is originally attributed to, but it's the seven plus or minus two number of things that you can hold in your head at one time. And on that podcast, it's one of the first couple episodes, if folks want to check it out, they're talking about how exactly that. The number of things that people are capable of reasoning about at one time is limited. And when you're writing a for loop, and you've got three different things, you've got the initialization of your index, then you've got the the criterion for exiting, and then you've got the incrementor, or decrementor, that's already like three, three of your seven things, and you haven't even gotten to the body of what you're trying to do. And it's interesting that you said that, and that that's actually the second time, the first time I heard it was on a functional programming podcast, which there's probably something to be said there, because this idea of not looping is also... Exists very largely in functional programming languages, like Haskell, you're not using loops to do reductions, you're using your generic reduction algorithms that are folds and whatnot. Rich, I think you were gonna say something.
[RP] 26:05
Is funny you should mention the seven plus or minus two. So I think I read a interview of Arthur Whitney, where that number crops up and it's something like a rule of thumb for like a line of APL, how much stuff should you have on there? And the answer might be seven plus or minus two ideas, I think, you know, it's hard to say that they're primitives or whole functions or something else. But yeah, the idea of that being a good sort of unit to stick with. Instead of like, your piece at a time is always an atom, your piece at a time is one thing that might be built up of seven plus or minus two other things, something like this. I thought that was a quite funny coincidence, or maybe not coincidence, but just a motif in the programming intellectuallists world, or meta programmers? I don't know.
[ST] 26:58
I wanted to just quote Whitney too. He spoke at the 2004 meeting of the British APL Association at the Royal Society in London. And he said, it is theoretically impossible for a k program to outperform hand-coded C. Yet, k programs routinely outperform hand-coded C is, what makes it possible is that it's way easier to find your errors in four lines of k than in 400 lines of C.
[CH] 27:34
I'm not surprised Arthur would say something like that. And I'm also incredibly jealous that you got to see Arthur speak. Because I know that is a rare, it's a rare occurrence. Yeah, Arthur, if you're listening, you should seriously consider coming on this podcast as what would be hands down probably the best episode that we would produce.
[RP] 27:56
Certainly the most hotly anticipated, yeah.
[CH] 27:59
Oh, yeah. For those that, I think it was mentioned in the first episode, but Arthur is the mastermind and the creator of the the k language, which then also spawned, I think k four went on to be wordified and became q, which is the language that Kx the company that Stephen works for, sells.
[ST] 28:20
And is now becoming k9 in his new company, shakti.
[CH] 28:24
Yeah, yeah. So there's, this is probably, this is probably a whole other episode, so I'll just briefly say an anecdote, and then we'll put a pin in it. But Arthur has created different versions of k. And I think when I actually when I spoke to him, and called them versions, he said, “No, no no, they are not versions, they're different languages.”. He thinks of the more as like different dialects of Lisp, like you've got, you've got, you know, Lisp 1 and 1.5, and 2, but then there was Racket and Closure. And he views, even though they're called k 0, 1, 2, 3, I think they skipped 8, because they didn't want to get to a lawsuit with Kubernetes. And went straight to k9, but he views them as different languages. And every time he starts on a new one, he completely throws out everything. He just like, wipes everything, and he does the parser like the whole thing from scratch each time, which if you were talking about smart individuals, Arthur's got to be way, way up at the top of the list there, along with Stevan Apter. But yeah, well, we'll we'll save a whole episode for the different k’s. And maybe, yeah, we'll we'll try and twist Arthur's arm to come on. And so he can tell us, you know, firsthand.
[RP] 29:30
Can I bring something up though, I'd like to get people's thoughts on. Because recently discussing this with a colleague about some of the some of the ideas that have been in APL for a long time, start to gain a little bit of, you know, become a bit more well known with the increased popularity of functional programming languages, the idea of folds and maps and things like this, rather than all these explicit loops. And someone had asked something like “What's, so what's sort of the difference between APL programming and other functional languages?” And maybe and my colleague suggested maybe it's something like what, we neither of us have that much experience in functional languages. But in functional languages with functions as the first class citizens, are you kind of building up your solutions as modifying the functions themselves until you've reached this thing, which is more or less a pipeline that your data then goes through. Whereas in the APLs, you're always thinking about the transformations on the arrays, and the data manipulation piece by piece.
[CH] 30:35
I can answer and then because, because I, I've done, but I'm actually more curious to hear what Stephen and Bob have to say. I frequently now make these videos where I call, I call them on my YouTube channel I heart APL in Haskell, and I just, I solve them in those two languages, because they're my two favorite. And I take a look at the other solutions that have been top voted on, on leetcode. And they're always done in Java and C++. And honestly, I think that like the mental model for functional languages, and array programming languages is very, very similar. Primarily though, for me, in functional languages, it's function composition, you're just composing functions. So there's, there's a problem where it was calculate the sign of an array. So you're given negative positive and zeros in an array. And you just take the product of all those numbers and then take the sign. So if you end up with a positive number, it's 1; negative number, it's -1; and if you end up with zero, it's 0. And the way you solve that in APL in Haskell, are almost identical. I treated it as if you might overflow if you do the reduction first. So convert everything to a 1, -1, or 0, and then do your multiplies reduction, aka product. So in APL that's literally multiplies slash for your product, and then monadic multiplies, which is going to do the the sign over your elements. In, in Haskell, it's 'signum', or it's 'product' dot 'map' 'signum', so you're mapping 'signum', which is the 1, -1, 0 function and then just taking the product. So those are super, super similar. The difference I would say, is that functional languages are, they more focused on giving you the algorithms, and then you having to compose them together to get your solution. APL has this extra layer on top of that, where you have this, like, library of patterns, that like, are purely at least I have only seen them in these array programming languages, like the outer product pattern, pattern, where you do an outer product, and then you immediately do a reduction row-wise, to get some count of like the number of characters in a string or something. There's no equivalent of that in like functional programming languages, like you have to build up a specific thing called a counter, or you'd have to build that up. Whereas in APL, it's, it's, you know, four or five characters, type, type them into your keyboard, and you get your result. And that's why I find APL such an exciting language, because I only know a fraction of these patterns. And the more and more you study it, like the more patterns and idioms you start to see. And every, every once in a while you stumble across one and you're just like, "oh, my goodness, this is beautiful”. Like, I would have never thought of this in a language like Python or C++. But I will defer, and I'm interested to hear Stephen and Bob, your answers, Bob?
[BT] 33:19
Well, I think part of it, in this, sort of goes a little bit back to Richard's, what Richard was talking about as well, there's a level at which you're composing when you're using functional languages. And then, with APL or array languages, you're really looking at transformations. And I think the difference in my head is, rather than composing, which is putting things together, quite often, when I get into it, I'm taking things apart to focus on how I can make that transformation cleaner. So I'm actually removing things whenever I can. And I don't get the sense if your, your language is based on composition, it's a really different mindset, to start subtracting things. Whereas, I think, my sense of the array languages, because you're looking at a transformation, and you're always focusing on what that data is doing, you're always trying to make that as simple as you can. So your, if there's a way I get, you know, I, the number of times I've turned roundabout loops, and caught around going, “Hey, I could have done this” and just remove that whole string of transformations because suddenly it appeared to me, all I'm trying to do is move from A to B, I don't have to go through D, E, F, and all these other ones, I just go AB, that, I've done, okay, so that, but that's done. Now, I've taken this whole process, and it's just down to one step. And I think it's focusing on the transformations and the data that sort of opens my mind up to that.
[CH] 34:45
While you were saying that, this might be completely off, but I was thinking of the scene from Iron Man, the first Iron Man where he's designing his Iron Man suit with like a holographic – I don't know – system, and he like uses his hands to like, blow all the components up and then like, throw a couple away and then shrink it back down. Like that's what I was, and that's how I feel when I'm programming in array programming languages, you're just staring at all the components, like the, the primitive algorithms, it's a scan, it's a reduction, it's a partition. And you have the ability to just look at it, explode it, remove a couple, shrink it back down. And, like there's, there's nothing even remotely close to that. That kind of feeling when you're coding in C++. I love C++, but there's no Iron Man holographic feeling when you're, when you're coding that way.
[RP] 35:32
Didn't someone say on a comment on one of your videos, Conor, that APL looks like what programming is portrayed as in movies?
[CH] 35:43
Well, I'm not sure if that's a good thing or a bad thing, because movies a lot of the times, if you pause it on the screen, you're going “what is going on there?”. So I know we're we had sort of two topics that we wanted to touch on. And I think we've sort of alluded a little bit to it. But maybe we should try and more explicitly tackle the second one, which is how do we how do we get folks interested, or not interested, but for folks that are interested and that want to learn? What's the best way to go about it? Do we recommend a single language? Is it, is it, is it q? Is it, is it, is it J? Is it APL?
[RP] 36:17
That's a different episode.
[CH] 36:19
Oh, is it?
[RP] 36:22
A array languages face-off: there can be only one.
[CH] 36:27
We won't, we won't recommend which one but for those that are interested in exploring the paradigm. What are, you know, I know, Stephen, you had some thoughts because you've been spending some time thinking about, you know, in this day and age, you know, it's no longer the 60s in the 70s when APL was sort of, had been developed and there was a ton of excitement around, and excitement around it. It's a different, it's a different time now, and there's 100 languages to choose from, so...
[ST] 00:36:56
Well – I think it's the first thing that comes to my mind – is that I was completely deceived in the 70s when I encountered APL. I thought it was a wonderful language, I still do. I and my colleagues thought that everybody we're working with in APL was in there because it was a brilliant language. Now, in the 1970s, APL timesharing systems provided personal computing before there were personal computers. That was a big deal. You could sit in front of a keyboard and do your budgets and there were no graphical user interfaces, there was no Excel and spreadsheets, but you could sit in front of a keyboard and do your budgets or whatever it is you needed to do.
When personal computing, when personal computers came along, the people who were doing their budgets and stuff all moved onto them. It was way cheaper – and when you got screens and graphical user interfaces; a lot easier to use – and in a sense the tide went out and we lost all the people who were there just in order to get what APL made possible, and there was a minority of us left saying what a great language this is. And, you know, we were still developing it, still doing systems, it still is - still is a great language and I think I see a similar pattern with q. Most of the people I know doing q development these days are doing it because of what it provides access to. So, we should definitely come back to do the array languages face-off at some time. But probably if you're thinking about learning a language, you probably already know which one you're interested in because of what it provides access to.
How do you start? Well, it's fairly easy to get started, and there's all kinds of intros, either workshops or material you can look at online. And then I've come to the conclusion something quite. There's really quite an obstacle for solo students. And the reason it's there is that for most people who need to get something done - in whichever language - you do an intro, and then you get into the pool with other people who are already using the language.
I was referring earlier to that loopless programming: “hey look Ma no hands, I'm using an each instead of a for loop," and in the process of learning a language more thoroughly where someone comes along and shows you - no, actually the primitives will do that iteration. You can do that, this, that, or the other much more economically, with whatever expression.
When I consult my experience, I realized I learned almost all I know about APL from people around me. Most of the people I know who know the languages learned from people around them. I know very few people who learn the languages to any significant extent, excuse me, just from books or studying online material.
Programming languages flourish the same way that human languages do amongst people who use them. I'll admit there are exceptions, but here's my premise: most people learn these languages within a linguistic community. Now look at the experience of the last 25-30 years of people using k and q. Go through an intro and then you move on to work in an office with people around you using the language. In the Kx business that would be either the Kx company or our clients. If you're what I'm coming to call "outside the gates - outside the city gates" you are struggling to learn the language 'cause there's no one around you to show you "oh this is an easier way to do this or that" - there's limited online resources.
In the this new pandemic distributed world I think we're going to have to learn how to teach the array paradigm: vector programming; how to get a thorough grasp of the syntax of the language; not just what you sort of pick up from what you see. We're going to have to learn how to teach that and we're gonna have to teach it online, and it's a new thing. I don't think we've ever actually had to do this before because we've been learning in real life communities. My new thought for the week.
[BT] 00:41:56
It's interesting that you left it, like, right at that point - online communities - because I think that's exactly where the learning goes on now in the array languages. When I think about J, if I was giving somebody advice, there are some really good fundamental documents. I mentioned Henry Rich's J for C Programmers. It's so comprehensive and it's written from the basis of a procedural language which a lot of people already have access to understanding. So that's a really good document.
There are others you if you go on the J software site you'll find others. But the key thing, I think, is to get involved with the online forums, because if you're watching the online forums - like this week there was two really big sort of discussions going on back and forth, and these really, really top end programmers. The first one was trying to figure out how to track stock variations, and it was based on a real problem and then broken down to a simple subset and then they took this and then basically they riffed back and forth through the emails or the messages going back and forth. "We'll try it this way. Try it this way, no, no. If you do this is this. This is the failure here." And you could actually watch - it was Raul Miller was one of them. I can't remember who the other person was, my apologies to the other person 'cause it was really neat watching them back and forth building this kind of code that you - all you need to do is just read through the thread. You can see them as they're coming up with problems as they're solving the problems and it gives you insight into how they think.
And then the second big problem was somebody came on and said they were trying to translate Aaron Hsu’s tree building algorithm into J and they were having problems with it. They couldn't get the depth thing working and somebody - this went on for a couple of days, actually, people suggesting things and then somebody broke it and when they broke it, suddenly everybody could do it and they were looking to optimize it and it was just fascinating to watch them go through that process. And so I'm not saying if you got into the middle of that, you'd say, "oh, I know what's going on now," but if you have done some background reading and I recently heard a metaphor saying if you're if you're trying to do something without doing background reading without getting the concepts, it's like learning to play air guitar, you know all the moves, you just don't know really what you're doing, and there's no music coming out and I thought that was really - you know you're going to have to spend some time getting the concepts you have to spend some time doing some reading, but then as soon as you start to understand this stuff, hop on those kind of forums and the J forums are good for that. I think is it, is it APL Orchard, Richard?
[RP] 00:44:50
Yeah, when it got to me I was just going to do some flat out plugs of various resources 'cause this is what Stephen described is a problem we're very aware of where if you get to a certain point, you've been learning APL and then you're like, "oh, how do I just do? I want to do this thing. How do I do it?" And you ask an experienced APLer and usually you'll get a whole bunch of really good answers, but the question is like how do we get that knowledge out of the experienced APLer's heads and out into into the internet or whatever forum so people can get access to it without having to literally ask people.
On that note. So what's coming out? So Rodrigo Girão Serrão, who is working for Dyalog revamping the Mastering Dyalog book, which sort of an excellent, quite comprehensive resource.
Uh, a guy who - so you mentioned the APL Orchard - you can access that through the URL APL dot chat, and that's just a very lively stack exchange message/forum/thread. It's very active so you'll - if you ask questions there, you'll get answers definitely same day most of the time. A user of the orchard called Stefan Kruger has written a really excellent introduction to to APL, and I guess, specifically, Dyalog APL. The reason it's not published quite yet is 'cause it references one of the problems from this year's APL problem solving competition, so he's waiting for that to finish before he publishes it.
Uh, and then I myself. I'm working on – the URL is course.dyalog.com – and that's kind of supposed to get you hands on, uh, working with APL basically straight from the beginning and try to, as quickly as possible, get you over some of those initial “gotchas” and things that people find a little bit confusing when they start learning, but then if you take any of those resources to get you started and then, as you've mentioned Bob, once you have questions, talking to humans is still by far one of the best ways to get answers. So APL dot chat is a really excellent way to get those answers.
[ST] 00:47:00
This is very, very interesting 'cause I'm hearing the same thing from you both: background reading the study resources, mastering the material is essential, but the learning also is a collaborative experience. We need to - we need to generate and foster those collaborative experiences. There's a famous motivational speaker like I doubt he's still around, but see he was known in his time as Charlie Tremendous Jones. You can find a celebrated quote from him on the internet, it says: “five years from now you will be the same person – the same person you are now – apart from the people you meet and the books you read. Books you read; the people you meet.”
[CH] 00:47:45
Yep, that's definitely – there's some other version of that where you are the average of the five people you spend the most time with that kind of thing where – so you should choose those five people carefully and that five people evolves over time, right? So it basically speaks to exactly what that quote is saying, but the one link I was going to add on top of this, and I'm not sure if there's actually a J or a q equivalent, and if not, I would - I would highly recommend those communities thinking about setting something up like it - is tryapl.org, that is actually how I fell in to the APL rabbit hole, there's a much longer story, but at that point in time where I wanted to go and take it for a spin and just be like, oh, you know, let's go and see what APL is all about. I I knew that most languages, there's a lot of popular websites like REPL.it and like J Doodle and coding grounds that just they have like 50 plus different languages that you can just like write code and run in the browser.
And so that's typically the first place that I try and go. 'cause I don't, you know, I have nightmares from back in university of trying to get like C++ up and running on my computer and these days you don't typically have to do that. You can just go to some browser and they've got a compiler running in the background on some other box and you just write Hello World and it works, and so TryAPL.org was fantastic for that. I went there; it has all of these symbols as like a clickable keyboard at the top to get started and so very quickly it shows the keyboard shortcut to start typing them in yourself. But as you have no idea “how do I type the circle with a line through it?” like I don't know that as a beginner and that's J and q will have an advantage over over that, like, in that regard, 'cause it's just ASCII, so like you, even if you don't know what it means you can, you can still figure out what to type, and q actually has words, so that's potentially even the easiest to get started with, but you can very quickly, you know, type your first line like just. Type plus slash, iota five and, you know, start with iota five and then add the plus slash and for me that already is just like “Oh my goodness” like that would have been – that would have been like 40 characters in C++ like I'm already. I'm already typing a fraction of what I would in the language that I know and and so yeah, I'm not sure. Is there like a Try-J or a Try-q website that we can also either put in the show notes or mention?
[BT] 00:50:07
'bout the only one I can think of and and there's not one explicitly that is for J, But it's TIO.run, which is Try It Online, and J is one of the languages you can go in and do that with. I don't like their interface and I'm sure you know we can do better. And I agree with you, but TryAPL is, I found, a really good site. In fact, coming early on I did a little bit of APL years and years ago and then ended up going over to J, but I was re introducing myself to to APL with TryAPL. It's really good.
[ST] 00:50:43
TryAPL, is absolutely brilliant. Yes, you're right. We use something a little bit like it in our training courses. Are running q behind Python and Jupyter notebooks. But we don't have a public interface like that. It's been talked about. I'm impressed by your by what you say about it. I'm going to take that back. See if I can get something done.
[CH] 00:51:08
Done, I think yeah, and I don't think q is on TIO.run, but I definitely know that there's a couple different versions of k because there's I know oK and Kona are some implementations that are based on, I think, K7. Maybe one of them is K4.
[RP] 00:51:25
I don't think it's there anymore, but once a while ago, maybe last year I saw some version of k that was done in WebAssembly, so it was just like, you just went to this URL. It was in your browser running natively in your browser right there. I think it's been taken down now, but that was really cool. Just the console, just straight there. That's your whole browser when you're...
[CH] 00:51:47
And and to be fair, I think downloading, q, k, J and Dyalog APL or any other APLs - it's not complicated. It's pretty straightforward. But definitely, I just think as a starting point, uh, there's a way lower barrier to entry if we can just send people a URL that they click, and then they literally start typing and like the lower that barrier to entry is, I think, the more people we can get hooked on being like "holy smokes it - I typed five things and I - I did some massive computation that would be non trivial in sort of another language."
[BT] 00:52:24
I think we have those links on under resources on arraycast.com, so if people are interested they can go over there and take a look and see what we put together in addition to whatever show notes we put on with this episode.
[CH] 00:52:37
And I guess maybe - maybe a good thing to start to close out on. We talked a bit about community and how currently that is maybe the best way and I can speak to that. I think on average I'm a more curious person than than the average individual, so I will tend to, you know, just type away until things start to work, but it is at certain points like I would go to APL Orchard and be like I've been trying to figure this out for four hours and then, sure enough, someone just is like: "oh, you're just missing a semicolon in the brackets, it changes the indexing." And I'm like, oh, how would I have known that they're like? Yeah, that's not the best, but yeah, the APL Orchard is fantastic, so going forward are we going to - you know whether it's around this podcast or - are we going to try and set up some kind of place? Whether that's a slack or a discord, or I know Stephen at one point you mentioned that we - you might be setting something up. Where should people go to ask questions of us or of just you know it's? This is, sort of, this is a cross-community podcast, right? It's not just for APL, it's not just for J, it's not just for q. Do we have that yet or do we need to set that up?
[RP] 00:53:49
Oh, a fair shout on that... well, there is one thing that exists. I'm not going to necessarily recommend this as the place going forward, but there's a subreddit r/apljk where - I guess due to the relatively niche small audience of each of the individual languages - they've banded together for this subreddit. And that's been reasonably active recently so, I mean, I know I look at it so...
[CH] 00:54:14
Did we post our first episode on that subreddit?
[BT] 00:54:17
Yeah we did. Yeah, we're tracking well.
[CH] 00:54:21
Okay, so.
[BT] 00:54:21
So that's part of the feedback we got.
[CH] 00:54:23
So there's a starting point. If you do have questions or comments or feedback, definitely check out that subreddit. It sounds like we'll be posting all the episodes there.
[RP] 00:54:32
Should I also mention one other community activity that's happening soon? Is this an appropriate time to stick that in?
[CH] 00:54:40
Sure, yeah.
[RP] 00:54:42
Adám Brudzewsky, who was on the first episode of the podcast, is - I guess he's hosting it. He's certainly involved in organizing it. A new meet up called APL Campfire. APL has a fairly rich history. It's, like, quite an old language, and this particular meet up is for people to come and give their stories. You know about their APL experience. So, that's going to be - the next meet up is Sunday the 6th of June at 6:00 PM UTC. And that's going to be via Zoom. But I'm guessing, uh, yes, to find out how to join, you can go to apl.wiki/campfire. So that's a nice straightforward URL. If you want to hear some of the sort of historical context around APL, which we've heard a lot of it alluded to, alluded to a little bit today, but it is really fascinating generally.
[CH] 00:55:40
Yeah, that sounds super interesting. I will definitely be tuning in if I am free.
[BT] 00:55:44
And I would just say with the one thing that I found about the different communities is: I think everybody who studies these languages knows they can be difficult and knows that they can be challenging and has felt those challenges personally at different times. I found people really really friendly and open and willing to explain things I've I just – I've been amazed at. That is, how much of that goes on. And as Stephen talked about, I think that's because that's in the culture of how you learn, and if somebody else is taking the time to work with you and teach you things. You just feel naturally you're going to give back and you're going to explain things to people who don't know quite as well, and I found that really a positive thing about the languages and it's just in that culture, so there's nothing wrong with going on these sites, on a forum or anything, and just lurking. I think that's not a bad idea. You start to see what's going on, but don't be afraid of asking questions because everybody was there once.
[CH] 00:56:42
I think there's no better way to end it than that, so I'll once again thank all of our listeners for tuning in. I know at the tail end we went through a ton of links, so if you lost track, just be sure to check out the show notes. They'll all be there if you want to get started and start exploring these languages, so thanks for listening and we'll see you in two weeks.
[BT] 00:57:02
Happy array programming
[CH] 00:57:03
Happy array programming
[Music Theme] 57:05