Transcript
Transcript prepared by
Adám Brudzewsky, Bob Therriault, and Sanjay Cherian
00:00:00 [Alex Unterrainer]
There's the saying that you're the average of the five people you surround yourself. So if your are constantly reading hard KDB code, you're going to improve. You're going to start thinking differently. You're going to learn from the code you read, the different approaches to solve problems. And it's definitely going to make you smarter.
00:00:27 [Bob Therriault]
Welcome to episode 108 of ArrayCast. And if you're expecting Conor's voice, do not expect Conor's voice on this episode. I'm your host, Bob Therriault, and with me is a panel and a very special guest. And first, let's go around the panel starting with Stephen, moving on to Marshall and wrapping up with Adám.
00:00:47 [Stephen Taylor]
I'm Stephen Taylor, I like APL and q.
00:00:48 [Marshall Lochbaum]
I'm Marshall Lachbaum, I'm the creator of BQN, and various other array languages.
00:00:54 [Adám Brudzewsky]
I'm Adám Brudzewsky. I like many array programming languages too, like Stephen. I don't understand all of them, but I'm best at APL.
00:01:02 [BT]
And I'm Bob Therriault. And as you've heard many times, I'm a J enthusiast. And today, before we go to our guest, we've got a number of announcements. I'll start with Marshall and then go to Adám, and then I've got a couple of announcements. [01]
00:01:15 [ML]
All right. So you may be familiar with APL wiki, the website about APL that anyone can edit. I like editing this website and I've done it again. The page that I've made now is for a while I've been kind of collecting information about the nested array model, which is what most APLs use now. And this was introduced in the early 1980s, which is what most APLs use now. And this was introduced in the early 1980s. And there were two different competing implementations that still kind of picker and have diffraction patterns today. So I've written everything you need to know about the nested array model on the APL wiki page, nested array model. So if you're interested in history, you might check it out.
00:02:06 [AB]
I can definitely recommend the APL wiki, especially the parts of Marshall wrote. It was great to read. So if you're interested in history, you might check it out. I would like to just give a reminder that the APL Forge round is closing on Monday, the 23rd of June. The APL Forge is this competition where you can submit some project or application or module to Dialog, and then we have some experienced APLs look at it and hand out a prize to whatever submission we like the best. So you can check it out. We'll leave a link to that, but it's just forge.dialog.com too.
00:02:36 [BT]
And we will have those all in the show notes. And as well, if you're listening to this later than some of the first published part of the episode, you may have missed your deadline. So you've got probably this weekend if it's early. And then after that, you'll have to wait for the next round of the forge. Cause I think it's something, is this the second one, Adám, or the third one?
00:02:55 [AB]
It's the second one.
00:02:58 [BT]
It is the second one. Okay, good. Yeah.
00:02:59 [AB]
And then there'll be a year until the next deadline. So you can always submit. It's just a matter of waiting.
00:03:04 [BT]
Yep. That's a, I think a really neat way of doing it rather than, I know you you've taken the competition, so you've made them a bit easier for people who are entering into APL and then the forge is for the more advanced people that used to try and take on the tougher parts of the competition. I think that's a really good way to do it. And I, I think and then my two announcements, well, my first one is John Earnest, who's been a guest in the past and does a Decker and the language Lil, that's sort of the background of Decker and Decker is kind of like a HyperCard stack. And I guess whenever you mentioned HyperCard, you should mention Bill Atkinson who passed away recently. And certainly an amazing programmer and was the guy that developed a HyperCard, an Apple product, but really changed the way a lot of things were done in computing in those kinds of ways. But John has taken HyperCard and made it his own in an array format. I guess it's sort of based loosely on different versions of k. And what John does is he does the usual thing that I guess all sorts of groups do in December where they have their competitions and stuff. He also does a fantasy camp in July. If you want to learn how to write things in for Decker in Lil, there's lots of people participating in that. It's a lot of fun. It's very open. People can see each other's code. I haven't tried to participate in it yet, but I've seen people participating in it. It's it's it's very lighthearted. So if you're interested in learning about Lil or Decker, I recommend you look for the show notes in that. We'll have a link to the Decker fantasy cap for 2025. And then the second announcement I have is Aaron Hsu has, I think it's two talks, at least it's a slide deck and a talk and, and it was at a LambdaConf recently. And so Aaron always provides interesting things to think about and he's certainly done it this time. It's a really good presentation. Not everybody agrees with Aaron's points of view, but that's good because he makes people think so. And Aaron's been on past episodes here too. So if you're familiar with Aaron, take a look at that and I think you'd enjoy it. And with that, we'll move on to our special guest who is Alex Unterrainer and Alex is KDB+ enthusiasts. So we're sort of scratching that itch for all the q and KDB+ people. And he's also the originator of Defcon q. Alex, would you like to tell us about, well, I guess the things you're doing, how you started out in computing? What got you interested in this? And eventually we'll certainly want to know about Defcon q.
00:05:40 [AU]
Yeah. Thanks, Bob. Thanks for having me. So I started programming early 2000s in high school. I started with Turbo Pascal. Some of you might have heard about Turbo Pascal and then moved on to Delphi and then Java, kind of like a natural progression back in the days when Java was the mainstream programming language that was taught at universities or in schools. And then when I graduated, I didn't really want to study computer science. In 2006, like the Internet looked quite different from how it looks nowadays. And it wasn't as fancy as it is or cool as it is nowadays. So I decided to just start working rather than getting a degree. And I ended up in a small investment bank in Northern Italy. I worked there full time from 2006 until 2009, straight through the big financial crisis, which maybe a few listeners will remember. And then I decided to get a degree in economics and management. And it was then when I was reading about algorithmic trading and it kind of like clicked. I could combine my two passions, finance and computer science, by specializing in computational finance, which I then did. I did two masters in the UK and started my grad team at Citigroup. And it was really just by coincidence that I found out about KDB because my flatmate at the time was working in the KDB team at Citigroup. So when I joined, I was really keen to join that team as well, especially because I learned that KDB is mainly used in the financial industry. And for me, for someone who's interested in finance, but also in programming, it made clearly sense to stick to KDB. So I will always be close to finance rather than being just like a web developer or like an app developer. And yes, at the beginning, KDB looked very different to anything I was used to. But if there's one thing about me that you can say for certain, it's that I'm very stubborn. So I just stuck with it. And no matter how painful the learning experience was at the beginning, I just kept doing. There were plenty of times where I wanted to give up and switch career or do something else completely. But eventually I started to really enjoy it and reach the point where I wouldn't want to do any other programming language or to be specific, any other non-array programming language. I played around a little bit with k and Shakti and even looked at APL. And I have to say, I wouldn't want to go to any object-oriented programming language. It's just so much boilerplate coding that's quite boring. And nowadays you can let Chat-GPT [02] create that stuff for you and you stick with the interesting cool part. That's me.
00:08:46 [BT]
And I should mention as well, when you say stick to it, I have to know your background as a hockey player. Yes. And at least in my country, which is Canada, it's quite common that hockey players tend to stick with things and go until other people might stop. So that's entirely within the character.
00:09:02 [AU]
And I used to be a goalie. So the one thing they say about goalies is that they're all a little bit crazy. So maybe that's where my stubbornness comes from. Yeah.
00:09:14 [BT]
I've known a bunch of goalies and my son actually played goal as well. So I can verify that he's a little crazy in a good way. So did you find when you were starting with the more traditional languages and then you had to move to, got into the array language, it's interesting that you were working in finance before you knew the array languages.
00:09:34 [AU]
Yeah, that's correct.
00:09:34 [BT]
So other people were working around you using them, but you were in an area where you weren't programming, I guess.
00:09:40 [AU]
Yeah. So, I mean, to be fair, I did work in finance, but that was in Italy where I'm originally from in North Italy. So the level of finance we did there was quite simple. Like a bunch of ETFs, a few stocks or bonds. So it wasn't anything too complicated. So there wasn't really an exposure to advanced programming languages that you would use. There was maybe a little bit of VBA or Python, but that's about it. Like the real exposure to all the different advanced programming languages only came later when I joined Citigroup. But even like, I've done computer science classes before and nowhere ever was actually APL mentioned, which like now that I know about the history and how q developed, like coming from k and A+ and APL, I think this is something that every computer science class should be taught because like it gives you a complete different view of how to do things and it makes you reason and think in a different way. So I think every computer science degree should at least have a lecture about array programming languages and APL and stuff like that. I was familiar with functional programming languages, but KDB, for example, it's not a pure functional programming language. So I definitely think like university professors should introduce that. I do know that there were a couple of professors that actually taught k or q. Someone I worked with at UBS, he learned k at Loughborough University. There was a professor teaching k and I know in the US, I think, is it Dennis Asha or Dennis?
00:11:34 [ST]
Shasha.
00:11:35 [AU]
He was teaching q and I believe k as well.
00:11:38 [BT]
And I think he, is he NYU, Stephen?
00:11:41 [ST]
Yes he is, New York University teaching k there.
00:11:44 [AU]
And obviously we have Nick Psaris who is teaching a little bit of KDB at Carnegie Mellon.
00:11:50 [BT]
Which would naturally bring us around to Defcon q, right? Because that, well, you tell us about Defcon q.
00:11:56 [AU]
Yeah, so basically after struggling and to learn KDB on my own, because I, so my way into KDB isn't the typical route, like the majority of KDB developers come into this programming language. Like you would normally join one of the big KDB consultancies like Data Intellect or KX or First Derivatives and then do their grad team there and they train you or at least to some extent train you in KDB and then they send you off to a client. Whereas I at Citigroup in the grad team, I just got thrown into the cold water. The way I started off was by reading q for Mortals, which I still recommend to everyone who wants to start with KDB. And then just like, you know, trial and error and try different concepts, play around with the language. Then I thought I understand something and then I kept trying some other examples and I realized, oh, if I actually try this example, so I get a complete different output than I expected. So realizing that because of the many overloads, you know, you might have ignored the case that you weren't aware of. So going back, reading q for Mortals again, and then two years, nearly two years ago in September 2023, I just decided that I want to give back to the KDB community. And I started my blog, DefConQ, which is a blog about KDB. And I'm trying to explain and teach KDB to people who are not coming from one of those consultancies. People like me who want to learn KDB on their own or just are not coming through the grad team in one of the big consultancies.
00:13:47 [BT]
So when you're getting somebody who's interested, so if they're interested, you've got that little hook in them. How do you bring them into it? Do you select an easy project or you do step by step through something? How do you approach teaching them that?
00:14:04 [AU]
So I pretty much follow the principle of breaking things down into very small pieces and understanding what each of those pieces does. So I have two blog posts, one about my favorite go-to learning resources about KDB. And then I have one blog post that is how to read, understand and learn KDB. And what I did there, I took one of, I think it was a code snippet from Stephen. I found it somewhere. He was truncating the empty white spaces at the beginning and the end of a string, which can be done in a very cool one-liner in KDB. And it also contains the Zen monk's example, Stephen.
00:14:55 [BT]
I'll say Stephen's got his face in his palm right now.
00:14:59 [AU]
I absolutely love that example. It's such a great way of doing things in KDB. So basically I broke that down into simple steps and explained it. And then I just keep, you know, posting about important KDB concepts I think will help people become a better developer. So then I just explain important operators, F by, apply, amend that. I explain concepts such as attributes, functional selects, IPC. And I also recently started to do smaller projects. Like I wrote a small real-time application that fetches data from Yahoo Finance and streams it to a KDB tick setup. I also explained like the typical setup of like a KDB tick architecture, the architecture, everything you pretty much need to know to become like a solid KDB developer who can be productive in a production environment.
00:16:03 [BT]
I think from what I remember, that architecture was one of the things in your talk at Iverson College [03] last summer that you went through and explained the different pinch points that people fall into that if you're, you can, you can have all the power you want in a language, but if you haven't put your architecture together properly it's not going to run fast. You have to not only know the language, but know how to put the pieces together.
00:16:28 [AU]
Yes, that's correct. Like if you have a fast car, it's also needs a good driver. So architecture is very important.
00:16:34 [BT]
Yeah. I was going to say Formula One learns in Monaco. You can have fast cars and they don't go that quick on skinny streets. So does anybody have any other questions for Alex? Stephen, I know your name's come up a number of times and I know you're involved with teaching q and KDB and that sort of thing. What's your feeling about the some things that Alex is doing with Defcon q?
00:16:55 [ST]
Oh, Alex, you poor thing. You got to Citigroup following your finance track and then somehow became infected with the desire to learn KDB and you learned it alone. What made it so hard? You had to call upon your hockey goalkeeper character to keep battling through.
00:17:14 [AU]
So I think like when I started with KDB around 10 years ago, like the documentation wasn't as bad as it was in the very early days when you only had the list box and could only send an email to a group of very experienced developers and you would be afraid of asking something simple. And there was already q for mortals and Code KX, but what that website, for example, was lacking and still to some extent is lacking nowadays is like some concrete examples like how do I do X, Y, Z in KDB? And with other programming languages, like if you're a Python developer and you want to do something, you just Google and you get like a million hits. Whereas with KDB, that's very limited. Another thing that I found difficult as a non-native English speaker, the language on Code KX sometimes can be a little bit intimidating. So I just like to break down things in a very simple structure that like a five-year-old would understand it. Obviously that's like an excellent exageration, but I just like to make it as simple as possible. Another thing that I found difficult was, and this might be just the nature of KDB or any other array programming language, the level of intellect of people working with array programming languages is incredibly high. Some of the smartest people I know work with array programming languages or in KDB, APL or whatsoever. And it can be very intimidating for someone who is not at their level to ask a question. So I often, and I don't know, maybe it's just like self-confidence or imposter syndrome. I often struggle to ask someone very senior experience, like, can you explain me this? Just because they might be very intimidating. And I did find out that a lot of people struggle with the same. So I just thought, okay, like, you know, I'm gonna create my blog and just be very approachable. And there hasn't been one question I didn't answer. It doesn't matter how simple it was. I honestly have answered all the questions. We're just making up his reactions. Whether it's like KDB related or career wise, I just think people should collaborate more and that will help the community.
00:19:52 [ST]
Well bless you for not using the phrase, but I'm gonna drop it in here. This talk of super smart people brings us to the phrase, the q gods that I've been, that in my years of looking after code KX.com, for which clearly I owe you an apology. I did my best to get taken out of use. Did you come across this phrase, the q gods?
00:20:20 [AU]
Yes. And I do remember that you are not a big fan of the word q god. That's why I actually avoided it on purpose. Coming back to you looking after the KX website, I have to say when you were managing it, it started to improve significantly. Like your iterator white paper you rewrote was like very, very, very good. I really enjoyed that one. So I was actually really sad when I found out that you no longer are in charge of the documentation. So q gods, I mean, I probably don't hate the word as much. I don't know whether you hate it or you probably just dislike it. I'm not sure whether I dislike it as much as you do. I find it, it's like, you know, something short that you can throw in a bunch of people who are at a level that a lot of people might not reach. And there are definitely a handful of people I can think of, but I also have to say some of them are actually really approachable. So maybe q legends would be more appropriate because you can actually still talk to them and they will help you. Then there are others that are, you know, in that category that are not as easy to approach or don't want to help you at all or share their knowledge. And it's a pity, like, I don't know why someone wouldn't want to share their knowledge.
00:21:56 [BT]
I would say that you're probably selling yourself short. Is the way that you can approach and communicate with people. Not everybody is comfortable doing that. Not everybody is good at explaining things and those that are quite often you find are approachable. It's it's not that common that somebody who's comfortable talking to other people.
00:22:16 [BT]
I would say that you're probably selling yourself short as the way that you can approach and communicate with people. Not everybody is comfortable doing that. Not everybody is good at explaining things. And those that are, quite often you find are approachable. It's not that common that somebody who's comfortable talking to other people will try and isolate themselves because they're sort of in their zone. They like to talk and get feedback. And the people I've met that are able to communicate and are very, very intelligent, as you say, those people I've found are often looking at you with a twinkle in their eye because they're learning a lot from you as you're asking the questions. You know, I'm thinking of people like Stevan Apter and people like that, where you just go, you can tell they're listening very closely to your question, not because, you know, they want to answer it, but they're also listening closely because they're finding things out about you and the way you look at things and the way the world works by the question that you ask. And not everybody's wired that way. I think you are. So I think that's why you might not see that. But sometimes when you approach people and they're a little standoffish and stuff, you got to realize they're probably working outside their comfort zone.
00:23:13 [AU]
Yeah, I agree. There have definitely been occasions where I learned more from a question than I learned actually from anything else, because it really makes you sit down and think about whether you fully understood how something works. And then it also makes you think about a way how you can explain it to someone who might not understand it the way you understand it. And that just teaches you so much more about concepts than you can maybe learn from a book where it's only explained in one way.
00:23:46 [BT]
Yeah, I think from what I remember, it might have been Richard Feynman was saying, if you can't explain something to a five year old, you don't understand it yourself.
00:23:54 [AU]
That's correct.
00:24:02 [BT]
Yeah. And I think that you can massage that into maybe not being entirely accurate, but I think certainly basic concepts, if a person, if you can't explain it to somebody in a way that they can at least grasp the basic parts of it, then maybe you don't have a handle on the basic stuff as well.
00:24:13 [ML]
John Scholes [04] has a very nice quote related to this, which we can put in the show notes. I've linked it for the people who want to look. It's a muse about stupid questions and it ends with a soundbite. A stupid question is a portal into an alternative mindset. So he gives some examples where people asked what he thought at the time were stupid questions. And he realized later that they were actually, that they were good ideas for APL implementations.
00:24:41 [BT]
Yeah. And if people are not aware of John Scholes, he sadly passed away. But if you go back to some of the old Dyalog vidoes he talked, in addition to being very interesting insights into the language, some of them are hilariously funny and basically are comedy routines. He was a very, very clever man. And I guess probably best known by the one liner for the game of life is probably how we, if people know the name, they probably know it through that.
00:25:10 [ML]
Yeah, the video about that.
00:25:12 [ST]
I'm not done with the cute gods yet.
00:25:14 [AU]
Okay.
00:25:15 [ST]
My quarrel with them is, relates very much to what DEF CON q is about. I know personally some of the people who get designated the q gods. And just as you say, they're incredibly smart people and it can be an awakening just spending time with them. My quarrel with the term is that it suggests that they're doing stuff that we can't do. And I've got a very deep quarrel with that. I think I wrote, possibly even on CodeKX.com, the q is a language and it yields to study and practice like any other language. And that's what I admire so much about what you're doing with DEF CON q, the insistence that none of this is out of reach.
00:26:04 [AU]
Yeah, I do agree to some extent. It's definitely not out of reach because I am doing it and I'm by far not like, you know, the best or smartest programmer out there. I'm just incredibly stubborn. Where I see the difference is that some q gods might just like come up with things that no one else has seen before. So I don't want to say it's something they can do that others can't. It's just like they set the first step, like they introduce new ways of, I don't know, using KDB in a way that you might not have thought about. And then once it's there and you look at it, you're like, oh yes, this makes sense and you understand it. And then whenever you have a problem where you realize you can apply that pattern or that code snippet, then you repeat it. Maybe for some people it just takes longer to get to that stage and become a q god. It's more practice, more examples. But yeah, there are definitely people who handle KDB or q or any other array programming language in a different way. It's like, I don't know, people riding a bike and they are doing some incredible backflip that no one ever thought you could do it and they just go and do it, you know, like others try and fail and fall and get injured. And then there are some people, they just go on a bike downhill and there you go, triple backflip out of nowhere. I see it like that. So I don't want to say that it might not be something no one else can ever do again, but I think some people just take longer than others. So maybe they're not gods in that sense that, you know, they're unreachable, but they're definitely more talented than others. And that's fine. You know, that's with everything in life. You know, some people have different talents than others and for someone it just clicks and then maybe they understand it, but cannot explain it. And that's perfectly fine. Like you need different talents to, you know, grow your community. At least that's what I like to think.
00:28:51 [ST]
I love that interpretation. Because you could turn that around and say that the language kind of gives you access to some of the ways these very smart people think.
00:28:59 [AU]
To some of the ways these very smart people think it's an easier access than you know, reading 404 hundred lines of C code, you can look at 4 lines, OK.
00:29:02 [AU]
Oh, absolutely.
00:29:10 [ST]
It gives an easier access than, you know, reading 400 lines of C code. You can look at four lines of q code.
00:29:21 [AU]
It also makes you better. It makes you smarter. You know, like there's the saying that you're the average of the five people you surround yourself with. So if you're constantly keep reading hard KDB code, you're definitely going to improve, you're going to start thinking differently. You're going to learn from the code you read, the different approaches to solve problems. It's definitely going to make you smarter. I hope I didn't get dumber over the last 10 years.
00:29:37 [ST]
You mentioned the Zen monks pattern earlier, which is a very tiny little snippet. Would you like to explain that one on air?
00:29:46 [AU]
So basically, it's a nice way of using the scan operator to apply a function to an input, but also not to apply it. So that was the original idea of the Zen monks. You need two monks to do a task, one who doesn't do the task and one that actually does the task. And by using a scan, like a function, you can actually get the original input back and the modified input where the function is applied to that input. So the way it was used in that truncating code snippet, it was basically you had a Boolean mask and you wanted to reverse the Boolean mask, but you also wanted to keep the original one. So you could use that Zen monk pattern to get back the original Boolean mask and the reversed one. And then you would continue by comparing the different Boolean flags to each other. Let's say you have a, what is it? Palindrome or anagrams, like a word where you can read it from the front to the back.
00:30:53 [BT]
Palindrome.
00:30:53 [AU]
Thank you. I always mix those two up. So basically if you want to check if something is a palindrome, you can basically just use that Zen monk pattern to reverse the string, but also keep the original one. And then you just compare each element with each. And if they match, then you have a palindrome. If they don't, you don't. It's that simple. Like a few characters in q and you have a nice function that returns true or false if a string is a palindrome.
00:31:22 [ML]
Yeah. So the pattern is just a reverse scan for that particular part of it, right?
00:31:27 [AU]
Yes. But you can exchange reverse with the other function.
00:31:31 [ML]
So I mean, this is a little confusing from APL because it's not a scan along the array. It's actually the k form of scan that it's more like APL's power operator for the APL listeners.
00:31:45 [BT]
The APL listeners have just gone into an alternate universe.
00:31:50 [ML]
A portal into an alternative mindset.
00:31:51 [AU]
Someone else needs to confirm that my APL is not beyond the initial 10 question challenge.
00:31:57 [AB]
Marshall, can you explain that further? Because I don't get it.
00:32:01 [ML]
So k has a form of scan that...
00:32:05 [ST]
It works on monadic functions.
00:32:06 [ML]
Yeah. The distinguishing factor is that the operand is a monadic function. That's right. And it'll just apply this monadic function. It's like APL's power limit, actually. It'll apply the monadic function until the results stop changing or until they match the original argument, I think.
00:32:23 [ST]
It'll take a left argument to say how many times you want to do it. That's where it's like the APL power operator.
00:32:29 [AB]
Right.
00:32:30 [ML]
And for reverse, you reverse it once, then it's reversed. You reverse it twice. Well, that's the same as the original argument. So it stops there and it just returns the original array and the reversed one.
00:32:41 [ST]
I could do this in APL if the power operator would take a vector for the number of times I want it done. So it's like a zero and a one. So I get a function applied and also not applied.
00:32:55 [AB]
Oh, but wait.
00:32:56 [ML]
Yeah. So this is why it's a scan and not a fold, because it keeps every input, maybe.
00:33:01 [AB]
Right. No. Okay. I understand. But then you compare them after you do that. Compare the two that you get.
00:33:07 [ML]
If you're checking for a palindrome, yeah.
00:33:09 [AB]
But if it is already a palindrome, don't you only get one? Because it's already converged, so to say?
00:33:15 [ST]
Well, they'd be the same.
00:33:17 [BT]
From what I understand, you would get two, because you've got one that's the original that hasn't had anything done to it. And then you get the other one that's reversed.
00:33:24 [ML]
It's a good question. If the reverse is the same as the original.
00:33:28 [AB]
Yeah, we're already done.
00:33:29 [AU]
But then that means it's a palindrome.
00:33:31 [ML]
But it still gives you... It doesn't stop when those two match. Why is that? Or does it?
00:33:37 [ST]
Because you use the left argument of one to the derived function. So you say one reverse backslash. Yeah. And when you do the one function backslash, you get it applied zero and one times.
00:33:52 [ML]
Oh, okay. So you explicitly tell it how many times.
00:33:55 [AB]
OK, now now I'm with you.
00:33:56 [ML]
I've seen it with no number for like negation, for Boolean negation. And then you'll get the... As long as the argument isn't empty, you'll always get two results.
00:34:06 [AB]
I'm so happy for my Dyalog Version 20, which is almost out. It has an operator that allows me easily to especially compare an argument with some transformation in the argument so that palindrome checking is trivial. If the reverse matches, then yeah, it's a palindrome. I didn't think of doing it like that.
00:34:28 [AU]
I mean, there are probably many other solutions to this problem. Like many ways lead to Rome.
00:34:34 [ST]
Let me jump back to something Alex was saying earlier about people's willingness to share and teach because the origins of KDB in finance actually deterred people from doing this. I'm going back a while, but early KDB customer user meetings, you need to sign an NDA just to get in.
00:34:55 [AU]
Yeah. Like the fact that it's used in a very secretive area, it also makes it not super community friendly. But you could say the same thing about, you know, Python C++. [05] It's also used in financial industry, but you will still find plenty of tutorials how to do certain things.
00:35:15 [BT]
You mentioned a number of times about the community for people who are visiting DEF CON q. What sort of, are they talking to other members? Do they exchange ideas back and forth? Do they work through you on the blog? What's the process?
00:35:29 [AU]
So I don't have comments enabled on the blog mainly because I'm not a web dev and I don't want to look after any features. No, but I post my stuff mainly on LinkedIn because that's the way how you can reach the people who work with KDB. So I get comments on the blog posts and questions and requests. So that's one way. But I started to do community meetups. So in May, we had like a London meetup where we just had drinks, all the KDB developers in London. So it was kindly sponsored by KX. So we put up an event at one of the pubs in London and it got a really good turnout. Like there were like around 60 KDB developers, which is pretty good turnout. So everyone had a good time, was chatting with each other and just try to see what others are doing. And then yes, like on my posts on LinkedIn, I would also get comments or suggestions or why are you not doing this? Or can you explain that? And I do like to see that I started to make others interact with the community more often. So I can be quite provocative. I just like to poke the bear. So I posted about garbage collection and say that there is no garbage in KDB, which technically isn't correct. But still, I knew what I was doing with that. And I got like a senior KDB developer who sent me a message from LinkedIn saying that, look, it's technically not correct. And I was like, yeah, I know, but I wanted to create some engagement.
00:37:23 [BT]
Nice to meet you.
00:37:26 [AU]
And I also do that with other things. Like there are plenty of Python community websites or like companies that post on LinkedIn and they're like, oh, in Python, I can do this and that. And I comment, oh, interesting. I can do this in one line in queue. And sure, it's provocative, but it also might raise attention to KDB. And I do want people to explore KDB and start using it more because I think it's like a very great language to use in a lot of use cases. Like I pretty much use it for pretty much everything. Like if I need to calculate some stuff rather than taking out the calculator, I just open my console and do it in KDB. It's very efficient and elegant and you can do so much with so little and you're very productive. Like I remember when I was doing Java, you have to create this main class, you need to build, you need to compile all that stuff. Like by the time you opened your IDE, I already have a solution in KDB because some of those IDEs just take forever to load. I remember in Citigroup, I also had to work with C#. It took seriously, because the code base was so big, it took 15 minutes until like Microsoft, I don't remember what the ID was. Now it's VS Code, but back then it was something else. So it took, and I kid you not, it was 15 minutes until it just opened because the code base was so massive. And you can do a lot of KDB in 15 minutes.
00:39:02 [AB]
Or any of the array languages, I would say.
00:39:06 [AU]
Or any other array language, yeah.
00:39:09 [BT]
When we had Jonny Press on, he was talking, and it reminded me when you were talking about getting the KDB developers together, he was saying that there was a way for the community to interact between all the banks. They had a common message system that from time to time people would put a problem or a challenge up. And he said at that point, if it was a really good comment or a challenge, he said pretty much everybody who's working at the banks would basically put what their work was to the side and then work on that to see who could come up with the best solution. And he said you would have probably been able to track productivity across the banks in the UK by what questions were coming up. And he says occasionally Arthur would come back on and put in a really quick answer to something. And then he said everybody then, he said that was the rest of his day. He would just be looking at that, trying to figure out what was going on in that one liner and would learn something from it. But as far as what his duties were at the bank for that day, he said there wasn't that much of that going on. But it was a really, it was interesting. I don't think that communication form exists anymore, but it was interesting to say at that time, that was the real community developing area. And that's where people used other people's ideas to improve what they were doing, because there was a competition there to see who could come up with the most elegant, the quickest. First thing was to be the quickest, but after that it was to be elegant or to be short. And that made you better.
00:40:43 [AU]
That was the list box I mentioned at the beginning. That was in the old days before the documentation existed. They just had like an email distribution list. You can send your question to [it] and then everyone would get your email and you would just reply. And yeah, you're right. That's what Johnny mentioned. And he was saying whenever he saw Attila was working on it, he was like: "I don't even bother."
00:41:11 [BT]
[Laughs] Yeah, I don't need to. I'll find out soon enough.
00:41:14 [AU]
Yeah, yeah. I think the Code of Advent created that spirit a little bit. In December, a lot of people work on the Code of Advant problems and try to solve them with KDB. KX recently started a Slack channel, but I don't think it gets as much traction as the original list box did. They also have their forum where you can post questions. StackOverflow, I have to say, became more popular recently. A lot of people post on there. I remember I was speaking with someone who worked at StackOverflow and they told me the percentage of questions about KDB on StackOverflow; it's like close to zero.
00:42:02 [BT]
Yeah. And one of the things I've learned about putting together communities and those kinds of things is you can't make them happen. What you can do is you can provide the ingredients to have something happen. But then the biggest thing is when it starts to happen, to try and get out of the way as much as possible so that it can actually develop. Because if you'd go in and say: "oh, this is really cool. Let's do it this way," you've immediately alienated most of the people who are interested in coming in and being part of the community. So you have to set it up and then let it go and see where it goes. And there's a little bit of guidance that sometimes it needs just because ... well, Alex, we have people who come in and provoke other people! And sometimes that's not the nicest thing. Sometimes it's really creative and really good. So sometimes there has to be guidance with it. But generally you're trying to let things go as much as possible because it's the community itself that's going to grow. It's not you growing a community.
00:43:03 [AU]
Yeah. It's a very difficult one. Last year, Stephen started his Q201 project. [06] And when I found out about it, I was super excited. And then when Stephen reached out to me asking if I want to join and help, I was like: "wow, this is a big honor to be asked by Stephen to participate and help him with this project." And then, yeah, unfortunately, in my opinion, we didn't get the traction it deserves and the traction I would have expected. If you would have told me about that at the beginning of my journey, I would [been] like: "hell yes; Sign me up!". It doesn't matter what time of the day or where, I'm going to make it happen. I want to learn from Stephen. I don't understand why people will not take that opportunity to learn from one of the q gods, if you want to call them. I'm sure Stephen doesn't like that, but I don't know. I would have killed to get that opportunity.
00:44:10 [ML]
You're actually supposed to just kill a goat and then you've let some of the blood out as a sacrifice and that pleases the q gods [laughter].
00:44:18 [AU]
Exactly
00:44:19 [BT]
Sometimes it's a timing thing. Really good ideas don't take off and then they show up in some slightly altered form a year later and boom! They're gone. I've seen that happen. Not many times, but I've seen it happen. It just confuses the people who did the original version [laughs] because they kind of look at each other and go: "what's going on here? We did this. Why are you, how did you do that?" And then it turns out that it was just a matter of ... [sentence left incomplete]. I mean, people criticize Malcolm. Gladwell for being a bit glib in the way he discusses things, but he mentions there are certain people that if you can attract them into a group, the group will grow because they're connected to so many other people and their job is almost like glue within a community. If you get those people interested, it's more likely to take off. Again, more likely doesn't mean it will, but it's more likely.
00:45:11 [AU]
I was going to say, I have the feeling sometimes people want to learn KDB for the wrong reasons. And this might be just how the industry is. It's used in finance. People associate that with massive salaries and you can get rich overnight. Then you get news magazines advertising: "oh, KDB can earn X, Y, Z in a year!" And then people are like: "okay, I learned this and I will get rich overnight." Now, first of all, those salaries are most of the time exaggerated; they're inflated and nearly never realistic. And then the other thing is it doesn't need to be KDB to earn that kind of money. The same with C++; there are some people out there who are earning a fortune because they have 20 years of experience in C++, but that's the point. They have 20 years of experience in C++. So it doesn't come from nowhere that they're earning that money. And to get to that level, it needs a lot of dedication, sweat and blood sacrifices.
00:46:26 [AB]
[Laughs] Where's that goat again?
00:46:27 [ML]
You use the same goat twice! [laughter]
00:46:28 [AU]
So people just see those headlines and they're like: "okay, I can earn this money." And I sometimes get these messages. They're like people like: "hey, can you teach me KDB? I want to be a KDB expert in a week or a month." I've been doing it for 10 years; I'm nowhere close to retire. And I'm pretty sure if it would be that easy, a lot of people would be retired and happy. Happy days. But yeah, so that's the unfortunate thing that a lot of people get into this programming language for just the wrong reasons.
00:47:03 [BT]
So what would you say are the right reasons?
00:47:05 [AU]
The right reasons is if you do [to] like solve complex problems in a very simple way with as little as code as possible. And if you do like to broaden your horizon, if you want to start thinking in different ways and maybe even become smarter, then I think you should definitely look into array programming languages.
00:47:31 [AB]
What did you say? Difficult, hard problems? Big problems? I would say even for small things, it's worth it.
00:47:37 [AU]
Yes, definitely. But it might be an overkill, but yeah. You're right. You can use it for pretty much anything that's kind of like backend related database. I mean, I wouldn't build a website with KDB, but you sure can.
00:47:56 [BT]
So if I was going to sum it up, you're saying the wrong reasons might be just going after making a quick buck, but the right reason might be the quest for power [Alex agrees]. Because you can do things with this. And if you've got things that you want to do, this enables, to give you the power to be able to do things.
00:48:06 [AU]
Exactly. Yes, that's a pretty good summary.
00:48:13 [BT]
So how are these languages used? I mean, they're used in the finance industry. I happen to know they're also used in areas where you've got incredible amounts of data that need to be processed very quickly. Formula One teams are using them to be able to take all the sensors' [data] off the car; quickly figure out what's going on and fire it back to a person who can make an analysis of what's happening in real time. And I think there are groups using it to balance power grids and things like that. Are there other areas that you've heard of that people might not have thought about?
00:48:51 [AU]
It's also used in healthcare, like big pharma. In my opinion, you can use array programming languages in pretty any sector that's dealing with big data. And with the amount of information that just keeps growing, that's pretty much any sector. Everywhere you need to do fast calculations or vector operations, you just use an array programming language because it's based on arrays. Doing array additions, multiplications in KDB is just trivial. Works out of the box. You don't have to use a loop at all and it makes it so much more efficient. So pretty much everything where you are touching on big data, you can use an array programming language. And sure, you can use it for smaller problems as well, as Adám said. I do agree if we ignore for a moment licensing, I would just use it for anything.
00:49:49 [AB]
Shout out to all the array languages, including k implementations (but I guess no q implementations that are free to use for non-profit purposes or entirely free to use for that sake) where you can write your little website if you think that's a good idea.
00:50:07 [AU]
Yeah, definitely.
00:50:09 [BT]
Well, you can get access to q on a limited basis. Can you not, to work with?
00:50:14 [AU]
There's the personal edition that is free for just personal use. [07] You can download and start playing around with it. And yeah, I have it installed on my machine. I do all my hobby projects on my laptop, so I don't need a commercial license. So it's perfectly fine.
00:50:33 [AB]
Doesn't that one have to call home, all the time?
00:50:35 [AU]
Yes, you need an Internet connection. That's correct.
00:50:38 [BT]
But I'm just thinking in something like that, that licensing agreement would be that if you get to the point where you would require more, then chances are you've got an income that would support that. You can make a business case for it.
00:50:50 [AU]
Yeah. I believe Stephen mentioned once [that] the APL model is quite interesting, where it's free until a certain threshold is reached. And then (I don't know how much) it's a certain amount that they ask for.
00:51:03 [AB]
That's the default. You can negotiate a different thing as well. That's just, if you don't do anything, that's what you have to do. But if you're not making money off it, then that's an excellent model to use.
00:51:19 [BT]
And if I mention J, in the old days you could use J for free as a student license. And I have to admit to Eric at one point that I was using it as a student license because I was learning J; I wasn't a student [laughs], but they would let you go along with that. And now it's just wide open. You can use the language just by downloading it. So it's available that way. And I guess the other thing I would add is the different web-based versions that you can use to try these languages out. But I don't think q has one yet, does it?
00:51:52 [AU]
No, I don't think so. No.
00:51:53 [BT]
That's too bad because I think that's probably the easiest, when you can go to your browser and start up a webpage and actually try things with the language. I know BQN, TryAPL. Is it BQNPAD? Is that right, Marshall?
00:52:09 [ML]
That's one of them [chuckles].
00:52:11 [BT]
One of them. You have many [laughs], or a couple anyway.
00:52:14 [ML]
ngn/k has a nice online interface though.
00:52:17 [AU]
Yes. Yeah. That's k though. Honestly, I like the terminal the best. It's so easy to install and then you have your terminal. I know for example, KX, on their KX Canopy, uses Jupyter Notebooks to give you easy access. They have their sandboxes and you just spin them up. But I am not a fan of Jupyter Notebooks. So for me, the terminal is the simplest way to learn; to play around.
00:53:02 [AB]
Not Jupyter Notebook, but for just playing around, I think I found that awkward too. But the online "Try It Out" sites are not like Jupyter Notebook. They tend to be more interactive. Immediate results.
00:53:08 [AU]
Yeah. Just a simple REPL.
00:53:10 [AB]
And they can be a REPL. They can be fancier than that. BQNPAD has this very nice pre-calculation feature where you see the result as you type before you even press enter and then you commit to it. And I tend to get into some weird state where I can't get myself out again [chuckles].
00:53:28 [ST]
Let me tell a little story of how we used KDB in a community edition. I think this was back when there was still a 32-bit version of it. Some of my neighbors came to me to complain about the ice cream vans, that's parking illegally and tempting children onto dangerous parts of the road. And they said: "we see these guys being ticketed, but it doesn't seem to deter them; I wonder what's happening." Our local authority has an open data portal and you can go and find all the parking fines. And so I said: "well, let's go take a look at this." And the open data portal is set up with an API. So you can send it SQL queries and get [results] . And they'll come down in JSON format [08] and you put them into a spreadsheet and so forth. It's all manageable. But I find all that stuff kind of heavy going, and I got q. So I used a form of the API, which took a very large query [laughs] and basically gave me most of the dataset. It's like a million and a half records and this isn't going to fit into any man's spreadsheet. But just pulled that down into the REPL session and started running some queries as I thought about them: "What's this? How about that?". And I filtered it down. So I spent about 20 minutes with my neighbors and we found out that we couldn't match up their snapshots of the ice cream vans in illegal positions and being ticketed. And mysteriously, the tickets weren't showing up in the database. Something else is going on there.
00:55:20 [BT]
So they might've been getting warnings, not tickets.
00:55:23 [ST]
That was for my neighbors to follow up. That was something which would have been, a half a day's work with Excel spreadsheets or Panda datasets to pull that stuff apart and write all the queries. And we just suck it down into the KDB session and hack around it with a few one line queries. Somebody told me, years ago, it's like the difference between walking into a forest, carrying an axe and carrying a chainsaw.
00:55:56 [BT]
On a recent [episode of] one of Connor's other podcasts, I was listening to it (I do listen to other podcasts that Connor does), he was mentioning his dad ([who] actually in my area of the world is actually quite a well-known investigative reporter). And he was talking about the stages his dad went through when Google came on and then when other tools became available that hadn't previously been. Previously all the research would be done by going to a library; going through microfiche, microfilms, flashing screens up and having to go to a special machine. And that all changed of course, with the Internet's access to that. And I would guess now that a lot of the access for search engines has become more commodified. You have to learn other ways to get around [to] this information. And it's things like open portals. I would say if somebody was really interested either in investigative reporting or community activism, these are tools that are really useful because you can take the raw data and figure out what's going on; not have to rely on somebody else spinning it for you and telling you what they would like you to know. So if I was going to put my rebel hat on, I would say ([to] somebody listening to this kind of thing), if there's an area that they think should be improved in society, learning how to use data and information efficiently and well, is a very powerful tool. Actually, as Alex points out, it's not always intelligence, but most other people don't have the tools to know how to use it. In that way, information is actually being used against you because the people that have the tools to use it are using it. And if you don't know how to use it, you don't have that kind of influence.
00:57:45 [ML]
You would be a REPL rebel, is the specific kind [laughter]. You need to talk to David Bowie about that.
00:57:54 [BT]
[Laughing] I was going to say rebel rebel. I was hearing echoes of Bowie.
00:57:56 [ST]
Shape, REPL and roll.
00:58:01 [BT]
Oh yeah. We're going to descend into dad jokes soon, I think. Anyway, Alex, is there anything else you would want to let people know about DefconQ? How to get involved? You mentioned LinkedIn. Is that the easiest way to contact you? How do people get involved?
00:58:21 [AU]
So you can definitely check out my blog: https://defconq.tech. I also have a newsletter on substack, which is probably the easiest way to stay up to date. I do post on LinkedIn and on my blog and the substack all the time. So if you are on LinkedIn, you most likely have seen my content and might even be annoyed by it. If not, then follow me there. Visit the blog; subscribe to the newsletter. It's all free. No hidden costs. That's pretty much it. And yeah, if you're interested in getting into KDB, have a look at my blog. I think it's a good point to start.
00:59:03 [BT]
Yeah. And I think people now know from listening to you that if at times you might seem prickly, you're not that way at all. It's just a matter of trying to get people interested.
00:59:13 [AU]
Yeah, I'm actually quite a nice person.
00:59:16 [BT]
I can verify that after meeting you at Iverson College last summer. And also, if you've not been to Bolzano, you must go to Bolzano. I did a trip to Italy last year, and before I went, I was talking to Alex and he was telling me about Bolzano. And so we expressly, my wife and I, went up and spend a day in Bolzano. It's a wonderful community [laughs]. We learned a lot of history. We saw a lot of things. And I guess it'll be part of the Winter Olympics next year. Is that right?
00:59:48 [AU]
It is. Yeah, because the biathlon is going to be there in Antholz. And if you're a big tennis fan, then the current number one [among] men, Janik Sinner, is from South Tyrol, the region where I'm from. So yes, we produce good things there.
01:00:08 [BT]
Well, it's a really amazing part of the world. As I said, we only spent a day there, but really enjoyed it. Living in British Columbia, there's lots of mountains around and that always makes me feel like home. So I felt right at home in [Bolzano]. And anyway, with that, I'll thank you, Alex, for coming on and sharing all your information with us and hope people get interested. As you said, the power that it gives you is probably the best reason to be interested in these things, whether it's power of developing your awareness or intellect, or whether it's actually power of being able to shape information. I think these are things that will sustain you and keep you working at something. And I would also say (and I think Aaron, in that talk I was mentioning in the announcements talks about this) the question of why for a lot of people who've already learned computing science (traditional languages or procedural languages) find it difficult to do array languages. Whereas people who are not trained as computer scientists will use these languages and think they're actually fairly straightforward. So if you happen to be stumbling into this podcast and think: "well, I'm not a computer scientist." Well, don't let that stop you. Go to something like, TryAPL. If you're adventurous, go and download q and play around with it. Get a sense of what it's doing, because I think you'll find as you work with it, you're going to learn things, but you're also going to find it's not as hard as you thought it was. Because unlike a computer scientist, you're not going to have to unlearn things before you can really start playing around with the languages. That's actually a big deal. Anyway, thanks again for coming on, Alex. Really appreciate you. If you do want information, the shownotes are with us. You can reach us at contact@arraycast.com [09] A shout out to our transcribers. I think right now it's Sanjay. We haven't seen Igor come back. So somebody can put out a search for Igor. He was heading into a very busy time so it may be a while before he comes back on, but thanks to Sanjay. Thanks to Adám for doing the preliminary pass. I know Adám doesn't feel good about being thanked (because he says it's so easy now with the programs that he uses to get that first transcription) but it really is important. A lot of people reach out that way. And so with that, I guess it's time to say: Happy Array Programming!
01:02:35 [everyone]
Happy Array programming!
[music]