Transcript

Transcript prepared by Bob Therriault, Igor Kim and
Sanjay Cherian

Show Notes

00:00:00 [Henry Rich]

You don't have to drink the whole tacit programming kool-aid to use hooks and forks judiciously in the middle of essentially explicit code.

00:00:14 [MUSIC]

00:00:22 [Conor Hoekstra]

Welcome to Episode 73 of ArrayCast. I'm your host, Conor, and today with us, we have four panelists, plus a returning guest who we will get to introducing in a couple minutes after brief introductions. So we'll go around and start with Bob, go to Stephen, then to Adám, and then to Marshall.

00:00:38 [Bob Therriault]

I'm Bob Therriault. I'm a J enthusiast. And so the day is just perfect for me with our guest.

00:00:44 [Stephen Taylor]

I'm Stephen. I'm an APL and q enthusiast.

00:00:49 [Adám Brudzewsky]

I'm Adám and I am an APL enthusiast. No, I'm actually pretty enthusiastic about all the array languages, but I do my work in APL.

00:00:59 [Marshall Lochbaum]

I'm Marshall. I've worked with J and APL in the past. Now I'm doing BQN and Singeli and possibly more.

00:01:09 [CH]

And as mentioned before, my name is Conor, host of the show, polyglot programmer, and also massive a fan of all of the array languages. And with that, I think we have a few announcements. We'll go to Bob first for a short announcement. And then I think Adám's got a couple and Marshall's got three or so.

00:01:23 [BT]

And my announcement is hot off the press. [01] As yesterday, I was made aware of a video. Well, it's a video, but it's actually also a podcast. Everybody seems to be doing their podcasts on YouTube now. But this is ProgLangCast. And I'm not sure about one of the host name because I don't think I over caught it. But the other is Christopher Augustus. And what they're doing is they're taking a different language every two weeks, I think. And they're showing how to do what they call. Well, their thesis is that "Hello, World" is just too simple to really show anybody what's going on in the language. So they're referring to it as "Bye bye, Hello, World." And they're taking a different language each time. So in their third episode, they've taken APL. And it's interesting. I think Christopher has some experience with APL, as does the other host. One doesn't like symbolic languages so much as he refers to them. And Christopher does. And it's kind of interesting to watch it. He admits that this is not what APL is necessarily best at. It's not really showing off. But I think anything that promotes the array languages we can get behind. So I thought I'd make people aware of that. And you may see other array languages come into their languages that they're looking at in the future.

00:02:43 [CH]

So yeah, definitely link in the description if I haven't seen it yet. This is news to me, which is why I love hosting this podcast. So I'm going to go watch that podcast probably right afterwards. And yeah, link in the show notes if you want to check that out. Adám, over to you.

00:02:54 [AB]

Yeah, well, very related to that. I did watch that. It's a whole hour. And yeah, well, they're not, as they say themselves, they don't tend to be expert APLers or array language programmers. Both of them have used APL in the past long ago. And so I thought I would make like a reaction video to that where I solve the same problem, discuss some of the things they're doing and alternative ways of doing it. And also doing it in Dyalog APL instead of GNU APL gives some more features and a bit sleeker in some ways. So we'll have a link to that as well. And then a new one of these many, what should we call them, tools, services online. Something called the OmniBar, at least it's called like that for now. This is Madeline Bergani, who has made kind of like you have a language bar in the many interfaces for array languages, those that use at least non-ASCII symbols. And they can be used to explore the language. You generally can hover over them with a mouse and you get some kind of pop up about what they do. And she's put together already a list of what are there, it's like about 10 languages into one giant list of features per glyph. So it just goes through the glyphs and say, there's this meaning that could be given to this glyph, for example, plus can be conjugate and they can be addition. And then it lists which implementations, dialects of or specifications of APL support that meaning of the glyph. And then there's a little search bar and you can make small queries to compare them, see what are the features that this language adds over this one and what are the differences between this one and that one. What was new here? And it's kind of cool to go and explore. It's still a work in progress and it's open source, you can maybe contribute and it's being discussed actively pretty much every day in the APL orchard as well.

00:05:06 [CH]

Yeah, this is really cool because one of the dialects of APL that we haven't really talked about a ton on this podcast is NARS 2000, which is Bob Smith's APL, correct? And it has a ton of, like, I guess, extension glyphs, you'll call them things that don't exist in other APLs that I don't know a ton about. But if you go through this site, you can see that there's these, you know, integral and derivative that only exist in N2K, aka NARS 2000. So hopefully maybe we can get Bob on an episode in the future and we can talk about some of the things that exist in NARS that don't exist in other APLs. But yes, very cool website. We will be paying attention. Expect to hear updates when the site changes and gets updates as well. Anyways, over to, I believe, Marshall for the remaining announcements.

00:05:52 [ML]

So my first announcement relates to BQN's foreign function interface, which is how you call functions in C. So that lets you do, I guess we talked about it a bit on the games episode, it lets you interface with all these frameworks that are written in C. One real issue with this has been working with C functions that pass around pointers. And so what we've had in the past for this is that you can accept a pointer, but tell it convert it to a BQN array. And so it converts it, but it's just like the value of the pointer as an array of numbers or whatever. What we have now is a pointer object. So you can tell it, just give me the pointer and it will give you a namespace back that you can then, it remembers what type it is and you can read to it and write from it at different indices and cast it and pass it to more FFI functions and so on. So that's a pretty nice ease of use thing as far as the FFI is concerned. Then my second announcement, well, it might be two announcements. First is that Nik Nikolov, who is the developer of ngn/k, announced a few weeks ago that he was no longer going to maintain it and he has indeed stopped maintaining it. But the second announcement is that I've decided, you know, I don't really want this implementation to be unmaintained. So I've started working with it a little bit. I've fixed a few bugs now. I'm actually working on some of the, we talked about windowed minimum on this podcast. So I'm right now working to implement one of the algorithms that we talked about there. So that's faster. And these, I'm making pull requests and Nik is accepting those. So I'll keep maintaining the language however I can as this goes forward. I don't want to make any radical changes, but if you have, you know, a bug that you've run into in the past or some, something that was not yet implemented that you would really like to see done, you could mention that. I guess we'll link to Nik's chat room for k is probably the place to go for that. But you can also contact me directly and ask for anything you want fixed in ngn/k.

00:07:58 [CH]

Awesome. So links to everything that Marshall just mentioned and the previous panelists mentioned will be in the top of the show notes. If you want to go check those out. Without further ado, we are going to introduce today's guest. He is the runaway guest for most number of appearances. I actually checked before the episode recording started. I don't think there's even anyone with four or three. I think the only other, there's a couple of people, Tally Beynon had like a two-parter, John Earnest, we had two episodes within sort of three. So I think there are a couple of folks that we've had on a couple of times, but Henry has been on, this will be his fifth time. I went and got the numbers. Not only is he our most recurring guest. He was our first guest, which I always forget about because it was such a long time ago now. So in 2021, he was on episode six, 2022, episode 18, 2023, episode 48 and episode 50. And today in 2024 is the fifth time. Three of those episodes were covering J903 and then they changed it to J9.4. And today, I think primarily we're going to be talking about updates that have happened over the last year to the J language, which will be in the J9.5. Potentially we might talk about some other stuff. I realized just in the last couple of minutes that we actually haven't had Henry on since our Tacit number five, which is was all about the train modifiers that I think in between sort of episode six and 18, which were Henry's first two appearances. They were mentioned in the first one. And then I thought that was really cool. I kind of poked around into it, but like hadn't mastered it. And anyways, now I still haven't mastered it, but I understand it a lot better. But we will, we will throw that to the end. First, I will throw it over to Henry. And I don't know if you want to give a brief introduction because you, you know, I assume the regular listeners definitely know who you are at this point. And then we can hop into sort of what the, what the most exciting things in J9.5 are over the past year.

00:09:54 [HR]

Yes. Well, I'm the main developer of the J engine, the actual interpreter, having picked that up from Roger Hui seven or eight years ago. We try to do a release every year. And that's part of why I've been on a couple of times, since you're kind enough to let me talk about the release. In 9.5, this was fairly light on features, but there are some interesting points in it. First of all, we support modular arithmetic now with an adverb that you say plus M dot, and now you've got modular ad. Previously, we had made a shot at doing that with some special combinations. You know, if you, if you have a modulus operator with addition and you do a certain form, maybe we can recognize what you're trying to do. But that just breaks when it comes to division, because division was defined to produce fractions. And you have to have some, something else in the language to say that what you really meant was modular division. So we, we added that and while we were at it, Cliff Reiter contributed a modular matrix inverse package. So you can now take the inverse of a matrix with a modulus. I don't know who would use that, but it's, it seems to be among our current J users, it appears in a lot of problems like Project Euler and the like. So it's more for people's enjoyment than for heavy use, although the performance is pretty good.

00:11:21 [CH]

I was going to say that. I'm not sure about the modular matrix inverse, but my first thought when I saw this was that back in my, you know, quote unquote competitive coding days, when you get past the beginner and maybe it shows up a little bit in the intermediate problems, but a lot of the more like higher level problems, especially in the combinatorial space, they ask you, you know, based on a bunch of stuff like, you know, calculate this number N, which is a, represents a combinations or ways to do things. But when, in order for the time complexity of the problem to be, you know, whatever, complicated enough that it's a tricky problem to solve, that N is going to be overflowing in N32 or N64. So they'll say modulus some crazy prime number so that it fits into one of those two integer widths. And you would always have to go and modify your algorithmic solution because obviously your binary operators are not modular. So like potentially this is going to be a great little trick now that with J you don't need to worry about that. You can just use this modular arithmetic and get the answer that you need without any extra work.

00:12:21 [HR]

Yeah, that was the idea for people who are doing that. And it works for extended integers too. So even if the modulus is bigger than it'll fit into it, it still works for you. The second thing we did and the, what really took the bulk of the work on the release was something that probably no, very few users will ever notice. It's like redoing your plumbing. It had to do with the primitives of a fetch [02] and amend. Fetch is where you pull a subarray out of an array. It's basically array indexing. Amend is when you modify a portion of an array. And you could say, well, you know, what is there to do about that? Well, the question becomes then, what parts of the interpreter need to be made fast? And when, you know, when is it time to rewrite something to go faster? You can't make everything fast. There's a notion of a tasteful balance. It's something that seems like it does a lot of work. You can let that take a little bit more time than something that feels like it doesn't do as much work. So when you're fetching from an array, well, if you're fetching from an array, the user presents you with a set of rows and columns he wants to fetch from. How do you do that? The simplest way, the most general way, the easiest way to code would be take the set of rows, pull them out of the array. And then for each of those rows, take the set of columns and pull the columns out of those rows. So you've actually done two fetches. You fetch the rows, then you fetch the columns. You get the same answer. But it's easy to code. You don't have to do anything special. It's just one loop. You're doing one thing repeatedly. The problem, the reason I was forced to revisit this is that we have a user who has arrays that are of the shape, the first axis is two and the other two axes are huge. So if you follow this procedure, even if what you want to get is just, let's say the arrays are two by 10,000 by 10,000 and you want to fetch one lousy atom. So you want to fetch first axis is one, the second axis is 100, third axis is 1,000. Following this procedure, you would first fetch one item from the first axis and the result of that is a 10,000 by 10,000 array. And then you would fetch one item from the second axis, resulting in a list of 10,000 numbers. And then you fetch the single number you want. So you've done what, 100 million, you've created 100 million words of intermediate result to produce one lousy word of output. This is offensive and it's very slow. But what you, so the question is, how do you fix this in an elegant fashion? How do you fix this so that you're doing an appropriate amount of processing? And generally, the more the user specifies, the more work I can do. But if what he's doing, if additional specifications reduce the amount of the final result, I don't want to produce these big intermediate values. We've had this problem ever from the beginning, but Roger had a pretty good solution to it. He did a special case. And instead of going one axis at a time, he went two axes at a time. So you first fetch from a square array and that worked. Nobody noticed for years, I think, until we had this user who had arrays of this particular shape.

00:16:32 [ML]

No, I don't think that's common in in most languages.

00:16:35 [HR]

Well, it's 10,000 times better than it could be, right? Yeah, you have to fetch an entire row when all you want is an atom. Anyway, so what I wanted to end up with was a data structure where rather than going from the front, I would go from the back and cycle through the last axis, the fastest changing axis, and accumulate only the amount of data that I needed. So I did that. It involved completely rewriting the fetch verb, but now it doesn't have any of those anomalous, bad performing cases. There were a couple of others. I don't know about other APLs, but Jay supports complementary indexing.

00:16:55 [ML]

No, I don't think that's common in most languages.

00:16:58 [HR]

Okay, so when I specify an axis, I can say, normally I say, give me number one or give me three, four, and five. And J can say, give me everything except the first and the last. Everything except zero and minus one. And the implementation of that was to turn that into the list of the indexes that are desired and then read those. And you can see, well, that's going to be bad. At the very least, I'm producing a large index list when the user only gave me a couple of numbers. If the array is 100,000 long, I have to produce 99,000 indexes of the index I actually want. So while I was at it, I fixed that so the complementary indexing doesn't have to produce the intermediate results it did before. So that was the first change. That was a matter of figuring out what an effective implementation would be and then rewriting it. Amend was a much trickier problem.

00:17:39 [CH]

Wait, before we move to amend, just so that potentially I and maybe one or two listeners that aren't familiar with fetch. So I went to NuVoc, took a look, and it's the left brace, aka curly bracket for those that...Different countries in the world refer to parentheses, brackets, and braces differently, but it's the curly bracket. And then colon, colon. And it's dyadic, so it takes like an integer or an array. And then...

00:18:06 [HR]

No, wait, no. Well, well, well. In that case, I use the wrong.

00:18:09 [ML]

I was wondering about that.

00:18:10 [HR]

I used the wrong name then it's not the, it's just left brace.

00:18:13 [ML]

It's just called From right?

00:18:16 [HR]

From, that's from. Okay, sorry. Okay. Yeah, from takes a subarray. Fetch does sort of the same thing, except it then opens it if the result's a box.

00:18:18 [CH]

Gotcha.

00:18:26 [ML]

Well, it can do that multiple times too, right?

00:18:28 [HR]

And it does it recursively, yes. I'm sorry. I considered saying left brace, but I thought people wouldn't understand that. I should have said...

00:18:36 [CH]

Yeah, I mean, it is better, I think, colloquially to use the names of them, but that's what I was...

00:18:41 [HR]

Yeah, it's from. It's the big one, from, not fetch.

00:18:44 [CH]

OK. The question that I was going to ask is what's the relationship between like APL's pick and BQN's pick and from? Are they the same thing under different names?

00:18:58 [AB]

No, not exactly.

00:19:00 [CH]

I heard not exactly from Adam, but I didn't hear what Marshall said.

00:19:05 [ML]

I said no. The question is, do you want to have another... Do you want to schedule another episode?

00:19:07 [CH]

All right. Let's defer the differences of from/fetch.

00:19:13 [ML]

Yeah, there are many, many minor variations. And also, I mean, most of these languages have multiple selection primitives. Right, yeah. And none of them exactly correspond to each other.

00:19:21 [CH]

All right. But in general, these come from the same bucket of trying to pull out an atom or element of an arbitrarily ranked array is what we're dealing with.

00:19:34 [ML]

So, yeah, J's from is closest to APL's bracket indexing, actually, or S quad is pretty much the same thing.

00:19:42 [AB]

No, the bracket indexing is closest.

00:19:44 [ML]

So it's selecting. You can select along each dimension independently.

00:19:49 [AB]

And the complementary indexing comes from the bracket indexing allowing this special syntax with a semicolon, the limited segments, one per axis. And APL has always allowed an empty segment. So you'd have consecutive semicolons or just at the end, there could be open brackets, semicolon, some number. So trying to make that into a function has been tried many times, including in J. And so there's the idea with the advent of enclosed or boxed arrays that you could have some boxed thing. So a boxed empty list would stand in for an empty segment in a list of boxes. So it's replacing this semicolon segmented square bracket with a vector of boxes. And then you need an empty box to signify an empty segment. And that in APL would mean all of them. But that can, of course, then be extended. Well, if a boxed nothing means all of them, then maybe boxed something means all but these, in which case you can understand boxed nothing to mean all but none of them. That's all. So that's how that comes, that history goes from APL's square brackets into J's complementary indexing. And the squad indexing function in APL does not have this complementary thing at all.

00:21:15 [CH]

We will definitely, this is an ArrayCast promise to the listener, we will definitely do an episode at some point because this actually just came up on ADSP the other day when we did our phone tag episode is that bracket indexing was an equivalent of like a parallel algorithm. But then at the end of the day, like I said, I really steer clear of this because it's a super practical thing. But when you're doing short one-liners, a lot of the times it looks less pretty than compared to some alternative. But anyways, stay tuned, listener, for a future episode on indexing, selecting, picking across all the array languages, what the differences are. But yeah, we'll table that for now and Bob's going to say something and then we'll move on to Adám.

00:21:57 [BT]

Well, I was going to say the other thing that J has is it's got ace, which is a colon, and that actually stands in for the empty box. So it actually becomes really nice to be able to just drop an a colon in and you're essentially just creating that little empty box.

00:22:12 [AB]

The the A then stands for all right.

00:22:15 [BT]

Sure, if you want.

00:22:15 [CH]

If you want.

00:22:17 [ML]

If you want. If I can interrupt this one more time, I was wondering, so obviously this is not going to be as good as the dedicated implementation you talked about, but J has these virtual slices, right? So did you try when you index just instead of actually copying out a 10,000 element row, taking a slice of that array? Is there a problem with that or does it?

00:22:38 [HR]

Yes, Marshall, I did that and I was not going to bring it up. But yeah, if the result of the indexing, however complicated it may be, is a contiguous set of the original array, the whole thing returns a virtual variable, which is a pointer to that part of the array. So no data has to be moved. And that turns out to be used quite a bit. You know, you fetch a couple of contiguous rows of a table, it doesn't have to make a copy of it.

00:23:10 [ML]

All right, yeah.

00:23:11 [HR]

Yeah, well, if you hear in this future talk about array indexing, you'll really want to be talking about the amend. Amend was the second thing that had to be rewritten.

00:23:22 [ML]

I don't know, I think that pushes us up to a two-part episode.

00:23:27 [HR]

You're probably right, because amend is a real challenge in J. And I don't know if you've got brackets syntax, it's not so hard to, well, it's still got to be a challenge in APL2. The problem is that really alone among all the primitives, amend requires three arguments. The thing to be amended, the new stuff to put into it, and the places where the new stuff should go. I guess you could maybe box those and put them into one argument, but the decision that was made was that amend will be implemented as an adverb. [03] So, the places you want to modify are the left argument of the adverb. The symbol is the right brace, so you'd say, "567 right brace." And that together, it says, "Okay, this creates a verb that modifies items five, six, and seven." That verb can then take two arguments, the right argument to be modified, the left argument with the stuff to go into items five, six, and seven, and the operation does its job. Where this becomes problematic is when rank gets involved. This amend operation can be applied with a rank, but that rank only applies to the arguments, to the ultimate right and left arguments, the data to be modified and the data to be installed. The indexes are stuck, they're hidden inside the compound, which was made up of 567 brace. So, the question is, how do you produce a primitive that gives you the full complexity that you want to be able to modify different parts of the array with different data? How would you make that work? Well, it's tough. And the language designers made a mistake, and I think that should lead us to be humble, because if Ken Iverson and Roger Hui can make a mistake, we all are going to make mistakes. As it turns out, what they decided to do was, they said, "Okay, the selector, the thing that goes to the left of the brace, the selector can specify any, it can be a general selector. It can pull anything it wants to out of the array. The data that gets pulled out of the array, the system will remember where it came from and put the new values in the places where the original data came from." That turns out to be very hard to implement efficiently. As you can see, in general, I may have, the trouble is that there's a single selector, a single box can select an entire subarray. And if I allow the selector to be anything, it can be a list of these boxes. So, the specification of what is to be modified can be a set of very shaped stuff scattered all around the array. This poses a problem when it comes time to figuring out where, what places match up with the left operator that has the new values. In the end, what the implementation did was, it handled a couple of special cases reasonably well. And then in general, it created an index list. It first created a matrix that has the index list of every atom in the array. And then it fetches from that to find out what the atoms are that are to be modified. And then it uses that to control what data is stored. And this is where it falls down with appropriate use of time. Because, let's say you've got an array of a million characters. Maybe it's a two by two array of a thousand by thousand. The index list would be a two by thousand by thousand array, or a thousand by thousand by two array to identify the parts that are going to be modified. So again, even if I only want to modify one lousy character, I might have to create two million index numbers. This is just a problem. The only way I could figure out to solve it is just say, just don't allow that. So, with some trepidation, I instituted the restriction that in an amend, there can only be one block specified. So, only one box is allowed, right? That was the real problem. These multiple boxes that produce multiple discontiguous regions, they're not allowed. You can still have discontiguous regions, but you only can have one box specifying the region. That solves most of the problem, but it doesn't solve the problem of scatter amends, which is a very useful feature where, in this case, the indices for amendment are index lists of individual cells, and we just go down the cells one by one and write new data into them. Well, anyway, to cut to the chase, it turns out that there were enough degrees of freedom in the language between singly boxed arrays, doubly boxed arrays, and triply boxed arrays that I could support scatter writes and any individual box, any individual specification of a region, and do it very efficiently, similar to the way it was done for ROM, rather than it works from the bottom up and modifies only the cells that need to be modified. The problem was that this is an incompatible change. So, I recognized the two special cases that I thought were most likely, and that is cases where you have multiple boxes, but they're really all specifying the same shape of region. And I put it out there in the beta and said, "Here it is. If you've got any problems with this, we'll talk about how to solve them." And much to my surprise, I think, it turns out nobody in the whole J world ever did any of these pathological cases that took so much effort to support, because there was never any complaint about it.

00:29:24 [AB]

But is that policy in J about backwards compatibility at all? Because I've seen a few versions, minor versions even, or if they can even be called that anymore.

00:29:35 [HR]

Well, we've tried to get back to the minor/major idea. There were changes that were not backward compatible. I mean, we have users like everybody else. We try to stay as compatible as possible. If we think that the language was incorrectly specified, we'll make a change. Here, that was my opinion here. The workaround for it is very simple.

00:29:57 [AB]

Sure, but it still means still breaking change. And you say the specification of the language was wrong. It's not that there was some ambiguity or some misunderstanding or something. You're saying it wasn't good design, but it was still design.

00:30:13 [HR]

Right. It was a design that could not be implemented efficiently.

00:30:17 [AB]

And you say that nobody in the J world, I mean, I'm very impressed. Everybody in the J world immediately reads and uses beta versions and gets back to you. Either the J world is small or you have some very impressive network connections there.

00:30:36 [BT]

Well, I don't think Henry's relying on everybody immediately responding, but this happens over a year, right? There's a year between releases.

00:30:44 [AB]

I mean, maybe that's--

00:30:47 [HR]

No, Tom is absolutely right. There are plenty of people who just won't use a beta version. And if they're using some code that we've broken, then we've caused a problem.

00:31:00 [AB]

In Dyalog, [04] which is what I know about, of course, we have customers for whom it's very expensive to upgrade. They need to test out lots of stuff, so they don't go from version to version. And they might do extensive testing and catch things like this, but they can easily go five major versions up in a single upgrade. And it takes years and years for them to prepare. And by the time they finally manage to move from their old version to a new version, the new version isn't even supported anymore. That's just how it is. And it's important because we have people that rely on this to deliver food to schoolchildren. I think of the kids. We have customers that rely on this to deliver medical information about patients that need to be accessed real-time in emergency situations. You can't have people get blood transfusions of the wrong type because somebody messed up something or changed the specification of something. And Dyalog has a very strong--

00:32:03 [HR]

I think the bigger problem is not those people because if they're going to test it for a year before they switch versions, they're going to find this incompatibility. It's more the casual user who isn't going to test it for a year, but just finds out that his code doesn't work.

00:32:21 [AB]

Maybe if they've got code coverage, but lots of old systems out there that have well tested from real use. Are you really going to rely on the tests being comprehensive enough that they will catch everything? Even some edge case of some primitive changes? We had an example. I remember my father showing me this. You know about quad IO in APL. You can index from zero, index from one, which was fixed at zero for J. And then there's the IOTA primitive, which is like I. But it's different when you have a vector argument from what you have got in J. In APL, the extension takes a shape of an array and it gives you an index of an array of that shape. So it returns an array of that shape and every element is a box enclosure, which is a vector that is the indices for that element, rather than just a reveled product of the links. So then we have, or many APLs have like a shortcut for the empty vector, empty numeric vector. It's called Zilda. It looks like a zero that's overprinted with a tilde through it. And here's an interesting thing then. So what is IOTA Zilda or IOTA IOTA zero if you want? Well, if IOTA takes the shape of an array and returns an array of that shape where every element have the indices for that position, then an array of shape nothing, shape empty list, that's a scalar. And what are all the valid indices into a scalar? Well, there can only be one. It only has one element. And that's an empty list because you don't need any indices to pinpoint a specific element. So the correct result, logically, is an enclosed boxed empty list. And that's a collection. So it's a collection of all the arranged along zero dimensions, a collection of all the indices into a scalar, into a single element. However, traditionally, at least some APLs, I was using APL plus and Dyalog also did this, they returned the index origin as a scalar when you wrote IOTA Zilda, as if you could index into a scalar as if it was a vector somehow, but it was returned a scalar, not a vector. So it was a convenient way, rather than writing three characters, quad IO, to get the index origin, you could write two characters, IOTA Zilda, save a character or something. I don't know. I don't know why they did it like this, but clearly nobody would do that, right? You don't do CodeGolf in production system. You don't write IOTA Zilda instead of quad IO. It's just obscure. Surely when somebody noticed that this is the wrong result, we can just fix it. And so we fixed it and all hell broke loose.

00:34:58 [HR]

Yeah, I agree. That would be a big change to make. I wouldn't make that change. I should point out that this change to amend was, it would not produce a wrong result. It just makes some things illegal that used to be legal, so that if anybody's relying on the feature, they're going to hit an error.

00:35:19 [AB]

Yeah, okay. I guess that's slightly safer to hit an error.

00:35:22 [HR]

I mean, they won't like it, but...

00:35:24 [ML]

Yeah, and chances are if your code uses it once, it uses it all the time. I mean, it would be, I guess it's conceivable that you have code that usually uses one box, but sometimes adds another. But that would be pretty strange. So I would think that if you're going to have a problem with this, your program is just going to crash right away.

00:35:41 [HR]

Yeah, that's my thought too.

00:35:43 [AB]

I'm not really arguing with you about this particular case. I'm just a bit surprised about what I've observed about J's stability when it comes to the language. And certainly at Dyalog that, we would be very, very scared to change anything. Morten Kronborg occasionally mentions, our CTO, he occasionally mentions wanting to change a primitive.

00:36:04 [HR]

Yeah, I think that's a matter of judgment. It depends on if you're pretty sure that most of your users haven't been born yet, it makes you want to keep the language as pure as possible. The best thing is not to make a mistake in definition, but it's just going to happen. There are going to be times when you decide that it needs to be done differently. There was a question about branch cuts and some complex functions. And after consideration, it was pointed out that the branch cuts we use for arctangent or whatever it was, were just not right. Can I fix that? I mean, it's the same argument that you've got. It's just that it's much less likely that it's going to make somebody deliver the milk wrong. But it's still a bug. The question is, is it a bug that needs to be fixed or something you have to keep compatibility with?

00:36:52 [BT]

Well, and the user always has a choice whether they want to stay with what they've been using, balanced against what the new features would be.

00:36:58 [AB]

Yeah, I would argue with that. Sometimes the choice goes away.

00:37:01 [ML]

Eventually, it usually does.

00:37:03 [HR]

Yeah, we don't want to encourage that either.

00:37:05 [AB]

There can be many reasons why you can't just stay, right? If things happen with hardware, things happen with operating systems where you can't, right? Let's say some Mac user is using some legacy thing that only was compiled on 32-bit and just doesn't work on 64-bit. All of a sudden...

00:37:22 [ML]

Well, being open source does help a lot.

00:37:24 [AB]

But it's not trivial to convert, right? There have been things that...

00:37:27 [ML]

Yeah, I mean, if the build system doesn't work anymore, then you have to either move up to the current source and fix the problems in the source that you're having or fix the build system in the old one.

00:37:40 [AB]

Not everybody can or can afford that. So I don't think it's a very viable thing to just say, "Well, you don't want to upgrade. Stay in an old version." And what we do at Dyalog, shh, don't tell anyone, is we provide compatibility settings instead. So not the famous or infamous Quad ML. But if we have a large customer that, or even a smaller one, that relies on something that we change, then we can add either an option specifically for them or that's secret from everybody else because we don't want to promote that kind of usage.

00:38:13 [HR]

Well, you know, we might do that too. That's not something we would talk about.

00:38:17 [AB]

We don't know, right? Or we can provide some option that is known out there saying, "You can set this environment variable," and then it behaves slightly different. And very interesting, you mentioned the thing about complex circular functions. And we had an interesting case where we added complex numbers to Dyalog. Then most things were just fine. One thing did change, and there used to be kind of like an easter egg in old APL that if you took the cube root of a negative number, it has a real value. So it would return that, but it's not the primary value. So, I mean, more than causes an easter egg, at least they had that, because who would actually use that?

00:38:58 [ML]

Yeah, I mean, I don't really agree with that characterization. It's like, you know, you're programming in a language that only has real numbers, and you ask, "What's the cube root of minus eight?" Well, it returning the number that cubed is going to give you minus eight is not really that crazy. I mean, the concept of the primary root doesn't come from mathematics. It's just a convention that makes sense in order to have a continuous root function that's continuous in the…

00:39:25 [HR]

I feel strongly about this, because I had to teach Algebra 2 in high school. And in Algebra 2, what numbers do is what the TI calculator says it does.

00:39:35 [AB]

Hey, that's not obvious which one they change from version to version.

00:39:36 [HR]

Yeah, but the square root of minus eight is minus two on a TI calculator, unless the numbers… If it's the three, you can say minus eight to the one-third, you'll get minus two. If the one-third gets a little bit flaky, it's not… it has to be a complex number.

00:39:58 [AB]

That's not nice. But actually, so big deal. Who even uses complex numbers for actual money-making purposes? And I don't think anybody…

00:40:07 [HR]

If you're doing fast Fourier transforms, you will.

00:40:10 [AB]

Okay, but notice we added complex numbers. So nothing changed for people using complex numbers, because nobody used them, because they didn't exist. And nobody who was using real numbers other than for fun would probably rely on these roots of negative numbers giving real results instead of erroring. I haven't heard about anybody actually hitting this. However, what did happen is an entirely different part of the language. We had the system function, the utility thing, that is used to validate input. And so it would take things that are supposed to be numbers, and it will tell you whether or not it represents a valid number, a character vector, whether it represents valid numbers. And then it will also give you the actual number that it represents. And if people input garbage, ABC, whatever, they would tell you this is not a valid number. And if people input garbage like 1J2, it would tell you this is not a valid number. And one day, it started saying that 1J2 is a valid number, because 1J2 is how we represent the complex number. And that was probably a mistake. That validation function should probably have stayed real, or at least had an option of some sort to say also do complex, not just one day start accepting input that wouldn't be valid. And if you have somebody who's typing into a form how much money is in the account or some other thing, and they by mistake put in a J there, and you get a complex value, well, you can imagine things getting complex real quick.

00:41:39 [ML]

Yeah, so I guess this sort of thing is why the number parsing function in BQN, you might ask, why is there not just a simple function to parse a number? What we have is a system function called parseFloat. And that's specified out, we picked a particular syntax that we want to use for floating point numbers, which is not the same as BQN syntax, because it uses a minus instead of the high minus sign. And there we can just say, well, yeah, we picked what we're going to do, and that's never going to change. So we think this is a reasonable choice of floating point numbers. If you need a modification, maybe you can do something to your input or something like that. And then if we have parseComplex, then we can add that. So naming things, of course, is one of the hard problems of computer science, but there's one tip for you.

00:42:25 [AB]

Yeah, obviously, it would have been trivial to do this right. Yeah, I don't think it was by design that it's called quad VFI, the system function, it changed. I think it was because it hooks into the interpreter itself, parsing numbers, and the interpreter all of a sudden started understanding things as numbers that weren't numbers before. That kind of oversight is easy to make. I'm pretty sure if somebody had thought about it, they would have not agreed that that was a good decision to make. Now, of course, we can't really change it back. So what do you do? You have to provide some options for customers to make sure that data entry in forms doesn't accept complex numbers. And that's why I'm asking, really, come back to what is the policy for J? Or maybe there isn't one. Maybe there should be one.

00:43:09 [HR]

No, there is one. We try not to break old code. This was a case where if it was going to break old code, it would break it. It wouldn't produce invalid results, and the implementation is much better.

00:43:21 [AB]

I remember there were even some primitives that were, I remember, like one of those with a letter and a dot or something, that were removed from one version and then came back meaning something else. Yes. And that seems dangerous. If somebody skips a version, then they just get different results all of a sudden.

00:43:39 [HR]

Yeah. Yeah, when we took Calculus out, we decided that that was a blunder. It goes back 20 years to when J software, I think, was going to try to compete with Mathematica or something like that. I'm not exactly sure what the business model was, but the idea was we can do symbolic mathematics. [05] That's cute, but C is not the right language to program that. Roger realized that pretty early and froze the development, but we decided, you know, somebody might be able to make a decent package of this if they want to. So we decommitted the primitives and replaced them with scripts to do the same thing. And now, in fact, I think Jan-Peter Jakobs is working on, has been submitting improvements to that package. So that worked out the way we wanted it to. We shifted it from something that would not be worked on to something that can be worked on by the users.

00:44:33 [BT]

And I think the one, it was the T dot, which was the Taylor series, right? I don't know that that's been replaced at all. I know Raoul talks a lot about going in and writing it out.

00:44:42 [ML]

It's something about threads, isn't it?

00:44:44 [HR]

Now it's threads, yes.

00:44:45 [CH]

I think in general, this topic of API, ABI evolution is like a super important one, but it always comes down to tradeoffs. Like, obviously, guaranteeing 100% backwards compatibility is great in the sense that everything Adám's talking about. If you've got clients that are depending on this behavior, you know, everyone's heard of Hiram's law. If there's some...

00:45:09 [ML]

I mean, there's a few degrees of this. I mean, you don't guarantee that the bugs are going to stay in unless- Literally sometimes that is what the guarantee- There's one way to do that and that's to-

00:45:21 [HR]

No, yeah. I remember when I was a kid, one of my friends was working at an office. He found a bug in the square root routine. This is a COBOL shop. Somebody had written a subroutine to this square root. He gave the wrong value if the input was less than one. But should you fix it? Exactly. It's a hard question because it's used for calculating when to order material. People might be relying on that bug. Even something as blatant as that, it's not obvious whether you should change it or not.

00:45:52 [CH]

Literally in C++ land, they talk about bug for bug compatibility. Like 100 percent backwards compatibility means bug for bug. Like, yes, you can find a bug, but if you have guaranteed in a standard that you have 100 percent backwards compatibility, technically that means bug for bug. Across because we've got GCC, Clang, and MSVC, and then there's a plethora of other less well-known C++ compilers like Intel's, ICC, etc. They're different front ends. I've talked to compiler vendors that they have different bugs in each of the MSVC, Clang, and they have to match. They have different bugs implemented for each of the different compilers. It sounds absolutely nuts, but that is a part of the trade-off. Sure, you're guaranteeing that you're not going to break things, but you're also guaranteeing that you are codifying your mistakes that you make into stone and there has to be some path or migratory. We're going to unit test and if we are going to fix something, and there's been stories where in C++, like in C++ 11, they changed the implementation of std string from copy on write to not be copy on write. It broke the world. It broke the world and to this day, over a decade later at committee meetings, people talk about how the implications of like, "Oh, we're just changing the implementation. We're not actually doing anything at an API level." But people find a way to rely on stuff and then it breaks the world. The point being is that there are trade-offs at the end of the day, and it's a very hard problem to address. I think there's merits for both. I don't use Uiua for anything right now because currently they're changing everything under the sun from version to version to version to version. I think that's fantastic. I wish more languages would do what Uiua is doing and that they're experimenting. They're throwing spaghetti at the wall. They're throwing stuff in, taking it out the next version. It obviously is not a technology you're currently going to build a company on top of because that would be nuts. It's like, "Yo, we're going to go take this car, this new competitor to Tesla." They keep turning and taking the wheels off and calling back everything while we're in the middle of our road trip, but it'll be fine. No one does that. But it still means that probably the language that Uiua ends up at is going to have polished a bunch of stuff that they were experimenting with and they're going to end up at a better product versus not being able to do that. It's not to say that one's better, one's worse. It's just that there's trade-offs.

00:48:26 [AB]

But every language started like that, I'm sure. BQN [06] was also very experimental. APL was also very experimental once upon a time.

00:48:34 [ML]

No, I think the total number of changes in BQN that have ever been made is approximately one Uiua version since it was even written down since before it was public. If you started the public release, there are three or four added primitives, probably, character arithmetic, but the backward compatibility breaks are-

00:48:56 [AB]

After it was published, right? So the question is, but BQN absolutely did have an experimental-

00:49:02 [ML]

Well, after anybody other than me was aware of this language.

00:49:06 [AB]

Well, when it was an internal Dyalog project and wasn't called BQN yet, and there was all kinds of stuff.

00:49:15 [ML]

Yeah, no, but the first you heard the name BQN was when I went on the forum to talk about it and announce it publicly, correct?

00:49:22 [AB]

Yeah, I think so. But it doesn't matter what the name began. It was that idea of the language and how it developed. It was very much in flux, right? Talking about what kind of symbol is the whole thing about the superscripts.

00:49:34 [ML]

All right. I mean, if you're saying that there was a design period for the language, yeah, but it was not-

00:49:39 [AB]

And things went back and forth and various pairings of primitives. And even APL itself was like that. If you look at the early Iverson notation, every paper he came up with was somewhat different. With the same general thing, but it was somewhat different from the other ones. You can't just take one and read that and take the specification of the language and read another paper. It's not compatible with each other. Once they made an actual, I mean, there might even, I'm sure there were things back and forth, even when it was an internal thing at IBM before it was ever published. But once they published it, it became generally available on the 360, then it was set in stone and very little has changed. I'm always surprised that IBM actually changed one thing going from old APL to APL2. What's stronger, square bracket indexing or stranding? And that did break stuff when people were migrating. So I think, and J as well, right? Jay started off as APL backslash question mark and things changed. But the question is, do you stop that phase or can you keep going? How much are you allowed to change? What about bugs, bug for bug compatibility? I would love to change the definition of APL's monadic iota on a one element vector to be different than it is today. Can't do that. It would break so much.

00:51:04 [HR]

You can't do that.

00:51:06 [AB]

Yeah. Dyadic grade is a bit of a weird thing. The encode, that's represent primitive. Both in APL and J has a very strange behavior with a scalar as left argument. It just chops. It pretends the left argument is a one element vector, which gives you a useless result. It gives you a modulus. If you mean modulus, do modulus. Don't do a represent. The only difference would be comparison tolerance. But seriously, if you're relying on this, changing the primitive in order to get a modulus with zero comparison tolerance and fuzz, then you're just trying to make it obscure. I would love to change that, for example. Can you do that kind of thing?

00:51:48 [HR]

Yeah, we would agree. You can't change that. Similar to your Iota, the example, if you open an empty boxed list, that's a special case. It doesn't follow the rules for fills. It's the only example in the whole language that doesn't follow the fill rules. I'd like to change that.

00:52:08 [AB]

Here's a fun one that Dyalog changed. I don't know why they changed. But the change, it became different from all the other APL implementations that I know of. APL has strong prototypes of types in empty arrays. So the empty arrays remember what they were before they were empty. And so there's a fun case. If you take the empty character vector and concatenate it with an empty numeric vector, or vice versa, you have to have an empty vector come back out. But which type does it have? Numeric or character. So you have to choose either right or left. Should people rely on this? No.

00:52:47 [HR]

In J, that's well specified. There's a priority rule for those types. And we pick according to the rule. Anyway, let me continue on with the release. We've got the bulk of the work was this rewrite to "from" and "amend" with a happy ending, I believe. Last time I was on here, last year, we talked a little bit about structural tool or structural under, [07] which is for the listeners. Under where we in J speak, we would say U under V means to V, then do U and then apply the inverse of V. So undo the mathematically similarity transformations or, you know, it's a change of point of view. And the question is, can you extend that to cases where V doesn't actually have an inverse? Well, I mean, you can, but can you usefully extend it to cases where it doesn't have an inverse? And those I take it are called structural unders. That right?

00:53:50 [ML]

Well, I mean, that's for particularly structural functions that discard some elements.

00:53:55 [AB]

More like a selective.

00:53:56 [ML]

Yeah. I mean, that's one mechanism of extension there. It doesn't ... [sentence left incomplete].

00:54:01 [HR]

Anyway, all you're talking about last year, J didn't support any of those. In J 9.4, if the inverse of V didn't exist, you were out of luck. You couldn't do that. And it was pointed out in this podcast that there are some places where it might be useful to define those inverses. So I looked into it and I found that this is a really tough problem. I think in general, there would need to be a different symbol for "under" when it's using a true inverse versus when it's using a pseudo inverse. But I didn't want to go there. So what I've dipped my toe in is using the regular "under" symbol, but giving special definitions for special cases where no real inverse could possibly be defined. And so far, there have been only two of those cases that have come up. One is a case where the operation is the "ravel" operator (comma), [which] says destroy whatever the shape was and turn the object into a list. Now, you can't do that. You've destroyed the shape, it's gone. So clearly, this can't be meaningful if you're looking for a true inverse. But in the context of "under", it would be reasonable to say, well, really, if I started with an argument Y and the operation is destroy the shape of Y and replace it with a list, undoing that would be taking whatever list comes out and putting it back into the shape of Y. That's reasonable. It doesn't conflict with any prior definition of anything. So we implemented that. You can use comma, you "ravel" in file if you have whatever name you want to use for it. You can remove the structure from an argument and have it put back as the inverse. And more importantly, found a second example: is if you do a selection from an array and you want to undo that, the definition now is store back into the array. So "amend" the selected elements is used as the inverse for "take" the selected elements. And that turns out to use that quite a bit lately. It's a useful function. And also it gives the advantage for completely operating in place. So if you want to add one under selecting the first two items, it will just add one directly into those two items and it won't copy anything. No copies of data need to be done. It can be done completely in place.

00:56:44 [ML]

It'll pull out the first two items, right? So if you were taking 200, you might get it? Oh, so it's got some special recognition for the one plus thing.

00:56:48 [HR]

No. What's special in the language (the implementation) is that blocks are marked as being eligible for in place operation.

00:56:53 [ML]

Oh, so this is used for rank two, isn't it?

00:57:05 [AB]

But what if you overflow? What if you have [127, 127, 127], and then you try to add one to the second element?

00:57:14 [HR]

These are virtual blocks.

00:57:16 [AB]

Surely it has to rewrite the entire array. I'm sure it works, but I'm saying, but you can't always do it in place. Sometimes you have to ... [sentence left incomplete].

00:57:22 [HR]

Oh, right.

00:57:23 [AB]

Okay. Whatever the types might be, whatever, I mean.

00:57:26 [HR]

Yeah. You can overflow an integer. If it overflows an integer, it recopies the array. The way it works is, it creates a virtual block for the region that was selected, marks that block as eligible for in placing, sends that off to the verb, and the verb (say it's add one, or maybe translate by fetching from some other array), if that verb returns the same original block that's been modified, the operation says: "oh yeah, it happened in place; I'm done." If it returns some other block, then that result has to be copied in. The only special code is in the dual thing that knows that it's taking the inverse of the "from" operation. So it's highly efficient and useful for these big arrays that are being modified.

00:58:16 [ML]

Yeah. And so I'm glad I asked about whether "from" returned slices. So I gather that "from" is going to inspect its left argument and figure out whether it's a bunch of contiguous indices. [Henry agrees]. Yeah. So that's pretty neat.

00:58:28 [HR]

It gets tricky if there's more than one axis, but the leading axes select only one element. So it's possible that you can get a virtual result even from a multidimensional "from". But anyway, with the rewrite, it handles all that properly. That's very useful.

00:58:45 [AB]

Henry, you mentioned using "from" to modify the first few elements, but surely it would be more natural to use "take" to select the first few. So I want to add one to the first three. Surely I would want to use "one plus under three take" or something like that.

00:59:04 [HR]

Yes. Okay, you're right about that. The problem is that in the context of "under", "take" has an inverse.

00:59:14 [ML]

The inverse is known as "mistake" [others laugh]

00:59:17 [AB]

Yes [chuckles]. Yes, it pads with fill elements to counteract the taking, right?

00:59:25 [ML]

No, because it doesn't know how long it's supposed to be. "Drop" does.

00:59:29 [AB]

Oh, you're right. Yes.

00:59:31 [ML]

I think it just returns the same thing.

00:59:32 [HR]

Yeah. I forget what it is. It might or might not be well-defined or properly defined, but it has a defined inverse. And that's an example of one of those things that we're not going to change. You know, we would not change that because ... [sentence left incomplete]

00:59:47 [AB]

You need two different conjunctions for the structural "under", because you have a compatibility problem with the old meaning because you have inverses for ... [sentence left incomplete]

00:59:56 [HR]

Right.

00:59:57 [AB]

...interesting things.

00:59:59 [HR]

Exactly. That's where I came down to. If you want to try to generalize this, you need to have two different symbols. The cases I've found that are interesting so far are cases where the inverse really isn't defined. So I'm okay with that.

01:00:17 [AB]

Yeah, there's "ravel" , but you could also have the "raze", right? [08] That's like a super "ravel". We destroy all the structure.

01:00:25 [ML]

That should be doable.

01:00:26 [HR]

I don't know. Well, if we could define the inverse of that. Again, that may have a defined inverse for all I know. But anyway, these first couple of examples of structural "under" are interesting. They show the depth of the problem, and they also show that there are cases where it can be very beneficial.

01:00:43 [AB]

So just in case it was said and I am lost in the sauce, was the inverse of "take" actually ever mentioned? Or did we just say it exists?

01:01:03 [AB]

It is. The inverse of "take" can just be an identity function. And [for the] inverse of "drop", you can pad with prototypes. But [for the] inverse of "take", so think about it. So I'm taking the first three ... [sentence left incomplete].

01:01:09 [CH]

Trust me, I'm thinking about it. I just am not understanding it [laughs].

01:01:11 [AB]

Yeah. So then I want to say, what argument could I give such that taking the first three gives this result? Well, the same thing. If something has length three (which it has after you take three), then if you take the first three, you get the same thing. So a proper inverse of "take" is the identity function, but it's not a useful inverse.

01:01:33 [CH]

Am I the only one that didn't understand that?

01:01:35 [HR]

Well, I can tell you how it's defined in J. I've just looked it up. The inverse of "take 3" is not defined.

01:01:41 [ML]

Yeah, I was just trying it and got an error.

01:01:44 [HR]

If you just do the monad, just take the head, It is defined. It adds an axis.

01:01:51 [ML]

Yeah, yeah, that makes sense.

01:01:52 [AB]

Okay, that makes sense. But it didn't have to, right? [Henry doesn't sound convinced] Oh, no, no. Because the head does have to reduce. It's a reduction, right?

01:02:03 [HR]

Yeah, "head" reduces the rank.

01:02:05 [ML]

Yeah, so I mean, it could add fill elements for no reason.

01:02:08 [HR]

Anyway, so you're right. "Take 3" would be a candidate for ... [sentence left incomplete]

01:02:12 [ML]

So J does not implement mistake [Henry chuckles]!

01:02:15 [AB]

But Dyalog does.

01:02:18 [HR]

"Take 3" would be a candidate for structural under [Marshall agrees].

01:02:19 [AB]

And "head" and "tail" are also, right?

01:02:23 [HR]

Well, not "head", because "head" already has an inverse. So I can't change that inverse, no matter how much I'd like to [Adám agrees]. And I actually don't like to very much. I think that inverse is occasionally useful.

01:02:37 [AB]

Yeah, so we have even worse problems with Dyalog, trying to do structural "under". It has to be separate from the "under" or instead of it. We have an operator that's ... [sentence left incomplete]

01:02:49 [ML]

Well, so you don't yet have the worst problems because you haven't implemented the regular "under".

01:02:56 [AB]

Yeah [chuckles] that's true. That's the thing. And then people say: "when is it coming? when is it coming?" It's not that simple. You don't want to make mistakes like this. But when you spoke about the "amend" and how it works, we have a dyadic operator. So like a conjunction in Dyalog (the @ operator). And I quite often use that. And it basically implements part of the functionality you get from structural "under", in that it allows you to do operations at a certain subset of the array itself. And things like setting the first bit of a boolean vector to one is very convenient to do: "1@1".

01:03:36 [HR]

Oh, yeah, that's useful.

01:03:38 [AB]

And there's something you could do with structural under as "one under first". Whatever your function is for "first" or "head".

01:03:47 [HR]

Well, anyway, so the stuff we've talked about so far got us up to about September. We didn't release until December. Part of that was to leave a lot of time for people to check out the changes to "amend". But also we were busy on functions that we did not announce in 9.5, but did announce in the beta for 9.6. And those are new numeric precisions that we support. Since you guys have worked with languages that support different precisions of numbers, I wanted to try to learn from you. Now we currently support double-double floats; so 104-bit floating point.

01:04:28 [AB]

What?

01:04:29 [HR]

As a native precision.

01:04:30 [AB]

Hold on. Not 104, surely?

01:04:30 [ML]

That sounds lower than what I've heard. I've heard 105 or 106, but 104 might be the actual, yeah.

01:04:37 [HR]

You could argue 105, but you don't have complete ... [sentence left incomplete]. Because the representation of a number has to be in a canonical form, so that comparisons for equality were easy to do [Marshall agrees]. So you don't have a free hand on the second hidden bit. Maybe 105, perhaps. 53 bits on the top part and 52 in the low part.

01:05:00 [AB]

Oh, I think I now understand. Maybe I'm just the only one that doesn't know about low-level stuff, but we should introduce maybe this to the listener [chuckles].

01:05:07 [ML]

So, all that your number is, is two regular floating point numbers, which represents the exact sum of those floating point numbers. And because floating point has an exponent, one of the numbers is going to have a higher exponent and the other one's going to have a lower. So that second number pretty much just gives you extra bits.

01:05:30 [HR]

Yeah, you might be thinking of IEEE quad precision. That's not what this is. IEEE quad precision, has a 15-bit exponent and would give you more than 105 bits, but there's no native hardware support for that anywhere, I don't think. The double-double [is] exactly as Marshall said. You got a high part and a low part. That's supported reasonably well, particularly when you have the fuse "multiply accumulate" instructions. The implementation of that is pretty fast. And so that's what we use for an extended floating point. That's just for those cases where you just need 30 decimal digits of accuracy and there's no substitute for it. We have users in that case, so we're supporting them. But I'm more interested in your experience with shorter integral forms. We also introduced 4-byte integer and 2-byte integer and maybe 1-byte integer is coming. I mean, I've got a slot reserved for it. But how do you handle conversions between those forms?

01:06:40 [ML]

So what's the definition here? Are these types that the user can see? Like if you add two 2-byte integers together, will they wrap at the 2-byte boundary, or are they just implementation optimization?

01:06:52 [HR]

Well, this is a beta, so it's [chuckles] up in the air. But the way it's implemented currently, there are storage formats. So you have signed 2-byte integers. When you create one, the order of priority is, lowest priority is the standard J number. A J number is boolean or integer or float or complex; whatever it needs to be. Above that in priority is 2-byte integer and above that is 4-byte integer. So if I add a number to a 2-byte integer, I'll get a 2-byte integer. If I add two 2-byte integers, I'll get a 2-byte integer.

01:07:32 [ML]

So if you add 2-byte to 4-byte, then it increases the precision instead of squashing it down.

01:07:37 [HR]

Right. 4-byte is the highest precision. If you overflow on adding to a 2-byte integer, you get a domain error. The idea is if you specify the storage format, you really care for some reason [Marshall agrees]. Probably you've got billions of them and you don't want to just double the space without at least knowing that you did it.

01:07:58 [ML]

Yeah. So I haven't worked with this sort of thing. What Dyalog and BQN do is they just tell you numbers are numbers. And for Dyalog, you can set whether the number is a regular binary float or a decimal float. [In] BQN, everything we've got currently just says floats. It could say complex, but we haven't implemented that. And then under the hood, it optimizes that as 32-bit or 16 or 8-bit integers when it finds out that it can. So yeah, I don't really know much about how you deal with all these different types if the user is specifying them. So NumPy and Julia, I know, have those. And [for] k, I know you can do those in a k database. I don't know ... [sentence left incomplete]. Like most of the open source k implementations don't give you a lot of numeric types. So I don't know how Q and K4 really handle it. You can find the documentation on these, I'm sure, but not on how they're implemented. So yeah, I know a few languages do this, but it's not something I've worked with.

01:09:09 [HR]

Yeah, I'm implementing them for a use case for a user where the variables are pretty big. Big enough that they're close to blowing out the caches.

01:09:12 [ML]

Yeah, so they're trying to control the memory use.

01:09:15 [HR]

Right. And a savings of factor of 4 or factor of 2 would be worth reaching for. But I've got no experience on how they're going to be used. I expect most of the time they'll almost be read-only [Marshall agrees]. They're written so as to save space and will be converted to numeric for actual use.

01:09:32 [ML]

Yeah, and I mean, I guess you've already got all the different selection functions and stuff work on any width. So as long as you're not going to do arithmetic on them, you can move them around however you want, right?

01:09:42 [HR]

Right. Yeah. the "i." family and all the structural changes, reshape and copy and all that jazz.

01:09:51 [BT]

Would that be a case where you might want to use an immutable array? So in a case, you'd make it read-only?

01:09:56 [HR]

We don't have that in J.

01:09:56 [ML]

Well, I mean, the semantics are always that it's immutable. Like if you amend the array and you've got a copy saved somewhere else, it's going to do a copy to avoid changing your saved value of that array.

01:10:15 [CH]

Potentially, what we should be doing now is turning this to the listener. I mean, we've got, I don't know if it's a thousand, but we've got in the hundreds for sure of technically minded folks that are listeners here. So if any of you have had experience in this area, reach out to either us directly and we can pass it along to Henry, or I'm not sure, Henry, if you want to plug your email and they can just reach out to you directly.

01:10:42 [HR]

henryhrich@gmail.com. [09]

01:10:45 [CH]

There you go. And I definitely know that there's a ton of work in this space. The closest I've done anything to this is my work that I did on fixed point numbers. That is adjacent to floating point implementations. But yeah, I mean, they are becoming much, much more widely used in the space of machine learning and stuff. I'm not sure if that's what your customer's use case is, but the half floating point and quarter floating point. And the joke that I always hear is that, you know, when it comes to machine learning and AI it's just a guess anyway. So who cares if you're off by a little bit? [everyone laughs] It's a recommendation engine at the end of the day or something like [that]. I mean, certain applications are more serious, but to be totally honest, like a ton of it is just these recurrent or convolutional neural nets that are just doing image recognition or what Netflix show you want to watch next. And it's not the end of the world if the sixth recommendation versus the seventh are in reverse order or something like that.

01:11:47 [HR]

Well, it does lead to questions about what primitives to implement. If you're going to have very short integers or short floating point, you might need saturating addition [Marshall agrees]. And that just becomes a nightmare [laughs], right? I don't know how bad it is.

01:12:06 [ML]

Well, like, what do you do if somebody wants to saturating add 2-byte to 4-byte? It's like when everything's all the same type, it's good. But then when you have something that mixes types, then you're in trouble all of a sudden.

01:12:19 [HR]

Yeah. Well, what I'm doing with 2-byte plus 4-byte is to avoid this quadratic proliferation of routines to add [Marshall agrees]. I have arranged all the types in a priority order and I convert everything to the highest type.

01:12:33 [CH]

That's very similar to what fixed point does. There's like an algebra; if there's two different scales or representations, you convert to the same one and then you do the operation, which inevitably means that in certain cases you're going to lose some precision. So it's then put on the user to say: "okay, you do the adjustments of whichever one so that they're in the same version so that we're not automatically doing something that you didn't want". But yeah, there's always like four or five different ways you could do it. The question is which one do you implement and then force the user to hand roll some extra stuff if they want something different? And it's, it's never pleasant. You never [chuckles] finish implementing and go: "ah, this is completely intuitive and the user will never be surprised". That is not how it works, unfortunately. All right. We are just about to pass the one and a half hour mark. I'm not sure if there's anything else that you wanted to highlight either in the 9.5 release or the 9.6 beta release? Because I think if I recall, you mentioned that this stuff isn't in 9.5, but it's being designed for 9.6, correct?

01:13:36 [HR]

The new numeric precisions are not, yes. Everything else we talked about was in 9.5.

01:13:43 [CH]

All right. Well, if that's the case, then I guess we should let people know if they want to get their hands on either 9.5 [or] 9.6, what is the best place to send folks to? I mean, I'm sure we'll include the link in the show notes, but where can people find either the beta 9.6 or J 9.5?

01:13:59 [BT]

Oh, well, you just go to "system installation" essentially, on the wiki. It'll take you right there. Or you go to the J software site and follow your nose through to the downloads and that will take you into whatever you want. And you can download old versions too. You don't have to take the most [recent] version. But I think one thing I've learned out of all of this is if you're a J user, it's very important that you spend time playing with the beta because [chuckles] if there are things that you want to influence, that's your chance to do it. And you have roughly a year. I know there are things that are introduced through that year, but they don't really spring things on you. You're given some time to respond. And I'm sure there's no doubt when you're in a situation where it's fluid in a beta, nobody's going to try and create something that's going to create huge problems. They're going to make adjustments. But once that beta hits (once it becomes the working version), I suppose your option is to go to a most recent version that doesn't cause you a problem. I know my approach is to say, well, the user takes the balance between are the opportunities worth what staying with the old one changing is. And I understand that there are times that you're forced to change and that can happen. But I think in a lot of cases, if you keep up with the beta, you can find those things and you have some influence for sure. In general cases: contact@arraycast.com. You can give us the feedback. We're happy to have it and we can forward stuff to Henry as well as to Marshall. If you don't want to go directly to Marshall for the ngn/k issues that he might be encountering over the next while [Marshall agrees]. And looks like Adám's got something to say.

01:15:46 [AB]

Well, having influence on these things, shouldn't we be celebrating them? Sounds like this is the first real confirmation of the ArrayCast having direct influence on its own subject matter [chuckles].

01:16:02 [BT]

No, actually, I think it's the second because the invisible modifiers [10] was mentioned here [Henry agrees] and then got changed.

01:16:09 [AB]

We only spoke about them, but it didn't sound like it.

01:16:10 [CH]

Oh yeah, yeah.

01:16:15 [HR]

That was mentioned here. But the reason I brought them back is because otherwise, I was going to have to change all the other documentation. Old documentation referred to it. It was easier to reinstate the feature than to fix the old books. I have to say, though, I love the language. I think it's amazing what you can do with Ken's language. It has nothing. There's no punctuation, no operator. It's just spaces and parentheses.

01:16:45 [CH]

Yeah, we will have to have you back because we didn't end up getting to it. And I actually, I have a list (where's my list?); I got lots of stuff on my computer, but I have a list of YouTube videos to make. The title says three YouTube videos, but there's actually five listed because clearly there was three when I made the list and then I added a couple. And one of them is the last number five on the list. It's not necessarily my priority, but it's J video decoding. I'll actually read it. I'm not going to explain what it does because I don't even understand. I can't remember what video, but someone commented on one of my YouTube videos. And this is the expression and it's using a modifier chain: + (,@ (\/ /~ ) - )~ ) So that's like six parentheses, three slashes if you include both, two tildes, a plus, a minus, an at, and a comma and it's a modifier train. I remarked on some video or some podcast at some point. Then someone left a comment (I think it was a live stream), but I haven't been able to decode it. But then someone actually emailed. I want to say his name was Igor (I might just be making that up). But if you're listening to this podcast, someone went in like line by line, decoded it. And I do plan to turn that into a YouTube video. Anyways, maybe I'll make that YouTube video and we'll have you back on to revisit the topic of modifier trains. Because they are a beautiful, if not entirely practical, language feature that exists in no other language.

01:18:20 [HR]

I'm going to have to disagree with you about some of this. Igor Zhuravlov, I think you're talking about. He's a pistol [Conor laughs]. He catches a lot of our mistakes [Henry chuckles]. I don't think you're doing J a service if you put a sentence like that out and suggest that this is what J is like. I wouldn't write that. And the reason it was taken out is because so few people could use the modifier trains that way that it was just better to tell them: "do it with explicit modifiers". Use u and v and say what you mean in a way that it's easy to understand. There's really no reason that a user needs to write that complicated expression that you talked about.

01:19:14 [AB]

There's no reason the user needs to write forks and hooks either, right?

01:19:21 [HR]

No, I disagree with that. A fork in the middle of a sequence of verbs is a very quick way of saying what you mean. But this modifier trains ... well, you know what a fork does. You know what a hook does. But the modifier sequence "conjunction, adverb, conjunction" what does that mean? There are about 20 of them. And so to begin with, you're going to have to go look it up and see how to make the substitution. And okay, so you've made the substitution and [in] the example we were given, there are probably four or five different pieces that have to be decoded that way. You can do it. But if somebody looks at that and says, well, this is what J is about. Well, that's not what J is about. Nobody writes that way [Conor laughs]. That's not true. A few people ... there are a couple people who write that way.

01:20:11 [AB]

But aren't you expressing exactly the sentiment that people have towards the forks? You said, well, a fork, a hook, you know what that does. Well, if I didn't know what it did, then I'd had to go and look up what it did and then decode it: "okay, so this verb takes these arguments".

01:20:30 [HR]

That's right. But my point is there are two things. There's a fork; there's a hook and they occur a lot. And I can reasonably expect you to learn what those two things do. And you can carry that with you and internalize them and use them to debug a sentence or to look at a sentence and understand it. And you don't have to drink the whole tacit programming kool-aid to use hooks and forks judiciously in the middle of essentially explicit code, which is how I want people to realize that J should be used. If you try to do everything tacit, it makes it much harder. That's up to you. But with these modifier trains, it's another level of complexity and there's not much payback for it. With hooks and forks, at least I could ... [sentence left incomplete]. Yes?

01:21:25 [BT]

I want to propose that we have Henry back on for "tacit 6" [Conor laughs]

01:21:31 [CH]

[Laughing] I'm literally thinking to myself that like, that was maybe the best clip. Are we allowed to put that as the cold open though? Because 98% of the episode had nothing to do with this. But now he's saying, you don't have to drink the tacit kool-aid, you know?

01:21:46 [AB]

We should use that for the next tacit episode [chuckles].

01:21:49 [CH]

Here's another ArrayCast promise, or maybe this is just a CodeReport promise. When I do make this video, I will cut in ... [sentence left incomplete]. I'm not sure if I'll use the video recording (if everyone agrees) or just the audio. I'll prefix my video with: "this isn't the J that everyone knows". And then I will continue to make my video that you just told me is a disservice. But I'll make sure that I get your disclaimer in there that like, people don't write code like this, except for Igor and maybe one or two others out there.

01:22:15 [HR]

No, no, I don't think Igor does it either. I think Igor can understand it. I have written a couple of modifier trains that I use as utilities. A modifier train is a useful template. It's not bad if you give it a name and some commentary; you can describe what it's doing. But even at that, it's no better than just writing it out as an explicit definition. Both versions, when they're given u and v operands, boil down ... they produce the same verbs that eventually get executed. The advantage of the modifier trains is they are very elegant; they're pretty. Not pretty enough that people should learn to use them [chuckles]. But anyway, I just don't want people to think ... [sentence left incomplete]. J's hard enough, you know, without having to think that you've got to learn this entire meta language of modifiers.

01:23:17 [CH]

All right, perfect. This is great because we did it, folks. Tacit 5.3, I believe this is [others laugh], or I don't know if we're on 5.4. We're going to see if we can get a full set of 5.5s all the way to 0.9, and then we'll do Tacit 6. Leave it to your host, me, to turn any episode [laughs], even if it means extending it another 10 or 15 minutes to make Bob's job harder [laughs]. Anyways, I apologize, Bob. We will say thank you so much, Henry, for coming on. This is always a blast when we get to chat. Probably we're going to have you on before a year passes to get our J 9.6 update because it's clear whether the listener wants to hear it, we definitely want to talk about modifier trains again [laughs]. And we'll reach out to you and find time for that. And Bob's got one last thing to say.

01:24:06 [BT]

I just wanted to point out that when we do Tacit 6, it is not guaranteed to be backwards compatible [others laugh].

01:24:15 [CH]

With that, I think we will say Happy Array Programming!

[everyone]

Happy Array Programming!

[music]