Generators Everywhere
Wow! Thanks for reading my very first post in my very new newsletter here! I'm pretty excited to be writing these. This post is, well, my first post, so it might feel a bit wonky, as I figure out how to best format these, etc. But, I hope you'll bear with me. And, of course, if you have any feedback, please do hit reply and send it to me (I think it will come to me); I can take it! Especially if you have any feedback on “wow, this formatting made it hard to read,” because I’m still trying to figure out how to use substack’s editor. :)
One thing I'm learning by writing this first post is that I probably can stand to write less. :) So, I got to a certain point that feels like it is okay to stop there and continue next week.
Just to set the stage, these are going to be pretty informal. I want to share cool things that I find, as I'm looking through some lesser-mainstream, perhaps lesser-known languages. Bear in mind that, since I'm learning these languages as I writ these, some of my understanding might be a bit off, and almost assuredly some of my terminology will be a bit off. Instead, I'll be writing how I think of it. I bet that will change as I get more into the language, too.
Another important thing to note is that I am not writing this to teach you the languages. So, I'm going to generally skip things like "This is the syntax of the language," unless that is part of what is cool (say, for example, if and when we talk about an array language) and there are some certain assumptions I'll make.
For example, I'm going to suppose that folks can understand the following code snippet:
procedure main()
write("hello, world")
end
without going into the specific parts of it.
Starting us off, as I mention in the about for the whole project, is Icon. I had never heard of it when I saw Glenn Vanderburg respond to Hillel Wayne about it. Wow! I was missing out on a lot. This language is pretty cool, so far. And, best of all, it has a basis that has made me really stop and go "hmmm...." this has interesting potential.
Glenn mentioned it as a focus on text processing. I'm just getting to that part of it, but the early learning, specifically what I'm going to talk about here, is making me really excited. The core idea that is really turning me on is the following statement:
Almost everything is an expression, and every expression is a generator.
This sort of blew my mind, as I knew that there was some subtle meaning there. What does it mean for everything to be an generator, and how does this translate into writing code?
Let's start with the simplest. What is `5`. Well, let's write it out.
procedure main()
write(5)
end
#=> 5
A constant value, like `5`, generates a single value: `5`. So, when we ask it for the next value, it yields `5`. Let's look at a multi-value generator, though: `1 to 2`. What happens if we do the same:
procedure main()
write(1 to 2)
end
#=> 1
What about the `2`? We would expect it to print that out, as well, wouldn't we? Well, no, because we aren't triggering the processing of the generator. Instead, we need to tell icon that we want to evaluate this expression's generators. This is done with the `every` keyword. Let's try it with the single-value `5` generator:
procedure main()
every write(5)
end
#=> 5
This again prints out just 5. This is expected, since `5` is a generator that just yields a single value: `5`. Things get interesting, though, when we use a multi-value generator, such as `1 to 2`.
procedure main()
every write(1 to 2)
end
#=> 1
#=> 2
Now we are getting somewhere. We now have it exhausting the generator. What does it mean to "exhaust the generator," though. Well, in Icon, an expression either succeeds or fails (more on this really when we get to talking a bit about control flow). If it succeeds, then it yields a value. If it fails, then it doesn't, and it tells the runtime that this generator is done, move along, nothing more to see here. So, if we think about it, we can kind of think of these generators as a list of prospective values that ends in `&fail` (the Icon term for when an expression fails). So, we kind of have something like
5 -> [5, &fail]
1 to 2 -> [1, 2, &fail]
Of course, it isn't a literal, hard-coded list, we are just using that as a handy way to think about it right now.
Expressions are evaluated left-to-right, but generators are evaluated bottom-up. That is, when evaluating an expression, it builds up a hierarchy of the pieces, left to right, then evaluates them from bottom-up. Let's take our simple `every write(5)` as an example.
`write` is an expression, so is `5`. So, let's build this in a hierarchy.
`write`
|
`5`
So, now that we have this hierarchy, we can start at the bottom and begin evaluating. We'll look at this as a conversation between `every` and the expression hierarchy.
`every` (starting at the bottom): Hey, `5`, can you generate a value for me?
`5`: Yeah, success! Here is the value: `5`!
`every`: Great, let me traverse up the chain. Hey `write`, can you generate a value for me?
`write`: Well, I need a value in order to do something. Do you have one for me?
`every`: Yup! The generator gave me `5`. Here it is!
`write`: Awesome! *side-effect printing to output* I was successful. My success value is `5`.
`every`: Great, let me track down again. Hey, `5`, can you generate a value for me?
`5`: No, I'm exhausted. `&fail`
`every`: Oh no, let me backtrack up and see about `write`. Hey, `write`, can you generate another value for me?
`write`: Well, I need a value in order to do something. Do you have one for me?
`every`: Uh, not really, the generator below you failed.
`write`: Then I fail. Here you go `&fail`.
`every`: No higher place to backtrack to, so I'm stopping!
And we get the following output to the screen:
5
Now, let's do the same for `every write(1 to 2)`.
`write` is an expression, so is `1 to 2`. So, let's build this in a hierarchy.
`write`
|
`1 to 2`
So, now that we have this hierarchy, we can start at the bottom and begin evaluating:
`every`: Hey, `1 to 2`, can you generate a value for me?
`1 to 2`: Yeah, success! Here is the value: `1`!
`every`: Great, let me traverse up the chain. Hey `write`, can you generate a value for me?
`write`: Well, I need a value in order to do something. Do you have one for me?
`every`: Yup! The generator gave me `1`. Here it is!
`write`: Awesome! *side-effect printing to output* I was successful. My success value is `1`.
`every`: Great, let me track down again. Hey, `1 to 2`, can you generate a value for me?
`1 to 2`: Yeah, success! Here is the value: `2`!
`every`: Great, let me traverse up the chain. Hey `write`, can you generate a value for me?
`write`: Well, I need a value in order to do something. Do you have one for me?
`every`: Yup! The generator gave me `2`. Here it is!
`write`: Awesome! *side-effect printing to output* I was successful. My success value is `2`.
`every`: Great, let me track down again. Hey, `1 to 2`, can you generate a value for me?
`1 to 2`: No, I'm exhausted. `&fail`
`every`: Oh no, let me backtrack up and see about `write`. Hey, `write`, can you generate another value for me?
`write`: Well, I need a value in order to do something. Do you have one for me?
`every`: Uh, not really, the generator below you failed.
`write`: Then I fail. Here you go `&fail`.
`every`: No higher place to backtrack to, so I'm stopping!
And we get the following output to the screen:
1 2
This is cool, but let's make it a bit more complicated to really get a sense of this. How about this line of code:
every write((1 to 2) + (10 to 20 by 10))
Okay, there is a new thing here with the `by` keyword. That does just what we would think. Let's do a simple experiment to see it:
every write(10 to 20 by 10)
#=> 10
#=> 20
So, it generates the numbers with a specific increment. Leaving it off `1 to 10` is the same as `1 to 10 by 1`.
Okay, evaluating the more complex expression starts again with building up the tree. We have a binary operator in there `+`, which will make the drawing a bit more complicated to understand, but we'll cheat a bit and let it deal with the value before it, as well. We'll eventually get to a nicer syntax, but, for now, let's just deal with the complexity. When we break this into the hierarchy, we get something like:
`write`
|
`1 to 2`
|
`+`
|
`10 to 20 by 10`
Just like before, let's start at the bottom and start evaluating. Note: this is going to get confusing using this conversational form; we'll eventually use a better syntax once we get through this.
`every`: Hey, `10 to 20 by 10`, can you generate a value for me?
`10 to 20 by 10`: Yeah, success! Here is the value: `10`!
`every`: Great, let me traverse up the chain. Hey `+`, can you generate a value for me?
`+`: Well, I need a value in order to do something. Do you have one for me?
`every`: Yup! The generator gave me `10`. Here it is!
`+`: Great, I need the value next highest up, though, so I can do the operation.
`every`: Cool. Hey, `1 to 2`, can you generate a value for me?
`1 to 2`: Yeah, success! Here is the value: `1`!
`every`: Great! Hey `+` here are your two values, `1` and `10`, can you generate a value for me?
`+`: Yup! Here it is `11`.
`every`: Awesome. Hey, `write`, can you generate a value for me?
`write`: I need a value!
`every`: I've got one for you, `11`.
`write`: Awesome! *side-effect printing to output* I was successful. My success value is `11`.
`every`: Great, let me track down again. Hey, `10 to 20 by 10`, can you generate your next value for me?
`10 to 20 by 10`: Yeah, success! Here is the value: `20`!
`every`: Hey, `+`, can you generate a value for me?
`+`: Sure. I need two values.
`every`: Hey `1 to 2`, what is your current value?
`1 to 2`: I'm still on `1`, here it is.
`every`: Cool. Hey, `+`, here are the values, `20` and `1`.
`+`: Great. Here is my value: `21`.
`every`: Hey, `write`, can you generate a value for me?
`write`: I need a value!
`every`: I've got one for you, `21`.
`write`: Awesome! *side-effect printing to output* I was successful. My success value is `21`.
`every`: Great, let me track down again. Hey, `10 to 20 by 10`, can you generate your next value for me?
`10 to 20 by 10`: Argh! `&fail`
`every`: Doh! Let me backtrack up. Hey `+`? Here is `&fail`.
`+`: Oooh, that means I will generate `&fail`.
`every`: Keep backtracking up. Hey `1 to 2`, can you generate your next value for me.
`1 to 2`: Sure. Here it is `2`.
`every`: Okay, let me drop down again. Hey `10 to 20 by 10`, can you generate for me:
`10 to 20 by 10`: Sure! My value is `10`.
etc
etc
etc
And we end up outputting `12`, then `22`. So, in the end, we have outputted:
11
21
12
22
Notice a key thing here: we didn't generate a "new" value for the `1 to 2` generator as long as the lower-level generator `10 to 20 by 10` was still succeeding in generating values, instead we asked `1 to 2` for the current value. Once the bottom level `&fail`-ed, though, we "backtracked" up the hierarchy and then asked `1 to 2` to generate its next value. We then went down and restarted the `10 to 20 by 10` generator from the start. Interesting!
This does get confusing quickly and hard to track in this style, so we'll drop this conversational mechanic.
After going through this, though, we should be able to understand in a straight-forward fashion how something like this evaluates (try it out yourself):
procedure main()
every write(1 to 2, 3 to 4)
end
#=> 13
#=> 14
#=> 23
#=> 24
This is already pretty long, and I think this is a good spot to end for this week. Next week, we'll look at a couple more things to really dive into around generators, including using them as control flow. The goal at the end of next week's letter will be to understand how the following snippet generates the given output:
procedure main()
every writes(" ", (1 to 2) (1 to 3, 1 to 2))
end
#=> 1 1 2 2 3 3 1 2 1 2 1 2
Thanks again for reading this post! I really appreciate it! Here is a picture of my cats! Sir Nickel of Noosie is on the top platform, and Little Miss Louise is in the basket. Don't worry, they will show up more and more as I get a handle on these writings.