Tuesday, October 27, 2020

The Forever Project

“Any writer worth his salt writes to please himself…It’s a self-exploratory operation that is endless.” – Harper Lee, author of To Kill a Mockingbird
I often get asked if Dragons Abound is available as open-source, or if I intend to make it into an app, and my standard answer is “Dragons Abound is a personal growth project."  This posting is the longer explanation of what that means to me.

If you're anything like me, you probably find that the effort to get by in life takes up most of your time and energy.  You spend your life working, commuting, going to the grocery store, doing the laundry, and all the other sundry tasks required to stay afloat in our modern world.  But it's hard to live a happy and fulfilled life if all you do is what you must do to survive.  You've got to carve out opportunities to invest in yourself as well.  One of the ways I do that is a life practice I think of as “Forever Projects."  This idea has been part of my life for a few decades, and my approach and understanding of it has evolved over the years, so I wanted to write a bit about my idea of a Forever Project and how I think it benefits me.  (And maybe will benefit you.)

As I suggested above, the purpose of a Forever Project is to create opportunities for you to grow in ways that you aren't already afforded by your “regular" life.  Whatever you do for your day job, you're already investing a huge chunk of your life in developing and using those skills and feeding that interest.  You're probably getting about as much out of that as you can.  Conversely, if you put even a few hours into something that's new to you, you're very likely to achieve significant growth.  So this is the first element of a Forever Project:  to focus on some new topic where you aren't already investing time and energy.

Of course, you're the judge of what that means.  Maybe during the day you're programming movement routines for monsters in a first person shooter, and at night you come home and develop game sprites for an 8 bit gardening game.  For some people, that might be repetitive and stifling.  But maybe for you that provides ample opportunities for growth.  The point is that a Forever Project should take you off in a new direction.  Find a new way to invest in yourself.

Second, a Forever Project is one that - as the name implies - you can work on forever.  It's an open-ended topic that you can explore for a long time without running out of challenges.  Now I'm not the first to suggest this idea, and if you're interested in personal growth there's obvious value in picking a topic that you can study a long time.  But my focus is not so much on having an endless topic to explore, but on some of the deeper implications of what it means to work on something “forever."

Much of how we think about our lives has its roots in economics.  When you make a decision (“Should I go to see a movie tonight?") much of your thinking is likely some form of economic reasoning: “How much will it cost?" “Is this movie worth it?" “Is there something else I'd rather spend my money on?"  “How far away is the theater?" “Should I paint the den instead?".  All of these questions are about getting value from how you spend your resources.  This sort of reasoning helps us use our time and resources in fruitful ways.  

But an interesting thing about economic reasoning is that it breaks down completely for someone with an infinite lifetime.  In an infinite lifetime, you'll have an infinite amount of money, so it really doesn't matter what the movie would cost.  And you don't have to choose between seeing the movie and painting the den, because with an infinite lifetime you can do both.  In fact, any thinking along the lines of “I'm wasting my time" makes no sense if you live forever - you can waste an infinite amount of time and still have an infinite amount left!

Of course, you don't really have “forever" to work on any project.  But you can reason like you do, and that has some interesting implications.  And because we don't normally reason like this, I think it can provide some great benefits.

First of all, reasoning this way removes all pressure to “accomplish something."  With an infinite amount of time on your hands, you can turn off that critic inside your head that constantly second-guesses your decisions.  (The one that uses economic reasoning to help you maximize the benefit of every choice in your life.)  If you're doing a regular project - say, building a rocking chair - and you get distracted into spending hours reading about how trees grow and the biology of woody plants, you'll end up thinking that was wasted time - time that did not move you closer to your goal of building a rocking chair.  But if you're learning woodworking and you have forever to do that, then a detour into how trees grow is perfectly fine.  Even if you detour entirely and go off to get a Ph.D. in plant biology, that's fine.  You can always come back to woodworking a few thousand years from now.  When you remove the judgement of whether a pursuit is “worth the time" you'll free yourself listen more closely to the inner voice of what interests and matters to you.

Learning at the workbench

This attitude - listening to your inner voice free of the constraints of finite economics - is a powerful enabler for personal growth.  You'll find that when you spend your time in areas that are both interesting and important to you, you'll find something to learn and be motivated to do it - and that's precisely where you're most likely to accomplish some personal growth.

Last year I spent six months working on map borders, and I got a few questioning comments about spending so much time on that topic.  That's a perfectly understandable reaction from someone who looks at what I'm doing through the lens of limited time, and wonders why I would spend so much effort on a minor aspect of map creation.  But I'm not “building a map app"; I'm doing a Forever Project about procedural generation, computer art and many other things.  I've always had an interest in Celtic knots, so I spent a couple of months learning how they were drawn and how to  implement them in Dragons Abound.  I learned something new (to me), and that bit of personal growth is the reward - regardless of whether or not I ever “finish" Dragons Abound.

I'll mention one more benefit of the Forever Project before I talk more about how I practice it.

Whatever you do to earn a living - fix cars, teach ballet, or design ground systems for satellites (me) - it's likely that you have made an effort (possibly a very big effort) to be good at what you do.  That's a consequence of a market economy - people pay more for rare expertise.  You've probably seen the classic supply and demand curve:


When there's a high demand and a low supply, the price is high.  The better you get at a skill, the fewer people there are who are equally good at that skill, and so the value of that skill rises.  The best quarterbacks in the NFL get paid more than the mediocre quarterbacks, and the best ballet teachers get paid more than the mediocre ballet teachers.  So most people end up pouring their effort into being very good at something - playing the violin, programming, selling life insurance, and so on - so that they can make a good living.  But a relentless focus on building a single skill is not (for most people) the best formula for leading a happy life.  In fact, many people who do that eventually crack - they have what we (deceptively) call a “A Mid-Life Crisis."

We talk sometimes about the “Renaissance Man" - a polymath who is good at many different pursuits.  But why were there (seemingly) many polymaths in the Renaissance and few now?  Part of the answer is surely that the gentlemen and courtiers of the Renaissance faced less economic pressure to specialize and were more free to do what naturally pleased them and fed their growth - to study and practice skills in many different areas.  I suspect that many modern people would be like that given the opportunity.  (Although the idle rich might argue otherwise.)  Regardless, we cannot easily change the whole basis of today's society (although we should), but we can certainly recognize its limits and try to intentionally diversify our interests outside of our 9 to 5 work life.

A good Forever Project will encourage you to wander through a variety of topics.  Many of them will be new(-ish) to you, so you'll find it easy to learn something about them.  I spent two months on Celtic knots and came away with a better understanding of their history, the theory behind drawing them, and some notion of how to implement an algorithm to draw them.  I'm pretty sure that none of that will ever increase my salary.  But what it did do is increase the breadth of my personal knowledge, skills and interests.  And if you do the same, perhaps that will make you a happier, more fulfilled person.  Possibly even a Renaissance (wo)man :-)

Now let me talk a little bit about the practical aspects of finding a Forever Project and working on it.

Choosing a Topic

The most important aspect of picking a Forever Project is to to pick a topic and not a goal.  If your Forever Project is to create an Uber app to connect people in need with free rides from volunteer drivers, then you've missed the point.  You've picked a specific goal, and that will short-circuit all the “forever" reasoning.  (Which doesn't mean that you shouldn't spend your time doing that.  It just means that you shouldn't expect to get Forever Project benefits out of building an app.)  A bad Forever Project is to “make a wooden rocking chair"; a good Forever Project is to “learn woodworking."  I'll talk some more about what makes a good topic a little bit later, but right now I want to talk about the challenge of giving yourself a topic rather than a goal.

If you've never attended a graduate school, you might imagine that the biggest challenge is the difficulty of the material - mastering the highest levels of human knowledge in some subject area.  But in fact most graduate students do fine with the actual schoolwork - after all, people don't even try graduate school unless they're pretty smart and talented in a subject.  You'd expect them to be good at that.  The real challenge is something else entirely.

The big difference between undergraduate education and graduate school is direction.  As an undergraduate student, the professor directs your study (“Read Chapter 4 by Thursday") and provides the framework for applying your knowledge (“Prove theorem 4.1 using Turner's Method").  In graduate school (once you've hit the research phase), you must provide your own direction.  You decide what to study, how to study it, and how to apply that new knowledge in some interesting way.  For a lot of graduate students, this is a stumbling block.  They're excellent at studying or solving problems when someone else tells them what to do, but they cannot direct themselves effectively.  To be successful, they have to learn to provide direction from within themselves (intrinsically) that they are used to receiving from outside themselves (extrinsically).  This is a difficult skill to master, and is probably the biggest reason graduate students fail.

Like graduate school, a Forever Project is inherently an intrinsic effort.  It's something you're going to do under your own direction for your own benefit.  You cannot rely on anyone else to tell you what to do, help you figure out how to do it, or cheerlead when you're feeling stuck.  So you're going to need a lot of the same skills as a graduate student to succeed.  The good news is that for several reasons this is much easier for a Forever Project than for graduate school.

In graduate school you don't really get a free choice of what to work on -- you have to match your thesis topic to an advisor, you can't do anything that's been done before, and so on.  But in a Forever Project you can pick whatever you'd like to work on.  So you can pick something that has great personal interest to you, even if it's extremely niche and has been done twenty times before.

I talk about picking a topic of “great personal interest," but what does that mean, anyway?  A personal interest is something that you find fascinating for its own sake, because of who you are.  In other words, something that you find intrinsically interesting.  Not something that you think you should study because of the latest article on Medium about how to get ahead in your job, but something that you would feel compelled to explore regardless of whether it provides an immediate benefits or not.  In a class I took from William Glass he said “I don't write because I want to, I write because I have to."  The ideal Forever Project is that sort of topic to you; like an itch that you must scratch.

If there's no topic in your life like that, fine.  Maybe you're not at a point where you're ready to take up a Forever Project.  Go do other things until some topic rears up and smacks you on the nose.  Or maybe there's a topic that seems kind of interesting.  Start exploring it; maybe it will become a Forever Project topic.  And if not, at least you've learned that!

On top of being intrinsically interesting to you, a Forever Project topic needs to be open-ended.  After all, if you're going to work on something forever, it had better be a big topic.  But when I say “big topic" I don't mean one that is life-changing or of great importance, I mean one that provides many opportunities to learn new things and to explore in new directions.

Any topic has opportunities to learn skills and knowledge specific to that topic.  I call this direct learning.  Some topics have more direct learning opportunities than others.  For example, the yo-yo has some opportunities for direct learning.  There are different techniques to be mastered, and ways to arrange those techniques into tricks.  But on the overall scale of all possible topics, yo-yoing offers limited opportunities.  Compare the yo-yo to (say) woodworking, where there are many more techniques to learn, and many different ways to apply those techniques.  Ideally, you want to pick a Forever Project topic on the woodworking end of the spectrum rather than the yo-yo end.

Many crafts make good Forever Projects, with abundant learning opportunities.  The reason is that crafts are skilled work.  To do a craft, you have to learn and master skills and this “learning and practicing" is exactly the point of the Forever Project.  Another reason is that crafts are creative -- in the sense that means “to create something."  When you craft something, much of the interest is in solving the puzzle of how to use the skills you've mastered to create the item you desire to produce.  If you're a woodworker and want to build an end table, you have to think about what joints you will use to build the structure, how you will create the parts of the table, and so on.  Compare that to (for example) practicing yoga, where you use your skills to follow along to the directions of your teacher.  In both cases you have to master difficult skills, but in one case you also have to creatively apply those skills in problem-solving.  And that's an addition level of personal growth.

Sometimes the difference between a bad and a good Forever Project topic is just in the context and conception.  Yo-yo-ing might not be a great Forever Project itself, but if you think of it as the first step of (say) Circus Arts then it's easier to see how you might pursue that for a long time.  And if you really are on fire to learn the yo-yo as a Forever Project, then do it!  But keep yourself open to unexpected learning opportunities along the way.  If you get interested in balance board along the way, don't turn away because “it isn't yo-yo."  Try to remember that in the Forever Project it's the forward motion that counts, not where it takes you.

Topics also have opportunities for indirect learning.  Indirect opportunities are detours -- side trips that can take you down new paths away from the original topic.  For example, if your Forever Project was woodworking and you built a chair, you might decide to cane weave the seat, learning the skills necessary to do that.

And this could lead you into basket weaving, and the guy at the cane supplies store might show you the Adirondack pack basket he'd made, and that would get you interested in the history of animal trapping.
This wandering through connections to new topics is what I call indirect learning.

Some topics have more connections than other topics.  For example, if you decide to write a chess playing program, the indirect learning opportunities are limited.  You might get interested in some of the history of chess, or similar games, or different ways to write game-playing programs, but these topics are all pretty close to chess.  Conversely, if you decide to write a map-drawing program, you have a wide array of diverse topics to explore:  how land forms, how things are named, how maps are drawn, what makes art beautiful, etc.  In other words, you want to pick a Forever Topic that has a lot of interesting distractions!

Of course, in graduate school an interesting distraction is the last thing you want.  My Ph.D. topic was on creativity, but my adviser was very interested in the related topic of humor and that distraction caused me no end of pain.  But this is one of the areas where having forever to work on a project turns the conventional wisdom on its head.  You're doing this for your own personal satisfaction and growth.  There's no such thing as a distraction as long as you're finding opportunities to grow, so in a Forever Project you actually want a topic that continually throws up interesting distractions.

And on a practical level, if you work on something for a long time (even if not actually forever) you're inevitably going to have “dry" periods where you are bored with the topic.  Having a bunch of interesting distractions lets you pivot away from your dry well and do something else that is interesting and provides you with personal growth.

The lack of indirect learning possibilities was a shortcoming in one of my previous Forever Projects called Net Prophet which predicted college basketball games.  This was a topic that was of great intrinsic interest to me (I've been a college basketball fan for many years) and offered many interesting direct learning opportunities.  But it didn't really connect to many other topics that were interesting to me, so that when I'd exhausted the direct learning opportunities, I hadn't found any related topics to take up instead.

The Practice 

Once you've selected a topic for a Forever Project, how should you pursue that?

The most important principle to remember in working on a Forever Project is that it's not about Getting Things Done.  It's about moving forward in a way that is (1) interesting to you, and (2) helps you to grow.  So throw out all the usual maxims about getting the most out of your time, tips for finishing a project, and so on.  All you need is a pretty simple loop:  find something interesting in your project and then work on it until it's no longer interesting.  Rinse and repeat.

There, I've given you the secret to a rich and fulfilling life. :-)

There are a couple of practical ways I've found to keep the queue of interesting topics full.

The most straightforward is keeping a TODO list -- which is really more of an “Interesting Ideas" list, but that's not as catchy.  The idea is to capture any idea associated with the Forever Project.  These could be bugs, notions on how to improve the Project, side projects, etc.  Unlike a TODO list, you don't put dates on these, or prioritize them, or try to “work them off."  When you need inspiration or a task to work on next, you can scan the list and see what catches your fancy.  I also leave the completed tasks (marked as such) in the list.  This is useful to see what you've accomplished (which can be motivating!).  Those of you who have read my blog for a while have no doubt come across entries where I noted some problem or idea and said that it was “going onto the TODO list."  If you're doing it right, your TODO list should be growing faster than you can work it off.

Another good source of ideas comes from reading about similar work and otherwise participating in a community of enthusiasts.  For Dragons Abound I follow a number of people doing map generation (like Azgaar) and generative art (like Mewo2sketches).  For my Forever Project on predicting college basketball games, I eventually built a community of other people working on the same problem, and we exchanged notes and ideas via email.  However you find a community, you'll find that it can be a good source of inspiration for the TODO list and also to help steer you away from areas that might be unproductive (in the sense of not helping you grow).

Even if you have good ideas, many people struggle to find the motivation to work on a project.  I've found a number of ways to address the motivation problem that may also work for you.

Much has been written in recent years on the psychology of video games. Regardless of your thoughts about video games, you can harness some of the same psychological tricks that make video games compelling to make your Forever Project compelling.  One of these is the notion of an immediate feedback progress loop.  The visceral connection between action and reward is a powerful mechanic.  There's a whole genre of games that uses nothing but that connection between the simplest possible action (a click) and small reward -- and they can be quite addictive!

A good Forever Project topic has this kind of immediate feedback loop.  You can experience in a relatively short time scale the progress that you've made.  A good example is learning to juggle.  You start off occasionally making one throw.  Then you get two in a row.  More practice, three in a row!  And so on.  Compare this to something like painting, where the feedback loop is much longer and progress is much slower.  There's a reason more people learn to juggle well than learn to paint well.

Coding projects generally have a built-in feedback loop:  make a code change and run to see the difference.  In Net Prophet, I could implement some new predictive feature and then run it over many years of college basketball games and see immediately if it helped improve the predictions.  And in Dragons Abound, I can make a change and run to see how my maps have changed.

It's certainly difficult to stay interested in a topic that doesn't provide an immediate rewards, so that's the most important aspect of a reward system.  But you should think about structuring your Forever Project to achieve those rewards at a variety of time scales, from very immediate to very long-term.  For example, in Dragons Abound, I have occasionally (but intentionally) interrupted my usual plans to recreate a map I like.  This is an intermediate scale reward, something between the immediate satisfaction of the code-run loop and the long-term satisfaction of creating a full-featured map program.
Having rewards at different times scales lets you adapt your effort to your current level of motivation.

Another way to think about this feedback loop is the idea of tangible progress.  As you work on your Forever Project, what will you have to show for your work?  What discernible, obvious evidence will there be that you've invested time and effort into this project?  If there is nothing (or little) to show for large chunks of work, you're going to have trouble maintaining your motivation.  So you want to think about structuring your Forever Project so that you can see/experience your progress.  One of the great things about Dragons Abound as a Forever Project is that for most efforts, I can see the result the next time I generate a map.  Areas where that doesn't work (restructuring world generation to be template driven, for example) are much harder to finish.

Another psychological mechanism from video games that you can leverage is socialization.  The idea behind socialization is to add a group (social) aspect to an individual effort.  In video games we see this in things like multiplayer modes and group chats.  Humans are in many ways pack animals, and getting involvement and approval from others is a powerful and engaging motivation.  Certainly there have been some solitary great creators, but many more have benefited from creating within a community.

There are a number of different ways you can socialize your Forever Project.  One obvious method is the one I'm doing write now - blogging or writing about your project.  I particularly like blogging for socialization because it has a couple of non-obvious benefits.

The first benefit is that writing itself is an opportunity for personal growth.  Unless you're already a professional author, you've probably got a lot of room to grow as a writer.  (For most people, writing could be a Forever Project itself!)  So writing itself is one of those interesting distractions I talked about above.  Even if your Forever Project ends up being a dud, you may learn a lot from writing about it.

Beyond that, writing about your Forever Project forces you to think about your learning process.  Introspection is an important element for effective personal growth.  When you're trying to learn something, or to become better at something, taking the time to think about what you're doing, what worked and what didn't work, and how you are learning is a way of supercharging your progress.  And writing about your project forces you to do that introspection.  (And if you don't write, try to find other opportunities to do this introspection.)  So try to write more about your process than your progress; think in terms of “what did I learn doing this" or "how did I get from here to there" rather than “what did I accomplish."  Further, since you're usually writing for an audience that isn't as expert in your project as you are, you have to think about ways to explain your process.  That effort also helps clarify and cement your own understanding.

The second benefit is that you probably won't have much of an audience for your writing.  (That may not sound like a benefit, but stick with me.)  Socialization is a two-way street.  If people don't respond, or respond negatively, that's a motivation killer.  Compare two different situations where you tell a story at a party.  In the first situation, you tell the story in a corner to one or two people, who are totally interested and smiling.  In the second situation, you tell the story in the center of the party with a large group of people around you, but they're almost all bored and uninterested, talking amongst themselves and largely ignoring you.  The first situation sounds better, right?  Well, that's the non-obvious benefit of blogging.  There are a load of people out there blogging, and almost all of them are better writers and better looking than you.  Nobody is going to read your blog about frabulizing widgets unless they really care about frabulizing widgets.  So it's not going to be a big audience, but it should be an interested audience.  And I think you'll find that you get 90% of the benefits of socialization from a handful of readers as you would get from a sea of readers.

That last point also suggests that you should be looking for readers in a small ponds rather than in the ocean.  I link these blog posts in /r/proceduralgeneration, which is a fairly small community but with many of  those people already interested in map generation.  There's a community for development blogs, but posting there is like telling a story in a crowd of uninterested people.  Whatever your Forever Project topic is, there's surely an Internet community of like-minded people somewhere, so seek that out as your audience, and don't worry about whether or not your blog is “popular."

Another roadblock to practicing a Forever Project is simply finding time.  You might have interesting ideas, strong motivation, and still be stymied simply because you cannot make time to actually do anything.  I could tell you about prioritizing what is important to you and all of that advice, but I'm sure you've heard that before, so let me try to give more practical help.

The first step in creating time is to reduce the barrier to entry.  By this I simply mean that you need to arrange your Forever Project so that when you do find some time to work on it, you can start immediately and work productively until your time runs out.   Imagine, for example, that your Forever Project was skydiving.  Skydiving is not something you can do impromptu.  (Or so I imagine - I admit that I have never skydived.)  You have to go to an airport, arrange a flight, and so on.  So if you're a person who struggles to find time to invest in something like a Forever Project, this isn't a good topic choice for you.  Maybe Circus Arts is better - with 30 free seconds you can pick up some bean bags and practice a cascade.

Beyond selecting well, you can also make your Forever Project work environment start fast.  For example, for Dragons Abound I have a development environment that starts up in a few clicks.  What's more, I have three different computers I use depending upon my daily circumstances, and I've made sure that I have exactly the same development environment on each computer, so there's no cognitive roadbumps in figuring out how to do things on a particular computer.  And that means I can work on Dragons Abound whenever I find a free moment, regardless of which computer I have handy.  For woodworking, I try to always keep my woodshop organized and ready for use.  If I don't do that at the end of a working session, I know it will slow down the start of my next session - and maybe drain away the motivation and enthusiasm I had built up.  And time that I spend hunting for my fillister plane is time I won't be creating, learning or growing.

There's also a psychological barrier to entry when you start up any new task.  Switching mental states to get started on something takes some reserve of willpower.  But you'll often find that once you get started, you have more motivation and time than you realized.  A strategy you can try to trick yourself over this particular barrier is to commit to doing only one, small task.  When I was a graduate student trying to get my thesis done, I committed to writing for a half-hour a day, no matter what.  Even if that meant just revising my index.  But the interesting thing is that after I managed to get started writing, I would often keep writing for hours.  With Dragons Abound I may find a small amount of time after a work day when I don't have much energy left.  In those times, I pick out some small bug or feature to work on.  I often find that I wasn't lacking energy to work, I was just lacking energy to start.

On a practical level I enable this strategy by filling my TODO list with tasks of various sizes.  Some are very small (“Do gray scale by slapping filter on entire map") and some are very big (“Implement procedural compasses") and others are every size between.  This lets me pick something to work on that matches the time and motivation I have available.  I might choose to work on something long and accept that I will have to come back to it, but I also have the option to pick something that I can finish now - and close the feedback loop I mentioned above.  If you know that you will only have small time chunks available (e.g., you have two small kids), you can also spend some of those chunks working on your TODO list to break big tasks into smaller tasks.  It's much better to get something small done than to get discouraged because you never have time to tackle anything.

I also fill my TODO list with different sorts of tasks.  Sometimes I have enthusiasm to work on a blog post when I have no enthusiasm to code.  Or I have enthusiasm to sift through The Cartographer's Guild looking for interesting examples of compasses when I have no enthusiasm to code or write.

You can also take some advice that is given to writers:  “When you write, write and when you edit, edit."  The point is to not interrupt your creative / personal growth flow with self-criticism or polishing.  When I'm adding some new feature to Dragons Abound, I try to consciously accept that the implementation will be non-optimal.  It's important to have the fun of creating something new, and not to ruin that with worries about quality.  (And you have forever to improve it, after all!)  If  you can stay focused on the fun / creative aspect of working on your Forever Project you'll find that you get more value out of the time you can put towards.  And in turn, that will encourage you to find more time.

The final step is to avoid distractions.  I'm not talking here about your wife asking you to open a jar of pickles.  I'm talking about efforts that don't contribute to learning something and growing.  “Polishing" is a good example of a common distraction.  You can find yourself working very hard to eke the last few smidgens of perfection out of your project.  Sometimes that can be a learning process -- you can learn about optimization, or delve into user interface design, etc.  Sometimes you're trying to perfect a skill you've picked up in your Forever Project and put it to some use -- you've decided to become a professional juggler, say.  There's nothing wrong with that, but it's not likely to teach you much and you should recognize that.

Back to the Beginning (and Wherever That Leads)

To return to the question I posed at the beginning of this essay, that last point is one of the reasons I don't make Dragons Abound available and haven't tried to monetize it.  Doing either of those things would be a distraction from what I consider the important work - keeping myself happy with interesting challenges and opportunities for personal growth.  Of course, I might not always feel that way.  I'm putting a lot of work into Dragons Abound and doing some (I think) interesting and new things with it, so maybe it has a future as an app of some sort or another.  And that's another benefit of a Forever Project that bears mentioning: it doesn't have to remain a Forever Project forever.

Monday, October 5, 2020

Delta Drawn (Part 2)

At the end of the last post, I had created land part of a delta at the mouth of a river:
Now I need to fix the river so that it actually crosses the delta, and then add in the branches.

To start off, I'll send the river across to the middle point of the delta.  Rivers are normally created as part of a process that traces the flux of water across the Voronoi grid.  I can't do the same thing here but I'll have to make sure my new invented river course has all the proper attributes and values.  The first step is to find the midpoint of the new delta, and connect the existing river to that:
Well, that sort of worked.  But the new river segment is narrowing down to nothing at the mouth of the river.  The problem turns out to be a missing divisor in a calculation and that is easily fixed:
Okay, so now we have the river reaching the ocean, which is a good start.  Right now it's just a single straight segment from the old end of the river to the ocean.  We can break that up into a few segments and add some variation.
Now the final segment is fairly indistinguishable from the original river.  However, there's a slight trace of line across the mouth of the river.  This happens because I've drawn the river exactly to the shore, and (because lines have thickness) some of the coastline is still showing.  The solution is to project the river a little bit further into the ocean.
Now I need to fill in the rest of the criss-crossing rivers of the delta.

The basic idea is that I'll repeatedly branch one of the existing rivers in the delta and then run the new river to some point on the coastline of the delta.  Something like (bad graphics warning):
Each time I split off a branch, I'll split some of the river's flow off to go with it.  I don't care if rivers cross each other, but one thing I want to avoid is a river with an impossible or highly unlikely flow, like this:
To keep this from happening, I'll force branches to go somewhere close to the mouth of the river they are branching from.  The farther back they branch, the farther away from the mouth they can go.

I'll start by just creating a branch manually to see if I can get that to work:
That sort-of worked.  There are a couple of problems.  First, the river is hitting the shore at a very oblique angle, which causes problems.  Secondly, the start of the river where it branches off the main river is very narrow.

The latter problem occurs because prior to creating deltas, I've never had a river split off of another river.  I've only had rivers join with other rivers.  Every river starts from nothing, so Dragons Abound always draws rivers starting from a point and growing to their initial width.  What I need to do is add a test to detect when a river is branching off of another river, and then suppress the narrow start.
That seems to have worked.  The problem of the oblique approach to the coast is more troublesome; with an irregular coast and a thick river, making them meet seamlessly in all cases is challenging.  For now I'm going to ignore it and see if I can get the rest of the delta drawn.

My idea for drawing the delta is to start at the point furthest from the coast, pick one or more random spots on existing rivers to branch from, then move a little closer to the coast and repeat.  By varying the number of branches and the space between branches and other variables, I should be able to tweak until I get something I think looks good.  We'll see.

Here's the first attempt, with decorations turned off (so all the rivers are straight):
This looks surprisingly un-terrible.  One problem seems to be that the very short rivers right at the coastline are a mess, so it might make sense to end the river splitting early.
That looks better.  Let me now introduce some noise into the paths of the rivers.
That is starting to look okay.  One noticeable problem is that sometimes the river branches run through other branches.  I'd like to notice when a river hits another river and then terminate it there.  This would create short connectors between branches.  Unfortunately, that turned out to be very difficult to program.  First it was difficult to detect intersections and correct the rivers to merge, and then these short connectors broke all the succeeding delta creation.  Eventually I had a “duh" moment and realized I could keep the short connectors as rivers but drop them from all the subsequent delta processing, and then things more-or-less fell into place.

I also did some experimenting with various parameters for the branching trying to find a combination that would give me something pleasing to the eye and avoiding some of the more obvious problems.  Here are some examples at slightly enlarged size:

These all look pretty good to me, so I'm willing to declare victory on the basic concept.  

The last step is parameterize the code.  Up to this point I've hard-coded the delta onto the longest river on the map, and hard-coded the values for the parameters that control branching, etc.  Now that I've got a working version, I can go through and parameterize the code so that I can have more than one delta and the deltas can have different characteristics.

Here's a kind of interesting result after that work:
Here the generator has randomly plopped a lake down right on top of a delta.  I'm not sure what to think about that.  On the one hand, the code does a pretty good job of rendering the situation.  The northern-most river is implausible because it connects to the lake in two places, but other than that I'm not sure this isn't a real-world possibility.  There's a lot of weird terrain in the world.

Here's another weird one:
What the heck happened here?  Let's look at the river without the delta.
The problem is that the delta algorithm somehow put the mouth of the river on the far left of the delta rather than near the middle.  This occurred because the semicircular delta added to the map intersected with the coastline in more than two spots.  It cuts off a little bit of the bay just above the river bend, and the algorithm puts the river mouth there, causing the acute turn in the river.  The fix is to use the first and the last intersections rather than the first two intersections.
Oops.  Well, maybe not.  Without going into too many details, this happens because the delta happens to get added where the coast wraps around, so the “center" found by the algorithm is in the whole rest of the coast rather than on the delta!  This is actually a rather tricky problem to detect, much less to fix.  After thinking about it for a couple of days, I decided there wasn't a solution that didn't require a lot of painstaking bookkeeping.  So I've decided to simply detect this case and reject this delta.

Of course, in most programming jobs you can't simply decide to skip something because it would be hard.  But a map doesn't have to have a delta. If I decide to add a delta to the map, I can also “undecide" that later on if it is too hard to do.  Of course, if I do that too much I may run into trouble, but I've found that the 80 / 20 rule is a good guide -- I can throw out about 20% of the problem cases without too much negative impact.

Adding the tests to avoid the case above turned up a couple of other bugs in the implementation which took a while to correct.  I also had a few maps that took a very long time to run, so I made one of my periodic performance investigations.  One of the big culprits was the code that tests whether a point is within a polygon.  My implementation was based upon code I found here.  I did some research on efficient point in polygon algorithms, and found an interesting discussion on the topic by Dan Sunday.  Dan suggests that a proper implementation of the “winding number" algorithm is both more correct and faster than other implementations, so I tried a Javascript implementation of his algorithm, based on code by Vlad Lasky that you can find here.  This indeed was about 10% faster than the old algorithm.  

The more interesting discovery is that every once in a while a map takes much, much longer to run than normal.  I suspect this happens when the JIT compiler in the browser has to compile the Dragons Abound code, but I'm not sure, and there doesn't seem to be any easy way to tell.  (Send me an email or a comment if you know how to suss this out.)

That pretty much wraps up deltas for now.  Here's a random map (click for full-size) showing a delta in context:
This is literally just the first random map I generated, but it happens to have cluster islands opposite the mouth of the delta, which is a nice touch.