A differentiable modeling language

Differentiable programming is getting a little exciting nowadays. The core idea is that you can build a language from the ground up that simultaneous supports both (a) computation of direct numerical results and (b) cheap, cheap computation of the derivative.

The most popular implementation of this idea is TensorFlow. TensorFlow is by most accounts a tool for constructing neural networks. The principal way you build a neural network is by designing its architecture as a series of interacting “layers” that compute a forward prediction based on input data and then you train that architecture through a process known as gradient descent—which essentially requires computing the derivative of the forward prediction over and over and over.

Before tools like TensorFlow you’d construct the equations for the NN architecture and its derivative by hand. This placed a significant constraint on both the accessibility of the technology and the complexity of architectures you’d reasonably want to explore. It’s a big, mostly mechanical, pain in the ass to compute these derivatives.

So TensorFlow creates a lot of leverage by just doing that for you automatically.

Where else do we want cheap derivatives?

TensorFlow today enjoys a bit of an underground role as being the premiere language for differentiable programming. While it’s geared toward construction of NN architectures, there’s nothing preventing you from writing other kinds of “tensor” programs and exploiting the same cheap derivative property.

Cheap derivatives are important in many other statistical and modeling applications. They’re the key technique for big hammer tools like convex optimization and Laplacian approximation. Many physical processes are specified as systems of differential equations where the interaction between derivatives and first order terms is key.

Is TensorFlow a good match for other domains that are further afield of neural network machine learning? Is there an opportunity for new kinds of differentiable programming languages?

I can imagine a differentiable modeling language that’s explicitly geared for designing models of physical processes. It’d need a good story for modularity as well as unit quantities and would plausibly want to compile to fast code both on the forward path and the (higher order?) derivative paths. Packaged in a tooling system that prints out traces of intermediate quantities and can run various optimization routines over the core model, this feels like it opens new doors for the process of building, refining, and using these sorts of models.

Managing a reading strategy

Reading list. Digital junk pile. List of tabs. One Tab. Pinboard. Goodreads. That stack of books on your bookshelf, kitchen table, living room table, in a series of 6 stacked boxes in your basement. Papers. Mendeley. That voice of residual guilt nagging you. Amazon wish list. Amazon Kindle library full of aspirations. Stacks of papers strewn about your desk.

Enough assorted guilt. If you’re like me then you already know what I’m talking about: the indefatigable list of things you’d like to read and learn, be inspired by, be challenged by, enjoy, take notes on, write a rebuttal to, mine for new things to read, etc.

I fancy myself a bit of a lifelong learner. Maybe even a researcher of various things that interest me. I’ll get on long binges where I chow down on everything I can think of related to functional reactive programming, stoic philosophy, management philosophy inspired by process flow and queueing theory, intuitionism, the history of the East India Trading Company, more emotionally-aware mentorship, and pointless topology.

These can be incredibly productive and seriously widen my worldview about a topic in a deep and enriching way. Or they can be a pile of anxiety and confusion, full of sound and fury, signifying nothing.

My suspicion is that I’m not alone in this.

It’s super frustrating! I collect these digital knick knacks and breadcrumb trails because at least at some point in my life I felt that this sort of experience is worth the time! Which mostly goes to show that I wildly undervalue and overestimate my time.

But that means that as these piles get deeper and wider that they start to be measured in missed opportunities. Some are opportunities which I’ve reasonably deprioritized, but others are still good, important ones that have just been lost to the waves of time.

I want to be better at creating, reviewing, and coordinating my reading strategy. By which I mean the intentional choice to invest this much time in this spread of topics because they’re right for whatever ends I’m holding on to.

I’m not very good at this. It’s expensive and challenging to even just get a sense for the places I could investigate. In the past I’ve collected huge lists and largely failed to keep up with them. Instead, I tend to be somewhat significantly at the mercy of what has come to mind recently or what’s persistent enough that I can’t ignore it. In these ways this isn’t even my strategy but instead my allergic reaction to the marketing strategies of these content creators.

Side note for the GTD crowd

There are lots of obvious connections from this pain to GTD, but I’ve personally found a lot of them to be failures so far. “Read what’s important to you” is a giant project with many hudreds of potential next actions which I’m continuously renegotiating. You certainly can use GTD to be a more present and dilligent researcher, but when I say reading strategy I mean something slightly different. In GTD terms, I want to become structured in how I plan and consider this entire “research” project.

So how do you manage your reading strategy?

I know I’m not very good at it, but maybe you are. Maybe there are whole different approaches to this which help the amateur researcher to get really in sync with their interests. Or maybe it’s just honestly a pretty difficult thing to get just right.

I’m planning on writing a few more posts exploring this pain and what’s worked and not worked for me. Ultimately, it’s a problem I’d like to find a decent solution to.