I've done AI safety grantmaking for the Survival and Flourishing Fund and Lightcone Commons. It's hard. There's an endless amount to know, from which hardware verification mechanisms will survive a dedicated adversary, all the way to which founders have skeletons in their closets that will blow up into destructive scandals. And the best grantmakers - not including me - make mathematical models of the outcome distribution, or of where the marginal dollar stops being highly effective.

AI has been a godsend for all of this. It can research, model, and - as of this year - superforecast outcomes. It can do what would be hours of work for a human in minutes. I found myself using it more and more. But however much I used it, it always felt like I was the bottleneck. The parts I could outsource to AI were sparkling, perfect, exhaustive - but they were all tacked together with shoddy connective tissue formed by the Mark 1.0 human brain. So I wondered if I could automate grantmaking.

I started with the dumbest possible strategy. I assigned the best thing I could think of - a well-crafted US-China pause based on the AI Futures Project's Plan A - an arbitrary value of one million points.^[1] Then I generated about eighty lesser AI safety milestones - from MATS to HPMOR to the RAISE Act to the emergent misalignment paper - and subjectively assigned them some number of points based on how good they seemed relative to Plan A and each other. Then I fed it to the AI to give it a sense for how good a "point" was in this framework. Then I asked it to forecast how many points various grant proposals were worth. The system has since accrued lots of extra epicycles and features and complications, but that's still the skeleton.

As far as I can tell, this dumbest possible strategy worked well. The autograntor gives common-sensical results that match my impressions and the vague gestalt funding order of Lightcone grantors. Its reports surface all relevant information, and its reasoning is clear. The single highest cost-effectiveness grant it's found so far is also the one that got the most interest during the last Lightcone run.^[2] When I tested it on a bottom-tier proposal that no human evaluator even considered, it gave cost-effectiveness numbers two orders of magnitude worse than the genuinely-under-consideration grants that I had tested it on before.

This post will present details for people who want to examine these ideas or build on them further. The current version of the autograntor has four parts: a Milestones document, a FutureSearch connector, a PERCYs module, and a Common Law. I'll start by reviewing each, then give some sample grant analyses, then attach the files you need to run the autograntor yourself.

The Milestones

The milestones are the earliest place that this strategy could fail. Can I really translate my beliefs about what matters in AI safety into a list of milestones, each with a consistent point value?

This combines two questions. First, can I, Scott, generate a set of milestone point values that accurately and consistently conveys my own worldview? Second, if someone else were to translate their own worldview into milestones, would it agree with mine?

I chose to deprioritize the second question. Yes, I have different beliefs about AI safety than other smart people, but this is an inherent problem for my grantmaking, even if I did it manually. If we could solve the first problem, then I could be confident that the autograntor was correctly representing my opinions, and someone else who wanted to use an autograntor could confidently ask it to represent their opinions instead.

To investigate the first question, I asked four AIs - Opus, Fable, ChatGPT, and Grok - to guess how I, Scott, would fill them in.^[3] If all four got similar results, then "Scott's worldview" constrains the point values well; if all of our results were very different, then noise in the translation of a worldview into numbers overwhelms signal.

Two-way correlations ranged from 0.95 - 0.99, which might sound extreme, but some of the milestones were obviously many orders of magnitude better than other milestones - it's not hard to tell that achieving global pause with China is much better than successfully running the Manifest conference.

A better measure is the geometric-mean difference between two AIs across all milestones. This ranged from 1.6x (Opus vs. ChatGPT) to 2.6x (ChatGPT vs. Grok).

Since my true milestone value list came from taking the average of many different methods of calculation, the real variance is something like the square root of these numbers, ie from 1.3x to 1.6x. I consider this good enough to work with.^[4]

I would have liked to have gotten some humans to try to predict my point value within my world-view, but couldn't get anyone to agree to do this (except for one person who agreed but used an AI, unintentionally duplicating my procedure).

Although I don't think the second question (difference between people with different worldviews) matters for the success of the autograntor, I did investigate it via a survey where I asked people to fill in their own values. Nine people agreed, including four people whose names I recognized and who I expected to have good models (other people might also have good models, I just didn't recognize their names). The geometric mean difference between myself and those four people was 2.9x, 4.4x, 6.9x, and 8x, so much larger than the difference between different AIs trying to estimate my view. Outside that artificial limitation, geometric means were even higher - for example, one pair had a geomean difference of 15x! To give an example of where such differences came from, "Harry Potter And The Methods Of Rationality" was an especially controversial milestone - one person rated it as 1 point, another person as 75,000 points.^[5]

Everything below assumes that the Milestones list is good enough to work with.

The FutureSearch Connector