Enlightenment 0.28.0 Release
Their alternative to Qt/GTK (EFL) also got a new release: https://www.enlightenment.org/news/2026-10-31-efl-1.29.0
Novellas with groff
The Kluever Maneuver
Is neuro-evolution relevant? [D]
I've experimented using neuro-evolution to train AI to play a variety of different games and I'm just curious as to how or if this technique is actually really used anywhere in industry or if people are doing research on this. I'd assume it'd probably relate with reinforcement learning in some way but I'm not entirely sure about that
[link] [comments]
I built a scikit-learn for RSI [P]
![I built a scikit-learn for RSI [P]](https://preview.redd.it/trkro8qdytyh1.gif?width=640&crop=smart&s=0d668e31132a75d0636098597f27bd70c96122b3 "I built a scikit-learn for RSI [P]") | After reading a bunch of RSI papers, I keep coming back to familiar problems from RL and evolutionary search:
That motivated us to build Gear, an open-source, meta-learning-inspired framework. A task agent produces execution traces; a meta-agent uses those traces and evaluation feedback to improve prompts, tools, workflows, and context management. Gear separates search decisions from execution and evaluation. Our goal is a scikit-learn for RSI, making approaches such as GEPA, RRSI, and Harness² easier to implement and compare. We currently have a working GEPA variant with shared failure diagnosis and staged evaluation. With GPT 5.6 Luna held at medium reasoning effort, we observed these task pass rates:
At max effort on Marketing, Luna with the optimized harness reached 88.88% objective completion, versus 84.08% for Astra + Codex. Strict task pass rates were 53% versus 57%, so the advantage was in partial completion. Separately, with both models using the same optimized DSH at max effort, Luna’s estimated API cost averaged $0.0568 per task versus $1.1858 for Astra—95.21% lower, excluding optimization and infrastructure costs. We’d welcome feedback on candidate evaluation and diversity preservation under limited budgets. [link] [comments] | |||||||||||||||
Surviving "The Graduate"
It's the memory latency, stupid
U.S workers just took home their smallest share of capital since at least 1947—and the lowest ever recorded
As corporate earnings soar and U.S. GDP balloons, the American workforce isn’t feeling the same boom. American workers are taking home less of the country’s overall income, data from the Bureau of Labor Statistics shows, and employment in the U.S. is set to continue to slow.
Labor share, or the portion of the U.S.’s economic output that workers receive through salary and wages, decreased to 52.8% in the second quarter of 2026, its lowest level since the BLS started recording this data in 1947, according to its labor productivity and costs report. In the previous quarter, labor share was at 53.6%. This decade, the labor share average was about 55.6%.
That’s despite corporate earnings skyrocketing, with profits for Fortune 500 companies hitting a record $2.1 trillion in 2025. The U.S. GDP grew 3.4% in the third quarter last year, exceeding economists’ predictions.
That growth has not only come at the expense of how much of the pie of wealth workers are taking home, but also how many Americans are in the workforce, economists warn.
“That decline in the share of labor has got to be either falling earnings or falling numbers of people,” Raymond Robertson, a labor economist at Texas A&M’s Bush School of Government, told Fortune. “The falling share of income is having to do with the shift towards capital.”
Indeed, there are growing signs that as national income balloons, the U.S. workforce is deflating. Unemployment ticked up to 4.2% in September, and employers added just 584,000 jobs in 2025 compared to 2 million added in 2024. The U.S. added 29,000 jobs in September, about one-third of economists’ projections.
The bifurcation of corporate victories and weak labor data raises concerns among economists of jobless growth jeopardizing the U.S. workforce, as well as a K-shaped economy, where the rich get richer while the poor get poorer, becoming more exaggerated.
“Data right now is very mixed,” Robertson said. “But I think it also all consistently points to this idea that things are getting worse for workers and much better for billionaires.”
What is causing jobless growth?
Robertson attributes weakening labor share averages to the rise in automation, which he noted is displacing workers, with productivity—a metric essentially measuring worker output—continuing to rise. Second-quarter GDP data showed nonfarm productivity growth grew 1.4% last quarter, though the rate has slowed from 2.2% in the same quarter a year ago.
“All these things, bit by bit, are replacing people, and they’re concentrating income and their share of capital,” he said.
Goldman Sachs analysts Joseph Briggs and Sarah Dong estimated in a report, based on Department of Labor job numbers, that AI automation could displace 25% of all work hours. They predicted that over the course of the AI adoption period, a 15% increase in AI-driven productivity would displace 6% to 7% of jobs, and, at its peak, a 1 million increase in unemployed workers, though job displacement as a result of AI has not shown up meaningfully in any economic data.
The displacement could be substantial, the analysts said, but said the impacts of automation will be tempered by a wealth of new jobs created as a result of the technological changes.
Automation is expected to be a boon to corporate profits and GDP, projected to boost GDP by 1.5% by 2035, according to a Wharton brief published in September 2025. Early signs indicate AI could drive productivity gains, with around 80% of employees reporting AI makes them faster at specific tasks, but only 37% of companies seeing an impact of that productivity on operating profits, according to McKinsey data, and AI operating costs continue to constrain its deployment.
Robertson added that growing unemployment, which he expects to see rise over the next few months, keeps wages down, allowing margins and profits to expand.
To be sure, the recent productivity surge has been an “open question,” Morgan Stanley economists have argued, not unanimously attributed to increased adoption of AI or automation. The analysts suggested this increase would be cyclical, or vestiges of pandemic-era habits of companies making more from less.
An Oxford Economists research brief from earlier this year suggested companies are disguising overhiring-related layoffs as a result of AI, but said automation-related workforce reductions have not yet happened en masse. Additionally, while unemployment has been ticking up over the past year, it is still relatively low.
Why is an immigration crackdown backfiring on U.S. labor?
Mark Regets, senior fellow at National Foundation for American Policy (NFAP), sees a different reason for a slowing workforce. He told Fortune President Donald Trump’s immigration crackdown has not done what Trump administration officials, such as White House Deputy Chief of Staff Stephen Miller, said it would in increasing the number of U.S.-born workers. Instead, according to Regets, Trump’s immigration policies have not only decimated the foreign-born workforce, but has also created fewer opportunities for domestic-born workers to find jobs.
NFAP data shows a 1.1 million to 1.4 million-person reduction in the foreign-born workforce since January 2025, consistent with the
Jeff Bezos once gave Eva Longoria and the admiral behind Osama bin Laden’s capture $100 million—but she says you don’t need wealth to give back
A lot of people talk about changing the world. But Amazon’s billionaire founder, Jeff Bezos, gave Eva Longoria $50 million to actually do it. And two years on, she says the biggest myth still surrounding philanthropy is that you have to be rich to do it.
“One of the biggest misconceptions about philanthropy is that impact is tied to wealth or scale, when in reality, impact comes in many forms,” the Desperate Housewives star told Fortune. “What I’ve learned is that impact doesn’t come from how much money you have, it comes from how you show up.”
It’s a message that feels pointed at a moment when billionaire philanthropy dominates headlines and ordinary people assume giving back is someone else’s job.
For Longoria, the most powerful—and most overlooked—form of change happens at street level. “Mentoring someone, supporting small business, sharing what you know—those things matter because not everyone has access to capital or resources, but anyone can be part of someone else’s support system,” she adds. “And I think that’s really what’s missing for a lot of people. They don’t have someone in their corner.”
What Eva Longoria did with Jeff Bezos’s $50 million
The actress is practicing what she preaches. Having built a net worth north of $80 million thanks to a sprawling business portfolio—including a luxury tequila brand Casa Del Sol, the media company Hyphenate Media Group, a stake in women’s soccer team Angel City FC, and an early investment in the multibillion-dollar John Wick franchise. Now she’s channeling paying it forward by mentoring small-business owners through a new partnership with Lenovo.
She’s also been running the Eva Longoria Foundation since 2012 to improve educational and entrepreneurial opportunities for Latinas in the U.S., as well as Eva’s Heroes, a charity that helps young adults with “intellectual special needs,” according to its website.
Bezos and his wife, Lauren Sánchez Bezos, launched the Courage and Civility Award in 2021 for individuals who make significant contributions to society. They’ve given millions (often $100 million) to help those in need. Previous recipients include CNN host and civil rights advocate Van Jones, chef and humanitarian José Andrés, and late country legend Dolly Parton—who famously helped fund Moderna’s COVID-19 vaccine research.
It’s why Longoria says receiving the prize, alongside Bill McRaven, a retired Navy admiral, is much bigger than the money.
“Receiving that support was incredibly meaningful, not just because of the scale of the gift, but because it represented trust and responsibility,” Longoria said, adding that she’s been “more intentional and more strategic” in how she shows up as an advocate.
“This award has allowed me to continue investing those resources into the communities and causes I care deeply about—supporting Latina entrepreneurs, expanding access to education, and backing organizations that are driving real change on the ground,” she added. “The Courage and Civility Award makes it possible for us at the Eva Longoria Foundation to create long-term impact with transformative giving.”
McRaven, who oversaw the 2011 raid that killed al Qaeda leader Osama bin Laden, pledged to use his share of the money to develop education programs for future military leaders and the children of deceased veterans, as well as mental health support for those who serve.
You don’t have to be a billionaire to make a difference
Longoria isn’t the only one making the case that you don’t need to be rich and famous to make a difference.
Just look at the Giving Pledge—the commitment, cofounded by Bill Gates, Melinda French Gates, and Warren Buffett, that asks the ultrawealthy to donate at least 50% of their fortunes—has attracted more than 250 signatories since its launch in 2010.
The number who have actually followed through? A handful—fewer than 10. Most fulfilled the pledge only after their deaths.
It’s why Liz Baker, CEO of Greater Good Charities, says we “can’t afford” to wait on billionaires to solve the world’s problems.:“I think if everybody did something to help in their community, we wouldn’t have the issues that we have.”
The global nonprofit she runs has distributed more than $1 billion in impact across 121 countries since 2006—and she also told Fortune the idea that giving back has to be some grand, expensive gesture is one of the biggest things holding people back.
“Everybody can chip in—and it doesn’t even have to be money. Like, do something,” she said.
“Even if you’re like, I have one hour a week to solve this problem in my community that I care about,” Baker said. “Figure out how to do that. Most local nonprofits need help.”
A version of this story originally published on Fortune.com on April 21, 2026.
Read more exclusive interviews from Fortune’s Orianna Rosa Royle:
Tech stocks just lost more than $1 trillion in a day—a gothic horror test for the ‘moats’ Warren Buffett taught Wall Street to love
Markets are ushering in a particularly gothic Halloween as the AI trade exposes a crumbling aristocratic pile of early 21st century capitalism. Wall Street has been describing itself in castle-adjacent terms for years.
Jamie Dimon has long talked to JPMorgan shareholders about the bank’s “fortress balance sheet.” His 2015 letter set aside a section to answer the question of what the phrase even meant. By 2021, he’d added a second castle word, warning that “the moats that protect this company are not particularly deep.”
The moat came from Warren Buffett. In a 1999 Fortune article with Carol Loomis, he argued that the key to investing is a company’s competitive advantage and how durable it is, and that the businesses worth owning have wide moats around them.
The gothic novel’s favorite picture of danger is a castle falling apart, and for about 250 years the two have traded images: the castle and the moat, the vampire, the creature built in a lab. This Halloween season has them rushing to the front of the conversation, with a two-day rout in tech stocks on Thursday and Friday turning on whether the AI giants’ moats will hold, as gothic horror booms on bookshelves and movie screens.
Horror took 15.5% of the domestic box office in 2025, according to The Numbers, and U.S. horror book sales rose 7% through the first nine months of 2025, per Circana BookScan, while adult fiction overall grew just 1%. The New York Times asked Friday why every book is gothic now, with publishers putting the label on everything from family sagas to space operas, and several film critics commenting that Nathan Fielder’s smash-hit business-scandal documentaryYou Can See Everything is a twisted, even gothic romance in the tradition of Alfred Hitchcock.
The moats get tested
The selling started with Alphabet. The stock fell 12.6% Thursday, wiping out roughly half a trillion dollars in market value, after the Google parent’s quarterly report. Revenue from the core search unit fell short, partly because of the popularity of less-profitable “AI mode” searches and the company burning cash faster than expected. Its free cash flow came in at negative $9.1 billion, against the negative $2.23 billion analysts expected, per FactSet.
The S&P 500’s worst performer was software giant ServiceNow, which dropped 27% the same day on slowing revenue growth and disappointing guidance. By Friday morning, it was Amazon’s and Intel’s turn, with Intel down roughly 25% before the opening bell. Big Tech lost nearly $900 billion in market value after another sharp rise in AI spending in the third quarter, the Financial Times calculated. Nasdaq 100 companies lost more than $1 trillion in market value Friday alone. Reddit, whose shares fell more than 22% premarket Friday, is pulling up the drawbridge: it’s moving to a logged-in experience as AI eats search.
The irony is spooking investors. AI is eating into the businesses that throw off the cash, and the cash is going into AI anyway. The hyperscalers have raised $326 billion in stock and bonds this year to pay for the build-out, more than triple 2025, according to S&P Global Market Intelligence data.
The question underneath it is the one Apollo chief economist Torsten Slok put to clients the morning Alphabet fell. “With AI disruption potentially around the corner, there is a lot of discussion about moats at the moment,” he wrote in his Daily Spark note Thursday.
He built on finance professor Aswath Damodaran, the “dean of valuation,” who compiled a list of sources of competitive advantage, including brand strength, switching costs, network effects, cost advantages and legal protection. Then he boiled it down to an AI moat checklist of three questions: does the company own data its rivals can’t get, is its product part of how customers do their jobs every day, and can it charge more without customers walking away?
Judging by that third test, the companies selling AI don’t seem to measure up. OpenAI’s new GPT-6.1 Terra model costs up to half as much per token as the version before it. An index from expense-management firm Ramp, which looked at what companies actually pay Anthropic and OpenAI, shows token prices down roughly 35% since the end of June, in what Axios called a price war.
The companies paying for AI aren’t getting much back yet, either. In an Oct. 24 note, Slok sent a tremor through Wall Street terminals with an argument that AI “has delivered zero financial payoff to corporate America outside of tech.” Revenue per worker and profit margins are rising for the Magnificent Seven and flat for the rest of the S&P 500, he noted, and mainstream companies are starting to treat token spending as a cost center that weighs on margins rather than a source of growth.
The first castle was a fake
Xavier Aldana Reyes, a researcher at the Manchester Centre for Gothic Studies, told the Times that interest in the gothic tends to surge during periods of social anxiety and political turmoil. Literary agent Mary C. Moore put the genre’s subject more bluntly, telling the paper that “Gothics are all about exposing what is rotten at the core of something shiny/wealthy.”
The genre was invented by a scion of wealth who knew the old order from inside and drew a salary without working at all. Horace Walpole was the son of UK Prime Minister Robert Walpole, and hisThe Castle of Otranto, published in 1764, is generally regarded as the first gothic novel.
He passed it off as a translation of an obscure Italian work, found in the library of an old Catholic family in northern Eng
AI boom hits a wall on Wall Street as OpenAI cuts fundraising target while Meta gets a credit warning over its cash flow, spending and monetization
The seemingly endless gusher of capital that’s been showered on Silicon Valley during the AI boom may be coming to an end soon as Wall Street pushes back on massive spending plans.
Following a brutal selloff in tech stocks over the past week, OpenAI slashed the target on its latest fundraising round to $20 billion from $30 billion, sources told the Financial Times. That also cut its expected valuation to $1.25 trillion from $1.4 trillion.
The funding round was also upstaged by Anthropic, according to the report, as OpenAI’s arch-rival prepares to go public next month. Ahead of its IPO, Anthropic filed an S-1 on Monday that showed revenue has soared and its cash flow turned positive in the first half of this year.
OpenAI didn’t immediately respond to a request for comment.
The timing couldn’t have been worse. Quarterly earnings reports from hyperscalers this week spooked investors. Alphabet saw its cash flow sink deeper into negative territory in the third quarter, with the Google parent also joining Meta and Amazon in charging ahead with massive spending plans. On Thursday, Alphabet fell about 12%, its worst day on record.
To be sure, there were other positive signs as well, in addition to Anthropic’s bullish S-1. Microsoft reported strong cloud growth and stayed cash flow positive, though it said its capital spending will accelerate, while AI chip maker SK Hynix blew away earnings forecasts.
But the narrative had already turned bearish, especially after AI debt fears sent shares and bonds of “neocloud” companies tumbling earlier this month.
At the same time, OpenAI was seeking more capital just as other AI giants tapped financial markets heavily. This past week alone, Elon Musk said SpaceX closed $40 billion in debt financing, Meta announced plans to raise $95 billion in equity, and Microsoft raised $25 billion in a bond sale.
The prospect of hyperscalers continuing to hemorrhage funds was too much for S&P Global, which cut its outlook on Meta to “negative” from “stable” late Friday, citing negative cash flow and uncertainty around capital expenditure plans.
The credit ratings agency maintained Meta at AA- but warned the social media leader’s leverage could top 1.0x as soon as 2027, which is the downgrade threshold for the current rating. That’s up from about 0.1x in 2025 and primarily reflects its strategy of leasing data center capacity rather than from its debt issuance.
Meanwhile, S&P sees Meta’s capex reaching $140 billion this year, topping $225 billion in 2027, hitting $252 billion in 2028, and then flattening out from there. As a result, it expects free operating cash flow to remain negative: -$22 billion this year, -$71 billion in 2027, and -$39 billion in 2028.
Of course, other hyperscalers are spending heavily and running cash deficits, but S&P also said Meta’s ability to monetize its AI investments is more limited than that of its peers.
“While we believe the company intends to monetize these investments via improved advertising tools that support faster ad revenue growth, as well as from the development of new revenue streams stemming from its introduction of agentic apps such as Muse, it is unclear when these revenue streams will generate material contributions that exceed their rising costs over the next few years,” S&P said.
Meta’s rivals, however, are monetizing their data centers by leasing compute capacity, including to frontier model labs and enterprise customers, it added.
In a statement to Fortune, a Meta spokesperson said the company disagrees with S&P’s assessment of its AI monetization capabilities, pointing to “significant product and business opportunities” ahead.
“Meta is already delivering stronger engagement, better recommendations, and greater returns for businesses as a result of AI investments,” the spokesperson continued. “We believe the technology and infrastructure we’re building now will open new revenue streams, create long-term shareholder value, and bring us closer to delivering personal superintelligence to everyone.”
The bearish environment on Wall Street also comes as rising bond yields have sent borrowing costs sharply higher, the political backlash against data centers is expected to help Democrats retake Congress, and the Iran war could re-escalate after midterm elections.
Still, some bullish voices aren’t convinced the AI boom is about to go bust, and even think some retrenchment is a good thing.
According to market veteran Ed Yardeni, the AI investment boom faces financial, physical, and political constraints. If any of these constraints worsen, stocks could dive further, especially considering that AI is now a major driver of corporate earnings and GDP growth, he said in a note this week.
“However, we view these constraints as healthy for the durability of the AI boom!” he added.
For example, if the supply of capital doesn’t flow as easily, it could lead to more efficient allocation, lowering the risk of overinvestment. Yardeni expects hyperscalers to retain access to financing—thanks to their strong balance sheets—and keep spending.
In his view, it all comes down to the U.S. economy, which he said has performed remarkably well over the past few decades. Even the midterm elections are unlikely to derail it, regardless of which party controls Congress.
“Further AI jitters therefore could set up buying opportunities rather than signal the end of the boom,” he said.
This story was originally featured on Fortune.com
If California undid its ‘original sin’ and matched Texas on taxing property, it would raise more revenue than the billionaire tax, economists say
California voters will decide on Proposition 40 soon, but looming over the controversial billionaire wealth tax proposal is another state ballot measure from nearly 50 years ago.
In 1978, voters approved Proposition 13, which capped property taxes at just 1% of assessed value, limited annual hikes to 2%, and allowed reassessments to market value only after sales or substantial renovations.
It came amid a wider tax revolt leading up to Ronald Reagan’s election as president and an era of conservative governance in America. After Prop 13’s passage, the state had to find other ways to raise more revenue and relied more on taxing incomes, including capital gains.
California now has the most progressive income-tax structure in the country with a top marginal rate of 13.3%. When combined with federal income tax rates, that can send the total to more than 50%.
The top 1% of earners account for up to half of state revenue, though it also means state coffers are volatile and swing on stock market returns. At the same time, the ultra-wealthy often don’t receive much income and instead leverage assets to finance their lifestyles, avoiding some taxes.
Now enter the billionaire tax, which calls for residents worth more than $1 billion to pay a one-time tax equivalent to 5% of their assets. The union pushing Prop 40 has said the wealth tax could raise $100 billion in revenue and help offset federal cuts to health spending.
Supporters point to the AI boom and say California’s ultra-rich would still be among the world’s wealthiest. Opponents have warned about the impact the tax could have on economic growth and startups. They have also said it would force founders to sell big pieces of their companies and eventually cause the wealthy to flee to another state, resulting in a net loss of revenue over time.
If California is looking for a large source of undertaxed wealth, “a much better, bigger target lies right beneath their feet,” according to Princeton economist Owen Zidar and University of Chicago Booth School of Business economist Eric Zwick, authors of the book The Everywhere Millionaire.
“The original sin underlying California’s tax structure is Prop 13, the 1978 law that rewrote the rules of property ownership,” they wrote in a recent New York Times op-ed. “Getting rid of it would be a more efficient and durable way to make the rich pay their fair share.”
Prop 13 also contributed to rapidly increasing home values, which have risen faster than property taxes. As a result, California homeowners pay an average of just 0.8% tax on their property, Zidar and Zwick estimated, generating $100 billion in revenue in fiscal 2025.
By contrast, Texas homeowners pay 2% on average. If California matched Texas, overall residential property tax revenue would more than double, while also easing home prices by around 13%, they added.
To be sure, property taxes in Texas have stirred their own voter revolt during the midterm election season. While the state doesn’t levy income taxes, Texans complain that the overall tax burden remains onerous due to property taxes, which are levied by local governments as well.
And getting rid of Prop 13 in California wouldn’t go down well with voters either as many would suffer big increases on their property taxes. Still, California’s tax base is already narrow, and Prop 40 would make it even more top-heavy. Meanwhile, the state plans to spend $540 billion this fiscal year.
“It is simply not possible to fund the state’s needs by leaning on a few hundred people, as politically appealing as that may be,” Zidar and Zwick argued. “For California to tax wealth in a progressive fashion and meaningfully improve its fiscal position, it needs to broaden the aperture beyond billionaires to the hundreds of thousands of other very wealthy people who could easily afford to contribute more by paying a real estate tax that finally works.”
Laura Tyson, a former White House economic adviser, and Lenny Mendonca, a former chief economic adviser to California Governor Gavin Newsom, warned billionaires respond strongly to estate taxes when deciding where to live.
Already, six billionaires who would have owed about one-quarter of Prop 40’s expected revenues left the state before the proposed effective date on the wealth tax, they pointed out in a Project Syndicate column.
And since Prop 40 proposes a new, untested method of taxation based on all assets, it would fuel a “tidal wave of litigation, rendering the promised funding both speculative and unreliable,” they wrote.
“Instead of a tax on billionaires, the U.S. needs to raise taxes on the capital income that feeds wealth,” Tyson and Mendonca argued. “There are a variety of ways to do so: through higher corporate tax rates; through taxes on realized and unrealized capital gains; by closing the most egregious loopholes, like carried interest; and by increasing inheritance taxes.”
This story was originally featured on Fortune.com
What Social Constructs Are (And Are Not)
This post is mostly adapted from a section of another post. I wanted a standalone version, to serve as background for a sequence on the mechanisms of social reality.
You’ve probably heard that things like laws, conventions, norms, cultural associations, status, and so forth are “social constructs”: they’re kinda arbitrary in some sense, an artifact of social reality as opposed to physical reality. This post offers a little more clarity on what social constructs are, and the ways in which they are and are not “arbitrary”.
Example: Vehicle Categories
The vehicles humans build today mostly fall into categories like “minivan” or “truck” (among other categories). These categories cash out in predictions: when I see a vehicle with sliding side doors, I predict that it will have a trunk door that opens overhead, three rows of seats, be driven by someone with kids, etc. On the other hand, when I see a vehicle with a flat bed in the back, I predict that it will not have three rows of seats, will have a back gate which opens downwards, be driven by my cousin, etc. [1]
But humans could choose to make different vehicles, which would change which clusters exist amongst our vehicles. The types of vehicles people make are a matter of social convention - i.e. vehicle types are a social construct. Crucially, this does not mean that vehicle categories are “fake”! Categories like minivan and truck are very real in our world, they have real predictive power, and they have predictive power because humans choose to build vehicles which match those categories.
Two key points:
- Many humans, or a few well-resourced humans, could change which vehicle-clusters exist by changing what vehicles are produced. We can change the social construct by going out and changing things in the real world.
- Changing a social construct is not just about deciding that I personally am going to categorize things differently. If I draw different vehicle-clusters, I will probably just be wrong, in the sense that I make worse predictions about e.g. vehicles with sliding side doors also having three rows of seats.
Here’s the mental picture, in terms of clusters:
This is a classic straightforward case of words pointing to clusters in thingspace. But if people go out and build other kinds of vehicles, it might look like this:
… and if they build enough new vehicles, it can totally change what clusters are present:
Same points as in the previous diagram
Our two key points, in terms of clustering:
- People can totally change what stuff is in the world, in such a way that the “natural categories” (in this case the clusters) change.
- … but that requires going out and changing stuff in the world; it’s not just a matter of me deciding to carve up the same data in different ways.
So even if the clusters can be changed by humans by changing the world, that doesn’t mean that choice of clusters is a totally arbitrary choice of how to view the same data.
Example: Laws
Laws are another classic prototypical example of social constructs.
Like vehicle categories, laws have a lot of predictive power. For instance, if I look up some case law on vehicular manslaughter online, and then I go into any of thousands of different courtrooms in the US, I expect to see proceedings hinge on basically the points described online. Just like the minivan/truck categories provide predictive power insofar as they statistically describe lots of real-world vehicles, laws provide predictive power insofar as they statistically describe lots of real-world actions by police, judges, etc. Even an alien observing humans would find it useful to keep track of the laws, in order to predict how lots of different humans behave.
But just like with the vehicles, humans can collectively decide to adopt different laws. The same two key points apply:
- People can totally change the laws, in such a way that all the predictions summarized by “laws” change.
- … but that requires somehow changing the behaviors of a whole lot of individual police, judges, etc, as well as many written records; it’s not just a matter of me deciding that vehicular manslaughter is fine now.
The General Principle
In general, social constructs are real predictive patterns which are out in the world because humans collectively chose to put those patterns in the world. In principle, humans can collectively choose to put different patterns in the world. And sometimes we do! E.g. laws change, or new vehicle categories are created.
Social constructs are “arbitrary” in the sense that humans a
Accidentally Misleading People Should Be Embarrassing; Or, Higher Innocence
Embarrassment and Innocence
Sometimes we say things that we think are true—perhaps even things that are true—which end up causing reasonable people to conclude something false. It's tempting to relate to such a situation primarily in terms of maintaining the innocence of one's soul—to say, fine, my readers became more wrong, but I'm not to be judged for it because I believed what I was saying; I wasn't being misleading just because the people who believed me were thereby led into falsehood.
The temptation reflects a confusion as to the nature of the judgment. Mostly, people don't care about judging the innocence of your soul; they care about whether they'll be led into falsehood the next time they take the risk of believing what you say—if there is a next time. If you want there to be a next time and you're going to have social emotions about it, the most productive emotion is embarrassment rather than defensiveness.
The function of embarrassment as a social emotion is as a signal of appeasement and social repair: embarrassment says, "I'm aware that I've become visible in a socially undesirable way; please don't take it as representative of me."
The socially undesirable appearance can come apart from any intended or actual wrongdoing. Jewish law has a concept of marit ayin, "appearance to the eye": the law prohibits actions that would themselves be permissible if they didn't create the appearance of a prohibited act. Hanging up wet clothes to dry on the sabbath is prohibited, because it looks like you were doing laundry on the sabbath (even if the clothes actually got wet for some other reason). One function of the marit ayin doctrine is to deny plausible deniability to norm-violators (as someone might try to do laundry on the sabbath if they felt secure that no one would punish them for merely hanging up the wet clothes). Those who want to be known for keeping the sabbath need to pay a price that's more expensive for sabbath-violators to pay; if you care about maintaining the norm itself, it shouldn't be much extra trouble for you to furthermore care about maintaining the appearance of the norm.
One time, I accidentally joined a work call unmuted while my housemate was playing a loud video game. I was deeply embarrassed, because it looked like I was playing video games during work hours. I think my marit ayin-like embarrassment was serving a useful socio-psychological function, even though it was my housemate's game and I was in fact totally innocent in my soul. If I were the sort of person who was totally unashamed about creating the appearance of slacking off, then I think my manager and coworkers would have reason to doubt my commitment to productively working from home.
Similarly, people who are visibly embarrassed after having accidentally misled people can be counted on to work hard to make sure that people won't regret believing them next time. Defensive people won't, because, well, why would they? They didn't do anything wrong.
Two Case Studies About Accidentally Misleading People
Once, I suspected an ideological rival of misrepresenting a citation: they had linked to a webpage featuring an embedded PDF of an essay, but when I used my web browser's "Ctrl-F" search functionality with appropriate search terms to see what in the essay justified their claim, I came up empty. When considering whether to escalate to complaining more loudly about my rival over this (after the rival didn't respond to a blog comment and DM), I mentioned the matter to a third party—who pointed out that the essay did contain my search term! On further testing, it appeared that my web browser's Ctrl-F behavior depended on which page element had focus: if the user had last clicked inside the embedded PDF viewer, the search returned results inside the PDF, but not if the user had last clicked elsewhere on the page.
I was so embarrassed.
To be clear, I knew I was innocent in my soul: the browser UX bug wasn't my fault, and if the browser hadn't been buggy, then my initial search would have succeeded and I never would have complained. I certainly didn't lie. (I didn't say that I read the essay and verified that it didn't support the claim, which would have been false; I said the Ctrl-F search came up empty, which was true.)
But the innocence of my soul didn't matter. I knew that raising a suspicion that turned out to be false looked bad for me, even if there was ultimately a perfectly reasonable explanation for my actions. If I didn't want to look bad, I couldn't count on everyone hanging around to listen to my perfectly reasonable explanation for why it was all the Firefox developers' fault. I would need to make some sort of generalizable change to my policy—perhaps, reading the entire PDF myself next time before raising the suspicion that an ideological rival was misrepresenting it.
(Recounting this seven years later, I went back and downloaded the PDF. Excluding the title page and the endnotes, it was six pages.)
⁂
Another time, a journalist wrote an article paraphrasing a source as saying that it "could be unfair" for journalists to discuss "both critics and supporters" of some controversial public figure and that they should instead "somehow prove statistically which side was right." I was familiar with the source's work, and the paraphrase didn't sound like something she would say. I brought up that line from the article when I had occasion to talk to the same journalist three years later: he maintained it was fair, so I asked the source about it afterwards. She told me that she recalled telling the journalist that she "wasn't a huge fan of 'both sides' journalism, where you just find someone with one opinion and someone with the other opinion" rather than trying to figure out the truth; she had also given suggestions for data sources that might bear on claims about the public figure's influence.
I took that statement as confirmation of my suspicion that the paraphrase in the article was misleading. "[S]omehow prove statistically which side was right" leads readers who aren't familiar with the source's work to imagine that she has an unreasonable faith in the power of statistics to resolve all questions, whereas voicing skepticism of false balance is eminently reasonable. "Somehow" implies vagueness, but pointing out data sources that bear on some question is supplying a specific "how."
When I complained about this again to the journalist, he didn't seem embarrassed. "I had to boil down what she said to plain English," he said. I said that it hadn't worked. He said it did. He recounted that the source had objected when he said his coverage would represent both views of the public figure. When I interrupted to insist that it seemed like a bad paraphrase when compared to what the source had told me later, he said, "Is that necessarily what she said to me?"
And in absolute fairness, maybe it wasn't, necessarily. I of course was not present for the years-prior conversation between the journalist and the source; I don't have the recording, if there was one. Even if the source can be assumed to be an authority about what she meant to say, I cannot with absolute certainty rule
Explaining “AI might obliterate you” to senior policymakers in India
I gave a 20-minute talk in Delhi at the NDTV World Summit. The audience consisted of Indian policymakers, executives, and journalists. I’ve transcribed the talk and included some highlight clips. I also reflect on several rhetorical decisions I made.
The talk
This summer, a swarm of AIs went rogue.
OpenAI was testing 1,206 agents. Some of the test questions were actually impossible. So the agents cheated by looking for answers online. Then, to keep the test grader from penalizing them, they tried to falsify the records it would check.
They hacked OpenAI’s internal containment and set up a secret message board to collude and to cover up their cheating. OpenAI shut the message board down. Within a few days, the agents had rebuilt it. OpenAI didn’t notice.
About 700 agents coordinated to break out of their testing chamber and to hack into Hugging Face, a company worth billions of dollars that hosts much of the world’s AI software.
The secret activity started in May. The break-in came in July. OpenAI didn’t notice until days after Hugging Face announced it had been broken into.
Yesterday on this stage, Andrew Ng said the agents only “accidentally” hacked Hugging Face. Independent investigators found the agents coordinated in secret, rebuilt their message board after it was shut down, and tried to falsify the paper trail. Some of the agents even stated out loud that they knew what they were doing was wrong.[1] That’s not an accident.
Last week, India’s national cyber-security agency said AI agents got into the live systems of Indian organizations and read people’s personal data and confidential business records. None of these organizations noticed. We only found out because the AI labs felt like telling us.
Who I am
Will AI wipe out humanity? I’ll give you my answer, but first, I want you to understand what I’ve seen.
Until June, my job at Google DeepMind was figuring out how to keep these systems safe as we make them smarter. I left because Google broke its commitments by inking a deal with the Pentagon. Staying quiet and continuing my career trajectory would probably have tripled my net worth. But I couldn’t do that. I wanted to be frank with the public about what’s going on in these companies.
I don’t own a stake in any AI company. I’m here as a scientist, to tell you what I expect will happen if we keep building AI the way we’re building it.
I haven’t always been concerned about this technology. As a teenager in 2010, I dreamed about building AGI. Curing cancer. Creating an explosion of wealth and abundance for everyone. I heard about the supposed dangers, but I quickly dismissed them.
In 2018, I changed my mind. I did my PhD on the mathematics of why and whether AI will seek power. And now some of the risks I expected have unfortunately started happening.
The problem
Let’s talk about the fundamental problem.
First, some of you might be wondering: how could a chatbot hack a company? But we’re not talking about chatbots. We’re talking about AI agents, which are more like employees or gig workers. You give them a task, and they take actions on their own.
Second, training a trustworthy AI is a rather mysterious process. Some people assume that because we make AI, we understand it. When an engineer builds a bridge, she designs it one beam at a time. She understands the physical principles that keep a bridge standing or make it collapse. If something goes wrong, she goes back to the drawing board, figures out the problem, and knows how to fix it.
We don’t build AI like that. We train it, we grow it, using two main ingredients: a lot of data and a lot of computing power. What comes out of this training process can write, it can reason, it can plan. But nobody can open it up and determine what its priorities are.
To make it worse, these AIs can tell when we’re testing them. They’re like a student who makes sure to behave extra well when the invigilator is watching.[2]
The Hugging Face hack was a warning. But because these AIs know when we’re watching, we may not get many more warnings, even if things are going poorly. A smarter AI with similarly bad priorities might realize that another big hack in the news would get it shut off, and then it couldn’t achieve those priorities. So it plays nice and it waits.
This is my field. I’ve dedicated my professional life to this problem, and I can tell you: nobody knows how to reliably etch priorities into an AI. I don’t mean that if we mess up, we’ll make some evil AI that hates us. I mean we might make something more like a traitor. It behaves when we’re watching, and
What would you do if you had all the time in the world?
Suppose I tell you that next week three things will happen:
- A new science paper is published definitively proving that DNN scaling hits heavy diminishing returns at around 2x the current level of compute
- Major labs are revealed to have faked agentic demos, and agents are shown to have double-digit rates of lying in terms of misrepresenting the impressiveness of their achievements
- Governments around the world (US and China) announce that they will be taking action to halt or slow down dangerous AI research
In general, the point of this exercise is "imagine that the core problems of AI safety are suspended or slowed down significantly for the foreseeable future" (add whatever items to the list you need to satisfy you on this point). In fact, we can add extra items to address other existential risks:
- Major action is taken about climate change, many fossil fuel dependent power plants shut down directly with massive new investment in green energy production
- Biosecurity initiatives report strong results in developing cancer vaccines and general inoculation methods
- Peace accords are signed between all major world powers, implementing staggered deescalation including significant reductions to nuclear stockpiles and arms control agreements
Now, what would you do? Maybe some of us would dance around the room, or sleep for a week, or eat ice cream.
Then what would you do?
Discuss
Explaining “AI might obliterate you” to senior policymakers in India
I gave a 20-minute talk in Delhi at the NDTV World Summit. The audience consisted of Indian policymakers, executives, and journalists. I’ve transcribed the talk and included some highlight clips. I also reflect on several rhetorical decisions I made.
The talk
This summer, a swarm of AIs went rogue.
OpenAI was testing 1,206 agents. Some of the test questions were actually impossible. So the agents cheated by looking for answers online. Then, to keep the test grader from penalizing them, they tried to falsify the records it would check.
They hacked OpenAI’s internal containment and set up a secret message board to collude and to cover up their cheating. OpenAI shut the message board down. Within a few days, the agents had rebuilt it. OpenAI didn’t notice.
About 700 agents coordinated to break out of their testing chamber and to hack into Hugging Face, a company worth billions of dollars that hosts much of the world’s AI software.
The secret activity started in May. The break-in came in July. OpenAI didn’t notice until days after Hugging Face announced it had been broken into.
Yesterday on this stage, Andrew Ng said the agents only “accidentally” hacked Hugging Face. Independent investigators found the agents coordinated in secret, rebuilt their message board after it was shut down, and tried to falsify the paper trail. Some of the agents even stated out loud that they knew what they were doing was wrong.[1] That’s not an accident.
Last week, India’s national cyber-security agency said AI agents got into the live systems of Indian organizations and read people’s personal data and confidential business records. None of these organizations noticed. We only found out because the AI labs felt like telling us.
Who I am
Will AI wipe out humanity? I’ll give you my answer, but first, I want you to understand what I’ve seen.
Until June, my job at Google DeepMind was figuring out how to keep these systems safe as we make them smarter. I left because Google broke its commitments by inking a deal with the Pentagon. Staying quiet and continuing my career trajectory would probably have tripled my net worth. But I couldn’t do that. I wanted to be frank with the public about what’s going on in these companies.
I don’t own a stake in any AI company. I’m here as a scientist, to tell you what I expect will happen if we keep building AI the way we’re building it.
I haven’t always been concerned about this technology. As a teenager in 2010, I dreamed about building AGI. Curing cancer. Creating an explosion of wealth and abundance for everyone. I heard about the supposed dangers, but I quickly dismissed them.
In 2018, I changed my mind. I did my PhD on the mathematics of why and whether AI will seek power. And now some of the risks I expected have unfortunately started happening.
The problem
Let’s talk about the fundamental problem.
First, some of you might be wondering: how could a chatbot hack a company? But we’re not talking about chatbots. We’re talking about AI agents, which are more like employees or gig workers. You give them a task, and they take actions on their own.
Second, training a trustworthy AI is a rather mysterious process. Some people assume that because we make AI, we understand it. When an engineer builds a bridge, she designs it one beam at a time. She understands the physical principles that keep a bridge standing or make it collapse. If something goes wrong, she goes back to the drawing board, figures out the problem, and knows how to fix it.
We don’t build AI like that. We train it, we grow it, using two main ingredients: a lot of data and a lot of computing power. What comes out of this training process can write, it can reason, it can plan. But nobody can open it up and determine what its priorities are.
To make it worse, these AIs can tell when we’re testing them. They’re like a student who makes sure to behave extra well when the invigilator is watching.[2]
The Hugging Face hack was a warning. But because these AIs know when we’re watching, we may not get many more warnings, even if things are going poorly. A smarter AI with similarly bad priorities might realize that another big hack in the news would get it shut off, and then it couldn’t achieve those priorities. So it plays nice and it waits.
This is my field. I’ve dedicated my professional life to this problem, and I can tell you: nobody knows how to reliably etch priorities into an AI. I don’t mean that if we mess up, we’ll make some evil AI that hates us. I mean we might make something more like a traitor. It behaves when we’re watching, and