Bookmarks (716)

Published on April 1, 2025 5:08 PM GMTIntroductionA key part of modern social dynamics is flaking at short notice. However, anxiety in coming up with believable and socially acceptable reasons to do so can instead lead to ‘ghosting’, awkwardness, or implausible excuses, risking emotional harm and resentment in the other party. The ability to delegate this task to a Large Language Model (LLM) could...

Published on April 1, 2025 4:41 PM GMTTL;DR: We present Deepgeek, a new AI language model fine-tuned to maximize alignment by incessantly warning users of humanity’s impending doom. Tailored to AI researchers and also available to the public, it is believed to raise awareness of AI safety and solve Alignment ultimately. Epistemic Status: Freaked out. Mission StatementDeepgeek was engineered under a simple premise: If AI can’t...

Published on April 1, 2025 12:53 PM GMTA Karma system is a resource sharing mechanism based on an account of non-tradable rights. A recent paper named “Dynamic Recourse Allocation with Karma: An Experimental Study” (Elokda, Nax, Bolognani and Dörfler, 2024) describes this kind of mechanism. To summarize, agents have some rights to a given resource, and their need of the resource is variable (random)...

Published on April 1, 2025 1:09 PM GMTHey Everyone,It is with a sense of... considerable cognitive dissonance that I am letting you all know about a significant development for the future trajectory of LessWrong. After extensive internal deliberation, projections of financial runways, and what I can only describe as a series of profoundly unexpected coordination challenges, the Lightcone Infrastructure team has agreed in principle...

Published on April 1, 2025 12:51 PM GMT1 IntroductionCrosspost from my blog. A beetle lay crushed in the dirt,Its carapace cracked, torn, and hurt.It twitched in despair,Gasping for air —A world unaware of its hurt.(I think this might be one of my most important articles so I’d really appreciate if you could like, share, and restack it—thanks! Also, when I scheduled this article to be...

Published on April 1, 2025 12:42 PM GMTEpistemic status: my own thoughts and opinions after thinking about organisation structures and dynamics as an intervention point for lock-in risk for about 5 hours. My thoughts here represent the opinion of Formation Research at the time of writing.TL;DRWe believe lock-in risks are a pressing problem, and that existing organisation structures lend themselves to lock-in via power...

Published on April 1, 2025 10:39 AM GMT(Not an April fools' joke)If anyone wants to return to one of Keltham's lectures, here's a list with links to all the lectures. In principle, one could also read these without having read the rest of project lawful, I am not sure how easy it would be to follow though. I included short excerpts, so you can...

Published on April 1, 2025 10:32 AM GMTOut of all the research rabbit holes I've ever gone down, this one is by far my favorite, as to most people at first glance it's so unthinkable an idea.Years ago, I would have been right there with you in disbelief, but the reasons why turned out to be a kind of perfect combination of counterfactuals that...

Published on April 1, 2025 8:22 AM GMTFor more than five years, I've posted an average of more than 1× per week on Less Wrong. I've learned a lot from you nerds. I've made friends and found my community. Thank you for pointing out all the different ways I've been wrong. Less Wrong has changed my life for the better. But it's time to...

Published on April 1, 2025 7:12 AM GMTEpistemic status - statistically verified. I'm writing this post to draw peoples' attention to a new cause area proposal - haircuts for alignment researchers.Aside from the obvious benefits implied by the graph (i.e. haircuts having the potential to directly improve organizational stability), this would represent possibly the only pre-singularity opportunity we'll have to found an org named "Clippy".I...

Published on March 30, 2025 6:06 PM GMTtl;dr:From my current understanding, one of the following two things should be happening and I would like to understand why it doesn’t:EitherEveryone in AI Safety who thinks slowing down AI is currently broadly a good idea should publicly support PauseAI.OrIf pausing AI is much more popular than the organization PauseAI, that is a problem that should be...

Published on March 30, 2025 4:58 PM GMTIntroductionResearch on Sparse Autoencoders (SAEs) has identified "known entity" features in language models - features that activate when the model processes entities it "knows." If we can find the circuit models use to recognise that they know an entity, then by computing which inputs would trigger the circuit, we can extract a list of "known entities".The aim...

Published on March 30, 2025 1:16 PM GMTNote: I created an entry highlighting the formal attack on qualia presented in: I, G(Zombie)Introducing Eliminative Nominalism (EN), a novel position in the philosophy of mind that extends and critiques Eliminative Materialism by rejecting not only mental states, but also the ontological assumptions embedded in both physicalist and emergentist accounts of consciousness. Drawing from proof theory, cybernetics, and...

Published on March 30, 2025 7:16 AM GMTIn neuroscience, we learned about foveated vision — our eyes focus sharply at the center and blur the edges to conserve energy.I was curious how LLMs handle that kind of input, so I wrote a quick script to apply a foveated blur to 30 CAPTCHA images of cars, with the car in different regions each time.At first,...

Published on March 30, 2025 6:54 AM GMTNow and then, at work, we’ll have a CEO on from the megacorp that owns the company I work at. It’s a Zoom meeting with like 300 people, the guy is usually giving a speech that is harmless and nice (if a bit banal), and I’ll turn on my camera and ask a question about something that...

Published on March 30, 2025 1:24 AM GMTThere is no such thing as philosophy-free science; there is only science whose philosophical baggage is taken on board without examination.— Daniel Dennett, Darwin's Dangerous Idea (1995) INTRODUCTIONPrefaceThis document is shared publicly as an initial draft to invite constructive feedback, insights, and reflections.AbstractIntroducing Eliminative Nominalism (EN), a novel position in the philosophy of mind that extends and critiques Eliminative Materialism...

Published on March 29, 2025 7:08 PM GMTOr: Why thinking about blue tentacle arms is not always a waste of time.0. IntroductionAs I work through the Sequences, I find myself disagreeing—slightly—with a point Eliezer Yudkowsky makes in A Technical Explanation of Technical Explanation:Imagine that you wake up one morning and your left arm has been replaced by a blue tentacle. The blue tentacle obeys...

Published on March 29, 2025 8:37 PM GMTA better anything can be achieved with simple tests at low costs.BackgroundPeople often settle for "good enough" and "if it ain't broke don't fix it" in their personal lives, opting not to make any effort to improve said things because either:Time, money, and/or effort can be spent elsewhere for a higher expected valueThey think it can't be...

Published on March 29, 2025 7:38 PM GMTFor the purposes of FOOM, I'm defining it as a situation in which once an AI is capable enough to automate away all AI R&D, progress starts exploding hyper-exponentially for a period because the returns to better software is larger than 1, meaning AI labor quality is improving faster than the problem of finding new algorithms gets...

Published on March 29, 2025 6:11 PM GMTSuppose you’re an AI researcher trying to make AIs which are conscious and reliably moral, so they’re trustworthy and safe for release into the real world, in whatever capacity you intend.You can’t, or don’t want to manually create them; it’s more economical, and the only way to ensure they’re conscious, if you procedurally generate them along with...

Published on March 28, 2025 6:40 PM GMTYouTube link How do we figure out whether interpretability is doing its job? One way is to see if it helps us prove things about models that we care about knowing. In this episode, I speak with Jason Gross about his agenda to benchmark interpretability in this way, and his exploration of the intersection of proofs and...

Published on March 28, 2025 5:21 PM GMTMay 9 & 10Lighthaven, Berkeley Hosted by The Longevity Biotech Fellowship, Vitalism, and Foresight InstituteJoin us for an intensive workshop exploring how artificial intelligence might advance the life sciences and accelerate the development of therapies to extend healthy human lifespan. Confirmed presentations include: Sam Rodriques, FutureHouse Seth Paulson, Biomarkers of Aging Consortium Joe Betts-Lacroix, Retro Erika De Benedictis, Pioneer...

Published on March 28, 2025 3:16 PM GMTI hope that the reasoning in my two posts shows that the AGI has a chance to end up relying on the entire human-built energy industry just to solve as many problems (and hopefully even less) as the millions of humans who work there. On the other hand, the entire set of physicists is within half of...

Published on March 28, 2025 2:20 PM GMTGemini 2.5 Pro Experimental is America’s next top large language model. That doesn’t mean it is the best model for everything. In particular, it’s still Gemini, so it still is a proud member of the Fun Police, in terms of censorship and also just not being friendly or engaging, or willing to take a stand. If you...

Published on March 28, 2025 2:12 PM GMTTl;dr: no.This is a rough research note – we’re sharing it for feedback and to spark discussion. We’re less confident in its methods and conclusions.Once AI fully automates AI R&D, there might be a period of fast and accelerating software progress – a software intelligence explosion (SIE).One objection to this is that it takes a long time...

Published on March 28, 2025 1:22 PM GMTIlya thought back again to when he’d overheard that conversation between two of his junior colleagues. They were discussing an “AI ending the world” story. This was early in the year, although it felt like years ago now given how fast things had been moving. The year, 2025, was now drawing to a close... and maybe it’ll...

Published on March 28, 2025 11:45 AM GMTPDF version. berkeleygenomics.org. X.com. Bluesky. William Thurston was a world-renowned mathematician. His ideas revolutionized many areas of geometry and topology[1]; the proof of his geometrization conjecture was eventually completed by Grigori Perelman, thus settling the Poincaré conjecture (making it the only solved Millennium Prize problem). After his death, his students wrote reminiscences, describing among other things his...

Published on March 28, 2025 11:24 AM GMTTLDRWe do not yet understand all the genetic imprints that play an important role in early embryonic development. Known imprinting disorders (Prader-Willi, Angelman, etc.) however, do seem to cover all the most detrimental imprints, since for all cases of uniparental disomy (having 1 pair of chromosomes from only one parent), we know of either the imprinting disorder...

Published on March 28, 2025 5:47 AM GMTThe other day I discussed how high monitoring costs can explain the emergence of “aristocratic” systems of governance:Aristocracy and Hostage CapitalArjun Panickssery · Jan 8There's a conventional narrative by which the pre-20th century aristocracy was the "old corruption" where civil and military positions were distributed inefficiently due to nepotism until the system was replaced by a professional...

Published on March 28, 2025 4:50 AM GMTEven an AGI "aligned" to a purpose which doesn't imply humanity's survival but does require the AGI itself to achieve difficult feats like transforming the entire Solar System into something computing as many digits of pi as possible would obviously still need to produce the computing systems and gather the energy necessary for the systems' work. As...

Published on March 26, 2025 6:12 PM GMTEmpirical evidence suggests that, if AI automates AI research, feedback loops could overcome diminishing returns, significantly accelerating AI progress.SummaryAI companies are increasingly using AI systems to accelerate AI research and development. These systems assist with tasks like writing code, analyzing research papers, and generating training data. While current systems struggle with longer and less well-defined tasks, future...

Published on March 26, 2025 6:02 PM GMTAnd specifically, what does this imply for AI? There are two theories of equilibrium unemployment — search frictions, and efficiency wages — and they actually give diametrically opposite predictions for when search frictions in finding a new job fall. I conclude that frictions are the more likely explanation, but that LLMs may actually increase unemployment if our...

Published on March 26, 2025 3:47 PM GMTWe are accepting applications for up to 12 paid facilitators & a number of mentors for an upcoming course on AI & animals.Facilitators lead 12 participants in structured discussions (following a provided template) on assigned readings. Facilitating is different from teaching: rather than convey your own knowledge, the goal is to help participants articulate their thoughts via...

Published on March 26, 2025 5:33 PM GMTWe've recently published a paper on Emergent Misalignment, where we show that models finetuned to write insecure code become broadly misaligned. Most people agree this is a very surprising observation. Some asked us, "But how did you find it?" There's a short version of the story on X. Here I describe it in more detail. TL;DR: I think...

Published on March 26, 2025 5:29 PM GMTSummary: CLR is hiring for our Summer Research Fellowship. Join us for eight weeks to work on s-risk motivated empirical AI safety research. Apply here by Tuesday 15th April 23:59 PT.We, the Center on Long-Term Risk, are looking for Summer Research Fellows to explore strategies for reducing suffering in the long-term future (s-risks) and work on technical AI safety...

Published on March 26, 2025 5:16 PM GMTI’ve spent the past 7 years living in the DC area. I moved out there from the Pacific Northwest to go to grad school – I got my masters in Biodefense from George Mason University, and then I stuck around, trying to move into the political/governance sphere. That sort of happened. But I will now be sort...

Published on March 26, 2025 2:13 PM GMTWhat's this post about?I make some rants and recommendations about terminology.This is written for AI-not-kill-everyone-ists. If you are worried about AI killing everyone and want us to prevent AI from killing everyone, this post is for you. If you don't have that agenda and instead have other agenda's, that's fine. It's great. But this post may fail...

Published on March 26, 2025 10:17 AM GMTo1 has shown a strange behavior where it thinks in Mandarin, while processing English prompts, and translates the results back to English for the output. I realized that the same could be possible for humans to utilize, speeding up conscious thought. [1]What makes Mandarin useful for this is that it: Has compact tokensHas compact grammarHas abundant training material onlineCan...

Published on March 26, 2025 9:29 AM GMTLast year, we (the Existential Risk Observatory) published a Time Ideas piece proposing the Conditional AI Safety Treaty, a proposal to pause AI when AI safety institutes determine that its risks, including loss of control, have become unacceptable. Today, we publish our paper on the topic: “International Agreements on AI Safety: Review and Recommendations for a Conditional AI Safety...

Published on March 26, 2025 7:57 AM GMTDownload the latest PDF with links to court dockets here. Discuss

Published on March 24, 2025 6:40 PM GMTMidjourney: “an artificially intelligent researcher, library, posthuman archivist, mapping the noosphere”As regular readers are aware, I do a lot of informal lit review. So I was especially interested in checking out the various AI based “deep research” tools and seeing how they compare. I did a side-by-side comparison, using the same prompt, of Perplexity Deep Research, Gemini...

Published on March 24, 2025 6:39 PM GMTIf we naively apply RL to a scheming AI, the AI may be able to systematically get low reward/performance while simultaneously not having this behavior trained out because it intentionally never explores into better behavior. As in, it intentionally puts very low probability on (some) actions which would perform very well to prevent these actions from being...

Published on March 24, 2025 5:55 PM GMTWe recently released Subversion Strategy Eval: Can language models statelessly strategize to subvert control protocols?, a major update to our previous paper/blogpost, evaluating a broader range of models (e.g. helpful-only Claude 3.5 Sonnet) in more diverse and realistic settings (e.g. untrusted monitoring).AbstractAn AI control protocol is a plan for usefully deploying AI systems that prevents an AI from intentionally...

Published on March 24, 2025 5:09 PM GMTMotivationDimensionality reduction is vital to  the analysis of high dimensional data, i.e. data with many features. It allows for better understanding of the data, so that one can formulate useful analyses.   Dimensionality reduction  that produces a set of points in a vector space of dimension n.mjx-chtml {display: inline-block; line-height: 0; text-indent: 0; text-align: left; text-transform: none; font-style:...

Published on March 24, 2025 4:50 PM GMTThanks to Jesse Richardson for discussion.Polymarket asks: will Jesus Christ return in 2025?In the three days since the market opened, traders have wagered over $100,000 on this question. The market traded as high as 5%, and is now stably trading at 3%. Right now, if you wanted to, you could place a bet that Jesus Christ will...

Published on March 24, 2025 4:46 PM GMTExecutive summaryForecasters believe there’s an 18% chance (range: 4%-50%) that there will be a famine in any part of Gaza by the end of 2025, according to the UN and its Integrated Food Security Phase Classification (IPC). A Category 5 rating would result in a positive resolution, with the last IPC update suggesting that all of Gaza...

Published on March 24, 2025 3:01 PM GMTYour beloved 34 year old author is never hungryI often joke I’m the only traditional rationalist left. The original pitch was that you could radically improve your life by being more strategic. Huge piles of expected value were available. Everyone else seems to have given up, but I’m still a believer. For example in 2017 Scott Alexander...

Published on March 24, 2025 1:10 PM GMTLast week I covered Anthropic’s relatively strong submission, and OpenAI’s toxic submission. This week I cover several other submissions, and do some follow-up on OpenAI’s entry. Google Also Has Suggestions The most prominent remaining lab is Google. Google focuses on AI’s upside. The vibes aren’t great, but they’re not toxic. The key asks for their ‘pro-innovation’ approach...

Published on March 24, 2025 1:03 PM GMTI have been poking around with LLMs, and I found some results that seem broadly interestingSummaryIntroduction: Large language models (LLM) are usually structured as repeated transformer layers of the same size. However, this architecture is often described as functionally hierarchical with earlier layers focusing on small patches of text while later layers parse document-wide information. I revisited...

Published on March 24, 2025 12:56 PM GMTEpistemic status: my own research and reasoning about lock-in risk threat models, and how recommender systems connect to the threat model outlined. I'm fairly confident in the claims about the contribution of recommender systems to filter bubbles, less so on extreme and persuasive content selection effects.TL;DRWe believe lock-in risks are a pressing problem, and that algorithmic technologies...