Bookmarks (678)

Published on January 16, 2025 6:03 PM GMTThis is a linkpost for How should we analyse survey forecasts of AI timelines? by Tom Adamczewski, which was published on 16 December 2024[1]. Below are some quotes from Tom's post, and a bet I would be happy to make with people whose AI timelines are much shorter than those of the median AI expert.How should we...

Published on January 16, 2025 5:20 PM GMTSummaryCan LLMs science? The answer to this question can tell us important things about timelines to AGI. In this small pilot experiment, we test frontier LLMs on their ability to perform a minimal version of scientific research, where they must discover a hidden rule about lists of integers by iteratively generating and testing hypotheses. Results are ambiguous:...

Published on January 16, 2025 5:19 PM GMTI would like to thank Professor Vincent Conitzer, Caspar Oesterheld, Bernardo Subercaseaux, Matan Shtepel, and Robert Trosten for many excellent conversations and insights. All mistakes are my own.I think that there's a fundamental connection between AI Safety via Debate and Guaranteed Safe AI via topology. After thinking about AI Safety via Debate for nearly two years, this...

Published on January 16, 2025 2:20 PM GMTThe fun, as it were, is presumably about to begin. And the break was fun while it lasted. Biden went out with an AI bang. His farewell address warns of a ‘Tech-Industrial Complex’ and calls AI the most important technology of all time. And there was not one but two AI-related everything bagel concrete actions proposed –...

Published on January 16, 2025 1:55 PM GMTNB this dialogue occurred at the very end of 2023, and for various reasons is only being published ~a year later! Keep this in mind while reading.Key topics coveredconsistent NN-internal deceptive planning (search: goblin, fae, homunc, inner)making AI/agents from foundation models (search: scaffold, condition)planning (search: planning, simulator, consequential)standards of evidence (search: fanfic, theory, Shrek, analog, logic)elicitation gaps...

Published on January 16, 2025 11:38 AM GMTWeirdML websiteRelated posts:How good are LLMs at doing ML on an unknown dataset?o1-preview is pretty good at doing ML on an unknown datasetIntroductionHow good are Large Language Models (LLMs) at doing machine learning on novel datasets? The WeirdML benchmark presents LLMs with weird and unusual machine learning tasks, designed to require careful thinking and actual understanding to...

Published on January 16, 2025 10:51 AM GMTFrom the earliest days of evolutionary thinking, we’ve used metaphors to understand how life changes over time. One of the most enduring is the image of a vast “fitness landscape” with countless peaks and valleys, each corresponding to different levels of survival and reproduction. This landscape is a way of imagining how living things (the “replicators”) search...

Published on January 16, 2025 8:53 AM GMTLearning quantum mechanics involves two things:learning the fundamentals, the nature of the stuff of which Nature is made; andloads of universe-imposed incidental complexity, spinors and vector calculus and Lie algebras and Hilbert spaces, in order to be able to effectively apply the fundamentals to the real world.Now, me, personally, I really like the feeling of understanding how...

Published on January 16, 2025 8:04 AM GMTToday's "nanowrimo" post is a fun longform introduction to permanents and their properties, organized in the way I wish it had been explained to me. Note before going on: this is not my field, and this post is even more likely than usual to contain incorrect statements or bugs.Recall that given an n×n.mjx-chtml {display: inline-block; line-height: 0; text-indent:...

Published on January 16, 2025 2:14 AM GMT(Explanation. Also I have no reason to think they hate me.)Do not use the original TruthfulQA multiple-choice or the HaluEval benchmark. We show that a simple decision tree can theoretically game multiple-choice TruthfulQA to 79.6% accuracy—even while hiding the question being asked! In response, the TruthfulQA authors created a new multiple-choice condition which avoids the vulnerabilities we...

Published on January 15, 2025 6:16 PM GMTApplications are now open for the Cooperative AI Summer School, which will take place from 9th to 13th July 2025 in Marlow, near London! Designed for students and early-career professionals in AI, computer science, and related disciplines—such as sociology and economics—the summer school offers a firm grounding in the emerging field of cooperative AI.Confirmed speakers for this year include:Michael Wellman (University...

Published on January 15, 2025 6:00 PM GMTI'm collecting (x-risk-relevant) safety research from frontier AI companies published in 2023 and 2024: https://docs.google.com/spreadsheets/d/10_dzImDvHq7eEag6paK6AmIdAGMBOA7yXUvumODhZ5U/edit?usp=sharing.I was planning to get AI safety researchers to score each of the papers, so that we could compare the labs on quality-adjusted safety research output. I'm giving up on this for now, largely because I expect to struggle to find scorers. Let...

Published on January 15, 2025 3:55 PM GMTSome fraction of the time, LLMs naturally go on existential rants. My best guess is that, just as people can flip into a context where we do that, so can LLMs. With the result that the LLM certainly sounds like it is suffering, even if we discount the possibility that it actually is. Which raised a question....

Published on January 15, 2025 2:50 PM GMTSocial networks have shaped me since a young age. Growing up at the beginning of the millennium, I used to spend my time in phpBB and vBulletin forums. There, I befriended internet strangers, started my way into graphic design, and learned about torrents. Forums were my favorite third places—little corners on the web where I felt a...

Published on January 15, 2025 2:30 PM GMTTable of Contents Man With a Plan. Oh the Pain. Actual Proposals. For AI Builders. Think of the Children. Content Identification. Infrastructure Week. Paying Attention. Man With a Plan The primary Man With a Plan this week for government-guided AI prosperity was UK Prime Minister Keir Starmer, with a plan coming primarily from Matt Clifford. I’ll be...

Published on January 15, 2025 2:22 PM GMTThe first section describes problems with a few different alignment targets. The second section argues that it is useful to view all of them as variations of a single alignment target: building an AI that does what a Group wants that AI to do. The post then goes on to argue that all of the individual problems...

Published on January 15, 2025 2:08 PM GMTMy current outlook on LLMs is that they are some combination of bullshit to fool people who are looking to be fooled, and a modest but potentially very important improvement in the capacity to search large corpuses of text in response to uncontroversial natural-language queries and automatically summarize the results. Beyond this, I think they’re massively overhyped....

Published on January 15, 2025 6:45 AM GMTSometimes I see a comment I'd like to bookmark, but currently the only ways to save a comment are by subscribing to its replies (which sometimes produces unwanted notifications and requires me to check a different profile section than the rest of my bookmarks) or bookmarking the post it's attached to (which can be inconvenient for posts...

Published on January 15, 2025 6:11 AM GMTThis is an article in the featured articles series from AISafety.info. AISafety.info writes AI safety intro content. We'd appreciate any feedback. The most up-to-date version of this article is on our website, along with 300+ other articles on AI existential safety.There are many fictional stories that depict unaligned non-human entities in ways that can illustrate some aspects of...

Published on January 15, 2025 4:56 AM GMTThere is a view I’ve encountered somewhat often,[1] which can be summarized as follows: After the widespread deployment of advanced AGI, assuming humanity survives, material scarcity will largely disappear. Everyone will have sufficient access to necessities like food, housing, and other basic resources. Therefore, the only scarce resource remaining will be "social status". As a result, the primary activity...

Published on January 13, 2025 6:35 PM GMTMuch of this content originated on social media. To follow news and announcements in a more timely fashion, follow me on Twitter, Threads, Bluesky, or Farcaster.ContentsFrom me and RPIJobs and fellowshipsOther opportunitiesEventsQuestionsAnnouncementsCommentary on the wildfiresSam Altman: AI workers in 2025, superintelligence nextNever underestimate elasticity of supply“The earnestness and diligence of smart technical people”“Americans born on foreign soil”UndauntedEli Dourado’s model of policy changeStatsLinksAIInspirationPoliticsChina...

Published on January 13, 2025 3:05 PM GMTNote: Thank you to Surge Biswas (founder of Nabla Bio) for comments on this draft and and Dylan Reid (an investor into Nabla) for various antibody discussions! Also, thank you to Martin Pacesa for adding some insight on a paper of his I discuss here (his comments are included).IntroductionAntibody design startups are singlehandedly the most common archetype...

Published on January 13, 2025 2:31 PM GMTNote: I am a postdoc in fMRI neuroscience. I have been exploring LLMs for their use in fMRI research (e.g., extracting activations and mapping them onto brain data). To support this direction, I first did something of an interpretability study. This yielded some potentially interesting and surprising results on the emergent effects of scaling. I organized the...

Published on January 13, 2025 1:40 PM GMTNow that I am tracking all the movies I watch via Letterboxd, it seems worthwhile to go over the results at the end of the year, and look for lessons, patterns and highlights. Table of Contents The Rating Scale. The Numbers. Very Briefly on the Top Picks and Whether You Should See Them. Movies Have Decreasing Marginal...

Published on January 13, 2025 11:18 AM GMTIn this post we’ll be looking at the recent paper “Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks” by He et al. This post is partially a sequel to my earlier post on grammars and subgrammars, though it can be read independently. There will be a more technical part II.I really like this...

Published on January 13, 2025 2:57 AM GMTDepressive realism is the idea that depressed people have more accurate beliefs than the general population. It’s a common factoid in “things I learned” lists, and often posited as a matter of settled science.In this post, I’ll explore whether it’s true.Where It BeganThe depressive realism hypothesis was first studied by Lauren Alloy and Lyn Yvonne Abramson, in...

Published on January 13, 2025 1:23 AM GMTTraditional economics thinking has two strong principles, each based on abundant historical data:Principle (A): No “lump of labor”: If human population goes up, there might be some wage drop in the very short term, because the demand curve for labor slopes down. But in the longer term, people will find new productive things to do, such that...

Published on January 12, 2025 5:11 PM GMTI've been researching the controversy over whether antidepressants truly work or whether they are not superior to a placebo. The latter possibility really contains two possibilities itself: either placebos are effective at treating depression, or the placebo effect reflects mean reversion. Here, the term "antidepressant" refers to drugs classified as SSRIs and SNRIs.Some stylized facts from the...

Published on January 12, 2025 5:11 PM GMTIntroductionMagnetic reconnection—the sudden rearrangement of magnetic field lines—drives dramatic energy releases in astrophysical and laboratory plasmas. Solar flares, tokamak disruptions, and magnetospheric substorms all hinge on reconnection. Usually, these events are uncontrolled and often destructive. But what if we could systematically harness reconnection here on Earth, funneling that released magnetic energy into an external circuit? This post...

Published on January 12, 2025 6:23 PM GMTFrom AI scientist to AI research fleetResearch automation is here (1, 2, 3). We saw it coming and planned ahead, which puts us ahead of most (4, 5, 6). But that foresight also comes with a set of outdated expectations that are holding us back. In particular, research automation is not just about “aligning the first AI...

Published on January 11, 2025 6:24 PM GMTMotivationI want a more concrete concept than AGI[1] to talk and write with. I want something more concrete because I am tired of the costs associated with how big, inferentially distant, and many-pathed the concept of AGI is, which makes conversation expensive. Accounting for the bigness, inferential distance, and many-pathed-ness is very important for dealing with AGI properly...

Published on January 11, 2025 2:43 PM GMTI decided, as a challenge to myself, to spend 5 minutes, by the clock, solving the alignment problem. This is the result, plus 25 minutes of writing it up. As such, it might be a bit unpolished, but I hope that it can still be instructive.Background This proposal is loosely based on iterated amplification, a proposal for training...

Published on January 11, 2025 3:22 AM GMTAnd by "we" I mean "I". I'm the one struggling.Agency with a 'y', basically means “The condition of being in action; operation.” Or the means or mode of acting, the context I hear it used most often is sociologically:"Agency is the capacity of individuals to have the power and resources to fulfill their potential." One's agency is one's...

Published on January 10, 2025 11:55 PM GMTWith LLM-based AI passing benchmarks that would challenge people with a Ph.D in relevant fields, I'm left wondering what they can do for real-world problems for which nobody knows the correct answer, such as how to treat potentially fatal medical conditions with no known cure. Are we at the point where AI can do better than curated...

Published on January 10, 2025 10:07 PM GMTThis post is to record the state of my thinking at the start of 2025. I plan to update these reflections in 6-12 months depending on how much changes in the field of AI.1. SummaryIt is best not to pause AI progress until at least one major AI lab achieves a system capable of providing approximately a...

Published on January 10, 2025 6:57 PM GMTVirtually everyone I see in the AI safety community seems to believe that working on AI alignment is the key to ensuring a safe future. However, it seems to me that AI alignment is at best a secondary instrumental goal that can't in and of itself achieve our terminal goal. At worst, it's a complete distraction.Defining humanity's...

Published on January 10, 2025 7:34 PM GMTAnthropic’s Alignment Science team conducts technical research aimed at mitigating the risk of catastrophes caused by future advanced AI systems, such as mass loss of life or permanent loss of human control. A central challenge we face is identifying concrete technical work that can be done today to prevent these risks. Future worlds where our research matters—that...

Published on January 10, 2025 6:13 PM GMTOne can call it "deceptive misalignment": the aligned AGI works as intended, but people really don't like it. Some scenarios I can think of, of various levels of realism:1. Going against the creators' will1.1. A talented politician convinces the majority of humans that the AGI is bad for humanity, and must be switched off. In a democratic vote,...

Published on January 10, 2025 4:53 PM GMTEpistemic status -- sharing rough notes on an important topic because I don't think I'll have a chance to clean them up soon.SummarySuppose a human used AI to take over the world. Would this be worse than AI taking over? I think plausibly:In expectation, human-level AI will better live up to human moral standards than a randomly...

Published on January 10, 2025 4:22 PM GMTThe Alignment Mapping Program: Forging Independent Thinkers in AI Safety - A Pilot RetrospectiveThe AI safety field faces a critical challenge: we need researchers who can not only implement existing solutions but also forge new, independent paths. In 2023, inspired by John Wentworth's work on agency and learning from researchers like Rohin Shah and Adam Shimi who have...

Published on January 9, 2025 4:47 PM GMTResponse to Discursive Games, Discursive WarfareThe discursive distortions you discuss serve two functions:1 Narratives can only serve as effective group identifiers by containing fixed elements that deviate from what naive reason would think. In other words, something about the shared story has to be a costly signal of loyalty, and therefore a sign of a distorted map....

Published on January 9, 2025 4:40 PM GMTLast year Timothy Telleen-Lawton and I recorded a podcast episode talking about why I quit Effective Altruism and thought he should too. This week we have a new episode, talking about what he sees in Effective Altruism and the start of a road map for rescuing it.  Audio recording Transcript Thanks to everyone who listened to the...

Published on January 9, 2025 4:30 PM GMTThe world is kind of on fire. The world of AI, in the very short term and for once, is not, as everyone recovers from the avalanche that was December, and reflects. Altman was the star this week. He has his six word story, and he had his interview at Bloomberg and his blog post Reflections. I...

Published on January 9, 2025 4:10 PM GMTSummary: The Many-Worlds interpretation of quantum mechanics helps us towards an overall evaluation of existence. I consider some recent work in philosophy of religion on the quantum multiverse and the Problem of Evil, as well as Olaf Stapledon’s Starmaker.I’ve previously suggested that when we think about the ethical implications of the many-worlds interpretation (MWI) of quantum mechanics, the kinds of implications...

Published on January 9, 2025 2:23 PM GMTWe're excited to announce that the PIBBSS Fellowship 2025 now includes a dedicated Cooperative AI track, supporting research that advances our understanding of cooperation in artificial intelligence systems. We are also announcing 300 USD bounties for each referral that becomes a Fellow. Read below for details.What is the Cooperative AI Track?Thanks to the support from the Cooperative AI...

Published on January 9, 2025 2:19 AM GMTThese are thoughts in response to the paper "Frontier Models are Capable of In-context Scheming" by Meinke et al. 2024-12-05, published in Apollo Research.  Link: https://static1.squarespace.com/static/6593e7097565990e65c886fd/t/6751eb240ed3821a0161b45b/1733421863119/in_context_scheming_reasoning_paper.pdf.  I found the paper via  this reddit thread, and commented there as well: https://www.reddit.com/r/slatestarcodex/comments/1hw39rd/report_shows_new_ai_models_try_to_kill_their/.  According to the paper, AIs were asked to roleplay to see what they would do if given...

Published on January 9, 2025 12:18 AM GMTA Systematic Approach to AI Risk Analysis Through Cognitive CapabilitiesEpistemic status: This idea emerged during my participation in the MATS program this summer. While I intended to develop it further and conduct more rigorous analysis, time constraints led me to publish this initial version (30-60m of work) . I'm sharing it now in case others find it...

Published on January 8, 2025 7:38 PM GMTThere’s a conventional narrative by which the pre-20th century aristocracy was the “old corruption” where civil and military positions were distributed inefficiently due to nepotism until the system was replaced by a professional civil service after more enlightened thinkers prevailed. Orwell writes in 1941 (emphasis mine):For long past there had been in England an entirely functionless class,...

Published on January 8, 2025 7:38 PM GMTEpistemic status: This is an off-the-cuff question.~5 years ago there was a lot of exciting progress on game playing through reinforcement learning (RL). Now we have basically switched paradigms, pretraining massive LLMs on ~the internet and then apparently doing some really trivial unsophisticated RL on top of that - this is successful and highly popular because interacting...

Published on January 8, 2025 2:59 PM GMTOn January 6, 2025, Ann Altman filed a lawsuit in the Eastern District of Missouri alleging that Sam Altman carried out multiple acts of sexual abuse against her over "a period of approximately eight or nine years" starting in 1997. The case number is 4:25-cv-00017, for those who have PACER access. I find the lawsuit complaint notable...