2023 in Reviewlife, review
30 Dec 2023
22 minute read
Table of Contents #
- Content I liked
2023 was. 2023 was… Settling into college in Boston, settling into a lot of things, picking the stable path in many ways, but finding new strength and resolve within; notably in my perspectives on community and what I want to accomplish. These are big questions that for me right now lie split between understanding the world and making it better.
Lot of firsts; doing my first real job this summer at Dust, where I explored product design and coded with really competent people… Seeing a bit of the world, traveling alone, in Tokyo actually, unexpectedly – there’s something vulnerable and uncomfortable about walking around a city, 15 km a day, alone, and seeing what it’s like, that really brought out a lot of good things for me… Teaching a class for the first time, on abstract algebra to high schoolers… Going to my first rave…
It feels like on a higher level a lot of this year has been about noticing conflicts – France vs US, grind vs pure aesthetics, impact vs pure aesthetics, exploration vs exploitation (sounds cliche but actually), me vs world, math vs software, safety vs innovation, ambition vs nonambition (actually) etc.. . and feeling that these crutches actually explain a lot of my internal confusion about key questions. It’s been an existential year. I’m realizing more and more that I have existential tendencies, and that this is also the lens I use to understand and interact with people. My reading this year reflects this!
In a few words though, it was also a year of mathematics, of grinding it and getting better; of philosophy and getting clearer frames for what I want from/for the world; of writing, reading and poetry; of climbing; of software and of AI…
My goals are always kind of shooting upwards, and it’s easy to find lists of things intimidating, but remember the process is often not as cool as it sounds.
Last year’s #
- publish a paper on theory of deep learning / model interpretability (8/10, Quantization Model of Neural Scaling):
- get an interesting/meaningful internship this summer (9/10, worked at Dust, was very cool)
- reach a new level of math ability (8/10, grinded math and got better)
- listen to more music, have stalled in college, want to publish an inventory of songs that I resonate with (5/10)
- publish at least one substantial coding project (wrote too much private code this year, or didn’t launch what I was building) (3/10)
- write >= 5 blog posts (10/10)
- learn to accept my decisions and their results (2/10, still suck)
- get back on track with my personal projects (3/10)
2024 Goals #
- #blog posts + book reviews >= 10
- try theoretical research and decide if it’s a fit (eg theory of DL)
- be the main person on a research project I was excited about
- write a poetry collection
- one substantial coding project
- learn more math/physics
- one climbing V7, consistently get V5s
- get a good anki setup/figure out uzay x spaced repetition
- get better at technical writing
- get a better perspective on my position and goals relevant to AI
What I wanted to do #
- publish a paper (or at least submit) (10/10)
- actually publish something related to my work on Espial (3/10)
- code a functional programming project, not sure about what, maybe a clean/intuitive math markup language using AI (3/10, did some tensor programming in Rust)
- want to build a better interface to my music, AI could be powerful here (0/10)
- publish agency project (0/10)
What I did #
This summer I interned at Dust, a startup focused on developing AI tools for enterprise work. This fits into my interest in using AI to help humans come up with ideas and learn with technology, and I was particularly interested in how insights in my work could be applied to how individuals work, even outside of organizations. I was given a lot of freedom and started with some fixes and building some infrastructure (like a web extension) for personal indexing. Then I got interested in building a new interfadce and started working on Gens, a power user tool that would allow users to integrate relevant documents, ideas, and notes from different data sources to build workflows with their AI assistants. I worked closely with the technical cofounder, Stan to develop the core functionality using React, TypeScript, and building on their Rust backend. My goal was to craft a writing interface where users could pull insights from their data sources to assist the creative process. On top of a bunch of coding, I also found myself thinking a lot through user workflows, testing prototypes, and gathering feedback to refine the user design.
This is the learning community I’m working on at MIT! We did some stuff mostly spring semester, but I need to kick things up a bit more next year.
We threw a blogathon which was very successful where people wrote something new every 30 mins, [see submissions here], did a learnathon where people learned/taught each other stuff, and we’re doing a weekly story reading group.
I also taught abstract algebra to a bunch of highschoolers in 2 hour classes for [Splash], which was fun and it seems we actually managed to give them the big ideas of groups, group actions, subgroups, quotients/homomorphisms!
This year started with me exploring some ideas in January related to developmental interpretability and how gradients of neural networks can be used to find clusters in input space that correspond to mechanisms in how the model solves separata sub tasks. I sort of ambled a bit and then the method lead to something that worked on LLMs, allowing us to find clusters of inputs for different subtasks the LLM was solving, like completing sequences of numbers for example (this was a cluster). This work was my contribution, particularly in the toy setting of sparse parity, to the Quantization hypothesis of neural scaling paper, that has a lot of cool ideas and proposes a model for LLM scaling and emergence.
I am quite interested in deep learning theory and think understanding generalization/inductive biases/the effect of regulariation is a key/super cool research question I want to work on. I want to try more mathy types of research given that I’ve done a lot of coding.
In the later third of the year I’ve been looking more at program synthesis, particularly after being impressed with Dreamcoder. I think there’s interesting stuff at the intersection of program synthesis and intepretability, and I also think it’ll be important for reasoning, but I’m not working on that.
I want to get better models of research and fields. It’s been hard! Which is to be expected, but I am making slow progress. Motivation and interest need to be constantly maintained which is difficult, but I want to feel like I am doing research that I own, that I am leading and fully owning up to, and I don’t think what I’ve done for now is at that point. Excited to try to get there.
What I want to do #
At MIT I’ve been feeling like I don’t really enjoy CS classes in a formal context with exams, probably because of my self-taught background, and I think I’d rather just implement things and develop CS knowledge like that. I’ve also been enjoying Rust, so I think implementing primitives in networking, databases, concurrency, etc… in Rust or another low-level language would be good.
I’ve been toying around with some better interfaces for language model x knowledge stuff for cyborgism. I think there are more useful ways of organizing data at smaller scales for personal workflows. I’m particularly interested in better self-organizing setups with automated recall and very high quality connections, but doing something here would be nice.
I also want to consolidate my Anki setup and review past courses, I think otherwise I easily lose a lot of context that would be useful if I want to build up a web of knowledge.
I want to do a research internship in industry this summer but getting one has been hard. Generally looking to work on hard open problems that matter at least a little, let me know! Otherwise I’ll just do my own thing.
What I wanted to learn #
- I want to get back to coding in languages I really like, for example Crystal or Rust/Haskell. I want to get comfortable doing functional programming for practical projects (5/10, did some Rust over the summer and got better at that)
- get to a new level in terms of manipulating abstractions in math (8/10, still a noob on an absolute scale)
- cool physics next semester (10/10, E&M was really nice)
- reinforcement learning (3/10, learned some of the theoretical framings)
- get better at cooking good meals quickly (1/10, i suck)
- stats (2/10, can’t get interested)
- get a deeper understanding of linear algebra (5/10, did abstract algebra not linear, but read ML)
- more philosophy, on ethics (8/10, read a lot of this)
- ~concurrent programming, maybe (4/10, did some in Rust over the summer)
What I learned #
I’m thinking of changing the format for this section up. A lot of this entire post is about learning, so I’ll just be going into things with some more detail.
This internship working at an early stage startup taught me a lot. I learned that independence requires accountability and clear planning to effectively deliver, especially in environments where things are moving fast and no one really knows what product the world wants. I want to be very independent but it is also pretty hard when you don’t know what you’re doing in a given context.
On another note, I find it kind of hard and excruciating to do the last 5% that turns a prototype into something a bit polished, and that can be key in some contexts. Gens was somewhat of a power user interface with a lot of complexity and to deliver in that scenario is precisely where you need a lot of product design. Product design is so important and hard! You can put so much thought into building designs and interfaces that naturally give your users the intuitions/reflexes to use your tools.
Working at a startup in the future seems fun. The founders were playing with a lot of constraints in terms of technical output and social considerations around funding and team dynamics/etc… and are doing well. I made a lot of good memories and am excited to see what they get up to.
On the technical side:
- did more Rust and learned a lot here, got better
- learned React from scratch
- did more async stuff/profiling
The spring semester became a semester somewhat purely focused on my courses, as I grew a bit overwhelmed with a pretty heavy courseload, and didn’t have much time for anything else. I think I had to do this to feel what it was like and decide not do it again, probably (gotta live your ego!).
8.022, Electricity and Magnetism, 10/10. This is MIT’s notorious version of E&M, with only like ~40 people enrolled instead of 700, because it’s more proof focused/mathy. It was really good, but easily took 12-15 hours a week. Recitations were high quality and the problems were thought-provoking. Makes me want to do more physics.
- 18.702, Algebra II, 8/10, covering fields, rings, some representation theory, culminating in Galois theory. This course gave me a new jump in mathematical maturity (? I think), and definitely does a lot of cool stuff, but the problems were sometimes too computational and it took a lot of time. I’m not sure how deep into pure mathematics I want to go, and this is another question I must answer!
- 21L.004 Reading Poetry, 8/10, was quite good and gave me some nice perspective and reference on (mostly) black poetry in the US.
- 6.1220, 6/10, Design and Analysis of Algorithms, the second algos course at MIT. It covered a wide array of topics, but the difficulty was concentrated in exams as opposed to problem sets, and it was very much breadth as opposed to depth, which I didn’t love. Did not end up spending much time on it.
- 6.101, N/A, Fundamentals of Programming, I knew the content and expedited this course because it’s a requirement, but think it’s well designed and intense if you don’t know much programming.
For the fall, I was trying to take it more light, but ended up going still a bit too hard on courses, albeit much more chill than the spring.
See post on classes.
- 18.404, Theory of Computation, 10/10, by the renowned Mike Sipser. I recommend his book and this course. Good problems, very interesting ideas, and a good framing for a lot in computer science. Pretty chill workload.
- 21W.771, 10/10
- 2.678, Electronics for mechanics, 9/10, a great electronics class where I got to understand how electronic systems work and bridge the gap with E&M, reverse engineer circuits, build stuff, and grow in general. Highly recommend! Ends with a control/EE project to make a line following algorithm for a toy car.
- 18.100B, Analysis, 8/10, this class was taught quite unconventionally and taught a lot of content in analysis, measure theory and functional analysis. It felt like every 1.5h lecture had 3h of content, which was cool but also bad because I still haven’t had time to absorb everything properly. Problems were sometimes a bit rote, but overall it was good.
- 6.S898, Deep Learning, 7/10, it was actually quite good to review and think about the ideas in depth, especially if I think I’m going to work in the field, and at the end I did a research project on interpreting [OthelloGPT], which was fun, but I felt like the problem sets were more about coding/debugging than I cared for and too many of the lectures were material I knew.
For next semester, I’m planning on taking a lighter load and actually really focusing on my projects. It’s just hard to because it feels like there’s always so much to learn, but everything is a tradeoff and it feels like AI is just blazing ahead. Nonetheless, here is the long list of things that seem interesting for next semester. Bolded are those I am trying to learn in 2024.
- 18.102 functional analysis
- 18.212 Algebraic Combinatorics
- 18.821 Project lab in mathematics
- 18.650 Stats
- 8.04 quantum
- 8.044 stat mech
- 8.09 classical mech III (if i take II during IAP)
- 6.7800 Inference and Information
- 6.4710[J] Evolutionary Biology: Concepts, Models and Computation
- 6.5840 Distributed Computer Systems Engineering
- 6.5230 Advanced Data Structures
- 6.8200 Sensorimotor Learning (does RL)
- 2.679 Electronics for Mechanical Systems II
- 2.000 Explorations in Mechanical Engineering
- 2.12 Introduction to Robotics
- 2.124[J] Robotics: Science and Systems
- 2.670 Mechanical Engineering Tools
Humanities (need CI H):
- 4.657 Design: The History of Making Things
- 17.57[J] Soviet and Post-Soviet Politics and Society: 1917 to the Present
- 21G.041[J] Foundations of East Asian Literature and Culture: From Confucius to the Beats (seems cool)
- 21W.745 Advanced Essay Workshop
- 21W.747 Rhetoric
- 17.41 Introduction to International Relations
- 21W.771 Advanced Poetry Workshop
Too many things to do in one life…
For the fall, I’m also interested in:
- 8.05 Quantum II
- How to make (almost) Anything
- Performance Engineering
- Complex Analysis
- Advanced Algos
I’ve been reading a lot and thinking a decent amount about time/choice/commitment and existentialist ideas in general, and this effort is giving me a better idea of what some type of philosophical work/question asking look like. I think I want to maybe attempt to write something about this soon, but on a high level this year studying some philosophy has been useful for me to get some exposure to other systems of meaning, and see how philosophers think about what values they are living their lives to, in a moral relativist fashion, for example in Beyond Good and Evil.
I think EA is one such system, but it also doesn’t truly resonate and reading this stuff helped me realized that by staring down into the abyss.
I’ve been noticing a lot related to the conflict between the idea of life as a continuous series of choices leading to a continuous series of becomings vs a perspective of letting yourself flow into what will happen and feel the world as it turns you into what you become. [College and choice is related], but more upcoming on this.
More stuff I want to learn for 2024 #
Completes other sections.
- concurrent stuff
- computer architecture
- OS design
- info theory
- piano :/
What I wanted to write #
~ I forgot to write this last year.
What I wrote #
I wrote more this year. This market also motivated me a bit. I wrote so much I became an MIT admissions blogger! I think I have a much better grip of how to write about myself since this year, mostly through just sheer grinding and reading. The application to the MIT blogger job helped a lot, and my travels. One of my goals for next semester is to now get better at technical writing and pedagogy, which I’ve started a bit but want to improve at.
Posts: (in reverse order)
- last year’s review
- how I learn (math): I don’t like this post anymore
- Review of Nietzsche’s Beyond Good and Evil, recommend
- College and choice, paralleling the whole girardian terror stuff
- L’Insoutenable Legerete de l’Etre
- Climbing and becoming, recommend!
- On time and noise
- Semester of poetry
- Explaining my Fall23 classes
- and this year’s review!
I also wrote a lot of poetry, in and outside of classes at MIT. This is more personal so you can ask to see it if you know me, and I’d recommend my poetry post, but it’s been a great experience in terms of letting myself accept forms of writing that are intentionally diffuse and allow you to sink into memories and emotion.
What I want to write #
- a primer on abstract algebra
- philosophy poasting
- more poetry
- technical posts that link ideas across subjects
- AI and TCS?
- thoughts on rationality
- something about communities
- what should I write? tell me!
Content I liked #
What I read #
This year was kinda bathed in existentialism, which makes sense. A good year for books.
I need to write up reviews for a lot of stuff, will do that next week and update.
L’Insoutenable Legerete de l’Etre (Unsufferable Lightness of Being), Kundera, 10/10, a literature story packed in with personal philosophy. Review in french.
- Ficciones, Borges: 10/10, need to review/condense, read this in the original Spanish which was a fun challenge, Borges has a beautiful flowing syntax and creates a mathematical, imaginative set of worlds in this work.
- Crime and punishment, Dostoevsky: 9.5/10, very good, review incoming, this book had been important for some of my friends and is widely appreciated so I was happy to read it.
- Old Man and the Sea, Hemingway, 9/10, need to review
- The Stranger (L’Etranger), Camus, 8.5/10, need to review
- Steppenwolf, Herman Hesse, 8/10, review in french english
- Siddhartha, Herman Hesse 7.5/10, pretty good, but mostly ideas I’ve already somewhat thought about on meditation and becoming. Hesse never went to India though? It’s interesting this book was a hit in the 20th century.
- Dubliners, Joyce (7/10), some of these stories were interesting, but I felt like the overall vibe was something I didn’t love. It kind of reminded me of the vibe of the English stories I read like 7 years ago.
- Nausea (La Nausée), Camus, 6/10, I feel like this book is doing something interesting, but it’s taking a lot of time to do it, and not doing it particularly compellingly. It’s describing a man floating between nihilism and powerful moments of life (an intense feeling of becoming), particularly a past life. It’s making you swallow this grim pill of existentialism and ennui in an interesting way, coupled with using memories and settings to make you feel becoming, but it’s too slow with it.
Non fiction #
- Beyond good and Evil - Nietzsche: very good for me
- Existentialisme et humanisme - Sartre: need to review, pretty good but also not too many new things based on the other stuff I’d read this year.
- Crafting Interpreters - Robert Nystrom (in progress). I read the first half of this one, and am keeping the second for when I implement a programming language in Rust. It was solid but I think the second half is probably the most insightful, the first one didn’t seem that infodense.
- Theorie de l’information - Antoine Chambert Loir (in progress), info theory book, halfway through, pretty good but highly theoretical as is the French tradition.
- Algebra, Artin - for math courses, was good although problems are sometimes too computational
- Baby Rudin - analysis textbook, very good
- Theory of Computation, Sipser - highly recommend
- Art of doing science and engineering, not my favorite, didn’t find it that good for me personally
- Structure of Scientific Revolutions, also not my favorite, might need to do a closer read.
What I want to read #
From Goodreads, some highlights:
- some CS foundations textbooks
- The Book of Disquiet
- Recommended Poetry
- Problem Solving through Problems / Proof and refutations
- Beginning of Infinity
In no particular order except the order my search found stuff in. These are all, in my own opinion, quite good. Probably missing some, but going back to my bookmarks, I’m happy to see how people are just writing good stuff!
- Book Review: Going Infinite
- All of Zhengdong Wang
- Augmenting Long Term Memory
- Performance – Wren language
- Deep Neural Nets: 33 years ago and 33 years from now
- What is PL research and how is it useful?
- Gratuities at your discretion
- Concubines: The Key to Dynastic China’s Success
- Half an hour before dawn in SF
- Recipe for training NNs
- About Ran Prieur
- Using spaced repetition systems to see through a piece of mathematics
- Firelight blogathon
- Lectures on Program Synthesis
- Theoretical Limitations of Autoregressive Models
- Aesthetics and Ethics
- You don’t need to work on hard problems
- My experience crafting an interpreter with Rust
- holding on to spring
- art and work
- otoro ML
- life changing classes at MIT
- suffering artist
- 10 weeks
- Why Transformative AI is really, really hard to achieve
- On Seeing through and Unseeing: The Hacker Mindset
- 2022 letter
- things i wish i could do
- the uneasy relationship between deep learning and classical stats
- ml theory with bad drawings
- the life of poetry
- things you learn dating cate hall
What should I read next year?
Subscribe to the blog's newsletter: