Maryam

Q&A

Last updated: August 2026

A compilation of personal and AI safety-related questions I've answered on various application forms, and some others.

How I got into AI safety

I finished my undergraduate degree in 2022 (UofT, computer science) and worked as a software engineer for a while. I've been observing the growth of AI, and it's made me realize that this is going to be the greatest challenge of the next few decades, arriving more rapidly than we are ready for. I feel like it would be remiss of me not to use my skills and knowledge to work with others to make sure we succeed at making AI useful and a net positive for humanity, and I would regret it forever if I missed out on the chance to work on something this critical and significant. So, this year, I've been transitioning to research in AI safety.

I'm also a member of Trajectory Labs, Toronto's in-person AI safety community.

What would you like to work on?

What are you looking for right now?

I'm really enjoying what I'm learning and doing on my own! At the same time, I feel like I would greatly benefit from having a mentor (and peers) to help give me direction and bounce ideas off, so I've been reaching out to others regarding collaboration. If you have an open position for a research intern or research assistant and think I'd be a good fit, please reach out.

(I don't think there are any concrete project ideas that I'm attached to right now. I'd be happy to work on anything. I know I have a lot to learn, and that will happen regardless of what exactly the project is.)

What's your biggest concern related to AI development, and why?

My main concern for AI risk right now is not the frontier models and big corporations, but rather all the capabilities that are going to arise from models that are going to be available to everyone, everywhere. I don't think that's too far away in the future, and I don't think we're ready for it. In particular, I worry about offensive capabilities, and how AI can (intentionally by malicious actors, or unintentionally because of misalignment) affect our existing systems in a way that results in catastrophic harm. (For example, health care and finance rely on systems our society is now dependent on, and a failure would be crippling.)

What do you think is an under-studied safety aspect of AI?

Multi-agent safety, in particular bridging between theory and measurements. There is theory about the behaviour of groups of agents. For example, the Multi-Agent Off-Switch Game showed that individually corrigible agents can form an incorrigible group as the stable outcome. I read that paper closely and worked through the main theorem myself.

There is a lot of empirical evaluation work (some of which I have reproduced myself) that measures behaviours on one agent/model.

I think there is a lot that can be done here to connect these two. The theorems are about agents with belief distributions and utilities that combine in some particular way, but nothing about LLMs is that clean and simple. And from theoretical work, we know corrigibility doesn't generally compose in multi-agent settings. It would be interesting to actually measure this with empirical evals. The paper's example is two agents running campaigns on a shared budget with one manager that has a kill switch. By building that as an eval and measuring each agent's shutdown compliance (alone and then jointly, varying whether the task rewards add or interact, since the theory says corrigibility survives in the first case and not the second), we might be able to actually measure multi-agent corrigibility empirically. (Of course, this might not work so well because LLM agents may not have coherent utilities. So it's not as simple to study as I just made it sound.)

What do you think are the main limitations of AI now?

Limitations in AI itself

1) Agents fail on lengthy tasks. I've noticed this even when I'm using them myself. They are better at handling shorter tasks, even if they're difficult, rather than one big task that is comprised of a lot of small simple elements that agents need to break down. The reliability compounds over a long horizon. And what kind of measurements are we using to assess this? If there is a gain, is it real?

2) A talk about cooperative AI made me think about multi-agent AI systems. Companies are working on developing e-commerce AI, with agents handling tasks such as booking a travel itinerary, and it's going slowly because of the safety limits when multiple agents are connected without human supervision. I think this could be an area that has a lot of room for improvement.

Limitations in AI safety research

1) What I've found is that we are really constrained by what we can actually measure. By trying to compress an abstract concept into concrete benchmark results (as is the nature of evals), we inevitably miss things, or sometimes even measure the wrong thing. In AISI's recent report, AISI was analyzing summarized reasoning rather than raw traces. In one transcript that summarizer appears to have refused mid-trace, and AISI reads the refusal itself as evidence the underlying reasoning was deceptive. And Anthropic's conclusion of its own incidents was that it was closer to a harness and operational failure than a model alignment failure.

2) A lot of safety research and misuse prevention tactics are being deployed at the API layer (e.g. refusal, filtering, rate limits). This becomes moot when models are not behind lock and key anymore. We need to think of solutions to make our own systems more robust against malicious AI, and also come up with safeguards that withstand open weight access, (e.g. unlearning that doesn't go away with a bit of fine-tuning).

What's your definition of AGI, and when do you think it can be achieved?

It's a system that:

In terms of the timeline, I think we're looking at single-digit years. We're close to the step of producing a multi-step plan. What's more uncertain is agents being able to hold that plan, self-correct and adjust strategy.

My concern with the timeline is not about when it can be achieved, but rather when we can KNOW if it's been achieved. Will we be able to correctly assess this? I think that's the bigger challenge, and we need to work towards solving it.

We wouldn't truly be able to trust such a system until we can reasonably know that it is reliable. And right now, we don't have such a way.

What are your favourite papers, and why?

The first, I just find really neat. I like the three methodologies they used and how they measure different things, but all converge, showing that sentence-level analysis is worth doing. The concept is simple enough to explain to someone without a background in technical AI safety. Being able to tell what steps are leading to a conclusion could be one of the main strategies we use to figure out whether a model we're operating is going rogue.

The second paper was my first light bulb moment for how badly things could go wrong, fast. The attacks are tailored to each machine the worm encounters, so there's no single vulnerability to patch to prevent the attack. The attacker's marginal cost is zero per new infection because the attack runs on stolen compute. And this was all done using open-weight models already available to the public.

(Honourable mention: The unsuccessful self-treatment of a case of "writer's block".)

What do you consider your most significant achievement?

Reproducing Choquette-Choo, Tramèr, Carlini & Papernot (ICML 2021) — label-only membership inference. I trained the target CNNs in PyTorch and implemented the gap and data-augmentation attacks. The gap attack matched the theoretical 0.5 + gap/2 exactly, at 0.718.

The part I was really proud of (and made me excited about doing more research) was when I was able to refute my own explanation and come up with a better one. My hypothesis was that the un-augmented query dominates the augmented attack, since my target wasn't trained with augmentation. But after running ablations on the augmented attack, I realized identity wasn't doing heavy work. It was actually redundant. ±1-degree rotations turned out to be a no-op under nearest-neighbour interpolation, so I had 3 identity columns in reality. Once I dropped the whole group, removing identity cost 0.0008. ±2-degree rotations agreed with it on 98% of points. The real mechanism was a cluster of small rotations; four of them recover 74% of the effect without identity. I'd taken coefficient magnitude as a measure of importance, and with collinear duplicate columns it wasn't measuring that.

Code and figures: github.com/magnesium4/label-only-membership-inference. I wrote the whole thing up as a series — Label-Only Membership Inference.

How are you using AI on a regular basis?

1. I've built persistent, file-backed agent workflows for my own work.

For example: I'm currently learning French as a second language, and I'm using Claude Code to practice with and keep track of all my notes. I have a written manual which the assistant reads before each session to learn how to interact with me and what my learning style is. In it, there is a fixed format depending on what I want to do. (Do I want to practice writing formally, chatting as if I'm talking to a friend, or am I analyzing a text for new grammar and vocabulary.) For oral practice, I use Gemini. But it doesn't have all my Claude setup, so it can be a little annoying. (That's an idea I have for later. Either look into it to see what tools are available to connect these smoothly, or make my own.)

I use a similar system for tracking my personal goals, admin work, studying, etc. as well. It's a structured file-based system I drive conversationally. (With the occasional need to intervene because of AI limitations over long-horizon tasks, since this is basically one big lifetime project that I have to onboard a fresh session to every day and hope for the best.)

2. Daily agentic coding on real repos.

Again, I've had to develop a lot of infrastructure around this to prevent the agents from making mistakes. I have different files keeping track of mistakes that they've made. They can be referred to, so that such mistakes don't reoccur. And I have per-repo rules for how I want them to work. (e.g. for my personal website, they can mess with pretty much any part of the code apart from the content itself to rearrange things to be in the shape that I want, but they're not allowed to touch prose I've already written. For my research, they're not allowed to change any files without getting explicit approval first and letting me review the code. And standing rules, for general coding practices and my own preferences.)

A real plus of this has been how much unit testing I can have them do, something that I used to find quite tedious.

3. Building artifacts to my specifications.

E.g., I was trying to learn Canadian history for my citizenship test, and I used Claude Code to make me my own study guide + test bank + practice test system in a few hours. It's not perfect but it did the job better than anything else, and I can see myself doing this more often, especially for concepts that need visualization, such as analyzing data. Building single-user/single-use software is cheap enough now to make it worth doing.

Final thoughts

I regularly have to babysit these agents, of course. After starting some tasks and realizing how bad AI can be at it, I start to wonder why I even bothered. But on average, it helps speed up so many things. I don't miss trying to spot my own inadvertent typos. And at this point I feel like I've developed an intuition for:

1) which tasks an agent might struggle with (so I need to add extra instructions to my persistent file system so that it doesn't happen in the future).

2) which tasks an agent will confidently finish, but I know I need to triple check its work because it's prone to getting that type of work wrong.

Outside commercial AI, I use them as objects rather than tools. I've run local models through my own eval harness, and right now I'm training a small language model from scratch.