Feedback to Software Carbon Intensity for AI Specification

[bgamazay]

Misc comments:

1. Life Cycle Stages should reference ones already defined in ISO - these are very close but I think aligning makes the most sense.

2. Design and Development stage:
   a. explicitly call out mid- and post-training to be included in this phase (rather than just pre-training) - this is a growing component
   b. explicitly call out that it shouldn't just be the final training run, but any test runs (you mention the entire training duration, but there are often many training runs done before the final one)

3. Consumer vs Provider: I think in practice there is a lot of gray area in these definitions. I recommend checking out the EU AI Act definitions, aligning to what's already out there

4. Consumer Functional Unit: for LLMs I think Query should be the FU rather than Token. LLMs behave very differently, especially with the rise of reasoning models, and they can generate a massively different amount of tokens from model to model. The query is really what is desired by the end user

5. Provider Functional Units: I don't think normalizing training by anything really makes sense or provides any useful information - instead a useful thing you can add here would be to think about how this phase can be properly attributed (amortized?) to the per inference CO2 value

6. Examples: Have you done any real-world testing of this spec? The reality is much, much more messy, complicated, and various depending on small decisions compared to what's in your examples. For instance, you say "Carbon emitted of inference servers" - just that one line raises about 50 questions in my head ranging from what's included in the energy portion (GPU, CPU, RAM, PUE, idle energy) how to calculate carbon (LB, MB, regional vs global) to how the model is configured (data precision, batching, quantization, etc.).
   I understand you are trying to be general, but there is such a wide range of variability depending on different choices that the results of this spec won't be comparable as currently written. I highly recommend integrating learnings from real-world implementations of this before proceeding.

Overall thought:
The European Committee of Standardization (CEN) is currently working (with leading experts) on a "Guidelines and metrics for the environmental impact of artificial intelligence systems and services" standard which goes into significant technical depth. ISO standardization is a goal of theirs, I believe, and I fear the space could become confused and fragmented if your spec beats them to the punch. Therefore, I discourage you from proceeding with ISO standardization of this spec as it's currently written, as it would confuse the industry, and ultimately be detrimental to the greening of software. I still think it could be useful if it has real-world numbers to ground it.

[jawache]

Thanks @bgamazay for this detailed feedback. This is my personal feedback, it's scheduled for review by the working group in next weeks call and they'll respond with the consensus view as well.

Already Addressed in FAQ

ISO Lifecycle Stage Alignment (1): This was discussed and implemented in issue #91. The specification's lifecycle stages are now aligned with ISO/IEC 22989:2022 and ISO/IEC 5338:2023 vocabulary.

Consumer vs Provider Definitions (3): The persona-based approach is covered in our FAQ under "Why does the specification use a persona-based approach?". The boundaries are designed around carbon accountability and agency rather than legal definitions, which is why they may differ from EU AI Act terminology.

Token vs Query for LLMs (4): This was discussed in the working group. The rationale for tokens is covered in our FAQ under "Why use different functional units for Consumers and Providers?" - alignment with commercial pricing models enables direct "cost vs. carbon" comparisons. There's also been work presented to the group by Antarctica on normalising tokens between providers. While this hasn't made it into the specification yet, it's an understood issue being tracked.

Provider Functional Units / Amortization (5): The FAQ covers this under "Why not use 'total emissions per version' for Providers?" and "Why aren't training emissions amortized to Consumers?". This was extensively discussed in the original working group - there was no consensus on an amortization approach, which is why we moved towards two separate persona-based scores: one for Providers (includes training) and one for Consumers (focuses on inference).

Real-World Testing (6): There have been initial calculations, and more are expected now that the specification is public. We're in conversations with various organisations about developing meaningful case studies.

Not Covered In the FAQ

Mid-training and Post-training Explicit Inclusion (2): Good point about explicitly calling out fine-tuning, RLHF, and experimental/test runs. We'll bring this to the working group for discussion.

CEN/ISO Coordination (7)

We're in active conversation with CEN. They have reviewed our specification and indicated it's in alignment with their approach. They'll share their report with us when they're able to disclose publicly, and we'll bring it to the working group for discussion at that point.

[navveenb]

Thanks @bgamazay for your valuable feedback.
Thanks, @jawache , for your response and for laying out the current status clearly.

To extend what Asim has shared, we are actively working with organisations to develop real-world case studies to validate the specification, the underlying calculations, and its practical applicability. As with the ISO SCI specification, this effort is grounded in actual implementations, with the goal of refining and evolving the specification based on empirical evidence and practitioner feedback.

All case studies developed as part of this work will be made publicly available (similar to ISO SCI), ensuring transparency and enabling the broader community to review, provide feedback, and learn from real deployment scenarios.

If you’re aware of organisations that may be interested in contributing to case studies, please feel free to share—we’d be happy to engage and support.

[jawache]

WG:

• @j-turnbull to take an action item to include mid and pre/post training in the spec.