clarifications and inputs regarding submitted questions

[seanmcilroy29]

From Sreenivasan Lakshminarayanan - Amadeus

Request clarification and input on my questions below, as we (Amadeus) plan to calculate and report "CO2 for AI" for our AI workloads.

1. In the example illustrated in the Spec, it is imperative that two separate SCI scores should be calculated and reported ( one from Provider, one from the consumer ). May I know how to handle the scenarios where we have "no or less" visibility in Provider's reporting?

2. For example, our Provider Microsoft has the following portals/ mechanisms to report CO2: a) "Emissions Impact Dashboard for Azure" b) "Emissions Impact Dashboard for Microsoft 365" and c) CO-Pilot in Microsoft Sustainability Manager. However, none of these seems to be specific to "CO2 for AI". Thus, the provider's reporting is a black box for us. How do we go about this scenario?

3. Even if we obtain Provider's CO2 for AI reporting, we are unsure of their mertic and calculation methodology. Thus, how do we consider Provider's CO2 reporting to be clubbed with our Consumer's side ( based on SCI for AI specifications ) of reporting values ?

4. We already have CARMEN which can calculate CO2 based on GSF's impact framework and SCI formula. Do you have any guidelines to 'extend' the existing "SCI" to "SCI for AI" ?

[seanmcilroy29]

1. In the example illustrated in the Spec, it is imperative that two separate SCI scores should be calculated and reported ( one from Provider, one from the consumer ). May I know how to handle the scenarios where we have "no or less" visibility in Provider's reporting?

So you are encouraged to also report a provider score but only if your product does some additional fine tuning over the model you are using, so let's say you took the Mistral model an dfine-tuned it, the provider score for your product would be the fine-tuning ONLY of the mistral LLM.

If all you are doing is calling inference on Microsoft provided AIs then there is no need to calculate Microsoft's provider score, in an ideal world Microsoft should calculate their own Provider and Consumer scores for their AI's your not required to calculate your vendors scores (however, realistically you may have to estimate them as part of yoru consumer calculation)

2. For example, our Provider Microsoft has the following portals/ mechanisms to report CO2: a) "Emissions Impact Dashboard for Azure" b) "Emissions Impact Dashboard for Microsoft 365" and c) CO-Pilot in Microsoft Sustainability Manager. However, none of these seems to be specific to "CO2 for AI". Thus, the provider's reporting is a black box for us. How do we go about this scenario?

If you are fine tuning a model (or creating it from scratch) you should include the emissions from all the infra used to create the model. I suspect from your use case you don't need this?
If you are running inference on Azure, TBH I'm not sure, I can add contacts we have in Microsoft to this thread to help provide guidance if you're open to that?

3. Even if we obtain Provider's CO2 for AI reporting, we are unsure of their mertic and calculation methodology. Thus, how do we consider Provider's CO2 reporting to be clubbed with our Consumer's side ( based on SCI for AI specifications ) of reporting values ?

If you are just consuming an AI (and then doing other things) then your Consumer score will be an estimate of the inference costs of using a provider's AI plus the rest of the ifra used to provide the AI service to yoru end users.

4. We already have CARMEN which can calculate CO2 based on GSF's impact framework and SCI formula. Do you have any guidelines to 'extend' the existing "SCI" to "SCI for AI" ?

SCI for AI is mostly a guide regarding what you must include in the calculation and what functional units you must use, so the overal framework I don't think would change, it's just clarity regarding what you included and how you are reporting it.
If you can share more info about the type of application you are propising to calculate, I think we can provide a bit more guidance, we expect there will be a lot of thing we need to figure out as there are unfortunately few use cases here but if you can share more info we can ideates here.

[lsreenivas]

Thank you so much for your elaborate answers; really appreciate it. Just to make sure that I fully grasped your answers, I require some more clarifications, please.

1. You quoted : ".......

in an ideal world Microsoft should calculate their own Provider and Consumer scores for their AI's your not required to calculate your vendors scores (however, realistically you may have to estimate them as part of yoru consumer calculation)"

. I can understand that we (Amadeus, the consumer) need not calculate Provider's (Microsoft, for example) scores, and we should rely on what we receive from the Provider as Provider's scores. However, I could not understand what you meant by

"Microsoft should calculate their own Provider and Consumer scores for their AI's"

. While the Provider is responsible for "calculating and then reporting" his own scores for LLM ( development and design, Model training, deployment etc ), I could not understand how the Provider can calculate scores for the Consumer's usage ?

2. To be clear on our ( Amadeus, the consumer ) side, we have no plans to fine tune any model. We will be only consuming the Provider's AI - meaning that our Consumer boundary shall be restricted to [ inference, operation, API Gateways, monitoring, caching and storage etc.]. Thus, as a consumer, we can follow the guidelines from GSF's SCI for AI to calculate our own Consumer's side of the score. My question is related to the "overall" score that we are planning to Report. As illustrated in this Example

Calculating the Consumer score is within our boundary. But, for the Provider's score we ought to rely on the Provider's report. However, the Provider's reporting and Calculation are "Blackbox" to us. For example, if we look at Microsoft's "Emission Impact Dashboard for Azure" which displays CO2 values for the usage of Azure cloud services, their calculation method was something which was originally validated by Stanford University in 2018. And, they claim they recently updated their calculation method in Feb 2024. Thus, does it even make sense for a consumer to report both ( Provider Score + Consumer Score ) even though they are completely different calculation methods? Or, do you advice the consumers to report only the consumer side of score (without worrying about Provider's score, as it would go in Provider's reporting anyways ? ).

Thank you, again, for your time and help..! @jawache , @seanmcilroy29 , @jmcook1186 , @navveenb

[jawache]

Hi @lsreenivas

I wonder if some examples might help:

Let's imagine you are someone who's created an LLM from scratch

You publish two scores a Provider Score which is say the Carbon per FLOP for the training of your LLM.
A consumer score which is Carbon per Token for the inference of your LLM.

Let's imagine you are someone who has fine-tuned an LLM and deployed it yourself.

You publish two scores a Provider Score which is say the Carbon per FLOP for the fine-tuning of your LLM.
A consumer score which is Carbon per Token for the inference of your LLM.

Let's impage you are someone who has taken an AI and perhaps say just added RAG ontop of it to give an enhanced LLM experience to your users.

You only publish a Consumer score.

Let's imagine you are someone who has an application which "uses" AI in someway, but AI is just one component, let's say an Image Processing App. The end user doesn't see your product as an LLM, it's just some app that has used an LLM inside it somewhere.

You are not in scope for SCI for AI, you publish a base SCI score, let's say Carbon per User. In the ideal world the provider of your LLM has published a Consumer (Carbon per Token) score and you use that in the computation of your Carbon per Token.

Does that help clarify?