Next Climinator steps

[suung]

• Complete advocate base
• Review if we have all prompts and queries correctly
• Implement follow up questions

[marchburg]

I would add the following tasks:

• Separate between RAG-Advocates and Model-Only-Advocates to implement "The general GPT4 model (OpenAI 2023) serves as one Advocate”
• Claim Decomposition as Part of the ClaimPreprocessing "Figure 1 depicts the operational flow of the Climinator framework, where the initial claim is parsed by an LLM into subclaims"
• Advocate Implementations: IPCC, WMO, AbsCC, S1000, NIPCC
• Evaluate also the two other Label Options: Panel A seven classes, Panel B five classes

[suung]

@marchburg thanks

few thoughts

• model only would be good yes I didn't prioritize it because in the paper it didn't perform better but good idea

• claim decomposition: It's in that version of climinator, it might be out of the revised version Tobi and Markus told me that it wasn't effective... For me it is unclear for sure, what exactly do we decompose. And there is also no benchmark in the climinator paper that we could compare against. I would therefore rather leave it out or wait after the paper is published to see if that changed. @leippold wht do you say?

• wmo and more data sources I started in the Build web interface #30 which is not idea, should extract it but before you start you might want to take a loook

• flexibility for label option evaluateion : I would find this important so we can also experiment with that (e.g. 3 label true, mostly correct and correct)

[leippold]

Dear all Here is the final version of the climinator paper. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4731802 There is still an embargo on it, so we cannot advertise it yet. Indeed, we do not have an objective benchmark. Also, the decomposition with today's LLMs is already somehow encoded, so forcing the model to additionally decompose is not really helpful. Also, what then remains is that one would aggregate the subclaims into a final verdict. The articles that we were using from the academic literature are documents that we are not allowed to share, unfortunately, Also, I am afraid that this corpus is becoming old... I wonder whether one can scrape e.g., carbonbrief.... , some other source that can be easily scraped, https://theconversation.com/, where scientists give a little bit a more personal (and honest!) view about the state of the climate... (according to a chat I had with Katharine Hayhoe)... cheers m

…

________________________________ From: suung ***@***.***> Sent: Tuesday, February 4, 2025 11:05 AM To: climateandtech/factchecker ***@***.***> Cc: Markus Leippold ***@***.***>; Mention ***@***.***> Subject: Re: [climateandtech/factchecker] Next Climinator steps (Issue #32) @marchburg<https://github.com/marchburg> thanks few thoughts * model only would be good yes I didn't prioritize it because in the paper it didn't perform better but good idea * claim decomposition: It's in that version of climinator, it might be out of the revised version Tobi and Markus told me that it wasn't effective... For me it is unclear for sure, what exactly do we decompose. And there is also no benchmark in the climinator paper that we could compare against. I would therefore rather leave it out or wait after the paper is published to see if that changed. @leippold<https://github.com/leippold> wht do you say? * wmo and more data sources I started in the #30<#30> which is not idea, should extract it but before you start you might want to take a loook * flexibility for label option evaluateion : I would find this important so we can also experiment with that (e.g. 3 label true, mostly correct and correct) — Reply to this email directly, view it on GitHub<#32 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AHEG63HIDBMJ76J4543TFFT2OCGIFAVCNFSM6AAAAABU7JPVXCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMMZTGQZDOOJTGM>. You are receiving this because you were mentioned.Message ID: ***@***.***>