Large Language Models have emerged as a transformative technology and are increasingly being adopted in modern businesses to:

• Elevate customer service standards
  
• Streamline internal processes
  
• Optimize workflow automation
  
• Enhance data processing capabilities, and 
  
• Create content in diverse knowledge domains. 

However, effectively unlocking these benefits demands a critical decision - whether to opt for a paid model or open-source, balancing performance with protecting sensitive data.  

To clarify these choices, our paper explores the landscape of state-of-the-art language models as of March 2024. It provides a robust and repeatable method to evaluate Large Language Models across various knowledge domains, facilitating meaningful comparisons for any given use case:

• Structured, repeatable approach 
  
• Systematic evaluation of costs and benefits associated with each LLM
  
• Providing stakeholders with clear insights into the value proposition
  
• Informing decisions concerning alignment with organizational objectives, resource constraints and regulatory demands. 

Finally, our paper not only provides insights for financial services firms looking to optimize or modernize their legacy systems using LLMs but also gives advice on how to adapt and integrate these models into their environments.