import litellm 
 
# To use openai. 
response = litellm.completion(model="o3-mini", messages=[{"content": "Hello", "role": "user"}]) 
 
# To use deepseek. 
response = litellm.completion(model="deepseek/deepseek-reasoner", messages=[{"content": "Hello", "role": "user"}]) 

This simplified approach helped us compare models without worrying about implementation complexities. 

DeepSeek vs. OpenAI – model overview

Before diving into the AI model benchmarking results, let’s define key concepts and introduce the core specifications of the tested models. 

One of the most important parameters to consider in LLM benchmarking is the context window—the maximum input size a model can process at once. This is measured in tokens, which represent chunks of text rather than individual words. A larger context window allows the model to handle more extensive documents in a single request, which is particularly important when working with long statistical reports. 

The pricing for LLMs is typically based on token usage, which can vary depending on the type of tokens being processed. There are generally three types of tokens involved in LLM pricing: 

1. Input Tokens: These are the tokens representing the user’s input, such as the text or prompt sent to the model for processing. The cost of input tokens is charged based on the number of tokens provided by the user in each request. 
2. Cached Input Tokens: Some models offer a caching mechanism, where previously used inputs are stored and reused in subsequent requests, reducing the need for reprocessing. This is often charged at a lower rate than fresh input tokens, as the model does not need to process them again from scratch. 
3. Output Tokens: These tokens represent the text or response generated by the model. Output tokens are charged based on the amount of text the model generates in response to the user’s input. 

The models selected for this comparison are among the latest releases from the past several months. While they differ in pricing and capabilities, we aim to assess whether these differences translate into measurable performance variations. Below is a breakdown of the key characteristics of DeepSeek-R1, OpenAI 4o-mini, and OpenAI o3-mini: 

	DeepSeek-R1	OpenAI 4o-mini	OpenAI o3-mini
Context Window	128,000 tokens	128,000 tokens (with a maximum output of 16,384 tokens)	200,000 tokens (with a maximum output of 100,000 tokens)
Release Date	January 2025	July 2024	January 2025
Pricing (per 1 million tokens)	Input: $0.55 Cached input: $0.14 Output: $2.19	Input: $0.15 Cached input: $0.075 Output: $0.60	Input: $1.10 Cached input: $0.55 Output: $4.40
Input Formats	Text	Text, Images (including PNG, JPEG, GIF, WEBP)	Text
Output Formats	Text	Text	Text

PDF file used for testing

The document used for testing is composed of several chapters of report on the Polish and worldwide maritime economy in 2017-2020. The report is 50 pages long and includes various statistics and analysis of cargo traffic, shipping, shipbuilding, and other maritime industries. The data in the file is formatted in tables and text. Most of the data is presented in tables, with additional explanations and summaries in the surrounding text. Example pages of the document used for testing: 

Testing Methodology

We conducted a series of tests using the following maritime economy-themed prompts and a PDF file providing context information. Here are example prompts regarding information included in the PDF: 

• • Summarize the key economic findings from a maritime report. 

• • What is the total cargo turnover of Polish sea ports in 2020? 

• • What are the main cargo types handled by Polish sea ports? 

• • Which countries are the main trading partners of Poland in seaborne trade? 

• • What is the average age of ships in the Polish maritime transport fleet? 

• • What are the key economic indicators for the Polish shipbuilding industry? 

As mentioned before, we compared the following models: 

• • OpenAI’s 4o-mini 

• • OpenAI’s o3-mini 

• • DeepSeek’s deepseek-resoner (R1) 

We measured the following metrics: 

• • Inference Time – This refers to the time it takes for the model to generate a response after receiving a prompt. A lower inference time means faster responses, which is crucial for real-time applications and large-scale document processing. 

• • Token Usage – LLMs process and generate text in units called tokens. A token can be a word, part of a word, or even a punctuation mark. The total token usage includes both input tokens (the user’s query or document) and output tokens (the model’s generated response). The more tokens used, the higher the cost of the request. 

• • Response Cost – This is calculated as token usage × model pricing (per 1,000 or 1,000,000 tokens, depending on the provider). Since different models have different pricing structures, comparing response costs helps determine which model is more cost-effective for large-scale use cases. 

Test Results

Here are the summarized results from our tests (each test was repeated several times): 

Model	Average Inference Time (s)	Average Response Cost ($)	Average Input Tokens	Average Output Tokens
DeepSeek R1	57.2	0.0039	63961.7	751.6
o3-mini	13.8	0.0755	63251.5	1162.5
4o-mini	9.5	0.0511	62538.0	1046.5

Key Observations

• • Inference Time: DeepSeek consistently demonstrated longer inference times compared to both OpenAI models. This could be a significant factor for applications that prioritize fast processing. 

• • Response Cost: DeepSeek showed a competitive advantage in terms of cost, particularly for output tokens. Despite the longer inference time, DeepSeek’s overall cost per request remains lower than OpenAI o3-mini and 4o-mini. The lower response cost of DeepSeek can be attributed to its caching mechanism, which reduces the need to reprocess input data. Most of the input content, particularly the PDF file’s contents, was cached, leading to significant savings in processing costs. This caching system allowed DeepSeek to handle repeated queries more efficiently, making it a cost-effective option for processing large documents. 

• • Output Variability: The models varied in style and the level of detail in their responses. This is important depending on the context and user requirements (e.g., high-level summaries vs. detailed analysis). 

• • LiteLLM API: LiteLLM made it extremely easy to track cost, token usage, and response time directly from the API responses, enabling a straightforward comparison between models. 

Conclusion

Our tests highlight the advantages of using unified APIs for LLM benchmarking. LiteLLM significantly simplified the process, allowing us to focus on LLM efficiency assessment and evaluating AI language models. While DeepSeek R1 demonstrated competitive cost-effectiveness, particularly due to its caching mechanism, it was by far the slowest model in our tests, with an average inference time of 57.2 seconds. In contrast, OpenAI o3-mini and 4o-mini provided significantly faster response times, making them more suitable for real-time applications. 

Artykuł OpenAI vs. DeepSeek: A Technical Comparison Using Unified APIs pochodzi z serwisu Inero Software - Software Consulting.

The end of the year is a time for summaries. In the world of IT, many interesting things have happened, so in this article, we decided to focus on AI. The development of artificial intelligence and its media presence accelerated to an unprecedented scale. Tools based on Large Language Models (LLMs) have been popularized and made widely available to users from various industries, not just technological ones. We decided to summarize the year with Andrzej Chybicki, the CEO of Inero Software. Here is the list he identified as the key 5 events of the past year. 

Fact 1: OpenAI – artificial intelligence becomes widely accessible 

OpenAI played a tremendous role in popularizing the field of artificial intelligence in the context of human language understanding. In 2022, they released ChatGPT, and in the following months, they presented new, improved models. These advancements not only improved the performance of existing applications but also opened new avenues for AI in healthcare, environmental science, administration, marketing, and more.  

In 2023, ChatGPT saw remarkable advancements, featuring enhanced learning algorithms for improved accuracy and nuanced conversations, personalized user interactions, expanded language support for global accessibility, and broader application integration. OpenAI emphasized ethical considerations and bias reduction, incorporated real-time learning for up-to-date content, improved multimedia interaction capabilities, and boosted the tool’s robustness and reliability. Additionally, ChatGPT was tailored for specific industries, providing specialized functionalities and knowledge, marking a significant leap in AI technology and user-centric applications. 

Expert Insight 

OpenAI was the first widely recognized large language model. In the coming years, we are likely to see various versions of LLMs designed for specific applications – in fact, this has been happening for a few months now. OpenAI, despite being a pioneer, at least in terms of recognizability, is not always considered the best model for everything. The direction of development is certainly popularization in a similar way as it was with computers (i.e., LLMs like PCs) and specialization, meaning specialized language models designed for specific applications or even entities or people.  

Fact 2: GitHub Copilot – a leader in AI/LLM implementation 

One of the key roles in the development of artificial intelligence is played by Microsoft, which collaborates with OpenAI. Over the past year, Microsoft has continued to refine its vision of Microsoft Copilot. Let’s focus on the solution for developers: GitHub Copilot. In 2023 it underwent significant changes and enhancements. Here are the key updates: 

In 2023, GitHub Copilot introduced several significant enhancements to bolster its role in AI-driven software development. The GitHub Copilot Chat, now generally available and powered by OpenAI’s GPT-4, provides more accurate code suggestions and explanations, using natural language to aid developers in various languages. This feature is integrated with both the GitHub platform and its mobile app, supporting coding, pull requests, and documentation. Additionally, GitHub Copilot Enterprise was introduced to tailor the tool to specific organizational needs, helping developers quickly adapt to their organization’s codebase and streamline tasks like documentation and pull request reviews, aimed at boosting enterprise-level productivity and security. The GitHub Copilot Partner Program was launched, integrating Copilot with various third-party developer tools and services, thereby creating a broad ecosystem that enhances the capabilities of developers using AI. Finally, GitHub unveiled new AI-powered security features in its Advanced Security suite, including a real-time vulnerability prevention system and application security testing features to detect and remediate code vulnerabilities and secrets, further securing the software development process. 

Expert Insight 

Thanks to its collaboration with OpenAI, Microsoft became a leader in AI/LLM implementation worldwide in 2023. Microsoft’s strategy in this area is based on using the LLM model to support (but not replace) as many activities and processes using Microsoft products as possible. Particularly important was ensuring an appropriate level of SLA (aligned with other Azure services) and data security. Among the most significant changes, apart from the mentioned GitHub Copilot (which aims to support developers in coding), are Copilot plugins available in practically all of this company’s flagship products (Word, Excel, PowerPoint, Outlook). 

In December 2023, Microsoft also presented the CoPilot Studio solution, which enables the creation of low-code/no-code IT systems with significant support from the OpenAI model. This effectively allows for the easy expansion of existing Azure low-code solutions such as Azure Agents with conversational bots or AI-supported database adapters. Although CoPilot Studio is not yet available in its final form, Microsoft clearly communicates development directions and the advantages that developers, engineers, and users can experience from its use. From the presentations of Microsoft representatives, it can be inferred that Microsoft’s goal is to lower the entry threshold for creating and implementing new advanced AI solutions, as using low-code platforms does not require as deep technical knowledge as traditional coding. We can expect widespread interest in these solutions not only from the largest companies using MS Azure in the coming years. Currently, among experts, the question is not “whether to use AI” but how to implement it to not fall behind the competition. Those entities that create a coherent strategy for incorporating AI-based products into their processes in the coming years will be able to significantly benefit from the revolution that is already taking place. 

Fact 3: The European AI Act: A Regulatory Milestone 

On 14 June 2023, the European Parliament adopted its negotiating position on the AI Act. Parliament’s priority is to make sure that AI systems used in the EU are safe, transparent, traceable, non-discriminatory and environmentally friendly. Parliament also wants to establish a technology-neutral, uniform definition for AI that could be applied to future AI systems. The AI Act sets different rules for different AI risk levels. 

The new rules establish obligations for providers and users depending on the level of risk from artificial intelligence. While many AI systems pose minimal risk, they need to be assessed. 

Unacceptable risk 

Unacceptable risk AI systems are systems considered a threat to people and will be banned. They include: 

• Cognitive behavioral manipulation of people or specific vulnerable groups: for example voice-activated toys that encourage dangerous behavior in children 
• Social scoring: classifying people based on behavior, socioeconomic status or personal characteristics 
• Real-time and remote biometric identification systems, such as facial recognition 

Some exceptions may be allowed: For instance, “post” remote biometric identification systems where identification occurs after a significant delay will be allowed to prosecute serious crimes but only after court approval. 

High risk 

AI systems that negatively affect safety or fundamental rights will be considered high-risk and will be divided into two categories: 

1) AI systems that are used in products falling under the EU’s product safety legislation. This includes toys, aviation, cars, medical devices and lifts. 

2) AI systems falling into eight specific areas that will have to be registered in an EU database: 

• Biometric identification and categorisation of natural persons 
• Management and operation of critical infrastructure 
• Education and vocational training 
• Employment, worker management and access to self-employment 
• Access to and enjoyment of essential private services and public services and benefits 
• Law enforcement 
• Migration, asylum and border control management 
• Assistance in legal interpretation and application of the law. 

All high-risk AI systems will be assessed before being put on the market and also throughout their lifecycle. For more information, visit the European Parliament website. 

*source: https://www.europarl.europa.eu 

Expert Insight 

Ensuring security and confidentiality of data is certainly one of the most important issues concerning the implementation of AI solutions. Many experts indicate that despite the good intentions of the European Commission, the proposed solutions may contribute to reducing the competitiveness of the domestic AI market, which in effect will increase the distance between Europe and leaders in this field (i.e., the USA and China). I personally share these concerns. Here, a good example might be the similar situation that occurred about 15 years ago when cloud computing was being implemented. At that time, the EU also created a regulation governing the rules of access and data confidentiality (GDPR), which to this day is the regulatory basis in this area. At the same time, the largest solutions that most in the EU use are those developed in the USA, where the priority was the free development of technology, and only secondarily the legal framework. Unfortunately, many indications suggest that a similar situation might occur with AI. 

Fact 4: Gemini: new model from Google 

Without a doubt, the launch of Gemini was the most prominent premiere in the latter part of 2023, generating significant buzz. It is a result of large-scale collaborative efforts by teams across Google. It was built from the ground up to be multimodal, which means it can generalize and seamlessly understand, operate across, and combine different types of information including text, code, audio, image, and video. 

Gemini 1.0 was trained to recognize and understand text, images, audio, and more at the same time, so it better understands nuanced information and can answer questions relating to complicated topics. This makes it especially good at explaining reasoning in complex subjects like math and physics. 

During the presentation on the release of the Gemini API for developers, a lot of time was dedicated to AI Studio, a browser-based, free tool for code creation. The second focus was on Vertex AI, a more advanced program that allows for “both training and deploying ML (machine learning) models and AI applications.” Google offers the option to transfer a preliminary project developed in AI Studio to Vertex AI, to add additional features available within the larger platform of Google Cloud. 

Expert Insight 

Google has officially joined the large language model (LLM) race. The most intriguing aspect of what they propose is that their model will operate in three versions: Ultra (the most feature-rich), Pro, and Nano, with the latter being designed for mobile phones. It’s still unclear whether Nano will run entirely on client devices (smartphones) or if it will simply be a thin client and a kind of extension of Google Assistant. It’s also worth emphasizing that Google, like Microsoft, will offer Gemini services as elements of its flagship products, such as Google Sheets, Google Docs, and others. 

Fact 5: Advancements in Natural Language Processing (NLP) 

2023 witnessed remarkable progress in the field of Natural Language Processing. Researchers and companies globally made significant strides in improving the accuracy and versatility of NLP models. These advancements have led to more sophisticated understanding and the generation of human language by machines, paving the way for more intuitive and natural human-computer interactions. This year saw the deployment of advanced NLP in various applications, from customer service chatbots to complex data analysis tools, revolutionizing how we interact with technology daily. This progress in NLP technology not only enhanced existing applications but also opened new possibilities for AI in fields such as education, content creation, and multilingual communication. 

Expert Insight 

AI technologies are increasingly breaking the barrier of understanding natural language, gradually blurring the line between structured data previously used in IT systems and human knowledge. It seems that the creation of AGI (Artificial General Intelligence), a machine matching or even surpassing the average human in many aspects, is now just a matter of time. The challenge for the world of science, business, and politics will now be to direct the development of AI in a way that serves the broadly understood humanity and does not cause threats that many (probably rightly) fear. 

The last 12 months have been rich in interesting AI releases. The presentation of new large language models has opened up a range of possibilities for their implementation in everyday tasks, both in programming work and creative teams. European authorities are trying to keep up with these changes and adapt legal regulations to be in line with the current technological situation. In the coming months, we will certainly see more premieres, as leading players like Google and Microsoft compete to create solutions that utilize artificial intelligence. 

Artykuł A year under the sign of artificial intelligence development pochodzi z serwisu Inero Software - Software Consulting.