import litellm 
 
# To use openai. 
response = litellm.completion(model="o3-mini", messages=[{"content": "Hello", "role": "user"}]) 
 
# To use deepseek. 
response = litellm.completion(model="deepseek/deepseek-reasoner", messages=[{"content": "Hello", "role": "user"}]) 

This simplified approach helped us compare models without worrying about implementation complexities. 

DeepSeek vs. OpenAI – model overview

Before diving into the AI model benchmarking results, let’s define key concepts and introduce the core specifications of the tested models. 

One of the most important parameters to consider in LLM benchmarking is the context window—the maximum input size a model can process at once. This is measured in tokens, which represent chunks of text rather than individual words. A larger context window allows the model to handle more extensive documents in a single request, which is particularly important when working with long statistical reports. 

The pricing for LLMs is typically based on token usage, which can vary depending on the type of tokens being processed. There are generally three types of tokens involved in LLM pricing: 

1. Input Tokens: These are the tokens representing the user’s input, such as the text or prompt sent to the model for processing. The cost of input tokens is charged based on the number of tokens provided by the user in each request. 
2. Cached Input Tokens: Some models offer a caching mechanism, where previously used inputs are stored and reused in subsequent requests, reducing the need for reprocessing. This is often charged at a lower rate than fresh input tokens, as the model does not need to process them again from scratch. 
3. Output Tokens: These tokens represent the text or response generated by the model. Output tokens are charged based on the amount of text the model generates in response to the user’s input. 

The models selected for this comparison are among the latest releases from the past several months. While they differ in pricing and capabilities, we aim to assess whether these differences translate into measurable performance variations. Below is a breakdown of the key characteristics of DeepSeek-R1, OpenAI 4o-mini, and OpenAI o3-mini: 

	DeepSeek-R1	OpenAI 4o-mini	OpenAI o3-mini
Context Window	128,000 tokens	128,000 tokens (with a maximum output of 16,384 tokens)	200,000 tokens (with a maximum output of 100,000 tokens)
Release Date	January 2025	July 2024	January 2025
Pricing (per 1 million tokens)	Input: $0.55 Cached input: $0.14 Output: $2.19	Input: $0.15 Cached input: $0.075 Output: $0.60	Input: $1.10 Cached input: $0.55 Output: $4.40
Input Formats	Text	Text, Images (including PNG, JPEG, GIF, WEBP)	Text
Output Formats	Text	Text	Text

PDF file used for testing

The document used for testing is composed of several chapters of report on the Polish and worldwide maritime economy in 2017-2020. The report is 50 pages long and includes various statistics and analysis of cargo traffic, shipping, shipbuilding, and other maritime industries. The data in the file is formatted in tables and text. Most of the data is presented in tables, with additional explanations and summaries in the surrounding text. Example pages of the document used for testing: 

Testing Methodology

We conducted a series of tests using the following maritime economy-themed prompts and a PDF file providing context information. Here are example prompts regarding information included in the PDF: 

• • Summarize the key economic findings from a maritime report. 

• • What is the total cargo turnover of Polish sea ports in 2020? 

• • What are the main cargo types handled by Polish sea ports? 

• • Which countries are the main trading partners of Poland in seaborne trade? 

• • What is the average age of ships in the Polish maritime transport fleet? 

• • What are the key economic indicators for the Polish shipbuilding industry? 

As mentioned before, we compared the following models: 

• • OpenAI’s 4o-mini 

• • OpenAI’s o3-mini 

• • DeepSeek’s deepseek-resoner (R1) 

We measured the following metrics: 

• • Inference Time – This refers to the time it takes for the model to generate a response after receiving a prompt. A lower inference time means faster responses, which is crucial for real-time applications and large-scale document processing. 

• • Token Usage – LLMs process and generate text in units called tokens. A token can be a word, part of a word, or even a punctuation mark. The total token usage includes both input tokens (the user’s query or document) and output tokens (the model’s generated response). The more tokens used, the higher the cost of the request. 

• • Response Cost – This is calculated as token usage × model pricing (per 1,000 or 1,000,000 tokens, depending on the provider). Since different models have different pricing structures, comparing response costs helps determine which model is more cost-effective for large-scale use cases. 

Test Results

Here are the summarized results from our tests (each test was repeated several times): 

Model	Average Inference Time (s)	Average Response Cost ($)	Average Input Tokens	Average Output Tokens
DeepSeek R1	57.2	0.0039	63961.7	751.6
o3-mini	13.8	0.0755	63251.5	1162.5
4o-mini	9.5	0.0511	62538.0	1046.5

Key Observations

• • Inference Time: DeepSeek consistently demonstrated longer inference times compared to both OpenAI models. This could be a significant factor for applications that prioritize fast processing. 

• • Response Cost: DeepSeek showed a competitive advantage in terms of cost, particularly for output tokens. Despite the longer inference time, DeepSeek’s overall cost per request remains lower than OpenAI o3-mini and 4o-mini. The lower response cost of DeepSeek can be attributed to its caching mechanism, which reduces the need to reprocess input data. Most of the input content, particularly the PDF file’s contents, was cached, leading to significant savings in processing costs. This caching system allowed DeepSeek to handle repeated queries more efficiently, making it a cost-effective option for processing large documents. 

• • Output Variability: The models varied in style and the level of detail in their responses. This is important depending on the context and user requirements (e.g., high-level summaries vs. detailed analysis). 

• • LiteLLM API: LiteLLM made it extremely easy to track cost, token usage, and response time directly from the API responses, enabling a straightforward comparison between models. 

Conclusion

Our tests highlight the advantages of using unified APIs for LLM benchmarking. LiteLLM significantly simplified the process, allowing us to focus on LLM efficiency assessment and evaluating AI language models. While DeepSeek R1 demonstrated competitive cost-effectiveness, particularly due to its caching mechanism, it was by far the slowest model in our tests, with an average inference time of 57.2 seconds. In contrast, OpenAI o3-mini and 4o-mini provided significantly faster response times, making them more suitable for real-time applications. 

Artykuł OpenAI vs. DeepSeek: A Technical Comparison Using Unified APIs pochodzi z serwisu Inero Software - Software Consulting.

Artykuł What are AI Agents and how can they help your company pochodzi z serwisu Inero Software - Software Consulting.

Optimization of Data Collection Process Thanks to AI Algorithms

In an era of progress and numerous achievements in natural language processing, we see increasing possibilities in the analysis and inference based on data collected in unstructured textual archives and documents. Therefore, streamlining business operations and implementing NLP (Natural Language Processing) and LLM (Large Language Models) tools become essential. The digitization of business processes moves many tasks into the virtual world, and controlling and analyzing a large volume of collected data becomes a significant challenge.

Save time

Of course, time is crucial. In the age of constant haste, laborious analytical tasks requiring focus, precision, and diligence become a burden for efficient and often dispersed teams. The modern business operates swiftly. Many industries, such as accounting, law firms, insurance, or transportation, place significant emphasis on the quality of documentation, which significantly impacts group efficiency.

Let’s take the transportation sector as an example. TSL companies handle dozens of freight shipments daily, each generating a large amount of documentation, including invoices, contracts, policies, delivery notes, and driver logs. When added to the internal administrative workflow, it’s easy to overlook an unreadable document (e.g., hastily scanned), which can disrupt the document circulation process at a later stage and require corrections, thus significantly extending the task list. This could affect cash flow. Therefore, optimizing the data collection process with the right quality becomes crucial.

Traditional methods are becoming less effective

Traditional software development methods are becoming less effective when dealing with large databases. They often rely solely on structurally processed data and predefined rules, which may not be suitable for working with unstructured text and the subtleties of natural writing. The classical approach to software development faces challenges when dealing with large knowledge bases.

Working on DocsQuality, we lower the barrier for teams to access NLP (Neuro-Linguistic Programming) and LLM (Large Language Model) tools, enabling them to easily retrieve data from unstructured texts and documents.

The role of DocsQuality software is to assess document attributes before they are imported into the accounting workflow. It evaluates real-time factors such as readability, clarity, and file quality. The software immediately highlights issues, for example, with email attachments, signaling that the system should not accept the document into the workflow at this stage.

NLP and LLM models

Combining and applying NLP and LLM models is an innovative method for assessing document quality. During software development, NLP can be used to extract text from image documents, utilizing Optical Character Recognition (OCR) techniques to transform image text into editable text, which can then be evaluated for quality. On the other hand, LLM models can read and understand text in PDF documents, allowing for assessment of readability, grammar, syntax, and spelling correctness, followed by content analysis, inference, and understanding.

The engineers at Inero Software are continuously developing the functionalities offered by DocsQuality. Creating new software in such a dynamic environment requires constant improvement and searching for optimal solutions. The potential of integrating NLP and LLM tools into this software will enable even more precise documentation analysis in the future.

DocsQuality software

If you are interested in DocsQuality software, we invite you to visit our website at https://docsquality.com/ and contact us via email at hi@docs-quality.com. We would be happy to answer all your questions and provide information about its functionalities and implementation. DocsQuality can be successfully integrated with an existing ERP system, expanding the offered functionalities to include effective monitoring of document quality, and allowing the identification of problematic files before they are introduced into the accounting workflow.

Artykuł Optimization of Data Collection Process Thanks to AI Algorithms pochodzi z serwisu Inero Software - Software Consulting.