AI - Inero Software - Software Consulting https://inero-software.com/category/ai/ We unleash innovations using cutting-edge technologies, modern design and AI Mon, 17 Nov 2025 12:21:20 +0000 en-GB hourly 1 https://wordpress.org/?v=6.9.1 https://inero-software.com/wp-content/uploads/2018/11/inero-logo-favicon.png AI - Inero Software - Software Consulting https://inero-software.com/category/ai/ 32 32 153509928 Implementing an AI-Powered Telephony Service Center with ElevenLabs & LiveAPI https://inero-software.com/enterprise-ai-telephony/ Mon, 17 Nov 2025 11:18:27 +0000 https://inero-software.com/?p=8238 Implementing an AI-Powered Telephony Service Center with ElevenLabs & LiveAPI Over the past year, advancements in real-time AI models and high‑fidelity speech synthesis have accelerated the development of AI-driven telephony systems. At Inero, we’ve had the opportunity to integrate modern telephony solutions with LiveAPI technology and ElevenLabs’ voice engine to…

Artykuł Implementing an AI-Powered Telephony Service Center with ElevenLabs & LiveAPI pochodzi z serwisu Inero Software - Software Consulting.

]]>

Implementing an AI-Powered Telephony Service Center with ElevenLabs & LiveAPI

 

Over the past year, advancements in real-time AI models and high‑fidelity speech synthesis have accelerated the development of AI-driven telephony systems. At Inero, we’ve had the opportunity to integrate modern telephony solutions with LiveAPI technology and ElevenLabs’ voice engine to create a human‑like, responsive, GDPR‑compliant communication experience for a major corporate client.

This article combines two perspectives: a high-level overview of LiveAPI and ElevenLabs technology, and a behind‑the‑scenes look at our practical engineering experience while delivering a real-world AI telephony solution.

1. What Makes LiveAPI and ElevenLabs a Powerful Combination?

 

LiveAPI solutions such as OpenAI Realtime API and Google Gemini Live API shift the paradigm from static prompts to streaming, interactive communication. These systems support real‑time audio input, low‑latency responses, natural interrupt handling, and multimodal context.

ElevenLabs complements this with industry‑leading voice synthesis. Its realistic, expressive voices and advanced prosody control enable AI agents that sound convincingly human. For telephony environments, this matters — clients expect clarity, confidence, and a pleasant conversational tone.

Diagram showing the interaction flow between a user, a LiveAPI voice model, and ElevenLabs TTS in a real-time AI telephony system.

2. Why GDPR Compliance Shapes the Choice of API in Europe

For European organisations, GDPR compliance is not optional — it defines which AI vendors can be used in production. Although both OpenAI and Google offer real-time APIs, enterprises operating in the EU often restrict use to providers ensuring transparent, EU‑aligned data governance. In practice, this means that Gemini Live API was the viable choice for our implementation, while OpenAI was excluded despite strong technical capabilities.

3. Our Practical Experience Integrating Telephony with LiveAPI and ElevenLabs

Below we outline the key lessons, challenges, and engineering decisions from our implementation.

 

3.1 Project Context

Our client — a large corporate organisation — required a system capable of handling outbound and inbound calls automatically, while maintaining a tone and responsiveness extremely close to human interaction. The goal was not a simple IVR or menu system, but a natural, fully conversational experience driven by real‑time AI.

 

3.2 Technology Stack and Constraints

We evaluated both OpenAI and Gemini Live APIs to compare latency, contextual reasoning and streaming quality. However, due to GDPR compliance requirements, the production system was designed around Gemini Live API. ElevenLabs provided the speech synthesis layer, offering high realism and consistent quality across telephony channels.

Telephony with AI Processing Pipeline

3.3 Key Engineering Challenges

Beyond typical engineering concerns like audio quality, session stability, and call routing, the most demanding challenge was not purely technical — it was understanding how real users communicate over the phone. Subtle behaviors such as interruptions, hesitation, changing tone, or switching context required careful analysis and extensive testing.

We also dealt with several micro‑issues, such as premature call termination, incorrect end‑of‑utterance detection, and managing the timing between user speech and AI responses.

 

3.4 What We Built Ourselves

AI models are inherently non‑deterministic and cannot be fully controlled like classic software components. To ensure predictable and business‑aligned outcomes, we developed backend modules responsible for:
• Conversation flow supervision
• Session state tracking
• Monitoring and logging voice interactions
• Handling edge cases and ambiguous user inputs

ElevenLabs’ tooling, especially the Hard Disk service, significantly supported our workflow, but the orchestration layer was built entirely by Inero.

 

3.5 What We Learned

The most important insight: designing a telephony AI system requires deep understanding of the user’s context, combined with the business objectives of the project. Quick prototyping and iterative PoC testing were essential — allowing us to validate conversational patterns early, reveal unexpected user behavior, and refine the interaction design.

Ultimately, success depended on aligning the AI’s conversational style with how real customers naturally speak, pause, and respond during a phone call.

4. GDPR Considerations in AI Telephony

 

All audio handling, session storage, and logging were designed according to GDPR principles: strict data minimisation, no training on user audio, encrypted transmission, and optional anonymisation of transcriptions. Where possible, processing was routed through EU‑aligned infrastructure.

Conclusion

Implementing an AI‑driven telephony service center requires more than connecting APIs — it requires understanding users, managing nuanced conversational flows, and ensuring full compliance with EU regulations. Our experience shows that LiveAPI technologies combined with ElevenLabs can deliver highly human‑like, responsive, and scalable communication channels for enterprise clients.

Artykuł Implementing an AI-Powered Telephony Service Center with ElevenLabs & LiveAPI pochodzi z serwisu Inero Software - Software Consulting.

]]> 8238 Chatbot, Agent or AI Assistant? Find Out Which Solution Is Best for Your Business https://inero-software.com/chatbot-agent-or-ai-assistant-find-out-which-solution-is-best-for-your-business/ Thu, 08 May 2025 08:57:21 +0000 https://inero-software.com/?p=7947 Artificial intelligence and Large Language Models are buzzwords heard in nearly every industry. Many companies are wondering how to use them safely and which solution will be the most effective. There are plenty of options—and they’re often hard to tell apart. In this article, we break them down in a clear and easy-to-understand way.

Artykuł Chatbot, Agent or AI Assistant? Find Out Which Solution Is Best for Your Business pochodzi z serwisu Inero Software - Software Consulting.

]]>

Artificial intelligence and Large Language Models are buzzwords heard in nearly every industry. Many companies are wondering how to use them safely and which solution will be the most effective. There are plenty of options—and they’re often hard to tell apart. In this article, we break them down in a clear and easy-to-understand way.

AI can take on many roles in a company—as a chatbot, assistant, agent, data analysis tool, content generator, or knowledge search engine. So how can you choose the solution that best fits your employees’ needs? It helps to understand what each option has to offer.

Chatbot – answers questions, provides explanations, and handles requests

This is the most common use of AI in areas such as customer service and sales. An AI chatbot based on a large language model, such as ChatGPT, can hold natural conversations, understand the context of inquiries, and deliver accurate answers—24/7, in multiple languages, and without human involvement.

 

These solutions are typically implemented on websites, in messaging platforms (like Messenger or WhatsApp), or within helpdesk systems, where they assist with answering questions, tracking orders, or providing product information. As a result, they significantly automate customer service, reduce operational costs, and improve customer satisfaction ratings.

 

For the purposes of this article, we define a chatbot as an AI interface primarily intended for external users—in other words, it operates “outside the company.” This definition distinguishes it from AI agents, which perform more complex tasks within internal processes by integrating with systems, databases, or APIs.

https://www.incone60.eu/seastat

 

AI Agent – a tool designed to carry out specific tasks

Unlike a chatbot, which interacts with external users, an AI agent operates within the organization and supports employees by automating specific business processes. It’s not a one-size-fits-all tool—it’s built with a clearly defined purpose in mind, such as document processing, data analysis, or integration with ERP systems.

 

Thanks to large language models like Gemini or Claude, an AI agent can understand context, make decisions, and trigger specific actions—without human input. It can run in the background, process data from multiple sources, manage files, or handle email inboxes. Each AI agent is tailored to the company’s individual needs and specific tasks. Only then can it offer real value instead of becoming just another generic tool.

Want to see how this works in practice?


Check out our case study: Meet your personal AI agent-a case study for a freight forwarding company – where we describe how we built an agent integrated with an email inbox.

AI Assistant – supports users in daily work by operating contextually and “in the background”

Unlike a chatbot that answers questions or an agent that automates a specific process, an AI assistant is a tool that works alongside employees in real time—it understands context, suggests next steps, and makes tasks easier within familiar applications.

 

It’s typically integrated into a specific work environment, such as a word processor, spreadsheet, CRM, or project management tool. The assistant doesn’t replace the user—it actively supports them in making decisions, writing, analyzing data, or planning.

 

AI assistants like GitHub Copilot, Notion AI, or Google’s Workspace assistant show how this technology can genuinely boost team productivity and reduce time spent on routine tasks. From a business perspective, a well-designed assistant can improve work quality, reduce errors, and make onboarding new employees easier.

Other Business Applications of Large Language Models

The possibilities go far beyond chatbots, assistants, or agents. These models can take on specialized roles, supporting tasks such as document processing, data analysis, or content creation. They’re increasingly used to automatically summarize reports, extract information from unstructured sources (like emails, PDFs, or scanned forms), or answer natural-language questions based on internal documentation.

 

LLMs can also assist marketing teams by generating suggestions for ad copy, product descriptions, or sales messages tailored to the company’s style. In analytics departments, they provide faster access to data—generating database queries, interpreting results, and presenting insights in a way that’s easy for non-technical users to understand. These applications often don’t require building a new tool from scratch, but rather integrating the AI model into existing company systems. This way, the technology supports specific tasks—right where it’s needed.

AI Models and Data Security

Business owners and managers still approach AI tools with caution, mainly because they’re unsure how to ensure the security and confidentiality of processed data. We’ve explored these topics in previous publications that are worth reviewing.

 

In the article “AI User Privacy: An Analysis of Platform Policies”, we outlined the data privacy and model training policies followed by major AI providers such as OpenAI, Google Gemini, Microsoft’s Azure OpenAI, and Anthropic’s Claude.

 

For those considering an on-premise solution, we recommend the blog post “Top Lightweight LLMs for Local Deployment” There, we reviewed several top open-source lightweight LLMs and explained how to run them on a local Windows machine—even with limited GPU resources.

Choosing the right AI tool for your company depends primarily on the goal it’s meant to achieve. A chatbot works best where quick and accessible customer service is key. An AI agent can automate repetitive internal processes and improve information flow between systems. An AI assistant provides day-to-day support for employees—offering suggestions, summaries, or preparing data for further use.

 

Large language models also allow integration with existing processes—without the need to build a dedicated tool from scratch. However, implementing AI-based technology requires a well-thought-out decision, taking into account both efficiency and data security. If you’re looking to adopt AI in your company and need an experienced partner to guide you through the process, get in touch with us.

Bring AI into Your Business

We provide professional consulting and end-to-end implementation of tools based on large language models.
Contact Us

Artykuł Chatbot, Agent or AI Assistant? Find Out Which Solution Is Best for Your Business pochodzi z serwisu Inero Software - Software Consulting.

]]> 7947 AI User Privacy: An Analysis of Platform Policies https://inero-software.com/ai-user-privacy-an-analysis-of-platform-policies/ Wed, 30 Apr 2025 08:35:35 +0000 https://inero-software.com/?p=7890 In this article, we’ll break down the data privacy policies of top AI platforms. You will also learn what to do to ensure your data is not used for training Large Language Models (LLM).

Artykuł AI User Privacy: An Analysis of Platform Policies pochodzi z serwisu Inero Software - Software Consulting.

]]>

Ever wondered where your data goes when you interact with AI cloud platforms? Or is it used to train future models? In this article, we’ll break down the data privacy policies of top AI platforms. You will also learn what to do to ensure your data is not used for training Large Language Models (LLM).

Major AI cloud providers have become increasingly transparent about their data usage policies – especially when it comes to training models. While most platforms, particularly those offering enterprise-level services, do not use your inputs and outputs for training by default, the fine print matters. Understanding how these services handle your data – and how you can maintain control – is essential.

In this article, we’ll break down the data privacy and model training policies of top AI platforms, including OpenAI, Google Gemini, Microsoft’s Azure OpenAI and Anthropic’s Claude. You’ll learn:

    • How AI platforms use your data and whether your data is used to train models by default
    • How to prevent AI from using your data opt, if needed
    • Where your data is stored (data residency), and
    • What compliance measures (like GDPR) apply

Adopting AI isn’t just about prompt engineering or model performance. It’s also about knowing where your data goes—and how to ensure it stays under your control.

Here’s what you need to know:

OpenAI – Data Usage and Privacy

OpenAI treats your data differently based on how you interact with its services:

ChatGPT App (Web/Mobile)

When you chat with ChatGPT, your conversations may be used to train AI models – unless you manually opt out. To prevent your data from being used:

    • Go to Settings → Data Controls → Improve the model for everyone and toggle it off.
    • Even with the opt-out, OpenAI stores chats for 30 days for abuse monitoring before deletion.

OpenAI API and ChatGPT Enterprise

If you’re a developer or a business using OpenAI’s API or ChatGPT Enterprise, there’s no need to opt out. By default, OpenAI does not use API or Enterprise data to train its models, and your data stays private. You don’t need to do anything to opt out – it’s already protected. You can choose to share data to help improve the model, but only if you want to.

Data Residency

OpenAI’s servers are mostly based in the United States, and currently, if you’re using the API directly, you can’t choose where your data is stored. That means your data is processed within OpenAI’s own infrastructure – protected by strong security, but not necessarily hosted in your country. 

 

However, there’s some progress for enterprise users. OpenAI recently introduced an option for eligible enterprise API customers that allows data to be stored in Europe, provided there’s a specific agreement in place. 

 

If regional data residency is important for your business – say, for GDPR or internal compliance – you might want to consider using Azure OpenAI, which hosts OpenAI’s models on Microsoft’s cloud. With Azure, you can choose a region like Western Europe or Asia, and all data processing and storage will stay within that geography. 

 

We’ll dive into Azure more in the next section – but in short: OpenAI handles your data securely, but for strict control over where it lives, a partner cloud service like Azure may be a better fit. 

Google (Gemini) – Google’s Approach to Your Data

Google’s foray into generative AI includes Gemini, a next-generation model that powers products like Google Gemini (the chatbot) and various enterprise AI offerings on Google Cloud. Here’s how they handle your data: 

 

Gemini App 
 

By default, Google does save your Gemini chat history to your account (much like search history) and may use it to improve their service. However, Google provides a “Gemini Activity” setting to control this.  

 

To manage this: 

    • Visit Gemini Activity settings. 
    • Pause Gemini Activity to stop saving chats and prevent them from being used in AI model training data sources. 
    • You can also delete existing conversation history. 

Turning off Gemini Activity means your new chats won’t be used to improve their machine learning services, nor will they be seen by human reviewers, unless you explicitly submit them as feedback. This gives regular users a way to opt out, similar to ChatGPT’s opt-out toggle. 

To stop saving your conversations, go to the Activity tab and toggle Gemini Apps Activity. You can also delete your past conversations. 

API and Vertex AI

If you’re using Google Cloud’s Vertex AI platform: 

    • Your prompts and outputs are not used to train AI models without explicit permission. 
    • Data may be cached briefly (up to 24 hours) for performance but remains within your selected geographic region. 
    • Businesses can opt for a zero-retention policy for maximum privacy. 
Data residency

Data residency is a strong point for Google: you can choose which geographic region your AI service runs in (e.g. EU or US data centers), and Google will process and store data in that region to meet any data localization requirements. 

Microsoft Azure OpenAI – Enterprise Data Protection by Design

Training Policy

Microsoft’s Azure OpenAI Service lets companies use OpenAI’s models through the trusted Azure cloud platform. Privacy is a major selling point here. Microsoft is very explicit: any data you send into Azure OpenAI is not used to train the underlying models or improve Microsoft’s or OpenAI’s services .

 

Microsoft’s Azure OpenAI Service essentially hosts OpenAI’s models (GPT-4, GPT-3.5, etc.) within the Microsoft Azure cloud. Microsoft has specifically designed this service for enterprises that require strong privacy protections. Key aspects are: 

    • Any data you input into Azure OpenAI – prompts, completions (model outputs), embeddings, fine-tuning data – is not used to train the AI models.  
    • Your inputs and outputs “are NOT available to other customers, are NOT available to OpenAI, and are NOT used to improve OpenAI models”.  
    • Microsoft only retains data as needed to provide the service and monitor for misuse. In fact, prompts and outputs on Azure are stored only temporarily (up to 30 days) by default, and solely for abuse detection purposes. After 30 days, those prompts are deleted. If even this temporary storage is a concern (say, for ultra-sensitive data), Microsoft offers a process called “modified abuse monitoring” where you can request that even the 30-day storage be bypassed, meaning no prompts are retained at all. Typically, you’d need approval for this exception, but it’s an option for high-security scenarios. 
Data Residency

Because it’s on Azure, you also benefit from easily choosing the region and complying with data residency requirements. When setting up Azure OpenAI, you deploy the service to an Azure region (for example, East US, West Europe, Southeast Asia, etc.). All processing and data storage for inference will occur within that region or its geographical boundary. So, if you deploy in Western Europe, your data isn’t leaving Europe – crucial for GDPR compliance. Azure itself meets numerous compliance standards (SOC 2, ISO 27001, etc.), and these extend to Azure OpenAI as an Azure service. 

Anthropic (Claude) – A Privacy-First AI Assistant

Training Policy

Anthropic, the company behind the Claude AI assistant (Claude 2 and newer versions), has emphasized a privacy-conscious approach from the outset. Anthropic adopts an opt-in approach: 

    • By default, Anthropic does not use your conversations or data to train its models. This applies to both their commercial offerings (Claude for Work, Anthropic API) and consumer products (Claude Free, Claude Pro) – your prompts and Claude’s responses aren’t automatically used for model training.  
    • They only use data if you deliberately opt-in, such as by providing explicit feedback. For instance, if you click a thumbs-up/down in a Claude interface or send data to their feedback channels, you’re essentially saying “you can learn from this”. 

For enterprise clients, Anthropic offers Claude Team/Enterprise, which not only guarantees no training on your data but also provides admin controls. One such feature is custom data retention settings. By default, Anthropic’s systems might retain your inputs/outputs indefinitely for your account (though not for training). However, Claude Enterprise admins can set a retention policy – for example, you might set it to delete all conversation data after 30 days, 60 days, etc., with 30 days being the current minimum. These controls aim to support compliance with regulations like GDPR. 

Data Residency

Anthropic is a newer player, and currently, when you use their API directly, you don’t explicitly choose a data region – it’s likely hosted in the US by Anthropic (or possibly through cloud providers like AWS in the US region). However, Anthropic models are also available through partners, which can help with data residency. For example, Anthropic’s Claude is offered via Amazon Bedrock (AWS’s AI service) and via Google Cloud Vertex AI. If you use Claude through one of these platforms, you can take advantage of AWS’s or Google’s region controls. 

Conclusion

Understanding the data collection practices of LLM providers is crucial for AI compliance, customer trust, and corporate governance. Whether you’re focused on compliance, customer trust, or internal data governance, these insights help you make informed decisions. Choose providers that align with your privacy values – and always review your settings. 

 

Here’s a comparison of major platforms: 

Provider Default Data Training Web App Setting Data Residency Options GDPR/CCPA Compliance Privacy Policy
OpenAI No (API) Opt-out available No; (unless used via Azure Microsoft) Yes Consumer privacy
Google No (Cloud + Gemini) No training by default Broad region control Yes Enterprise privacy, Gemini privacy, Vertex AI
Azure No N/A Full regional control Yes Azure, OpenAI privacy
Anthropic No No training by default No (unless used via partners) Yes API users, Claude.ai users

For maximum privacy and control, local deployment (on-premises models) is always an alternative. This avoids cloud storage concerns entirely. You can read more about local deployment here. 

Let's talk about AI agents

Ready to bring AI into your business? Let us help you get started.
Contact us

Artykuł AI User Privacy: An Analysis of Platform Policies pochodzi z serwisu Inero Software - Software Consulting.

]]> 7890 Top Lightweight LLMs for Local Deployment https://inero-software.com/top-lightweight-llms-for-local-deployment/ Thu, 17 Apr 2025 09:50:46 +0000 https://inero-software.com/?p=7843 In this post, we’ll explore several top open-source lightweight LLMs and how to run them on a local Windows PC—whether CPU-only or with a limited GPU—for document processing tasks.

Artykuł Top Lightweight LLMs for Local Deployment pochodzi z serwisu Inero Software - Software Consulting.

]]>
Running large language models (LLMs) on your own hardware has become increasingly feasible thanks to lightweight LLMs—models with relatively small parameter counts that deliver strong performance without requiring server-grade GPUs. In this post, we’ll explore several top open-source lightweight LLMs and how to run them on a local Windows PC—whether CPU-only or with a limited GPU—for document processing tasks. We also include a benchmark comparing the models in terms of accuracy and inference speed, helping you choose the right model for your local environment and use case. 

What Are Lightweight LLMs (and Why Run Them Locally)?

“Lightweight” LLMs are models typically in the range of ~1–8 billion parameters – far smaller than GPT-3 class models – often optimized to run on a single GPU or even CPU. They are usually released as open models with freely available weights. These models trade some raw power for efficiency, but recent research and clever engineering (better data, distilled training, efficient attention mechanisms, etc.) have dramatically improved their capabilities. Many can now match or beat much larger models on specific benchmarks. 

Local deployment of such models is valuable for several reasons: 

    • Privacy & Security: All data stays on your machine, which is crucial for confidential documents like insurance contracts. You’re not sending sensitive text to a third-party API. 
    • Cost Savings: Once downloaded, local models run for free – no API usage fees or cloud compute bills. This can make a big difference if you process large volumes of documents regularly. 
    • Latency & Offline Access: Local inference eliminates network latency. Responses can be near-instant on a GPU, and you can operate entirely offline. This is useful for on-site workflows or when internet access is restricted. 
    • Customization: With local models you have full control – you can adjust parameters, prompts, or fine-tune models to better fit your domain (e.g. insurance data) without vendor limits. 

In short, lightweight LLMs put AI capabilities directly in your hands, on hardware you own. Next, we’ll compare some of the leading open models that are well-suited for local document processing. 

Comparing Top Lightweight LLMs

Lightweight open-source large language models (LLMs) are becoming a practical choice for organizations looking to run AI workloads locally. They offer a strong balance between performance, speed, and resource requirements—making them ideal for document summarization, extraction, and classification without relying on cloud infrastructure.  

We’ll focus on the following open-source models (each with downloadable checkpoints) that have a good reputation for quality relative to their size: 

    • Llama 3.1 – 8B parameters (Meta AI) 
    • StableLM Zephyr – 3B parameters (Stability AI) 
    • Llama 3.2 – 1B/3B parameters (Meta AI) 
    • Mistral – 7B parameters (Mistral AI) 
    • Gemma 3 – 1B and 4B variants (Google DeepMind) 
    • DeepSeek R1 – 1.5B and 7B variants (DeepSeek AI) 
    • Phi-4 Mini – 3.8B parameters (Microsoft) 
    • TinyLlama – 1.1B parameters (community project) 
  • These models range from very small (under 1 GB on disk) to mid-sized (~5 GB). All can be run in inference mode on a 16 GB GPU (often even in half-precision or 4-bit quantized form) and many are workable on CPU with enough RAM and patience. Table 1 summarizes their characteristics: 
Model Size on Disk (quantized) Max Context Licence
Llama 3.1 (8B) 4.9GB 128k tokens Open-source
StableLM Zephyr (3B) 1.6GB 4k tokens Only non-commercial use
Llama 3.2 (3B) 2.0GB 128k tokens Open-source
Mistral (7B) 4.1GB 32k tokens Open-source (Apache 2.0)
Gemma 3 (4B) 3.3GB 128k tokens Open-source
Gemma 3 (1B) 0.8GB 32k tokens Open-source
DeepSeek R1 (7B) 4.7GB 128k tokens Open-source (MIT licence)
DeepSeek R1 (1.5B) 1.1GB 128k tokens Open-source (MIT licence)
Phi-4 Mini (3.8B) 2.5GB 128k tokens Open-source
TinyLlama (1.1B) 0.6GB 2k tokens Open-source
Table 1: Lightweight LLMs for local use – model sizes and maximum context window. 

Notes: “Max Context” is the maximum sequence length (tokens) the model can process in one go.  

Next, let’s look at each model’s pros and cons, especially in the context of document tasks: 

    • Llama 3.1 (8B): Powerful general-purpose model; moderate size and strong multilingual capabilities. Heavy for CPU-only systems; requires chunking for long documents. 
    • StableLM Zephyr (3B): Ultra-lightweight, good for basic QA/extraction. Limited by small parameter count and commercial license restrictions. 
    • Llama 3.2 (3B): Excellent summarization and retrieval; long context support (128k tokens). Smaller size affects complex reasoning accuracy. 
    • Mistral (7B): Best overall performer for its size; highly efficient inference. Ideal for detailed summarization tasks. 
    • Gemma 3 (4B/1B): Offers multimodal capabilities and extensive multilingual support. The 4B model balances capability and speed; the 1B model best suited for simple tasks. 
    • DeepSeek R1 (7B/1.5B): Balanced efficiency and comprehension for general NLP tasks; limited complex reasoning compared to Mistral. 
    • Phi-4 Mini (3.8B): Exceptional reasoning, math, and logical capabilities; perfect for analytical document processing. English-focused. 
    • TinyLlama (1.1B): Extremely lightweight; suitable for basic text extraction/classification tasks. Limited contextual understanding. 

The models reviewed above cover a wide range of sizes and capabilities. Larger variants like Llama 3.1 and Mistral perform well on complex summarization and multilingual tasks but are less suited for CPU-only setups. Mid-sized models such as Llama 3.2 and Gemma 3 (4B) handle long inputs efficiently with reasonable performance. Smaller models, including TinyLlama and StableLM Zephyr, are lightweight and fast, making them practical for basic extraction or classification tasks. 

Models Benchmarking: Document Extraction and Summarization

Here we outline a simple model benchmarking plan covering two common document-processing tasks: 

  1. Information Extraction: We evaluated how well each model can extract specific fields from a policy or certificate. Specifically, we prompted each model to find the policy number, insured name, VAT ID, address and insurance period in the document text and return the structured output – clean JSON response with all the needed values.
  2. Summarization: Each model generated a concise summary of an insurance policy, covering key points such as coverage, exclusions, and conditions.We rated the summaries on clarity, correctness, factual accuracy and readability and penalized heavily fabricating information. 

We used 11 documents and ran all tests using Ollama (you can read about running model with Ollama here). The benchmarks were performed on a PC equipped with an NVIDIA GeForce RTX 2060 and 6 GB VRAM. To ensure consistent results, each model was run with temperature set to 0 for the extraction task (to produce deterministic outputs), and with a fixed temperature of 0.7 for summarization. For the extraction task, we also used structured outputs:  

 

{ 
        "model": "deepseek-r1:7b", 
        "prompt": "You are an assistant that extracts insurance-related information from a given input text. You must extract and return only the following fields: - policy_number,- insurance_period,- insured (company or person name),- nip (tax identification number),- address (of the insured). Return the output as a **clean JSON object** — not as a string, not inside quotes, and without any commentary. If a field is missing, use 'Not found'. Document text: ", 

    "stream": false, 
    "format": { 
    "type": "object", 
    "properties": { 
      "policy_number": { 
        "type": "string" 
      }, 
      "insurance_period_start": { 
        "type": "string" 
      }, 
      "insurance_period_end": { 
        "type": "string" 
      }, 
      "insured": { 
        "type": "string" 
      }, 
      "insured_nip": { 
        "type": "string" 
      }, 
      "insured_address": { 
        "type": "string" 
      } 
    }, 
    "required": [ 
      "policy_number", 
      "insurance_period_start",  
      "insurance_period_end", 
      "insured", 
      "insured_nip", 
      "insured_address" 
    ] 
  } 
} 
Examples of insurance certifacates. 

The table below presents the benchmark results. Extraction accuracy refers to the number of documents (out of 11) where the model successfully extracted all key fields. Token/sec indicates the model’s inference speed — how quickly it generates responses. 

Model Summarization Extraction Accuracy Tokens/sec
Llama 3.1 (8B) High-quality, no hallucinations 10/11 13.49
StableLM 3B Average quality, typos/hallucinations 4/11 56.51
Llama 3.2 (3B) Concise yet comprehensive summary, no hallucinations 8/11 49.49
Mistral 7B Extensive summary, factually correct 8/11 29.01
Gemma 3 4B Concise yet comprehensive summary, no hallucinations 10/11 13.37
Gemma 3 1B Concise yet comprehensive summary, no hallucinations 4/11 73.46
DeepSeek 7B Concise yet comprehensive summary, no hallucinations 6/11 16.39
DeepSeek 1.5B Very poor, frequent hallucinations/errors 0/11 66.45
Phi-4 Mini 3.8B Very concise summaries, factually correct 9/11 39.31
TinyLlama 1.1B Poor quality, severe hallucinations 2/11 107.34
Table 2: Benchmarking results.  

This scatterplot visualizes the trade-off between extraction accuracy and inference speed (measured in tokens per second) 

The benchmarking results reveal significant variations among the tested models.  

    • Bottom-right models – Llama 3.1 (8B), Gemma 3 (4B), and Phi-4 Mini (3.8B) – excel in summarization quality and extraction accuracy, consistently providing concise and accurate outputs. Phi-4 Mini seems to offer a good trade-off between speed and accuracy. 
    • Mistral 7B, DeepSeek 7B, Llama 3.2 generate detailed and informative summaries, though their extraction performance is more moderate. 
    • On the other hand, smaller models (on the top-left side of the chart) like StableLM Zephyr (3B), Gemma 3 (1B) and TinyLlama (1.1B) show significantly weaker extraction accuracy and are prone to frequent hallucinations. However, they benefit from faster inference times. Their limited context windows (e.g., 4k tokens) may contribute to these shortcomings. Overall, they may be suitable for only very basic tasks. 

Choosing the Right Model for Your Needs

When selecting a language model for document extraction or summarization, it’s all about balancing accuracy, speed, and hardware constraints. Below is a quick breakdown to help you pick the best fit—whether you need high precision, fast inference, or something lightweight for basic tasks. 

    • High Accuracy & Reasonable Speed: Choose Phi-4 Mini (3.8B), Gemma 3 (4B), or Llama 3.1 (8B) for robust extraction and summarization accuracy. 
    • Fast Inference & Moderate Accuracy: Opt for Llama 3.2 (3B) or StableLM Zephyr (3B) for simpler tasks on limited hardware. 
    • Balanced Performance (Accuracy-Speed Tradeoff): Mistral (7B) provides strong general-purpose capability suitable for detailed document summarization tasks. 
    • Low Resource Environments (Basic Tasks): Consider TinyLlama (1.1B) for quick extraction or classification on minimal hardware if accuracy isn’t critical. 

Conclusion

Lightweight LLMs are increasingly viable solutions for local deployment, particularly in document-intensive industries such as insurance. Models such as Phi-4 Mini, Gemma 3 (4B), and Mistral 7B provide strong performance in summarization, extraction, and classification tasks. Carefully balancing model size, inference speed, and accuracy ensures optimal outcomes, empowering organizations with affordable, private, and responsive AI solutions directly on owned hardware. 

This might interest you

Optimization of Back-Office Processes with AI Agent Implementation: A Practical Example
Read the full text

Artykuł Top Lightweight LLMs for Local Deployment pochodzi z serwisu Inero Software - Software Consulting.

]]> 7843 How to Prepare Your Company for AI Agent Implementation https://inero-software.com/how-to-prepare-your-company-for-ai-agent-implementation/ Tue, 08 Apr 2025 08:45:46 +0000 https://inero-software.com/?p=7741 This article explains what to focus on before deploying an AI agent, which areas of the business need to be well-prepared, and how to avoid common mistakes.

Artykuł How to Prepare Your Company for AI Agent Implementation pochodzi z serwisu Inero Software - Software Consulting.

]]>

Implementing an AI agent in a company is not only a technological challenge but also a strategic one. As more businesses consider using artificial intelligence in their daily operations—from customer service to document analysis—successful implementation requires careful planning. This article explains what to focus on before deploying an AI agent, which areas of the business need to be well-prepared, and how to avoid common mistakes.

There are many areas where AI can be helpful. From automating routine tasks, supporting customer service and data analysis, to streamlining decision-making processes and creating intelligent assistants that support team workflows. The potential is enormous—but the key lies in properly preparing the organization for this change.

Stages of AI Assistant Implementation

The process of implementing an AI assistant in an organization can be divided into several stages, each requiring specific actions. From analyzing business needs, selecting the right language model, and preparing the infrastructure, to integrating with existing systems and testing—each step impacts the overall effectiveness of the solution.

The key stages are:

  1. Needs analysis and readiness assessment
  2. Data and content preparation
  3. Solution design
  4. Assistant development and configuration
  5. Testing and pilot phase
  6. Deployment and maintenance

Needs analysis and readiness assessment

To ensure the best results from implementing an AI agent, start by asking yourself: which tasks and areas have the most potential for optimization through the use of artificial intelligence?

When looking for an answer to this question, it’s worth carefully analyzing your company’s current structure, processes, and employee responsibilities. This will help identify so-called “bottlenecks” that may affect the quality of services provided. These might include, for example:

    • long response times to quote requests

    • teams overloaded with routine tasks

    • lack of consistency in customer communication

    • manual processing of documents and data

    • difficulties in quickly accessing internal company knowledge

Based on this analysis, you’ll be able to identify areas for improvement as well as the people who will directly benefit from the support of AI assistants.

The second area that should be reviewed is the existing infrastructure. Implementing an AI assistant doesn’t require a large amount of hardware. If the company doesn’t want to invest in new machines, it can opt to use cloud services such as Azure, AWS, or Google Cloud.

Data is a crucial part of the preparation process. To fully leverage the potential of dedicated AI solutions, it’s important to understand that training the model behind the assistant requires datasets stored in digital form. These should be well-organized and kept in a central repository or database. The less structured the data, the higher the cost of implementing the assistant—and the greater the risk that the solution won’t meet expectations.

Data and content preparation

At this stage, it’s essential to gather all materials that contain important company knowledge—this may include PDF, Word, and Excel documents, website content, FAQ sections, emails, or data from databases.

 

Next, the collected information needs to be properly prepared—organized, cleaned of unnecessary content (e.g., unreadable PDFs), standardized where possible, and exported to CSV or JSON files (e.g., emails).

 

In some cases, such as when planning further model customization (fine-tuning), it will also be necessary to label the data or prepare a dedicated training set in the form of instructions and expected responses, for example:

{"prompt": "What documents are required to sign an OCS agreement?", "response": "The following documents are required to sign an OCS agreement: ..."}

Solution design

At this stage, decisions are made about the technical design of the solution. It’s important to define what type of assistant will best meet the company’s needs—whether it’s a simple chatbot answering questions, a more advanced assistant with access to company knowledge (so-called RAG – Retrieval-Augmented Generation), or an agent capable of independently performing specific tasks such as making bookings, generating reports, or sending emails.

The next step is selecting the appropriate technologies, including the large language model (LLM) that will power the assistant—such as GPT-4, Claude, Mistral, LLaMA, or Gemini—depending on specific needs and requirements related to privacy, cost, and integration capabilities.

Finally, it’s worth preparing a list of functions the assistant should perform and planning integration with other systems used in the company—such as the CRM, knowledge base, or email.

Assistant development and configuration

At this stage, both the technical backend and the user-facing part of the assistant (frontend) are developed. This could be, for example, a chat interface on the website, a button that launches the assistant in an application, or a widget integrated with tools like Slack. You can read more about how AI agent integration with the Slack communication platform can look here >>LINK

In parallel, the selected language model is deployed—via services such as Azure OpenAI, OpenAI API, Anthropic (Claude), Google Vertex AI (Gemini), or locally using open-source models like LLaMA, Mistral, or Mixtral.

 

If the assistant is meant to use internal company knowledge, a RAG (Retrieval-Augmented Generation) mechanism needs to be configured—enabling it to search and match relevant documents to user queries.

 

Finally, integrations with other systems—such as CRM, ticketing systems, or email—are implemented, allowing the assistant to meaningfully support the team’s day-to-day work.

Testing and pilot phase

After implementation, thorough testing of the solution is essential. The first step is functional testing—checking whether the assistant correctly understands user intent, responds in line with company documentation, and handles different types of queries appropriately.

 

The next phase is testing with end users (UAT – User Acceptance Testing), which helps assess how well the assistant performs in real-world scenarios and whether it meets employees’ expectations.

 

Based on feedback and observations, iterative improvements are made—such as adjusting responses, adding new documents to the knowledge base, or refining prompts and the agent’s logic. This phase is often repeated several times until a satisfactory level of quality is achieved.

Deployment and maintenance

After completing the testing phase, the assistant is deployed to the target infrastructure—this may be a public cloud (e.g., Azure, AWS, GCP), on-premise servers, or a hybrid solution, depending on security and availability requirements. More about this is covered later in the article.

 

It’s also necessary to set up monitoring, which allows you to track things like token usage, query frequency, error rates, and the quality of generated responses. This enables quick issue resolution and cost optimization.

In daily use, it’s important to keep the data up to date—adding new documents, removing outdated information, and updating the knowledge base the assistant relies on.

 

Over time, as business needs evolve, it may be worth considering retraining or fine-tuning the model—e.g., every few months—to better align it with the organization’s specific context.

 

Finally, it’s important to provide technical support and user assistance to ensure the solution is not only technically reliable but also convenient and intuitive for everyday use.

Data privacy

In the “Deployment and maintenance” section, we discussed the available options for choosing the infrastructure on which the AI agent will be deployed.

Each solution has its pros and cons. Choosing an on-premise setup gives you full control over the data, but it requires a dedicated machine with specific parameters.

Another option is using a public cloud service, such as Azure. Microsoft clearly states that data submitted to the Azure OpenAI service is not used to train or improve OpenAI or Microsoft models (source).

According to Microsoft, prompts and responses are not shared with other customers or OpenAI. Azure operates in full isolation mode: when using GPT-4 on Azure, no information from your conversations is shared with OpenAI LLC. Microsoft has confirmed this in a Data Processing Addendum (DPA).

AI decision accountability

It’s important to remember that formal and legal responsibility for the outcomes of an AI agent’s actions and the data it processes lies with the entity that implemented and oversees the solution—most often.

  1. the organization (e.g., the company that deployed the assistant),
  2. the system administrator,
  3. the individual making decisions based on AI suggestions (e.g., a customer service representative, recruiter, or doctor).

How to reduce risk?

  1. Human-in-the-loop (HITL) – A human must approve important decisions, while AI only supports the process (e.g., the assistant drafts a response, but a person approves it).
  2. Clear disclaimers and warnings – The AI should inform users: “I am an AI assistant – please verify my responses before making a decision.”
  3. Source verification – The AI assistant should, where possible, cite sources for its answers or indicate when it doesn’t know rather than guessing. Using RAG enables precise control over the knowledge base.

Summary

The process of implementing an AI agent must be well-planned and carefully considered. It may seem challenging at first, but with proper preparation, it can deliver long-term benefits. If you need support, feel free to contact us.

 

AI Agent in Your Company?

Write to us and find out how an AI Agent can support your company.
Contact

Artykuł How to Prepare Your Company for AI Agent Implementation pochodzi z serwisu Inero Software - Software Consulting.

]]> 7741 Deploying LLMs Locally: A Guide to Ollama and LM Studio https://inero-software.com/deploying-llms-locally-a-guide-to-ollama-and-lm-studio/ Fri, 04 Apr 2025 08:53:42 +0000 https://inero-software.com/?p=7692 Whether you’re building a custom chatbot, agent, an AI-powered code assistant, or using AI to analyse documents offline, local deployment empowers you to experiment and innovate without relying on external services.

Artykuł Deploying LLMs Locally: A Guide to Ollama and LM Studio pochodzi z serwisu Inero Software - Software Consulting.

]]>

Local deployment of Large Language Models (LLMs) is becoming increasingly popular among developers, tech enthusiasts, and professionals in industries like insurance and transport. Unlike cloud-based APIs, local LLM deployment offers greater privacy, offline accessibility, and complete control over resource optimization and inference performance. 

Running models like Llama 2 or Mistral directly on your hardware means your data stays on your machine — ideal for privacy-sensitive tasks such as processing insurance documents or working with proprietary transport data. There are no recurring API costs, and the performance depends solely on your system. Whether you’re building a custom chatbot, agent, an AI-powered code assistant, or using AI to analyse documents offline, local deployment empowers you to experiment and innovate without relying on external services. 

In this guide, we’ll explore two powerful tools that make this possible: Ollama and LM Studio. We’ll walk through installation, usage, and customization, helping you pick the best option for your goals. 

Getting Started with Ollama (CLI Tool)

Ollama is a lightweight, open-source command-line tool for running LLMs locally. It acts as a model manager and runtime, making it easy to download and execute open-source models (like Llama 2, Mistral, CodeLlama, etc.) on your machine. Ollama is available for macOS, Linux, and Windows, and it includes a local REST API for integration into applications. 

1. Install Ollama on Your System: Download the installer for your platform from the official Ollama website or use a package manager.

On Windows, download the OllamaSetup.exe from the website and run it. On Linux, you can install Ollama with one command: 

curl -fsSL https://ollama.com/install.sh | sh 

After installation, open a terminal/command prompt and verify it’s installed by checking the version: 

ollama --version 

This should display the installed Ollama version, confirming it’s ready to use, e.g.: 

ollama version is 0.6.2 

2. Download an LLM Model (“Pull” a Model): Ollama has a built-in model library. You can search their catalog on the website or simply pull a known model by name. For example, to download the 7B parameter Llama 2 chat model, run: 

ollama pull llama2:7b-chat 

This command fetches the model weights to your machine (it may take a while, as models are multiple GBs in size). You only need to pull a model once; afterward it’s stored locally. You can list all downloaded models with ollama list if needed. 

3. Run the Model Locally: Once downloaded, you can execute the model with the ollama run command. This will launch an interactive session where you can enter prompts and get responses. For example: 

ollama run llama2:7b-chat >>> What is the capital city of Poland? 

After running the above, Ollama will load the model and you’ll see an >>> prompt. You can then type your questions or instructions. The model (here Llama 2 7B chat) will generate a response to each prompt. For instance, you might ask “What is the capital of France?” and get an answer like “Paris is the capital of France.” printed in the terminal. Internally, the first run may take a bit to initialize, but subsequent prompts are answered interactively. Tip: You can also pass a one-off prompt directly in the command, e.g. ollama run llama2:7b “What is the capital city of Poland?“ will output a single response and return to the shell. 

You can also start Ollama as a background server with ollama serve. This enables the REST API on localhost:11434, which developers can use to integrate the model into apps via HTTP calls. You can ask the model by sending POST request, e.g.: 

curl http://localhost:11434/api/generate -d '{ 
  "model": "llama2:7b-chat", 
  "prompt": "What is the capital city of Poland?" 
}' 

The API returns newline-separated JSON objects, chunk by chunk, as the model generates the response: 

{ 
    "model": "llama2:7b-chat", 
    "created_at": "2025-04-02T15:19:17.1569954Z", 
 
    "response": "The", 
    "done": false 
} 
{ 
    "model": "llama2:7b-chat", 
    "created_at": "2025-04-02T15:19:17.268992Z", 
    "response": " capital", 
    "done": false 
} 
{ 
    "model": "llama2:7b-chat", 
    "created_at": "2025-04-02T15:19:17.3796491Z", 
    "response": " city", 
    "done": false 
} 
... 
{ 
    "model": "llama2:7b-chat", 
    "created_at": "2025-04-02T15:19:21.3106413Z", 
    "response": " Warszawa", 
    "done": false 
} 
{ 
    "model": "llama2:7b-chat", 
    "created_at": "2025-04-02T15:19:21.4619772Z", 
    "response": ").", 
    "done": false 
} 
{ 
    "model": "llama2:7b-chat", 
    "created_at": "2025-04-02T15:19:21.6296267Z", 
    "response": "", 
    "done": true, 
    "done_reason": "stop", 
    "total_duration": 5337417000, 
    "load_duration": 8625100, 
    "prompt_eval_count": 28, 
    "prompt_eval_duration": 854952300, 
    "eval_count": 15, 
    "eval_duration": 4472807400 
} 

If you set stream: false, the response is a single JSON object: 

curl http://localhost:11434/api/generate -d '{ 
  "model": "llama2:7b-chat", 
  "prompt": "What is the capital city of Poland?", 
  "stream": false 
}

You can also set a number of model parameters such as temperature by adding field options: 

curl http://localhost:11434/api/generate -d '{ 
  "model": "llama2:7b-chat", 
  "prompt": "What is the capital city of Poland?", 
  "options": { 
 "temperature": 0.2   
  } 
  "stream": false 
}'

4. Customize Models: Ollama supports a Dockerfile-like syntax called a Modelfile to create custom LLM variants. These let you: 

    • Start from an existing model (like llama3) 
    • Add custom system prompts 
    • Inject user-defined data (e.g., instructions, context) 
    • Set model parameters, like temperature 

Here is the simple example how you can create your custom assistant for processing insurance documents: 

FROM llama2:7b-chat 
 
PARAMETER temperature 0.7 
 
SYSTEM """ 
 
You are an assistant that extracts insurance-related information from a given input text. 
 
You must extract and return only the following fields: 
- policy_number 
- insurance_period 
- insured (company or person name) 
- nip (tax identification number) 
- address (of the insured) 
 
Return the output as a **clean JSON object** -- not as a string, not inside quotes, and without any commentary. If a field is missing, use "Not found". 
Example output format: 
{ 
  "policy_number": "...", 
  "insurance_period": "...", 
  "insured": "...", 
  "nip": "...", 
  "address": "..." 
} 
 
""" 
TEMPLATE """ 
{{ .System }} 
Input: 
{{ .Prompt }} 
Response: 
""" Lorem ipsum dolor sit amet, consectetur adipiscing elit. Ut elit tellus, luctus nec ullamcorper mattis, pulvinar dapibus leo.

To use Makefile, save it in a directory, e.g. insurance-assistant and create the custom model:

ollama create insurance-assistant -f insurance-assitant/Modelfile 

Then, you can use your model by providing the proper model name in a request: 

 curl http://localhost:11434/api/generate -d '{ 
  "model": "insurance-extractor", 
  "prompt": "", 
  "stream": false 
}' 

Ollama is purely CLI-based, so there’s no graphical interface. However, this makes it powerful for automation – you can pipe input/output, log responses to files, or call the Ollama API from code. In summary, with just a few commands, you have a privacy-protecting LLM running on your PC, ready to answer questions or assist in coding, all without any internet connection needed. 

Getting Started with LM Studio (Desktop App)

LM Studio is a user-friendly desktop application that lets you download and run local LLMs via a graphical interface. It’s cross-platform (Windows, macOS, Linux) and ideal for beginners who prefer not to use the command line. With LM Studio, you can chat with models in a nice UI, manage model downloads, and even run a local server to use the model in other apps. 

 

1. Install and Launch LM Studio: Download the installer for your OS from the LM Studio website and install it. After installation, launch the LM Studio app. The first time you open it, you’ll be prompted to download an AI model. You can choose from a list of popular open-source models. For example, you might select a smaller model like “Mistral 7B” or an instruction-tuned Llama2 variant to start. 

2. Run Your First Chat: Once the model is downloaded, LM Studio will load it into memory. You can then start a new chat session in the app. The interface typically has a text box where you can enter your prompt or question, and the model’s response will appear in the chat window. Simply type a query (for example: “What’s the capital of France?” or “Explain quantum physics simply.”) and hit Enter. The AI’s answer will be displayed as the “Assistant” reply in the chat. LM Studio conveniently shows the generation metrics: 

    • number of input and output tokens, 
    • tokens per second – you can see how fast the model is generating text, 
    • context occupancy, 
    • system resources usage (RAM and processor usage). 

3. Explore the Features: The LM Studio GUI provides additional features accessible to both beginners and advanced users: 

    • Model Library: A “Discover Models” or catalog section where you can download new models or update existing ones. You’re not limited to one model – you can have multiple models stored and switch between them. This means you have a wide selection: from small 3B parameter models for speed, up to 70B models if your system can handle them. 
    • Chat Interface: The main chat screen (as shown above) is where you interact with the model. Each new prompt you enter is answered by the model in a conversational format. You can have multi-turn dialogues, just like chatting with ChatGPT. There’s no need to manage a prompt history manually – the app keeps the conversation context. 
    • Advanced Settings: On the side panel, LM Studio offers configuration knobs for those who want more control. You can set a system prompt (a role or instruction that guides the AI’s behavior globally), adjust generation settings like temperature (creativity vs. consistency) and top-p or top-k sampling for controlling randomness, max tokens for responses, etc. These options let you fine-tune how the model responds without writing any code. For instance, you could set a system instruction like “You are a helpful coding assistant,”. This is a friendly way to customize behavior, though it’s not as extensive as programmatic control in a CLI tool. 
Advanced settings – simple example of AI assistant for processing insurance documents
    • Local API Server: For developers, LM Studio includes a “Local LLM Server” mode. Just switch to Developer tab, choose the model, and toggle Start button. It enables an API endpoint on localhost that mimics the OpenAI API, allowing other programs to send requests to your local model. This is powerful if you want to integrate the local LLM into your own applications (for example, connecting a chatbot UI or using the model for AI features in an IDE) while still benefiting from privacy and not relying on external services. 
Developer tab – you can enable local LLM server hosting your customized LLM. 

Using LM Studio is as simple as chatGPT – type and get answers – but entirely running on your hardware. The user-friendly interface lowers the barrier to entry, since you don’t need to use the terminal or remember commands. You get immediate, interactive AI responses, with buttons and menus to manage everything. 

Ollama vs. LM Studio: Tool Comparison

Both Ollama and LM Studio let you run LLMs locally, but they cater to slightly different audiences and use-cases. Here’s a comparison of key aspects to help you understand their differences: 

    • Interface & Ease of Use: LM Studio provides a polished graphical user interface, which makes it extremely approachable for beginners. It’s point-and-click with an integrated chat window, so no technical knowledge is required to get started. Ollama, on the other hand, is a command-line interface (CLI) tool (with an optional REST API). It offers a lot of power and flexibility but does require comfort with the terminal to use effectively. Beginners might find Ollama’s learning curve steeper, whereas LM Studio feels more plug-and-play. 
    • Supported Models: Both tools support a wide range of open-source LLMs. LM Studio can load any model in GGUF format (the standard for llama.cpp), meaning models like Llama 2 (7B, 13B, 70B), Mistral, Vicuna, Alpaca, CodeLlama, etc., as long as you have the hardware for them  
    • Use Cases Suited: Because of the above differences, LM Studio is excellent for users who want a personal ChatGPT-like assistant on their PC with minimal setup. It’s great for interactive Q&A, brainstorming, or casual use – you launch it when you need it, type queries, get answers. Ollama is ideal for developers or those who want to incorporate LLMs into projects or workflows. If you plan to experiment with prompts in scripts, fine-tune model behaviors, or build an app (like a chatbot, a coding assistant integration, etc.) that calls a local model, Ollama’s CLI and API give you that flexibility. 

Conclusion and Recommendations

Deploying LLMs locally has opened up a world of possibilities for developers and enthusiasts. We’ve discussed Ollama and LM Studio – two excellent tools that make local AI accessible. To recap some guidance on choosing between them: 

    • Choose LM Studio if you want a plug-and-play AI chat experience with a friendly GUI. It’s perfect for beginners or those who prefer not to tinker with command lines. You get quick setup, easy model downloads, and a nice chat interface for interactions. This might be best for someone who just wants an “offline ChatGPT” for personal use, note-taking, or idea generation without fussing over configurations. It’s also a convenient way to demo LLM capabilities to non-technical users (since it feels like a normal app). 
    • Choose Ollama if you want more control, automation, or integration. Developers and power users will appreciate its flexibility – you can script it, run it headless on a server, integrate the local LLM into your own apps via the API, and fine-tune model behavior with Modelfiles . If you’re comfortable with a terminal and want to customize how the AI works (beyond what a GUI allows), Ollama is a better fit. It’s also lightweight if you intend to run background AI services continuously. 

Finally, remember that the LLM itself (the model you choose) is as important as the tool. Spend time finding a model that suits your task – whether it’s a concise summarizer or a creative storyteller – and fits your hardware. Both Ollama and LM Studio make it easy to swap models, so you’re not locked in. The ecosystem of open-source models is growing rapidly, which means running a powerful AI on your own device is only getting easier and more common. 

In summary, deploying LLMs locally with these tools gives you the best of both worlds: AI capabilities similar to cloud services, but with privacy, control, and zero ongoing cost. Whether you go with a command-line power tool like Ollama or a user-friendly app like LM Studio, you’ll be joining the cutting edge of local AI development. Happy experimenting, and enjoy your new personal AI running right on your machine! 

Artykuł Deploying LLMs Locally: A Guide to Ollama and LM Studio pochodzi z serwisu Inero Software - Software Consulting.

]]> 7692 OpenAI vs. DeepSeek: A Technical Comparison Using Unified APIs https://inero-software.com/openai-vs-deepseek-a-technical-comparison-using-unified-apis/ Fri, 14 Mar 2025 13:35:14 +0000 https://inero-software.com/?p=7564 In this post, we conduct a comparative analysis of three popular LLMs—OpenAI’s GPT based models: 4o-mini and o3-mini, and open-source DeepSeek R1—to evaluate their effectiveness in reading and analyzing statistical data from large PDFs.

Artykuł OpenAI vs. DeepSeek: A Technical Comparison Using Unified APIs pochodzi z serwisu Inero Software - Software Consulting.

]]>

Large language models (LLMs) are increasingly used to analyze and extract insights from extensive documents, including lengthy statistical reports in PDF format. However, not all models perform equally when processing large files, especially those exceeding 50 pages. In this post, we conduct a comparative analysis of three popular LLMs—OpenAI’s GPT based models: 4o-mini and o3-mini, and open-source DeepSeek R1—to evaluate their effectiveness in reading and analyzing statistical data from large PDFs. Our assessment focuses on three key factors: accuracy, response time, and cost estimation for each model. 

To ensure a fair comparison, we utilized LiteLLM, a unified API that simplifies multi-model LLM benchmarking. By standardizing interactions across different LLM providers, LiteLLM allowed us to focus on evaluating LLM performance metrics rather than implementation differences. 

A Unified API Approach

Comparing open-source and proprietary LLMs from different providers can be challenging due to their varying APIs. To standardize our testing, we utilized LiteLLM, a library that provides a consistent interface for interacting with multiple LLMs. This allowed for easier switching between models and facilitated a more objective AI model comparison. Here is how easy it is to switch models using LiteLLM’s unified API: 

import litellm 
 
# To use openai. 
response = litellm.completion(model="o3-mini", messages=[{"content": "Hello", "role": "user"}]) 
 
# To use deepseek. 
response = litellm.completion(model="deepseek/deepseek-reasoner", messages=[{"content": "Hello", "role": "user"}]) 

This simplified approach helped us compare models without worrying about implementation complexities. 

DeepSeek vs. OpenAI – model overview

Before diving into the AI model benchmarking results, let’s define key concepts and introduce the core specifications of the tested models. 

One of the most important parameters to consider in LLM benchmarking is the context window—the maximum input size a model can process at once. This is measured in tokens, which represent chunks of text rather than individual words. A larger context window allows the model to handle more extensive documents in a single request, which is particularly important when working with long statistical reports. 

The pricing for LLMs is typically based on token usage, which can vary depending on the type of tokens being processed. There are generally three types of tokens involved in LLM pricing: 

  1. Input Tokens: These are the tokens representing the user’s input, such as the text or prompt sent to the model for processing. The cost of input tokens is charged based on the number of tokens provided by the user in each request. 
  2. Cached Input Tokens: Some models offer a caching mechanism, where previously used inputs are stored and reused in subsequent requests, reducing the need for reprocessing. This is often charged at a lower rate than fresh input tokens, as the model does not need to process them again from scratch. 
  3. Output Tokens: These tokens represent the text or response generated by the model. Output tokens are charged based on the amount of text the model generates in response to the user’s input. 

The models selected for this comparison are among the latest releases from the past several months. While they differ in pricing and capabilities, we aim to assess whether these differences translate into measurable performance variations. Below is a breakdown of the key characteristics of DeepSeek-R1, OpenAI 4o-mini, and OpenAI o3-mini: 

DeepSeek-R1 OpenAI 4o-mini OpenAI o3-mini
Context Window 128,000 tokens 128,000 tokens (with a maximum output of 16,384 tokens) 200,000 tokens (with a maximum output of 100,000 tokens)
Release Date January 2025 July 2024 January 2025
Pricing (per 1 million tokens) Input: $0.55
Cached input: $0.14
Output: $2.19		 Input: $0.15
Cached input: $0.075
Output: $0.60		 Input: $1.10
Cached input: $0.55
Output: $4.40
Input Formats Text Text, Images (including PNG, JPEG, GIF, WEBP) Text
Output Formats Text Text Text

PDF file used for testing

The document used for testing is composed of several chapters of report on the Polish and worldwide maritime economy in 2017-2020. The report is 50 pages long and includes various statistics and analysis of cargo traffic, shipping, shipbuilding, and other maritime industries. The data in the file is formatted in tables and text. Most of the data is presented in tables, with additional explanations and summaries in the surrounding text. Example pages of the document used for testing: 

Testing Methodology

We conducted a series of tests using the following maritime economy-themed prompts and a PDF file providing context information. Here are example prompts regarding information included in the PDF: 

    • Summarize the key economic findings from a maritime report. 
    • What is the total cargo turnover of Polish sea ports in 2020? 
    • What are the main cargo types handled by Polish sea ports? 
    • Which countries are the main trading partners of Poland in seaborne trade? 
    • What is the average age of ships in the Polish maritime transport fleet? 
    • What are the key economic indicators for the Polish shipbuilding industry? 

As mentioned before, we compared the following models: 

    • OpenAI’s 4o-mini 
    • OpenAI’s o3-mini 
    • DeepSeek’s deepseek-resoner (R1) 

We measured the following metrics: 

    • Inference Time – This refers to the time it takes for the model to generate a response after receiving a prompt. A lower inference time means faster responses, which is crucial for real-time applications and large-scale document processing. 
    • Token Usage – LLMs process and generate text in units called tokens. A token can be a word, part of a word, or even a punctuation mark. The total token usage includes both input tokens (the user’s query or document) and output tokens (the model’s generated response). The more tokens used, the higher the cost of the request. 
    • Response Cost – This is calculated as token usage × model pricing (per 1,000 or 1,000,000 tokens, depending on the provider). Since different models have different pricing structures, comparing response costs helps determine which model is more cost-effective for large-scale use cases. 

Test Results

Here are the summarized results from our tests (each test was repeated several times): 

Model Average Inference Time (s) Average Response Cost ($) Average Input Tokens Average Output Tokens
DeepSeek R1 57.2 0.0039 63961.7 751.6
o3-mini 13.8 0.0755 63251.5 1162.5
4o-mini 9.5 0.0511 62538.0 1046.5

Key Observations

    • Inference Time: DeepSeek consistently demonstrated longer inference times compared to both OpenAI models. This could be a significant factor for applications that prioritize fast processing. 
    • Response Cost: DeepSeek showed a competitive advantage in terms of cost, particularly for output tokens. Despite the longer inference time, DeepSeek’s overall cost per request remains lower than OpenAI o3-mini and 4o-mini. The lower response cost of DeepSeek can be attributed to its caching mechanism, which reduces the need to reprocess input data. Most of the input content, particularly the PDF file’s contents, was cached, leading to significant savings in processing costs. This caching system allowed DeepSeek to handle repeated queries more efficiently, making it a cost-effective option for processing large documents. 
    • Output Variability: The models varied in style and the level of detail in their responses. This is important depending on the context and user requirements (e.g., high-level summaries vs. detailed analysis). 
    • LiteLLM API: LiteLLM made it extremely easy to track cost, token usage, and response time directly from the API responses, enabling a straightforward comparison between models. 

Conclusion

Our tests highlight the advantages of using unified APIs for LLM benchmarking. LiteLLM significantly simplified the process, allowing us to focus on LLM efficiency assessment and evaluating AI language models. While DeepSeek R1 demonstrated competitive cost-effectiveness, particularly due to its caching mechanism, it was by far the slowest model in our tests, with an average inference time of 57.2 seconds. In contrast, OpenAI o3-mini and 4o-mini provided significantly faster response times, making them more suitable for real-time applications. 

Artykuł OpenAI vs. DeepSeek: A Technical Comparison Using Unified APIs pochodzi z serwisu Inero Software - Software Consulting.

]]> 7564 Meet Your Personal AI Agent: A Case Study for a Freight Forwarding Company https://inero-software.com/meet-your-personal-ai-agent-a-case-study-for-a-freight-forwarding-company/ Fri, 21 Feb 2025 11:27:19 +0000 https://inero-software.com/?p=7341 AI-driven tools are becoming increasingly prevalent across various industries, streamlining processes from simple graphic design and translations to advanced document, email, and database analysis. In this article, we will present a practical business application of an AI assistant in action. AI Agents have a wide range of applications, and their…

Artykuł Meet Your Personal AI Agent: A Case Study for a Freight Forwarding Company pochodzi z serwisu Inero Software - Software Consulting.

]]>
AI-driven tools are becoming increasingly prevalent across various industries, streamlining processes from simple graphic design and translations to advanced document, email, and database analysis. In this article, we will present a practical business application of an AI assistant in action.

AI Agents have a wide range of applications, and their full potential is still being discovered. The main advantages of AI-powered assistants include:

1. Automating Routine Processes

AI agents can handle repetitive tasks such as customer inquiries, document analysis, and data management. By automating these processes, businesses can reduce operational costs and improve efficiency.

 

2. Personalized Customer Interactions

By analyzing data, AI agents can provide personalized recommendations and tailored offers, enhancing customer engagement and improving overall user experience.

 

3. Speed and Availability

AI operates 24/7, delivering instant responses and real-time support. This is particularly valuable in industries that require quick reaction times, such as e-commerce, finance, and logistics.

 

4. Advanced Data Analysis

AI-powered agents can process vast amounts of data in a short time, identifying patterns and correlations that support better business decision-making.

 

5. Optimizing Decision-Making Processes

With predictive modeling, AI assists in demand forecasting, risk management, and supply chain optimization, helping organizations make more informed decisions.

 

6. Seamless Integration with Existing Systems

Modern AI solutions can be easily integrated into existing ERP, CRM, and analytics platforms, enhancing their capabilities and improving overall system efficiency.

A Practical Example of AI Agent Use in the Transport Industry

AI agents can be applied across various industries, including banking, sales, and human resource management. In this text, we will focus on a freight forwarding company that handles anywhere from a few to dozens of shipments daily.

 

 

Freight forwarders deal with constant communication and the verification of numerous documents. Each of these tasks takes time—a resource that is often in short supply—making errors more likely when the workload is high.

 

 

How can time management be improved? By automating repetitive and predictable tasks. This is where an AI Agent comes in. Here’s an example of an AI assistant we developed, powered by Google’s Large Language Model, Gemini.

 

One possible application is the following scenario:

 

 

A freight forwarder receives an email that should include an insurance policy along with proof of payment. The AI Agent automatically, without needing to be prompted, checks whether the email contains the required attachments. If they are included, it proceeds to verify the following details:

In the Insurance Policy:

      • Policy number
      • Insurance period and whether it is currently valid
      • Insured party details, including tax identification number and address
      • Bank account number for premium payment

In the Payment Confirmation:

      • Payment reference
      • Amount
      • Bank account number
      • Payment date
      • Whether the transfer corresponds to the submitted policy (e.g., based on the reference, account number)

The AI Agent then transfers the extracted data into a designated Excel file, which is continuously updated. The data file can be formatted accordingly, for example, by highlighting entries in red where the insurance policy is invalid or the payment has not been verified. 

In this simple way, instead of searching through their inbox for the right emails, the freight forwarder can check the Excel file to see if the documents have been received from a specific sender and whether they are correct. This saves a significant amount of time and ensures data accuracy.

 

There are many ways to further develop our AI Assistant. It can be integrated with other tools, such as Slack or other communication platforms, to send notifications about missing documents or generate automated email responses. An AI-powered agent can be tailored to the specific needs of a company, a department, or even an individual role.

Do you want to explore the possibilities of AI Agents?​

Schedule a meeting. We’d be happy to discuss the possibilities.
SCHEDULE A MEETING

Artykuł Meet Your Personal AI Agent: A Case Study for a Freight Forwarding Company pochodzi z serwisu Inero Software - Software Consulting.

]]> 7341