Computer vision and object detection are increasingly used in the automation of business processes. Along with the dynamic development of technology, especially artificial intelligence, there are many new innovative business applications for this type of algorithms. In one of our recent posts, we outlined how to build a custom R-CNN based detector. In this article, we will walk you through how to do the same with YOLO v5, which has grown in strength in recent years. One of the important aspects over and above other solutions is the speed of inference. As part of this article, we will present the specified use case and all steps of its implementation. Enjoy your reading!

Use case

Signatures are still one of the most common methods of document authentication. Especially for enterprise applications, identification of signed and unsigned copies of documents in digital repositories may be time-consuming and a challenging task. However, automation supported by machine learning can make it easier. In this context, we describe the capabilities of YOLO v5 detector, and we will discuss issues of detecting invoice signatures. Let’s start with an example as shown in the figure below. For the purpose of this study, we prepared a fake invoice, fake data of the seller, buyer or the product itself. Let’s investigate now, how we can focus on hand-written signatures (which is fake too :-)).

Collecting Data

One of the first steps is to collect a set of images to train your model. For our use case, we have prepared a small 10 sample training set of invoices filled with different data and augmented this collection by a dedicated Python script. To make things easier, we used a Roboflow, which is a useful tool for data tagging Here you can upload your dataset and make a quick annotation process like on this GIF.

Additionally, we can generate more output images with random values ​​of rotation, saturation, exposure, noise, blur and other types of transformations. Moreover, with the help of this tool, we can determine our training, validation and test data split – the default is 70%, 10%, 10%. This is an important thing to prevent overfitting our model, you can read more about it on this blog. Thanks to the described tool, we can export our dataset in the YOLO v5 Pytorch format and put it in the directory of our project. The figure below shows the selection and target tree, where one of the most important files is the data.yml that will be used for training.

In our case after the first step of augmentation by a Python script, we generate 69 images. Additionally, we use a noise option in Roboflow, which allows us, in the end, export 169 annotated examples. Our images were ultimately split as shown in the figure below. 

And here you can see some of the train examples… 

Model configuration and architecture

The next step in the whole process is to define the configuration and architecture of the YOLO model. We may build our own network structure, although in version 5 we are provided with one of the following models:

• YOLOv5s,
• YOLOv5m,
• YOLOv5l,
• YOLOv5x.

We may use any of them but we must remember to assign appropriate target file parameter value called nc – number of classes. In our use case, it’s 1. The models differ from each other with the number of parameters used, speed of frames per second (FPS), accuracy and others… On the figure below you can see how they cope with the same COCO dataset. Click on the image below to learn more.

Training

What’s important, the Ultralytics company provides us with YOLO v5 developed in PyTorch, a framework specialized in machine learning. Thanks to this, we may download their repository from GitHub and train our own detector. Nothing simpler, but what else do we need for everything to work fine? 

To run each of the scripts we must install the dependencies contained in the requirements.txt. If we are using pip, we can use the following command in the terminal. Please remember that you must be in the project directory. 

pip install -r requirements.txt

In the installation process, one thing may be a problem. Namely, the PyTorch library…

The different operating system, package in use, programming language etc. may require a specified command. You will find everything you need in this tutorial.

Let’s assume you have passed all the requirements and start our training!

One line of code is enough to run this process, but we need to make sure that we have prepared two important files – data.yaml and for example yolov5l.yaml. The first one you should have from the data collection stage and the second one can be found in the YOLO v5 repository, in the Models directory. 

That’s it! Let’s run the training by the following command: 

python train.py --data dataset/data.yaml --cfg models/yolov5l.yaml --weights ''

The above command is the simplest possible, additionally, you can define the following options or parameters: 

• img-size,
• batch-size,
• epochs, 
• name,
• no-save,
• cache…

For the purposes of this article, we conducted a relatively short training for images resized to 416×416, batch size 32 and 1500 epochs. When the process is complete, the result will be a weight file named best.pt.

Perhaps you may stop your learning process if your weights are optimal for you at the moment. We stopped our training after 650 epochs, which took about 15 hours in total. Please note that the PC used does not have a dedicated graphics card.

Detailed metrics of our training:

• precision – 0.93507,
• recall – 0.96429,
• mAP_0.5 – 0.94755,
• mAP_0.5:0.95 – 0.48702,
  

where:

precision – measures how accurate are your predictions,

recall – measures how good you find all the positives,

mAP_0.5 – mean average precision for IoU* = 0.5,

mAP_0.5:0.95 – mean average precision for IoU* from 0.5 to 0.95 with a step size of 0.005,

* IoU (Intersection over Union) – measures the overlap between 2 boundaries. Find out more about these metrics here.

Inference on test images

Now, when we have our trained model trained after the learning stage, we may go to inference on test images. As you remember, one of the directories exported from Roboflow was called test. We can use it by putting in the terminal following line: 

python detect.py --weights best.pt --source dataset/test --conf 0.6 --img-size 600

where conf is model confidence – higher required makes fewer predictions. 

Finally, we may see the result visualization. The approximate inference time on one test image 416×416 was ~ 0.3 s, while for 700×700 ~ 0.8s.

Summary

For the implementation of the described detector, we used only 10 images, extending the dataset by augmentation process. Thanks to the Roboflow tool, it was possible to quickly annotate and export the data to the YOLO format. The learning process allowed us to obtain a target detector that effectively recognizes signatures on invoices.

Please note that with such a small dataset it is only adapted to recognize similar invoices. If we wanted to expand the possibilities of our detector, we would have to equip ourselves with a better graphics card and more, and more data. Additionally, you may be tempted to add a new class, e.g. to separate the objects into the Signature of the Seller and the Buyer.

Maybe you will see a new article in the future with more recognition classes. Stay tuned!

In Inero Software – software development agency, we are experts of digital transformation and smart automation. We provide knowledge and expertise on how to successfully use cutting edge technologies and data to shape corporate digital products of the future.

For more information, visit us on our website or follow us on LinkedIn. 

Artykuł Digital Cloud Document Repositories – how to identify signatures in scanned PDF documents pochodzi z serwisu Inero Software - Software Consulting.

Artykuł How Advanced Routing and Optimization can support an emergency response actions – use case study pochodzi z serwisu Inero Software - Software Consulting.

SchedulerFactory schedulerFactory = new StdSchedulerFactory();
Scheduler scheduler = schedulerFactory.getScheduler();

public class FileDownloadJob extends QuartzJobBean {

   @Override
   protected void executeInternal(JobExecutionContext jobExecutionContext) throws JobExecutionExceptions{
       // target code of performed operations
   }
}

JobKey jobKey = JobKey.jobKey((task.getId());

JobDataMap jobData = new JobDataMap();
jobData.putAll(new HashMap<>() {{
 	  put("email", user.getEmail());
  	  put("name", user.getName());
          put("progress", task.getProgress());
}});

JobDetail jobDetail = newJob().ofType(FileDownloadJob.class)
       				.withIdentity(jobKey)
      				.setJobData(jobData)
      				.build();

Trigger trigger = newTrigger()
       .startNow()
       .build();

scheduler.scheduleJob(jobDetail, trigger);

public class FileDownloadJob extends QuartzJobBean implements InterruptableJob

@Override
public void interrupt() {
   _interrupted = true;
}

if (_interrupted) {
   break;
}

scheduler.interrupt(jobKey);

jobDataMap.put("nextJob", nextTask.getId());

Artykuł Java: Scheduling the execution of background tasks using Spring pochodzi z serwisu Inero Software - Software Consulting.

One of our teams field of specialization is the use of artificial intelligence for modeling and optimizing complex transport models. From the practical point of view, the optimal planning of transport resources and supervision of their implementation is one of the key tasks in the area of ​​supply chain management.  In this type of solutions, using artificial intelligence and advanced computing IT systems can bring  observable benefits. One of the solutions this post refers to is the optimization of redistribution network of the supply chain (RCSN).

Pros, cons and uses

RCSN is an alternative to the centralized distribution networks (CDN) that consist of one central warehouse for the storage and distribution of goods. CDN is effective if we carry out large, repetitive orders to destinations that are close to each other. If our supply chain get’s more complicated we can add some a number of local warehouses to place the goods closer to final destination. While the CDN generally simplifies transport management, it does not allow to optimally distribute costs of supply for complicated logistic models. Though the RCSN is more complex and more expensive to manage compared to the classic versions of warehousing, with the support of a specified IT solutions it guarantees a stable level of services, maintaining the performance of order management and distribution. 

Use case

In this article, we will show how Inero Software team deals with the above-mentioned problems in an example use case. Let’s assume that we have a company with many branches and we need to exchange the necessary materials between them for proper functioning. To optimize costs and efficiency, we don’t always need to complete another large order from the supplier, but we exchange materials between nearby branches. It is a much better and faster solution, and one of the most important aspects is that it minimizes the workload. 

Let us assume, that in our showcase , we have 4 drivers and 40 company branches between, where each branch belongs to 1 of 4 groups( indicated by pin color in figure below). In the given use case, it was assumed that the goods may be stored and exchanged only between locations from the same group (a pin of the same color on the map), but the transport flow must be maximized between all locations. In order to optimize the total supply chain, it was assumed that the closest locations in a given group are the warehouses in the vincity of the particular driver’s home-base. This means that before the commodities are exchanged between individual points, the driver must pick up the goods from the defined warehouse and store them at the end of the daily route. 

In addition, when generating routes by the system, optimization algorithms takes under consideration following constraints: 

• each driver starts and finishes his/hers route at the warehouse. This means that the driver cannot leave any goods in the car for the night before/after the route;
• goods are intended to be transferred among one group and cannot not be stored in the local warehouse (destination) of other groups. However, commodities dedicated for different groups, can be transported by the same vehicle;
• every destination (location) can be considered as a local warehouse;
• the driver cannot exceed the driving time and daily mileage limit;

Moreover, some other specific limitations can be defined in the system, e.g. that the driver can only move in a restricted area, maximum 9 hours a day, can be defined. In this context the goal of the optimization is to maximize cargo flow between different branches of the company. 

The picture below shows an example of the result optimization of redistributing goods using our solutions.  

As you can see in the picture above, the system has generated a driving plan for each driver for each day of the week. Thanks to this, we can ensure the exchange of the necessary goods in a given area, even several times a week.  

As part of defining individual locations of the local warehouses, each of them was given a unique id in the range 10-49.  The illustration below shows the generated driver’s work plan divided into individual days. 

It can be seen that the simulation requirements were met. If we visit a certain group on a given day, we initially pick up the goods and finally store them in the same unit belonging to the same group.  

If the problem would require the exchange of goods between larger areas, e.g. from the western part of Poland to the northern part, etc., the implemented software could be extended with drivers’ meeting points and the exchange. Then, the cargo can be delivered depending on the demand to specific branches of the company. This process is based on the previously generated solution. The system’s algorithms determine which locations have been selected for drivers’ routes. On this basis, the route-based, the system finds the closest point on the map where drivers can potentially meet. If there are such places, they are added at certain times, (e.g. 12pm to 1pm, Monday) to the driver’s schedule. In the next stages, the system intelligently selects the remaining locations, maintaining continuity and increasing the area of exchange.

The figure below shows the location of meeting points on the Monday route for drivers’ routes (Szczecin – Gdańsk) and (Wrocław – Legionowo).   

Cargo swap points have been given a unique id to include them along the route. 

• (Szczecin – Gdańsk) – 100, 
• (Legionowo – Wrocław) – 101.

As you can see, meetings are held at designated times. Thanks to this operation, we exchange goods between larger areas, and in the next days we can distribute it to other locations. 

Solution

In our solutions, we use methods and techniques of artificial intelligence to model the problem of redistribution of goods and then optimize the commodities flow in distributed network. In situations where a large amount of calculations is required, various heuristic methods come to the rescue. They allow to minimize calculation costs and speeding up finding the optimal solution. Heuristic searches are suitable for complex problems. The artificial intelligence estimates quality of available solutions, and every iteration improves output result. In the case of a logistic redistribution network, every step of the optimization, extends the resulting node (i.e. in our case the route) with the node with the lowest value of the predefined evaluation function. 

In our solutions, for example, we use the greedy algorithm. The model at each step makes the optimal choice at the moment, based on the given evaluation function. In the case of transport problems, it will certainly be the number of kilometres imposed converted into costs compared to the current fuel price. The algorithm, by adding new customers along the route, makes the optimal decision, i.e. the cheapest one at the moment. 

The simplest form of the evaluation function can be defined as: 

cost = new_distance - current_distance

where:

new_distance – total distance to the new point from the driver’s location, 

current_distance – current distance, from the driver’s location to the actually last point. 

Summary

The use of IT systems supporting the supply chain redistribution network definitely allows for an optimized flow of goods. In addition, it can provide many additional analytical tools, such as: 

• order and invoice reports,
• archiving statistics,
• delivery time information system, 

and many others tailored to customer needs… 

Artykuł Redistribution supply chain network pochodzi z serwisu Inero Software - Software Consulting.

Thanks to the development of computing technology and artificial intelligence in recent years, applications using machine learning and systems based on neural networks have more and more business uses. Moreover, the advancement of technology has made machine learning, adapting advanced solutions based on neural networks cheaper and allows for relatively quick creation of solutions that have direct application, such as tracking objects on the camera image (tracking cars in parking lots, detecting people on construction sites) or extracting information from scans of documents (finding amounts and contractor data on an invoice). 

In the described material, we presented the methodology that we use as part of our ML-based work using R-CNN networks, which due to the universality of the structure and architecture of the model, work well in a wide range of areas. It is a type of neural network specifically designed to detect objects, which at least one layer is a convolution layer, allowing to distinguish the features of the manipulated image. Based on the input image, R-CNN determines potential ROI’s (regions of interest) by selective search segmentation. Then each of them is analyzed by a neural network to obtain the probability of the object classification. 

R-CNN detectors are widely used in: 

• driver assistance systems in autonomous cars, 
• face recognition, 
• anomalies detection,  
• health risk assessment and biomarkers of aging discovery, 
• Intelligent multi-site surveillance system, 

and many others… 

Building your own R-CNN detector is a simple task that does not require a lot of work. In this article you will see an example of its use and the next steps in the process. 

Use case

In order to demonstrate the possibility of building Your own detector based on R-CNN, the use case in this article will be the date detection on the transport document. Your solution may have different elements or the number of classes of objects to be recognized. A sample document is shown in the illustration below. As You can see, there are many irrelevant elements on this picture. In this case, we are only interested on each date. 

Software architecture

Programming tools enabling the implementation of the system in Python are, for example: 

• Tensorflow, 
• Keras, 
• OpenCV, 
• Scikit-learn. 

Once we have configured development environment, the first step in building the system is collecting the dataset. In order to make a R-CNN based detector with a high measure of accuracy, remember that: 

• the collection of images must be extensive and diverse, 
• as input to the model must be of the same dimension,
• eliminate duplicates that can cause network over-matching. 

For a good start, in the described case of the article it is enough to prepare even a few photos of the elements that we want to recognize, preferably against the background of other objects. Why such preferences? In the next step of the algorithm, should be a positive and negative samples generated from the prepared set. 

In order to generate the samples, we will use selective search segmentation.  The possibility of using this algorithm is provided by the OpenCV library.  This function uses hierarchical clustering to group pixels and then combine them into one based on color, texture or composition. The following is an implementation of the search code on the base image. The output of the process is a set of a potential ROI’s, depending on the size of the photo, even over 2.000 item’s.  

selective_search = cv2.ximgproc.segmentation.createSelectiveSearchSegmentation()
selective_search.setBaseImage(image)
selective_search.switchToSelectiveSearchFast()
rects = selective_search.process()

To divide the generated ROI’s, for example, we can use a metric called IoU (Intersection over Union). It’s defined as the intersection area divided by area of the union of a predicted bounding box and ground-truth box. Using this factor will allow You to filter when building a sample set. However, before we proceed to this operation, You should manually prepare the ground truth.  There are many different object tagging programs available on the internet. In this article, we will use the VoTT (Visual object Tagging Tool) originally developed by the Commercial Software Engineering (CSE) group at Microsoft.  

The above image shows the tagged date in the sample photo.  The operation should be repeated for all prepared data and exported to the appropriate format (JSON, XML, etc.). The generated files contain the exact coordinates and labels of previously marked objects. This allows us to compare our tagged data with those that will return a selective search segmentation. In this algorithm, we decided to set the IOU to 0.5, because of the good prediction result. You can modify this ratio depending on Your preferences and check whether the resulting data set is consistent with the assumptions. After executing the script, the filtered data should look like this (on the left – positives, on the right – negatives).  

As You can see, the positive samples present a variety of dates, while the negative other image elements.  On such a dataset, we can move on to building the R-CNN neural network model. A helpful library in this aspect will be the previously mentioned Tensoflow with Keras.  

After loading samples of the created model architecture You should split into test and training data. We can quickly do this with a single line of code using scikit-learn, when we can define the size of the test data (test_size) and the shuffle value (random_state). 

(self.train_x, self.test_x, self.train_y, self.test_y) = train_test_split(self.data, self.labels,
                                                                                  test_size=0.1,
                                                                                  stratify=self.labels,
                                                                                  random_state=42)

The Tensorflow library is already so extensive that it provides us with ready-made network architectures. The implemented algorithm uses the MobileNetV2 model, containing pre-trained weights from ImageNet. Here we define the dimension of each tensor, i.e. in this case the image, which is the input of the model. 

self.model = MobileNetV2(weights="imagenet", include_top=False, input_tensor=Input(shape=(224, 224, 3)))

To the ready model architecture, optionally we can add layers to improve the network. In this use case have been used: 

• pooling – combining the spatial data created by the convolution layer, minimizing the input data in both dimensions, 
• flatten – minimazing each tensor to one dimensional, 
• dense – enabling the connection of nodes in each subsequent layer using the activation function, 

• droput – consisting in a random data rejection. Allows for regularization to prevent overfitting. It also proved the flexibility of the neural network and the margin of response to input data.  

The next step in creating Your own R-CNN based detector is to compile the model with defined optimizer, metrics and loss function.  

self.model.compile(loss="binary_crossentropy", optimizer=Adam(lr=self.lr), metrics=["accuracy"])

This algorithm uses the Adam’s optimizer, the acurracy metric and the loss function binary crossentropy. The selected parameters are popular for similar use cases, but in the detailed Tensorflow documentation you can find something specific for Your solution.  

Now we can proceed to the training process. By properly defining the parameters of a function, we can do it with a line of code:  

self.model.fit(
    x=None, y=None, batch_size=None, epochs=1, verbose=1, callbacks=None,
    validation_split=0.0, validation_data=None, shuffle=True, class_weight=None,
    sample_weight=None, initial_epoch=0, steps_per_epoch=None,
    validation_steps=None, validation_batch_size=None, validation_freq=1,
    max_queue_size=10, workers=1, use_multiprocessing=False
)

Remember to choose the right learning parameters. Each epoch is modified with indivudal weights, which affects the quality of the network. In the algorithm results section of this article, You can see the parameters we selected and information on how to modify them. 

After completing the network training process, it is enough to save the model to a file and proceed to the next stage, i.e. object detection on test data. Now, for each input image, we once again perform a selective search segmentation and use the built-in model to predict individual ROI’s. For our own solution, we can define a minimum probability for which the regions will be considered valid. Typically, this value is in the range of 95-99%. Finally, all ROI’s that were filtered out in the previous operations can be marked on the target image by using simple function of the OpenCV library. The next chapter presents the effects of the R-CNN based detector constructed in this article. 

Results and summary

To implement detecor in this article were used only 10 images. It allowed to generate about 150 positive and 1000 negative samples. An important aspect of the network training itself is the selection of parameters. These include learning rate, batch size and number of epochs. The Tensorflow library provides tools for evaluating the solution, such as Tensorboard or the classification report. In the first one, we can observe different types of graphs, such as the accuracy and loss of the learning process every epoch. Additionally, the report automatically predicts the test data, returning the final precision. Thanks to this, even if we select parameters experimentally, we can deduce their further modification in order to improve the network.  

The created dataset and selected parameters (learning rate = 0.00001, batch size = 32, epochs = 100) allowed to obtain the final accuracy – 97%. Considering that the image’s collection was relatively small, good results were achieved. At the end of this article, let’s show the effects of the implemented date detector on sample images of transport documents.  

You can see in the pictures that the detecor is doing very effectively. If we would like to expand the developed detector, we would have to: 

• prepare an extensive set of data, various date cases and their location on the document, 
• train the neural network again, 
• depending on the metrics used, modify the parameters of the learning process, 
• perform a detection on test images. 

By following a few steps outlined in this article, You can create Your own solution by classifying a variety of objects. You can even upgrade your detector by adding more recognition classes. 

Inero Software provides knowledge and expertise on how to successfully use cutting edge technologies and data to shape corporate digital products of the future.

In the blog post section you will find other articles about IT systems and more!

Artykuł Few tips on how to create custom class detection system using R-CNN pochodzi z serwisu Inero Software - Software Consulting.