Introduction
Algonia Platform is a collaboration tool between developers, data scientists and system integrators that simplifies project creation and delivering to end users.
Architecture
Algonia Platform contains three parts:- Algonia Marketplace - online algorithms marketplace.
-
Algonia Visual Designer - on premise tool used by developers and data scientist for publishing algorithms and projects onto Algonia Marketplace.
On the other hand it's used by end users to visually combine downloaded algorithms into workflows.
-
Algonia Engine - lightning fast and low memory usage on-premise environment that executes visual workflows.
Algonia Engine is by far the most powerful visual programming language ever created, with more raw processing horsepower than anything available at any price.
Algonia terms
Algo consumersSystem integrators, business developers, data scientists, non-developers and developers that download pre-prepared algorithms and projects from Algonia Marketplace. Algo consumers can download individual Algos and combining them by themselves or already prepared Projects.
Consuming algorithms inside Algonia Platform doesn't require programming skills as development in Algonia Visual Designer is just drag'n'dropping downloaded algorithms, visually combining them and setting their properties.
Algo providers
Developers contribute with their existing or new algorithms, libraries and projects. Uploading algorithm to Algonia Marketplace is a simple procedure.
Every algorithm uploaded to Algonia Marketplace immediately becomes part of the existing ecosystem and therefore can be combined with other algorithms. Algo Providers can find new ways to monetize their software or to promote themselves by free algorithms contribution.
Algo
Smallest unit of Algonia Platform, represented in visual form. It's a package of functionality that can be combined with other Algos.
Visual Workflow
Connected Algos forming some application logic.
Project
Set of Visual Workflows that can be executed on workstations, servers and embedded devices.
Project Package
Distributable package of Project that can be installed on premises.
Algonia benefits
Algonia Engine is, at it's core, highly portable environment with unbeatable execution speed. Because of it's "box" development approach it forces systems modularity from the ground up.Lower development costs
System development doesn't start from scratch, but it's based on combining pre-prepared, already established algorithms. Visual approach is easy to understand, even for non-developers. Users with different roles can actively participate in development process. In case of need for specialized algorithms, open calls can be initiated and those algorithms will be delivered to you.
Lower maintenance costs
Systems are upgrading constantly. For that reason parts of application have to be modular enough when they need to be upgraded or replaced.
In many cases upgrade is just not possible or it heavily breaks existing functionality. Algonia handles this in the most painless way. It's just a matter of replacing or adding visual boxes.
Non-Intrusive
Complex systems are usually created with different technologies and tools. There is no tool that rule them all as everyone of them has different purpose.
Algonia Platform does not dictate you in any way in which portion must be used. But instead is built as highly flexible system for outer integrations.
That's why Algonia Platform offers unlimited extending flexibility and can behave in:
- Main role: Algonia's visual algorithms represents majority of system. Customer can inject custom scripts that fine tune workflow and manipulate data anywhere between those algorithms.
- Supporting role: Algonia's algorithms are involved to external system in minority. Algonia Engine could be included in 3rd party software as reference (library) where visual workflows can be called from client application. There is no restriction on clients application programming languages, because Algonia Engine is highly portable component written in C.
Developer friendly
Algorithm providers can easily pack their existing libraries, without need for major changes in their code.
We are aware the importance of debugging and for that reason tools like Visual Studio and Web-PDB are seamlessly integrated into Algonia Platform.
Once the package is created, it can be debugged inside Algonia Visual Designer with tools that developers are already familiar with.
See Algonia Platform in action.
Hello World Project
Preparing environment
If you prefer Docker, go to Docker Hello World tutorial, otherwise download portable Algonia Designer and Algonia Engine bundles according to your OS and extract it to your local folder: This tool is needed for creating projects with downloaded algorithms from Algonia Marketplace.Introduction
The tutorial will explain how to download Project from Algonia Marketplace and execute it on on-premise environment.This will be application that recognize objects from images downloaded from web.
It contains two Algos - first one downloads images from the web and the other one recognizes objects from those images.
Algos are connected into Visual Workflow and bundled into Project.
Downloading project
Navigate to Algonia Marketplace and select Algo offers:
Search for Object Recognition Project by typing #5 into search box (search is made by id instead of name).
Click on project to open it's details and then click on Download link.
Installing Project
Open Algonia Designer, click on Project->Install from toolbar and select ObjectRecognitionFromWebImages.algoproj package that you downloaded from Algonia Marketplace.Project contains three Algos. Don't worry if you don't understand them at the moment, here's just brief info:
- First one is Start and serves as entry point to application.
- Second is ImageDownloader which downloads images.
- Third is ObjectRecognition which recognize objects from downloaded images.
Downloading and installing Algos
Algo and Project installation are similar. We have two Algos in our Project, first one is ImagesDownloader with id #4 and second one ObjectDetection with id #3.Repeat same download procedure as you did for ObjectRecognitionFromWebImages Project.
To install Algos open Algo -> Install from the toolbar and search for downloaded ObjectDetection.algo and ImageDownloader.algo packages.
Running Project
Now you can run Project by pressing F5 or click Engine->Run from toolbar.* If you're using Dockers set properties first:
- Docker image name: tvinko/algonia:1.0.0-py38-netcore31 (pulled image from docker hub)
- Docker container name: object_recognition (name of the container that will be spawned from Docker Image)
You'll see recognition results in output log (name of the recognized object and it's confidence score).
If you open Project root by click on Project -> Reveal in explorer and navigate to assets\Images\results folder, you'll see images with marked detected objects.
ImageDownloader reads assets\urls.txt file and downloads images from that list. You can freely add your own image urls.
Now you can click on Stop Project and Close to get back to main view
Adopting Project To Different Environments
Congratulations, you just executed your first Algonia Project.You still might question yourself how can you adopt Algos to your needs. For example, you might have urls file on different location or you need results to be saved somwhere else.
It's impossible for algorithm author to know in advance how would end user environment look like and which algorithms are going to be connected to his own. That's why author exposes properties which end user can set to adopt it to their situation.
Let's start with ImageDownloader properties. Click on ImageDownloader node and Add args button.
You can see default values for those two properties:
First property is UrlFilePath which represents path to file that contains list of images urls. Second, ImagesDirectory, is path where downloaded images will be saved. Both paths are relative to Algonia Engine root directory.
Now click on ObjectRecognition node and Add args button to examine it's properties:
You can see that ImagesDir property points to location where ImagesDownloader will save images from web. This is data source location of images that ObjectRecognition algo will recognize objects from.
ImagesResultsDir property defines path where resulting images will be saved. Results are images with marked detected objects.
In the next tutorial you'll create a Distributable Package of your Project
Hello world package
In the previous tutorial you execute workflows directly from Algonia Designer. Thats perfectly normal in development phase, because you're creating application logic and you want to see your changes immediately.But when you're satisfied with results and you want to install it at your environment or ship it to your customer, there are at least two drawbacks when you execute it from Algonia Visual Designer
- It needs human intervention - someone needs to run it
- It's has significantly slower cold start when started from Algonia Visual Designer
Creating package
Packaging your application is simple process. Start Algonia Visual Designer, open template that you created in the previous tutorial and click on Template->Create package from menu bar.Packaging process could take few minutes, depending on size of your application. Result is portable AlgoniaEngine.zip package with all needed dependencies.
This package contains AlgoniaEngine executable that executes your workflows.
Manual optimizations
Automatic packaging process creates complete package - even with references that might not be needed.You can significantly reduce the size of the package, but this involves some manual work and understanding of Algonia Engine project structure.
Algonia Engine contains following folders:
- libs - contains all installed algorithms grouped by authors. Make sure that this folder contains just algorithms that are needed in your current project.
- NETCoreRuntime - Algonia Engine hosts complete .NET Core runtime for algorithms that are written in .NET Core. You can delete this folder if your project doesn't contains algorithms written in .NET Core.
- templates - folder with templates that is accessed only in development phase. When you create package, template is compiled and automatically published to Algonia Engine root. You can delete this folder if you don't have any project assets saved here.
Hello World From Docker
What is Docker?
If you're a Software Engineer or a Data Scientist, you probably would have heard about docker by now.Docker is an software tool that lets you package one or more applications together with all their dependencies in containers, thus making them easier to use. It provides the ability to package and run an application in a loosely isolated environment called a container.
You can download Docker or read more about it on official page
Algonia Platform And Docker
Docker separates file system to create isolation between the host system and containers. It also provides other features (networking) that can be applied to Algonia platform to run Algos from untrusted providers in more secure way.Algonia Platform offers out of the box Docker integration, however make sure that you also apply good practices for maximum security
Algonia Engine Docker Image is published on docker repository.
Installation
After Docker setup, make sure that you're running it in Linux Container mode.Next step is to pull Algonia Engine Docker Image. Image tag contains Algonia Engine, Python and .Net Core versions:
tvinko/algonia:algonia_engine_version-python_version-netcore_version
Open command prompt and enter following command (check Algonia Docker repository for latest versions):
docker pull tvinko/algonia:1.0.0-py38-netcore31
Downloaded Docker Image contains Algonia Engine version 1.0.0, Python version 3.8 and NetCore version 3.1.
After Algonia Engine Image is installed, download portable Algonia Designer according to your OS (other platforms are coming soon) and extract it to your local folder: Download and install Project (#5) and Algos (#3,#4) from Algonia Marketplace. You can follow instructions from Hello World Tutorial.
Running The Project
Start Algonia Designer, click Project->List and open ObjectRecognitionFromWebImages Project.We must define Docker Image that we're going to use. Algonia Designer will automatically run Docker Container with Algonia Engine inside and execute current Project.
Click on Settings->Open from toolbar and set following properties:
- Docker image name: tvinko/algonia:1.0.0-py38-netcore31 (pulled image from docker hub)
- Docker container name: object_recognition (name of the container that will be spawned from Docker Image)
Click Save to save configuration and return to main view and execute Project with click on Engine->Run Docker.
You can view execution log from Docker Logs window:
Now you can click on Stop Engine to delete container and return to main window.
Under The Hood
This section will give you overview how Algonia Engine Docker Image is created, so that you can apply your own security meassures on the top of it if necessary.Dockerfile that builds Image is:
FROM python:3.8.6-slim-buster
WORKDIR /home/algonia
RUN useradd docker
COPY ./out .
RUN cp libZenCommon.so /usr/lib && cp libZenCoreCLR.so /usr/lib
RUN apt-get update && apt-get install -y --no-install-recommends libgdiplus && apt-get install -y libgomp1
ENV DOTNET_SYSTEM_GLOBALIZATION_INVARIANT=true
RUN chown -R docker:docker /home/algonia/templates
RUN chown -R docker:docker /home/algonia/tmp
RUN mkdir /home/docker
RUN chown -R docker:docker /home/docker
USER docker
CMD ["sh", "-c", "/home/algonia/templates/${TEMPLATE_NAME}/start.sh ${TEMPLATE_NAME}"]
FROM python:3.8.6-slim-buster
RUN useradd docker
COPY ./out .
It's essential to know that Algonia Engine Container have full access to templates.
It's advisable that executing Project is only one under templates folder, as malicious Algo could change content of other Projects.
After all needed dependencies are installed, we change user:
RUN chown -R docker:docker /home/algonia/templates
RUN chown -R docker:docker /home/algonia/tmp
RUN mkdir /home/docker
RUN chown -R docker:docker /home/docker
USER docker
CMD ["sh", "-c", "/home/algonia/templates/${TEMPLATE_NAME}/start.sh ${TEMPLATE_NAME}"]
Along with executing Algonia Engine, this script installs all needed prerequisities defined by Algo providers. For example, this script can read Python Requirements.txt file and install components defined in requirements file.
You can see content of the script by clicking on Settings->Open from Algonia Designer toolbar and click on Modify Docker Script button.
Standard content of the script is:
#### START USER SCRIPT ####
#### END USER SCRIPT ####
#### START AUTO GENERATED - DO NOT MODIFY ####
/home/algonia/AlgoniaEngine /home/algonia/templates/$1/$1.compiled 0;
#### END AUTO GENERATED - DO NOT MODIFY ####
/home/algonia/AlgoniaEngine /home/algonia/templates/$1/$1.compiled 0;
Algorithm publishing
In the previous tutorials, we learn how to setup environment and execute Algonia Projects as end users. The following tutorials are resources for Algo Providers and System Integrators.
Let's define what is involvement of each group:
-
System Integrators are "gluing" Algos downloaded from Algonia Marketplace into Visual Workflows and Projects. They can fine-tune workflows by writting helper scripts between Algos - adopting
Algonia Project to end users environments.
Projects can also be published back to Algonia Marketplace.
- Algo Providers are contributing with algorithms and libraries - once the library is published to Algonia Marketplace, it becames part of the whole ecosystem and accessible to System Integrators.
- System Integrators can write helper scripts inside this Algo to fine tune Visual Worflow.
- Algo Providers can reference their libraries from it and include them into Algonia Platform. In this case it acts as wrapper between their algorithms and Algonia Platform.
Next Steps
In the next tutorials, you'll learn how to upload C# and Python algos on Algonia Marketplace.Publishing Algo to Algonia Marketplace is a two level operation:
- Algo Providers import their libraries into Algonia Designer locally. In this step they can test and debug how library interacts with other Algos.
- When they're satisfied with results, Algonia Designer creates package with all needed information and dependencies. Packages are *.algo files ready for upload to Algonia Marketplace.
C# Algonia Basics
Content coming soon...C# Algos
This tutorial demonstrates publishing C# algorithm on the Algonia Marketplace.We'll create simple library that downloads images from the web, but the techniques you'll learn in the tutorial are fundamental to publish any algorithm on Algonia Marketplace.
The tutorial is based on Hello World example. We're going to create ImagesDownloader from the scratch and publish it on Algonia Marketplace.
Open your favourite C# IDE and create new .NET Core 3 Class Library with ImageDownloaderAlgo class:
public class ImageDownloaderAlgo
{
string _urlsPath;
string _destinationPath;
public ImageDownloaderAlgo(string urlsPath, string destinationPath)
{
this._urlsPath = urlsPath;
this._destinationPath = destinationPath;
}
}
Constructor will be called by Algonia Engine and two arguments will be passed:
- urlsPath - path to file with urls of images
- destinationPath - path to directory where downloaded images will be saved
Now, we're going to define our main algorithm logic - downloading images from web.
public string Download()
{
List<DownloadStatus>
statuses = new List<DownloadStatus>();
if (Directory.Exists(Path.Combine(Environment.CurrentDirectory, _destinationPath)))
{
Console.WriteLine(string.Format("Clearing {0} dir...", Path.Combine(Environment.CurrentDirectory, _destinationPath)));
Directory.Delete(Path.Combine(Environment.CurrentDirectory, _destinationPath), true);
}
Directory.CreateDirectory(Path.Combine(Environment.CurrentDirectory, _destinationPath));
foreach (var url in File.ReadAllLines(_urlsPath))
{
using (WebClient client = new WebClient())
{
try
{
Console.WriteLine(string.Format("Downloading {0}", url));
client.DownloadFile(new Uri(url), Path.Combine(Environment.CurrentDirectory,
_destinationPath, Path.GetFileName(new Uri(url).LocalPath)));
statuses.Add(new DownloadStatus(url, 200));
}
catch (WebException webEx)
{
statuses.Add(new DownloadStatus(url, (int)((HttpWebResponse)webEx.Response).StatusCode));
}
catch
{
statuses.Add(new DownloadStatus(url, 500));
}
}
}
return JsonSerializer.Serialize<List<DownloadStatus>>(statuses);
}
Code is pretty much self explanatory.
First we check if destination directory exists and create one if doesn't.
Then we read urls from file line by line and download images from url to local path.
We save response status code for each downloaded image and return collection back to Algonia Engine in JSON form.
It's a good practice that algorithm returns some meaningful information, because you never know in advance which algorithm will be connected yours.
Maybe this information is not cruical in our case, but imagine online store that would like to check for broken images on their web store. Their algorithm would definitely need statuses of images.
To sucessfully compile this code, we'll also need DownloadStatus class:
using System;
using System.Collections.Generic;
using System.Text;
public class DownloadStatus
{
public string Url { get; set; }
public int Status { get; set; }
public DownloadStatus(string Url, int Status)
{
this.Status = Status;
this.Url = Url;
}
}
Here is complete code of our algo:
using System;
using System.Collections.Generic;
using System.IO;
using System.Net;
using System.Text.Json;
namespace Algonia
{
public class ImageDownloaderAlgo
{
string _urlsPath;
string _destinationPath;
public ImageDownloaderAlgo(string urlsPath, string destinationPath)
{
this._urlsPath = urlsPath;
this._destinationPath = destinationPath;
}
public string Download()
{
List<DownloadStatus>
statuses = new List<DownloadStatus>();
if (Directory.Exists(Path.Combine(Environment.CurrentDirectory, _destinationPath)))
{
Console.WriteLine(string.Format("Clearing {0} dir...", Path.Combine(Environment.CurrentDirectory, _destinationPath)));
Directory.Delete(Path.Combine(Environment.CurrentDirectory, _destinationPath), true);
}
Directory.CreateDirectory(Path.Combine(Environment.CurrentDirectory, _destinationPath));
foreach (var url in File.ReadAllLines(_urlsPath))
{
using (WebClient client = new WebClient())
{
try
{
Console.WriteLine(string.Format("Downloading {0}", url));
client.DownloadFile(new Uri(url), Path.Combine(Environment.CurrentDirectory,
_destinationPath, Path.GetFileName(new Uri(url).LocalPath)));
statuses.Add(new DownloadStatus(url, 200));
}
catch (WebException webEx)
{
statuses.Add(new DownloadStatus(url, (int)((HttpWebResponse)webEx.Response).StatusCode));
}
catch
{
statuses.Add(new DownloadStatus(url, 500));
}
}
}
return JsonSerializer.Serialize<List<DownloadStatus>>(statuses);
}
}
}
There is nothing new till now, we created an ordinary C# library in Visual Studio.
Next step is to prepare package in Algonia Designer and publish it on Algonia Marketplace.
Steps for package creation:
- Open Algonia Designer and click on Temlate -> List. Select ObjectRecognition that you created in hello world tutroial.
- Delete ObjectRecognition algorithm, as we'll replace it with our own.
- Click on Node->Add and select CS node. We will wrap our algorithm inside this node