Realtime Image Processing (JavaFX, OpenCV)

This will be somewhat similar to my previous post about ant algorithm simulator, so i would recommend you to check it out first to get an idea of JavaFX. Realtime image processing is one of my hobby projects (started at 2010) that uses JavaFX and OpenCV. It is basically a desktop application where you can edit images from a video or a webcam feed in realtime and see the results. There are multiple operation types and you can apply them on top of each other.

There is a video that explains what it is doing. So i will not go into the details of the program. But i will tell you the story of this application, how did i converted it into a JavaFX Maven project and how can you run it. Just like the ant algorithm simulator, i will trash my own app and talk about possible future improvements. First things first, you can watch the video here.

What is image processing

Since artificial intelligence is very popular these days, there is not much need to explain what is image processing. Because most of you probably know this term from artificial intelligence studies like detecting humans or counting objects or preventing crashes. But before all this, image processing algorithms are improved drastically so that you can apply them on images without much processing power.

An image in computer vision is basically just a matrix. It has rows and columns and a depth. Lets call each of these units pixels. These pixels are combined of 3 color information (depth) and sometimes 1 opacity information, which is called alpha. Most of the time you are not much bothered with opacity but the color information can be defined with different methods. For example if you use additive or subtractive or digital colors, you would get RGB (red, green, blue) or CMY (cyan, magenta, yellow) or HSI (hue, saturation, intensity) compositions. These 3 colors means that every pixel has 3 components. So our matrix is (M x N x 3) martix. Keep this in mind.

After you define the matrix, all of the image processing infrastructure is built on top of this data structure. But matrix means linear algebra. So yeah, it is all about mathematics. Nothing fancy about images or any artificial intelligence. I know some of the math that requires to filter an image but applying it on the image efficiently is a whole different game. That is why we have libraries like OpenCV or OpenCL or some others probably for python or c++.

Let me give you the brief history of this application. When i was a bachelor student, i have learned C# (2006 - 2011) and showed the courage to make my graduation project about image processing by using Java Swing. My mentor suggested me to use JMF (Java Media Framework) for image processing and showed me an example. I had basic java back then but it took around 5 months to understand java, JMF and image processing techniques because it was realtime and reading a video or a webcam feed. That project taught me java. I remember the first time i was able to draw a line on the video, it took me 2 months.

Anyway, i have completed it, graduated and years passed. When it was 2016 i have seen that the JMF framework is no longer supported and maintained and has problems with 64 bit systems, i decided to re-write it. I have found out that there is a new library called OpenCV and it works on java. Also, i have learned JavaFX and loved it. So i have turned it into a JavaFX project with OpenCV. It was firstly a Netbeans project and i have uploaded the video on youtube. Years passed again :) I have decided to upgrade it to OpenCV 4 (it was using 3) and make it a maven project. This blog post is the result of this desicion. Enough with the history. Let me briefly go over the technical details of runing the project.

Running the application

This project was developed with Netbeans using JDK 8 and JavaFX without maven. You needed to configure OpenCV on your computer to be able to run it. Then JavaFX got open sourced to community and removed from JDK. So if you want to create a JavaFX maven project, you better use JDK 11. After you install JDK, you can fork and clone the codes from github. There is no need to configure any OpenCV or JavaFX since they are inside the pom file as dependencies. You should be able to clean and install the maven application. Here is the pom dependencies and plugins required.

			
<dependencies>
	<!-- https://mvnrepository.com/artifact/org.openjfx/javafx-controls -->
	<dependency>
		<groupId>org.openjfx</groupId>
		<artifactId>javafx-controls</artifactId>
		<version>15.0.1</version>
	</dependency>
	<!-- https://mvnrepository.com/artifact/org.openpnp/opencv -->
	<dependency>
		<groupId>org.openpnp</groupId>
		<artifactId>opencv</artifactId>
		<version>4.3.0-3</version>
	</dependency>
</dependencies>

<build>
	<sourceDirectory>imageprocessing/src</sourceDirectory>
	<resources>
		<resource>
			<directory>imageprocessing/src</directory>
			<excludes>
				<exclude>**/*.java</exclude>
			</excludes>
		</resource>
	</resources>
	<plugins>
		<plugin>
			<artifactId>maven-compiler-plugin</artifactId>
			<version>3.8.1</version>
			<configuration>
				<source>${java.version}</source>
				<target>${java.version}</target>
			</configuration>
		</plugin>
		<plugin>
			<groupId>org.openjfx</groupId>
			<artifactId>javafx-maven-plugin</artifactId>
			<version>0.0.5</version>
		</plugin>
	</plugins>
</build>
			
		

But there are couple of weird situations here. The main class of the application is the class called Launcher.java. It launches the JavaFX window because the javafx main class is Application and it is not part of java, remember. It can't load on startup. So you need to run it from the Launcher.class main method. Also you can't use module-info.java file (that comes with JDK 9+) because OpenCV is creating problems with that module infrastructure. Also you can't use JDK 12 or above because OpenCV initialization (OpenCV.loadShared() method) causes exceptions. OpenCV + maven = problems.

This project doesn't have an executable jar file like the ant algorithm because JRE can't see the main method for some reason. It also gets around 80 megabytes because it inclued opencv libraries too. You can add maven plugins inside the pom file to create your own executable and deal with this problem yourself if you want. When you try to run the application, it finds 2 main classes. One of them is my own and the other one is some JavaFX css handler class that has a main method for some reason. You can ignore that one. If you can handle all these weird problems you can try applying "grayscale" + "edge detection" + "negative colors" with this specific order and get a filter like this.

Difficulties and deficits

It is time to be harsh on myself :) I have developed these codes like 10 years ago and updated them 5 years ago. The naming convention is inconsistent, just like ant algorithm. I have used translateX and translateY properties for interface components to replace them even though i have used VBox or HBox. The screen is being generated with code and lets you chose the number of operations. This is causing lots of lines of code. But on the other hand, the application knows how many operations can be fit into your screen.

The main problem in this kind of image processing application is probably video codecs. Because the opencv library is very efficient at matrix operations so most of the time, it is not causing much performance issues. The video or webcam feed would probably be less than 30 frames per second and fullhd resolution. So unless you are applying 10 different filters on top of each other, you wouldn't have much performance issues. I have developed a multithreaded effect in this application and it is horrendous compared to native OpenCV functions :)

But if you want to render a video and conduct image processing on it, than you would have to deal with codecs. I don't think OpenCV or any other library is mainly focused on being a video player. You can create a video with images with OpenCV but playing a video is complicated. If you are not dealing with videos and you only have webcams, you may not have much trouble.

The code has no comments :) The main class is more than 1K lines of code like ant algorithm. I wanted to make the interface flexible so that it can support as many as 20 operations to be applied. This made the operations code a little bit too coupled with the interface. Also i should have implemented some sort of design pattern. Because this feels like a mixture of command and chain of responsibility. I haven't implamented a functionality for stopping the video while playing so if you play a video and want to see something in it, you would have to restart it.

Download, have fun, learn

This sounded like a mobile game ad. In this post, i have introduced my image processing application. The mobile version of the application that uses OpenCV 3 for Android is in the play store. But it is not maintained like the desktop application. Image processing is used in face recognition, object detection and identification or even hiding information. There are probably more areas like augmented reality that can really utilize image processing. You can catch very subtle disturbances or nuances in images which would be almost impossible for your eyes to detect. My program is designed to be an experimental tool for learners. It is also a good and up-to-date example of JavaFX, OpenCV, Maven trio. See you at the next post :)