Using C# to run Python Scripts with Machine Learning Models

Python Data Ecosystem is the most popular package of libraries and frameworks for Data Science projects using Machine Learning (ML) algorithms today. It includes more than 1000+ developed libraries and frameworks. A huge Python international community from all around the world is constantly maintaining existing and developing new libraries and frameworks in Python programming language.

Being able to apply Machine Learning algorithms to business decisions is very important for companies today. When and how companies should apply Machine Learning algorithms based on their existing software and hardware infrastructure? Suppose you work for a company where the main programming language implementation could be C++, C#, Java or any other compiled language except Python or R. A simple question could be: Can any of these main programming languages solve all their Data Science needs? If not, what do you do in this case when the company wants to keep their main development language? I would like to propose a very simple solution for this very common problem. It is very simple because with a couple lines of C# code, for example, I can run any Python script file with any developed Machine Learning model. It will allow me to input/output any required model parameters in C# too.

Image Classification Project

ml_model = MLPClassifier()
hyper_parameter_candidates = [{"hidden_layer_sizes":[(20), (50), (100)],
"max_iter":[500, 800, 1000],
"activation":["identity", "logistic", "tanh", "relu"],
"solver":["lbfgs", "sgd", "adam"]}]
scoring_parameter = "accuracy"
cv_fold = KFold(n_splits=5, shuffle=True, random_state=1)

In my previous blog Refactoring Python Code for Machine Learning Projects. Python “Spaghetti Code” Everywhere! I presented a generic function tune_hyperparameter_model() to create an optimized classification model using GridSearchCV() and RandomizedSearchCV() methods.

Using the print_searchcv_result() function shown in the Python class code below ImageFileConvertFlattenArrayClass(), the best score and hyperparameters were determined as:

mean:0.985 std:(+/-0.006) for {"activation":"identity", "hidden_layer_sizes":100, "max_iter":500, "solver":"sgd"}

The table shows good statistical results of 97.34% accuracy score with the image test dataset.

Using the Python Pickle library the classification model file was saved locally as image_classification.pkl. Now that we have the model created let’s find out how C# can call it and pass data to and from it using input/output any parameters.

Running Python Script File in C#

The code below represents a simple C# class to run a Python script file with any developed Machine Learning model. The public method ExecutePythonScript() does all the work. As you can see the Python script file is passed as argument for the StartInfo object process (Arguments = filePythonScript). The StandardOutput method returns the output text contained by the Python print() function. This function evaluates each expression in turn and writes the resulting object to standard output. The StandardError method returns any occurred errors from the Python script file. In the catch block, I get the exception message to keep the code consistent.

using System;
using System.Diagnostics;
namespace RunPythonScript
{
/// <summary>
/// Machine Learning C# - Python
/// </summary>
public class MLSharpPython : IMLSharpPython
{
public readonly string filePythonExePath;
/// <summary>
/// ML Sharp Python class constructor
/// </summary>
/// <param name="exePythonPath">Python EXE file path</param>
public MLSharpPython(string exePythonPath)
{
filePythonExePath = exePythonPath;
}
/// <summary>
/// Execute Python script file
/// </summary>
/// <param name="filePythonScript">Python script file and input parameter(s)</param>
/// <param name="standardError">Output standard error</param>
/// <returns>Output text result</returns>
public string ExecutePythonScript(string filePythonScript, out string standardError)
{
string outputText = string.Empty;
standardError = string.Empty;
try
{
using (Process process = new Process())
{
process.StartInfo = new ProcessStartInfo(filePythonExePath)
{
Arguments = filePythonScript,
UseShellExecute = false,
RedirectStandardOutput = true,
RedirectStandardError = true,
CreateNoWindow = true
};
process.Start();
outputText = process.StandardOutput.ReadToEnd();
outputText = outputText.Replace(Environment.NewLine, string.Empty);
standardError = process.StandardError.ReadToEnd();
process.WaitForExit();
}
}
catch (Exception ex)
{
string exceptionMessage = ex.Message;
}
return outputText;
}
}
}

The interface of the MLSharpPythonclass is defined below.

namespace RunPythonScript
{
public interface IMLSharpPython
{
string ExecutePythonScript(string filePythonScript, out string standardError);
}
}

Because a C# class was created, it is a best practice to also we implement a unit test class program for it. I often see class libraries created without implementing interfaces/unit test and the results are usually poor or sub optimized performance with no way of knowing why. That is why we implement unit tests as a standard practice. The code below implements a simple unit test for ExecutePythonScript() method. No a lot of explanation here, it’s standard C# unit test code.

using Microsoft.VisualStudio.TestTools.UnitTesting;namespace RunPythonScript.Tests
{
[TestClass()]
public class MLSharpPythonTests
{
private static string filePythonExePath = Properties.Settings.Default.FilePythonExePath;
private static string folderImagePath = Properties.Settings.Default.FolderImagePath;
private static string filePythonNamePath = Properties.Settings.Default.FilePythonNamePath;
private static string filePythonParameterName = Properties.Settings.Default.FilePythonParameterName;
[TestMethod()]
public void ExecutePythonScriptTest()
{
string standardError;
string expectedOutputText = "1";
string fileNamePythonExe = filePythonExePath;
MLSharpPython mlSharpPython = new MLSharpPython(fileNamePythonExe);
string imagePathName = folderImagePath + "Image_Test_Name.png";
string fileNameParameter = $"{filePythonNamePath} {filePythonParameterName} {imagePathName}";
string actualOutputText = mlSharpPython.ExecutePythonScript(fileNameParameter, out standardError);
Assert.AreEqual(expectedOutputText, actualOutputText);
}
}
}

Here is a passed test result after running.

Summary
Last Test Run Passed (Total Run Time 0:00:02.120212)
1 Test Passed

Now that I have developed the class and its interface/unit test, I can create a simple program to run a Python script file. Because we should not hardcode variables in any programming language, I created an app.config file shown below to set Python Exe file path, folder image path, Python script file path and name, and input parameter name.

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
<applicationSettings>
<RunPythonScript.Properties.Settings>
<setting name="FilePythonExePath" serializeAs="String">
<value>C:\FilePythonExePath\python.exe</value>
</setting>
<setting name="FolderImagePath" serializeAs="String">
<value>C:\FolderImagePath\</value>
</setting>
<setting name="FilePythonNamePath" serializeAs="String">
<value>C:\FilePythonNamePat\image_use_model_pkl_classification_csharp.py</value>
</setting>
<setting name="FilePythonParameterName" serializeAs="String">
<value>-image_path_name</value>
</setting>
</RunPythonScript.Properties.Settings>
</applicationSettings>
</configuration>

Let’s look at a C# console application to show how to run a Python script file using the MLSharpPython class. I believe I have commented it very well so anyone should be able to understand and use it in their C# company Data Science projects. As you can see from the code the output text of the ExecutePythonScript() method will determine if the passed test image file belongs to category 0 or 1.

using System;namespace RunPythonScript
{
public class Program
{
// Get config settings
private static string filePythonExePath = Properties.Settings.Default.FilePythonExePath;
private static string folderImagePath = Properties.Settings.Default.FolderImagePath;
private static string filePythonNamePath = Properties.Settings.Default.FilePythonNamePath;
private static string filePythonParameterName = Properties.Settings.Default.FilePythonParameterName;

static void Main(string[] args)
{
string outputText, standardError;

// Instantiate Machine Learning C# - Python class object
IMLSharpPython mlSharpPython = new MLSharpPython(filePythonExePath);
// Test image
string imagePathName = folderImagePath + "Image_Test_Name.png";
// Define Python script file and input parameter name
string fileNameParameter = $"{filePythonNamePath} {filePythonParameterName} {imagePathName}";
// Execute the python script file
outputText = mlSharpPython.ExecutePythonScript(fileNameParameter, out standardError);
if (string.IsNullOrEmpty(standardError))
{
switch (outputText.ToLower())
{
case "1":
Console.WriteLine("Image category 1");
break;
case "0":
Console.WriteLine("Image category 0");
break;
default:
Console.WriteLine(outputText);
break;
}
}
else
{
Console.WriteLine(standardError);
}
Console.ReadKey();
}
}
}

Below is an example of the program result for category 1 image.

Result: Image category 1

Python Script File

Let’s look at the Python code below and explain what it does. First of all, we’ll need three Python Data Ecosystem libraries: NumPy, OpenCV and Pickle (Python object serialization library). The class ImageFileConvertFlattenArrayClass includes the image_flatten_pixel_nparray_opencv() method to resize and flatten the images array from 2D to 1D. The generic print_exception_message() method is used to print the occurred exception messages.

To run this code the parameter -image_path_name needs to be added and passed as argument to the start main ‘__main__’. The main() function open, load and close the classifier model file image_classification.pkl. The predict method of the loaded classifier model determines if the passed image is category 0 or 1. This Python code has been commented very well for you to understand easily.

# python libraries
import os
import sys
import time
import traceback
import argparse

# python data ecosystem libraries
import numpy as np
import cv2
import pickle
import config
class ImageFileConvertFlattenArrayClass(object):
"""
image file convertor to flatten array class
"""

def __init__(self):
pass

def image_flatten_pixel_nparray_opencv(self, image_path_name, nparray_dimension = "1d", image_resize_factor=None):
"""
resize and flatten the image from 1d to 2d array
:param image_path_name: image path and file name
:param nparray_dimension: nparray dimension type
:param image_resize_factor: image resize (reshape) factor
"""
image_nparray = None
try:
image = cv2.imread(image_path_name, cv2.IMREAD_GRAYSCALE)
image_height, image_width = image.shape
if image_resize_factor is not None:
image_width_resize = int(image_width / image_resize_factor)
aspect_ratio = image_height / image_width
image_height_resize = int(aspect_ratio * image_width_resize)
image_dimension = (image_width_resize,
image_height_resize)
image = cv2.resize(image, image_dimension, interpolation=cv2.INTER_AREA)
image_width, image_height = image_width_resize, image_height_resize
image_nparray = np.fromstring(image.tobytes(), dtype=np.uint8)
if (nparray_dimension == "1d"):
image_nparray = image_nparray.reshape((1, image_width_resize * image_height_resize))
except:
exception_message = sys.exc_info()[0]
print("An error occurred. {}".format(exception_message))
return image_nparray

def print_searchcv_result(self, classifier_model):
"""
print grid or randomized search cv results: best score and best parameters
:param classifier_model: defined classifier model
:return none
"""
try:
print("Scores:")
means = classifier_model.cv_results_["mean_test_score"]
standard_deviations = classifier_model.cv_results_["std_test_score"]
for mean, standard_deviation, parameter in zip(means, standard_deviations, classifier_model.cv_results_["params"]):
mean = float("{0:0.3f}".format(mean))
standard_deviation = float("{0:0.3f}".format(standard_deviation * 2))
print("mean:{} (std:+-{}) for {}".format(mean, standard_deviation * 2, parameter))
print()
print("Best Score:")
print(float("{0:0.3f}".format(classifier_model.best_score_)))
print()
print("Best Parameters:")
print(classifier_model.best_params_)
print()
except:
exception_message = sys.exc_info()[0]
print("An error occurred. {}".format(exception_message))

def print_exception_message(self, message_orientation="horizontal"):
"""
print full exception message
:param message_orientation: horizontal or vertical
:return None
"""
try:
exc_type, exc_value, exc_tb = sys.exc_info()
file_name, line_number, procedure_name, line_code = traceback.extract_tb(exc_tb)[-1]
time_stamp = " [Time Stamp]: " + str(time.strftime("%Y-%m-%d %I:%M:%S %p"))
file_name = " [File Name]: " + str(file_name)
procedure_name = " [Procedure Name]: " + str(procedure_name)
error_message = " [Error Message]: " + str(exc_value)
error_type = " [Error Type]: " + str(exc_type)
line_number = " [Line Number]: " + str(line_number)
line_code = " [Line Code]: " + str(line_code)
if (message_orientation == "horizontal"):
print( "An error occurred:{};{};{};{};{};{};{}".format(time_stamp, file_name, procedure_name, error_message, error_type, line_number, line_code))
elif (message_orientation == "vertical"):
print( "An error occurred:\n{}\n{}\n{}\n{}\n{}\n{}\n{}".format(time_stamp, file_name, procedure_name, error_message, error_type, line_number, line_code))
except:
pass

def main(image_path_name):
"""
main function start program
:param image_path_name: image path and file name
"""
try:
# instantiate the object for image array flatten class
image_array_flatten_class = ImageFileConvertFlattenArrayClass()

# check is image path file name exists
if (os.path.exists(image_path_name) == False):
print("File {} not found.".format(image_path_name))
exit()

# resize and flatten the image from 2d to 1d array
nparray_dimension = "1d"
image_resize_factor = 10
image_1d_nparray = image_array_flatten_class.image_flatten_pixel_nparray_opencv(image_path_name, nparray_dimension, image_resize_factor)
# create the data frame
X_real = pd.DataFrame(image_1d_nparray)

# data frame normalization
X_min = config.X_MIN; X_max = config.X_MAX
X_real = (X_real.astype("float32") - X_min) / (X_max - X_min)
# open and close the mlp classifier pickle model
project_directory_path = os.path.dirname(os.path.realpath(__file__))
mlp_classifier_model_pkl = open(os.path.join(project_directory_path, "image_classification.pkl"), "rb")
mlp_classifier_model_file = pickle.load(mlp_classifier_model_pkl)
mlp_classifier_model_pkl.close()

# run the predict method and validate for image category 1 or 0
y_predict_file = mlp_classifier_model_file.predict(X_real)
if y_predict_file == 1:
print("1")
else:
print("0")
except:
image_array_flatten_class.print_exception_message()

# main top-level start program
if __name__ == '__main__':
arg_parse = argparse.ArgumentParser()
arg_parse.add_argument("-image_path_name")
arguments = arg_parse.parse_args()
image_path_name = arguments.image_path_name
main(image_path_name)

Deployment Requirements

Conclusion

I really believe that after this blog is published many .NET shop companies will use the power of Python Data Ecosystem to tackle their Data Science projects needs along with known Machine Learning frameworks like scikit-learn, TensorFlow, Caffe, PyTorch, Keras, Neon, etc. Feel free to contact me with your feedback and let me know how I can help!

Senior Software Engineer and Senior Data Analyst. I have been specialized in Data Analytics and Machine Learning using Python Data Ecosystem.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store