Refactor computation data handling to use filepaths (#126)

Changed the internal representation of algorithms and datasets within the service from byte slices to file paths, writing received data directly to temp files. This modification allows for handling potentially large data sets without the need to load them entirely into memory, improving the memory efficiency and scalability of the service. Additionally, it aligns the call signature of external algorithms with the new approach, updating documentation and examples accordingly. Updated the linear regression example for consistency with the new data handling process.

Resolves issues with memory bloat when processing large datasets.

Signed-off-by: SammyOina <sammyoina@gmail.com>
This commit is contained in:
Sammy Kerata Oina
2024-05-13 18:16:39 +03:00
committed by GitHub
parent 226704cf0d
commit c274521faf
3 changed files with 49 additions and 33 deletions
+3 -3
View File
@@ -6,8 +6,8 @@ import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
dataset = sys.argv[1]
iris = pd.read_csv(io.StringIO(dataset))
csv_file_path = sys.argv[2]
iris = pd.read_csv(csv_file_path)
# Droping the Species since we only need the measurements
X = iris.drop(['Species'], axis=1)
@@ -30,7 +30,7 @@ joblib.dump(log_reg, model_buffer)
model_bytes = model_buffer.getvalue()
# Define the path for the Unix domain socket
socket_path = sys.argv[2]
socket_path = sys.argv[1]
# Create a Unix domain socket client
client = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)