mirror of
https://github.com/ultravioletrs/cocos.git
synced 2026-08-07 07:14:50 +00:00
Refactor computation data handling to use filepaths (#126)
Changed the internal representation of algorithms and datasets within the service from byte slices to file paths, writing received data directly to temp files. This modification allows for handling potentially large data sets without the need to load them entirely into memory, improving the memory efficiency and scalability of the service. Additionally, it aligns the call signature of external algorithms with the new approach, updating documentation and examples accordingly. Updated the linear regression example for consistency with the new data handling process. Resolves issues with memory bloat when processing large datasets. Signed-off-by: SammyOina <sammyoina@gmail.com>
This commit is contained in:
committed by
GitHub
parent
226704cf0d
commit
c274521faf
@@ -6,8 +6,8 @@ import pandas as pd
|
||||
from sklearn.model_selection import train_test_split
|
||||
from sklearn.linear_model import LogisticRegression
|
||||
|
||||
dataset = sys.argv[1]
|
||||
iris = pd.read_csv(io.StringIO(dataset))
|
||||
csv_file_path = sys.argv[2]
|
||||
iris = pd.read_csv(csv_file_path)
|
||||
|
||||
# Droping the Species since we only need the measurements
|
||||
X = iris.drop(['Species'], axis=1)
|
||||
@@ -30,7 +30,7 @@ joblib.dump(log_reg, model_buffer)
|
||||
model_bytes = model_buffer.getvalue()
|
||||
|
||||
# Define the path for the Unix domain socket
|
||||
socket_path = sys.argv[2]
|
||||
socket_path = sys.argv[1]
|
||||
|
||||
# Create a Unix domain socket client
|
||||
client = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
|
||||
|
||||
Reference in New Issue
Block a user