SorterBot Cloud Documentation¶
arm_commands module¶
Contains the higher level commands that control the robotic arm. These commands are not directly controlling the servo motors, they are doing higher level operations which consists of series of lower level steps.
-
class
arm_commands.ArmCommands(config_path)¶ Bases:
object-
close()¶ Closes the session: stops the camera in case it’s recording, moves the arm to starting position and neutralizes the servos.
-
infer_and_sort()¶ Controls the process of localizing objects and moving them to the containers on the Raspberry side. First it takes images for inference, sends them for processing, and after all of them were successfully processed, requests the session’s commands from the cloud service. After the commands are received, executes them one by one and returns the arm to the initial position.
-
record_training_video()¶ Records a video which later can be used to create a training dataset by utilizing sorterbot_labeltool. After the video is recorded, it will be uploaded to the appropriate s3 bucket.
-
reset_arm()¶ Instructs the arm to return to the start position.
-
send_image_for_processing(image_path, step, is_after)¶ Takes an image from disk, opens it and send the image bytes directly to the Cloud service. It creates a new connection for each image, where the image metadata is sent as headers of the initial HTTP handshake.
- Parameters
image_path (str) – Path of the image on disk to be uploaded and processed.
step (int) – Identifies the image, corresponds to the pulse width of servo 0 where the image was taken. Used to correctly place the log after upload was done.
is_after (bool) – Boolean representing is the current image recording session is for creating an overview stitched image after the objects have been moved to the containers.
- Returns
success (bool) – Boolean representing if image processing was successful.
step (int) – Identifies the image, corresponds to the pulse width of servo 0 where the image was taken. Used to correctly place the log after upload was done. It is passed through the function and used to report the execution results.
-
take_pictures(steps, is_after=False)¶ Takes pictures for inference. After the pictures are taken, they will be sent over WebSockets to Cloud service to start inference and locate objects of interest. This function will wait for all the images to be processed, and finally it will check if all of them were successfully processed.
- Parameters
steps (list of ints) – List containing pulse widths of servo 0 where images should be taken.
- Returns
all_success (bool) – Boolean indicating if all the images were successfully processed.
is_after (bool) – Boolean representing is the current image recording session is for creating an overview stitched image after the objects have been moved to the containers.
-
camera module¶
This module controls the Raspberry Pi’s camera, which is used to make training videos and take inference pictures.
logger module¶
Python Logger to format logs and send them to the Control Panel using HTTPHandler.
-
class
logger.Logger(config_path)¶ Bases:
object
magnet module¶
This module controls the magnet on the robotic arm which is used to pick up objects. I Grove Electromagnet is used, which has it’s own control electronic and can be turned on and off by supplying a logical high and logical low signal to the appropriate input.
main module¶
-
class
main.Main(heart_rate=3, is_dev=False)¶ Bases:
objectManages WebSocket connections to Cloud service and Control Panel and starts new sessions as instructions from the user arrives from the Control Panel. The connections are opened when this file is run and periodically checked with pings if they are still working. If a connection is not responding, it is automatically closed and reopened. If the WebSocket server is down, the client will try to reconnect periodically. At the end of every heartbeat, connection status is reported to the Control Panel.
- Parameters
heart_rate (int) – Length of interval in seconds in which the connection checks and status reports are executed.
is_dev (bool) – Controls whether the development or production config file should be loaded.
-
connect_cloud(cloud_host=None)¶ Opens a WebSockets connection to the Cloud service.
- Parameters
cloud_host (str) – Optional parameter to specify the host of the connection. If none is provided, the host saved to the config file will be used.
- Returns
connection_success – 0 or 1, representing if connecting to the Cloud service succeeded. Integer values are used instead of bool, because they are directly JSON serializable.
- Return type
int
-
async
connect_control()¶ Opens a WebSockets connection to the Control Panel using the address saved in arm_config.yaml. If the connection fails, it will keep trying until it succeeds.
-
async
get_cloud_host_and_connect()¶ Retrieves the latest address of the Cloud service from the Control Panel and attempts to connect using the new host. If the connection succeeds, saves the new host to the config file.
- Returns
connection_success – 0 or 1, representing if the attempted connection using the new host succeeded. Integer values are used instead of bool, because they are directly JSON serializable.
- Return type
int
-
async
heartbeat()¶ Heartbeat function, which is executed in a certain interval to check WebSocket connection statuses and set up connections as needed. First, it checks if there is an open and responsive connection to the Cloud service. If not, it will try to open a connection using the host address saved in arm_config.yaml. If the connection cannot be opened, retrieves the Cloud host address from the Control Panel and tries to establish a connection using that. If it succeeds, the new host will be saved to arm_config.yaml. Finally, the connection status is sent to the Control Panel. In case there is a healthy connection to the Cloud service and the user pressed the start button, a new session will be initiated. If there is no connection, the session will not be started even if the start button was pressed.
- Returns
should_start_session – 0 or 1, representing if the session should be started. Integer values are used instead of bool, because they are directly JSON serializable. If there is no connection to the Cloud service, the session cannot be started.
- Return type
int
-
load_config()¶ Loads the contents of the config file (arm_config.yaml)
servo_control module¶
Contains the lower level insturctions that are manipulating the servos directly. These functions are called from the high level counterpart, arm_control.
-
class
servo_control.ServoControl(servos=14, 15, 18, 24, 25, start_positions=1425, 500, 1800, 1780, 1150)¶ Bases:
object-
execute_command(cmd)¶ Executes a single command on a single servo. Since servos move as fast as they can to the given position, which results in a too fast and unstable movement, instead of immediatly giving the command to move to the final positions, a series of intermediate positions are generated and supplied to the servo in a certain frequency to slow and smooth the movement. Except when recording the dataset (where even periods between movements are important) a sine smoothing is applied to further slow the movement at the beginning and end of every command. In order the achieve this, first a trajectory is generated as a list of intermediate positions, then the trajectory is executed by sendin each position to the servo and pausing for a short time between the steps.
- Parameters
cmd (tuple) – Tuple containing the command to be executed. cmd[0] is the servo index (not pin number!), which retrieves the pin number from the servos parameter of the __init__ function. cmd[1] is the pulse width where the servo should move, and cmd[2] is optionally the desired speed of the movement. It has to be one of the keys from self.speeds. If nothing is supplied, defaults to “fast”. Used to slow down movement when recording training video.
-
execute_commands(commands, parallel=False)¶ Executes the commands supplied.
- Parameters
commands (tuple) – Each element in the tuple is one command, which is another tuple, including the polar coordinates the destination. The first element corrensponds to the pulse_width which servo0 needs to receive in order to move to the correct position, and the second element corrensponds to the distance in pixels from the top of the picture used for inference.
parallel (bool) – If this parameter is true, the commands are executed simultaneously, utilizing multi-threading. This results in faster and smoother movement, but not always applicable. For example after picking up an object, the arm need to move further up to avoid accidentally bumping into other objects when moving sideways.
-
init_arm_position(is_inference=False)¶ Instructs the arm to move to inital position for taking a video for training or for taking pictures for inference.
-
move_to_position(end_pos, is_container=False)¶ Instructs the arm to move to the supplied destination. On the first run, it fixes servo2 and servo3 to avoid unexpected movements. It always start with moving servo2 to position in order to avoid bouncing into objects. The variables servo_2_pos and servo_3_pos are functions of servo_1_pos and were determined by recording 7 datapoints of the servo angles for correct position for item pickup, then polynomial curves were fitted, which gave the constants used below.
- Parameters
end_pos (tuple) – A tuple containing the polar coordinates of the object. The first element corresponds to the pulse_width which servo0 needs to receive in order to move to the correct position, and the second element corrensponds to the distance in pixels from the top of the picture used for inference.
is_container (bool) – If it is true, the arm will move to a higher position, which is suitable for dropping off an object.
-
neutralize_servos()¶ Neutralizes servos, which means that it sends a 0 pulse width to each of them, which will make them release the shafts, so they become moveable with hand.
-
storage module¶
Handles uploads to AWS s3 and creation of local folder for files to be saved.
-
class
storage.Storage¶ Bases:
object-
create_next_session_folder()¶ Creates a folder for the inference images names sess_{datetime}.
- Returns
curr_sess_path – Absolute path of the current session folder.
- Return type
str
-
create_next_train_folder()¶ Creates a folder for the training videos, named as simple integers starting from 1. It will find the highest existing number and increment it to get the current folder name. If the highest number is an empty folder, that will be used instead of creating a new one.
- Returns
next_folder – Absolute path of the next folder for training videos.
- Return type
str
-
upload_file(bucket, path, arm_id='')¶ Uploads a file to s3.
- Parameters
bucket (str) – Bucket name on AWS s3.
path (str) – Path of the file to be uploaded.
arm_id (str) – Identifier of the arm.
-