SAS Cloud Analytic Services: How To

Scale Down a CAS Server

Scaling down a CAS server to zero pods when not in use means that the CAS operator deletes worker pods and controller pods created by the operator on behalf of that particular CAS Deployment object. PVCs, service accounts, service objects, Ingress, and any other objects in cas-smp or cas-mpp overlays are not deleted.

Perform these steps to scale down your CAS server to zero pods:

1. Locate the CAS server that you want to scale down.
   kubectl -n name-of-namespace get casdeployments

   kubectl -n name-of-namespace get casdeployments

   An example output:

   NAME		AGE
   default		20h

2. To scale down a CAS server to zero pods:
   kubectl -n name-of-namespace patch casdeployment default --type=json -p='[{"op": "add", "path": "/spec/shutdown", "value": true}]'

   kubectl -n name-of-namespace patch casdeployment default --type=json -p='[{"op": "add", "path": "/spec/shutdown", "value": true}]'

3. Wait for all the pods to get deleted:
   kubectl -n name-of-namespace wait --for=delete -l casoperator.sas.com/server=default pods

   kubectl -n name-of-namespace wait --for=delete -l casoperator.sas.com/server=default pods

   The user might see any one of the following messages:

   • pod/sas-cas-server-default-controller condition met

     This message indicates that it waited until the pods were deleted, and it is now safe to scale back up.

   • error: no matching resources found

     This message indicates that the pods have been already deleted. It is now safe to scale back up.

   • error: timed out waiting for the condition on casdeployments/default

     This message indicates that the pods did not finish terminating after a short time-out period. A longer time-out value can be specified if necessary. For example, --timeout 120s.

Scale Up a CAS Server

To scale up your CAS server, set the shutdown variable to false.

kubectl -n name-of-namespace patch casdeployment default --type=json -p='[{"op": "add", "path": "/spec/shutdown", "value": false}]'

Restart a CAS Server

To restart a CAS server, you must scale down and scale up the CAS server. See Scale Down a CAS Server and Scale Up a CAS Server.

Delete the CAS Server Pods

You might need to delete the CAS server pods in your deployment at some point in time. For example, one or more of your CAS pods might become evicted or might be stuck pending waiting on a precondition.

1. Locate the CAS server that you want to delete.
   kubectl -n name-of-namespace get casdeployments

   kubectl -n name-of-namespace get casdeployments

   An example output:

   NAME		AGE
   default		20h

   The sample commands apply to the default instance of the server, whether it is a single-machine server (SMP) or a distributed server (MPP). If your deployment includes more than one server, specify the server instance name.

2. Delete the CAS server.
   kubectl -n name-of-namespace delete pods -l casoperator.sas.com/server=default

   kubectl -n name-of-namespace delete pods -l casoperator.sas.com/server=default

   Note: The CAS operator will restart the CAS server pods after they are deleted.
   TipTo delete all the CAS servers:

   kubectl -n name-of-namespace delete pods -l app.kubernetes.io/managed-by=sas-cas-operator

   kubectl -n name-of-namespace delete pods -l app.kubernetes.io/managed-by=sas-cas-operator

Convert an SMP CAS Server to an MPP CAS Server

To understand more about the SMP CAS server and the MPP CAS server, see What is the CAS Server, SMP, and MPP? in Getting Started with SAS Viya Platform Operations .

Perform these steps to convert an SMP CAS server to an MPP CAS server:

1. Perform the steps in Remove a CAS Server.
2. Perform the steps in Change the Number of Workers for MPP CAS in SAS Viya Platform: Deployment Guide.
3. Redeploy the software using the original deployment method.

   As appropriate, perform the steps in Deploy the Software in SAS Viya Platform: Deployment Guide.

4. After the CAS server has restarted, verify that your SMP CAS server is successfully converted to an MPP CAS server.
   kubectl -n name-of-namespace get pods | grep cas-server

   kubectl -n name-of-namespace get pods | grep cas-server

   Here is example output:

   sas-cas-server-default-controller   1/1  Running  0  28s
   sas-cas-server-default-worker-0     1/1  Running  3  107s
   sas-cas-server-default-worker-1     1/1  Running  3  107s

Add a Backup Controller

You can add a backup controller to your CAS server only if it is deployed in MPP mode.

To add a backup controller:

1. Perform the steps in Remove a CAS Server.
2. Perform the steps in Add a Backup Controller for MPP CAS in SAS Viya Platform: Deployment Guide.
3. As appropriate, perform the steps in Deploy the Software in SAS Viya Platform: Deployment Guide.

Change the Number of CAS Worker Nodes

You can add additional worker nodes to your CAS server only if it is deployed in MPP mode.

The administrator can choose to change the number of worker nodes temporarily or permanently depending on the needs of the deployment.

For temporary changes, use kubectl patch to alter the value of spec.workers. The administrator can make temporary changes based on the usage and the resources needed at that point in time.

kubectl -n name-of-namespace patch casdeployment name-of-casdeployment --type=json -p='[{"op": "add", "path": "/spec/workers", "value": number-of-worker-nodes}]'

If the administrator determines that the number of CAS worker nodes that were provisioned in the original deployment do not align with the typical CAS resource requirements, a permanent change to the number of CAS workers can be made by performing the instructions in Change the Number of Workers for MPP CAS in SAS Viya Platform: Deployment Guide.

Enable Table Rebalancing

Data tables can be automatically rebalanced depending on the following scenarios:

1. The administrator reduces the number of workers by changing spec.workers in the CASDeployment custom resource to a smaller value. For more information, see cas.SCALEDOWNMODE.

   All the data on loaded tables is rebalanced across the worker pods that are not marked to be terminated.

2. The administrator adds more workers by changing spec.workers in the CASDeployment custom resource to a larger value.
3. A worker pod fails and is successfully restarted by the operator within 60 seconds, which results in a new worker. However, the total number of workers is the same as when it was originally configured.
4. A worker pod fails and cannot be successfully restarted by the operator that results in a smaller number of workers than specified in spec.workers. The failure might have occurred due to an underlying Kubernetes node failure where there might be no appropriate nodes available in the cluster to start the replacement pod.

   In scenarios 2, 3, and 4, automatic table rebalancing depends on the values set for CAS_GLOBAL_TABLE_AUTO_BALANCE, CAS_SESSION_TABLE_AUTO_BALANCE environment variables, and the tableRedistUpPolicy option settings.

By default, automatic table rebalancing is disabled. This functionality can be enabled by setting the following global environment variables before deploying your CAS server. To set the variables, the cas-add-environment-variables.yaml transformer file needs to be modified. For more information, see Change the Number of Workers for MPP CAS in SAS Viya Platform: Deployment Guide.

• CAS_GLOBAL_TABLE_AUTO_BALANCE

  Automatic balancing of promoted tables is controlled by this environment variable. Setting this variable to true or 1 enables this functionality. This variable can also be set to a value of background.

  background: Each global table that is configured to be rebalanced is locked, and a copy of the table is created that includes data on worker pods that are added when the number of worker pods is increased. When the copy is completed, the original table is deleted and replaced with the copy. During this operation, an action can read the locked table, and actions that attempt to alter the locked table have to wait for the copy to be completed.

• CAS_SESSION_TABLE_AUTO_BALANCE

  Automatic balancing of session local tables is controlled by this environment variable. Setting this variable to true or 1 enables this functionality.

To decide whether a table should be rebalanced depends on these factors:

• Tables that are rebalanced are able to leverage all the workers when these tables are used as input for future actions.
• If large tables are required to be rebalanced, then it increases the amount of time required to increase the number of workers.
• Setting the tableRedistUpPolicy value on tables and CASLIBs, and the settings of the sessionTableRedistUpPolicy session option and the tableRedistUpPolicy server option controls the time needed to add nodes to the server, and controls the performance of actions using specific tables after new worker nodes are added to the server.

Add a CAS Server

You can create a new CAS server instance by running the create-cas-server.sh script that generates Custom Resources (CR).

Running this script generates all the artifacts that are necessary for deploying a new CAS server in the SAS Viya platform Kubernetes framework, bundled into one directory that can be added to the base kustomization.yaml file.

Perform these steps to add a new CAS server:

1. Navigate to the $deploy/sas-bases/examples/cas/create directory of your deployment assets. Run the create-cas-server.sh script and specify the instance name to label the server. (A complete list of options is displayed by running the script with the --help flag).

   The following command generates a top-level directory cas-instanceName that has CR, permstore, data, PersistentVolumeClaim (PVC) definitions, and other artifacts that are required for the new CAS server.

   ./create-cas-server.sh -i instanceName

   ./create-cas-server.sh -i instanceName

2. Copy the new directory to the $deploy/site-config directory of your deployment assets and reference this path in the resources section of the base kustomization.yaml file. This ensures that the new CAS server is included when the manifest is rebuilt.

   kustomization.yaml:

   ... resources: - site-config/cas-shared-instanceName ...

   ...
   resources:
   - site-config/cas-shared-instanceName
   ...

3. Redeploy the software using the original deployment method. As appropriate, follow the instructions in Deploy the Software in SAS Viya Platform: Deployment Guide. The required CAS pods and services are created for the new CAS server.

   Verify the new CAS server:

   kubectl -n name-of-namespace get pods | grep instanceName

   kubectl -n name-of-namespace get pods | grep instanceName

   Here is an example output:

   sas-cas-server-instanceName-controller	3/3	Running		0	1m

   Verify the PVC definitions:

   kubectl -n name-of-namespace get pvc | grep instanceName

   kubectl -n name-of-namespace get pvc | grep instanceName

   Here is an example output:

   NAME                            STATUS   VOLUME  CAPACITY   ACCESS MODES   STORAGECLASS   AGE
   cas-{{ INSTANCE }}-data          Bound    pvc     1Gi        RWX            nfs-client     1m
   cas-{{ INSTANCE }}-permstore     Bound    pvc     100Mi      RWX            nfs-client     1m

You can run the create-cas-server.sh script with more options. The following example creates a new directory called cas-sample and generates a new CAS CR with two worker nodes and a backup controller:

./create-cas-server.sh --instance sample --output . --workers 2 --backup 1

Remove a CAS Server

If you want to remove your SMP or MPP CAS server permanently, modify sas.cas.instance: delete configuration instance. For more information, see Edit Configuration Instances.

However, you can also delete your CAS deployment if you want to remove your SMP or MPP CAS server.

1. Locate the CAS server that you want to delete.
   kubectl -n name-of-namespace get casdeployments

   kubectl -n name-of-namespace get casdeployments

   Here is an example output:

   NAME         AGE
   default     22h
   

2. Delete the CAS deployment. It deletes all the running pods in your SMP or MPP CAS cluster.
   kubectl -n name-of-namespace delete casdeployment default

   kubectl -n name-of-namespace delete casdeployment default

Edit Configuration Instances

The configuration for the CAS server is specified in configuration instances. These configuration instances contain configuration options or Lua code that is executed when the CAS server starts.

Because you cannot access or edit these instances directly in a Kubernetes environment, SAS Environment Manager enables you to edit the configuration instances. See Edit Server Configuration Instances in SAS Environment Manager: User’s Guide for general information.

The CAS server has the following configuration instances:

• sas.cas.instance.config: config

  The SAS administrator can set environment variables or configuration options or both in the CAS server and sessions.

• sas.cas.instance.config: delete

  Deletes all configuration instances defined for the CAS deployment when the CAS server is shutdown or reset.

  When the CAS server is reset, the configuration instances are redefined with their default values. Make sure to back-up and re-apply any of the configuration instances that you want to keep. You might want to copy the contents to a file, reset the CAS server, copy back the contents from the file to configuration instances, and reset the CAS server again for the changes to take effect.

• sas.cas.instance.config: logconfig

  When you click on this instance, you can see an XML formatted log configuration file. The SAS administrator can set the level of logging. Make sure that the XML file is formatted correctly for the CAS server to start.

  If the XML file is formatted incorrectly and becomes invalid, then the default deployed logging configuration is used so that the CAS server starts.

• sas.cas.instance.config: sessionlogconfig

  When you click on this instance, you can see an XML formatted session log configuration file. It is recommended not to modify the level of session logging unless required.

• sas.cas.instance.config: settings

  The SAS administrator can set environment variables in the CAS server and sessions.

• sas.cas.instance.config: startup

  Start-up scripts are executed before a CAS session connects to the CAS server.

To edit the CAS server configuration instance:

1. Refer to the steps in Edit Server Configuration Instances in SAS Environment Manager: User’s Guide.
2. In the list of all services, select the service name that matches the name of your CAS deployment instance. In this example, the name of the service name is cas-shared-default. You can enter cas in the Filter field to quickly locate the service.
3. In the Configuration area, locate the configuration instances that are associated with the configuration that you want to modify. The instances are expanded by default. Collapse the instances to easily find the one that you want to edit.
   
4. Click beside the instance that you want to modify.
5. After you are finished, click Save.
   Note:

   • The CAS server should be restarted for the changes to take effect. See Restart a CAS Server.
   • Use Lua-formatted syntax for the following configuration instances: sas.cas.instance.config: config, sas.cas.instance.config: delete, and sas.cas.instance.config: startup.
   • Use bash-formatted syntax when you add environment variables to sas.cas.instance.config: settings configuration instance.

Here is an example to illustrate how to set an environment variable in a configuration instance:

1. Set the env.CASACTIONHISTORYLEN environment variable.
2. Click beside the sas.cas.instance.config: config instance.
   
3. After you are finished, click Save. Restart the CAS server for the changes to take effect.

If the CAS server is not running, the SAS administrator has an option to review and edit the configuration instances for that CAS deployment.

1. Refer Step 3. Using the drop-down list, choose Definitions.
2. Search for sas.cas.instance.config. Review and edit the configuration instances.

Set umask Value

In many circumstances, it might be desirable to control the permissions of files created from CAS sessions. To set permissions on files created from CAS sessions, umask values can be set. A SAS administrator can set the umask value in the sas.cas.instance.config: settings configuration instance of the CAS server.

To set the umask value:

1. All the CAS configuration instances are disabled by default, which means that processing is not enabled for any modifications to the configuration instances in SAS Environment Manager. To allow the SAS administrator to modify the scripts in the configuration instances, set SAS_ALLOW_ADMIN_SCRIPTS to TRUE in the sas-shared-config configMap. For more information, see the README file at $deploy/sas-bases/overlays/sas-programming-environment/README.md (Markdown format) or $deploy/sas-bases/docs/sas-programming-environment-configuration-tasks.htm (HTML format).
2. Edit the sas.cas.instance.config: settings configuration instance to set the umask value as required. See Edit Configuration Instances.

   Here is an example:

   Set the umask value, so that all files for all users have a permission of rw-r--r-- (owner:read and write; group:read; other:read):

   umask 022

   umask 022

Mount persistentVolumeClaims for the CAS Server

Data storage in containers is ephemeral. When a container is deleted, the stored data is lost. For durable storage, data should be maintained in persistent volumes outside the Kubernetes cluster. Data remains intact, regardless of whether the containers that the storage is connected to are terminated.

Using a hostPath volume for storage is supported, but it is a potential security risk. SAS recommends that you use hostPath volumes only if you are knowledgeable of those risks.

For instructions to connect data storage outside the cluster to your SAS Viya deployment, see Mount persistentVolumeClaims for the CAS Server in SAS Viya Platform: Deployment Guide.

Set HTTP Proxy Environment Variables for CAS Sessions

If your CAS host requires a forward HTTP proxy for requests to resources such as S3 or ADLS, set the HTTP_PROXY and HTTPS_PROXY environment variables in the sas.cas.instance.config: settings configuration instance in SAS Environment Manager. For more information, see Edit Configuration Instances.

if [[ $CAS_OPTIONS == *session* ]]; then
 export HTTP_PROXY=http://proxy.example.com/
 export HTTPS_PROXY=http://proxy.example.com/
fi

Add CAS Server to SAS Viya Programming-Only Deployment

To add the default CAS server to your existing SAS Viya Programming-Only deployment, perform the following steps:

1. Review instructions for resource requirements for your CAS server. See CAS Server Resources in System Requirements for the SAS Viya Platform.
2. Provision CAS NodePool nodes as required for your environment. See Plan the Workload Placement in SAS Viya Platform: Deployment Guide.
3. Add CAS default resources to the base kustomization.yaml file of your deployment.

   Uncomment or add the following lines to the resources: section of the base kustomization.yaml file:

   - sas-bases/overlays/cas-server - sas-bases/overlays/cas-server/auto-resources

   - sas-bases/overlays/cas-server 
   - sas-bases/overlays/cas-server/auto-resources 

   Uncomment or add the following line to the transformers: section of the base kustomization.yaml file:

   - sas-bases/overlays/cas-server/auto-resources/remove-resources.yaml

   - sas-bases/overlays/cas-server/auto-resources/remove-resources.yaml

4. Redeploy the software using the original deployment method chosen for your SAS Viya Programming-Only deployment. As appropriate, perform the steps in Deploy the Software in SAS Viya Platform: Deployment Guide.

Remove CAS Server from SAS Viya Programming-Only Deployment

To remove the default SAS Cloud Analytic Services (CAS) server from your existing SAS Viya Programming-Only deployment, perform the following steps:

1. Delete the existing CAS server. For instructions, see Remove a CAS Server.
2. Delete CAS default resources from the base kustomization.yaml file of your deployment.
   • Delete or comment out these lines from the resources: section of the base kustomization.yaml file :
     - sas-bases/overlays/cas-server - sas-bases/overlays/cas-server/auto-resources

     - sas-bases/overlays/cas-server 
     - sas-bases/overlays/cas-server/auto-resources 

   • Delete or comment out the following line from the transformers: section of the base kustomization.yaml file:
     - sas-bases/overlays/cas-server/auto-resources/remove-resources.yaml

     - sas-bases/overlays/cas-server/auto-resources/remove-resources.yaml

   IMPORTANT Removing existing CAS Server resources from a SAS Viya Programming-Only deployment removes all resources associated with that CAS Server, including the CAS data and CAS backup persistent volume claims. SAS recommends retaining the data by copying your CAS server backups to an alternative file system. For instructions, see Examples of Transferring Backup Data between the SAS Viya Platform Environment and the Local File System in SAS Viya Platform: Backup and Restore.
   Note: When the CAS server is removed, you can also remove other related CAS transformers, such as cas-auto-restart, as they are no longer necessary.
3. Redeploy the software using the original deployment method chosen for your SAS Viya Programming-Only deployment. As appropriate, perform the steps in Deploy the Software in SAS Viya Platform: Deployment Guide.