Dev-PlayBook

## SIRIUS Developer Playbook
---
### Before Start

1. Install google cloud SDK:

    https://cloud.google.com/sdk/

2. Initialize gcloud SDK and authenticate:

    `gcloud init`

3. Install Kubernetes:

    https://kubernetes.io/docs/tasks/tools/install-kubectl/

4. Authenticate kubectl:

    `gcloud container clusters get-credentials cluster-sirius`

---

### Monitor the current status

- `kubectl get pod`

    You should see two or more pods with status "Running". One called "mongo-0" is running the MongoDB container. The other called "sirius-dev-**********-*****", is running the SIRIUS server. These names are going to be used later.

    Note: The `pod` in Kubernetes has the same concept as a Docker container.

- `kubectl get service`

    Running services and their EXTERNAL-IP are shown here.

---

### Launch a terminal in the containers

- `kubectl exec -ti POD_NAME bash`

    `POD_NAME` is the name of the pod to log in. i.e.

    `kubectl exec -ti mongo-0 bash`

    Will open a bash terminal inside the `mongo-0` pod. You can then run the Mongo shell there. (MongoDB authentication needed)

    `kubectl exec -ti sirius-pod-***** bash`

    Replace the `sirius-pod-*****` with the actually running pod name, then you will get a terminal inside the sirius pod. You will see a terminal at `/app/sirius` which contains the source code. The Nginx config is located at `/etc/nginx/`.

    You can also edit the sirius source code here locally. But this is **NOT** recommended because you will lose all the changes when the pod is terminated. (triggered by deployment)

---

### Manually deploy SIRIUS to Kubernetes

1. You should have Docker installed locally.

    https://docs.docker.com/install/

2. Checkout a github branch or commit

    `git checkout your_branch`

3. Build the Docker image

    Make sure you're in the SIRIUS root folder

    `cd SIRIUS-backend`

    Build the image with remote name

    `docker build -t us.gcr.io/valis-194104/sirius:YOUR_TAG .`

    Here `YOUR_TAG` should be changed to a unique tag you want, for exapmle you can use BRANCH_DATE like `FIX_0828`.

4. Test the newly built image locally

    a(1). If you have a local MongoDB instance running, you can run the docker container

    `docker run -ti -p 5000:5000 --link some-mongo:mongo us.gcr.io/valis-194104/sirius:YOUR_TAG bash`

    a(2). Else, if you don't have a local MongoDB instance, you can port-forward the cloud MongoDB

    `kubectl port-forward mongo-0 27017`

    Then run the docker container

    `docker run -ti -p 5000:5000 us.gcr.io/valis-194104/sirius:YOUR_TAG bash`

    b. Now you have a terminal at `/app/sirius` running inside the docker container, you can run the tests by

    `python -m unittest discover`

    c. If all tests passed, you can also manually launch the flask server

    `flask run -h 0.0.0.0`

    d. When the server is running, your webpack frontend launched by `npm run dev` will be able to connect to `localhost:5000`

5. Deploy to Kubernetes cloud server

    ** CAREFUL: This operation will affect other developers/users who might be using the cloud server.

    a. Push the Docker image to Google Cloud container registry

    `gcloud docker -- push us.gcr.io/valis-194104/sirius:YOUR_TAG`

    Here `YOUR_TAG` is the unique tag you previous used when building the image.

    b. Patch the deployment with the new image

    `kubectl patch deployment sirius-dev -p '{"spec":{"template":{"spec":{"containers":[{"name":"sirius-dev","image":"us.gcr.io/valis-194104/sirius:YOUR_TAG"}]}}}}'`

    Here `YOUR_TAG` is the unique tag you previous used when building the image.

    c. Check the deployment is successful

    `kubectl get pod`

    In the results, you should see the `sirius-pod-***` has been updated, with a very short `AGE`. You might also catch the old pod that is being terminated.

---

### Manually restart cloud SIRIUS/Mongo server

The simplest way to remove an old pod and start a new one is deleting the pod

`kubectl delete pod sirius-dev-****`

Here `sirius-dev-****` is the running pod name. The deployment will immediately create a new pod.

Same for the MongoDB pod

`kubectl delete pod mongo-0`

Note: During the booting up of the new pod, the service will be unavailable, so this operation is not recommended. The better way to start a new pod is deploy an update.

---

### Rebuild the MongoDB Database

The rebuilding of the cloud database is made very simple by the python script `rebuild_mongo_database.py`, but this operation will take quite some time to finish and should be very carefully done. There are in general two ways to do this.

#### I. Delete then rebuild the current database

**IMPORTANT** During the rebuilding of MongoDB database, the queries will result in empty or wrong results, and they will take very long time until the rebuilding process finished with building the search index. Therefore, this is generally not recommended.

1. To prevent a deployement update from interupting the building process, we create a single pod

    `cd SIRIUS-backend/config/kubernetes/dev`

    `kubectl create -f single-pod.yaml`

    After this, you should see a new `sirius-pod` start running in `kubectl get pod` output.

2. Get a bash terminal on the new pod

    `kubectl exec -ti sirius-pod bash`

3. Set your MongoDB username and password with write permission

    `export MONGO_UNAME=USERNAME`

    `export MONGO_PWD=PASSWORD`

4. Goto the local cache folder

    `cd /cache`

    This folder is located on a local SSD of the node, which provides the best R/W performance. If the `gene_data_tmp` folder exists here, the downloaded data files inside it will be used to skip the downloading.

5. Run the rebuild script

    `/app/sirius/tools/rebuild_mongo_database`

    Note: Only 5 ENCODE bed files will be uploaded by default. To parse and upload the rest of the ENCODE bed files, you can go to `gene_data_tmp/ENCODE/` then run `/app/sirius/tools/automate_encode_upload.py -s 5 -e 2000`

6. Clean up

    Exit the `sirius-pod`, then delete it

    `kubectl delete pod sirius-pod`

#### II. Hot-swap the database

**Note:** The hot-swap is done by building a new database while the old one is up and running, then swap the SIRIUS backend to use the new database. This will allow the running server to be almost untouched, but require more disk space.

Step 1-4 is the same as above.

5. Change the source code to use another database

    Edit `/app/sirius/mongo/__init__.py`, find the two lines

    ```
    db = client.database
    #db = client.database1
    ```

    This means `database` is in use. Switch the `#` sign at start of line to the other one, e.g.

    ```
    #db = client.database
    db = client.database1
    ```

    Note: If `database1` is in use, you should switch back to use `database`.

    Save this change. This change will tell the rebuild script to use the new database.

6. Run the rebuild script in the `/cache` folder

    `/app/sirius/tools/rebuild_mongo_database`

    After this step, a new database (`database1`) is created. At this point, the old database (`database`) is still up and used by SIRIUS server.

7. Swap the SIRIUS server to use the new database

    a. Update the `sirius/mongo/__init__.py` in your local git repository the same way as above.

    b. Deploy this change. For dev server, you can simply push this change to the master branch, Circle-CI will run the tests then deploy the update.

    After this step, the SIRIUS server should be connected to the newly built database.

8. Clean up

    a. Delete the old database

    Inside `sirius-pod`, run command to drop the old database

    `python -c "from sirius.mongo import client; client.drop_database('OLD_DATABASE')"`

    **CAREFUL** `'OLD_DATABASE'` should be the name of the old database. (Don't drop the new one!)

    b. Delete the `sirius-pod`

    `kubectl delete pod sirius-pod`

---

## For production server

**IMPORTANT** The production server is serving our website http://valis.bio. The operations on the production server should be done with extra care.

The production server was setup to use another namespace in Kubernetes. To check the status

`kubectl get pod --namespace production`

All other `kubectl` commands also work with the option `--namespace production`.

#### Deployment to the production server

The deployment to the production server is not configured in Circle-CI. To manually update the SIRIUS production server, first go to the root SIRIUS folder, then run the script:

`SIRIUS-backend$ ./config/kubernetes/production/build_and_deploy.sh`