# Purpose
-KarmaNotes is an online database of college lecture notes. KarmaNotes empowers college students to participate in the free exchange of knowledge.
+KarmaWorld is an online database of college lecture notes. KarmaWorld
+empowers college students to participate in the free exchange of knowledge.
+
+# Naming
+
+The repository and the project are called KarmaWorld. One implementation
+of KarmaWorld, which is run by FinalsClub Foundation, is called
+[KarmaNotes](https://www.karmanotes.org/).
# Pre-Installation
`fabfile.py` (`{project_root}/fabfile.py`) and `README.md`
(`{project_root}/README.md`).
+## External Software Dependencies
+
+### pdf2htmlEX
+
+KarmaWorld uses [pdf2htmlEX](https://github.com/coolwanglu/pdf2htmlEX) as
+a dependency. pdf2htmlEX is used to convert uploaded PDF notes into HTML.
+
+An [outdated version of pdf2htmlEX](https://github.com/FinalsClub/pdf2htmlEX)
+is available which includes the
+[patch](https://github.com/FinalsClub/pdf2htmlEX/commit/3c19f6abd8d59d1b58cf254b7160b332b3f5b517)
+required for pdf2htmlEX to correctly work with KarmaWorld.
+
+Newer versions can be used by applying the patch by hand. It's a fairly
+simple two-line modification that can be done after installing
+pdf2htmlEX.
+
+### SSL Certificate
+
+If you wish to host your system publicly, you'll almost certainly want
+an SSL certificate signed by a proper authority.
+
+You may need to set the `SSL_REDIRECT` environment variable to `true` to
+make KarmaWorld redirect insecure connections to secure ones.
+
+Follow [Heroku's SSL setup](https://devcenter.heroku.com/articles/ssl-endpoint)
+to get SSL running on your server with Heroku.
+
## External Service Dependencies
-Notice: This software makes use of external third party services which require
+Notice: A number of services are required even if running the KarmaWorld web
+service [locally](#local). Some of the services are recommended, and some are
+completely optional even if running the web service on Heroku.
+
+This software makes use of external third party services which require
accounts to access the service APIs. Without these third parties available,
-this software may require considerable overhaul. These services have
-API keys, credentials, and other information that you must provide to KarmaNotes
-as environment variables. The best way to persist these environment variables is
-by using a `.env` file. Copy `.env.example` to `.env` and populate the fields as
-required.
+this software may require considerable overhaul. These services have API keys,
+credentials, and other information that you must provide to KarmaWorld
+as environment variables.
+
+The best way to persist these API keys in environment variables is by using a
+`.env` file. Copy `.env.example` to `.env` and populate the fields as required.
+
+Many of these services have free tiers and can be used without charge for
+development testing purposes.
+
+* Reminder
+ * Copy `.env.example` to `.env` and populate the environment variables there.
+* Required Services
+ * [Google Drive](#google-drive)
+ * [Filepicker](#filepicker)
+ * [PostgreSQL](#postgresql)
+ * [Celery](#celery-queue)
+* Optional but recommended
+ * [IndexDen](#indexden): enables searching through courses, notes, etc
+ * [Heroku](#heroku): the production environment used by karmanotes.org
+ * it might not be possible to run KarmaWorld on Heroku using a free
+ webapp.
+ * [Amazon S3](#s3-for-static-files): for static file hosting
+* Entirely optional (though used in the production environment)
+ * [Twitter](#twitter): share updates about new uploads
+ * [Amazon Mechanical Turk](#amazon-mechanical-turk): generate quizzes, flashcards, etc
+ * [Amazon CloudFront](#amazon-cloudfront-cdn)
+ * [Amazon S3](#s3-for-filepicker): store files uploaded to Filepicker
+ * Filepicker does not support S3 storage in its free tier
### Heroku
This project has chosen to use [Heroku](www.heroku.com) to host the Django and
See README.heroku for more information.
+#### pdf2htmlEX on Heroku
+If using Heroku, the default
+[KarmaNotes Heroku buildpack](https://github.com/FinalsClub/heroku-buildpack-karmanotes)
+will [include](https://github.com/FinalsClub/heroku-buildpack-karmanotes/blob/master/bin/steps/pdf2htmlex)
+the [required version of pdf2htmlEX](#pdf2htmlex).
+
+### Celery Queue
+Celery uses the Apache Message Queueing Protocol for passing messages to its workers.
+
+For production, we recommend using Heroku's CloudAMQP add-on, getting your own CloudAMQP account, or
+running a queueing system on your own. The `CLOUDAMQP_URL` environment variable must be set correctly
+for KarmaWorld to be able to use Celery. The `CELERY_QUEUE_NAME` environment variable
+must be set to the name of the queue you wish to use. Settings this to something unique
+allows multiple instances of KarmaWorld (or some other software) to share the same queueing server.
+
+For development on localhost, `RabbitMQ` is the default for `djcelery` and is well supported. Ensure
+`RabbitMQ` is installed for local development.
+
+### PostgreSQL
+
+PostgreSQL is not necessarily required; other RDBMS could probably be fit into
+place. However, the code was largely written assuming PostgreSQL will be used.
+Change to another system with the caveat that it might take some work.
+
+There are many cloud providers which provide PostgreSQL databases. Heroku has
+an add-on for providing a PostgreSQL database. Ensure something like this
+is made available and installed to the app.
+
+For local development, ensure a PostgreSQL is running on localhost or is
+otherwise accessible.
+
### Amazon S3
The instructions for creating an [S3](http://aws.amazon.com/s3/) bucket may be
[found on Amazon.](http://docs.aws.amazon.com/AmazonS3/latest/gsg/CreatingABucket.html)
-Two, separate buckets will be needed in production: one for static file hosting
+Two, separate buckets may be used in production: one for static file hosting
and one as a communication bus with Filepicker.
+#### S3 for Filepicker
+
This software uses S3 to store files which are sent to or received
from Filepicker. Filepicker will need to know the S3 bucket name, access key,
and secret key.
bucket, because the software will upload files to the Filepicker S3 bucket
through Filepicker's API and it will link to or download files from the
Filepicker S3 bucket through Filepicker's URLs. This will be covered in the
-Filepicker section below.
+[Filepicker section](#filepicker).
+
+#### S3 for static files
This software uses S3 for hosting static files. The software will need to
update static files on the S3 bucket. As such, the software will need the
assessible.
In the settings for the Cloudfront Distribution, copy the "Domain Name" from
-General settings as a host URL (`//foo.cloudfront.net/`) and paste the URL into
-both the `CLOUDFRONT_URL` environment variable and the `CLOUDFRONT_DOMAIN`
-environment_variable (used for the Boto `AWS_S3_CUSTOM_DOMAIN` setting, an
-explanation is [here](https://coderwall.com/p/8kjr3q)).
+General settings and set `CLOUDFRONT_DOMAIN` to it. For example, `abcdefghij.cloudfront.net`.
### Amazon Mechanical Turk
Mechanical turk is employed to generate human feedback from uploaded notes.
This service is helpful for generating flash cards and quizzes.
-
-This service might be optional and it might cause unexpected charges when
-deployed. If the required environment variable is not found, then no errors will occur and no mechanical turk tasks will be created, avoiding any unexpected
+This service is optional and it might cause unexpected charges when
+deployed. If the required environment variable is not found,
+then no errors will occur and no mechanical turk tasks will be created, avoiding any unexpected
costs.
The `MTURK_HOST` environment variable is almost certainly
This may be done with a Google Apps account with administrative privileges, or ask
your business sysadmin.
-Follow [Google's instructions](https://developers.google.com/drive/delegation)
-to create a Google Drive service account.
+Follow [Google's instructions](https://developers.google.com/drive/web/auth/web-server)
+to create a Google Drive service account. If using Google Apps, it is worth
+looking at [these instructions](https://developers.google.com/drive/delegation).
+
+Populate the `GOOGLE_USER` environment variable with the email address of the
+user whose Google Drive will be accessed. This is typically your own email
+address.
+
+Google Drive used to use p12 files by default. Now a new-style JSON file is
+downloaded by default when creating new credentials. Until the code has been
+[updated](https://github.com/FinalsClub/karmaworld/issues/437) to use the
+new-style JSON file, make sure to click the `Generate a new P12 key` button.
+
+While on the Credentials page (with the `Generate a new P12 key` button
+visible), note the Service account Email address. It will have a format like
+`numbers-alphanumerics@developer.gserviceaccount.com`. Copy this value and
+paste it into the `GOOGLE_SERVICE_EMAIL` environment variable.
Convert the p12 file into a Base64 encoded string for the
`GOOGLE_SERVICE_KEY_BASE64` environment variable. There are many ways to do
with open('file.p12', 'r') as f:
print binascii.b2a_base64(f.read)
-Copy the contents of `client_secret_*.apps.googleusercontent.com.json` into the
-`GOOGLE_CLIENT_SECRETS` environment variable.
-
### Filepicker
-This software uses [Filepicker.io](https://www.inkfilepicker.com/) for uploading
+This software uses [Filepicker](https://www.filepicker.com/) for uploading
files. This requires an account with Filepicker.
Filepicker can use an additional third party file hosting site where it may
send uploaded files. This project, in production, uses Amazon S3 as the third
party. See the Amazon S3 section above for more information.
+In development, an S3 bucket will not be necessary. The Free Plan should
+suffice.
+
Create a new App with Web SDK and provide the Heroku App URL for the
Application's URL. You'll be given an API Key for the App. Paste this into the
`FILEPICKER_API_KEY` environment variable.
Find the 'App Security' button on the left hand side of the web site. Make sure
-'Use Security' is enabled. Generate a new secret key. Paste this key into the
+'Use Security' is enabled. Generate a new app secret. It might require
+reloading the page to see the new secret. Paste this secret into the
`FILEPICKER_SECRET` environment variable.
If you have an upgraded plan, you can configure Filepicker to have access to
supply the credentials for the user with access to the Filepicker S3 bucket.
### IndexDen
-KarmaNotes uses IndexDen to create a searchable index of all the notes in the
+KarmaWorld uses IndexDen to create a searchable index of all the notes in the
system. Create an free IndexDen account at
[their homepage](http://indexden.com/). You will be given a private URL that
accesses your IndexDen account. This URL is visible on your dashboard (you
Set the `INDEXDEN_PRIVATE_URL` environment variable to your private URL.
Set the `INDEXDEN_INDEX` environment variable to the name of the index you want
-to use for KarmaNotes. The index will be created automatically when KarmaNotes
+to use for KarmaWorld. The index will be created automatically when KarmaNotes
is run if it doesn't already exist. It may be created through the GUI if
desired.
environment variables `TWITTER_CONSUMER_KEY`, `TWITTER_CONSUMER_SECRET`,
`TWITTER_ACCESS_TOKEN_KEY`, `TWITTER_ACCESS_TOKEN_SECRET`.
-### SSL Certificate
+# Local
+
+## Configuring foreman
+
+KarmaNotes runs on Heroku as a webapp and thus makes use of a Procfile. While
+not strictly necessary, KarmaWorld can use the same basic Procfile which is
+convenient and consistent.
+
+To use the Procfile locally, we recommend using `foreman`. To install `foreman`
+and other Heroku tools, install the
+[Heroku toolbelt](https://toolbelt.heroku.com/).
+
+Ensure environment variables are available to `foreman` by copying
+`.env.example` to `.env` and update those variables as appropriate for your
+local system.
+
+## pdf2htmlEX
+
+This project uses [pdf2htmlEX](https://github.com/coolwanglu/pdf2htmlEX) as
+a dependency. pdf2htmlEX is used to convert uploaded PDF notes into HTML. It
+needs to be installed on the same system that KarmaWorld is running on.
+
+### using their source
+
+See their instructions at
+[https://github.com/coolwanglu/pdf2htmlEX/wiki/Building](https://github.com/coolwanglu/pdf2htmlEX/wiki/Building).
+
+Make sure to [patch](https://github.com/FinalsClub/pdf2htmlEX/commit/3c19f6abd8d59d1b58cf254b7160b332b3f5b517)
+the source code to expose two variables.
+
+### using our fork
+
+You can use FinalsClub's [outdated version of pdf2htmlEX](https://github.com/FinalsClub/pdf2htmlEX).
+See their installation instructions above, but don't worry about patching.
+
+### using their PPA
-If you wish to host your system publicly, you'll need an SSL certificate
-signed by a proper authority.
+You can use [their upstream PPA](https://launchpad.net/~coolwanglu/+archive/ubuntu/pdf2htmlex).
-If you are working on local system for development, a self signed certificate
-will suffice. There are plenty of resources available for learning how to
-create one, so that will not be detailed here. Note that the Vagrant file will
-automatically generated a self signed certificate within the virtual machine.
+ apt-add-repository ppa:coolwanglu/pdf2htmlex
+ apt-get update
+ apt-get install pdf2htmlex
-The certificate should be installed using nginx.
+Then patch the javascript on your system by running this code in the shell.
-# Local Install
+ cat >> `dpkg -L pdf2htmlex | grep pdf2htmlEX.js` <<PDF2HTMLEXHACK
+ Viewer.prototype['rescale'] = Viewer.prototype.rescale;
+ Viewer.prototype['scroll_to'] = Viewer.prototype.scroll_to;
+ PDF2HTMLEXHACK
-KarmaNotes is a Heroku application. Download the [Heroku toolbelt](https://toolbelt.heroku.com/).
+## Install
-To run KarmaNotes locally, do `foreman start`. Before your first run, there are
-a few setup steps:
1. `virtualenv venv`
1. `source venv/bin/activate`
1. `pip install -r requirements.txt`
+ * on Debian systems, some packages are required for pip to succeed:
+ * `apt-get install python-dev libpython-dev python-psycopg2 libmemcached-dev libffi-dev libssl-dev postgresql-server-dev-X.Y`libxml2-dev libxslt-dev
1. `pip install -r requirements-dev.txt`
+
+## Configuration
+
+Make sure [External Service Dependencies](#external-service-dependencies) are
+satisfied. This includes running a local database and RabbitMQ instance as
+desired.
+
+ 1. configure `.env` as per [instructions](#external-service-dependencies)
1. `foreman run python manage.py syncdb --migrate --noinput`
1. `foreman run python manage.py createsuperuser`
1. `foreman run python manage.py fetch_usde_csv ./schools.csv`
- 1. `foreman run python manage.py import_usde _csv ./schools.csv`
+ 1. `foreman run python manage.py import_usde_csv ./schools.csv`
1. `foreman run python manage.py sanitize_usde_schools`
+* `fetch_usde_csv` downloads school records and stores them to `./schools.csv`. This file name
+ and location is arbitrary. As long as the same file is passed into `import_usde_csv` for
+ reading, everything should be fine.
+
+* `fetching_usde_csv` requires `7zip` to be installed for processing compressed
+ archives. On Debian-based systems, this entails `apt-get install p7zip-full`
+
+If using `DJANGO_SETTINGS_MODULE='karmaworld.settings.dev'` in `.env`, static
+file hosting should be done by local files. `DEFAULT_FILE_STORAGE` should be
+set to `django.core.files.storage.FileSystemStorage`.
+
+If using `DJANGO_SETTINGS_MODULE='karmaworld.settings.prod'` in `.env`, static
+file hosting is done by `DEFAULT_FILE_STORAGE` defined in `.env`.
+
+## Run
+
+Make sure you are inside your virtual environment (`source venv/bin/activate`).
+
+If the code has changed or this is the first run, make sure any modified static
+files get compressed with `foreman run python manage.py compress`. Static files
+then need to be uploaded correctly with `foreman run python manage.py
+collectstatic`.
+
+Run `foreman start`. `foreman` will load the `.env` file and manage running all
+processes in a way that is similar to that of Heroku. This allows better
+consistency with local, staging, and production deployments.
+
+To run web-only, but no celery or beat, run `foreman start web` to specify
+strictly the web worker.
+
+Press ctrl-C to kill foreman. Foreman will run Django's runserver command.
+If you wish to have more control over how this is done, you can do
+`foreman run python manage.py runserver <options>`. For running any other
+`manage.py` commands, you should also precede them with `foreman run` like just shown.
+This simply ensures that the environment variables from `.env` are present.
# Heroku Install
-KarmaNotes is a Heroku application. Download the [Heroku toolbelt](https://toolbelt.heroku.com/).
+KarmaNotes runs on Heroku as a webapp. This section addresses what was done
+for KarmaNotes so that other implementations of KarmaWorld can be run on
+Heroku.
+
+Before anything else, download the [Heroku toolbelt](https://toolbelt.heroku.com/).
-To run KarmaNotes on Heroku, do `heroku create` and `git push heroku master` as typical
-for a Heroku application. This will deploy KarmaNotes to Heroku with a supporting buildpack.
+To run KarmaWorld on Heroku, do `heroku create` and `git push heroku master` as typical
+for a Heroku application. Set your the variable `BUILDPACK_URL` to
+`https://github.com/FinalsClub/heroku-buildpack-karmanotes` to use a buildpack
+designed to support KarmaNotes.
You will need to import the US Department of Education's list of accredited schools.
1. Fetch USDE schools with
`heroku run python manage.py fetch_usde_csv ./schools.csv`
1. Upload the schools into the database with
- `heroku run python /manage.py import_usde _csv ./schools.csv`
+ `heroku run python /manage.py import_usde_csv ./schools.csv`
1. Clean up redundant information with
`heroku run python /manage.py sanitize_usde_schools`
projects / oweals / karmaworld.git / blobdiff