# Purpose
-KarmaNotes is an online database of college lecture notes. KarmaNotes empowers college students to participate in the free exchange of knowledge.
+KarmaWorld is an online database of college lecture notes. KarmaWorld
+empowers college students to participate in the free exchange of knowledge.
+
+# Naming
+
+The repository and the project are called KarmaWorld. One implementation
+of KarmaWorld, which is run by FinalsClub Foundation, is called
+[KarmaNotes](https://www.karmanotes.org/).
# Pre-Installation
`fabfile.py` (`{project_root}/fabfile.py`) and `README.md`
(`{project_root}/README.md`).
+## External Software Dependencies
+
+### pdf2htmlEX
+
+KarmaWorld uses [pdf2htmlEX](https://github.com/coolwanglu/pdf2htmlEX) as
+a dependency. pdf2htmlEX is used to convert uploaded PDF notes into HTML.
+
+An [outdated version of pdf2htmlEX](https://github.com/FinalsClub/pdf2htmlEX)
+is available which includes the
+[patch](https://github.com/FinalsClub/pdf2htmlEX/commit/3c19f6abd8d59d1b58cf254b7160b332b3f5b517)
+required for pdf2htmlEX to correctly work with KarmaWorld.
+
+Newer versions can be used by applying the patch by hand. It's a fairly
+simple two-line modification that can be done after installing
+pdf2htmlEX.
+
+### SSL Certificate
+
+If you wish to host your system publicly, you'll almost certainly want
+an SSL certificate signed by a proper authority.
+
+You may need to set the `SSL_REDIRECT` environment variable to `true` to
+make KarmaWorld redirect insecure connections to secure ones.
+
+Follow [Heroku's SSL setup](https://devcenter.heroku.com/articles/ssl-endpoint)
+to get SSL running on your server with Heroku.
+
## External Service Dependencies
-Notice: This software makes use of external third party services which require
+Notice: A number of services are required even if running the KarmaWorld web
+service [locally](#local). Some of the services are recommended, and some are
+completely optional even if running the web service on Heroku.
+
+This software makes use of external third party services which require
accounts to access the service APIs. Without these third parties available,
-this software may require considerable overhaul. These services have
-API keys, credentials, and other information that you must provide to KarmaNotes
-as environment variables. The best way to persist these environment variables is
-by using a `.env` file. Copy `.env.example` to `.env` and populate the fields as
-required.
+this software may require considerable overhaul. These services have API keys,
+credentials, and other information that you must provide to KarmaWorld
+as environment variables.
+
+The best way to persist these API keys in environment variables is by using a
+`.env` file. Copy `.env.example` to `.env` and populate the fields as required.
+
+Many of these services have free tiers and can be used without charge for
+development testing purposes.
+
+* Reminder
+ * Copy `.env.example` to `.env` and populate the environment variables there.
+* Required Services
+ * [Google Drive](#google-drive)
+ * [Filepicker](#filepicker)
+ * [PostgreSQL](#postgresql)
+ * [Celery](#celery-queue)
+* Optional but recommended
+ * [IndexDen](#indexden): enables searching through courses, notes, etc
+ * [Heroku](#heroku): the production environment used by karmanotes.org
+ * it might not be possible to run KarmaWorld on Heroku using a free
+ webapp.
+ * [Amazon S3](#s3-for-static-files): for static file hosting
+* Entirely optional (though used in the production environment)
+ * [Twitter](#twitter): share updates about new uploads
+ * [Amazon Mechanical Turk](#amazon-mechanical-turk): generate quizzes, flashcards, etc
+ * [Amazon CloudFront](#amazon-cloudfront-cdn)
+ * [Amazon S3](#s3-for-filepicker): store files uploaded to Filepicker
+ * Filepicker does not support S3 storage in its free tier
### Heroku
This project has chosen to use [Heroku](www.heroku.com) to host the Django and
See README.heroku for more information.
-#### pdf2htmlEX
-This project uses [pdf2htmlEX](https://github.com/coolwanglu/pdf2htmlEX) as
-a dependency. pdf2htmlEX is used to convert uploaded PDF notes into HTML. If
-using Heroku, the default [KarmaNotes Heroku buildpack](https://github.com/FinalsClub/heroku-buildpack-karmanotes)
+#### pdf2htmlEX on Heroku
+If using Heroku, the default
+[KarmaNotes Heroku buildpack](https://github.com/FinalsClub/heroku-buildpack-karmanotes)
will [include](https://github.com/FinalsClub/heroku-buildpack-karmanotes/blob/master/bin/steps/pdf2htmlex)
-an [outdated version of pdf2htmlEX](https://github.com/FinalsClub/pdf2htmlEX)
-which is [patched](https://github.com/FinalsClub/pdf2htmlEX/commit/3c19f6abd8d59d1b58cf254b7160b332b3f5b517).
+the [required version of pdf2htmlEX](#pdf2htmlex).
### Celery Queue
Celery uses the Apache Message Queueing Protocol for passing messages to its workers.
For production, we recommend using Heroku's CloudAMQP add-on, getting your own CloudAMQP account, or
running a queueing system on your own. The `CLOUDAMQP_URL` environment variable must be set correctly
-for KarmaNotes to be able to use Celery. The `CELERY_QUEUE_NAME` environment variable
+for KarmaWorld to be able to use Celery. The `CELERY_QUEUE_NAME` environment variable
must be set to the name of the queue you wish to use. Settings this to something unique
-allows multiple instances of KarmaNotes (or some other software) to share the same queueing server.
+allows multiple instances of KarmaWorld (or some other software) to share the same queueing server.
For development on localhost, `RabbitMQ` is the default for `djcelery` and is well supported. Ensure
`RabbitMQ` is installed for local development.
-### Postgresql RDBMS
+### PostgreSQL
PostgreSQL is not necessarily required; other RDBMS could probably be fit into
place. However, the code was largely written assuming PostgreSQL will be used.
The instructions for creating an [S3](http://aws.amazon.com/s3/) bucket may be
[found on Amazon.](http://docs.aws.amazon.com/AmazonS3/latest/gsg/CreatingABucket.html)
-Two, separate buckets will be needed in production: one for static file hosting
+Two, separate buckets may be used in production: one for static file hosting
and one as a communication bus with Filepicker.
+#### S3 for Filepicker
+
This software uses S3 to store files which are sent to or received
from Filepicker. Filepicker will need to know the S3 bucket name, access key,
and secret key.
bucket, because the software will upload files to the Filepicker S3 bucket
through Filepicker's API and it will link to or download files from the
Filepicker S3 bucket through Filepicker's URLs. This will be covered in the
-Filepicker section below.
+[Filepicker section](#filepicker).
+
+#### S3 for static files
This software uses S3 for hosting static files. The software will need to
update static files on the S3 bucket. As such, the software will need the
This may be done with a Google Apps account with administrative privileges, or ask
your business sysadmin.
-Follow [Google's instructions](https://developers.google.com/drive/delegation)
-to create a Google Drive service account.
+Follow [Google's instructions](https://developers.google.com/drive/web/auth/web-server)
+to create a Google Drive service account. If using Google Apps, it is worth
+looking at [these instructions](https://developers.google.com/drive/delegation).
+
+Populate the `GOOGLE_USER` environment variable with the email address of the
+user whose Google Drive will be accessed. This is typically your own email
+address.
+
+Google Drive used to use p12 files by default. Now a new-style JSON file is
+downloaded by default when creating new credentials. Until the code has been
+[updated](https://github.com/FinalsClub/karmaworld/issues/437) to use the
+new-style JSON file, make sure to click the `Generate a new P12 key` button.
+
+While on the Credentials page (with the `Generate a new P12 key` button
+visible), note the Service account Email address. It will have a format like
+`numbers-alphanumerics@developer.gserviceaccount.com`. Copy this value and
+paste it into the `GOOGLE_SERVICE_EMAIL` environment variable.
Convert the p12 file into a Base64 encoded string for the
`GOOGLE_SERVICE_KEY_BASE64` environment variable. There are many ways to do
with open('file.p12', 'r') as f:
print binascii.b2a_base64(f.read)
-Copy the contents of `client_secret_*.apps.googleusercontent.com.json` into the
-`GOOGLE_CLIENT_SECRETS` environment variable.
-
### Filepicker
-This software uses [Filepicker.io](https://www.inkfilepicker.com/) for uploading
+This software uses [Filepicker](https://www.filepicker.com/) for uploading
files. This requires an account with Filepicker.
Filepicker can use an additional third party file hosting site where it may
send uploaded files. This project, in production, uses Amazon S3 as the third
party. See the Amazon S3 section above for more information.
+In development, an S3 bucket will not be necessary. The Free Plan should
+suffice.
+
Create a new App with Web SDK and provide the Heroku App URL for the
Application's URL. You'll be given an API Key for the App. Paste this into the
`FILEPICKER_API_KEY` environment variable.
Find the 'App Security' button on the left hand side of the web site. Make sure
-'Use Security' is enabled. Generate a new secret key. Paste this key into the
+'Use Security' is enabled. Generate a new app secret. It might require
+reloading the page to see the new secret. Paste this secret into the
`FILEPICKER_SECRET` environment variable.
If you have an upgraded plan, you can configure Filepicker to have access to
supply the credentials for the user with access to the Filepicker S3 bucket.
### IndexDen
-KarmaNotes uses IndexDen to create a searchable index of all the notes in the
+KarmaWorld uses IndexDen to create a searchable index of all the notes in the
system. Create an free IndexDen account at
[their homepage](http://indexden.com/). You will be given a private URL that
accesses your IndexDen account. This URL is visible on your dashboard (you
Set the `INDEXDEN_PRIVATE_URL` environment variable to your private URL.
Set the `INDEXDEN_INDEX` environment variable to the name of the index you want
-to use for KarmaNotes. The index will be created automatically when KarmaNotes
+to use for KarmaWorld. The index will be created automatically when KarmaNotes
is run if it doesn't already exist. It may be created through the GUI if
desired.
environment variables `TWITTER_CONSUMER_KEY`, `TWITTER_CONSUMER_SECRET`,
`TWITTER_ACCESS_TOKEN_KEY`, `TWITTER_ACCESS_TOKEN_SECRET`.
-### SSL Certificate
-
-If you wish to host your system publicly, you'll need an SSL certificate
-signed by a proper authority.
-
-Follow [Heroku's SSL setup](https://devcenter.heroku.com/articles/ssl-endpoint)
-to get SSL running on your server.
-
-You may set the `SSL_REDIRECT` environment variable to `true` to make KarmaNotes
-redirect insecure connections to secure ones.
-
# Local
## Configuring foreman
-KarmaNotes is a Heroku application and makes use of a Procfie. To use the
-Procfile locally, we recommend using `foreman`. To install `foreman` and other
-Heroku tools, install the [Heroku toolbelt](https://toolbelt.heroku.com/).
+KarmaNotes runs on Heroku as a webapp and thus makes use of a Procfile. While
+not strictly necessary, KarmaWorld can use the same basic Procfile which is
+convenient and consistent.
+
+To use the Procfile locally, we recommend using `foreman`. To install `foreman`
+and other Heroku tools, install the
+[Heroku toolbelt](https://toolbelt.heroku.com/).
Ensure environment variables are available to `foreman` by copying
`.env.example` to `.env` and update those variables as appropriate for your
This project uses [pdf2htmlEX](https://github.com/coolwanglu/pdf2htmlEX) as
a dependency. pdf2htmlEX is used to convert uploaded PDF notes into HTML. It
-needs to be installed on the same system that KarmaNotes runs on.
+needs to be installed on the same system that KarmaWorld is running on.
### using their source
1. `source venv/bin/activate`
1. `pip install -r requirements.txt`
* on Debian systems, some packages are required for pip to succeed:
- * `apt-get install python-dev libpython-dev python-psycopg2 libmemcached-devlibffi-dev libssl-dev postgresql-server-dev-X.Y`
+ * `apt-get install python-dev libpython-dev python-psycopg2 libmemcached-dev libffi-dev libssl-dev postgresql-server-dev-X.Y`libxml2-dev libxslt-dev
1. `pip install -r requirements-dev.txt`
+
+## Configuration
+
+Make sure [External Service Dependencies](#external-service-dependencies) are
+satisfied. This includes running a local database and RabbitMQ instance as
+desired.
+
+ 1. configure `.env` as per [instructions](#external-service-dependencies)
1. `foreman run python manage.py syncdb --migrate --noinput`
1. `foreman run python manage.py createsuperuser`
1. `foreman run python manage.py fetch_usde_csv ./schools.csv`
* `fetching_usde_csv` requires `7zip` to be installed for processing compressed
archives. On Debian-based systems, this entails `apt-get install p7zip-full`
+If using `DJANGO_SETTINGS_MODULE='karmaworld.settings.dev'` in `.env`, static
+file hosting should be done by local files. `DEFAULT_FILE_STORAGE` should be
+set to `django.core.files.storage.FileSystemStorage`.
+
+If using `DJANGO_SETTINGS_MODULE='karmaworld.settings.prod'` in `.env`, static
+file hosting is done by `DEFAULT_FILE_STORAGE` defined in `.env`.
+
## Run
Make sure you are inside your virtual environment (`source venv/bin/activate`).
# Heroku Install
-KarmaNotes is a Heroku application. Download the [Heroku toolbelt](https://toolbelt.heroku.com/).
+KarmaNotes runs on Heroku as a webapp. This section addresses what was done
+for KarmaNotes so that other implementations of KarmaWorld can be run on
+Heroku.
+
+Before anything else, download the [Heroku toolbelt](https://toolbelt.heroku.com/).
-To run KarmaNotes on Heroku, do `heroku create` and `git push heroku master` as typical
+To run KarmaWorld on Heroku, do `heroku create` and `git push heroku master` as typical
for a Heroku application. Set your the variable `BUILDPACK_URL` to
`https://github.com/FinalsClub/heroku-buildpack-karmanotes` to use a buildpack
designed to support KarmaNotes.
projects / oweals / karmaworld.git / blobdiff